duplicates - How to do de-duplication on records from AWS Kinesis Firehose to Redshift? -


i read document of official aws kinesis firehose doesn't mention how handle duplicated events. have experience on it? googled use elasticcache filtering, mean need use aws lambda encapsulate such filtering logic? there simple way firehose ingest data redshift , @ same time has "exactly once" semantics? lot!

you can have duplication on both sides of kinesis stream. might put same events twice stream, , might read event twice consumers.

the producers side can happen if try put event kinesis stream, reason not sure if written or not, , decide put again. consumer side can happen if getting batch of events , start processing them, , crash before managed checkpoint location, , next worker picking same batch of events kinesis stream, based on last checkpoint sequence-id.

before start solving problem, should evaluate how have such duplication , business impact of such duplications. not every system handling financial transactions can't tolerate duplication. nevertheless, if decide need have such de-duplication, common way solve use event-id , track if processed event-id already.

elasticcache redis place track event-id. every time pick event processing, check if have in hash table in redis, if find it, skip it, , if don't find it, add table (with ttl based on possible time window such duplication).

if choose use kinesis firehose (instead of kinesis streams), no longer have control on consumer application , can't implement process. therefore, either want run such de-duplication logic on producer side, switch use kinesis streams , run own code in lambda or kcl, or settle de-duplication functions in redshift (see below).

if not sensitive duplication, can use functions in redshift, such count distinct or last_value in window function.


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more

ruby on rails - Seeing duplicate requests handled with Unicorn -

recently upgraded our ec2 instances c3.large in order boost more unicorn workers handle increased traffic our site. (6xworkers per machine - 2 ec2 instances). when did this, started seeing duplicate records being created! looks somehow, maybe 2 unicorn workers attempting process same request? in nginx logs see 1 [post] request particular rails controller - yet 2 records being created in database. i'm hitting kind of race condition issue - have no idea how debug this. doesn't happen time, when it's frustrating well. noticed 2 database records within 5 seconds of each other. (and yes disable button when user clicks submit button). unicorn 4.8.2 ruby 1.9.3 rails 3.2.14 any great! thanks! although may not fix problem, upgrade ruby 2.0.0. there blog mentioning idea upgrade ruby 2.0.0 if using unicorn. for example: https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app specifically: if using ruby 1.9, ...
Read more