mapreduce - Hadoop Reduce side what is really occurred? -


i'm reading "hadoop: definitive guide" , have questions.

in chapter 7 "how mapreduce works" on page 201, author said in reduce side

when [a] in-memory buffer reaches threshold size (controlled mapreduce.reduce.shuffle.merge.percent) or [b] reaches threshold number of map outputs (mapreduce.reduce.merge.inmem.threshold), merged , spilled disk.

my questions (4 questions) conditions a , b.

in condition a default values config hadoop 2 that

  1. will 0.462(0.66 * 0.70) of 1gb memory reducer full (462mb), merge , spill begin?

in condition b default values config that

  1. when 1000 pairs of key-value map output 1 mapper have been collected in buffer, merge , spill begin?

  2. in above question 1 mapper correct or more 1 mapper different machine?

in following paragraph, author said that

as copies accumulate on disk, background thread merges them larger, sorted files

  1. it correct intended purpose when spill file wants write disk before spill exist in disk merge current spill?

please me better understand happening in hadoop.

thanks in advanced

please see image
each mapper run in different machine
can explain precise meaning specified enter image description here

  1. will 0.462(0.66 * 0.70) of 1gb memory reducer full (462mb), merge , spill begin?
    yes. trigger merge , spill disk.
    default each reducer 1gb of heap space, of 70% mapreduce.reduce.shuffle.input.buffer.percent used merge , shuffle buffer, 30% might needed reducer code, , other task specific requirements. when 66% mapreduce.reduce.shuffle.merge.percent of merge , shuffle buffer full merging (and spill) triggred, make sure space if left process sorting , during sort other incoming files not waiting memory space.
    refer http://www.bigsynapse.com/mapreduce-internals


  1. when 1000 pairs of key-value map output 1 mapper have been collected in buffer, merge , spill begin?
    no. here map outputs may refer partitions of map output file. (i think). 1000 key value pair might small amount of memory.
    in slide 24 of presentation cloudera same refered segments

    mapreduce.reduce.merge.inmem.threshold segments accumulated (default 1000)


  1. in above question 1 mapper correct or more 1 mapper different machine?
    understanding, 1000 partitions can come different partitions. partitions of map output files bieng copied parallely different nodes.


  1. it correct intended purpose when spill file wants write disk before spill exist in disk merge current spill?
    no, (mapper output) partition file in memory merged other (mapper output) partition files in memory create spill file , written disk. might repeated several times buffer fills up. many files have been written disk. increase efficiency these files in disk merged larger file background process.

source:
hadoop definitive guide. 4th edition.
http://answers.mapr.com/questions/3952/reduce-merge-performance-issues.html
https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
http://www.bigsynapse.com/mapreduce-internals
http://answers.mapr.com/questions/3952/reduce-merge-performance-issues.html


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more

ruby on rails - Seeing duplicate requests handled with Unicorn -

recently upgraded our ec2 instances c3.large in order boost more unicorn workers handle increased traffic our site. (6xworkers per machine - 2 ec2 instances). when did this, started seeing duplicate records being created! looks somehow, maybe 2 unicorn workers attempting process same request? in nginx logs see 1 [post] request particular rails controller - yet 2 records being created in database. i'm hitting kind of race condition issue - have no idea how debug this. doesn't happen time, when it's frustrating well. noticed 2 database records within 5 seconds of each other. (and yes disable button when user clicks submit button). unicorn 4.8.2 ruby 1.9.3 rails 3.2.14 any great! thanks! although may not fix problem, upgrade ruby 2.0.0. there blog mentioning idea upgrade ruby 2.0.0 if using unicorn. for example: https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app specifically: if using ruby 1.9, ...
Read more