Why MongoDB db.col.count() shows more documents than was inserted -


using java driver mongodb trying insert 25,637,015 documents mongodb cluster. documents retrieved sql server database , inserted empty mongodb sharded collection (called col) in multithreaded fashion (8 concurrent threads). process took 2 hours. interesting , puzzling went on over 6(!) hours after program has finished.

firstly, hard drives in cluster node computers continued spin crazy. secondly, , more importantly, db.col.count() ran less second interval continued render different results:

mongos> db.col.count()  25694898 mongos> db.col.count() 25694917 mongos> db.col.count() 25695154 mongos> db.col.count() 25695207 mongos> db.col.count() 25695422 mongos> db.col.count() 25695493 mongos> db.col.count() 25696024 mongos> db.col.count() 25696130 mongos> db.col.count() 25698565 mongos> db.col.count() 25695145

what more intriguing these counters while going , down greater number of inserted documents: 25,637,015. had been smaller speculate documents went sort of queue , being processes, greater?!

like said after 6 hours stabilized: hard drives stopped spinning , mongos> db.col.count() has rendered correct number: 25637015.

if of importance. have 2 replica sets in sharded cluster. each replica set has 2 data nodes , 1 arbiter node. run 3 config servers. , 3 mongos. spread between 4 centos boxes (virtual) running on windows hosts. source sql server on yet physical machine. balancer not disabled duration of insert or anytime after. mongodb version 2.2.6 64 bit.

any idea mongodb doing 6 hours after java program has finished inserting? why count high?

thank

for of drivers, mongodb uses memory enhance write performance. insertion first goes memory , journal, returns @ once. moment data not on disk yet. more information, have @ write concern section of mongodb manual. that's why collection keeps growing.

as count returns more accurate number issue, there's jira issue it. see if answers question. unfortunately it's not fixed yet.

edit:
time spent, it's hard sure. depends on hardware, disk. helpful run mongostat , mongotop , see what's going on. once know if insertion still running, you'll know if count result makes sense. here found related jira issue explaining count operation in sharded clusters. may lead situation. however, happens when the server migrating. before going further, please let me know how sharded cluster built. what's shard key?


Comments

Post a Comment

Popular posts from this blog

get url and add instance to a model with prefilled foreign key :django admin -

Image
i have page in each contest shown in row. each row contains contest id , link other model names have foreign key contest. news model: class news(models.model): title = models.charfield(max_length=50, null=true) description = models.textfield() time = models.timefield( default= datetime.datetime.now().time()) contest = models.foreignkey(contest) i want use admin adding , changing these models. want when user clicks on 'news' has foreign key contest id=1, shows list of news value foreign key. use {% instance in instances %} <tr class="{% cycle 'row1' 'row2' %}"> <td>{{ instance.title }}</td> <td>{{ instance.starttime }}</td> <td>{{ instance.timespan }}</td> <td>{{ instance.date }}</td> <td><a href="/adm...
Read more

css - Make div keyboard-scrollable in jQuery Mobile? -

from can tell, although jquery-mobile-powered pages can contain divs overflow set scroll or auto, , these divs can scrolled one-screen bar or mouse wheel, cannot scrolled using arrow keys, page-up/page-down, or home/end. instead, official "page" div (with data-role="page") absorbs input. perhaps other divs can't acquire focus, i'm not sure. is there way around this? edit: jsfiddle of simple example: https://jsfiddle.net/qogz0shx/ <script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquerymobile/1.4.5/jquery.mobile.min.js"> </script> <link rel="stylesheet" href="https://code.jquery.com/mobile/1.4.5/jquery.mobile-1.4.5.css"/> <style> #outer { overflow:scroll; height: 50vh; width: 50vw; } #inner { height: 500vh; width: 500vw; } ...
Read more

ruby on rails - Seeing duplicate requests handled with Unicorn -

recently upgraded our ec2 instances c3.large in order boost more unicorn workers handle increased traffic our site. (6xworkers per machine - 2 ec2 instances). when did this, started seeing duplicate records being created! looks somehow, maybe 2 unicorn workers attempting process same request? in nginx logs see 1 [post] request particular rails controller - yet 2 records being created in database. i'm hitting kind of race condition issue - have no idea how debug this. doesn't happen time, when it's frustrating well. noticed 2 database records within 5 seconds of each other. (and yes disable button when user clicks submit button). unicorn 4.8.2 ruby 1.9.3 rails 3.2.14 any great! thanks! although may not fix problem, upgrade ruby 2.0.0. there blog mentioning idea upgrade ruby 2.0.0 if using unicorn. for example: https://www.digitalocean.com/community/articles/how-to-optimize-unicorn-workers-in-a-ruby-on-rails-app specifically: if using ruby 1.9, ...
Read more