0

We are using mongo-cxx-driver -3.1.1 . We have 1 router, 1 config server and 3 shards on cluster. (No replica sets). The shard key is the "_id" field and we execute all sorts of queries, some may be scatter gather, aggregate and some use the shard key. When we perform continuous inserts (in that process all the above queries may be executed), we see that mongo shard disk I/O reported by collectd for the 3rd shard is twice as much as for the other 2 shards. Shard 1 and 2 have more or less same disk I/O numbers. However, the third shard is consistently double. collectd doesn't report any other processes running either. Also, when we stop the stimulus (for continuous insert) we see that the disk I/O becomes 0. Hence, proving that there is nothing else running. Any inputs towards this will be helpful.

0

Problem is your sharding key (_id) what is not "hashed". Because _id is "linear" (first 4 bytes are unix timestamp) all inserts are done to "last" chunk (chunk which upper bound is $MaxKey).

When _id is used as sharding key (and you use ObjecID and not create you own unique _id) it should be sharded {"_id":"hashed"} and not {"_id":1}

Hashed distributes insert load to every shard.

| improve this answer | |
  • Thanks for the quick response. I guess I didn't provide all the information necessary. The "_id" field is over written with a unique value that our application creates and that is the shard key. And we do sh.shardCollection("database.collection", { _id: "hashed" }) This would create hashed shard key ? – Chitra Vaidyanathan Nov 7 '17 at 15:05
  • Yes, when sharding key is hashed it is distributed automatically evenly between shards so that there is no hot spots. If you look what chunks has been moved, is most of chuks moved away from 3rd to other shards? – JJussi Nov 7 '17 at 18:50
  • Any other inputs regarding the huge difference in disk I/O – Chitra Vaidyanathan Nov 9 '17 at 11:08
  • No, not really. As you wrote, somehow all inserts go to 3rd shard and that, of course, generates lots of load. The solution is to find out WHY inserts go there when they should not go there. Use sh.status(true) to check what chunks are at last shard and maybe you figure out why all (or most of) writings goes there. – JJussi Nov 9 '17 at 15:48
  • @ChitraVaidyanathan To expand the answer: if the chosen shard key is monotonically increasing, you end up with the described problem. It does not really matter wether it is an ObjectId or a counter and a shard key does not even have to be unique. – Markus W Mahlberg Nov 11 '17 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.