We have huge collections ( around 3.5 TB ) that is sharded with 2 shards on mongodb. Since the collection is growing and load is getting increased on existing shards we decided to add 2 new shards for this cluster. However we see data is filling up very slowly on new shards , while still keeping high on old shards ( after a week on one collection we have around 1.7% of data on new shards, while 48% on old once , and second collection is even less - 0.5% ). All data in both collections is having ttl expiring indexes of 2 months, however it looks like this mechanism is also not improving the process ( theoretically data should get expired in old shards and therefore the balance should become better )

Is there anything anything that can be done to speed up migration of data to new shards?

  • What is the output of sh.status() for this collection? Maybe try sh.enableAutoSplit in case you run MongoDB version 7. Or try configureCollectionBalancing Jan 22 at 7:18
  • The output says balancing true, The number of chunks for one collection is 15K for shards 1 and 2 , and approximately 1.5K for shards 3 and 4 For second the number of chunks for shards 1 and 2 are 63-65K and 1-1.1K for shards 3 and 4 Unfortunately i am running 4.4 , so both commands are not available Jan 22 at 13:17
  • That's really a lot. As mentioned already in another comment, you should consider Hashed Sharding. MongoDB version 4.4 will go end-of-life in one week. Jan 22 at 14:22
  • We do use hashed sharding, and it was working great with almost equal balance between 2 shards. T Jan 22 at 18:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.