I had a 3 shard set up, and ran out of capacity, so I added 3 more shards. (Each shard is a replica set). But the data isn't spread across the cluster evenly. I have my chunkSize set to the standard 64mb:
mongos> db.settings.find( { _id:"chunksize" } )
{ "_id" : "chunksize", "value" : 64 }
I thought this meant that when a chunk hits 64mb, it splits into two equal chunks both of size 32mb. That's what is demonstrated here. Is that not correct?
Here is my sharding distribution:
mongos> db.accounts.getShardDistribution()
Shard rs_0 at rs_0/mongo_rs_0_member_1:27018,mongo_rs_0_member_2:27019,mongo_rs_0_member_3:27020
data : 137.62GiB docs : 41991598 chunks : 1882
estimated data per chunk : 74.88MiB
estimated docs per chunk : 22312
Shard rs_1 at rs_1/mongo_rs_1_member_1:27018,mongo_rs_1_member_2:27019,mongo_rs_1_member_3:27020
data : 135.2GiB docs : 41159069 chunks : 1882
estimated data per chunk : 73.56MiB
estimated docs per chunk : 21869
Shard rs_2 at rs_2/mongo_rs_2_member_1:27018,mongo_rs_2_member_2:27019,mongo_rs_2_member_3:27020
data : 219.92GiB docs : 69739096 chunks : 1882
estimated data per chunk : 119.66MiB
estimated docs per chunk : 37055
Shard rs_3 at rs_3/mongo_rs_3_member_1:27018,mongo_rs_3_member_2:27019,mongo_rs_3_member_3:27020
data : 101.52GiB docs : 30650628 chunks : 1882
estimated data per chunk : 55.23MiB
estimated docs per chunk : 16286
Shard rs_4 at rs_4/mongo_rs_4_member_1:27018,mongo_rs_4_member_2:27019,mongo_rs_4_member_3:27020
data : 103.38GiB docs : 31071379 chunks : 1883
estimated data per chunk : 56.22MiB
estimated docs per chunk : 16500
Shard rs_5 at rs_5/mongo_rs_5_member_1:27018,mongo_rs_5_member_2:27019,mongo_rs_5_member_3:27020
data : 101.1GiB docs : 30516395 chunks : 1881
estimated data per chunk : 55.04MiB
estimated docs per chunk : 16223
Totals
data : 798.77GiB docs : 245128165 chunks : 11292
Shard rs_0 contains 17.23% data, 17.13% docs in cluster, avg obj size on shard : 3KiB
Shard rs_1 contains 16.92% data, 16.79% docs in cluster, avg obj size on shard : 3KiB
Shard rs_2 contains 27.53% data, 28.45% docs in cluster, avg obj size on shard : 3KiB
Shard rs_3 contains 12.7% data, 12.5% docs in cluster, avg obj size on shard : 3KiB
Shard rs_4 contains 12.94% data, 12.67% docs in cluster, avg obj size on shard : 3KiB
Shard rs_5 contains 12.65% data, 12.44% docs in cluster, avg obj size on shard : 3KiB
What's up with this? How can the first 3 shards/replica sets have an average size greater than 64mb when that is set to be chunkSize? Rs_2 is 119mb! Rs_2 has 27.53% of the data when it should have 16.6%.
I have very high cardinality in my shardkey, and it is not in a monotonically increasing one.
What should I do here? I can manually find chunks that are large and split them, but that is a pain. Do I lower my chunkSize? Is there some service/call I need to run to do this automatically?