I created a sharded mongodb cluster with 2 shards. Then I created a sharded collection by range and inserted 10000 documents with random documents with random values for the sharded key using Java. For some reason right after the insertion, the chunks are evenly distributed among the shards, however the number of documents are not.
This is what I see right after the insertion of the 10000 documents (in addition to existing 2 documents)
Shard shard0000 at 127.0.0.1:27017
data : 2.65MiB docs : 4402 chunks : 7
estimated data per chunk : 387KiB
estimated docs per chunk : 628
Shard shard0001 at 127.0.0.1:27018
data : 6.02MiB docs : 10002 chunks : 8
estimated data per chunk : 770KiB
estimated docs per chunk : 1250
Totals
data : 8.67MiB docs : 14404 chunks : 15
Shard shard0000 contains 30.57% data, 30.56% docs in cluster, avg obj size on shard : 631B
Shard shard0001 contains 69.42% data, 69.43% docs in cluster, avg obj size on shard : 631B
First I don't understand why I see 14,404 documents instead of 10,000 that I inserted.
To my surprise, the next day, I come and run ```db.myCollection.getShardDistribution()``, and suddenly I see a completely different picture. The data suddenly became well distributed.
So I started experimenting and found out that if I run the following commands:
use config
db.settings.update(
{ _id: "balancer" },
{ $set: { activeWindow : { start : "11:06", stop : "11:07" } } },
{ upsert: true }
)
And wait for that time window to pass the data is suddenly making sense:
mongos> db.myCollection.getShardDistribution()
Shard shard0000 at 127.0.0.1:27017
data : 2.65MiB docs : 4402 chunks : 7
estimated data per chunk : 387KiB
estimated docs per chunk : 628
Shard shard0001 at 127.0.0.1:27018
data : 3.36MiB docs : 5600 chunks : 8
estimated data per chunk : 431KiB
estimated docs per chunk : 700
Totals
data : 6.02MiB docs : 10002 chunks : 15
Shard shard0000 contains 44.03% data, 44.01% docs in cluster, avg obj size on shard : 631B
Shard shard0001 contains 55.96% data, 55.98% docs in cluster, avg obj size on shard : 630B
In my Java code that inserts 10000 documents, all I do is generate 10,000 Document objects and run an insertMany(documents) in the end.
List<Document> documents = new ArrayList<>();
for (int i = 0; i < 10000; i++) {
Document document = new Document();
document.append("name", generateName())
...
documents.add(document);
}
collection.insertMany(documents);
So what is going on here?
Note: I set the max chunk size to 1MB just to experiment with sharding Thanks
Update:
A the key on which I'm sharding is a random string generated like this:
private static String generateName() {
StringBuilder name = new StringBuilder();
name.append(RandomStringUtils.randomAlphabetic(1).toUpperCase());
name.append(RandomStringUtils.randomAlphabetic(5, 10).toLowerCase());
return name.toString();
}
So for the large number of documents, the distribution should be fairly uniform.
