2 Adding more information about the sharding key
source | link

Update:

A the key on which I'm sharding is a random string generated like this:

 private static String generateName() {
        StringBuilder name = new StringBuilder();

        name.append(RandomStringUtils.randomAlphabetic(1).toUpperCase());
        name.append(RandomStringUtils.randomAlphabetic(5, 10).toLowerCase());

        return name.toString();
    }

So for the large number of documents, the distribution should be fairly uniform.

Update:

A the key on which I'm sharding is a random string generated like this:

 private static String generateName() {
        StringBuilder name = new StringBuilder();

        name.append(RandomStringUtils.randomAlphabetic(1).toUpperCase());
        name.append(RandomStringUtils.randomAlphabetic(5, 10).toLowerCase());

        return name.toString();
    }

So for the large number of documents, the distribution should be fairly uniform.

1
source | link

MongoDB uneven document sharding - and Migration Windows

I created a sharded mongodb cluster with 2 shards. Then I created a sharded collection by range and inserted 10000 documents with random documents with random values for the sharded key using Java. For some reason right after the insertion, the chunks are evenly distributed among the shards, however the number of documents are not.

This is what I see right after the insertion of the 10000 documents (in addition to existing 2 documents)


Shard shard0000 at 127.0.0.1:27017
 data : 2.65MiB docs : 4402 chunks : 7
 estimated data per chunk : 387KiB
 estimated docs per chunk : 628

Shard shard0001 at 127.0.0.1:27018
 data : 6.02MiB docs : 10002 chunks : 8
 estimated data per chunk : 770KiB
 estimated docs per chunk : 1250

Totals
 data : 8.67MiB docs : 14404 chunks : 15
 Shard shard0000 contains 30.57% data, 30.56% docs in cluster, avg obj size on shard : 631B
 Shard shard0001 contains 69.42% data, 69.43% docs in cluster, avg obj size on shard : 631B

First I don't understand why I see 14,404 documents instead of 10,000 that I inserted.

To my surprise, the next day, I come and run ```db.myCollection.getShardDistribution()``, and suddenly I see a completely different picture. The data suddenly became well distributed.

So I started experimenting and found out that if I run the following commands:

use config 
db.settings.update(
   { _id: "balancer" },
   { $set: { activeWindow : { start : "11:06", stop : "11:07" } } },
   { upsert: true }
)

And wait for that time window to pass the data is suddenly making sense:

mongos> db.myCollection.getShardDistribution()

Shard shard0000 at 127.0.0.1:27017
 data : 2.65MiB docs : 4402 chunks : 7
 estimated data per chunk : 387KiB
 estimated docs per chunk : 628

Shard shard0001 at 127.0.0.1:27018
 data : 3.36MiB docs : 5600 chunks : 8
 estimated data per chunk : 431KiB
 estimated docs per chunk : 700

Totals
 data : 6.02MiB docs : 10002 chunks : 15
 Shard shard0000 contains 44.03% data, 44.01% docs in cluster, avg obj size on shard : 631B
 Shard shard0001 contains 55.96% data, 55.98% docs in cluster, avg obj size on shard : 630B

In my Java code that inserts 10000 documents, all I do is generate 10,000 Document objects and run an insertMany(documents) in the end.

  List<Document> documents = new ArrayList<>();

  for (int i = 0; i < 10000; i++) {
     Document document = new Document();
     document.append("name", generateName())
                    ...
     documents.add(document);
  }

  collection.insertMany(documents);

So what is going on here?

Note: I set the max chunk size to 1MB just to experiment with sharding Thanks