0

I created a cluster with two shards. Or so I thought. In my design, first server has mongos_router shard1 (3 nodes) and config servers. Second server only has shard 2 nodes. After I configured shards I enabled sharding for every database. And when I look the output of sh.status() I see the output below:

...
  {
    database: {
      _id: 'wins_emission',
      primary: 'shard1rs',
      partitioned: false,
      version: {
        uuid: UUID('aebf94cf-6069-41ba-9a91-f91a944071b1'),
        timestamp: Timestamp({ t: 1711952615, i: 3000 }),
        lastMod: 1
      }
    },
    collections: {}
  },
  {
    database: {
      _id: 'wins_healthcheck',
      primary: 'shard2rs',
      partitioned: false,
      version: {
        uuid: UUID('663cb5f7-b7b3-4f40-9f52-2c3d1969fb65'),
        timestamp: Timestamp({ t: 1711952305, i: 4 }),
        lastMod: 1
      }
    },
...

I understood this as databases will be distributed among shards. And I was expecting the data among nodes not to be same. For example, notifications table have 17.7k docs. And I'm expecting these docs to be shared among nodes. Like shard1-first node have 4k, shard1-second node have 4k etc. But it isn't working like that. Every node in every shard has the same amount 17.7k . I may be misunderstood this.

I tried sharding at collection level for notifications table. I created a hashed shard key. And then executed sh.shardCollection() command. And now my first shard has 4.7k docs among its own nodes whereas shard2 has 12.9k in itself. Now this made me think of these questions.

  1. Do I need to shard every collection to make use of a sharded cluster?
  2. Should I shard every collection or only the ones which will hold big data like logs.
  3. Why do all the nodes in a shard have the same amount of docs. Aren't they supposed to distribute the data in itself?

Any help is appreciated.

Edit 1: Output of the getShardDistribution of a specific collection;

{
  data: '382.25MiB',
  docs: 7006,
  chunks: 3,
  'estimated data per chunk': '127.41MiB',
  'estimated docs per chunk': 2335
}
---
Shard shard2rs at shard2rs/172.16.114.129:27020,172.16.114.129:27021,172.16.114.129:27022
{
  data: '579.78MiB',
  docs: 10578,
  chunks: 1,
  'estimated data per chunk': '579.78MiB',
  'estimated docs per chunk': 10578
}
---
Totals
{
  data: '962.03MiB',
  docs: 17584,
  chunks: 4,
  'Shard shard1rs': [
    '39.73 % data',
    '39.84 % docs in cluster',
    '55KiB avg obj size on shard'
  ],
  'Shard shard2rs': [
    '60.26 % data',
    '60.15 % docs in cluster',
    '56KiB avg obj size on shard'
  ]
}


1

1 Answer 1

0

MongoDB distributes data in chunks. The default chunk size is 128 MiBytes.

Thus, the collection data size must be greater than 128 MiBytes, otherwise it is not distributed.

Regarding your questions:

  1. Do I need to shard every collection to make use of a sharded cluster?
  2. Should I shard every collection or only the ones which will hold big data like logs.
  • No, there is no need to shard all collections. Typically you enable sharding only on the big collections.
  1. Why do all the nodes in a shard have the same amount of docs. Aren't they supposed to distribute the data in itself?

In earlier version of MongoDB the data was sharded to get evenly distributed number of documents/chunks. In newer release of MongoDB the data is sharded to get evenly distributed amount of data.

See this example in my sharded cluster:

{
  data: '238.38GiB',
  docs: 65898591,
  chunks: 28,
  'Shard shard_02': [
    '25.05 % data',
    '26.49 % docs in cluster',
    '3KiB avg obj size on shard'
  ],
  'Shard shard_04': [
    '24.97 % data',
    '26.85 % docs in cluster',
    '3KiB avg obj size on shard'
  ],
  'Shard shard_03': [
    '25 % data',
    '19.84 % docs in cluster',
    '4KiB avg obj size on shard'
  ],
  'Shard shard_01': [
    '24.96 % data',
    '26.81 % docs in cluster',
    '3KiB avg obj size on shard'
  ]
}
6
  • The collection has 1gb data. When I enabled sharding in collection level It migrated two chunks to shard2 and now shard2 has 12.9k docs whereas shard1 has 4.7k. Apr 2 at 10:00
  • What is the cardinality of your sharding key? Do the documents have similar size? Sharding is done to distribute the amount of data, not the number of documents. What is the output of db.collection.getShardDistribution()? Apr 2 at 10:59
  • I pasted the output in edit. And I dont get what you mean by cardinality. The sharding key is hashed if thats what you asked. Apr 2 at 13:22
  • "cardinality" means "number of different values" (related to total number of values) Apr 2 at 13:26
  • I created the shard key on _id so if I understood correctly it would be as many as the document count. Apr 2 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.