0

When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is but the docs indicate query performance doesn't scale well due to the scatter gather pattern used.

In Elasticsearch, similar behaviour occurs except the queries on those shards are executed concurrently. Is that the case with MongoDB as well? Or does Mongos scatter the queries to each of the shards and gather the results in sequence and that is why it's inefficient? I couldn't any detailed information anywhere on how this works internally

1

3 Answers 3

1

See mongos Broadcast Operations:

mongos instances broadcast queries to all shards for the collection unless the mongos can determine which shard or subset of shards stores this data.

After the mongos receives responses from all shards, it merges the data and returns the result document. The performance of a broadcast operation depends on the overall load of the cluster, as well as variables like network latency, individual shard load, and number of documents returned per shard. Whenever possible, favor operations that result in targeted operation over those that result in a broadcast operation.

Looks like, mongos queries the shards concurrently. At least term "broadcast" is a clear indication for that. The response time of your query is mainly determined by the slowest response from all the shards I would assume.

5
  • Thank you, WernFried! But what's concerning is "favour operations that result in targeted operation over broadcast operation" - does this mean our broadcast queries might still appear to be slow to the clients just because mongos didn't favour for some reason?
    – asgs
    Jul 13 at 23:28
  • It means, you should make a smart choice of your shard key. When you have a smart shard key, then the majority of queries include the shard key field. Using _id (no matter if hashed or not) is rather a poor choice, I guess. Jul 14 at 6:23
  • that's what I want to know why _id (hashed or not) is a poor choice. it gets the docs distributed evenly across the shards, reads are executed in parallel
    – asgs
    Jul 18 at 11:05
  • If the query has to run only on a single shard, then the overall load is just lower. The mongos does not have to wait for the slowest shard. _id would be a poor shard key, mainly because your typical query does not include the _id. Of course, if the majority of your queries include then _id then hashed _id would be a very good key. Jul 18 at 11:20
  • assuming you're querying docs on fields which are indexed, I don't see why a particular shard would be slower than others if the docs to scan on all those shards are fairly uniformly distributed. I understand about the overall load but the load gets divided among the shards which are there. So, instead of 1 shard doing the work for a given query, N shards do 1/Nth of work for the same query. I don't know if I'm wrong in this regard
    – asgs
    Jul 18 at 14:28
1

The answer is yes.

The scattered query is what you get when you use _id as the sharding key. The query is sent to all shards and result is "gathered" from them. As here is said

If a query does not include the shard key, the mongos must direct the query to all shards in the cluster. These scatter gather queries can be inefficient. On larger clusters, scatter gather queries are unfeasible for routine operations.

5
  • But my question is, do these shards execute such queries concurrently or one after the other as orchestrated by mongos?
    – asgs
    Jul 12 at 19:18
  • Yes, queries are running same time (parallel) at all shards, and results are combined into one.
    – JJussi
    Jul 14 at 4:29
  • thank you. if they are running in parallel, why is it not feasible on larger clusters?
    – asgs
    Jul 18 at 11:06
  • When query is scatter-gather, it eats resources from all shards even the shard don't have data for that query. Especially when single shard have more data (and indexes) what fits to that nodes memory. System needs to "swap" data between memory (where search is done) and disk. It's always better (in cluster) send query only to those nodes what have requested data. Not all of them.
    – JJussi
    Jul 19 at 12:43
  • if we use the hashed _id as the shard key, then pretty much every shard holds the results or at least exposes the possibility of holding the matching documents. I'm not sure if it's the data swap between RAM and Disk that is causing the problem here
    – asgs
    Jul 20 at 18:41
0

When we shard a collection by the MongoDB's _id field, we could achieve the most uniform distribution there is,

No, this statement may not be correct. In fact, it is not recommended to have the _id field as shard key, when you have the default ObjectId as the shard key. This is because, it is a monotonically changing value (because of the timestamp embedded in it), and a monotonically changing values like timestamps and dates don't make good shard keys. With monotonically increasing values, even data distribution is not possible.

In case, you are having a field other than ObjectId for _id for the shard key, the above note may not apply.

So, how to overcome this issue? There are some ways, and the common ones are the combining the _id field with another field to makeup the shard key (shard key can be made of more than one field). Another way is to use Hashed Sharding - which allows even distribution of data among the shards (even when used with monotonically changing fields).


... but the docs indicate query performance doesn't scale well due to the scatter gather pattern used.

Shard key is one of the most important aspects of a sharding solution. It is used for evenly distributing data among the shards. When you use a shard key as part of your query filter, the query will be fast and is a Targeted operation - the query is directed to one shard only. And, if your query filter doesn't include the shard key, then it will be Broadcast / Scatter-Gather operation - an inefficient and slow query and the query is directed to all the shards.

5
  • When we say _id as a shard key, we mean the hashed value of it. So, that's not a concern. But how scatter gather resulting in quering all shards is inefficient as against a query a specific targeted shard is not clearly documented anywhere. If the query filters are indexed, it shouldn't matter whether one shard queries the corresponding indexed keys and the docs in the disk in one go or multiple shards do their job in the 1/Nth fashion assuming the data distribution is even. The only overhead is in the mongos node (which has received the client request) to aggregate the results from all shards
    – asgs
    Aug 20 at 9:22
  • Even if a query filter has an indexed field(s), if its not of shard key, mongos sends the query to all shards. And, the matching documents are retrieved from all shards (or from one shard or from multiple shards) - hence an inefficient operation.
    – prasad_
    Aug 20 at 14:08
  • even if it's sent to all N shards, each shard will do only 1/Nth of the work involved to produce the result set as against one shard doing the full work. I don't see how N shards all working together is inefficient except for the fact that mongos needs to merge/aggregate the results from all shards before sending back to client
    – asgs
    Aug 23 at 11:17
  • 1
    In general, this broadcast operations may involve network, other ops on the shards, merging the results, etc. - hence its inefficient.
    – prasad_
    Aug 23 at 11:27
  • ok, got it. thank you, Prasad
    – asgs
    Aug 23 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.