9

I have mongodb shard cluster, shard key is hashed. It has 2 shard replica sets. Each replica set has 2 machines.

I did an experiment by adding another 2 shard replica sets, and it starts to rebalance.

However, after a while I find out that chunk migration is rather to be slow. It takes 1 hour to move 1.4GB of data.

It makes me worry, it means I have to wait for 13 days to complete 500GB of chunk migration!

I am new on this stuff and I don't have a god feel whether it is slow, fast, or normal. But still, those number does not convince me.

Additional notes on experiment: - using m3 medium aws machines - no other process running, only chunk migration - default mongodb sharding installation with no further configuration - shardkey is using hashed at object id (_id) - max chunk size 64MB

10

Update: April 2018

This answer was correct at the time of the question, but things have moved on since then. Since version 3.4 parallelism has been introduced, and the ticket I referenced originally has been closed. For more information I cover some of the details in this more recent answer. I will leave the rest of the answer as-is because it remains a good reference for general issues/constraints as well as valid for anyone on an older version.

Original Answer

I give a full explanation of what happens with a chunk migration in the M202 Advanced course if you are interested. In general terms, let's just say that migrations are not very fast, even for empty chunks, because of the housekeeping being performed to make sure migrations work in an active system (these still happen even if nothing but balancing is happening).

Additionally, there is only one migration happening at a time on the entire cluster - there is no parallelism. So, despite the fact that you have two "full" nodes and two "empty" nodes, at any given time there is at most one migration happening (between the shard with the most chunks and the shard with the least). Hence, having added 2 shards gains you nothing in terms of balancing speed and just increases the number of chunks which have to be moved.

For the migrations themselves, the chunks are likely ~30MiB in size (depends on how you populated data, but generally this will be your average with the default max chunk size). You can run db.collection.getShardDistribution() for some of that information, and see my answer here for ways to get even more information about your chunks.

Since there is no other activity going on, for a migration to happen the target shard (one of the newly added shards) will need to read ~30MiB of data from the source shards (one of the original 2) and update the config servers to reflect the new chunk location once it is done. Moving 30MiB of data should not be much of a bottleneck for a normal system without load.

If it is slow, there are a number of possible reasons why that is the case, but the most common for a system that is not busy are:

  • Source Disk I/O - if the data is not in active memory when it is read, it must be paged in from disk
  • Network - if there is latency, rate limiting, packet loss etc. then the read may take quite a while
  • Target Disk I/O - the data and indexes have to be written to disk, lots of indexes can make this worse, but usually this is not a problem on a lightly loaded system
  • Problems with migrations causing aborts and failed migrations (issues with config servers, issues with deletes on primaries)
  • Replication lag - for migrations to replica sets, write concern w:2 or w:majority is used by default and requires up to date secondaries to satisfy it.

If the system was busy then memory contention, lock contention would usually be suspects here too.

To get more information about how long migrations are taking, if they are failing etc., take a look at the entries in your config.changelog:

// connect to mongos
use config
db.changelog.find()

As you have seen, and as I generally tell people when I do training/education, if you know you will need 4 shards, then it's usually better to start with 4 rather than ramp up. If you do, then you need to be aware that adding a shard can take a long time, and initially is a net negative on resources rather than a gain (see part II of my sharding pitfalls series for a more detailed discussion of that).

Finally, to track/upvote/comment on the feature request to improve the parallelism of chunk migrations, check out SERVER-4355

  • Thanks, this explain chunk migration mechanism a lot more than mongodb documentation. – rendybjunior Oct 31 '14 at 12:00
  • I'll join your course for sure. :) What do you think about the speed I mentioned before? Is it normal or slow? I know this question is relative based on many aspects. But I am asking for your own opinition – rendybjunior Oct 31 '14 at 12:29
  • It seems a little slow based on your description, but I would have to benchmark medium instances to be sure. Your current rate might be all they are capable of, or you might have one of the problems I mentioned in the answer. One control you can try is a manual chunk move - turn off the balancer and essentially do it yourself to see if there are any problems and what impact a move has on the source/target systems. You can find the relevant details on moveChunk here: docs.mongodb.org/manual/reference/method/sh.moveChunk – Adam C Oct 31 '14 at 13:28
  • Just to add that chunk mirgation has low priority on mongoDB and even on High Performance Systems can take some time if they are busy. – Antonios Nov 1 '14 at 20:20
  • @Antonis - not sure what you mean by priority, a chunk migration is a read from the source shard (same as any other read) and a write on the target shard (with the aforementioned write concern), there is no prioritisation of these operations versus others. They will be slow on busy systems, but not because of any inherent priority difference. – Adam C Nov 2 '14 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.