How to speed up MongoDB chunk moving between shards

Question

On a MongoDB 3.6 in a replica set and WiredTiger engine we have a large collection with 7.7TB of data. We started to see severe performance issues (extreme CPU usage and slowdown of data replication) which we somehow solved by temporary using a new empty database. The data is basically usage logs and DB pattern is that mostly there are inserts (of new usage data) and occasionally data reads for certain users or reports.

To scale the system and solve performance issues in the long run, we now created two shards where the idea is to have a fast shard (with data of recent 2 months) and a slow shard (with all historical data). Fast shard run on n1-standard-8 (8 vCPUs, 30 GB memory) machines with SSD disks, while slow shard is on n1-highmem-2 (2 vCPUs, 13 GB memory) machines with rotational disks.

The database is not under any load (there are no new usage log inserts) and basically does nothing. The sharding process has started, but the speed of moving chunks is slow and will take around 40 days to move all the chunks. There are 200000 chunks left to be moved from fast to slow shard with average chunk size 32MB and it takes about 18 seconds to move one chunk.

What can be done to speed up this chunk migration process?

UPDATE

I have learned that if source shard is on SSD disks, then the chunk moving has almost double speed.

Answer 1

Things like how many shards we have, which database are sharded, and the configuration setting of your shard cluster. But one of the most important pieces of information that the config servers hold, is the mapping of the chunks to shards.

By default MongoDB takes ChunkSize=64 MB

By default , MongoDB takes 64 megabytes as the default chunk size. That means that if a chunk is about 64 megabytes, or within 64 megatons range , it will be split.

1MB<=chunkSize<=1024MB

We can define a chunk size between the values of one megabyte and 1024 and one gigabyte. The chunk size is configurable during run time. So if we decide to change a chunk size, we can easily do so. But before we go in changing your chunk size.

for further your ref here and here

Answer 2

At the end I found another way to speed up chunk moving. Here is the procedure that I tried on a smaller part of the dataset (80 million records) and the estimate is that in that way it will take about 16 days for entire migration.

If you try this, please make sure that you have a backup in place in case something goes wrong.

Here is the procedure:

1. Count the records to move (on shard rs0)

db.use.find({$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]}).count()

2. Dump the sample on shard rs0

mongodump --host <rs0-host> --db=db --collection=use --archive --gzip --query '{$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]}' > dump.gz

3. Delete the data on shard rs0 (connect directly to rs0 mongod, not through mongos)

db.use.deleteMany({$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]})

4. Move chunks (connect to mongos)

use config
sh.stopBalancer()
var lower=ObjectId("58B60F00e4b03547ad945a8a");
var upper=ObjectId("58BCA680e4b03547ad945a8a");
var query={shard: "rs0", ns: "db.use", "max._id":{$gte: lower}, "min._id": {$lte: upper}};
var cursor=db.chunks.find(query);

cursor.forEach(function(d) {
 print( "chunk: " + d.min._id ); 
 sh.moveChunk("db.use", { "_id" : d.min._id }, "rs1");
});
sh.setBalancerState(true)

5. Restore dump (connect to mongos)

mongorestore --host <mongos> --db=db --collection=use --gzip --archive=dump.gz