0

On a MongoDB 3.6 in a replica set and WiredTiger engine we have a large collection with 7.7TB of data. We started to see severe performance issues (extreme CPU usage and slowdown of data replication) which we somehow solved by temporary using a new empty database. The data is basically usage logs and DB pattern is that mostly there are inserts (of new usage data) and occasionally data reads for certain users or reports.

To scale the system and solve performance issues in the long run, we now created two shards where the idea is to have a fast shard (with data of recent 2 months) and a slow shard (with all historical data). Fast shard run on n1-standard-8 (8 vCPUs, 30 GB memory) machines with SSD disks, while slow shard is on n1-highmem-2 (2 vCPUs, 13 GB memory) machines with rotational disks.

The database is not under any load (there are no new usage log inserts) and basically does nothing. The sharding process has started, but the speed of moving chunks is slow and will take around 40 days to move all the chunks. There are 200000 chunks left to be moved from fast to slow shard with average chunk size 32MB and it takes about 18 seconds to move one chunk.

What can be done to speed up this chunk migration process?

UPDATE

I have learned that if source shard is on SSD disks, then the chunk moving has almost double speed.

  • Basically nothing! Slowest part of movechunk is delete part at the end of process. Delete is done one document at time, what makes it slow process. Anyway, (clustered; sharded) setup where you have nodes with "new" data and then you have "archive" nodes, is very tricky to produce because you need "timestamp" at sharding key and timestamp is monotonically increasing what is big "no-no" as sharding key. Check Monotonically Changing Shard Keys at is.gd/Ndy50a – JJussi Apr 9 '18 at 10:38
0

Things like how many shards we have, which database are sharded, and the configuration setting of your shard cluster. But one of the most important pieces of information that the config servers hold, is the mapping of the chunks to shards.

By default MongoDB takes ChunkSize=64 MB

By default , MongoDB takes 64 megabytes as the default chunk size. That means that if a chunk is about 64 megabytes, or within 64 megatons range , it will be split.

1MB<=chunkSize<=1024MB

We can define a chunk size between the values of one megabyte and 1024 and one gigabyte. The chunk size is configurable during run time. So if we decide to change a chunk size, we can easily do so. But before we go in changing your chunk size.

for further your ref here and here

  • What do you say, that changing chunk size could make a difference in chunk moving speed? – ssasa Apr 9 '18 at 12:23
  • Change of chuck size makes moving one chunk faster, BUT now you have more chunks to be moved --> moving takes same amount of time. If you have 640MB of data at 10 chunks or 50 chunks don't make difference, both ways you are moving same amount of data (documents) – JJussi Apr 10 '18 at 9:23
0

At the end I found another way to speed up chunk moving. Here is the procedure that I tried on a smaller part of the dataset (80 million records) and the estimate is that in that way it will take about 16 days for entire migration.

If you try this, please make sure that you have a backup in place in case something goes wrong.

Here is the procedure:

  1. Count the records to move (on shard rs0)
db.use.find({$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]}).count()
  1. Dump the sample on shard rs0
mongodump --host <rs0-host> --db=db --collection=use --archive --gzip --query '{$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]}' > dump.gz
  1. Delete the data on shard rs0 (connect directly to rs0 mongod, not through mongos)
db.use.deleteMany({$and:[{_id:{$gt: ObjectId("58B60F00e4b03547ad945a8a")}}, {_id: {$lt: ObjectId("58BCA680e4b03547ad945a8a")}}]})
  1. Move chunks (connect to mongos)
use config
sh.stopBalancer()
var lower=ObjectId("58B60F00e4b03547ad945a8a");
var upper=ObjectId("58BCA680e4b03547ad945a8a");
var query={shard: "rs0", ns: "db.use", "max._id":{$gte: lower}, "min._id": {$lte: upper}};
var cursor=db.chunks.find(query);

cursor.forEach(function(d) {
 print( "chunk: " + d.min._id ); 
 sh.moveChunk("db.use", { "_id" : d.min._id }, "rs1");
});
sh.setBalancerState(true)
  1. Restore dump (connect to mongos)
mongorestore --host <mongos> --db=db --collection=use --gzip --archive=dump.gz

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.