On a MongoDB 3.6 in a replica set and WiredTiger engine we have a large collection with 7.7TB of data. We started to see severe performance issues (extreme CPU usage and slowdown of data replication) which we somehow solved by temporary using a new empty database. The data is basically usage logs and DB pattern is that mostly there are inserts (of new usage data) and occasionally data reads for certain users or reports.
To scale the system and solve performance issues in the long run, we now created two shards where the idea is to have a fast shard (with data of recent 2 months) and a slow shard (with all historical data). Fast shard run on
n1-standard-8 (8 vCPUs, 30 GB memory) machines with
SSD disks, while slow shard is on
n1-highmem-2 (2 vCPUs, 13 GB memory) machines with
The database is not under any load (there are no new usage log inserts) and basically does nothing. The sharding process has started, but the speed of moving chunks is slow and will take around 40 days to move all the chunks. There are 200000 chunks left to be moved from fast to slow shard with average chunk size 32MB and it takes about 18 seconds to move one chunk.
What can be done to speed up this chunk migration process?
I have learned that if source shard is on SSD disks, then the chunk moving has almost double speed.