1

I have to move a MongoDB Collection (Size: 1TB+) from Sharded MongoDB Cluster to another[5 shards on both clusters]. During this operation, it is assumed that the accompanying program will be offline, so no need to worry about inconsistent data, but I need to minimize the downtime.

I've tried testing it with mongodump and mongorestore, but looks like that is taking ages to finish.

Please share your experiences for such kind of scenarios.

Note: Cluster has only one collection, so I am open at cluster level sync as well.

  • Did you try pre splitting the collection before doing the mongorestore? – Mani Mar 19 at 20:42
  • @Mani Sorry I created confusion, but my question is more of, how I can move/migrate collection from one cluster to another. – andy_l Mar 20 at 8:24
  • As you mentioned in the question you are using mongodump and mongorestore to move the data from one cluster to another, what I am saying is before using the mongorestore presplit the collection in destination cluster and proceed with mongorestore. It will improve the performance of mongorestore. – Mani Mar 20 at 15:07
0

As per MongoDB documentation Create Chunks in a Sharded Cluster

If you want to ingest a large volume of data into a cluster that is unbalanced, or where the ingestion of data will lead to data imbalance, such as with monotonically increasing or decreasing shard keys. Pre-splitting the chunks of an empty sharded collection can help with the throughput in these cases.

EXAMPLE

To create chunks for documents in the myapp.users collection using the email field as the shard key, use the following operation in the mongo shell:

for ( var x=97; x<97+26; x++ ){
    for ( var y=97; y<97+26; y+=6 ) {
        var prefix = String.fromCharCode(x) + String.fromCharCode(y);
        db.adminCommand( { split: "myapp.users", middle: { email : prefix } } );
    }
}

This assumes a collection size of 100 million documents.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.