0

I have a shareded collection which I'm trying to migrate from an existing production Cluster running v 2.4 to a new one running 2.6.

As both clusters are shareded, I'm using mongoexport from the old and mongoimport to the new so the data will go through the config servers. The way I'm doing it is the call mongoexport process and pipe the results into mongoimport process.

I'm guessing that the export | import process affects the size of the exported json which fails to be imported back to the new cluster. Is there any way I can avoid having this problem or even ignore some documents that cause this error? (I'd rather lose a few than lost all 8 million records).

Is there a way to go around this?

Thanks in advance

Improve this question
2
  • Are you trying to do a live migration to a parallel/new cluster running 2.6, or trying to migrate your 2.4 cluster to 2.6?
    – Stennie
    Commented Aug 28, 2014 at 6:40
  • Live migration between 2 different clusters. Almost done too, just these 2 collections remaining.
    – Meny Issakov
    Commented Aug 28, 2014 at 6:41

1 Answer 1

Reset to default
2

For data migration (or backup) between MongoDB servers, you should be using mongodump and mongorestore (binary backups) rather than mongoimport/mongoexport (text backups).

Backups (and restores) of sharded collections need to be done through a mongos.

There are several reasons to use mongodump/mongorestore:

  • binary backups preserve type fidelity in your BSON documents; mongoexport to a text format can lose type information because JSON can only represent a subset of the BSON types

  • you can dump/restore full databases (rather than being limited to a single collection)

  • information on index definitions is included so they can be recreated by mongorestore

If you're trying to do a live migration by dumping/exporting records, this is going to be problematic because your dump will not represent a single point in time if there is write activity on the collections you are dumping.

You definitely want to disable the balancer on the source cluster before taking your mongodump, to ensure there are no active data migrations.

You may also want to look into a live migration tool like hydra.

Improve this answer
6
  • Regarding the Mongodump/restore part. If I'm not mistaken the mongodump is done directly on the mongod nodes, and so does the restore. isn't that problematic to perform a restore on a new cluster where the config servers aren't aware of the data and chunks distribution?
    – Meny Issakov
    Commented Aug 28, 2014 at 6:56
  • Sorry, I should have made that clearer. You need to mongodump and mongorestore via the mongos.
    – Stennie
    Commented Aug 28, 2014 at 6:57
  • Isn't mongodump/restore performed directly via the mongod nodes? If I'll perform dump on the config server, this means I'll also get other stuff from the config server such as shards metadata/chunks metadata and more, most of them are not relevant in the new cluster as most of the data has already been migrated. I've seen hydra before, I'll give it another look and see if it can solve my problem. Thanks
    – Meny Issakov
    Commented Aug 28, 2014 at 7:03
  • 1
    No, if you dump/restore via the mongos this will correctly route requests to the shards. You only get the config metadata if you dump the config database. Since you only have some specific collections to dump you can specify the -c (or --collection) name to mongodump. If you dump directly from a shard mongod this may include orphaned data (the mongos knows what data belongs to a shard; the mongod does not). Restoring directly to a shard mongod will add data the config servers don't know about and cause issues for sharded collections.
    – Stennie
    Commented Aug 28, 2014 at 7:07
  • That's great news. I didn't knew that!. i'll give it a shot, thanks.
    – Meny Issakov
    Commented Aug 28, 2014 at 7:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.