2

We are using MongoDB GridFS to create a document archive. I am creating a sharded cluster at our Primary DataCenter. I need a working backup at our offsite Disaster Recovery DataCenter. This will be a very large database, ~6TB. I will be adding 20k new documents daily. I need to be able to take incremental backups at the end of our batch cycle and apply them to the offsite cluster. We do not want to use replication because we want the clusters to be entirely independent, but essentially identical. We will never delete from the collection. We are running Red Hat.

Does anyone have any suggestions about how to keep the databases in sync?

3
  • This SO post offers some relevant answers.. It has some content in the comments about incremental backups that I think can help in your case
    – vmachan
    Jan 27, 2016 at 19:30
  • Has anyone ever used mongo-connector? Do you think it will do what I need? github.com/mongodb-labs/mongo-connector Jan 28, 2016 at 16:27
  • You could make it do what you want (similar to mongoriver mentioned in my answer) but neither tool is officially supported, and as I also mentioned you will need to add your own logic and verification to the process so you can be satisfied that all the data is mirrored offsite.
    – Adam C
    Feb 4, 2016 at 18:39

1 Answer 1

1

You can look into tailing the oplogs of the shards yourself, thereby sending the data to a cluster in a remote data center incrementally. Similarly you can look into tools like mongoriver from Stripe or mongo-connector from MongoDB labs if you don't want to do all the work yourself from scratch (but then are relying on a third party keeping that tool up to date or an unsupported labs project). You will still need to wrap these utilities in some logic based on your needs and do some manual data verification to validate date between the two sites, and getting point-in-time snapshots of a cluster is non-trivial.

The only "out of the box", supported, way to do this (that I am aware of) is to use the on-premise backup capabilities of Ops Manager which is a paid offering from MongoDB themselves (full disclosure: I used to work for MongoDB). It's a complicated process to keep a full copy of a sharded cluster in a separate location, so really your choices are to invest your own time and resources, or pay for a product to take care of it for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.