Sharded MongoDB backup solution

Question

I am looking for a way that I can backup my 12 MongoDB shards into a single server. Disk space is not an issue but getting the backups to be on a single node would be the best. I am not sure if there is a way to run more than one MongoDB server on a single machine or if there is a way to have all 12 shards merged into a single MongoDB instance. I am running on AWS and will have the primary data in US-East but want the backups in US-West. I dont mind if there is lag on the backup itself, but trying to avoid the app servers for waiting for the backup to be written.

Answer 1

You need to be a little more clear about your requirements:

• Do you want to keep all 12 databases intact (12 instances versus all 12 in a single instance)?
• Are you intending to serve any queries from the backup servers?
• Do you want multiple snapshots?

You can approach this from many different ways, but I will cover the most common approaches:

Put secondaries in US-West

This approach means you have a full, complete copy of your databases on the west coast. You can make them hidden and priority 0 to make sure they don't serve slaveOK reads or become primary by accident. To make them primary (in a disaster recovery scenario) you will need to re-configure the set manually if they are hidden and priority 0.

This is quite an easy approach, though it could be expensive. You can run multiple mongod processes on a single host but they will be competing for resources and it will still have to be fast enough to keep up with the primaries on the east coast.

Use EBS snapshots

Assuming you are running with journaling enabled, you can just take an EBS snapshot of your data folder (dbpath) and you can back that up to S3 or use it to "seed" hosts on the west coast.

Mongodump

mongodump can be used to dump out all of your data from the Eastern hosts to the Western hosts. However, the output (BSON files) will need to be restored before it can be used.

Generally I would recommend a combination of the first two in order to combine the best resilience options with the most flexibility.

Answer 2

Ideally you would have one or more replica set. Within the replica set I suggest you always have odd numbers for the number of members with one node that is hidden locate in us west. To do so into a three members replica set you can use something like this:

{
"_id" : "rs0",
"version" : 1,
"members" : [
         {
                 "_id" : 0,
                 "host" : "mongodb0.example.net:27017"
         },
         {
                 "_id" : 1,
                 "host" : "mongodb1.example.net:27017"
         },
         {
                 "_id" : 2,
                 "host" : "mongodb2-us-west-region.example.net:27017",
                 "priority": 0,
                 "hidden": true
         }
        ]
 }

If you keep this sort of config to do backups on a mongo cluster in aws is as simple as have a list of hidden nodes in us-west region. Obtain their instance id, lock the database - call aws api to describe the ebs volumes you have attached to those instance id and ask to snapshot it, after all this is done unlock the database. Also you need to take care of the automatic balancer - first stop one of the config server (by stop i mean you put a lock - if you can afford to shut it down is even simpler - than just copy the files) - mongodump the config database and start the balancer.

As an operation sequence you should do the following

• stop balancer
• lock the db
• snapshot the db
• make a backup of the balancer (mondodump or copy the files)
• unlock the db
• start the balancer