26

I was given the task to set up incremental backups for MongoDB replicaset, as start point, of course, I googled it and could not find anything on MongoDB docs, I did find however this question on Stack Overflow, which encouraged to develop my own solution as didn't find Tayra very active.

I read about oplog and realized it was very easy to develop something to replay the log, but it turns out that I didn't have to as mongorestore does that for me.

Now I have a working solution with bash scripts and it was quite easy, that's the reason I am asking here if there is any flaw in my logic, or maybe something that will bite me in the future.

Below how I implemented that:

Full backup procedure

  1. lock writes on a secondary member db.fsyncLock()
  2. Take snapshot
  3. Record last position from oplog

    db.oplog.rs.find().sort({$natural:-1}).limit(1).next().ts
    
  4. Unlock writes db.fsyncUnlock()

Incremental backup procedure

  1. lock writes on a secondary member
  2. Dump oplog from the recorded oplog position on full (or latest incremental ) backup:

    mongodump --host <secondary> -d local -c oplog.rs -o /mnt/mongo-test_backup/1 
        --query '{ "ts" : { $gt :  Timestamp(1437725201, 50) } }'
    
  3. Record latest oplog position (same way as for full backups)

  4. Unlock writes

Full backup restore procedure

  1. stop all instances of mongod
  2. copy snapshot to data dir of the box which will be the primary, but make sure to exclude all local* and mongod.lock this restore technique is called reconfigure by breaking mirror
  3. Start primary
  4. reconfigure replicaset
  5. start secondaries without any data, let them perform the initial sync. Or copy the data from the new primary with fresh local database

Restore incremental backup

When we created incremental backup it stored it like this:

/mnt/mongo-test_backup/1/local/oplog.rs.bson
/mnt/mongo-test_backup/1/local/oplog.rs.metadata.json

We're instered on oplog.rs.bson but we will have to rename it, so here are the steps:

  1. change directory to the backup: cd /mnt/mongo-test_backup/1/local
  2. delete the json file rm *.json
  3. rename the bson file mv oplog.rs.bson oplog.bson
  4. restore it:

    mongorestore -h <primary> --port <port> --oplogReplay /mnt/mongo-test_backup/1/local
    

I have it all scripted, I may commit it on GitHub later.

The question is if there is any flaw in the logic? I am a bit suspicious as the procedure is quite straight forward and still I couldn't find it documented anywhere.

  • 2
    What version of Mongo are you using? If you are using wiredtiger, then the first item you referenced with db.fsyncLock() is a problem. MongoDB Inc claims "With WiredTiger, the fsync command with the lock option cannot guarantee that the data files do not change. As a result, do not use these methods to ensure consistency for the purposes of creating backups." link – SDillon Jul 24 '15 at 12:29
  • 1
    @SDillon using 3.0.4 but not using WiredTiger, at least not yet. I we decide to use it, instead of lock writes we will have to stop mongod all together. It's a fair point thanks – Tiago Jul 24 '15 at 12:35
  • I found the following tool for incremental backup github.com/EqualExperts/Tayra hope this will help – Ahmad Abuhasna Jan 19 '16 at 10:05
  • 1
    "Changed in version 3.2: fsync command with the lock option can ensure that the data files do not change for MongoDB instances using either the MMAPv1 or the WiredTiger storage engines, thus providing consistency for the purposes of creating backups." – safety Jan 10 '17 at 8:39
  • Normal (and absolutely easiest) way to do incremental backups is use LVM and snapshots. docs.mongodb.com/manual/tutorial/… – JJussi May 5 '17 at 14:24
3

To answer your question. No! There is no fail on your logic and it should work without problems. However, if LVM snapshots can be used, it's better way to do backups.

  • How do you do incremental backups of an LVM snapshot? Thanks! – TanisDLJ Sep 4 '17 at 12:56
  • LVM snapshots are incremental by nature. Snapshot are moments in time and there are only changes recorded. – JJussi Sep 4 '17 at 13:54
  • Just a snapshot taken yes, it is incremental. But if you archive the snapshot then is a full backup. You cannot archive different incremental backups like duplicity does, for instance. And you cannot simply start creating snapshots every 30 min for incremental backups as it would hit the performance really bad. – TanisDLJ Sep 4 '17 at 14:06

protected by Community Aug 29 '16 at 17:47

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.