6

We have full lvm snapshot backup. We are trying to take oplog backup in every hour and save as a file. We like to apply the oplog backup on top of full backup if we need to. We like to know how we can do this. We are taking everyday LVM snapshot full backup. Our Oplog is 80GB, which can hold 30 days of oplog. But we are facing issue to apply it. I can take the oplog backup in every hour. But question is : how to apply it? We don't want duplicate or any missing data.

  • How often you take a full backup? The problem here lie on 1) You might get duplicates between oplog snapshot and 2) you might miss events if oplog cant fit one hour event. To be honest i am really interested in find a way implement your idea – Antonios Aug 4 '14 at 21:02
4

The oplog is idempotent, you can run through the operations in it as many times as you want and you won't get duplicates or issues unless you run the operations on a set of data files in an odd state.

However, it should be noted that as long as you have the journal as part of the LVM snapshot, re-running the oplog is not necessary for a consistent backup.

With that said, if you do have a copy of the oplog, you can "replay" it in two ways:

  1. Start up a new MongoDB instance, use it to access the oplog backup, and then create a custom script to read each operation and apply it in order (has the benefit of allowing you to filter and being rather quick)
  2. Use mongodump/mongorestore to replay the oplog (see below)

To use mongodump in this way, you first have to dump it out into BSON format so that it can be resored:

mongodump --dbpath /path/to/folder/with/oplog/files -d local -c oplog.rs -o oplogDump

Next you move the bson file out of the folder (this is basically for convenience):

mkdir oplogRestore
mv oplogDump/local/oplog.rs.bson oplogRestore/oplog.bson

Now you can use mongorestore to replay the oplog and apply it to a given running instance:

mongorestore --host host:port --oplogReplay oplogRestore

You can also use the oplogLimit option if you only wish to replay up to a certain point. See Asya's excellent answer here for more on that.

Please note that none of this will be particularly quick, nor is it strictly necessary (as mentioned above).

| improve this answer | |
  • I followed the procedure, but it did not work. oplogReplay --oplogLimit 1407127315:363 /data/test/oplogR connected to: 10.98.141.231:27017 Tue Aug 5 13:30:57.391 The oplogLimit is not newer than the last oplog entry on the server.root@mongo-dba-test sarba]# mongorestore -h 10.98.141.231 --port 27017 --oplogReplay oplogR connected to: 10.98.141.231:27017 Thu Aug 7 09:07:22.495 No oplog file to replay. Make sure you run mongodump with --oplog. – user3895930 Aug 7 '14 at 13:05
  • renaming oplog.rs.bson to oplog.bson is not only for convenience. mongorestore searches explicitly for oplog.bson (it would be already named oplog.bson if it would come from full-dump with mongodump --oplog) – davisca Feb 22 '17 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.