Incremental backup MongoDB from oplog

Question

I was trying to take backup using oplog based on ts using below code

 import pymongo
    import traceback
    from pymongo.cursor import CursorType
    c = pymongo.MongoClient('mongodb://localhost:28001')
    oplog = c.local.oplog.rs
first = next(oplog.find().sort('$natural', pymongo.DESCENDING).limit(-1))
ts = first['ts']
while True:
    cursor = oplog.find({'ts': {'$gt': ts}}, cursor_type=CursorType.TAILABLE_AWAIT, oplog_replay=True)
    while cursor.alive:
        for doc in cursor:
            ts = doc['ts']
            print doc['ns']
            db,collection=str.split(str(doc['ns']),'.',1)

It is working well when we have less opLog size but as i am using it on production where oplog size is 48 GB around it is not working.

I cannot alter oplog size.Please suggest some way to take incremental backup.

Answer 1

Incremental backup procedure

• lock writes on a secondary member

• Dump oplog from the recorded oplog position on full (or latest incremental ) backup:

  mongodump --host <secondary> -d local -c oplog.rs -o /mnt/mongo-test_backup/1 
  --query '{ "ts" : { $gt :  Timestamp(1437725201, 50) } }'
  

• Record latest oplog position (same way as for full backups)

• Unlock writes

Click Here for detailed information

Answer 2

I'm thinking to do incremental backup using mongodump,like followings.

mongodump -d local -c oplog.rs -q "{ts : { \"\$gt\" : { \"\$timestamp\" : { \"t\" : 1494476525, \"i\" : 0 } } }}" --port 27022 --archive=/home/mongodb/testoplog.tar.gz --gzip >> testoplog.log 2>&1