MongoDB - Mongorestore is not working as expected

Question

A Very Good Day,

I have the user in the mongodb like below who has the superuser privileges (I confirmed the role setting using show users command)

{
    "_id" : "admin.mongoadmin",
    "user" : "mongoadmin",
    "db" : "admin",
    "roles" : [
        {
            "role" : "readWrite",
            "db" : "admin"
        },
        {
            "role" : "root",
            "db" : "admin"
        }
    ],
    "mechanisms" : [
        "SCRAM-SHA-1",
        "SCRAM-SHA-256"
    ]
}

when I try to restore the oplog using the mongorestore, I get the error :

Failed: restore error: error applying oplog: applyOps: not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1552828309, 1), h: 4632811839329880092, v: 2, op: "c", ns: "admin.$cmd", o: { create: "system.keys", idIndex: { v: 2, key: { _id: 1 }, name: "id", ns: "admin.system.keys" } }, o2: {} } ], $db: "admin" }

mongorestore -u admin -p password --authenticationDatabase=admin --oplogFile 0000000000_0_oplog.bson  --oplogReplay --oplogLimit=1552828432 --dir='/oplog/temp'
2019-03-17T13:47:36.945+0000    preparing collections to restore from
2019-03-17T13:47:36.945+0000    replaying oplog
2019-03-17T13:47:36.962+0000    Failed: restore error: error applying oplog: applyOps: not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1552828309, 1), h: 4632811839329880092, v: 2, op: "c", ns: "admin.$cmd", o: { create: "system.keys", idIndex: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.keys" } }, o2: {} } ], $db: "admin" }

NOTE : I specified the oploglimit (--oplogLimit=1552828432) with the last value I got from the bsondump Is this correct? Or Am I missing anything?

(i.e)

{"ts":{"$timestamp":{"t":1552828432,"i":79}},"t":{"$numberLong":"1"},"h":{"$numberLong":"-2072015676601300967"},"v":2,"op":"i","ns":"inventory.hari","ui":{"$binary":"avdlGH8AS1eBPXRytlO1Yg==","$type":"04"},"wall":{"$date":"2019-03-17T13:13:52.139Z"},"o":{"_id":"79","name":"Hari","role":"Developer","isEmployee":true}}
{"ts":{"$timestamp":{"t":1552828432,"i":80}},"t":{"$numberLong":"1"},"h":{"$numberLong":"-6279494628130059002"},"v":2,"op":"u","ns":"inventory.hari","ui":{"$binary":"avdlGH8AS1eBPXRytlO1Yg==","$type":"04"},"o2":{"_id":"79"},"wall":{"$date":"2019-03-17T13:13:52.139Z"},"o":{"_id":"79","name":"WD_Userjava.util.Random@9a7504c","role":"Developer","isEmployee":true}}

Edited :

I did the restore like below

mongorestore -u mongoadmin -p password --authenticationDatabase admin -d itsm_inventory -c hari 0000000000_0_oplog.bson

It seems successful

I inserted like this

{ "_id" : "0", "name" : "WD_Userjava.util.Random@2bbf4b8b", "role" : "Developer", "isEmployee" : true }
{ "_id" : "1", "name" : "WD_Userjava.util.Random@7a765367", "role" : "Developer", "isEmployee" : true }

but it is restored like below

{ "_id" : ObjectId("5c91eac58fe872c878289d83"), "ts" : Timestamp(1552888901, 2), "t" : NumberLong(1), "h" : NumberLong("-3584326290393879639"), "v" : 2, "op" : "i", "ns" : "itsm_inventory.hari", "ui" : UUID("b5f574f1-c55a-4a5b-8729-4483f56f5c9d"), "wall" : ISODate("2019-03-18T06:01:41.062Z"), "o" : { "_id" : "1", "name" : "WD_Userjava.util.Random@7a765367", "role" : "Developer", "isEmployee" : true } }
{ "_id" : ObjectId("5c91eac58fe872c878289d84"), "ts" : Timestamp(1552888901, 3), "t" : NumberLong(1), "h" : NumberLong("-3618415120025822301"), "v" : 2, "op" : "i", "ns" : "itsm_inventory.hari", "ui" : UUID("b5f574f1-c55a-4a5b-8729-4483f56f5c9d"), "wall" : ISODate("2019-03-18T06:01:41.062Z"), "o" : { "_id" : "2", "name" : "WD_Userjava.util.Random@76b0bfab", "role" : "Developer", "isEmployee" : true } }

Mani's answer edit :

mongorestore -u mongoadmin -p password --authenticationDatabase=admin --oplogFile 0000000000_0_oplog.bson  --oplogReplay --oplogLimit=1553063755 --dir='/oplog/temp'
2019-03-21T10:46:42.435+0000    preparing collections to restore from
2019-03-21T10:46:42.435+0000    replaying oplog
2019-03-21T10:46:42.457+0000    Failed: restore error: error applying oplog: applyOps: not authorized on admin to execute command { applyOps: [ { ts: Timestamp(1552888562, 3), h: -8964353497436574374, v: 2, op: "c", ns: "admin.$cmd", o: { create: "system.keys", idIndex: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.keys" } }, o2: {} } ], $db: "admin" }

Anyhelp is appreciated. Thanks in advance.

Answer 1

As per MongoDB jira blog here mongorestore --oplog will not apply oplog entries captured during the mongodump operation, and exit with an error when attempting to use mongorestore with the restore role and a MongoDB deployment that uses access control. For additional information, please review the documentation on the restore role.

As it's a affects the MongoDB version like 2.6.10,3.04. To avoid such error need to update the MongoDB with current version. The restore role does not provide users access to the applyOps command, which mongorestore uses with the --oplogReplay option.

WORKAROUNDS

To restore data using the mongorestore --oplogReplay option, users may create a role that grants anyAction on anyResource. Do not assign this role to any other user. This role provides full unrestricted access to the system.

Alternatively users can disable authentication while restoring data by starting mongod instances without the --auth option during the installation and then re-enabling --auth after completing the restoration.

After conversation with OP (Harry) & Edit

How to restore a ".bson" file in MongoDB?

C:\Program Files\MongoDB\Server\4.0\bin>mongorestore -d test -c oplog C:\data\oplog.bson
2019-03-20T10:06:35.981+0300    checking for collection data in C:\data\oplog.bson
2019-03-20T10:06:35.985+0300    restoring test.oplog from C:\data\oplog.bson
2019-03-20T10:06:37.058+0300    no indexes to restore
2019-03-20T10:06:37.058+0300    finished restoring test.oplog (10138 documents)
2019-03-20T10:06:37.059+0300    done

In the above mongorestore command where the test is the database name & oplog is the collection name. The C:\data\oplog.bson is the .bson file path location.

For further your ref here

Answer 2

Based on the oplog file you uploaded, the minimum timestamp is 1552888560 and maximum timestamp is 1553063754. But the --oplogLimit=1552828432 you mentioned in your question is not within the limit, which is less than the minimum timestamp value. That is the reason none of the documents were restored.

Run the following command from the folder where 0000000000_0_oplog.bson file exists.

mongorestore -u admin -p password --authenticationDatabase=admin --oplogFile 0000000000_0_oplog.bson  --oplogReplay --oplogLimit=1553063755 --dir='/oplog/temp'

Also, make sure no .bson file present in the /oplog/temp folder.

Answer 3

I had the same issue yesterday when I tried to perform PITR. I am still not able to figure out why it didn't worked when my admin user had "root" role assigned. However, I disabled the "authorization" and tried, it worked. But that's not we all want to be doing in production right.

But then as another workaround, I tried different approach.

1. Either copy oplog to new collection directly (db.copyCollection) or taking a dump and then restoring it to some test collection temporarily.

2. Goal is to edit the documents where "ns: admin.$cmd" or any operations entries related to "admin" db exists in actual oplog dump taken.

3. Once edit has been done, take mongodump again on that collection so you will get ".bson" file in order to restore it to oplog.

4. We could have edited the initial oplog dump file ".bson" by converting it using bsondump but mongorestore --oplogReplay will only work with bson files and there was no way I could have convert json into bson except importing it back to new collection then taking mongodump. Hence, step 3.