4

We accidentally deleted the directory rm -rf /data/db which was our MongoDB path, and thanks to extundelete, we recovered it and got the directory /data/db.

Here are our files in the directory, and the files was generated under MongoDB version 3.4.

enter image description here

Folder diagnostic.data:

Folder diagnostic.data

Folder journal:

Folder journal

Step 1: try to run mongod as normal

a) We ran mongod --port 27017 --dbpath /data/db --bind_ip_all and mongo, and expected there should have a user-defined database wecaXX, But, it did not show up.

> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB

In Robo3T

enter image description here

b) Then I tried to run mongod --port 27017 --dbpath /data/db --bind_ip_all --repair. The result was:

2019-03-25T14:10:02.170+0800 I CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] MongoDB starting : pid=23018 port=27017 dbpath=/data/db 64-bit host=iZj6c0pipuxk17pb7pbaw0Z
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] db version v4.0.7
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] git version: 1b82c812a9c0bbf6dc79d5400de9ea99e6ffa025
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] allocator: tcmalloc
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] modules: none
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] build environment:
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     distmod: ubuntu1604
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     distarch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten]     target_arch: x86_64
2019-03-25T14:10:02.191+0800 I CONTROL  [initandlisten] options: { net: { bindIpAll: true, port: 27017 }, repair: true, storage: { dbPath: "/data/db" } }
2019-03-25T14:10:02.191+0800 W STORAGE  [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2019-03-25T14:10:02.194+0800 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten]
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2019-03-25T14:10:02.194+0800 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=256M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2019-03-25T14:10:02.818+0800 E STORAGE  [initandlisten] WiredTiger error (17) [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists Raw: [1553494202:818725][23018:0x7f6119074a40], connection: __posix_open_file, 715: /data/db/WiredTiger.wt: handle-open: open: File exists
2019-03-25T14:10:02.818+0800 I STORAGE  [initandlisten] WiredTiger message unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.1
2019-03-25T14:10:03.832+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:832267][23018:0x7f6119074a40], txn-recover: Main recovery loop: starting at 4/11366912 to 5/256
2019-03-25T14:10:03.832+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:832674][23018:0x7f6119074a40], txn-recover: Recovering log 4 through 5
2019-03-25T14:10:03.898+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:898252][23018:0x7f6119074a40], txn-recover: Recovering log 5 through 5
2019-03-25T14:10:03.964+0800 I STORAGE  [initandlisten] WiredTiger message [1553494203:964880][23018:0x7f6119074a40], txn-recover: Set global recovery timestamp: 0
2019-03-25T14:10:03.998+0800 I RECOVERY [initandlisten] WiredTiger recoveryTimestamp. Ts: Timestamp(0, 0)
2019-03-25T14:10:03.999+0800 E STORAGE  [initandlisten] WiredTiger error (17) [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists Raw: [1553494203:999855][23018:0x7f6119074a40], WT_SESSION.create: __posix_open_file, 715: /data/db/_mdb_catalog.wt: handle-open: open: File exists
2019-03-25T14:10:04.000+0800 I STORAGE  [initandlisten] WiredTiger message unexpected file _mdb_catalog.wt found, renamed to _mdb_catalog.wt.1
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten]
2019-03-25T14:10:04.015+0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 3824 processes, 65535 files. Number of processes should be at least 32767.5 : 0.5 times number of files.
2019-03-25T14:10:04.020+0800 I STORAGE  [initandlisten] createCollection: admin.system.version with provided UUID: 47d8713d-ac61-4081-83bf-60209ad60a7d
2019-03-25T14:10:04.034+0800 W ASIO     [initandlisten] No TransportLayer configured during NetworkInterface startup
2019-03-25T14:10:04.036+0800 I COMMAND  [initandlisten] setting featureCompatibilityVersion to 4.0
2019-03-25T14:10:04.036+0800 I STORAGE  [initandlisten] repairDatabase admin
2019-03-25T14:10:04.037+0800 I STORAGE  [initandlisten] Repairing collection admin.system.version
2019-03-25T14:10:04.040+0800 I STORAGE  [initandlisten] Verify succeeded on uri table:collection-0-4352287918877967674. Not salvaging.
2019-03-25T14:10:04.048+0800 I INDEX    [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" }
2019-03-25T14:10:04.048+0800 I INDEX    [initandlisten]          building index using bulk method; build may temporarily use up to 500 megabytes of RAM
2019-03-25T14:10:04.055+0800 I STORAGE  [initandlisten] finished checking dbs
2019-03-25T14:10:04.055+0800 I STORAGE  [initandlisten] WiredTigerKVEngine shutting down
2019-03-25T14:10:04.056+0800 I STORAGE  [initandlisten] Shutting down session sweeper thread
2019-03-25T14:10:04.057+0800 I STORAGE  [initandlisten] Finished shutting down session sweeper thread
2019-03-25T14:10:04.140+0800 I STORAGE  [initandlisten] shutdown: removing fs lock...
2019-03-25T14:10:04.140+0800 I CONTROL  [initandlisten] now exiting
2019-03-25T14:10:04.140+0800 I CONTROL  [initandlisten] shutting down with code:0

c) After repairing I re-ran mongod --port 27017 --dbpath /data/db --bind_ip_all, it still showed nothing (the same result as step a)).

Step 2: Use wiredTiger tool

Since that didn't work as I expected, I started to find any other tools or approach that may help me. Here is the link Recovering a WiredTiger collection from a corrupt MongoDB installation which introduces how to use WiredTiger to recover collections. And I decided to have a try.

I created a folder db_backup and put all my files into it. And I created a folder wiredtiger-3.0.0 under db_backup.I installed wiredtiger in folder wiredtiger-3.0.0:

enter image description here

a) To salvage the collection

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/# cd ./wiredtiger-3.0.0
root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../db_for_wt -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-23--360946650994865928.wt
        WT_SESSION.salvage 100

Based on the article above, it means that we have successfully salvaged this collection.

Error Occured:

b) To dump the collection

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../../db_backup -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R dump -f ../collection.dump collection-23--360946650994865928
lt-wt: cursor open(table:collection-23--360946650994865928) failed: No such file or directory

Note that the command above was indeed ended without .wt.

I have checked my directory argument and I found no fault. In the picture, the file collection-23--360946650994865928.wt is right Here.

enter image description here

The snapshot of collection-23--360946650994865928.wt just has been salvaged.

We could see some English or Chinese characters in it. And that data truly is from one of collection of database wecaXX.

enter image description here

Questions:

1) Now, I'm stuck by dumping the collection. Does anyone know what's wrong with that?

2) collection-23--360946650994865928.wt contains our most important data. If we could not restore the whole database, extracting data from that file will be still very useful. However, if we do copy paste manually, not all the characters are well written; they still need to be decrypted. Does anyone know how to do that?

3) is it possible that we did not recover all the files (or the entire body of all the files) under /data/db?

Any comment or answer will be appreciated!!

  • what is the MongoDB version(x,y,z)? – Md Haidar Ali Khan Mar 25 at 10:27
  • As i am able to see that from your screenshot of Robo3T, only visible to admin and local databases (system databases). There is no any user defined database available . Could you update the same things. – Md Haidar Ali Khan Mar 25 at 10:31
  • ,After connecting with mongo cell , can you run the command such as show dbs; and update the screenshot of that. – Md Haidar Ali Khan Mar 25 at 10:37
  • Same question asked twice dba.stackexchange.com/questions/232972/… – Mani Mar 25 at 16:37
0

As per your screenshot of Robo3T where only showing the System database (i.e., admin & local). There is no any user defined database is available with your connected host.

Eventhough the files you have generated under the mongodb version is 3.4 & the starting of MongoDB db version is sppearing as v4.0.7.

I will somewhere guess that you have been loose your user created database.

Suppose that if user database wil be there then automatically it will after host connection with Robo 3T-1.2 tool.

For example as the screenshot of Robo 3T - 1.1 appears in my environment.

enter image description here

For verification you can also check with mongo shell, after connecting the mongo shell you can check the command such as show dbs;

It will apears like that

> show dbs
DynamicsAX  0.006GB
ERP         0.086GB
admin       0.000GB
blog        0.000GB
config      0.000GB
enron       0.207GB
local       0.000GB
mflix       0.060GB
test        0.001GB

Note:- Right Now, Before doing any rocket science first you have to take mongodump of your data directory .

As per mention screenshot by you here the database Recovery is a user defined database.

As per your error mongodump error log

root@iZj6c0pipuxk17pb7pbaw0Z:/data/db_backup/wiredtiger-3.0.0# ./wt -v -h ../../db_backup -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R dump -f ../collection.dump collection-23--360946650994865928
lt-wt: cursor open(table:collection-23--360946650994865928) failed: No such file or directory

It's unable to mongodump successfully.

Note: The mongodump will fail if you’re using a version of MongoDB < 3.2, as 3.2 is built against WiredTiger 2.7.

After edit the question of OP

Now, I'm stuck during dumping the collection. Are there anything wrong with me?

Yes, Because the mongodump not went successfully.

In the collection-23--360946650994865928.wt, have we really got the whole data? Even if we could not recover the database, Could we extract data from these *.wt files?

As here already community wiki has been updated the answer, the *.wt are binary data files used by the WiredTiger storage engine. Individual files are not usable as a standalone backup. If you want to take a file copy backup of a MongoDB dbPath you need to include all of the files using a Supported File Copy Backup Method.

for further you ref here , here , here , here and here

  • "I will somewhere guess that you have been loose your user created database" ==> but we do see our most-wanted data in collection-23--360946650994865928.wt, they are not lost. – SoftTimur Mar 25 at 12:06
  • My files were generated under MongoDB 3.4 or higher, The error is "No such file or directory". So I think there must be some other reasons for "dump error". – 国贵棠 Mar 25 at 12:39
  • In one of your ref links.How to restore from .wt files? Compared his files in /db He got a file WiredTiger.turtle that I didn't see in my folder Would it be a cause? – 国贵棠 Mar 25 at 13:06
  • I am not sure if the command ./wt dump is as same as mongodump.So does it really related to mongodump? – 国贵棠 Mar 25 at 13:41
  • The WiredTiger.turtle file contains information required by the WiredTiger storage engine to understand the other data files. It should be along with the WiredTiger all files. – Md Haidar Ali Khan Mar 25 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.