MongoDB hashtable namespace index max chain reached error after machine crash

Question

I have a database machine running standalone MongoDB v2.6.12 on Ubuntu 14.04. It is running well until the machine crashed (by power failure).

MongoDB won't start since then outputting the following error in log.

2017-11-07T16:19:12.149+0800 ***** SERVER RESTARTED *****
2017-11-07T16:19:12.151+0800 [initandlisten] MongoDB starting : pid=1978 port=27017 dbpath=/edx/var/mongo/mongodb 64-bit host=external-db
2017-11-07T16:19:12.151+0800 [initandlisten] db version v2.6.12
2017-11-07T16:19:12.152+0800 [initandlisten] git version: d73c92b1c85703828b55c2916a5dd4ad46535f6a
2017-11-07T16:19:12.152+0800 [initandlisten] build info: Linux build5.ny.cbi.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2017-11-07T16:19:12.152+0800 [initandlisten] allocator: tcmalloc
2017-11-07T16:19:12.152+0800 [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/edx/var/mongo/mongodb", journal: { enabled: true }, smallFiles: true }, systemLog: { destination: "file", logAppend: true, path: "/edx/var/log/mongo/mongodb.log" } }
2017-11-07T16:19:12.154+0800 [initandlisten] journal dir=/edx/var/mongo/mongodb/journal
2017-11-07T16:19:12.154+0800 [initandlisten] recover begin
2017-11-07T16:19:12.154+0800 [initandlisten] info no lsn file in journal/ directory
2017-11-07T16:19:12.155+0800 [initandlisten] recover lsn: 0
2017-11-07T16:19:12.155+0800 [initandlisten] recover /edx/var/mongo/mongodb/journal/j._0
2017-11-07T16:19:12.155+0800 [initandlisten] recover cleaning up
2017-11-07T16:19:12.155+0800 [initandlisten] removeJournalFiles
2017-11-07T16:19:12.156+0800 [initandlisten] recover done
2017-11-07T16:19:12.466+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.469+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.469+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.472+0800 [initandlisten] waiting for connections on port 27017
2017-11-07T16:19:12.483+0800 [IndexRebuilder] Fatal Assertion 16253
2017-11-07T16:19:12.487+0800 [IndexRebuilder] 0x121df81 0x11bd689 0x11a019d 0xa2d5e5 0xa2d703 0x904f70 0xba15e4 0xba24b8 0x11a3532 0x1262d49 0x7f8f424c9184 0x7f8f417cdffd

It is unable to run with --repair which will result in this error BadValue Can't have journaling enabled when using --repair option.

And I don't think it is really a namespace problem, because a clone machine from the snapshot before the crash will run without any problem.

The other question you guys marked that I'm "duplicateing" is in completely different situation. As I'm not even able to start MongoDB, how can I run command in it?

Also I tried to disable journal and get the following error.

You specified --repair but there are dirty journal files. Please
restart without --repair to allow the journal files to be replayed.
If you wish to repair all databases, please shutdown cleanly and
run with --repair again.

And if I remove --repair I get the same error: hashtable namespace index max chain reached:1335 errors in the log as in the beginning.

Answer 1

OK I found my own answer.

Because the journal file seems corrupted, this solution will abandon all changes in the journal file so the database might be restored to a slightly earlier state before the crash

According to MongoDB documentation,

At default intervals of 60 seconds, MongoDB asks the operating system to flush the shared view to disk.

the period of losing data should be within less than 60 seconds before crash by default.

The steps are as follows:

1. Turn off journal in the config. For me it's in /etc/mongod.conf and change journal=true to journal=false.

2. In the MongDB data directory (e.g. /var/lib/mongodb/, or for me it's /edx/var/mongo/mongodb/), remove both journal folder and mongod.lock file.

3. Run mongod with --repair option like sudo -u mongodb mongod -f /etc/mongod.conf --repair and wait. This operation may need a lot of disk space.

4. Turn on journal to journal=true as opposite to step 1, and start the server normally. E.g. sudo service mongod start.