0

I have a database machine running standalone MongoDB v2.6.12 on Ubuntu 14.04. It is running well until the machine crashed (by power failure).

MongoDB won't start since then outputting the following error in log.

2017-11-07T16:19:12.149+0800 ***** SERVER RESTARTED *****
2017-11-07T16:19:12.151+0800 [initandlisten] MongoDB starting : pid=1978 port=27017 dbpath=/edx/var/mongo/mongodb 64-bit host=external-db
2017-11-07T16:19:12.151+0800 [initandlisten] db version v2.6.12
2017-11-07T16:19:12.152+0800 [initandlisten] git version: d73c92b1c85703828b55c2916a5dd4ad46535f6a
2017-11-07T16:19:12.152+0800 [initandlisten] build info: Linux build5.ny.cbi.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2017-11-07T16:19:12.152+0800 [initandlisten] allocator: tcmalloc
2017-11-07T16:19:12.152+0800 [initandlisten] options: { config: "/etc/mongod.conf", net: { port: 27017 }, security: { authorization: "enabled" }, storage: { dbPath: "/edx/var/mongo/mongodb", journal: { enabled: true }, smallFiles: true }, systemLog: { destination: "file", logAppend: true, path: "/edx/var/log/mongo/mongodb.log" } }
2017-11-07T16:19:12.154+0800 [initandlisten] journal dir=/edx/var/mongo/mongodb/journal
2017-11-07T16:19:12.154+0800 [initandlisten] recover begin
2017-11-07T16:19:12.154+0800 [initandlisten] info no lsn file in journal/ directory
2017-11-07T16:19:12.155+0800 [initandlisten] recover lsn: 0
2017-11-07T16:19:12.155+0800 [initandlisten] recover /edx/var/mongo/mongodb/journal/j._0
2017-11-07T16:19:12.155+0800 [initandlisten] recover cleaning up
2017-11-07T16:19:12.155+0800 [initandlisten] removeJournalFiles
2017-11-07T16:19:12.156+0800 [initandlisten] recover done
2017-11-07T16:19:12.466+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.469+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.469+0800 [initandlisten] error: hashtable namespace index max chain reached:1335
2017-11-07T16:19:12.472+0800 [initandlisten] waiting for connections on port 27017
2017-11-07T16:19:12.483+0800 [IndexRebuilder] Fatal Assertion 16253
2017-11-07T16:19:12.487+0800 [IndexRebuilder] 0x121df81 0x11bd689 0x11a019d 0xa2d5e5 0xa2d703 0x904f70 0xba15e4 0xba24b8 0x11a3532 0x1262d49 0x7f8f424c9184 0x7f8f417cdffd

It is unable to run with --repair which will result in this error BadValue Can't have journaling enabled when using --repair option.

And I don't think it is really a namespace problem, because a clone machine from the snapshot before the crash will run without any problem.

The other question you guys marked that I'm "duplicateing" is in completely different situation. As I'm not even able to start MongoDB, how can I run command in it?

Also I tried to disable journal and get the following error.

You specified --repair but there are dirty journal files. Please
restart without --repair to allow the journal files to be replayed.
If you wish to repair all databases, please shutdown cleanly and
run with --repair again.

And if I remove --repair I get the same error: hashtable namespace index max chain reached:1335 errors in the log as in the beginning.

  • 2
    Start your mongod to maintenance mode without journaling! – JJussi Nov 7 '17 at 13:14
  • OK I found my own answer. 1. Turn off journal in the config. 2. In the MongDB data directory, remove both journal folder and mongod.lock file. 3. Run mongod with --repair and wait. 4. Turn on journal in the config and start the server. – Myles Fang Nov 8 '17 at 2:47
  • @MylesFong you should post that as an answer and accept it. – LowlyDBA Nov 8 '17 at 14:15
  • @LowlyDBA Sure. I remember I cannot find the "Answer" button when I write the above comment. – Myles Fang Nov 9 '17 at 2:18
1

OK I found my own answer.

Because the journal file seems corrupted, this solution will abandon all changes in the journal file so the database might be restored to a slightly earlier state before the crash

According to MongoDB documentation,

At default intervals of 60 seconds, MongoDB asks the operating system to flush the shared view to disk.

the period of losing data should be within less than 60 seconds before crash by default.

The steps are as follows:

  1. Turn off journal in the config. For me it's in /etc/mongod.conf and change journal=true to journal=false.

  2. In the MongDB data directory (e.g. /var/lib/mongodb/, or for me it's /edx/var/mongo/mongodb/), remove both journal folder and mongod.lock file.

  3. Run mongod with --repair option like sudo -u mongodb mongod -f /etc/mongod.conf --repair and wait. This operation may need a lot of disk space.

  4. Turn on journal to journal=true as opposite to step 1, and start the server normally. E.g. sudo service mongod start.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.