Mongo DB Can't Start

Question

We have had a mongodb running fine for a few weeks and all of sudden it has gone down.

We are unable to start it back up as it is complaining that the dbpath (/data/db/) does not exist. I've tried routing through the documentation and have seen others suggest using the dbpath flag, however still no luck.

We are starting it with sudo service mongodb start and the config file dbpath config points to a directory with our db in.

Any advice would be greatly appreciated.

Answer 1

Since you are running out of space for the journal files, here is what the docs says:

journal

journal Default: (on 64-bit systems) true

Default: (on 32-bit systems) false

Set to true to enable operation journaling to ensure write durability and data consistency.

Set to false to prevent the overhead of journaling in situations where durability is not required. To reduce the impact of the journaling on disk usage, you can leave journal enabled, and set smallfiles to true to reduce the size of the data and journal files.

Note You must use nojournal to disable journaling on 64-bit systems.

journalCommitInterval

Default: 100 or 30

Set this value to specify the maximum amount of time for mongod to allow between journal operations. Lower values increase the durability of the journal, at the possible expense of disk performance.

The default journal commit interval is 100 milliseconds if a single block device (e.g. physical volume, RAID device, or LVM volume) contains both the journal and the data files.

If different block devices provide the journal and data files the default journal commit interval is 30 milliseconds.

This option accepts values between 2 and 300 milliseconds.

To force mongod to commit to the journal more frequently, you can specify j:true. When a write operation with j:true is pending, mongod will reduce journalCommitInterval to a third of the set value.

For more clarification, please read Recover MongoDB Data following Unexpected Shutdown. You may need to run validate() on your data collections to verify integrity.

You should also read about Journaling

SUGGESTIONS

• Looking back at this phrase If different block devices provide the journal and data files the default journal commit interval is 30 milliseconds.
• You should make store the MongoDB journals on a different disk from the Data.
• You may also want to shrink the journaling files if you haven't already done so.
• You should see if the journal files are RAM-based. If they are and if you ran out of memory, that may explain your crash.
• See Documentation on --smallfiles

Answer 2

To explain the effect you are seeing. When you run without --smallfiles (the default) the journal files that are created will be at least 3GB (3 x 1GB) and can be larger with a high load system. The journal needs to be big enough to hold a decent chunk of the data you are inserting, and you don't want to be waiting around to allocate a new file for a journal commit (every 100ms), hence the journal files are pre-allocated for speed.

Running with --smallfiles reduces this initial size to (3 x 128MB) 384MB. Hence I am guessing you had less than 3GB but more than 384MB of free space. Details are documented here:

http://docs.mongodb.org/manual/reference/configuration-options/#smallfiles

As an extra bit of advice, for the best performance I would recommend placing the journal on a different volume than your data for maximum performance.