Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

Evening we have had a mongodb running fine for a few weeks and all of sudden it has gone down.

We are unable to start it back up as it is complaining that the "dbpath (/data/db/) does not exist" I've tried routing through the documentation and have seen others suggest using the dbpath flag, however still no luck.

We are starting it with "sudo service mongodb start" and the config file dbpath config points to a directory with our db in.

Any advice would be greatly appreciated.

Thanks in advance. Gary

share|improve this question
    
DISCLAIMER Not MongoDB Guy Questions: Is the lock file still around? Look for /data/db/mongod.lock. Is it a zero byte file ? –  RolandoMySQLDBA Jul 16 '13 at 21:11
    
Hi Rolando, thanks for coming back to me. Yes the lock file is still around and it is also zero bytes. I have actually just got it back up so I now need to identify why it went down. I can't see anything obvious in the logs. However when I did just get it started it suggested I have "Insufficient free space for journal files " even though there seems to be enough free space on the server. Using smallfiles flag has allowed it to start but to be honest I'm not sure what it does. Apart from the obvious. –  Gary Jul 16 '13 at 21:19
add comment

2 Answers 2

up vote 2 down vote accepted

Since you are running out of space for the journal files, here is what the docs says:

journal

journal Default: (on 64-bit systems) true

Default: (on 32-bit systems) false

Set to true to enable operation journaling to ensure write durability and data consistency.

Set to false to prevent the overhead of journaling in situations where durability is not required. To reduce the impact of the journaling on disk usage, you can leave journal enabled, and set smallfiles to true to reduce the size of the data and journal files.

Note You must use nojournal to disable journaling on 64-bit systems.

journalCommitInterval

Default: 100 or 30

Set this value to specify the maximum amount of time for mongod to allow between journal operations. Lower values increase the durability of the journal, at the possible expense of disk performance.

The default journal commit interval is 100 milliseconds if a single block device (e.g. physical volume, RAID device, or LVM volume) contains both the journal and the data files.

If different block devices provide the journal and data files the default journal commit interval is 30 milliseconds.

This option accepts values between 2 and 300 milliseconds.

To force mongod to commit to the journal more frequently, you can specify j:true. When a write operation with j:true is pending, mongod will reduce journalCommitInterval to a third of the set value.

For more clarification, please read Recover MongoDB Data following Unexpected Shutdown. You may need to run validate() on your data collections to verify integrity.

You should also read about Journaling

SUGGESTIONS

  • Looking back at this phrase If different block devices provide the journal and data files the default journal commit interval is 30 milliseconds.
  • You should make store the MongoDB journals on a different disk from the Data.
  • You may also want to shrink the journaling files if you haven't already done so.
  • You should see if the journal files are RAM-based. If they are and if you ran out of memory, that may explain your crash.
  • See Documentation on --smallfiles
share|improve this answer
    
Thanks Roland for such a thorough answer much appreciated. I've been having a look through and sure enough the RAM on the cloud instance probably wasn't enough, we've upped our spec (Is there anyway in logs that we could say for certain this was the issue). Clearly journalling is not something we want to disable, I'm just intrigued as to why --smallfiles is not used as default, does it affect performance? –  Gary Jul 16 '13 at 22:40
1  
Smallfiles setting would put a cap on diskspace usage for journaling. Most large installations of MongoDB would shell out the $$$$ for magnum hardware and disk. Large journaling files would not be the concern. When hardware and disk are at a premium, I can see skimping on journaling as a means of not worrying about crashes due to logs and increasing write performance at the same time (naturally at the risk of not having consistent data after a crash). Smaller journal files may result in more disk I/O and more open files if you have a large number of databases. –  RolandoMySQLDBA Jul 17 '13 at 15:18
add comment

To explain the effect you are seeing. When you run without --smallfiles (the default) the journal files that are created will be at least 3GB (3 x 1GB) and can be larger with a high load system. The journal needs to be big enough to hold a decent chunk of the data you are inserting, and you don't want to be waiting around to allocate a new file for a journal commit (every 100ms), hence the journal files are pre-allocated for speed.

Running with --smallfiles reduces this initial size to (3 x 128MB) 384MB. Hence I am guessing you had less than 3GB but more than 384MB of free space. Details are documented here:

http://docs.mongodb.org/manual/reference/configuration-options/#smallfiles

As an extra bit of advice, for the best performance I would recommend placing the journal on a different volume than your data for maximum performance.

share|improve this answer
    
Thanks apologies I can't accept all the answers, have upped everyone for their help and accepted Rolandos for his thorough explanation. Help is/was much appreciated. –  Gary Jul 18 '13 at 19:49
    
Hey Adam, great answer, especially the placing the journal on a different volume than your data for maximum performance, just like what this blog says of MySQL : yoshinorimatsunobu.blogspot.com/2009/05/…. +1 for your answer. –  RolandoMySQLDBA Jul 18 '13 at 19:59
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.