MongoDB optimization on high data

Question

I have a mongoDB collection with 42M registers with 1M new registers every day.

We found a low performance on mongoDB querying. showing in mongostat the fields: qr|qw ar|aw with values of 300-500.

First move was to check index and make new indexes for querys that may took so long.

Now we are getting a good performance most of the time.

But in some cases (random time) we get a lot of connections in pending status, all slow on querys.

But, the top command shows a good performance on processing speed and ram.

The question is:

• How I can check more further what are the not optimized querys.

• In new relic show the INSERT command in this specific collection take the 62% of "most time consuming", there is any way to speed this up?

• And why can be the causes of the pending connections? The strange thing is: when the numbers starts going up never stops until website crash.

EDIT:

We finally solved a big part of the problem doing this steps:

• Move mongo to a new server withouth a openvz, seems mongo gives troubles with openvz
• New filesystem and saas.
• Create new indexes based on our needs.

Mongo is now running pretty well, so thanks all for your help :)

Answer 1

First, let me say that with the information given it will be hard to get at the root cause - it is usually an iterative process that takes multiple attempts to track down the culprit. In the interest of answering your "what next?" portion of the question rather than identifying the root cause, read on.....

First, a couple of recommendations:

1. Get the host into MMS (it's free) - see http://mms.10gen.com - so you can graph stats over time and get a view of the issues without having to be sitting on the box running commands
2. Get munin-node installed too, so you can correlate ops etc. with IO (install docs for MMS explain this).

Next, a couple of quick checks for common causes:

• What is your filesystem/kernel? - these generally need to be ext4/XFS and recent enough to have fallocate working (2.6.23 and 2.6.25 respectively) so that new file allocation is not slow
• Assuming you don't get MMS and munin installed, get iostat output to match up with mongostat to determine if IO is the root cause for the bottleneck
• Do you do any periodic batch updates that grow the documents significantly (i.e. that would cause moves)? Moves are expensive and can cause IO to get backed up
• Is your disk up to the data volume you are writing to it? MongoDB fsyncs to disk every 60 seconds by default, if the volume that needs to be synced after 60 seconds is massive (say because of an insert spike) then you can also run into issues

That's not an exhaustive list, I have seen other issues cause this, but that should get you started down the right path.