4

I have a mongoDB collection with 42M registers with 1M new registers every day.

We found a low performance on mongoDB querying. showing in mongostat the fields: qr|qw ar|aw with values of 300-500.

First move was to check index and make new indexes for querys that may took so long.

Now we are getting a good performance most of the time.

But in some cases (random time) we get a lot of connections in pending status, all slow on querys.

But, the top command shows a good performance on processing speed and ram.

The question is:

  • How I can check more further what are the not optimized querys.

  • In new relic show the INSERT command in this specific collection take the 62% of "most time consuming", there is any way to speed this up?

  • And why can be the causes of the pending connections? The strange thing is: when the numbers starts going up never stops until website crash.

EDIT:

We finally solved a big part of the problem doing this steps:

  • Move mongo to a new server withouth a openvz, seems mongo gives troubles with openvz
  • New filesystem and saas.
  • Create new indexes based on our needs.

Mongo is now running pretty well, so thanks all for your help :)

4
  • 1
    Please post output of db.currentOps() when the event is occurring. Jul 17, 2012 at 15:25
  • Thanks for the advice. We are checking in that command the querys that take more time in execution, that's is right? In other hand, we have a lot of querys with "too long for logging" or something like that. Now is working fine so i can't give you a log yet
    – nax
    Jul 17, 2012 at 15:29
  • 1
    I think you mean that the query is too large to be logged - usually caused by the query being a huge $in list or similar. In that case you are either going to have to turn on profiling to get the details or work it out from the application side. Aaron is right - db.currentOp() is a good start to figuring out what is running at the time of slowness
    – Adam C
    Jul 18, 2012 at 0:05
  • Yes that was the output, but we finally used profiling to get the more expensive querys, and we are working to fix this! thanks !
    – nax
    Jul 18, 2012 at 12:04

1 Answer 1

2

First, let me say that with the information given it will be hard to get at the root cause - it is usually an iterative process that takes multiple attempts to track down the culprit. In the interest of answering your "what next?" portion of the question rather than identifying the root cause, read on.....

First, a couple of recommendations:

  1. Get the host into MMS (it's free) - see http://mms.10gen.com - so you can graph stats over time and get a view of the issues without having to be sitting on the box running commands
  2. Get munin-node installed too, so you can correlate ops etc. with IO (install docs for MMS explain this).

Next, a couple of quick checks for common causes:

  • What is your filesystem/kernel? - these generally need to be ext4/XFS and recent enough to have fallocate working (2.6.23 and 2.6.25 respectively) so that new file allocation is not slow
  • Assuming you don't get MMS and munin installed, get iostat output to match up with mongostat to determine if IO is the root cause for the bottleneck
  • Do you do any periodic batch updates that grow the documents significantly (i.e. that would cause moves)? Moves are expensive and can cause IO to get backed up
  • Is your disk up to the data volume you are writing to it? MongoDB fsyncs to disk every 60 seconds by default, if the volume that needs to be synced after 60 seconds is massive (say because of an insert spike) then you can also run into issues

That's not an exhaustive list, I have seen other issues cause this, but that should get you started down the right path.

1
  • well thanks! maybe i wasn't clear in what i looking for, but you give me the points to start! we found in top that the reading/writing was causing a bottleneck. So we gonna start fixing this, then we gonna host into MMS and check what querys need to be optimized, anyway we check all our querys in mongo and we gonna create index based on this. Thanks a lot!
    – nax
    Jul 18, 2012 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.