Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I'm running mongodb 1.4.7 - I've a node server set up which logs incoming requests in a logs collection:

{'u':'user123', 'l':'log data'}

In addition, once the request happens, the server will check if a users collection contains a user123 in this case via the .find({'u': 'user123'})

Running top I see this:

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
4709 mongodb   20   0 1568m 435m 303m R 176.3 11.0 370:22.58 mongod

This, however, only happens every so often, most of the time it sits at about 1-2% CPU and running mongotop shows that mongo isn't particularly stressed:

           mdb.users         1ms         1ms         0ms
           mdb.impressions   1ms         1ms         0ms

I'm getting a decent amount of traffic. What steps can I take to identify the source of resource usage spikes?

share|improve this question
    
Are you sure that is your MongoDB version? I think you might mean 2.4.7 - the 1.4 branch only ever got to 1.4.4 and that was released June 2010 –  Adam C Nov 4 '13 at 18:36
    
Do you have numa enabled - you can check in Mongo logs at start-up to see if there is a warning about numa - it can cause this sort of symptom. –  Asya Kamsky Nov 6 '13 at 8:26

2 Answers 2

There is no one simple answer here, it will very much depend on your usage. CPU can be consumed in many ways. I'll list the most common (in my experience with MongoDB) and how you can try to determine if this is the case for your spikes.

Before I do though, I will say that a CPU spike is not necessarily a problem - people get obsessed about spikes in CPU, increases in the load metric, but what really matters is the performance of your application - is it actually suffering during these spikes? If not, then it's probably not that important - by all means monitor it and be aware in case things get worse and you do see an impact, but spikes happen on busy systems without it being a problem per se.

Common Causes (and how to narrow it down):

1 - A periodic, intensive database operation

Table scans (non-indexed queries), massive index updates, in-memory sorts and probably a few others can all tax a database process from the CPU side of things. To determine if that is the cause here, there are a few tools that you should be using:

This is going to come up in this list more than once, but specifically you will want to look at the number of ops being executed at a given time, btree hits, and the hardware stats (munin-node required). These will give you an insight into the work that the database is doing. User CPU is what you have listed above, but is there a system CPU component too? MMS will graph them over time and let you correlate the spikes in CPU with other events in general.

If you know the times that you are seeing the spikes, take a look in the logs - are there any slow operations around that time? Any index builds? If the system is being generally slowed down there might be a lot of false positives in here, so look for trigger events, common themes etc.

The excellent mtools by Thomas Rueckstiess can make this analysis significantly easier, allowing you to graph events, look at specific slices of time etc.

This will tell you what databases and collections MongoDB is spending time on. Now, if you only have one collection and no replication, then this will not be terribly helpful, but if you have a multitude to pick from, then it can be very effective at narrowing things down, especially if you combine it with the data from MMS to get you specific time frames to analyze. Unlike MMS though, you have to run this in advance of the issue so that the timeframe of your suspicious spike is covered.

This tool provides data similar to that in MMS, but at a per-second granularity (MMS Monitoring is 60 seconds), so if the spikes are short lived, this can be a valuable piece of additional data. Similar to mongotop, it must be run manually beforehand to cover an event and then the data gathered/examined.

2 - An IO Bottleneck

Slow IO can cause CPU usage to spike. Again, MMS Monitoring with munin-node installed will give you an idea as to the nature of the CPU usage - if IOWait is high then you will want to pay attention to the whether the traffic is read or write bound (or both) and looking at the page fault levels can also be enlightening (high levels can mean that data is not fitting into memory, or "cold" data is being paged in at that time).

Essentially, all of the same tools are in play here as in the first scenario, but this time you can add:

IOStat deserves an essay in and of itself, so I will be brief - you will want to look at %util as a broad first measure, then look for further information to determine the source of the slowness. The await metric can tell you if queueing is an issue, svctime can tell you if the device is being slow, high reads or writes at the time of the spikes can tell you what particular IO limit you are running into. Again, similar to the mongostat/mongotop tools, you can run this command to a much finer granularity than you will see in MMS.

3 - Connection Related Churn

This is somewhat more rare, but a badly configured driver, or an application that gets out of control can sometimes cause a massive amount of connections to be opened/closed in a small time frame while leaving the concurrent connection level fairly constant. You can use a combination of MMS (connection graph) and analysis of the log files (as long as you are not running with --quiet) to track this down.

There are quite a few other things to look at here, but those are where I would start, and if you master the tools listed you will be well on your way to troubleshooting those other possibilities.

share|improve this answer

I use mongodb in a similar way, and I have seen similar CPU maxing issues. (In fact, from time to time, I see mongod peg the cpu for some minutes, even when my write activity is relatively low.)

I attribute this to mongodb doing a few things:

  • The mongo db (in our case, anyway) has a fixed max size, and is circular. So periodically, it has to "eat its tail"
  • We are often at the edge of avail disk space, which at times (a few times a week) causes a lot of thrashing

I have never been able to fully diag this, and in our setup, we are ok with dropping data once in awhile (it's just log data). We are working on a new bigger mongo server... so that may solve it.

share|improve this answer
    
capped collections do NOT generate CPU load when they "roll over" - it's a non-event. –  Asya Kamsky Nov 6 '13 at 8:25
    
@AsyaKamsky OK! I can't tell what mongo does that spikes cpu, just painting up the overall situation. All seem to agree that disk thrashing can cause high mongo cpu, and I do have that at times. –  samsmith Nov 8 '13 at 6:08
    
if you have queries that do sorts, or you do mapreduce and/or aggregations, that will cause CPU load. –  Asya Kamsky Nov 9 '13 at 21:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.