Mongo 3 Intermitted Crashes

Question

Currently running two VMs with fresh installs of Ubuntu 14.04 x64 and Mongo 3 Stable. Each VM has the same specs with 8 cores and 56GBs of RAM. Our processing VM is doing great under heavy loads, however our production VM seems to have Mongo crashing once every few hours and can't pinpoint why or how.

My config for the production Mongo instance looks like this

Here is the last three logs outputted. The first two links are cut near the bottom as they were too big, and the last is a full log top to bottom (as it was a short lived run).

https://gist.github.com/dustinblackman/e32778344ec5f21af0d8 https://gist.github.com/dustinblackman/acb0a3aef77114275de4 https://gist.github.com/dustinblackman/908d63e7bd386aae6bc7

The applications that are communicating with this instance are using the latest PHP, Python, and NodeJS drivers.

Does anything in the logs stand out to anyone? All that I see is that the only proper operation that happens before a crash is auth (and I've tried disabling auth, but no deal). Any help is appreciated. Thanks!

Answer 1

Open up /var/log/messages and search for mongo.

If you see "oom-killer" or "Out of memory" then this points to your problem.

We had that problem when running a mongos and multiple mongods on the same machine, for testing purposes.

Since there is no way to limit mongo's memory usage, we just added a 4GB swapfile, so there was some leeway when memory was at its limit. (Using swap is the recommendation here.)

More on the OOM killer here.