2

So our MongoS instances use up A LOT of memory. More than even the shards and sometimes crash because of it.

Cluster structure

  • 3x mongos
  • 3x mongod Shards
  • 3x mongod Replicas (one for each shard)
  • 3x mongod Config Servers

Each of these is running on its own dedicated server with 64GB of RAM, except the config servers which are running on smaller servers, but they are not relevant here.

Situation

Our MongoS instances use up even more memory than that, but only after some intensive workload. Usually after some huge import or aggregation. And the problem is that they never drop it afterwards. If we do not do huge imports, the memory will stay between 6 and 10GB, but during an aggregation it will jump up to 95-99% memory usage and will stay there even after the aggregation is finished resulting in us getting alerts for high memory usage on our monitoring software all the time. But that's not the main problem, the problem is it crashes.

Problem

Now, I know mongo uses as much memory as it has available and frees it up as necessary, however, it doesn't really free it up correctly. It often crashes, in fact it's bad enough for me to have the following cronjob:

pgrep mongos || service mongos restart

The above cronjob pretty much checks if mongos is running and if not restarts it, i.e. starts it back up after a crash. And after a crash and restart, the memory naturally drops all the way to 5 to 10GB and stays there permanently, unless we do another heavy aggregation or import. In other words, mongo doesn't really need that memory and according to all the issues on stackoverflow, all the docs and what not, mongo is supposed to let the OS tell it when to free up memory. It's supposed to release memory when something or maybe even mongo itself needs to use it for something else, but it doesn't do that and crashes instead.

Thankfully, this isn't a super major problem as our 3 MongoS are behind a load balancer, so if one goes down the other 2 can keep our services online. But it's still an issue I don't want to have to deal with in production, especially as our database and load grows.

I've found this - https://jira.mongodb.org/browse/SERVER-24303 and have enabled aggressive decommit, but that did not help.

Thanks

  • 1
    Could you please list your OS and Mongo version when you get a chance? – Ali Razeghi Jun 10 '16 at 16:18
  • @AliRazeghi sure, all servers are on Mongo 3.2.3 and Debian 8. Currently, 2 of the mongos servers are using 8GB each, while the 3rd is using all 62.6GB out of 64GB and 19GB of swap. All of it is coming from the mongos process. – Sam Jun 14 '16 at 7:10
  • MongoDB tries to load working set into memory for efficient read/write operations. This shouldn't be an issue if your MongoDB instances are running on standalone server. I'd suggest to look into logs and see what is reason of crash? – Saleem Jun 16 '16 at 0:16
  • @Saleem it very rarely crashes. Usually just makes the server unresponsive for a bit, which sends us alerts. If I do get a crash, naturally I'll check the logs. But that might not happen for a while. – Sam Jun 16 '16 at 7:08
  • The mongos should not be behaving as described - I would recommend opening a SERVER ticket to investigate - you will likely need to have the dump from an crash, and some examples of the aggregation and other usage that causes that sort of memory usage – Adam C Jun 18 '16 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.