So our MongoS instances use up A LOT of memory. More than even the shards and sometimes crash because of it.
- 3x mongos
- 3x mongod Shards
- 3x mongod Replicas (one for each shard)
- 3x mongod Config Servers
Each of these is running on its own dedicated server with 64GB of RAM, except the config servers which are running on smaller servers, but they are not relevant here.
Our MongoS instances use up even more memory than that, but only after some intensive workload. Usually after some huge import or aggregation. And the problem is that they never drop it afterwards. If we do not do huge imports, the memory will stay between 6 and 10GB, but during an aggregation it will jump up to 95-99% memory usage and will stay there even after the aggregation is finished resulting in us getting alerts for high memory usage on our monitoring software all the time. But that's not the main problem, the problem is it crashes.
Now, I know mongo uses as much memory as it has available and frees it up as necessary, however, it doesn't really free it up correctly. It often crashes, in fact it's bad enough for me to have the following cronjob:
pgrep mongos || service mongos restart
The above cronjob pretty much checks if mongos is running and if not restarts it, i.e. starts it back up after a crash. And after a crash and restart, the memory naturally drops all the way to 5 to 10GB and stays there permanently, unless we do another heavy aggregation or import. In other words, mongo doesn't really need that memory and according to all the issues on stackoverflow, all the docs and what not, mongo is supposed to let the OS tell it when to free up memory. It's supposed to release memory when something or maybe even mongo itself needs to use it for something else, but it doesn't do that and crashes instead.
Thankfully, this isn't a super major problem as our 3 MongoS are behind a load balancer, so if one goes down the other 2 can keep our services online. But it's still an issue I don't want to have to deal with in production, especially as our database and load grows.
I've found this - https://jira.mongodb.org/browse/SERVER-24303 and have enabled aggressive decommit, but that did not help.