We have a replica set of production mongoDB 3.6.10 , large volume of 5.5 TB , over 10,000 connections anytime , applications read both from primary and secondary servers , we see almost every month a crash in the primary server while every month we see increase in the non-mapped virtual memory. Is it recommended to do a planned restart every month in order to clean the non-mapped virtual memory and to avoid server crashes? Thanks alon
Is it recommended to do a planned restart every month in order to clean the non-mapped virtual memory and to avoid server crashes?
No, scheduled restarts are not a recommended practice. You should investigate and address the source of your issue rather than restarting your server periodically.
If your deployment is correctly configured, MongoDB shouldn't crash when out of memory. Restarting your
mongod processes will temporarily free up memory, but it is expected that a database server will try to take advantage of available memory as needed for your workload.
If you are running Linux, one likely cause for unexpected process shutdown is the Out-of-Memory (OOM) Killer which looks for processes using a lot of memory (like a database server) and shuts them down if there is insufficient memory or swap space. If the OOM Killer decides to kill
mongod, the telltale log message is normally in your system logs -- try grepping for "oom-killer" or "killed process" in
/var/log/messages. See Taming the OOM Killer for more background if this sounds potentially applicable to your environment.
If your overall data size is 5.5TB, I assume there is also general contention for server resources unless you have a relatively small working set or very large host instances.
One source of non-mapped memory usage is your current connection count.