We have a 3-node Mongodb replica set that resides entirely in AWS. The primary is an i3.2xlarge (8 cores 61G RAM), while the secondary members are i3.xlarge (4 cores 30.5G RAM). These instances are each backed by two SSD EBS volumes, with one of the pair formatted XFS solely for the Mongodb installation. The Mongodb installations use the WiredTiger storage engine. The mongod.conf is here.
We have a roughly weekly ETL process that uses Pentaho to write to our replica set via a Tomcat8-based REST API. This process has completed smoothly for the last two years until our upgrade to Mongodb 3.6 at the end of February. During the first run of this process on Mongodb 3.6, the mongod process ran the instance out of memory. The OS invoked oom-killer to stop the process. After the subsequent election, the process then also killed the new primary.
We’d previously never configured swap space for the instances, but after this incident, swap space was configured on each, on a [RAM x 2] sized AWS NVMe instance store volume. Since implementing the swap space, any intensive process will continue to run the instance down to <1G of memory, then roll over into the swap space. We’re no longer experiencing a mongod service crash, but the system will regularly use up all available memory, and up to half of the swap space. As an example, the system was in such a state yesterday, and while restarting the service freed up the memory and swap space, the nightly dump has left the system using 26/60G of memory. It seems that the service isn’t releasing memory/killing threads once they’re no longer actively in use (see below image)..
Other system details:
- Ubuntu 14.04.5
- 4.4.0-112-generic #135~14.04.1-Ubuntu SMP
- Mongodb 3.6.2
- transparent_hugepage & defrag are set to “never”
- readahead is set to 16K
- numactl is installed, allowing NUMA to be disabled via the init script