2

We have a 3-node Mongodb replica set that resides entirely in AWS. The primary is an i3.2xlarge (8 cores 61G RAM), while the secondary members are i3.xlarge (4 cores 30.5G RAM). These instances are each backed by two SSD EBS volumes, with one of the pair formatted XFS solely for the Mongodb installation. The Mongodb installations use the WiredTiger storage engine. The mongod.conf is here.

We have a roughly weekly ETL process that uses Pentaho to write to our replica set via a Tomcat8-based REST API. This process has completed smoothly for the last two years until our upgrade to Mongodb 3.6 at the end of February. During the first run of this process on Mongodb 3.6, the mongod process ran the instance out of memory. The OS invoked oom-killer to stop the process. After the subsequent election, the process then also killed the new primary.

We’d previously never configured swap space for the instances, but after this incident, swap space was configured on each, on a [RAM x 2] sized AWS NVMe instance store volume. Since implementing the swap space, any intensive process will continue to run the instance down to <1G of memory, then roll over into the swap space. We’re no longer experiencing a mongod service crash, but the system will regularly use up all available memory, and up to half of the swap space. As an example, the system was in such a state yesterday, and while restarting the service freed up the memory and swap space, the nightly dump has left the system using 26/60G of memory. It seems that the service isn’t releasing memory/killing threads once they’re no longer actively in use (see below image).24hr % avail. Memory. Service was restarted at 10:15am, and dump started at 2am..

Other system details:

  • Ubuntu 14.04.5
  • 4.4.0-112-generic #135~14.04.1-Ubuntu SMP
  • Mongodb 3.6.2
  • transparent_hugepage & defrag are set to “never”
  • readahead is set to 16K
  • numactl is installed, allowing NUMA to be disabled via the init script
  • Before upgradation of MongoDB 3.6.2 , what version are there is your environment? And what was storage engine on that time (i mean to say that "MMAPv1" or "WiredTiger" . – Md Haidar Ali Khan Mar 7 '18 at 6:30
  • Try limiting WiredTiger disk cache size. See docs.mongodb.com/manual/reference/configuration-options/… – Saleem Mar 7 '18 at 7:55
  • Prior to 3.6.2, our environment was 3.4.13; also on WiredTiger. – Travis Mar 7 '18 at 12:53
  • how many active connections are there? Could you share the db.serverStatus() output? – weishann Mar 7 '18 at 16:33
  • @weishann There are currently 20 open connections. During an ETL push, that number will top out at ~425. db.serverStatus() output is here: pastebin.com/nrM7qUB2 – Travis Mar 8 '18 at 13:09
0

As per your system details the readahead is set to 16K. But as per MongoDB BOL for the WiredTiger storage engine. Set the readahead setting to 0 regardless of storage media type (spinning, SSD, etc.).

Setting a higher readahead benefits sequential I/O operations. However, since MongoDB disk access patterns are generally random, setting a higher readahead provides limited benefit or performance degradation. As such, for most workloads, a readahead of 0 provides optimal MongoDB performance. In general, set the readahead setting to 0 unless testing shows a measurable, repeatable, and reliable benefit in a higher readahead value. MongoDB Professional Support can provide advice and guidance on non-zero readahead configurations.

As MongoDB documentation for the WiredTiger, consider the following recommendations on Linux environment:

  1. Turn off atime for the storage volume containing the database files.
  2. Set the file descriptor limit, -n, and the user process limit (ulimit), -u, above 20,000, according to the suggestions in the ulimit reference. A low ulimit will affect MongoDB when under heavy use and can produce errors and lead to failed connections to MongoDB processes and loss of service.
  3. Disable Transparent Huge Pages. MongoDB performs better with normal (4096 bytes) virtual memory pages. see Transparent Huge Pages Settings.
  4. Disable NUMA in your BIOS. If that is not possible, see MongoDB on NUMA Hardware.
  • As stated above, transparent huge pages and NUMA are disabled. The xfs volume for the db installation also has the noatime flag set. Max user processes are > 245K. I will look into decreasing readahead and increasing max open files. – Travis Mar 7 '18 at 13:09
0

For anyone looking for the answer to this issue in the future, we discovered that it was due to a bug in the 3.0 > 3.2 releases of MongoDB. Unfortunately, we had to spend contractor money to figure it out, but at least we have our replica set functional again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.