0

I operate a fairly busy HTTP backend over a MongoDB cluster consisting of a primary instance and two secondary nodes. Most commands and queries are served by the primary DB node, and on the surface, things run smoothly and there are no critical performance issues. However, based on a recent MongoDB log analysis, I found that there seems to occur some spikes or clusters of slow (> 100ms) operations over time. Here's a sample as visualized through MTools/mlogvis:

Slow queries (> 100ms) on a 60 minute window

As the HTTP backend isn't timed in any way that could cause acute bursts of activity, I'm struggling to find an explanation for these spikes that appear roughly at 1:10 intervals. Are them a normal artifact from the way that the logs are stored? Do they reflect some underlying infrastructure condition? Or maybe nothing to worry about?

For context, this is a MongoDB 3.2.22 cluster running on Ubuntu 18.04 instances. The HTTP application is a load balanced .Net Core 3.1 API that connects to the database through the standard MongoDB.Driver.

All insights will be very helpful. Thank you very much.

5
  • Did you check the slow queries? They are also written to log file. Mar 8 at 10:47
  • @WernfriedDomscheit yes, but none are really hurting the database with full scans or locks. Also, these same queries perform under 100ms most of the time.
    – Humberto
    Mar 8 at 12:35
  • 1
    It is difficult to guess based on limited information, and slow queries are tip of the iceberg for data that should be reviewed. The 3.2 release series is also well into legacy versions (first released Dec 2015, End-of-Life in Sept 2018). If performance and support is a concern, I recommend planning an upgrade to a more modern non-EOL version of MongoDB (currently 4.2+), as there have been significant improvements in behaviour and diagnostics that would make this issue easier to investigate if it is still reproducible.
    – Stennie
    Mar 12 at 5:21
  • 1
    Some important missing details are your storage engine (MMAPv1 or WiredTiger) and any mongod configuration settings you have changed from the default. A ~70s interval sounds suspiciously close to the default 60s storage engine checkpoint/flush, but there are many possibilities including resource contention (Disk, RAM, network, CPU) or blocking operations (periodic server-side JavaScript eval?). Since there appears to be a regular interval, I suggest you try to correlate slow queries with some other time series metrics like I/O, replication activity, and command execution.
    – Stennie
    Mar 12 at 5:21
  • @Stennie thanks, I will look into these details. Unfortunately, for the time being, it is difficult to correlate metrics from the different application components; it's something we're working on though.
    – Humberto
    Mar 22 at 21:29

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.