1

This is a mongoDB replica set with 1 primary, 1 secondary and 1 arbiter.

Suddenly the reads from disk into cache and cache page evictions increased 2x-3x and stay elevated since then (all metrics below from primary unless noted otherwise):

enter image description here

enter image description here

Indices, queries, average document size are unchanged. As far as I understand the theory:

  • Wired Tiger uses 1.5GB of 4GB RAM for internal cache
  • Eviction happens because the cache is at or beyond 80% of 1.5GB. (Not because the pages grow beyond the 8MB threshold, because the ops/s and network traffic are unchanged so there's not more data or ops; and not because the pages are too dirty as the evicted pages are unmodified)

I don't understand:

  • How can this happen so suddenly when all other metrics do not show any significant change?
  • Why do the metrics not go back to normal during off-peak times?

More metrics

  • Operations/s:

enter image description here

  • Index usage

enter image description here

  • CPU Usage:

enter image description here

  • Burst Credits CPU were used also before, but Burst Credits Disk did go up slightly due to the increased reads from disk:

enter image description here

  • Storage size did not increase significantly:

enter image description here

  • Memory usage:

enter image description here

Environment

Update Oct 18, charts from secondary instance:

(environment changed on Oct 16, hence metrics go down)

enter image description here

enter image description here

enter image description here

enter image description here

Improve this question
3
  • 1
    I assume the current charts are all from the primary. The cache graph definitely suggests a change in workload or cache pressure. A few likely possibilities to investigate: were there any changes in your secondary state (perhaps down or resyncing) that might correlate with the increase? Were there any backups being taken via mongodump (or perhaps a new backup agent enabled)? It has been several days since you posted this; have the metrics returned to their previous average?
    – Stennie
    Oct 18, 2019 at 2:50
  • Yes, charts are from primary, the metrics did not go down (until we changed to the next higher M3 plan yesterday with 8GB RAM). No apparent events on the secondary, no down, resync, etc. I added secondary charts with the cursor at the time of the incident. What puzzles me is that, even if there has been a one-time cache pressure, why would the metrics not return to normal? Thanks for taking a look btw.
    – Manuel
    Oct 18, 2019 at 3:15
  • @Stennie I may have an explanation. I observed that the "cache read" increases significantly with the number of connections between our app server instances and the DB. Apparently it makes a difference, whether we run 5 fast or 10 slow app server instances that connect to the DB, given the same request load. For total app server performance it doesn't make any difference to us, because 5 fast instances handle the same load as 10 slow instances, but it seems to make a difference for the DB. Surely, each connection has an overhead, but apparently it can be so significant. Can you confirm that?
    – Manuel
    Oct 25, 2019 at 2:00

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.