This is a mongoDB replica set with 1 primary, 1 secondary and 1 arbiter.
Suddenly the reads from disk into cache and cache page evictions increased 2x-3x and stay elevated since then (all metrics below from primary unless noted otherwise):
Indices, queries, average document size are unchanged. As far as I understand the theory:
- Wired Tiger uses 1.5GB of 4GB RAM for internal cache
- Eviction happens because the cache is at or beyond 80% of 1.5GB. (Not because the pages grow beyond the 8MB threshold, because the ops/s and network traffic are unchanged so there's not more data or ops; and not because the pages are too dirty as the evicted pages are unmodified)
I don't understand:
- How can this happen so suddenly when all other metrics do not show any significant change?
- Why do the metrics not go back to normal during off-peak times?
- Index usage
- CPU Usage:
- Burst Credits CPU were used also before, but Burst Credits Disk did go up slightly due to the increased reads from disk:
- Storage size did not increase significantly:
- Memory usage:
- hosted on mLab: Dedicated Cluster Standard AWS M2 plan
- mongoDB 3.6.12 (WiredTiger)
- 4GB RAM
- 60 GB of SSD block storage (General Purpose EBS)
Update Oct 18, charts from secondary instance:
(environment changed on Oct 16, hence metrics go down)