0

I'm encountering cache pressure issues in MongoDB, leading to early termination of buckets in my time series collections within a single database. Here's an overview of my system setup and the challenges I'm facing:

MongoDB Version: Community Edition 7.0.2

System Configuration: 128GB RAM, 12 CPUs, and 2TB SSD x3 (PSS).

Data Ingestion Rate: Approximately 120 million active socket data entries per day into time series collections.

Time Series Collection Configuration: Each collection uses deviceId as the metaField and has a granularity of minutes. Collections are created on a daily basis for the devices.

PS: Please Ignore the count of closedBuckets, I have realized after a while. That's why its so big in the first place.

Issue Description:

After beginning to save documents into the collections, I've observed that the on-disk size of my collections grows up to 3 times larger than normal in time(normal being 8GB and, in some cases, expanding to 24GB which is worst case with same amount of data). In an attempt to fine-tune the WiredTiger parameters to address this issue, I haven't yet succeeded.

Following the parameter adjustments, I restarted MongoDB, which cleared all WiredTiger caches, temporarily alleviating the cache pressure. However, this did not resolve the underlying problem, as numberOfBucketsClosedDueToCachePressure began to increase again within 4 days.

enter image description here

So Once this kicks into play, I have restarted the mongodb and All wiredtiger cache is deleted. So There was no longer cache pressure for a while(around 4 days) as I have configured the mongodb with these stats,

db.adminCommand( { "setParameter": 1, "wiredTigerEngineRuntimeConfig": "eviction=(threads_min=8,threads_max=16),checkpoint=(wait=60),eviction_dirty_trigger=10,eviction_dirty_target=5,eviction_trigger=95,eviction_target=80"})

And I also have increased --WiredTigerCacheSizeGB to 90GB rather than default.

But It didn't resolved my problem and numberOfBucketsClosedDueToCachePressure is started again after in 4 days, below,

enter image description here

And then I again restarted the Mongod. So Here is the metric I saved before the last restart,

db.serverStatus().wiredTiger.cache

{
  'application threads page read from disk to cache count': 152451515,
  'application threads page read from disk to cache time (usecs)': Long('8845187692'),
  'application threads page write from cache to disk count': 129757266,
  'application threads page write from cache to disk time (usecs)': Long('2717874335'),
  'bytes allocated for updates': 579813314,
  'bytes belonging to page images in the cache': Long('8548491672'),
  'bytes belonging to the history store table in the cache': 1237909068,
  'bytes currently in the cache': Long('9596083637'),
  'bytes dirty in the cache cumulative': Long('54447326651345'),
  'bytes not belonging to page images in the cache': 1047591964,
  'bytes read into cache': Long('23141148943904'),
  'bytes written from cache': Long('54251712735224'),
  'checkpoint blocked page eviction': 2215368,
  'checkpoint of history store file blocked non-history store page eviction': 0,
  'eviction calls to get a page': 530564173,
  'eviction calls to get a page found queue empty': 24519253,
  'eviction calls to get a page found queue empty after locking': 3357282,
  'eviction currently operating in aggressive mode': 0,
  'eviction empty score': 0,
  'eviction gave up due to detecting a disk value without a timestamp behind the last update on the chain': 60,
  'eviction gave up due to detecting a tombstone without a timestamp ahead of the selected on disk update': 0,
  'eviction gave up due to detecting a tombstone without a timestamp ahead of the selected on disk update after validating the update chain': 0,
  'eviction gave up due to detecting update chain entries without timestamps after the selected on disk update': 0,
  'eviction gave up due to needing to remove a record from the history store but checkpoint is running': 0,
  'eviction passes of a file': 214853840,
  'eviction server candidate queue empty when topping up': 1069582,
  'eviction server candidate queue not empty when topping up': 19090331,
  'eviction server evicting pages': 0,
  'eviction server slept, because we did not make progress with eviction': 48373131,
  'eviction server unable to reach eviction goal': 0,
  'eviction server waiting for a leaf page': 5952,
  'eviction state': 32,
  'eviction walk most recent sleeps for checkpoint handle gathering': 10413,
  'eviction walk target pages histogram - 0-9': 168185695,
  'eviction walk target pages histogram - 10-31': 24765093,
  'eviction walk target pages histogram - 128 and higher': 0,
  'eviction walk target pages histogram - 32-63': 6502801,
  'eviction walk target pages histogram - 64-128': 15400251,
  'eviction walk target pages reduced due to history store cache pressure': 0,
  'eviction walk target strategy both clean and dirty pages': 23101693,
  'eviction walk target strategy only clean pages': 1769786,
  'eviction walk target strategy only dirty pages': 189982361,
  'eviction walks abandoned': 3425789,
  'eviction walks gave up because they restarted their walk twice': 154422380,
  'eviction walks gave up because they saw too many pages and found no candidates': 9295851,
  'eviction walks gave up because they saw too many pages and found too few candidates': 769560,
  'eviction walks reached end of tree': 328742014,
  'eviction walks restarted': 0,
  'eviction walks started from root of tree': 166309038,
  'eviction walks started from saved location in tree': 48544802,
  'eviction worker thread active': 20,
  'eviction worker thread created': 13435,
  'eviction worker thread evicting pages': 501503267,
  'eviction worker thread removed': 13429,
  'eviction worker thread stable number': 14,
  'files with active eviction walks': 0,
  'files with new eviction walks started': 174319634,
  'force re-tuning of eviction workers once in a while': 7878,
  'forced eviction - do not retry count to evict pages selected to evict during reconciliation': 2721,
  'forced eviction - history store pages failed to evict while session has history store cursor open': 2652,
  'forced eviction - history store pages selected while session has history store cursor open': 14880,
  'forced eviction - history store pages successfully evicted while session has history store cursor open': 12228,
  'forced eviction - pages evicted that were clean count': 12261,
  'forced eviction - pages evicted that were clean time (usecs)': 30563,
  'forced eviction - pages evicted that were dirty count': 282910,
  'forced eviction - pages evicted that were dirty time (usecs)': 1108835767,
  'forced eviction - pages selected because of a large number of updates to a single item': 171045,
  'forced eviction - pages selected because of too many deleted items count': 88,
  'forced eviction - pages selected count': 5234782,
  'forced eviction - pages selected unable to be evicted count': 4939611,
  'forced eviction - pages selected unable to be evicted time': 75319296,
  'hazard pointer blocked page eviction': 3411193,
  'hazard pointer check calls': 507033786,
  'hazard pointer check entries walked': Long('6127148292'),
  'hazard pointer maximum array length': 2,
  'history store table insert calls': 751612906,
  'history store table insert calls that returned restart': 0,
  'history store table max on-disk size': 0,
  'history store table on-disk size': Long('22507274240'),
  'history store table reads': 1,
  'history store table reads missed': 0,
  'history store table reads requiring squashed modifies': 0,
  'history store table resolved updates without timestamps that lose their durable timestamp': 0,
  'history store table truncation by rollback to stable to remove an unstable update': 7275,
  'history store table truncation by rollback to stable to remove an update': 39,
  'history store table truncation to remove all the keys of a btree': 0,
  'history store table truncation to remove an update': 29,
  'history store table truncation to remove range of updates due to an update without a timestamp on data page': 0,
  'history store table truncation to remove range of updates due to key being removed from the data page during reconciliation': 0,
  'history store table truncations that would have happened in non-dryrun mode': 0,
  'history store table truncations to remove an unstable update that would have happened in non-dryrun mode': 0,
  'history store table truncations to remove an update that would have happened in non-dryrun mode': 0,
  'history store table updates without timestamps fixed up by reinserting with the fixed timestamp': 0,
  'history store table writes requiring squashed modifies': 8,
  'in-memory page passed criteria to be split': 634736,
  'in-memory page splits': 111872,
  'internal page split blocked its eviction': 145,
  'internal pages evicted': 62160564,
  'internal pages queued for eviction': 6516997,
  'internal pages seen by eviction walk': Long('3931843331'),
  'internal pages seen by eviction walk that are already queued': 2715118,
  'internal pages split during eviction': 2504,
  'leaf pages split during eviction': 115096679,
  'maximum bytes configured': Long('96636764160'),
  'maximum milliseconds spent at a single eviction': 41,
  'maximum page size seen at eviction': 10579040,
  'modified pages evicted': 285306123,
  'modified pages evicted by application threads': 2978,
  'operations timed out waiting for space in cache': 0,
  'overflow keys on a multiblock row-store page blocked its eviction': 0,
  'overflow pages read into cache': 0,
  'page split during eviction deepened the tree': 5,
  'page written requiring history store records': 140600952,
  'pages currently held in the cache': 183870,
  'pages evicted by application threads': 158968,
  'pages evicted in parallel with checkpoint': 214114332,
  'pages queued for eviction': 2015353761,
  'pages queued for eviction post lru sorting': 2015991125,
  'pages queued for urgent eviction': 96013667,
  'pages queued for urgent eviction during walk': 267220,
  'pages queued for urgent eviction from history store due to high dirty content': 0,
  'pages read into cache': 328889408,
  'pages read into cache after truncate': 0,
  'pages read into cache after truncate in prepare state': 0,
  'pages removed from the ordinary queue to be queued for urgent eviction': 3887,
  'pages requested from the cache': Long('19954273331'),
  'pages seen by eviction walk': Long('15722170027'),
  'pages seen by eviction walk that are already queued': 156099924,
  'pages selected for eviction unable to be evicted': 35366638,
  'pages selected for eviction unable to be evicted because of active children on an internal page': 184980,
  'pages selected for eviction unable to be evicted because of failure in reconciliation': 4569982,
  'pages selected for eviction unable to be evicted because of race between checkpoint and updates without timestamps': 0,
  'pages walked for eviction': Long('540580921486'),
  'pages written from cache': 677215717,
  'pages written requiring in-memory restoration': 128931795,
  'percentage overhead': 8,
  'recent modification of a page blocked its eviction': 0,
  'reverse splits performed': 34287,
  'reverse splits skipped because of VLCS namespace gap restrictions': 0,
  'the number of times full update inserted to history store': 417535460,
  'the number of times reverse modify inserted to history store': 334077349,
  'total milliseconds spent inside reentrant history store evictions in a reconciliation': 0,
  'tracked bytes belonging to internal pages in the cache': 570786066,
  'tracked bytes belonging to leaf pages in the cache': Long('9025297571'),
  'tracked dirty bytes in the cache': 1294875087,
  'tracked dirty pages in the cache': 11464,
  'uncommitted truncate blocked page eviction': 0,  'unmodified pages evicted': 186358588
}

It resolved after I restart. But Its expected to be this issue happen again in 4 days.

So Any recommendations are welcomed.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.