usage percentage of Mongodb internal cache

Question

I've ran two performance test on a MongoDB server with identical environment/settings AFAIK. I found that the throughput was 10% apart.

When I inspected the mongostat logs, I found that Mongodb in the first test with faster throughput used 95% of its cache_size. In the second test, it stayed at 80%.

I rebooted the server system between the runs, but didn't change any of the settings of the server or the client. mongodb log reports the exact same setting in both tests:

First test: 
2019-08-14T00:42:32.588-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

Second test (after reboot):
2019-08-14T01:11:15.722-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

Here is a sample from mongostat of the first test:

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0 104209     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.7G 107|0 128|0  27.0m   40.2m  272 Aug 14 00:51:13.759
    *0 102610     *0     *0       0     0|0  0.0% 94.9%       1 96.7G 94.6G 109|0 128|0  26.6m   39.6m  272 Aug 14 00:51:44.092
    *0 104724     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.6G  87|0 128|0  27.1m   40.4m  272 Aug 14 00:52:13.767

And a sample from the second test:

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0  94009     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 80.9G 127|0  81|0  24.4m   36.3m  272 Aug 14 01:17:29.289
    *0  93680     *0     *0       0     0|0  0.0% 80.3%       0 83.1G 81.0G   0|0 107|0  24.3m   36.2m  272 Aug 14 01:17:59.119
    *0  93068     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 81.0G 130|0 128|0  24.1m   35.9m  272 Aug 14 01:18:29.123
    *0  92599     *0     *0       0     0|0  0.0% 80.0%       0 83.1G 80.9G 137|0 128|0  24.0m   35.8m  272 Aug 14 01:18:59.186

• What could have caused the difference in the cache percentage usage and throughput?
• is it safe to change explicitly this settings so Mongodb uses more of its cache?
• Can I do this setting change from the command line ? (I'm starting mongodb from command line and not as a service)

Answer 1

By default in MongoDB 3.6.4, WiredTiger will try to keep dirty percentage (size of modified data in the cache relative to configured cache size) to below 5% of configured cache size, and the overall cache usage below 80% of configured cache size. The default cache size is described in WiredTiger memory in the documentation

Once the percentages goes beyond these numbers, WiredTiger will try much harder by making application threads that wants to do an operation in WiredTiger (such as the threads doing your inserts) to help with cache eviction (which could involve flushing to disk) before it's allowed to do the work. This acts as a built-in throttling mechanism to ensure that the storage engine and the disk are not extremely overwhelmed by incoming work

Thank you!

Answer 2

1. What could have caused the difference in the cache percentage usage and throughput? MongoDB uses both file system as well as WiredTiger's cache when required

2. Is it safe to change explicitly this settings so Mongodb uses more of its cache? Yes. It is totally safe to change the cache settings

3. Can I do this setting change from the command line ? The setting has be done in your mongod config file under storage option

Refer the below link on how you should decide on the cache size https://docs.mongodb.com/manual/faq/storage/

Similar to the below:

storage:
  engine: wiredTiger
  wiredTiger:
    engineConfig:
      cacheSizeGB: 7

Thank you!

-Anban