0

I've ran two performance test on a MongoDB server with identical environment/settings AFAIK. I found that the throughput was 10% apart.

When I inspected the mongostat logs, I found that Mongodb in the first test with faster throughput used 95% of its cache_size. In the second test, it stayed at 80%.

I rebooted the server system between the runs, but didn't change any of the settings of the server or the client. mongodb log reports the exact same setting in both tests:

First test: 
2019-08-14T00:42:32.588-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

Second test (after reboot):
2019-08-14T01:11:15.722-0400 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=96122M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),

Here is a sample from mongostat of the first test:

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0 104209     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.7G 107|0 128|0  27.0m   40.2m  272 Aug 14 00:51:13.759
    *0 102610     *0     *0       0     0|0  0.0% 94.9%       1 96.7G 94.6G 109|0 128|0  26.6m   39.6m  272 Aug 14 00:51:44.092
    *0 104724     *0     *0       0     9|0  0.0% 94.9%       0 96.7G 94.6G  87|0 128|0  27.1m   40.4m  272 Aug 14 00:52:13.767

And a sample from the second test:

insert  query update delete getmore command dirty  used flushes vsize   res   qrw   arw net_in net_out conn                time
    *0  94009     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 80.9G 127|0  81|0  24.4m   36.3m  272 Aug 14 01:17:29.289
    *0  93680     *0     *0       0     0|0  0.0% 80.3%       0 83.1G 81.0G   0|0 107|0  24.3m   36.2m  272 Aug 14 01:17:59.119
    *0  93068     *0     *0       0     9|0  0.0% 80.9%       1 83.1G 81.0G 130|0 128|0  24.1m   35.9m  272 Aug 14 01:18:29.123
    *0  92599     *0     *0       0     0|0  0.0% 80.0%       0 83.1G 80.9G 137|0 128|0  24.0m   35.8m  272 Aug 14 01:18:59.186
  • What could have caused the difference in the cache percentage usage and throughput?
  • is it safe to change explicitly this settings so Mongodb uses more of its cache?
  • Can I do this setting change from the command line ? (I'm starting mongodb from command line and not as a service)
  • 1
    What MongoDB version are you using? Please also post what queries are you using – kevinadi Aug 19 at 4:29
0
  1. What could have caused the difference in the cache percentage usage and throughput? MongoDB uses both file system as well as WiredTiger's cache when required

  2. Is it safe to change explicitly this settings so Mongodb uses more of its cache? Yes. It is totally safe to change the cache settings

  3. Can I do this setting change from the command line ? The setting has be done in your mongod config file under storage option

Refer the below link on how you should decide on the cache size https://docs.mongodb.com/manual/faq/storage/

Similar to the below:

storage:
  engine: wiredTiger
  wiredTiger:
    engineConfig:
      cacheSizeGB: 7

Thank you!

-Anban

  • Thanks, but AFAIK, the settings in both tests where exactly the same. My system has 192GB RAM. so wiredTiger takes by default 50% which is almost 96GB. In the first test MongoDB uses 97GB = which is 95% of the allocated, while the second test is using 81G which is only 80% of the allocated cache. My question is what why in the first test, MongoDB stopped loading more data to the cache once it reached 80% while in the first test 95% of the cache_size was used???? – Yuval Aug 20 at 12:20
0

By default in MongoDB 3.6.4, WiredTiger will try to keep dirty percentage (size of modified data in the cache relative to configured cache size) to below 5% of configured cache size, and the overall cache usage below 80% of configured cache size. The default cache size is described in WiredTiger memory in the documentation

Once the percentages goes beyond these numbers, WiredTiger will try much harder by making application threads that wants to do an operation in WiredTiger (such as the threads doing your inserts) to help with cache eviction (which could involve flushing to disk) before it's allowed to do the work. This acts as a built-in throttling mechanism to ensure that the storage engine and the disk are not extremely overwhelmed by incoming work

Thank you!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.