1

It is stated in the MongoDB documentation FAQ (https://docs.mongodb.com/manual/faq/storage/#wiredtiger-storage-engine) :

Avoid increasing the WiredTiger internal cache size above its default value.

which should be 50 % ou 60 % of RAM by default

and

The storage.wiredTiger.engineConfig.cacheSizeGB limits the size of the WiredTiger internal cache. The operating system will use the available free memory for filesystem cache, which allows the compressed MongoDB data files to stay in memory.

I don't understand this statement. Does the WT cache contains the working set in terms of compressed data and index ? If so, do we need to keep extra memory for some kind of memory mapped files for MongoDB data files ?

My db.serverStatus().mem shows this :

{
    "bits" : 64,
    "resident" : 10518,
    "virtual" : 12131,
    "supported" : true,
    "mapped" : 0,
    "mappedWithJournal" : 0
}

So I would guess I've no memory mapped bytes, then I can increase de WT cache ?

Thanks !

Improve this question

2 Answers 2

Reset to default
3

Do we need to keep extra memory for some kind of memory mapped files for MongoDB data files

The mem.mapped and mem.mappedWithJournal metrics in serverStatus output are only applicable to the MMAPv1 storage engine, so are expected to be 0 with WiredTiger. These metrics predate the storage engine API (and WiredTiger) so will likely be moved to a storage-engine specific section of serverStatus at some point.

Does the WT cache contains the working set in terms of compressed data and index ?

The WiredTiger internal cache is used for uncompressed data which is a different representation from the compressed data format on disk. Memory outside the WiredTiger cache is available for filesystem cache (which caches the on-disk representation, including compression) as well as other temporary RAM requirements such as per-connection overhead (1MB per connection), in-memory sorts, aggregations, and JavaScript contexts.

Your working set is the total set of data and indexes that your application frequently accesses. For best performance this should ideally fit into RAM. A larger WiredTiger cache will allow for more uncompressed data in RAM at the expense of memory available for other purposes. If your data compresses significantly, more free memory for the filesystem cache will allow for a larger total working set in RAM (uncompressed + compressed data).

So I would guess I've no memory mapped bytes, then I can increase de WT cache ?

As per the MongoDB documentation, you should generally leave the WiredTiger cache at the default size (or possibly even decrease this). You can always try to experiment with tuning the cache size for your workload and deployment, but setting this too large may adversely affect performance.

Improve this answer
4
  • Thanks for your answer, very helpful. I'm still not sure I understand fully. Does it mean that the compressed data in memory is just memory-mapped pages of MongoDB data managed by the kernel ? If so, it would mean that the WT cache and the memory-mapped pages can contain duplicate information, one handled by the WT cache, and the other handled by the kernel. If so, what's the point of keeping both of them ? Thanks !
    – rudi bruchez
    Jun 12, 2017 at 19:56
  • Compressed data in memory is stored in the filesystem cache. This is normal O/S file caching, not a memory-mapped view of the data file (as would be the case with a memory-mapped storage engine like MMAPv1). WiredTiger uses block-level compression to efficiently compress pages of data containing multiple documents. In order to manipulate individual documents, pages from the data files have to be uncompressed into the internal WiredTiger cache. Reading compressed pages from the filesystem cache is faster than reading those from disk.
    – Stennie
    Jun 12, 2017 at 22:46
  • @rudibruchez For more background, see: New Compression Options in MongoDB 3.0 and Does the MongoDB 3.2 WiredTiger compression include stuff stored in RAM.
    – Stennie
    Jun 12, 2017 at 22:50
  • thanks for the clarification, I guess it is still debatable as how to balance both caches. I've seen that the WiredTiger developers changed the rule from 50 % to 60 % RAM for the internal cache, and some say that increasing the WT cache improved their performance. I'll keep investigating
    – rudi bruchez
    Jun 13, 2017 at 10:54
0

Answer is no! First, because you are watching wrong information. Because you are using wiredTiger, you should look db.serverStatus().wiredTiger.cache

Lot's of information but

"bytes currently in the cache" : 88404

Tell you current situation

"maximum bytes configured" : 19327352832

Maximum cache size (50% of memory)

"bytes read into cache" : 43999,
"bytes written from cache" : 88126

How much data has been read from disk (and wrote back) since startup

MongoDB also needs additional memory outside of this cache for aggregations, sorting, connection management, and the like, so it is important to make sure you leave MongoDB with enough memory to do its work. If not, there is a chance MongoDB can get killed by the OS Out of memory (OOM) killer.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.