4

I am using MongoDB (3.2.6) as storage for my application and since a couple of weeks MongoDB started to crash usually every 1 or 2 weeks. The crash seems to come from an issue during WiredTiger memory allocation here is the full stacktrace of the last crash:

2016-10-09T18:25:36.389+0200 E STORAGE  [thread1] WiredTiger (12) [1476030336:325695][993:0x7f559c19a700], file:index-1108-8065571460375661294.wt, WT_SESSION.checkpoint: memory allocation of 18624 bytes failed: Cannot allocate memory
2016-10-09T18:25:36.547+0200 E STORAGE  [thread1] WiredTiger (12) [1476030336:422567][993:0x7f559c19a700], file:index-1108-8065571460375661294.wt, WT_SESSION.checkpoint: checkpoints cannot be dropped when in-use: Cannot allocate memory
2016-10-09T18:25:37.347+0200 I COMMAND  [conn10793] command admin.$cmd command: ping { ping: 1 } keyUpdates:0 writeConflicts:0 numYields:0 reslen:37 locks:{} protocol:op_query 208ms
2016-10-09T18:25:37.386+0200 E STORAGE  [thread1] WiredTiger (12) [1476030337:254969][993:0x7f559c19a700], checkpoint-server: checkpoint server error: Cannot allocate memory
2016-10-09T18:25:38.448+0200 E STORAGE  [thread1] WiredTiger (-31804) [1476030337:484838][993:0x7f559c19a700], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-10-09T18:25:38.677+0200 I -        [conn10894] Fatal Assertion 28559
2016-10-09T18:25:38.994+0200 I -        [WTJournalFlusher] Fatal Assertion 28559
2016-10-09T18:25:38.994+0200 I -        [TTLMonitor] Fatal Assertion 28559
2016-10-09T18:25:40.978+0200 I -        [thread1] Fatal Assertion 28558
2016-10-09T18:25:41.207+0200 I -        [thread1] 

The server it's running on has 7Gb RAM and is exclusively used for Mongo.

free -m
             total       used       free     shared    buffers     cached
Mem:          6811       6140        670          0         28        634
-/+ buffers/cache:       5478       1332
Swap:         2047       1009       1038

Several dbs are running on it with a total size around 925MB while WiredTiger cache status is :

db.serverStatus().wiredTiger.cache
"bytes currently in the cache" : 969927022,
"bytes read into cache" : 800513540,
"bytes written from cache" : 1349774879,
"maximum bytes configured" : 3221225472, 
[...]

According to the documentation, WiredTiger should use by default

60% of RAM minus 1 GB, or [...] https://docs.mongodb.com/manual/faq/storage/#to-what-size-should-i-set-the-wiredtiger-cache

Based on 7GB RAM we get ~ 3.2Gb so it seems good.

Now how can 925Mb databases can bust a 3.2Gb cache ? I know that we have to take the indexes into account but even by indexing every fields (which is not the case) it should not fill the 2+ Gb remaining ? Last crashed occurred yesterday so the provided stats are just after a fresh restart. Could it be that the way I'm using Mongo a memory leak occurs somewhere slowly increasing the memory consumption ? I may completely miss the point as I am new to Mongo so any leading advice or article would be a relief.

I found this issue : https://jira.mongodb.org/browse/SERVER-24408 but it does not provide any further explanations.

EDIT 12/10:

3 days later the cache status is the following :

"bytes currently in the cache" : 1319874144,
"bytes read into cache" : 1054001990,
"bytes written from cache" : 8936911271,
  • What's the SHMMAX set to on the server? – Philᵀᴹ Oct 10 '16 at 10:24
  • From ipcs -lm > max total shared memory (kbytes) = 8388608 – DessDess Oct 10 '16 at 10:32
0

Potentially the unix box is forking the process from your application and causing a duplicate process to try to allocate the same amount of memory but crashing when it does so.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.