0

MongoDB crashed twice in 6 hours today due to invariant() failure. We have been using MongoDB 3.6.3 for more than a year now, but this is the first time that we saw this issue. The extract from the log files is given below for reference -

Log from 1st failure:

2019-02-27T08:44:51.411+0530 E STORAGE [conn3385299] WiredTiger error (12) [1551237291:411637][2344:2000827264], file:collection-17055--5635302636308412295.wt, WT_CURSOR.search: C:/MREYE/MongoDB/data\collection-17055--5635302636308412295.wt: handle-read: ReadFile: failed to read 4096 bytes at offset 720363520: Not enough storage is available to process this command.

: Not enough space 2019-02-27T08:44:51.412+0530 F -
[conn3385299] Invariant failure: seekRet resulted in status UnknownError: 12: Not enough space at src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp 1734

2019-02-27T08:44:51.412+0530 F - [conn3385299]

***aborting after invariant() failure

Log from 2nd failure:

2019-02-27T13:44:11.594+0530 E STORAGE [conn20] WiredTiger error (12) [1551255251:594637][54524:2000827264], file:collection-17071--5635302636308412295.wt, WT_CURSOR.search: memory allocation of 28696 bytes failed: Not enough space

2019-02-27T13:44:11.594+0530 F - [conn20] Invariant failure: seekRet resulted in status UnknownError: 12: Not enough space at src\mongo\db\storage\wiredtiger\wiredtiger_record_store.cpp 1734

2019-02-27T13:44:11.595+0530 F - [conn20]

***aborting after invariant() failure

Note:

The MongoDB is on C drive and there is sufficient space left (276 GB) on that drive. The MongoDB data folder size itself is only 65 GB. Any idea what might have caused this issue?

0

It looks like the problem is not with your physical memory.

What is the RAM size you are using? Are there any other applications that are consuming more RAM memory?

Refer the below thread for more details

Invariant failure: seekRet resulted in status

  • Thanks for your response! the RAM size is 32 GB. When I checked memory usage immediately after the crash, there was 14 GB available. Also, we do not have any other resource-intensive applications running on that server. Moreover, nothing was changed on the server in the last 1 year, and this was the first time we saw this invariant failure issue. – Chandana Mar 8 at 6:32
  • Is the issue still persists? – Mani Mar 8 at 6:34
  • I restarted the server after the 5th crash and we haven't seen this issue since then - (this was 4 days back). Nevertheless, I wanted to understand what could have gone wrong suddenly without any change / update / new install done on the system. I was wondering if there was some fault with the storage layer – Chandana Mar 8 at 10:06
  • Were you creating any index on existing collections that time on a large collection or any memory-consuming operation? Are you using default wired tiger cache size or any specific size mentioned in the config? Also you can search for any issues already reported like this is mongo in jira.mongodb.org/projects/SERVER/issues, if you don't see any existing issues, raise a new issue. – Mani Mar 8 at 18:50
  • Yes, the crashes were mostly seen during data import. So it is possible that it was at the time when indexes were being created. Nevertheless, the same data import jobs have been running everyday on schedule since the last 1.5 years, but this issue had never occurred before. Also, we are using default wired tiger cache size. – Chandana Mar 11 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.