Cannot restart or repair mongodb database after the crash (***aborting after invariant() failure) [closed]

Question

After an unexpected crash, I can't restart or --repair the mongodb database anymore. While restarting the server with systemctl restart mongodb I am getting the following:

2020-09-16T20:25:23.835+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1600280723:835687][23452:0x7f4a58059a40], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_block_read_off, 302: sizeStorer.wt: fatal read error: WT_ERROR: non-specific WiredTiger error Raw: [1600280723:835687][23452:0x7f4a58059a40], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_block_read_off, 302: sizeStorer.wt: fatal read error: WT_ERROR: non-specific WiredTiger error

2020-09-16T20:25:23.835+0200 E STORAGE [initandlisten] WiredTiger error (-31804) [1600280723:835692][23452:0x7f4a58059a40], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1600280723:835692][23452:0x7f4a58059a40], file:sizeStorer.wt, WT_SESSION.open_cursor: __wt_panic, 523: the process must exit and restart: WT_PANIC: WiredTiger library panic

2020-09-16T20:25:23.835+0200 F - [initandlisten] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 420 2020-09-16T20:25:23.835+0200 F - [initandlisten]

***aborting after fassert() failure

2020-09-16T20:25:23.851+0200 F - [initandlisten] Got signal: 6 (Aborted).

When I am trying repair with mongod --dbpath **/** --repair I am getting the following:

2020-09-16T20:05:09.921+0200 I STORAGE [initandlisten] repairDatabase admin
2020-09-16T20:05:09.921+0200 I STORAGE [initandlisten] Repairing collection admin.system.users
2020-09-16T20:05:09.921+0200 I STORAGE [initandlisten] Verify failed on uri table:collection-0-4776197172926482174. Running a salvage operation.
2020-09-16T20:05:09.922+0200 F - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database_impl.cpp 228
2020-09-16T20:05:09.922+0200 F - [initandlisten]

***aborting after invariant() failure

2020-09-16T20:05:09.929+0200 F - [initandlisten] Got signal: 6 (Aborted).`

Answer 1

After an unexpected crash, I can't restart ... the mongodb database

Time to exercise your Recovery Strategy, put your Recovery Plan into operation and get your database back from the Backups you've taken of it.

Your database is actually remarkably resilient.
Left to its own devices (and all other things being equal) it will recover from a server crash all by itself. This is called Instance Recovery and all the "real" DBMSs do it completely automatically.

However, if your database can't recover on its own, then it's time to seriously consider putting that database out of it's misery and resurrecting it from Backup. Blithely jumping with "repair" tools is a Sure-Fire(TM) way to lose Data.

Now, don't get me wrong - losing Data is not always bad ... but only if it's written down in your Recovery Strategy, which would define, amongst other things:

• How long you can take to recover your database, and
• How much Data you're allowed to lose in doing so.

Your Backup plan should support your Recovery Strategy.

NB: No sensible Recovery Strategy should ever rely on being able to "repair" the database which, by that point, is basically broken. Always make sure you can get back to a stable, known, Good State (i.e. the last Backup) and that you have practised performing that recovery work.