Starting with question 2:
I have found no clear answer, but it is now obvious to me that fs snapshots is not sufficient for MongoDB version 4.0.19. We'll upgrade MongoDB if we can and will start using mongodump in addition to the fs snapshots. We'll probably stop MongDB before snapshots are made. At this point I do not know how corrupt files goes undetected by MongoDB, or if a changed backup strategy will protect us. I hope starting MongoDB after backups will at lease expose corrrupted files as soon as they apare.
Question 1 solved:
@Jjussi pointet us to a link that explained how to recover data using the wt-tool: https://www.fatalerrors.org/a/wiredtiger-tool-restores-mongodb-data.html
From docs and forums we initially tought the MongoDB 4.0+ could handle this with the --repair option, but that turned out to be wrong. We compiled the wt-tool and managed to repair the corrupted file. We still lost four days worth of data. All snapshots contained corrupted versions of the file in question, dated well before the snapshots own dates.
Matching collection names with filenames on disk was a challenge. The initial runs with the '--repair' option left log entries for most of the files. I discovered that field names could be found in clear text in the collection files, so thats how I identfied the files not mentioned in the logs.
Please refere to the link abow, but a quick recap would be:
Backup the directory of the corrupt database
Compile wt-tool and repair the corrupt file
Create a new database (delete content of the dbdir if you have a backup), and create the collections you need.
Make a list of collections and their corresponding filenames from the newly created database. I used the code snippet below.
Map collection names and filenames from the old database by checking logfile and/or grep -l for known field names.
Stop mongod, replace newly created files with files from the corrupt database, and ofcourse the file(s) fixed by wt-tool. Keeping the new filenames.
Run /usr/local/bin/mongod --dbpath /var/db/mongodb --repair -vv
Assuming aok so far, start mongodb and open the mongo shell.
Re-index all collections:
db.getCollectionNames().forEach(function(coll_name) {
var coll = db.getCollection(coll_name);
coll.reIndex();
});
Code snippet listing collections an their filenames (without .wt):
function findFileNames() {
var adminDb = db.getSiblingDB("admin");
var dbList = adminDb.runCommand({ "listDatabases": 1 }).databases;
for (var i in dbList) {
var dbToSearch = db.getSiblingDB(dbList[i].name);
var collectionNames = dbToSearch.getCollectionNames();
for(var i = 0; i < collectionNames.length; i++){
var name = collectionNames[i];
var stats = dbToSearch.getCollection(name).stats();
print (dbToSearch + '.' + name + ' ' + stats.wiredTiger.uri);
}
}
}
print (findFileNames())
