2

Background

I have a directory full of .gz files that were produced by mongodump on a mongod running MongoDB 3.4.x, which I am writing into a different instance running MongoDB 3.4.2 using mongorestore ... --numInsertionWorkersPerCollection 8 --drop --nsFrom ... --nsTo .... All but one are getting loaded properly. They all take between 1 and 3 hours to load the data, and then another 2-4 hours to rebuild indexes. The problematic file hangs before the step below, and I usually kill the process externally after about a day. I've reproduced the scenario 4 or 5 times.

finished restoring database.collection (N documents) 

During this time the mongorestore process is still visible from the shell, but does not have a corresponding entry in db.currentOp(), and the size of the database is not increasing.

These errors were returned right before the log line "restoring indexes for collection database.collection from metadata":

2017-03-13T09:30:04.218-0400    demux namespaceHeader: {database collection true 726142452659775357}
2017-03-13T09:30:04.218-0400    demux checksum for namespace database.collection is correct (726142452659775357), 54599167880 bytes
2017-03-13T09:30:04.218-0400    demux finishing (err:corruption found in archive; I/O error reading length or terminator ( gzip: invalid header ))

I have found a couple reports of mongorestore hanging (e.g.), and some documentation of issues with /s in namespaces in older versions (e.g.), but I haven't found a description of quite my situation.

Question

Now I find myself with some indication that a file I have loaded may have been corrupt, or at least had a corrupt header. Issuing realistic queries to the collection in question and its indices returns results that look reasonable. How can I definitively detect or rule out "corruption" of this type? What indeed does "corrupt" mean in this context? Can I expect to find it in the database, or only within the zip file?

Update: At @mustaccio's suggestion I ran gzip -t on the problematic file and it reported "trailing garbage". Is this sufficient to pin the problem reported by MongoDB on a bad input and stop looking for corruption in the database?

  • Can you confirm the specific version of mongodump used and your command-line options? In particular I'm wondering if you used mongodump's built-in compression and archiving or compressed after the fact using a standalone gzip program. – Stennie Mar 22 '17 at 0:32
  • So this was definitely using the built-in --gzip option for mongodump and mongorestore in MongoDB 3.2+? You mention reproducing the problem 4 or 5 times -- was this with different backups but always affecting the same collection, or were different backups or collections affected? – Stennie Mar 23 '17 at 7:07
  • @Stennie The problem appeared on the same collection in the same file each time. I can't guarantee that the archive was generated with mongodump --archive v.3.2+, but circumstantial evidence strongly points in that direction. Thank you for your persistence! I see the note in the gzip docs you linked that the -q option precludes trailing garbage from appearing, and I am far from having expertise in MongoDB or Go, but I don't see that option passed in the mongodump code. Would you expect mongodump --archive to guarantee no trailing garbage? – WAF Mar 23 '17 at 13:11
  • I've made a few attempts to reproduce the issue with the provided details, but haven't managed to (yet). It's interesting that you always have a problem with the same collection; to be extra clear, can you confirm this happened repeatedly in different backups rather than successive mongorestore attempts? Does this collection have any special options (eg. capped, TTL index, ...)? Also, what storage engine and O/S are you using? I've been assuming default storage engine for 3.2+ (WiredTiger) and some flavour of Linux. If you can repro the problem at will, mongodump --version would be great. – Stennie Mar 23 '17 at 13:33
  • @Stennie Sorry for not clarifying. The repeated attempts were only to mongorestore the same archive file. The original database no longer exists in the form it did at the time of the mongodump. There are no special options on the collection in question, just a healthy amount of data ("storageSize" : 18050703360), and a bunch of indexes. Both mongods are using wiredTiger and running on Linux. – WAF Mar 23 '17 at 19:54
2

demux finishing (err:corruption found in archive; I/O error reading length or terminator ( gzip: invalid header ))

This error message refers to processing of the gzip archive rather than the BSON data within. Your test with gzip -t implies that the contents are likely OK and the error may be due to ignorable padding at the end of the archive (see: gzip complains with trailing garbage ignored).

All but one are getting loaded properly. That one reproducibly hangs (indefinitely)

If the problem is easily reproducible, I would raise a bug report in the MongoDB Jira TOOLS project including more details such as the specific versions of mongodump and mongorestore and command line parameters used as well as how long you waited. It seems reasonable for mongorestore to have some sort of timeout or error handling for this case rather than hanging indefinitely; I also wouldn't expect mongodump to create files that the same version of mongorestore cannot handle.

Now I find myself with some indication that a file I have loaded may have been corrupt, or at least had a corrupt header. Issuing realistic queries to the collection in question and its indices returns results that look reasonable. How can I definitively detect or rule out "corruption" of this type? What indeed does "corrupt" mean in this context? Can I expect to find it in the database, or only within the zip file?

It's unlikely that you would be able to restore corrupt BSON documents into MongoDB, but there may be some nuances depending on how/where data is corrupted. In most cases invalid data will trigger an exception/assertion when the server attempts to read the BSON, but valid BSON does not necessarily mean that the field contents are as expected.

To be entirely certain you would have to compare the document counts or checksums of the restored data against the original data (assuming you still have a copy other than the gzip archive).

  • 1
    There's a MongoDB server command that will probably help, but could potentially return a false negative depending on data provenance and versions of tools/server involved. You can try using the dbHash command to calculate an MD5 checksum against all documents in a collection. This command is sensitive to BSON field reordering and I have seen a few historical edge cases (usually related to replication) where documents may have the same contents on a field-by-field comparison but fail to match based on the order of fields. – Stennie Mar 23 '17 at 7:21
  • I expect dbHash will be fine for a mongodump and mongorestore comparison as long as there are no writes to the source collection between the time you dump & restore. However, if the results differ I would try an alternate verification approach rather than assuming corruption. dbHash also involves reading all of the documents in a collection in order to calculate the MD5 checksum, so less resource intensive approaches such as verifying document counts and data (not file) size may be preferable if they satisfy your expectations on verification. – Stennie Mar 23 '17 at 7:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.