We had a previous client who's server crashed. They were running MySql 5 on Debian Lenny and doing daily backups. When they contacted us to help them restore the database from the backups we realized that the backups were only capturing 20% of the data - they were doing 30 days of rolling backups but at some point the db grew so much that it couldn't hold all 30 days and was just erroring out (I guess?). This was all done on their end, we only found out about it after the fact.

The drive from the server was accessible. I was able to copy out the entire /var/lib/mysql directory and save that. They were using InnoDB. Once we realized the backups were useless we created an identical VM with Debian Lenny and same version of MySql with the idea of replacing the /var/lib/mysql folder with the one from the crashed server. We did that, replaced the /var/lib/mysql and the /var/etc/mysql folder as well just in case the configuration had something in it. This seemed like the only logical step to take.

mySql wouldn't start - errors about pagetables and hex dumps. Tried recovery options 1-6 with nothing, mysql wouldn't start. Finally we ran innodbcheck against the ibdata1 file and it just came back with one issue 'Fail: page 2965 invalid'. It's a 200 GB ibdata1 file and I believe 99% of it are documents they were storing in the database. Is there a way to correct this error and get this working again?

New contributor
esen is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • Are you trying to start mysqld with ibdata1 and the ib_logfiles intact? If so, you could try starting an instance with the ibdata1 file alone, without the ib_logfiles. It sounds like it's trying to do crash recovery on startup, and this is touching the corrupt page(s). If you let it initialize new log files, it might skip that part, and if that allows it to start up, you could try to mysqldump as many tables as you can without touching the corrupt parts. Jun 22 at 20:41
  • Other than that, I am afraid I don't have any ideas. This database might be unrecoverable. This might just be a learning experience for the client: a backup system must verify the backups, or they're no good. Jun 22 at 20:42
  • Thanks Bill - I did try that but no go. As a last hail mary I just plugged the drive into a different computer hoping it would at least boot and start mysql. Sure enough it booted, a little wonky but still working. I was able to myqdump everything except a range in one table. I did the dump direct to an external drive mounted because I couldn't trust that drive. It looks like we saved most of the data but we haven't dumped the DocumentData table which makes up the bulk of the database although not as important as the other tables.
    – esen
    Jun 22 at 23:07
  • You could also try recovering a subset of the DocumentData table by using mysqldump --where '<expression>' to dump ranges of rows, hoping to avoid touching the corrupted page(s). Good luck. Jun 22 at 23:26


Your Answer

esen is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.