3

System: Windows Server 2012 R2, MariaDB 10.2.8

We are currently trying to recover from a quasi-crash (it was a normal shutdown, but mysqld won't start up again) and are having problems with the redo log of InnoDB during start up. If we start with innodb_force_recovery=0, the process gets to the point where it logs

2019-06-14  9:29:41 14580 [Note] InnoDB: Starting final batch to recover 1150 pages from redo log.

and then gets stuck there. Since there are no further log entries from that point it is not clear if it takes more time or silently fails. We tried waiting for 22 hours, with no visible progress. The process ran with <1% CPU usage and unchanging memory usage, so presumably it stops silently.

We then tried increasing numbers of innodb_force_recovery and at 6 the server finally starts up again by skipping the redo log entirely. Unfortunately this also starts with read-only. Thus clearing the log by using innodb_fast_shutdown=0 doesn't appear to do anything on shutdown.

What we need now is a way to clear the redo log with mysqld being offline. Simply deleting the log files ib_logfile0 and ib_logfile1 doesn't seem to work either, since they somehow get recreated during start up, which runs into the same problems as above. We're assuming the data somehow is saved in ibdata1, but we can't delete it from there without also deleting the tablespace.

As a last resort we tried deleting the corrupted database by deleting it's directory from disk (since we can't delete it within MySQL as it only starts in read-only) and then starting with innodb_force_recovery=3, but once again the process tries to perform the redo log and then stops because the database doesn't exist.

It seems the best course of action is to somehow start mysqld without performing the redo log or clearing it beforehand, then deleting the corrupted database and restoring it from a backup. But we can't get it to start in write-mode without trying and failing with the redo log. Is there a way to clear it by directly manipulating the files?

3
  • How big is he database?
    – akuzminsky
    Jun 15, 2019 at 5:44
  • About 110GB in the data directory. Jun 16, 2019 at 8:13
  • The bigger the database the slower it will recover/start. Just give it more (a lot more) time.
    – mlazarov
    Feb 28, 2020 at 13:40

1 Answer 1

1

What you should be doing, is

  • kill mysqld process. Disable the MySQL service (or whatever it is called), if it runs as service.
  • upgrade to latest 10.2 (it is already on 10.2.24 as of today, you are using a very old version)
  • Enable the service and start again.

If it does not start again, take the mysqld process minidump while it is hanging, and submit a bug to JIRA https://jira.mariadb.org/browse/MDEV

This is definitely more helpful than trying to "recover" by deleting Innodb redo logs (this is one of the things you should not be doing).

2
  • You do not know why it hangs, and it is not supposed to be hanging, so it is a bug. if you communicate with MariaDB developers, you and they will get more insight, and the bug, if still present in newest version, will hopefully be fixed. Jun 14, 2019 at 15:38
  • Thank you for the reponse. For the purpose of getting the instance up and running we went with the route of dumping the data and reimporting into a clear instance, But since we still have the problematic instance I might try to solve it with your suggestion on a dev machine. If it still fails then, I will submit a bug report. Jun 16, 2019 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.