I have a Centos 7.9 server with MariaDB version 10.2.43. All of a sudden on May 2 2022, I have started receiving mysql down alerts. On checking I could see below log information in logs.
Error 2:- May 2 22:09:03 server1 mysqld: 2022-05-02 22:09:03 140534819583744 [ERROR] InnoDB: Page [page id: space=57618, page number=185] log sequence number 6629091300064 is in the future! Current system log sequence number 6629091254594. May 2 22:09:03 server1 mysqld: 2022-05-02 22:09:03 140534819583744 [ERROR] InnoDB: Your database may be corrupt or you may have copied the InnoDB tablespace but not the InnoDB log files. Please refer to https://mariadb.com/kb/en/library/innodb-recovery-modes/ for information about forcing recovery.
I have managed to recover the data using innodb_force_recovery to the value of 5, restore the databases from backups/dump. I have some doubts regarding the issue. I have checked entire log information for Mariadb and I could see below log entry repeatedly logged days before the InnoDB Crash.
[ERROR] mysqld: Got error 'Could not get an exclusive lock; file is probably in use by another process' when trying to use aria control file '/var/lib/mysql/aria_log_control'
I wanted to understand / Clarify few points regarding this issue.
Is the aria_log_control error a critical error and is it connected to InnoDB Crash? Why this error?
How come the Page log sequence number is in the future for Mariadb service? Is it related to some kind of disk corruption or memory issue?
How to monitor this on production environment and prevent the same?
Is this a bug with maridb 10.2.43? Will updating to latest stable 10.6 version will fix it?