0

My Issue

After shutting down and starting my system up, my MySQL daemon gives me this in the logs (the pertinent snippet is as follows):

2021-05-03T12:48:32.458594Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: fut0lst.ic:86:addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA thread 139822345225984
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
12:48:32 UTC - mysqld got signal 6 ;
Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
Thread pointer: 0x7fffe0c79cb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...

This is the second time I'm running into this issue. After struggling for a while the last time, I re-installed mysql server, which worked. I'm puzzled as to why this keeps happening and want to get to the root of the issue this time. I did nothing out of the ordinary (as far as I'm aware) to cause this error.

Troubleshooting

innodb_force_recovery = 3 (not below) in my.cnf enables the daemon to run. According to InnoDB recovery guide and the official link in the error description this has something to do with "So for example: if your MySQL server starts up on mode 3, but not mode 2, it might be a safe assumption to say that the crash has something to do with the transaction rollback process." I've added innodb_fast_shutdown = 0 and innodb_purge_threads = 1 and dropped all the non-system schemas as well, but to no avail.

SHOW FULL PROCESSLIST revealed an event listener and my current process, so I don't think there was much there.

Hdd space and memory is not an issue.

And so...

I'm not certain how to proceed with debugging from here on in. A nudge in the right direction would be appreciated. Thanks.

7
  • Two of the messages have hinted to hardware failure. If you have some bad blocks interfering with an InnoDB file, this could be a bad thing. Have you confirmed the health of the storage device? – matigo May 3 at 13:57
  • @matigo I have seen that mentioned in several places though I certainly hope that's not the case. I'll get some software to scan the drive now. – jvndev May 3 at 14:03
  • check the system log for other anomaly's – nbk May 3 at 15:17
  • @matigo After an overnight scan of several hours I learned that the drive is fine, thankfully. It's one of those things one postpones due to fear of bad news: youtube.com/watch?v=msGvEtmR970 – jvndev May 4 at 6:51
  • @nbk Currently I'm using /var/log/mysql/error.log as a source to track the issue. I did read that the datadir might also contain .err files. I assume they will be relevant to the schemas i.e. mysql.err. I don't see anything like that though which further indicates that this issue can't be due to a corrupt index, hanging rollback or what have you. Which system log are you referring to? – jvndev May 4 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.