Won't start: InnoDB: Corrupted page ... Tablespace was not found

Question

After a server crash (Ubuntu 16.04), when I try to start Mysql (Mysql 5.7) using innodb_force_recovery=0 it doesn't start and error.log shows:

InnoDB: Checksum mismatch in datafile: ./panel_financiero_v2/kpis_analytics.ibd, Space ID:93, Flags: 33. Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-troubleshooting-datadict.html for how to resolve the issue.
InnoDB: Corrupted page [page id: space=93, page number=0] of datafile './panel_financiero_v2/kpis_analytics.ibd' could not be found in the doublewrite buffer.
InnoDB: Tablespace 93 was not found at ./panel_financiero_v2/kpis_analytics.ibd.
InnoDB: Set innodb_force_recovery=1 to ignore this and to permanently lose all changes to the tablespace.
InnoDB: Cannot continue operation.

I just can start Mysql using innodb_force_recovery=6 (I can't start Mysql using innodb_force_recovery=1 as the error message advises). The .ibd and .frm files of the troubled table are in the right directory (no empty files); I even tried (just in case, with no real hope) to delete these two files (moving them to another directory) but it doesn't work neither.

Any way to repair or recover the table (bearing in mind that innodb_force_recovery=6 starts Mysql in read-only mode and I even can't create a new table)? Any way, at least, to start mysql normally (with innodb_force_recovery=0) even losing the information of that troubled table?

Answer 1

https://dev.mysql.com/doc/refman/5.7/en/repair-table.html says:

REPAIR TABLE works for MyISAM, ARCHIVE, and CSV tables.

In other words, REPAIR TABLE does nothing for InnoDB. InnoDB has its own automatic crash recovery that it runs on startup.

But InnoDB crash recovery only works for lost changes that were in progress at the time of the crash. It uses the redo log and the doublewrite buffer to reconstruct lost dirty pages. This process cannot reconstruct data that was corrupted at rest.

The error message actually tells you what to do:

InnoDB: Set innodb_force_recovery=1 to ignore this and to permanently lose all changes to the tablespace.

Read more details at https://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html:

1 (SRV_FORCE_IGNORE_CORRUPT) Lets the server run even if it detects a corrupt page. Tries to make SELECT * FROM tbl_name jump over corrupt index records and pages, which helps in dumping tables.

Start the server with innodb_force_recovery=1 after moving your tablespace files back where they belong.

Then you can dump the table using mysqldump, and it should read the pages it can read, skipping corrupted pages. Then you can make a new table to import from the dump:

CREATE TABLE mytable_new LIKE mytable;
RENAME TABLE mytable TO mytable_bad, mytable_new TO mytable;

Then re-import your dumped data.

I assume you do not need help with dumping a single table and importing the dump file.

---

Re your comment:

It seems like you have more extensive corruption. I suggest the following steps:

1. Start the MySQL server with innodb_force_recovery=6
2. Use mysqldump to dump all of the data.
3. Shut down MySQL server.
4. Back up the contents of your datadir just in case you want to get professional help to see if they can recover more of the data. Try https://twindb.com, they are the best experts in MySQL database recovery.
5. Initialize a new datadir (I would use a new disk device, and take the old one out of service because it now has unknown damage).
6. Restore the data dump you created with mysqldump.

Answer 2

We had the same problem in a Bitnami Wordpress appliance. We researched many websites and procedures to resolve this issue. None of the researched articles, blogs and posts had the solution we needed, but we were able to piece together the following.

1. review log file:

   cd /opt/bitnami/mysql tail -400 mysqld.log

ERRORS LISTED:

[ERROR] [MY-012224] [InnoDB] Checksum mismatch in datafile: ./bitnami_wordpress/wp_redirection_404.ibd 
[ERROR] [MY-012237] [InnoDB] Corrupted page [page id: space=2126, page number=0] of datafile './bitnami_wordpress/wp_redirection_404.ibd' could not be found in the doublewrite buffer. 
[ERROR] [MY-013183] [InnoDB] Assertion failure: fil0fil.cc:5551:err == DB_SUCCESS || err == DB_INVALID_ENCRYPT 

2. Identify problem and associated object:

   data/bitnami_wordpress/wp_redirection_404.ibd

3. Remove problem file:

   mv data/bitnami_wordpress/wp_redirection_404.ibd /tmp/.

4. Modify config file to start MySQL ignoring errors:

   vi /opt/bitnami/mysql/my.cnf

-- add 'innodb_force_recovery = 1' to end of section "[mysqld]"

5. Start MySQL in recovery mode:

   /opt/bitnami/ctlscript.sh start mysql

   /opt/bitnami/ctlscript.sh status MySQL

6. Modify config file to start MySQL in NORMAL mode:

   vi /opt/bitnami/mysql/my.cnf

-- remove 'innodb_force...' from section "[mysqld]"

7. Restart MySQL to verify it starts correctly:

   /opt/bitnami/ctlscript.sh start mysql

   /opt/bitnami/ctlscript.sh status MySQL

8. Verify Wordpress website renders correctly: Browse to http://websiteurl.company.com

9. Restore MySQL database from latest backups:

   • exit 'root' session to 'bitnami' account session

--Clear local MySQL database

cd /opt/bitnami/mysql/data/${dbname} ; /home/bitnami/blankwpdb.sh ${dbname} ${dbuser}

--Restore MySQL database from Primary Backup

cd /opt/bitnami/mysql/data/${dbname} ; cat ${current_mysql_bkup} | /opt/bitnami/mysql/bin/mysql -u ${dbuser} -p${dbpw} ${dbname}

--Restart MySQL and Apache HTTP Services

/opt/bitnami/mysql/scripts/ctl.sh restart
/opt/bitnami/apache2/scripts/ctl.sh restart

=====/blankwpdb.sh======
#!/bin/bash
EXPECTED_ARGS=2
E_BADARGS=65
MYSQL=`which mysql`
dbpw=`grep DB_PASSWORD /opt/bitnami/apps/wordpress/htdocs/wp-config.php | cut -d \' -f 4`
C1="DROP DATABASE IF EXISTS $1;"
C2="CREATE DATABASE $1;"
COMMANDS="${C1}${C2}"
if [ $# -ne $EXPECTED_ARGS ]
then
  echo "Usage: $0 dbname dbadmin"
  exit $E_BADARGS
fi
$MYSQL -u$2 -p$dbpw -e "$COMMANDS"

10. Verify Wordpress website renders correctly:

Browse to http://websiteurl.company.com

I hope this helps whoever continues to experience this particular issue now or in the future.

Answer 3

If you do not need the data from the corrupt table, you could try first discarding tablespace.

See here: https://dba.stackexchange.com/a/298144/237137