I have a MySQL instance acting as a replication slave and something recently caused it to crash. I suspected InnoDB page corruption and wanted to switch from one-shared-tablespace (for InnoDB) to file-per-table, so I went through this process mostly to good effect. (I had a server crash part of the way through the restore, but I re-ran from the middle and was able to get back up and running).
I restarted the replication slave and started catching back up with the master. Until sometime around the middle of the night when MySQL same down again.
No error messages are in any logs that I can find, other than in
Nov 17 04:12:32 myserver mysqld_safe: Number of processes running now: 0 Nov 17 04:12:32 myserver mysqld_safe: mysqld restarted Nov 17 04:12:32 myserver mysqld: 2017-11-17 04:12:32 0 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead. Nov 17 04:12:32 myserver mysqld: 2017-11-17 04:12:32 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecat ed. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). Nov 17 04:12:32 myserver mysqld: 2017-11-17 04:12:32 0 [Note] /usr/sbin/mysqld (mysqld 5.6.38-1~dotdeb+7.1-log) st arting as process 8811 ... Nov 17 04:12:33 myserver mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Attempts to start the server result in the same messages in my log file.
innodb_force_recovery=1 in my
my.cnf file allows me to start the server, albeit in read-only mode. But I can't seem to determine which table or tables are causing the crash.
I have run
innochecksum against every on-disk
*.ibd file I can find, as well as the primary
ibdata1 file, and no errors are reported.
How can I get more information about whatever is causing my server to crash/panic/quit/exit immediately after starting? I've tried running
mysqld directly with the
-v switch, but I get nothing more than I got in my
How can I turn-up the log-level on the mysqld process to get some actual information about what's going on? I have some multi-million-record tables, here, and simply wiping things out and restoring from a backup takes a really long time.
I'm including my logging configuration for reference:
#general_log_file = /var/log/mysql/mysql.log #general_log = 1 #slow_query_log = 1 #slow_query_log_file = /var/log/mysql/mysql-slow.log #long_query_time = 2 #log-queries-not-using-indexes # #server-id = 1 #log_bin = /var/log/mysql/mysql-bin.log expire_logs_days = 10 max_binlog_size = 100M #binlog_do_db = include_database_name #binlog_ignore_db = include_database_name # NOTE: Error logging goes to syslog due to /etc/mysql/conf.d/mysqld_safe_syslog.cnf log_error= syslog (this line comes from /etc/mysql/conf.d/mysqld_sage_syslog.cnf)
None of the files in
/var/log/ seem to have anything useful in them. When running
mysqld from the command-line (where logs are sent to stdout), the process exits without any indication as to why.