0

Setup:

Ubuntu 18.04.1 LTS
4.15.0-34-generic #37-Ubuntu
Docker: 18.06.1-ce build e68fc7a
mysql:5@sha256:e47e309f72c831cf880cc0e1990b9c5ac427016acdc71346a36c83806ca79bb4

mysql --version = Ver 14.14 Distrib 5.7.25, for Linux (x86_64) using  EditLine wrapper

Log:

07:17:39 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=6
max_threads=151
thread_count=5
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68196 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f9814012240
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f98580a4e80 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0x560da4deb81c]
mysqld(handle_fatal_signal+0x479)[0x560da4716879]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f987679b0c0]
mysqld(_Z32page_cur_search_with_match_bytesPK11buf_block_tPK12dict_index_tPK8dtuple_t15page_cur_mode_tPmS9_S9_S9_P10page_cur_t+0x167)[0x560da4eb2be7]
mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x989)[0x560da4fd81a9]
mysqld(+0xe682d9)[0x560da4f2c2d9]
mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x1b9e)[0x560da4f3307e]
mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x244)[0x560da4e2ee24]
mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x30f)[0x560da476830f]
mysqld(+0xac68f4)[0x560da4b8a8f4]
mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x105)[0x560da4b8dbb5]
mysqld(_ZN4JOIN4execEv+0x370)[0x560da4b86a30]
mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x233)[0x560da4bf5eb3]
mysqld(+0x61cf32)[0x560da46e0f32]
mysqld(_Z21mysql_execute_commandP3THDb+0x460d)[0x560da4bb972d]
mysqld(_Z11mysql_parseP3THDP12Parser_state+0x395)[0x560da4bbbff5]
mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xfc4)[0x560da4bbd0b4]
mysqld(_Z10do_commandP3THD+0x197)[0x560da4bbe487]
mysqld(handle_connection+0x278)[0x560da4c7af38]
mysqld(pfs_spawn_thread+0x1b4)[0x560da5153164]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f9876791494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f9874fddacf]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f98141c3520): SELECT `id`, `state`, `param`, `cursor_row`, `cursor_column`, `can_send`, `createdAt`, `updatedAt`, `userId`, `botId` FROM `states` AS `state` WHERE `state`.`userId` = 131913 AND `state`.`botId` = 3202 LIMIT 1
Connection ID (thread ID): 863486
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

Probebly unrelated but repeated a lot:

2019-06-12T14:52:32.090130Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 10295ms. The settings might not be optimal. (flushed=9 and evicted=0, during the time.)
2019-06-12T14:53:56.177026Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8427ms. The settings might not be optimal. (flushed=200 and evicted=0, during the time.)
2019-06-12T15:03:05.055209Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6080ms. The settings might not be optimal. (flushed=3 and evicted=0, during the time.)
2019-06-12T15:03:52.398646Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5042ms. The settings might not be optimal. (flushed=168 and evicted=0, during the time.)

more logs and settings


Default settings from the docker have not been changed. It runs fine and queries to all unaffected tables work fine, but any queries, even CHECK TABLE on the affected table crashed the server instantly.

I haven't attempted data recovery and simply dropped and restored the table from backups. However, this is the second time in 2 months that this has happened (to different tables, but both were tables with more writes compared to the rest).

How can I diagnose the cause of corruption or make it less likely?

What I've checked so far:

  • The included log is the first sign of anything going wrong, the previous log entry is from 30min before and unrelated. And while crash with a corrupted table is reproducible I have no way of reproducing the corruption and higher log verbosity is unideal on the production server under load.
  • dmesg | egrep -i 'killed process' is empty, the box has 4gb of ram and swap enabled. I don't think the tables got corrupted due to oom kill in the middle of a write.
  • Shot in the dark: Have you tried disabling the load of persistent buffer pool? innodb_buffer_pool_load_at_startup is enabled in your settings, which might increase the likelihood of errors if this file already got errors (should be caught by InnoDB checksums but you'll never know). – gertvdijk Jun 18 at 1:15
  • @gertvdijk I don't see any signs of MySQL restarting in the logs of the first crash, so I don't think a setting that applies only to startup would affect this. But I will change it anyway. The only downside is slowdown after restart until caches are warmed up again, correct? We won't find out if it helped until a few months to see if it corrupts tables again or not though... – Bor691 Jun 18 at 5:40
  • Additional information request. Post on pastebin.com and share the links. RAM size of your MySQL Host server A) complete (not edited) my.cnf or my.ini Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME D) SHOW FULL PROCESSLIST; E) complete MySQLTuner report F) SHOW ENGINE INNODB STATUS; AND Optional very helpful information, if available includes - htop OR top OR mytop for most active apps, ulimit -a for a linux/unix list of limits, iostat -xm 5 3 for IOPS by device and core/cpu count, for server workload tuning analysis. – Wilson Hauck Jun 20 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.