Ubuntu 18.04.1 LTS 4.15.0-34-generic #37-Ubuntu Docker: 18.06.1-ce build e68fc7a mysql:5@sha256:e47e309f72c831cf880cc0e1990b9c5ac427016acdc71346a36c83806ca79bb4 mysql --version = Ver 14.14 Distrib 5.7.25, for Linux (x86_64) using EditLine wrapper
07:17:39 UTC - mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=6 max_threads=151 thread_count=5 connection_count=5 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68196 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f9814012240 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f98580a4e80 thread_stack 0x40000 mysqld(my_print_stacktrace+0x2c)[0x560da4deb81c] mysqld(handle_fatal_signal+0x479)[0x560da4716879] /lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f987679b0c0] mysqld(_Z32page_cur_search_with_match_bytesPK11buf_block_tPK12dict_index_tPK8dtuple_t15page_cur_mode_tPmS9_S9_S9_P10page_cur_t+0x167)[0x560da4eb2be7] mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x989)[0x560da4fd81a9] mysqld(+0xe682d9)[0x560da4f2c2d9] mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x1b9e)[0x560da4f3307e] mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x244)[0x560da4e2ee24] mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x30f)[0x560da476830f] mysqld(+0xac68f4)[0x560da4b8a8f4] mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x105)[0x560da4b8dbb5] mysqld(_ZN4JOIN4execEv+0x370)[0x560da4b86a30] mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x233)[0x560da4bf5eb3] mysqld(+0x61cf32)[0x560da46e0f32] mysqld(_Z21mysql_execute_commandP3THDb+0x460d)[0x560da4bb972d] mysqld(_Z11mysql_parseP3THDP12Parser_state+0x395)[0x560da4bbbff5] mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xfc4)[0x560da4bbd0b4] mysqld(_Z10do_commandP3THD+0x197)[0x560da4bbe487] mysqld(handle_connection+0x278)[0x560da4c7af38] mysqld(pfs_spawn_thread+0x1b4)[0x560da5153164] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7f9876791494] /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f9874fddacf] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f98141c3520): SELECT `id`, `state`, `param`, `cursor_row`, `cursor_column`, `can_send`, `createdAt`, `updatedAt`, `userId`, `botId` FROM `states` AS `state` WHERE `state`.`userId` = 131913 AND `state`.`botId` = 3202 LIMIT 1 Connection ID (thread ID): 863486 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
Probebly unrelated but repeated a lot:
2019-06-12T14:52:32.090130Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 10295ms. The settings might not be optimal. (flushed=9 and evicted=0, during the time.) 2019-06-12T14:53:56.177026Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 8427ms. The settings might not be optimal. (flushed=200 and evicted=0, during the time.) 2019-06-12T15:03:05.055209Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 6080ms. The settings might not be optimal. (flushed=3 and evicted=0, during the time.) 2019-06-12T15:03:52.398646Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5042ms. The settings might not be optimal. (flushed=168 and evicted=0, during the time.)
Default settings from the docker have not been changed. It runs fine and queries to all unaffected tables work fine, but any queries, even
CHECK TABLE on the affected table crashed the server instantly.
I haven't attempted data recovery and simply dropped and restored the table from backups. However, this is the second time in 2 months that this has happened (to different tables, but both were tables with more writes compared to the rest).
How can I diagnose the cause of corruption or make it less likely?
What I've checked so far:
- The included log is the first sign of anything going wrong, the previous log entry is from 30min before and unrelated. And while crash with a corrupted table is reproducible I have no way of reproducing the corruption and higher log verbosity is unideal on the production server under load.
dmesg | egrep -i 'killed process'is empty, the box has 4gb of ram and swap enabled. I don't think the tables got corrupted due to oom kill in the middle of a write.