0

I have a 3 node MariaDB galera cluster running. It was running fine until I updated one node from MariaDB 10.3.14 to 10.3.16 (the other nodes are still on 10.3.14). Since then, this one node regularly crashes when writing to one or two specific tables.

I already resynced the node multiple times, but it keeps crashing when writing to one specific table. Writes to other tables are fine.

The configuration was unchanged during the update.

Any ideas what would cause this crash? My next try would be to reinstall MariaDB.

Jul  8 11:42:16 prod03 mysqld[14704]: 2019-07-08 11:42:16 10 [ERROR] [FATAL] InnoDB: Trying to read page number 828793444 in space 420612, space name praxiswilh/fibu_buchung, which is outside the tablespace bounds. Byte offset 0, len 16384
Jul  8 11:42:16 prod03 mysqld[14704]: 190708 11:42:16 [ERROR] mysqld got signal 6 ;
Jul  8 11:42:16 prod03 mysqld[14704]: This could be because you hit a bug. It is also possible that this binary
Jul  8 11:42:16 prod03 mysqld[14704]: or one of the libraries it was linked against is corrupt, improperly built,
Jul  8 11:42:16 prod03 mysqld[14704]: or misconfigured. This error can also be caused by malfunctioning hardware.
Jul  8 11:42:16 prod03 mysqld[14704]: To report this bug, see https://mariadb.com/kb/en/reporting-bugs
Jul  8 11:42:16 prod03 mysqld[14704]: We will try our best to scrape up some info that will hopefully help
Jul  8 11:42:16 prod03 mysqld[14704]: diagnose the problem, but since we have already crashed,
Jul  8 11:42:16 prod03 mysqld[14704]: something is definitely wrong and this may fail.
Jul  8 11:42:16 prod03 mysqld[14704]: Server version: 10.3.16-MariaDB-1:10.3.16+maria~bionic-log
Jul  8 11:42:16 prod03 mysqld[14704]: key_buffer_size=134217728
Jul  8 11:42:16 prod03 mysqld[14704]: read_buffer_size=2097152
Jul  8 11:42:16 prod03 mysqld[14704]: max_used_connections=5
Jul  8 11:42:16 prod03 mysqld[14704]: max_threads=502
Jul  8 11:42:16 prod03 mysqld[14704]: thread_count=17
Jul  8 11:42:16 prod03 mysqld[14704]: It is possible that mysqld could use up to
Jul  8 11:42:16 prod03 mysqld[14704]: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3226632 K  bytes of memory
Jul  8 11:42:16 prod03 mysqld[14704]: Hope that's ok; if not, decrease some variables in the equation.
Jul  8 11:42:16 prod03 mysqld[14704]: Thread pointer: 0x7ff184000c08
Jul  8 11:42:16 prod03 mysqld[14704]: Attempting backtrace. You can use the following information to find out
Jul  8 11:42:16 prod03 mysqld[14704]: where mysqld died. If you see no messages after this, something went
Jul  8 11:42:16 prod03 mysqld[14704]: terribly wrong...
Jul  8 11:42:16 prod03 mysqld[14704]: stack_bottom = 0x7ff6409c2e08 thread_stack 0x49000
Jul  8 11:42:16 prod03 mysqld[14704]: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x55e16635b53e]
Jul  8 11:42:16 prod03 mysqld[14704]: /usr/sbin/mysqld(handle_fatal_signal+0x515)[0x55e165df5fb5]
Jul  8 11:42:16 prod03 mysqld[14704]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7ff7aac9a890]
Jul  8 11:42:17 prod03 mysqld[14704]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7ff7aa1aee97]
Jul  8 11:42:17 prod03 mysqld[14704]: /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7ff7aa1b0801]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0xa24321)[0x55e1660c7321]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x4a1c9a)[0x55e165b44c9a]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0xad0ad5)[0x55e166173ad5]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0xa8859a)[0x55e16612b59a]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0xa639c2)[0x55e1661069c2]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0xa46a9d)[0x55e1660e9a9d]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x987620)[0x55e16602a620]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x989294)[0x55e16602c294]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x98b88f)[0x55e16602e88f]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x99bcbe)[0x55e16603ecbe]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x8e00e6)[0x55e165f830e6]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(_ZN7handler12ha_write_rowEPh+0x10f)[0x55e165e00b6f]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(_ZN14Rows_log_event9write_rowEP14rpl_group_infob+0x167)[0x55e165ee4847]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(_ZN20Write_rows_log_event11do_exec_rowEP14rpl_group_info+0x7d)[0x55e165ee4dcd]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEP14rpl_group_info+0x22f)[0x55e165ed8ebf]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(wsrep_apply_cb+0x512)[0x55e165d74cd2]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xcb)[0x7ff789aa746b]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(+0x224654)[0x7ff789af5654]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x158)[0x7ff789af8998]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x9e)[0x7ff789afb73e]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x178)[0x7ff789ad2028]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x72)[0x7ff789ad3172]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x12d)[0x7ff789afbf4d]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/lib/galera/libgalera_smm.so(galera_recv+0x2b)[0x7ff789b1689b]
Jul  8 11:42:17 prod03 mysqld[14704]: /usr/sbin/mysqld(+0x6d2922)[0x55e165d75922]
Jul  8 11:42:18 prod03 mysqld[14704]: /usr/sbin/mysqld(start_wsrep_THD+0x38f)[0x55e165d67a6f]
Jul  8 11:42:18 prod03 mysqld[14704]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7ff7aac8f6db]
Jul  8 11:42:18 prod03 mysqld[14704]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7ff7aa29188f]
Jul  8 11:42:18 prod03 mysqld[14704]: Trying to get some variables.
Jul  8 11:42:18 prod03 mysqld[14704]: Some pointers may be invalid and cause the dump to abort.
Jul  8 11:42:18 prod03 mysqld[14704]: Query (0x7ff64fb0eca9): INSERT INTO `XXX.FIBU_BUCHUNG  (id_buchung, id_profil, belegnummer, buchungsnummer, buchungsart, zahlungsdatum, wertstellungsdatum,  buchungstext, betrag_soll, betrag_haben, id_konto, id_gegenKonto, buchungstoken, isgegenbuchung,  kontoSH, gegenkontoSH, betragBN, steuerschluessel, steuerbetrag, date_fixation, date_cancellation, id_booking_cancellation ) VALUES ( 'XXX', 'XXX', '', 134, 0, '2019-07-08', '2019-07-08', 'XXX', 0, 105.96, 8100, 1200, '2019-07-08 11:42:16', 0, 'H', 'S', 'B', 0, 0, '2019-07-08', null, null )
Jul  8 11:42:18 prod03 mysqld[14704]: Connection ID (thread ID): 10
Jul  8 11:42:18 prod03 mysqld[14704]: Status: NOT_KILLED
Jul  8 11:42:18 prod03 mysqld[14704]: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on
Jul  8 11:42:18 prod03 mysqld[14704]: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Jul  8 11:42:18 prod03 mysqld[14704]: information that should help you find out what is causing the crash.
Jul  8 11:42:18 prod03 mysqld[14704]: Writing a core file...
Jul  8 11:42:18 prod03 mysqld[14704]: Working directory at /var/lib/mysql
Jul  8 11:42:18 prod03 mysqld[14704]: Resource Limits:
Jul  8 11:42:18 prod03 mysqld[14704]: Limit                     Soft Limit           Hard Limit           Units
Jul  8 11:42:18 prod03 mysqld[14704]: Max cpu time              unlimited            unlimited            seconds
Jul  8 11:42:18 prod03 mysqld[14704]: Max file size             unlimited            unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max data size             unlimited            unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max stack size            8388608              unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max core file size        0                    unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max resident set          unlimited            unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max processes             256645               256645               processes
Jul  8 11:42:18 prod03 mysqld[14704]: Max open files            16364                16364                files
Jul  8 11:42:18 prod03 mysqld[14704]: Max locked memory         16777216             16777216             bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max address space         unlimited            unlimited            bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max file locks            unlimited            unlimited            locks
Jul  8 11:42:18 prod03 mysqld[14704]: Max pending signals       256645               256645               signals
Jul  8 11:42:18 prod03 mysqld[14704]: Max msgqueue size         819200               819200               bytes
Jul  8 11:42:18 prod03 mysqld[14704]: Max nice priority         0                    0
Jul  8 11:42:18 prod03 mysqld[14704]: Max realtime priority     0                    0
Jul  8 11:42:18 prod03 mysqld[14704]: Max realtime timeout      unlimited            unlimited            us
Jul  8 11:42:18 prod03 mysqld[14704]: Core pattern: core
Jul  8 11:42:19 prod03 systemd[1]: mariadb.service: Main process exited, code=killed, status=6/ABRT
Jul  8 11:42:19 prod03 systemd[1]: mariadb.service: Failed with result 'signal'.
Jul  8 11:42:24 prod03 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Jul  8 11:42:24 prod03 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Jul  8 11:42:24 prod03 systemd[1]: Stopped MariaDB 10.3.16 database server.
4
  • Suggest you file a bug. Point out that this update is a rather narrow change (.14 to .16). I see that the Changelog for .15 just says "bump the VERSION", as if they did not make any changes there. Meanwhile, back off to .14. – Rick James Jul 9 '19 at 15:06
  • Have you run mysql_upgrade after upgrading? (This may happen automatically on Ubuntu, though, and it sounds like you're on Ubuntu ...) BTW, I've been running 10.3.15 Cluster in production, no issues. Quite a lot of changes beyond bumping the version AFAICT. – dbdemon Jul 10 '19 at 14:10
  • I backed off to .14 and strangely it crashed again today while accessing the same table 'XXX.FIBU_BUCHUNG'. So it seems not to be a problem with mariadb itself, but maybe with an other library which was updated in the same "apt-get" run. I did not update the other nodes since then and I really have no clue how to find the reason for this error. – homedom Jul 11 '19 at 10:50
  • I hope I finally got the problem fixed. It seems that there was a problem with one table which caused the issue after state transfer to the updated node. I backed up the table, dropped and restored it on the running cluster. After adding the node back into the cluster, it seems to be stable for the last few days. – homedom Aug 9 '19 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.