We have Percona MySQL Cluster running with three nodes on CentOS.
Server : Percona-XtraDB-Cluster-server-56-5.6.19-25.6.824.el6.x86_64
One node is crashing repeatedly.
Thread pointer: 0x18910ba30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f354c30fd38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8f36b5]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x6635c4]
/lib64/libpthread.so.0(+0xf710)[0x7f3b73fde710]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG19raw_get_current_logEP11 st_log_info+0x2e)[0x89b1be]
/usr/sbin/mysqld[0x8a0833]
/usr/sbin/mysqld[0x59eb2c]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13s t_plugin_intPvEijS3_+0x226)[0x6f7746]
/usr/sbin/mysqld(_Z28ha_start_consistent_snapshotP3THD+0x281 )[0x5a8d51]
/usr/sbin/mysqld(_Z11trans_beginP3THDj+0x114)[0x78a7f4]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x216c)[0x6e924c]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x65 8)[0x6ed768]
/usr/sbin/mysqld[0x6ed8c1]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0x1362)[0x6ef412]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1e3)[0x6f0f73]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x17f)[0x6b9bdf]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6b9db7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xae9b4a]
/lib64/libpthread.so.0(+0x79d1)[0x7f3b73fd69d1]
/lib64/libc.so.6(clone+0x6d)[0x7f3b724ddb5d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7effb4e03890): is an invalid pointer
Connection ID (thread ID): 202009
Status: NOT_KILLED
I am unable to understand why it is crashing and what should I look at.
I have already tried following but server still crashes with same output.
1. Removed Percona Server.
2. Removed all data under /var/lib/mysql
3. Rebooted server
4. Install Percona Server
5. Sync with cluster (full SST) - Copies all data from running node.
It still crashes with same error.
It does not crash immediately but randomly after about 18-22 hours.
