0

We have Percona MySQL Cluster running with three nodes on CentOS.
Server : Percona-XtraDB-Cluster-server-56-5.6.19-25.6.824.el6.x86_64
One node is crashing repeatedly.

Thread pointer: 0x18910ba30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f354c30fd38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8f36b5]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x6635c4]
/lib64/libpthread.so.0(+0xf710)[0x7f3b73fde710]
/usr/sbin/mysqld(_ZN13MYSQL_BIN_LOG19raw_get_current_logEP11 st_log_info+0x2e)[0x89b1be]
/usr/sbin/mysqld[0x8a0833]
/usr/sbin/mysqld[0x59eb2c]
/usr/sbin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_P13s t_plugin_intPvEijS3_+0x226)[0x6f7746]
/usr/sbin/mysqld(_Z28ha_start_consistent_snapshotP3THD+0x281 )[0x5a8d51]
/usr/sbin/mysqld(_Z11trans_beginP3THDj+0x114)[0x78a7f4]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x216c)[0x6e924c]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x65 8)[0x6ed768]
/usr/sbin/mysqld[0x6ed8c1]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3 THDPcj+0x1362)[0x6ef412]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x1e3)[0x6f0f73]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x17f)[0x6b9bdf]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6b9db7]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xae9b4a]
/lib64/libpthread.so.0(+0x79d1)[0x7f3b73fd69d1]
/lib64/libc.so.6(clone+0x6d)[0x7f3b724ddb5d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7effb4e03890): is an invalid pointer
Connection ID (thread ID): 202009
Status: NOT_KILLED

I am unable to understand why it is crashing and what should I look at.

I have already tried following but server still crashes with same output.
1. Removed Percona Server.
2. Removed all data under /var/lib/mysql
3. Rebooted server
4. Install Percona Server
5. Sync with cluster (full SST) - Copies all data from running node.

It still crashes with same error.
It does not crash immediately but randomly after about 18-22 hours.

  • Might be a bug, is 5.6.19 the latest version? because community is on 5.6.20. Is there a possibility to over-commit memory? – Antonios Aug 4 '14 at 15:37
  • Yes, it is latest version. Server has sufficient free memory. – Jai Aug 5 '14 at 0:51
  • I would suggest to turn on general query log to catch what was executed on crash time. This line smells like a bug: Query (7effb4e03890): is an invalid pointer, Did you open a case on Percona about it? – Antonios Aug 5 '14 at 8:12
  • It's probably worth your time to thoroughly test the server's RAM, and if that's good, check the disks for bad blocks. – Michael - sqlbot Aug 6 '14 at 1:51
  • You are absolutely right. It does seems like drive errors. Server is on LSI RAID 10 with 8 drives. MegaCLI is showing raid array optimal but one of the drive is showing errors, I believe that drive is faulty and needs to be replaced. Thanks. – Jai Aug 6 '14 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.