1

Heading

I have three nodes Galera cluster MariaDB, a node in June 4, 2016 suddenly crash, the following is log error

Red Hat Enterprise Linux Server release 6.5 Server version: 5.5.39-MariaDB-wsrep-log

160603 13:56:06 [Warning] WSREP: SQL statement was ineffective, THD: 29598800, buf: 143 QUERY: commit => Skipping replication
160603 13:56:06 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail.

Server version: 5.5.39-MariaDB-wsrep-log key_buffer_size=25165824 read_buffer_size=131072 max_used_connections=226 max_threads=802 thread_count=79 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1784371 K bytes of memory Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7f2980201000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7f2a5d9b2cd0 thread_stack 0x48000 160603 13:56:17 [Warning] WSREP: last inactive check more than PT1.5S ago (PT11.5678S), skipping check /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xa95ecb] /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6ebbc8] /lib64/libpthread.so.0[0x3da300f710] /usr/lib64/galera/libgalera_smm.so(_ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2_+0x1f)[0x7f4127f5d21f] /usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM13post_rollbackEPNS_9TrxHandleE+0x2e)[0x7f4127f7828e] /usr/lib64/galera/libgalera_smm.so(galera_post_rollback+0x68)[0x7f4127f91d88] /usr/sbin/mysqld[0x6925d5] /usr/sbin/mysqld[0x6926e8] /usr/sbin/mysqld(_Z17ha_rollback_transP3THDb+0xe6)[0x6ee8a6] /usr/sbin/mysqld(_Z15ha_commit_transP3THDb+0x1e2)[0x6eebb2] /usr/sbin/mysqld(_Z12trans_commitP3THD+0x45)[0x66e1b5] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x2430)[0x5a8220] /usr/sbin/mysqld[0x5acf54] /usr/sbin/mysqld[0x5ad33b] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1889)[0x5aefb9] /usr/sbin/mysqld(_Z10do_commandP3THD+0x11c)[0x5af73c] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x2eb)[0x660d3b] /usr/sbin/mysqld(handle_one_connection+0x4c)[0x660ddc] /lib64/libpthread.so.0[0x3da30079d1] /lib64/libc.so.6(clone+0x6d)[0x3da2ce8b6d]

Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f2a88c1f018): is an invalid pointer Connection ID (thread ID): 29598800 Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=on,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=off

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 160603 13:56:19 mysqld_safe Number of processes running now: 0 160603 13:56:19 mysqld_safe WSREP: not restarting wsrep node automatically 160603 13:56:19 mysqld_safe mysqld from pid file /PMS-db/data/lyxr-db04.pid ended

[MYSQLD]
user=mysql
basedir=/usr
datadir=/PMS-db
socket=/var/lib/mysql/mysql.sock
pid_file=mysqld.pid
port=3306
log_error=error.log
#log_output=FILE
#relay_log=relay-bin
### INNODB OPTIONS 
innodb_buffer_pool_size=90418M
innodb_additional_mem_pool_size=20M
innodb_flush_log_at_trx_commit=2
innodb_file_per_table=1
innodb_data_file_path = ibdata1:100M:autoextend
## You may want to tune the below depending on number of cores and disk sub
innodb_read_io_threads=8
innodb_write_io_threads=8
innodb_doublewrite=1
innodb_log_file_size=1024M
innodb_log_buffer_size=96M
innodb_buffer_pool_instances=8
innodb_log_files_in_group=2
innodb_thread_concurrency=0
#innodb_file_format=barracuda
innodb_io_capacity=2000
innodb_flush_method = O_DIRECT
innodb_locks_unsafe_for_binlog=1
innodb_autoinc_lock_mode=2
## avoid statistics update when doing e.g show tables
innodb_stats_on_metadata=0
engine_condition_pushdown=1
default_storage_engine=innodb
#thread_handling=pool-of-threads
#thread_pool_max_threads=2000
#thread_pool_size=32
#thread_pool_oversubscribe=3
#thread_pool_stall_limit=30
#thread_pool_idle_timeout=60

# CHARACTER SET
#collation_server = utf8_unicode_ci
#init_connect='SET NAMES utf8'
character_set_server = utf8

# REPLICATION SPECIFIC - GENERAL
#server_id must be unique across all mysql servers participating in replication.
server_id=1
#auto_increment_increment=2
#auto_increment_offset=SERVERID
# REPLICATION SPECIFIC
binlog_format=ROW
log_bin=binlog
log_bin_trust_function_creators=1
#relay_log=relay-bin
expire_logs_days=30
#log_slave_updates=1
# OTHER THINGS, BUFFERS ETC
key_buffer_size = 24M
tmp_table_size = 64M
max_heap_table_size = 64M
max_allowed_packet = 512M
sort_buffer_size = 256K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
skip_name_resolve
memlock=0
sysdate_is_now=1
max_connections=2000
thread_cache_size=512
query_cache_type = 0
query_cache_size = 0
table_open_cache=1024
lower_case_table_names=0
bind-address=192.168.16.11
slow_query_log = on  
log_queries_not_using_indexes=1
long_query_time  = 1
##
## WSREP options
##

# Full path to wsrep provider library or 'none'
wsrep_provider=/usr/lib64/galera/libgalera_smm.so

# Provider specific configuration options
wsrep_provider_options="gcache.size=8192M"

# Logical cluster name. Should be the same for all nodes.
wsrep_cluster_name="xmscluster"

# Group communication system handle
#wsrep_cluster_address=gcomm://

# Human-readable node name (non-unique). Hostname by default.
wsrep_node_name=lyxr-db01
# Address for incoming client connections. Autodetect by default.
wsrep_node_address=192.168.16.11

wsrep_node_incoming_address=192.168.16.11
wsrep_sst_receive_address=192.168.16.11
wsrep_sst_auth=repluser:foxhis

# How many threads will process writesets from other nodes
wsrep_slave_threads=16

# DBUG options for wsrep provider
#wsrep_dbug_option

# Generate fake primary keys for non-PK tables (required for multi-master
# and parallel applying operation)
wsrep_certify_nonPK=1

# Location of the directory with data files. Needed for non-mysqldump
# state snapshot transfers. Defaults to mysql_real_data_home.
#wsrep_data_home_dir=

# Maximum number of rows in write set
wsrep_max_ws_rows=131072

# Maximum size of write set
wsrep_max_ws_size=1073741824

# to enable debug level logging, set this to 1
wsrep_debug=0

# convert locking sessions into transactions
wsrep_convert_LOCK_to_trx=0

# how many times to retry deadlocked autocommits
wsrep_retry_autocommit=1

# change auto_increment_increment and auto_increment_offset automatically
wsrep_auto_increment_control=1

# replicate myisam
wsrep_replicate_myisam=1
# retry autoinc insert, which failed for duplicate key error
wsrep_drupal_282555_workaround=0

# enable "strictly synchronous" semantics for read operations
wsrep_causal_reads=0

# Command to call when node status or cluster membership changes.
# Will be passed all or some of the following options:
# --status  - new status of this node
# --uuid    - UUID of the cluster
# --primary - whether the component is primary or not ("yes"/"no")
# --members - comma-separated list of members
# --index   - index of this node in the list
#wsrep_notify_cmd=

##
## WSREP State Transfer options
##

# State Snapshot Transfer method
# ClusterControl currently DOES NOT support wsrep_sst_method=mysqldump
wsrep_sst_method=xtrabackup

# Address on THIS node to receive SST at. DON'T SET IT TO DONOR ADDRESS!!!
# (SST method dependent. Defaults to the first IP of the first interface)
#wsrep_sst_receive_address=

# SST authentication string. This will be used to send SST to joining nodes.
# Depends on SST method. For mysqldump method it is root:<root password>

# Desired SST donor name.
#wsrep_sst_donor=

# Protocol version to use
# wsrep_protocol_version=
[MYSQL]
socket=/var/lib/mysql/mysql.sock
#default_character_set=utf8
[client]
socket=/var/lib/mysql/mysql.sock
#default_character_set=utf8
[mysqldump]
max_allowed_packet = 512M
#default_character_set=utf8
[MYSQLD_SAFE]
pid_file=mysqld.pid
log_error=error.log
basedir=/usr
datadir=/PMS-db

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.