0

What I have done: I have restarted mysql on the JOINERNODE to apply some database settings including increasing the back_log and query_cache_size settings.

What I am seeing: When I start mysql on the JOINERNODE, I see socat launch and listen on port 4444 then stop about 1-2 seconds later.

Joiner MYSQL logs:

161107 19:16:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
161107 19:16:43 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.JKtTSU' --pid-file='/var/lib/mysql/JOINERNODE-recover.pid'
161107 19:16:56 mysqld_safe WSREP: Recovered position 3bf24a64-e806-11e5-8238-ea129650fffe:4050608949
161107 19:16:56 [Note] WSREP: wsrep_start_position var submitted: '3bf24a64-e806-11e5-8238-ea129650fffe:4050608949'
161107 19:16:56 [Note] WSREP: Read nil XID from storage engines, skipping position init
161107 19:16:56 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
161107 19:16:56 [Note] WSREP: wsrep_load(): Galera 25.3.5(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
161107 19:16:56 [Note] WSREP: CRC-32C: using hardware acceleration.
161107 19:16:56 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
161107 19:16:56 [Note] WSREP: Passing config to GCS: base_host = 170.71.77.88; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 1G; gcache.size = 4G; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = P30S; pc.weight = 1; protonet.ba
161107 19:16:56 [Note] WSREP: Service thread queue flushed.
161107 19:16:56 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
161107 19:16:56 [Note] WSREP: wsrep_sst_grab()
161107 19:16:56 [Note] WSREP: Start replication
161107 19:16:56 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
161107 19:16:56 [Note] WSREP: protonet asio version 0
161107 19:16:56 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
161107 19:16:56 [Note] WSREP: backend: asio
161107 19:16:56 [Note] WSREP: GMCast version 0
161107 19:16:56 [Note] WSREP: (0a872720-a551-11e6-9bc2-eedf51200d4b, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
161107 19:16:56 [Note] WSREP: (0a872720-a551-11e6-9bc2-eedf51200d4b, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
161107 19:16:56 [Note] WSREP: EVS version 0
161107 19:16:56 [Note] WSREP: PC version 0
161107 19:16:56 [Note] WSREP: gcomm: connecting to group 'galera_cluster', peer 'DONORNODE:'
161107 19:16:56 [Note] WSREP: declaring 4159cb50-6323-11e6-946f-f77a9bdd31e2 stable
161107 19:16:56 [Note] WSREP: Node 4159cb50-6323-11e6-946f-f77a9bdd31e2 state prim
161107 19:16:56 [Note] WSREP: view(view_id(PRIM,0a872720-a551-11e6-9bc2-eedf51200d4b,85) memb {
  0a872720-a551-11e6-9bc2-eedf51200d4b,0
  4159cb50-6323-11e6-946f-f77a9bdd31e2,0
} joined {
} left {
} partitioned {
})
161107 19:16:57 [Note] WSREP: gcomm: connected
161107 19:16:57 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
161107 19:16:57 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
161107 19:16:57 [Note] WSREP: Opened channel 'galera_cluster'
161107 19:16:57 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
161107 19:16:57 [Note] WSREP: Waiting for SST to complete.
161107 19:16:57 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0ad4f230-a551-11e6-9d07-8244e22ea7fb
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: sent state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: got state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb from 0 (JOINERNODE)
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: got state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb from 1 (DONORNODE)
161107 19:16:57 [Note] WSREP: Quorum results:
  version    = 3,
  component  = PRIMARY,
  conf_id    = 72,
  members    = 1/2 (joined/total),
  act_id     = 4053449525,
  last_appl. = -1,
  protocols  = 0/5/2 (gcs/repl/appl),
  group UUID = 3bf24a64-e806-11e5-8238-ea129650fffe
161107 19:16:57 [Note] WSREP: Flow-control interval: [23, 23]
161107 19:16:57 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 4053449525)
161107 19:16:57 [Note] WSREP: State transfer required:
  Group state: 3bf24a64-e806-11e5-8238-ea129650fffe:4053449525
  Local state: 00000000-0000-0000-0000-000000000000:-1
161107 19:16:57 [Note] WSREP: New cluster view: global state: 3bf24a64-e806-11e5-8238-ea129650fffe:4053449525, view# 73: Primary, number of nodes: 2, my index: 0, protocol version 2
161107 19:16:57 [Warning] WSREP: Gap in state sequence. Need state transfer.
161107 19:16:59 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'joiner' --address '170.71.77.88' --auth 'replication:replication' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '18657''
WSREP_SST: [INFO] Streaming with xbstream (20161107 19:16:59.521)
WSREP_SST: [INFO] Using socat as streamer (20161107 19:16:59.524)
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20161107 19:16:59.663)
161107 19:17:02 [Note] WSREP: Prepared SST request: xtrabackup|170.71.77.88:4444/xtrabackup_sst
161107 19:17:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
161107 19:17:02 [Note] WSREP: REPL Protocols: 5 (3, 1)
161107 19:17:02 [Note] WSREP: Service thread queue flushed.
161107 19:17:02 [Note] WSREP: Assign initial position for certification: 4053449525, protocol version: 3
161107 19:17:02 [Note] WSREP: Service thread queue flushed.
161107 19:17:02 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (3bf24a64-e806-11e5-8238-ea129650fffe): 1 (Operation not permitted)
   at galera/src/replicator_str.cpp:prepare_for_IST():447. IST will be unavailable.
161107 19:17:02 [Note] WSREP: Member 0.0 (JOINERNODE) requested state transfer from '*any*'. Selected 1.0 (DONORNODE)(SYNCED) as donor.
161107 19:17:02 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 4053450570)
161107 19:17:02 [Note] WSREP: Requesting state transfer: success, donor: 1
xbstream: Can't create/write to file './ibdata1' (Errcode: 17 - File exists)
xbstream: failed to create file.
2016/11/07 19:17:38 socat[18887] E write(1, 0x84b3e0, 8192): Broken pipe
WSREP_SST: [ERROR] Xbstream failed (20161107 19:17:38.164)
WSREP_SST: [ERROR] Data directory /var/lib/mysql/ may not be empty: lp:1193240 Manual intervention required in that case (20161107 19:17:38.167)
WSREP_SST: [ERROR] Cleanup after exit with status:32 (20161107 19:17:38.169)
WSREP_SST: [INFO] Removing the sst_in_progress file (20161107 19:17:38.172)
161107 19:17:38 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'joiner' --address '170.71.77.88' --auth 'replication:replication' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '18657': 32 (Broken pipe)
161107 19:17:38 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
161107 19:17:38 [ERROR] WSREP: SST failed: 32 (Broken pipe)
161107 19:17:38 [ERROR] Aborting

161107 19:17:38 [Warning] WSREP: 1.0 (DONORNODE): State transfer to 0.0 (JOINERNODE) failed: -22 (Invalid argument)
161107 19:17:38 [ERROR] WSREP: gcs/src/gcs_group.c:gcs_group_handle_join_msg():723: Will never receive state. Need to abort.
161107 19:17:38 [Note] WSREP: gcomm: terminating thread
161107 19:17:38 [Note] WSREP: gcomm: joining thread
161107 19:17:38 [Note] WSREP: gcomm: closing backend
161107 19:17:39 [Note] WSREP: view(view_id(NON_PRIM,0a872720-a551-11e6-9bc2-eedf51200d4b,85) memb {
  0a872720-a551-11e6-9bc2-eedf51200d4b,0
} joined {
} left {
} partitioned {
  4159cb50-6323-11e6-946f-f77a9bdd31e2,0
})
161107 19:17:39 [Note] WSREP: view((empty))
161107 19:17:39 [Note] WSREP: gcomm: closed
161107 19:17:39 [Note] WSREP: /usr/sbin/mysqld: Terminated.
161107 19:17:39 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended

Donor Mysql Logs:

161107 19:16:56 [Note] WSREP: declaring 0a872720-a551-11e6-9bc2-eedf51200d4b stable
161107 19:16:56 [Note] WSREP: Node 4159cb50-6323-11e6-946f-f77a9bdd31e2 state prim
161107 19:16:56 [Note] WSREP: view(view_id(PRIM,0a872720-a551-11e6-9bc2-eedf51200d4b,85) memb {
  0a872720-a551-11e6-9bc2-eedf51200d4b,0
  4159cb50-6323-11e6-946f-f77a9bdd31e2,0
} joined {
} left {
} partitioned {
})
161107 19:16:56 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
161107 19:16:56 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: sent state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: got state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb from 0 (JOINERNODE)
161107 19:16:57 [Note] WSREP: STATE EXCHANGE: got state msg: 0ad4f230-a551-11e6-9d07-8244e22ea7fb from 1 (DONORNODE)
161107 19:16:57 [Note] WSREP: Quorum results:
  version    = 3,
  component  = PRIMARY,
  conf_id    = 72,
  members    = 1/2 (joined/total),
  act_id     = 4053449525,
  last_appl. = 4053449492,
  protocols  = 0/5/2 (gcs/repl/appl),
  group UUID = 3bf24a64-e806-11e5-8238-ea129650fffe
161107 19:16:57 [Note] WSREP: Flow-control interval: [23, 23]
161107 19:16:57 [Note] WSREP: New cluster view: global state: 3bf24a64-e806-11e5-8238-ea129650fffe:4053449525, view# 73: Primary, number of nodes: 2, my index: 1, protocol version 2
161107 19:16:57 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
161107 19:16:57 [Note] WSREP: REPL Protocols: 5 (3, 1)
161107 19:16:57 [Note] WSREP: Service thread queue flushed.
161107 19:16:57 [Note] WSREP: Assign initial position for certification: 4053449525, protocol version: 3
161107 19:16:57 [Note] WSREP: Service thread queue flushed.
161107 19:16:57 [Warning] WSREP: Releasing seqno 4053449525 before 4053449526 was assigned.
161107 19:17:02 [Note] WSREP: Member 0.0 (JOINERNODE) requested state transfer from '*any*'. Selected 1.0 (DONORNODE)(SYNCED) as donor.
161107 19:17:02 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 4053450570)
161107 19:17:02 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
161107 19:17:02 [Note] WSREP: Running: 'wsrep_sst_xtrabackup --role 'donor' --address '170.71.77.88:4444/xtrabackup_sst' --auth 'replication:replication' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '3bf24a64-e806-11e5-8238-ea129650fffe:4053450570''
161107 19:17:02 [Note] WSREP: sst_donor_thread signaled with 0
WSREP_SST: [INFO] Streaming with xbstream (20161107 19:17:02.257)
WSREP_SST: [INFO] Using socat as streamer (20161107 19:17:02.260)
WSREP_SST: [INFO] Streaming the backup to joiner at 170.71.77.88 4444 (20161107 19:17:02.273)
WSREP_SST: [INFO] Evaluating innobackupex --defaults-file=/etc/my.cnf $INNOEXTRA --galera-info --stream=$sfmt ${TMPDIR} 2>${DATA}/innobackup.backup.log | socat -u stdio TCP:170.71.77.88:4444; RC=( ${PIPESTATUS[@]} ) (20161107 19:17:02.277)
2016/11/07 19:17:38 socat[57475] E write(3, 0x689200, 8192): Broken pipe
WSREP_SST: [ERROR] innobackupex finished with error: 1.  Check /var/lib/mysql//innobackup.backup.log (20161107 19:17:38.181)
WSREP_SST: [ERROR] Cleanup after exit with status:22 (20161107 19:17:38.185)
161107 19:17:38 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup --role 'donor' --address '170.71.77.88:4444/xtrabackup_sst' --auth 'replication:replication' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '3bf24a64-e806-11e5-8238-ea129650fffe:4053450570'
161107 19:17:38 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup --role 'donor' --address '170.71.77.88:4444/xtrabackup_sst' --auth 'replication:replication' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '3bf24a64-e806-11e5-8238-ea129650fffe:4053450570': 22 (Invalid argument)
161107 19:17:38 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup --role 'donor' --address '170.71.77.88:4444/xtrabackup_sst' --auth 'replication:replication' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --gtid '3bf24a64-e806-11e5-8238-ea129650fffe:4053450570'
161107 19:17:38 [Warning] WSREP: 1.0 (DONORNODE): State transfer to 0.0 (JOINERNODE) failed: -22 (Invalid argument)
161107 19:17:38 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 4053457328)
161107 19:17:39 [Note] WSREP: Member 1.0 (DONORNODE) synced with group.
161107 19:17:39 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 4053457328)
161107 19:17:39 [Note] WSREP: Node 4159cb50-6323-11e6-946f-f77a9bdd31e2 state prim
161107 19:17:39 [Note] WSREP: Synchronized with group, ready for connections
161107 19:17:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
161107 19:17:39 [Note] WSREP: view(view_id(PRIM,4159cb50-6323-11e6-946f-f77a9bdd31e2,86) memb {
  4159cb50-6323-11e6-946f-f77a9bdd31e2,0
} joined {
} left {
} partitioned {
  0a872720-a551-11e6-9bc2-eedf51200d4b,0
})
161107 19:17:39 [Note] WSREP: forgetting 0a872720-a551-11e6-9bc2-eedf51200d4b (tcp://170.71.77.88:4567)
161107 19:17:39 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
161107 19:17:39 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 23bcece3-a551-11e6-aef5-b26eb62eab01
161107 19:17:39 [Note] WSREP: STATE EXCHANGE: sent state msg: 23bcece3-a551-11e6-aef5-b26eb62eab01
161107 19:17:39 [Note] WSREP: STATE EXCHANGE: got state msg: 23bcece3-a551-11e6-aef5-b26eb62eab01 from 0 (DONORNODE)
161107 19:17:39 [Note] WSREP: Quorum results:
  version    = 3,
  component  = PRIMARY,
  conf_id    = 73,
  members    = 1/1 (joined/total),
  act_id     = 4053457328,
  last_appl. = 4053457319,
  protocols  = 0/5/2 (gcs/repl/appl),
  group UUID = 3bf24a64-e806-11e5-8238-ea129650fffe
161107 19:17:39 [Note] WSREP: Flow-control interval: [16, 16]
161107 19:17:39 [Note] WSREP: New cluster view: global state: 3bf24a64-e806-11e5-8238-ea129650fffe:4053457328, view# 74: Primary, number of nodes: 1, my index: 0, protocol version 2
161107 19:17:39 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
161107 19:17:39 [Note] WSREP: REPL Protocols: 5 (3, 1)
161107 19:17:39 [Note] WSREP: Service thread queue flushed.
161107 19:17:39 [Note] WSREP: Assign initial position for certification: 4053457328, protocol version: 3
161107 19:17:39 [Note] WSREP: Service thread queue flushed.
161107 19:17:39 [Warning] WSREP: Releasing seqno 4053457328 before 4053457329 was assigned.
161107 19:17:44 [Note] WSREP:  cleaning up 0a872720-a551-11e6-9bc2-eedf51200d4b (tcp://170.71.77.88:4567)

innobackup.backup.log from

161107 19:17:02 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

161107 19:17:02  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;port=3306;mysql_socket=/var/lib/mysql/mysql.sock' as 'replication'  (using password: YES).
161107 19:17:02  version_check Connected to MySQL server
161107 19:17:02  version_check Executing a version check against the server...
161107 19:17:03  version_check Done.
161107 19:17:03 Connecting to MySQL server host: localhost, user: replication, password: set, port: 3306, socket: /var/lib/mysql/mysql.sock
Using server version 5.5.38-MariaDB-wsrep-log
innobackupex version 2.3.4 based on MySQL server 5.6.24 Linux (x86_64) (revision id: e80c779)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 5005
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 2097152000
xtrabackup: using O_DIRECT
161107 19:17:35 >> log scanned up to (182177566706404)
xtrabackup: Generating a list of tablespaces
161107 19:17:35 [01] Streaming ./ibdata1
161107 19:17:36 >> log scanned up to (182177569423322)
161107 19:17:37 >> log scanned up to (182177571865574)
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.

My.cnf settings:

[client]
port = 3306
socket = /var/lib/mysql/mysql.sock

[mysqld]
back_log = 1000
expire_logs_days = 1
innodb_autoextend_increment = 64
innodb_buffer_pool_instances = 64
innodb_buffer_pool_size = 100G
innodb_log_buffer_size = 128M
innodb_thread_concurrency = 0
thread_cache_size = 512
server-id = 3
port = 3306
binlog_cache_size = 4M
binlog-do-db = zabbix
binlog_format = ROW
binlog-row-event-max-size = 8192
datadir = /var/lib/mysql
innodb_concurrency_tickets = 5000
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_log_file_size = 2000M
innodb_old_blocks_time = 1000
innodb_stats_on_metadata = OFF
ignore-db-dir = lost+found
join_buffer_size = 1M
log-bin = mysql-bin
log-error = /var/log/mysql/mysqld.log
max_allowed_packet = 64M
max_connect_errors = 10000
max_connections = 1000
max_heap_table_size = 256M
net_buffer_length = 8K
pid-file = /var/run/mysqld/mysqld.pid
query_cache_size = 64000000
query_cache_type = 1
read_buffer_size = 1M
relay-log-recovery = 1
relay-log-space-limit = 2G
replicate-do-db = zabbix
replicate-ignore-db = mysql, performance_schema, lost+found
slave-skip-error = 1062
socket = /var/lib/mysql/mysql.sock
sort_buffer_size = 1M
table_open_cache = 4096
tmp_table_size = 1G
wait_timeout = 28800
key_buffer_size = 16M
binlog-format = row
innodb_flush_neighbor_pages = cont
innodb_max_dirty_pages_pct = 30
innodb_io_capacity = 6000
log-slave-updates = true
read_rnd_buffer_size = 16M
relay-log-purge = 1
thread_concurrency = 24
tmpdir = /dev/shm

innodb_file_per_table

skip-slave-start

[sst]
streamfmt=xbstream

[mysqldump]
max_allowed_packet = 64M
quick

[mysql]
no-auto-rehash

[mysqlhotcopy]

interactive-timeout

[mysqld_safe]
pid-file = /var/run/mysqld/mysqld.pid

!includedir /etc/my.cnf.d/

[mysqld]
large-pages
skip-external-locking

[mysqld]
large-pages
skip-external-locking

[mysqld]
wsrep_provider = /usr/lib64/galera/libgalera_smm.so
wsrep_provider_options = gcache.size=4G; gcache.page_size=1G
wsrep_cluster_address = gcomm://cernzbxdb201.cernerasp.com
wsrep_cluster_name = galera_cluster
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 2
innodb_locks_unsafe_for_binlog = 1
wsrep_sst_method = xtrabackup-v2
wsrep_slave_threads = 64
wsrep_sst_auth =replication:replication
  • I have wiped out mysql and have reinstalled it. I am attempting to do a full sst but am still getting these same exact errors – Som3guy Nov 8 '16 at 5:57
  • Well the error suggests about the same... clear /var/lib/mysql/ and reattempt. WSREP_SST: [ERROR] Data directory /var/lib/mysql/ may not be empty: lp:1193240 Manual intervention required in that case (20161107 19:17:38.167) If you have already tried that, do you still get the same error!!!? – mysql_user Nov 8 '16 at 7:06
  • Yes I did. The same exact error. – Som3guy Nov 8 '16 at 7:36
  • How much RAM do you have? (Some settings seem quite high.) – Rick James Nov 12 '16 at 4:20
  • 512GB of RAM for mysql and two other apps, plus whatever the OS needs. – Som3guy Nov 20 '16 at 5:47
0

Here is what I did to fix this:

  1. I wiped out mysql.(Deleted all mariadb packages) Becuase the tablespace was corrupted.
  2. I have reinstalled all mariadb packages.
  3. I found I was only replicating the zabbix database so I deleted all files in /var/lib/mysql
  4. I set this setting wsrep_sst_method = xtrabackup-v2 as it was wsrep_sst_method = xtrabackup.
  5. I started mysql.
  6. Now I am waiting however long it takes to replicate 3.1TB from my donor node to the joiner.
|improve this answer|||||
  • Once replication catches up and is working as expected I will mark this as the answer, until then I will just post this here as this is what I have done and it seems to have fixed my issue. – Som3guy Nov 8 '16 at 7:42
  • wsrep_sst_method = xtrabackup-v2 was already set! but glad you could fix it. Have fun. – mysql_user Nov 8 '16 at 9:08
  • Yep, It was not part of my chef configuration. So when I wiped it out I ran chef-client to reinstall and it set that setting to xtrabackup-v2. – Som3guy Nov 8 '16 at 20:11
  • I did get this error in the middle of the replication however Error: failed to execute query FLUSH TABLES WITH READ LOCK: Lock wait timeout exceeded; try restarting transaction So I ran: set innodb_lock_wait_timeout=100 from 50. Then started the replication from scratch again. – Som3guy Nov 8 '16 at 20:13
0

If you are using systemd, try to increase service timeout for mysql/mariadb to some value big enough to finish sst (couple of hours). What is happening is that systemd sends signal to mysql after timeout expires (900s in my case), which closes socat on joiner side, and that breaks pipe on donor side too.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.