0

I'm trying to install a new galera cluster. The primary host started fine, but the secondaries are failing during the state transfer with rsync, and are not starting. I haven't been able to fix the problem

Here's the log:

Mar 19 09:43:14 vagrant-ubuntu-trusty-64 mysqld_safe: Starting mysqld daemon with databases from /var/lib/mysql
Mar 19 09:43:14 vagrant-ubuntu-trusty-64 mysqld_safe: WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.V1VQNk' --pid-file='/var/lib/mysql/node2-recover.pid'
Mar 19 09:43:14 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:14 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-1~trusty-wsrep) starting as process 7936 ...
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld_safe: WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] /usr/sbin/mysqld (mysqld 10.0.29-MariaDB-1~trusty-wsrep) starting as process 7986 ...
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Read nil XID from storage engines, skipping position init
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: wsrep_load(): Galera 25.3.19(r3667) by Codership Oy <[email protected]> loaded successfully.
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: CRC-32C: using hardware acceleration.
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Warning] WSREP: Could not open state file for reading: '/var/lib/mysql//grastate.dat'
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootsrap: 1
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 192.168.0.109; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; p
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: c.checksum = false; pc.
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(00000000-0000-0000-0000-000000000000:-1)
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: wsrep_sst_grab()
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Start replication
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: protonet asio version 0
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: Using CRC-32C for message checksums.
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: backend: asio
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: gcomm thread scheduling priority set to other:0 
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: restore pc from disk failed
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: GMCast version 0
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: EVS version 0
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: gcomm: connecting to group 'test', peer '192.168.0.102:,192.168.0.104:,192.168.0.109:'
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') connection established to 7bc8012e tcp://192.168.0.109:4567
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Warning] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') address 'tcp://192.168.0.109:4567' points to own listening address, blacklisting
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') connection established to 76006a4b tcp://192.168.0.102:4567
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 
Mar 19 09:43:18 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:18 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') connection established to 7b77d168 tcp://192.168.0.104:4567
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: declaring 76006a4b at tcp://192.168.0.102:4567 stable
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: declaring 7b77d168 at tcp://192.168.0.104:4567 stable
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Node 76006a4b state prim
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: view(view_id(PRIM,76006a4b,3) memb {
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #01176006a4b,0
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #0117b77d168,0
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #0117bc8012e,0
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: } joined {
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: } left {
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: } partitioned {
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: })
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: save pc into disk
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: gcomm: connected
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Opened channel 'test'
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Waiting for SST to complete.
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: STATE EXCHANGE: sent state msg: 7c16951b-0c88-11e7-ad86-7bb28eca7f7e
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: STATE EXCHANGE: got state msg: 7c16951b-0c88-11e7-ad86-7bb28eca7f7e from 0 (node1)
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: STATE EXCHANGE: got state msg: 7c16951b-0c88-11e7-ad86-7bb28eca7f7e from 1 (node3)
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: STATE EXCHANGE: got state msg: 7c16951b-0c88-11e7-ad86-7bb28eca7f7e from 2 (node2)
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Quorum results:
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011version    = 4,
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011component  = PRIMARY,
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011conf_id    = 2,
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011members    = 1/3 (joined/total),
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011act_id     = 7,
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011last_appl. = -1,
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011protocols  = 0/7/3 (gcs/repl/appl),
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011group UUID = 760199b2-0c88-11e7-be7d-f2d7ea489521
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Flow-control interval: [28, 28]
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 7)
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: State transfer required: 
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011Group state: 760199b2-0c88-11e7-be7d-f2d7ea489521:7
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: New cluster view: global state: 760199b2-0c88-11e7-be7d-f2d7ea489521:7, view# 3: Primary, number of nodes: 3, my index: 2, protocol version 3
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Warning] WSREP: Gap in state sequence. Need state transfer.
Mar 19 09:43:19 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:19 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '192.168.0.109' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --parent '7986' --binlog '/var/log/mysql/mariadb-bin' '
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Member 1.0 (node3) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 rsyncd[8036]: rsyncd version 3.1.0 starting, listening on port 4444
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Prepared SST request: rsync|192.168.0.109:4444/rsync_sst
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: REPL Protocols: 7 (3, 2)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Assign initial position for certification: 7, protocol version: 3
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Service thread queue flushed.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (760199b2-0c88-11e7-be7d-f2d7ea489521): 1 (Operation not permitted)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Warning] WSREP: Member 2.0 (node2) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Requesting state transfer failed: -11(Resource temporarily unavailable). Will keep retrying every 1 second(s)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Warning] WSREP: 0.0 (node1): State transfer to 1.0 (node3) failed: -141 (Unknown error 141)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Member 0.0 (node1) synced with group.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: declaring 76006a4b at tcp://192.168.0.102:4567 stable
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: forgetting 7b77d168 (tcp://192.168.0.104:4567)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Node 76006a4b state prim
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: view(view_id(PRIM,76006a4b,4) memb {
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #01176006a4b,0
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #0117bc8012e,0
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: } joined {
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: } left {
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: } partitioned {
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #0117b77d168,0
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: })
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: save pc into disk
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: forgetting 7b77d168 (tcp://192.168.0.104:4567)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: STATE EXCHANGE: sent state msg: 7c8f8b36-0c88-11e7-8bd7-33c58acb74e1
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: STATE EXCHANGE: got state msg: 7c8f8b36-0c88-11e7-8bd7-33c58acb74e1 from 0 (node1)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: STATE EXCHANGE: got state msg: 7c8f8b36-0c88-11e7-8bd7-33c58acb74e1 from 1 (node2)
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Quorum results:
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011version    = 4,
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011component  = PRIMARY,
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011conf_id    = 3,
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011members    = 1/2 (joined/total),
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011act_id     = 7,
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011last_appl. = 0,
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011protocols  = 0/7/3 (gcs/repl/appl),
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: #011group UUID = 760199b2-0c88-11e7-be7d-f2d7ea489521
Mar 19 09:43:20 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:20 [Note] WSREP: Flow-control interval: [23, 23]
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor.
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 7)
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: Requesting state transfer: success after 2 tries, donor: 0
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: GCache history reset: old(00000000-0000-0000-0000-000000000000:0) -> new(760199b2-0c88-11e7-be7d-f2d7ea489521:7)
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Warning] WSREP: 0.0 (node1): State transfer to 1.0 (node2) failed: -141 (Unknown error 141)
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: gcomm: terminating thread
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: gcomm: joining thread
Mar 19 09:43:21 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:21 [Note] WSREP: gcomm: closing backend
Mar 19 09:43:22 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:22 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') connection to peer 7bc8012e with addr tcp://192.168.0.109:4567 timed out, no messages seen in PT3S
Mar 19 09:43:22 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:22 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') turning message relay requesting off
Mar 19 09:43:24 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:24 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') connection to peer 76006a4b with addr tcp://192.168.0.102:4567 timed out, no messages seen in PT3S
Mar 19 09:43:24 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:24 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.0.102:4567 
Mar 19 09:43:25 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:25 [Note] WSREP:  cleaning up 7b77d168 (tcp://192.168.0.104:4567)
Mar 19 09:43:25 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:25 [Note] WSREP: (7bc8012e, 'tcp://0.0.0.0:4567') reconnecting to 76006a4b (tcp://192.168.0.102:4567), attempt 0
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: evs::proto(7bc8012e, LEAVING, view_id(REG,76006a4b,4)) suspecting node: 76006a4b
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: evs::proto(7bc8012e, LEAVING, view_id(REG,76006a4b,4)) suspected node without join message, declaring inactive
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: view(view_id(NON_PRIM,76006a4b,4) memb {
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: #0117bc8012e,0
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: } joined {
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: } left {
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: } partitioned {
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: #01176006a4b,0
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: })
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: view((empty))
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: gcomm: closed
Mar 19 09:43:26 vagrant-ubuntu-trusty-64 mysqld: 170319  9:43:26 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Mar 19 09:43:29 vagrant-ubuntu-trusty-64 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:7986) terminated unexpectedly. (20170319 09:43:29.239)
Mar 19 09:43:29 vagrant-ubuntu-trusty-64 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 8036 (20170319 09:43:29.242)
Mar 19 09:43:29 vagrant-ubuntu-trusty-64 rsyncd[8036]: sent 0 bytes  received 0 bytes  total size 0
Mar 19 09:43:29 vagrant-ubuntu-trusty-64 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20170319 09:43:29.751)
Mar 19 09:43:29 vagrant-ubuntu-trusty-64 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended
Mar 19 09:43:47 vagrant-ubuntu-trusty-64 /etc/init.d/mysql[8358]: 0 processes alive and '/usr/bin/mysqladmin --defaults-file=/etc/mysql/debian.cnf ping' resulted in
Mar 19 09:43:47 vagrant-ubuntu-trusty-64 /etc/init.d/mysql[8358]: #007/usr/bin/mysqladmin: connect to server at 'localhost' failed
Mar 19 09:43:47 vagrant-ubuntu-trusty-64 /etc/init.d/mysql[8358]: error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111 "Connection refused")'
Mar 19 09:43:47 vagrant-ubuntu-trusty-64 /etc/init.d/mysql[8358]: Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!
Mar 19 09:43:47 vagrant-ubuntu-trusty-64 /etc/init.d/mysql[8358]:

And here's the configuration file:

[mysqld]
transaction-isolation = READ-COMMITTED

key_buffer = 16M
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 1050
#expire_logs_days = 10
#max_binlog_size = 100M

log_bin=/var/lib/mysql/mysql_binary_log

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="test"
wsrep_cluster_address="gcomm://192.168.0.102,192.168.0.104,192.168.0.109"

# Galera Synchronization Configuration
wsrep_sst_method=rsync

# Galera Node Configuration
wsrep_node_address=192.168.0.109
wsrep_node_name=node2
1
  • How big is the database size? Does network connectivity working (ping, rsync)?
    – Sybil
    Mar 20, 2017 at 8:57

2 Answers 2

0

I found the Problem. I was using ansible for deployment which causes some error that has something to do with logging. See this thread for details

0

I had a similar issue, and had to get more details from the donor node's logs. In my case it was trying to use rsync for the sst transfer, and the port for rsync (4444) was not open on the firewall. To fix it, I ran these commands on all the nodes:

firewall-cmd --zone=public --add-port=4444/tcp --permanent
firewall-cmd --reload

That seemed to do the trick.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.