0

I have 2 nodes. One a VPS on NFO Servers and another on Azure. I have setup the firewalls/disabled the firewalls. The nodes are CentOS 7, Latest Kernel. The configs are as follows:

Node 1 (Master DB) /etc/my.cnf.d/server.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# network
connect_timeout = 60
wait_timeout = 28800
interactive_timeout = 28800
max_connections = 100000
max_allowed_packet = 256M
max_connect_errors = 1000
skip-host-cache
skip-name-resolve

# this is only for the mysqld standalone daemon
[mysqld]
#
# * Galera-related settings
#

connect_timeout = 60
wait_timeout = 28800
interactive_timeout = 28800
max_connections = 100000
max_allowed_packet = 256M
max_connect_errors = 1000
skip-host-cache
skip-name-resolve
[galera]

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://<NODE1IP>,<NODE2IP>"

## Galera Cluster Configuration
wsrep_cluster_name="cluster1"
## Galera Synchronization Configuration
wsrep_sst_method=rsync
## Galera Node Configuration
wsrep_node_address="<NODE1IP>"
wsrep_node_name="node1"

wsrep_provider_options="evs.inactive_timeout=PT45S;evs.install_timeout=PT30S;evs.keepalive_period=PT10S;evs.max_install_timeouts=8;evs.send_window=512;evs.suspect_timeout=PT30S;evs.user_send_window=256;gmcast.time_wait=PT5S;gmcast.peer_timeout=PT20S;pc.announce_timeout=PT20S"

[embedded]
[mariadb]
[mariadb-10.1]

Node 2 (Secondary Node) /etc/my.cnf.d/server.cnf

[server]

connect_timeout = 60
wait_timeout = 28800
interactive_timeout = 28800
max_connections = 100000
max_allowed_packet = 256M
max_connect_errors = 1000
#skip-host-cache
#skip-name-resolve

[mysqld]

connect_timeout = 60
wait_timeout = 28800
interactive_timeout = 28800
max_connections = 100000
max_allowed_packet = 256M
max_connect_errors = 1000
#skip-host-cache
#skip-name-resolve

[galera]

binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://<NODE1IP>,<NODE2IP>"

## Galera Cluster Configuration
wsrep_cluster_name="cluster1"
## Galera Synchronization Configuration
wsrep_sst_method=rsync
## Galera Node Configuration
wsrep_node_address="<NODE2IP>"
wsrep_node_name="node2"

wsrep_provider_options="evs.inactive_timeout=PT45S;evs.install_timeout=PT45S;evs.keepalive_period=PT10S;evs.max_install_timeouts=8;evs.send_window=512;evs.suspect_timeout=PT30S;evs.user_send_window=256;gmcast.time_wait=PT5S;gmcast.peer_timeout=PT20S;pc.announce_timeout=PT20S"

[embedded]

[mariadb]

[mariadb-10.1]

Node 2 LOG:

Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913375615104 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913375615104 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913375615104 [Note] WSREP: Opened channel 'cluster1'
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913375615104 [Note] WSREP: Waiting for SST to complete.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 253b55ca-0a28-11e8-9489-5eeaebba8209
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: STATE EXCHANGE: sent state msg: 253b55ca-0a28-11e8-9489-5eeaebba8209
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: STATE EXCHANGE: got state msg: 253b55ca-0a28-11e8-9489-5eeaebba8209 from 0 (node2)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: STATE EXCHANGE: got state msg: 253b55ca-0a28-11e8-9489-5eeaebba8209 from 1 (node1)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Quorum results:
Feb  4 21:53:37 node2 mysqld: version    = 4,
Feb  4 21:53:37 node2 mysqld: component  = PRIMARY,
Feb  4 21:53:37 node2 mysqld: conf_id    = 47,
Feb  4 21:53:37 node2 mysqld: members    = 1/2 (joined/total),
Feb  4 21:53:37 node2 mysqld: act_id     = 23,
Feb  4 21:53:37 node2 mysqld: last_appl. = -1,
Feb  4 21:53:37 node2 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
Feb  4 21:53:37 node2 mysqld: group UUID = 73533f81-0a01-11e8-a97c-8eced69c69dc
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Flow-control interval: [23, 23]
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Trying to continue unpaused monitor
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 23)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: State transfer required:
Feb  4 21:53:37 node2 mysqld: Group state: 73533f81-0a01-11e8-a97c-8eced69c69dc:23
Feb  4 21:53:37 node2 mysqld: Local state: 00000000-0000-0000-0000-000000000000:-1
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: New cluster view: global state: 73533f81-0a01-11e8-a97c-8eced69c69dc:23, view# 48: Primary, number of nodes: 2, my index: 0, protocol version 3
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Warning] WSREP: Gap in state sequence. Need state transfer.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030170368 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '74.91.120.198' --datadir '/var/lib/mysql/'   --parent '2389'  '' '
Feb  4 21:53:37 node2 mysqld: which: no ifconfig in (/usr/sbin:/sbin:/usr//bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
Feb  4 21:53:37 node2 rsyncd[2503]: rsyncd version 3.1.2 starting, listening on port 4444
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: Prepared SST request: rsync|74.91.120.198:4444/rsync_sst
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: REPL Protocols: 7 (3, 2)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: Assign initial position for certification: 23, protocol version: 3
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913282713344 [Note] WSREP: Service thread queue flushed.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (73533f81-0a01-11e8-a97c-8eced69c69dc): 1 (Operation not permitted)
Feb  4 21:53:37 node2 mysqld: at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Member 0.0 (node2) requested state transfer from '*any*'. Selected 1.0 (node1)(SYNCED) as donor.
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913039169280 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 23)
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: Requesting state transfer: success, donor: 1
Feb  4 21:53:37 node2 mysqld: 2018-02-04 21:53:37 139913030473472 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 73533f81-0a01-11e8-a97c-8eced69c69dc:23
Feb  4 21:53:56 node2 mysqld: 2018-02-04 21:53:56 139913047561984 [Note] WSREP: (24a276a0, 'tcp://0.0.0.0:4567') connection to peer 24a276a0 with addr tcp://74.91.120.198:4567 timed out, no messages seen in PT20S
Feb  4 21:53:56 node2 mysqld: 2018-02-04 21:53:56 139913047561984 [Note] WSREP: (24a276a0, 'tcp://0.0.0.0:4567') turning message relay requesting off
Feb  4 21:55:06 node2 systemd: mariadb.service start operation timed out. Terminating.
Feb  4 21:55:20 node2 yum[2914]: Installed: socat-1.7.3.2-2.el7.x86_64
Feb  4 21:55:45 node2 mysqld: 2018-02-04 21:55:45 139913039169280 [Warning] WSREP: 1.0 (node1): State transfer to 0.0 (node2) failed: -255 (Unknown error 255)
Feb  4 21:55:45 node2 mysqld: 2018-02-04 21:55:45 139913039169280 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():736: Will never receive state. Need to abort.
Feb  4 21:55:45 node2 mysqld: 2018-02-04 21:55:45 139913039169280 [Note] WSREP: gcomm: terminating thread
Feb  4 21:55:45 node2 mysqld: 2018-02-04 21:55:45 139913039169280 [Note] WSREP: gcomm: joining thread
Feb  4 21:55:45 node2 mysqld: 2018-02-04 21:55:45 139913039169280 [Note] WSREP: gcomm: closing backend
Feb  4 21:56:05 node2 mysqld: 2018-02-04 21:56:05 139913039169280 [Note] WSREP: view(view_id(NON_PRIM,24a276a0,48) memb {
Feb  4 21:56:05 node2 mysqld: 24a276a0,0
Feb  4 21:56:05 node2 mysqld: } joined {
Feb  4 21:56:05 node2 mysqld: } left {
Feb  4 21:56:05 node2 mysqld: } partitioned {
Feb  4 21:56:05 node2 mysqld: 7e3c9660,0
Feb  4 21:56:05 node2 mysqld: })
Feb  4 21:56:05 node2 mysqld: 2018-02-04 21:56:05 139913039169280 [Note] WSREP: view((empty))
Feb  4 21:56:05 node2 mysqld: 2018-02-04 21:56:05 139913039169280 [Note] WSREP: gcomm: closed
Feb  4 21:56:05 node2 mysqld: 2018-02-04 21:56:05 139913039169280 [Note] WSREP: /usr/sbin/mysqld: Terminated.
Feb  4 21:56:05 node2 systemd: mariadb.service: main process exited, code=killed, status=6/ABRT
Feb  4 21:56:05 node2 systemd: Stopped MariaDB 10.2.12 database server.
Feb  4 21:56:05 node2 systemd: Unit mariadb.service entered failed state.
Feb  4 21:56:05 node2 systemd: mariadb.service failed.
Feb  4 21:56:06 node2 mysqld: WSREP_SST: [ERROR] Parent mysqld process (PID:2389) terminated unexpectedly. (20180204 21:56:06.332)
Feb  4 21:56:06 node2 mysqld: WSREP_SST: [INFO] Joiner cleanup. rsync PID: 2503 (20180204 21:56:06.334)
Feb  4 21:56:06 node2 rsyncd[2503]: sent 0 bytes  received 0 bytes  total size 0
Feb  4 21:56:06 node2 mysqld: WSREP_SST: [INFO] Joiner cleanup done. (20180204 21:56:06.839)

NODE 1 LOG:

Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285264168704 [Note] WSREP: save pc into disk
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: Member 1.0 (node1) synced with group.
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 23)
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: Synchronized with group, ready for connections
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285264168704 [Note] WSREP: forgetting 6a8672dc (tcp://74.91.120.198:4567)
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: STATE_EXCHANGE: sent state UUID: b8044650-0a27-11e8-9fe5-43b0fd1b829f
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: STATE EXCHANGE: sent state msg: b8044650-0a27-11e8-9fe5-43b0fd1b829f
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: STATE EXCHANGE: got state msg: b8044650-0a27-11e8-9fe5-43b0fd1b829f from 0 (node1)
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: Quorum results:
Feb  5 03:50:34 node1 mysqld: version    = 4,
Feb  5 03:50:34 node1 mysqld: component  = PRIMARY,
Feb  5 03:50:34 node1 mysqld: conf_id    = 44,
Feb  5 03:50:34 node1 mysqld: members    = 1/1 (joined/total),
Feb  5 03:50:34 node1 mysqld: act_id     = 23,
Feb  5 03:50:34 node1 mysqld: last_appl. = 0,
Feb  5 03:50:34 node1 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
Feb  5 03:50:34 node1 mysqld: group UUID = 73533f81-0a01-11e8-a97c-8eced69c69dc
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: Flow-control interval: [16, 16]
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285255776000 [Note] WSREP: Trying to continue unpaused monitor
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: New cluster view: global state: 73533f81-0a01-11e8-a97c-8eced69c69dc:23, view# 45: Primary, number of nodes: 1, my index: 0, protocol version 3
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: REPL Protocols: 7 (3, 2)
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285247383296 [Note] WSREP: Assign initial position for certification: 23, protocol version: 3
Feb  5 03:50:34 node1 mysqld: 2018-02-05  3:50:34 140285297739520 [Note] WSREP: Service thread queue flushed.
Feb  5 03:50:39 node1 mysqld: 2018-02-05  3:50:39 140285264168704 [Note] WSREP:  cleaning up 6a8672dc (tcp://74.91.120.198:4567)
Feb  5 03:51:00 node1 mysqld: 2018-02-05  3:51:00 140285264168704 [Note] WSREP: (7e3c9660, 'tcp://0.0.0.0:4567') connection established to c7a9e931 tcp://74.91.120.198:4567
Feb  5 03:51:00 node1 mysqld: 2018-02-05  3:51:00 140285264168704 [Note] WSREP: (7e3c9660, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
Feb  5 03:51:00 node1 mysqld: 2018-02-05  3:51:00 140285264168704 [Note] WSREP: declaring c7a9e931 at tcp://74.91.120.198:4567 stable
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285264168704 [Note] WSREP: Node 7e3c9660 state prim
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285264168704 [Note] WSREP: view(view_id(PRIM,7e3c9660,46) memb {
Feb  5 03:51:01 node1 mysqld: 7e3c9660,0
Feb  5 03:51:01 node1 mysqld: c7a9e931,0
Feb  5 03:51:01 node1 mysqld: } joined {
Feb  5 03:51:01 node1 mysqld: } left {
Feb  5 03:51:01 node1 mysqld: } partitioned {
Feb  5 03:51:01 node1 mysqld: })
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285264168704 [Note] WSREP: save pc into disk
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 2
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: STATE_EXCHANGE: sent state UUID: c7feaa26-0a27-11e8-85f8-cf8f4662ea16
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: STATE EXCHANGE: sent state msg: c7feaa26-0a27-11e8-85f8-cf8f4662ea16
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: STATE EXCHANGE: got state msg: c7feaa26-0a27-11e8-85f8-cf8f4662ea16 from 0 (node1)
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: STATE EXCHANGE: got state msg: c7feaa26-0a27-11e8-85f8-cf8f4662ea16 from 1 (node2)
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: Quorum results:
Feb  5 03:51:01 node1 mysqld: version    = 4,
Feb  5 03:51:01 node1 mysqld: component  = PRIMARY,
Feb  5 03:51:01 node1 mysqld: conf_id    = 45,
Feb  5 03:51:01 node1 mysqld: members    = 1/2 (joined/total),
Feb  5 03:51:01 node1 mysqld: act_id     = 23,
Feb  5 03:51:01 node1 mysqld: last_appl. = 0,
Feb  5 03:51:01 node1 mysqld: protocols  = 0/7/3 (gcs/repl/appl),
Feb  5 03:51:01 node1 mysqld: group UUID = 73533f81-0a01-11e8-a97c-8eced69c69dc
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: Flow-control interval: [23, 23]
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: Trying to continue unpaused monitor
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: New cluster view: global state: 73533f81-0a01-11e8-a97c-8eced69c69dc:23, view# 46: Primary, number of nodes: 2, my index: 0, protocol version 3
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: REPL Protocols: 7 (3, 2)
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: Assign initial position for certification: 23, protocol version: 3
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285297739520 [Note] WSREP: Service thread queue flushed.
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: Member 1.0 (node2) requested state transfer from '*any*'. Selected 0.0 (node1)(SYNCED) as donor.
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285255776000 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 23)
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140282322433792 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'donor' --address '74.91.120.198:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'     '' --gtid '73533f81-0a01-11e8-a97c-8eced69c69dc:23' --gtid-domain-id '0''
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140285247383296 [Note] WSREP: sst_donor_thread signaled with 0
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140282322433792 [Note] WSREP: Flushing tables for SST...
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140282322433792 [Note] WSREP: Provider paused at 73533f81-0a01-11e8-a97c-8eced69c69dc:23 (137)
Feb  5 03:51:01 node1 mysqld: 2018-02-05  3:51:01 140282322433792 [Note] WSREP: Tables flushed.
Feb  5 03:51:20 node1 mysqld: 2018-02-05  3:51:20 140285264168704 [Note] WSREP: (7e3c9660, 'tcp://0.0.0.0:4567') turning message relay requesting off
Feb  5 03:53:09 node1 mysqld: rsync: failed to connect to 74.91.120.198 (74.91.120.198): Connection timed out (110)
Feb  5 03:53:09 node1 mysqld: rsync error: error in socket IO (code 10) at clientserver.c(122) [sender=3.0.9]
Feb  5 03:53:09 node1 mysqld: WSREP_SST: [ERROR] rsync returned code 10: (20180205 03:53:09.214)
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140282322433792 [ERROR] WSREP: Failed to read from: wsrep_sst_rsync --role 'donor' --address '74.91.120.198:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'     '' --gtid '73533f81-0a01-11e8-a97c-8eced69c69dc:23' --gtid-domain-id '0'
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140282322433792 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'donor' --address '74.91.120.198:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'     '' --gtid '73533f81-0a01-11e8-a97c-8eced69c69dc:23' --gtid-domain-id '0': 255 (Unknown error 255)
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140282322433792 [Note] WSREP: resuming provider at 137
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140282322433792 [Note] WSREP: Provider resumed.
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140282322433792 [ERROR] WSREP: Command did not run: wsrep_sst_rsync --role 'donor' --address '74.91.120.198:4444/rsync_sst' --socket '/var/lib/mysql/mysql.sock' --datadir '/var/lib/mysql/'     '' --gtid '73533f81-0a01-11e8-a97c-8eced69c69dc:23' --gtid-domain-id '0'
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140285255776000 [Warning] WSREP: 0.0 (node1): State transfer to 1.0 (node2) failed: -255 (Unknown error 255)
Feb  5 03:53:09 node1 mysqld: 2018-02-05  3:53:09 140285255776000 [Note] WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 23)
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285255776000 [Note] WSREP: Member 0.0 (node1) synced with group.
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285255776000 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 23)
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285247383296 [Note] WSREP: Synchronized with group, ready for connections
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285247383296 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285264168704 [Note] WSREP: forgetting c7a9e931 (tcp://74.91.120.198:4567)
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285264168704 [Note] WSREP: Node 7e3c9660 state prim
Feb  5 03:53:10 node1 mysqld: 2018-02-05  3:53:10 140285264168704 [Note] WSREP: view(view_id(PRIM,7e3c9660,47) memb {

Seems to be an Rsync error but I have all firewalls off/configured. Anyone have any suggestions. I am having nightmares with this.

  • The two log files show different time stamps. Is this because of different time zones? – dbdemon Feb 5 '18 at 12:10
1

A few observations / ideas:

  • You're doing geo-distributed replication since you're replicating across the Internet between two different hosting providers. Make sure you use appropriate config values to allow for the increased latency and connectivity failures. See e.g. Galera's documentation, including this part.
  • The first error for node 2 appears to be no ifconfig. Do you not have that installed?
  • The main issue appears to be this line in the log for node 2:

    Feb  4 21:53:56 node2 mysqld: 2018-02-04 21:53:56 139913047561984 [Note] WSREP: (24a276a0, 'tcp://0.0.0.0:4567') connection to peer 24a276a0 with addr tcp://74.91.120.198:4567 timed out, no messages seen in PT20S  
    

    This timeout somehow needs to be prevented. See the Galera doc link above, and go from there.

  • rsync is not the optimal SST method. Install Percona Tools' xtrabackup, or try the new mariabackup method.
  • I don't think MariaDB distributes SELinux policies that work with Galera (at least they didn't in the past), so therefore make sure to disable SELinux (or put it into permissive mode). If you don't do this, you risk that things like SST will be blocked.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.