0

I setup a 4-node Galera cluster on the following platform:

CentOS 8.1.1911, with MariaDB 10.3.17 and Galera 3.26.

Galera config:
wsrep_cluster_address="gcomm://192.168.10.47,192.168.10.48,192.168.11.47,192.168.11.48"

I setup a new cluster on node 1 running "galera_new_cluster". Then I started mariadb on the other nodes which all joined successully.

I wrote a script to examine the wsrep status on each node and noted the following values:

wsrep_cluster_status: Primary
wsrep_cluster_size: 4
wsrep_local_state_comment: Synced
wsrep_evs_state: OPERATIONAL

Then to test the cluster, I restarted each node starting with node 4, 3, 2 and then 1. I monitored the above status and noticed the cluster size reduced to 3 on a reboot of nodes 4, 3 and 2 and then went back to 4 after re-joining.

However, when rebooting node 1, it did NOT re-join the cluster successfully, and the status on each of the other 3 nodes was:

wsrep_local_state_comment: Initialising

The logs on node 1 showed:

2020-04-30 12:31:36 0 [Note] WSREP: gcomm: connecting to group 'Web_Cluster', peer '192.168.10.47:,192.168.10.48:,192.168.11.47:,192.168.11.48:'
2020-04-30 12:31:36 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') connection established to 26c8ad71 tcp://192.168.10.47:4567
2020-04-30 12:31:36 0 [Warning] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') address 'tcp://192.168.10.47:4567' points to own listening address, blacklisting
2020-04-30 12:31:36 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') connection established to 150abdb8 tcp://192.168.10.48:4567
2020-04-30 12:31:36 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2020-04-30 12:31:36 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') connection established to 2f355b90 tcp://192.168.11.48:4567
2020-04-30 12:31:36 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') connection established to c8c39206 tcp://192.168.11.47:4567
2020-04-30 12:31:38 0 [Note] WSREP: declaring 150abdb8 at tcp://192.168.10.48:4567 stable
2020-04-30 12:31:38 0 [Note] WSREP: declaring 2f355b90 at tcp://192.168.11.48:4567 stable
2020-04-30 12:31:38 0 [Note] WSREP: declaring c8c39206 at tcp://192.168.11.47:4567 stable
2020-04-30 12:31:39 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') connection to peer 26c8ad71 with addr tcp://192.168.10.47:4567 timed out, no messages seen in PT3S
2020-04-30 12:31:40 0 [Note] WSREP: (26c8ad71, 'tcp://0.0.0.0:4567') turning message relay requesting off
2020-04-30 12:32:07 0 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
         at gcomm/src/pc.cpp:connect():158
2020-04-30 12:32:07 0 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():209: Failed to open backend connection: -110 (Connection timed out)
2020-04-30 12:32:07 0 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1458: Failed to open channel 'Web_Cluster' at 'gcomm://192.168.10.47,192.168.10.48,192.168.11.47,192.168.11.48': -110 (Connection timed out)
2020-04-30 12:32:07 0 [ERROR] WSREP: gcs connect failed: Connection timed out
2020-04-30 12:32:07 0 [ERROR] WSREP: wsrep::connect(gcomm://192.168.10.47,192.168.10.48,192.168.11.47,192.168.11.48) failed: 7
2020-04-30 12:32:07 0 [ERROR] Aborting

I couldn't figure out why there was a connection time-out as all nodes could talk to each other. In the end the only resolution was to run "galera_new_cluster" on node 1 again.

When running:

systemctl status mariadb

The only difference I could see on node 1 compared to the others was the extra parameter:

9195 /usr/libexec/mysqld --basedir=/usr --wsrep-new-cluster
9051 /usr/libexec/mysqld --basedir=/usr
18193 /usr/libexec/mysqld --basedir=/usr
16338 /usr/libexec/mysqld --basedir=/usr

Any insight as to how I can get node 1 to re-join after a reboot would be appreciated, as this is a potential issue waiting to happen.

2
  • Did you restart each node before testing the next node? Or did you take down all 4 at the same time? – Rick James May 7 '20 at 21:41
  • Yes, the cluster size reduced to 3 on a reboot of nodes 4, 3 and 2 and then went back to 4 after re-joining. So each node was rebooted from a fully sync'd cluster. – mphodge May 10 '20 at 9:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.