what is the right way to restart a mariadb galera cluster?

Question

I am trying to restart cluster. I have two problems in stopping cluster and starting cluster

In stopping cluster, I follow the instruction https://software.mirantis.com/reference-documentation-on-fuel-folsom/galera/

For safe, I stop replication on node by set global wsrep_on='OFF' and then, I cannot stop it gracefully. I must kill it. It is not clean.

In starting cluster, I start the first node by service mysql start --wsrep-cluster-address=gcomm:// I am not sure it is the right way because it create new cluster.

The instruction is in https://mariadb.com/kb/en/mariadb/documentation/replication-cluster-multi-master/galera/getting-started-with-mariadb-galera-cluster/#state-transfer-failure not available to me.

What is the right way to me ?

Answer 1

As far as I read the right way is to use the init.d/systemd scripts of MySQL/MariaDB in order to stop a node. This will inform WSREP/Galera that the replication must stop and will save on the disk, in the MySQL/MariaDB data folder, in a file the last transaction number that the node has committed (So that next time it will boot, the nodes will compare their last committed transaction and sync).

You have to stop a node and wait a little bit (like a minute or so) before to stop the next node, to be sure the other nodes see that.

After that, it is very important to start again the nodes in the reverse ordre, so that the last node you shutdown is the first booting one, having the most up-to-date data. Then the next nodes will be able to sync with the first one.

Using wsrep_on='OFF' doesn't sound right to me, WSREP is embedded in MySQL, so when MySQL gracefully stops, WSREP knows what it has to do.

Answer 2

This is a terribly old question and in the meantime MariaDB Galera Cluster has improved a lot both in terms of usability and more core features.

I think you have always been able to just restart each node in turn, so in other words do a rolling restart, thereby avoiding disruption to apps. That is sort of the point of a high availability solution like Galera.

node1:

systemctl restart mariadb

(Or equivalent command on OSes without systemd.)

Wait until the command completes, then make sure the node has completed syncing with the cluster, which can take a little while on large/busy systems:

mariadb -e "show status
  where variable_name in ('wsrep_ready', 
    'wsrep_local_state_comment', 
    'wsrep_connected',
    'wsrep_cluster_size');"

You want that to say "Synced", the expected number of nodes and wsrep_connected and wsrep_ready both ON.

Then repeat on the other nodes.