Restarting Galera MySQL after proper Shutdown

Question

I already have read and followed dozens of documents describing a split brain situation and how to restart the cluster after then.

However, I am less in this situation as in the situation of shutting down the Cluster properly, that is, stopping all nodes with

systemctl stop mysql

So, the cluster is reachable until the last node is stopped. Then, I know which one was the last one and I should be able to restart the nodes without creating a new cluster, so that joining nodes perform an IST.

How do I achieve this?

Answer 1

I am not quite certain if I understand your question completely, but let's consider two scenarios:

After an orderly, manual shutdown of the cluster, you will know which node you shut down last. Just begin the startup sequence with that node, and bring them up manually one after the other.

But suppose your shutdown was accidental, due to a power outage or similar. You will not know which node considers itself "the last to go down". However, you can check the file grastate.dat on the individual cluster members and search for the flag safe_to_bootstrap: 1. If there is exactly one node that has this flag set to 1, boot that one first.

Otherwise, as described in https://galeracluster.com/2016/11/introducing-the-safe-to-bootstrap-feature-in-galera-cluster/, start mysqld with the --wsrep-recover switch, compare the UUIDs in the "Recovered position" line in the output and set the safe_to_bootstrap flag to 1 by hand on the node with the highest UUID. Then, proceed as above.

Those were the procedures for manually starting the nodes. I'm asking about a way to automate this in a more robust way in a different question.