1

Galera clusters, somewhat famously, have a requirement that nodes need to be shut down and brought back up again in a certain order.

There are a few questions here and on other sites about recovering from unclean shutdowns, which may cause split brain situations in the cluster. For manual restarts, I usually follow a procedure that relies on checking files and potentially even database state and is, frankly, too fragile and too much manual work for my taste.

I think it should be possible to encode this manual procedure right into the systemd unit files of a cluster, so that when the failure condition is resolved and machines boot back up, the cluster members also come back up in the correct sequence automatically. This will definitely involve listing all cluster member nodes in some central location, but should be re-usable apart from that.

So, is there a commonly used, best-practice way to re-start a Galera cluster after any sort of shutdown, automatically during system startup?

Improve this question

1 Answer 1

Reset to default
1

I would say the best practice would be to ensure that one node is always still running, when the others have stopped normally. (systemctl stop mysql/mariadb) Then restart one of the previously stopped nodes, and when that restarted node reports in logfiles that it connected to the running old node, and that it's ready to serve connections, you may restart the old one still running.

If all nodes have been stopped, you need to figure out manually which is the newest node (by using (ubuntu):galera_recovery, note the highest change no.) Maybe even have edit the safe_to_bootstrap line in the grastate.dat file.

That node then needs to restart the cluster (create the new unique ID for the others to connect to) (by using (ubuntu): galera_new_cluster).

The remaining nodes simply start db with 'systemctl start mariadb'.

Sounds as if there's a reason they didnt yet built that in to systemd ;) ?

Improve this answer
2
  • 1
    Thanks for the reply and welcome to DBA! However, it essentially just reiterates my manual recovery procedure linked above -- I was hoping for a more robust, more generally applicable and less manual procedure, that was the core of my question.
    – jstarek
    Commented Feb 24, 2021 at 13:31
  • Hi Yes, I thought it was more or less what everybody have to do when handling Galera. But I got to think it over again while writing it :) I hope you stumble over some better/more automated tools. And that you'll remember sharing them here if you get lucky ;) I'm only deaing with two small clusters, so I'll live - but I feel the cumbersome pain you'd get if you're responsible for a bunch of clusters or just many nodes. And if you don't pay top$ for tools like ClusterControl.
    – user266120
    Commented Feb 25, 2021 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.