MariaDB Galera Cluster - first node won't reconnect after restart... Ever

Question

Firewalls disabled, replication worked fine, IP's accessible..

All config copied from here: https://www.digitalocean.com/community/tutorials/how-to-configure-a-galera-cluster-with-mysql-5-6-on-ubuntu-16-04

As soon I try to restart the first node it won't reconnect.

I cannot restart the whole cluster as it supposed to be serving content 24/7 with at least 3 nodes available at all times...

Any idea how I can setup Galera so it would survive a restart of ANY node? (restart of a cluster is a pretty much fatal crash as leads to losses going in xx k of $ and the main reason behind selecting Galera was the low cost vs Microsoft (which can survive without a problem any number of resets while setup correctly)...

As I am trying to push Galera instead of Windows, I have to prove that indeed it is a commercially viable solution (which means that it have to survive main node restart, especially that we have dozens of "main" servers so "first" node is strictly theoretical).

I was testing in ~ 2 weeks ago and unfortunately, it ended up with a restart and data loss (due to having test servers offline)

Answer 1

With not so much detail in the question, I can't give a detailed answer, but I'd recommend starting with MariaDB's documentation about Galera Cluster since you're using MariaDB. See e.g. Getting Started with MariaDB Galera Cluster. The DigitalOcean documentation you referred to is for MySQL 5.6, so there may be details in there that works slightly differently on MariaDB.

Maybe the most crucial question about your specific problem is: What error message are you getting? And I mean the actual error message in the MariaDB error log, which is not necessarily what systemctl will show you in the shell when you run the start/restart/status commands. Once you have this, try your favourite search engine to find the answer. If that doesn't work, come back to DBA.SE and ask a more specific question which includes the error message(s).

Galera Cluster is an excellent technology, but if you don't have much experience with it yet, make sure to read carefully about its limitations, and make sure they're not in conflict with your particular use-case. There is a lot to learn, so be prepared to invest sufficient time and effort.