How do I add 2 missing nodes to MariaDB Galera Cluster using IST and not SST

Question

I inherited an issue where a MariaDB Galera Cluster has only one out of 3 nodes working.

Node1: up and running in production and is accepting reads and writes
Node2: fresh install it just need to sync with node 1
Node3: fresh install it just need to sync with node 1

If I run service mysql start on Node2/Node3 it changes the state of wsrep_local_state_comment from Synced to Donor/Desynced and a SST is triggered but I will have a lot of downtime.

I have a 2TB of databases and it's going to take about 5-7 hours for a newly added node to sync, and I can't afford the downtime (as an idea, during the day I have 1GB/minute of binlogs and at night about 4GB/hour of binlogs).

I take a db backup every night with Percona XtraBackup and I know how to restore it on a fresh server.

I have galera.cache set at 32GB.

Is there a way of restoring the other 2 nodes without triggering an SST?

I was thinking of restoring the backup of Node1 to Node2 maybe it will trigger an IST.

Answer 1

IST only works if you have a mostly up-to-date node. It does not work for a node with no data; you must use SST in this case.

There are may ways to do SST. Check to see which one might be fastest, or at least least invasive, for your situation.