How to restart MariaDB Galera cluster to change my.cnf

Question

I have 3 dbs servers with mariadb galera cluster.

I need few my.cnf change but each time I try to just restart with mysql restart command , replication gets affected.

So can anybody let me know proper way to restart mysql without downtime.

[mysqld]
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
bind-address=0.0.0.0

# Galera Provider Configuration
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

# Galera Cluster Configuration
wsrep_cluster_name="web_cluster"
wsrep_cluster_address="gcomm://172.16.1.173,172.17.1.173,172.18.1.173"

# Galera Synchronization Configuration
wsrep_sst_method=rsync
#wsrep_sst_auth=cluster-user:clusterpass
#wsrep_sst_method=mariabackup


# Galera Node Configuration
wsrep_node_address="172.16.1.173"
wsrep_node_name="web-db-01"

Answer 1

1. rsync is usually not a good SST method as it can block writes on the donor node while performing SST. Use mariabackup instead to avoid that.
2. Use wsrep_provider_options="gcache.recover=yes; gcache.size = xM;" where x is enough megabytes to hold all write sets for any expected downtime, if you want to be able to use IST and avoid SST after restarts. E.g. the amount of downtime expected for upgrading MariaDB on a single node. 128 is what I use, but it obviously depends on how much writing is usually happening on your cluster. gcache.recover=yes; is needed to try and use gcache-based IST when a node starts up instead of SST. See the MariaDB KB article on wsrep_provider_options for details.