MariaDB Galera 3-Node Cluster Sometimes Doesn't Serves

Question

I have 3 Node MariaDB Galera cluster. I am using "mariabackup" as the wsrep_sst_method, but the database system doesn't work properly. I can say that sometimes there are 100 connection requests per second from clients, so it is a bit bussy system but sometimes the db system doesnt serves for 2-6 seconds. Connections are lost or the db is unreachable, so when I check in that times the wsrep status is not synched. What should I do to build a 3 Node MariaDB system that can work properly for a bussy connection traffic? There must be something?

More Info: All nodes are in the same server as virtual machines and and I am using IP virtualization named Bonding in each virtual machine. It means each machine has two ethernet connections and the Bonding creates an virtual IP to represent the two IP addresses as just one IP address. So each machine has two ethernet connection bot we only use one virtual IP which represents them. Example Ethernet1 + Ethernet2 = bond0 I use Keepalived between the machine1 and machine2, so it also make it possible to use a virtual IP for machines. When the machine1 is available it routes the connections to this, otherwise to the machine2. So Machine1_bond0 + Machine2_bond0 = Keepalived_virtualIP I checked logs during problem in the connections but I could not see any log about keepalived or bond0. All resource usages are under %50 in bussy connection traffic, so the resources are all enough (200gb disc, 8gb ram for each machine, they only servers as db) All nodes-machines are writable-readable. All of them are Masters. Slowlogs are not turned on during the production. I think it has problem only during the backup operations.

Answer 1

"are in the same server as virtual machines" -- This means that they are competing for CPU and I/O, especially during the backup. Is this just a test? I would not deploy Galera with all nodes on a single machine.

When SST is involved, 2 of the 3 nodes are involved. The donor is busy fetching and sending data; the recipient cannot receive connections until the SST is finished.

Perhaps the simplest way to keep more capacity when doing an SST is to have a 4th node. Then 2 of the 4 nodes are free to handle connections.

Galera nodes need to be on different machines, even different geographical locations.

Do both (4th node; separate machines) if you can.

The slowlog is only a small burden. I recommend keeping it on, even on production systems. As a compromise, you could have it on in only one node.