0

I have 3 Node MariaDB Galera cluster. I am using "mariabackup" as the wsrep_sst_method, but the database system doesn't work properly. I can say that sometimes there are 100 connection requests per second from clients, so it is a bit bussy system but sometimes the db system doesnt serves for 2-6 seconds. Connections are lost or the db is unreachable, so when I check in that times the wsrep status is not synched. What should I do to build a 3 Node MariaDB system that can work properly for a bussy connection traffic? There must be something?

More Info: All nodes are in the same server as virtual machines and and I am using IP virtualization named Bonding in each virtual machine. It means each machine has two ethernet connections and the Bonding creates an virtual IP to represent the two IP addresses as just one IP address. So each machine has two ethernet connection bot we only use one virtual IP which represents them. Example Ethernet1 + Ethernet2 = bond0 I use Keepalived between the machine1 and machine2, so it also make it possible to use a virtual IP for machines. When the machine1 is available it routes the connections to this, otherwise to the machine2. So Machine1_bond0 + Machine2_bond0 = Keepalived_virtualIP I checked logs during problem in the connections but I could not see any log about keepalived or bond0. All resource usages are under %50 in bussy connection traffic, so the resources are all enough (200gb disc, 8gb ram for each machine, they only servers as db) All nodes-machines are writable-readable. All of them are Masters. Slowlogs are not turned on during the production. I think it has problem only during the backup operations.

2
  • 1
    Some more info, please. Ping distance between nodes. Typical CPU usage or Load Average. Are all 3 nodes writable? Is the slowlog turned on? (I may ask for its contents next.) Does it have trouble only during a backup?
    – Rick James
    Oct 8 at 23:14
  • @RickJames I added some info to the question that you asked but I dont understand which contents you means? Oct 11 at 5:34
1

"are in the same server as virtual machines" -- This means that they are competing for CPU and I/O, especially during the backup. Is this just a test? I would not deploy Galera with all nodes on a single machine.

When SST is involved, 2 of the 3 nodes are involved. The donor is busy fetching and sending data; the recipient cannot receive connections until the SST is finished.

Perhaps the simplest way to keep more capacity when doing an SST is to have a 4th node. Then 2 of the 4 nodes are free to handle connections.

Galera nodes need to be on different machines, even different geographical locations.

Do both (4th node; separate machines) if you can.

The slowlog is only a small burden. I recommend keeping it on, even on production systems. As a compromise, you could have it on in only one node.

3
  • It is sensible to have in seperated machines, but there are no load balancing for the 4th machine, so it will not be useful. Should I build load balancing? Oct 12 at 8:16
  • What are you using for "load balancing"? Typically, it can handle any number of nodes: 3, 4, whatever. Yes, add the 4th to the load balancer.
    – Rick James
    Oct 12 at 16:15
  • I am not using load balancing, I am only routing the connections to the db1 if it is available, otherwise to 2 with keepalived. Oct 13 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.