0

Suppose this mode, I have 3 node with Galera sql, each node is a master. I call them A,B,C. If I have a transaction like update, what will happens if I execute the transaction at node A, but the node B is suffering a net partition from A and C just now, and A, C is not aware that B is failed since the detection mechanism has not exceeded 5 secs(for example). Is that transaction will fail since not get all node agree ? Or will succ since get a quorum?

Another question: If I have a load-balance for 3 nodes, what will happens if I execute read at node B which suffering net partition from A and C just now. Is this read will failed? I have this question since there is a delay for recognizing node B's failure state.

Are all transactions will fail since not get all current nodes' agree in the failure detection period? After thie period, some node will be removed from cluster, at this time, the cluster will service ok?

2 Answers 2

0

When B becomes inaccessible from A and C,

  • A and C talk to each other and realize that they comprise more than 50% of the cluster, so they continue to accept writes. (I do not know whether the pending write you described will give up before the timeout.) Your code must check for errors after every COMMIT and restart the transaction just in case. (2/3 is a quorum.)

  • B can't see anyone else, so it stops accepting writes and aborts (at COMMIT) any transactions that have started on B. Again, the writers to B must catch this error and replay the transaction. (1/3 is not a quorum.)

  • After the network is fixed, the 3 nodes discuss the situation. Since B has been out of the cluster for some time, one of two things happens:

  • If the outage was brief enough, either A or C sends the missing updates to B. This is called IST.

  • If the outage has been "too long", either A or C sends the entire dataset to B. This is called SST.

More discussion: https://galeracluster.com/library/documentation/state-transfer.html

Note that COMMIT is when each node must agree, disagree, or be found missing.

I don't have the details on the load balancer, but I assume that it eventually figures out that B is in trouble and stops sending writes to it. Similarly, when it is synced back up, it will get writes again.

Yes, there will be a few seconds of hiccup throughout the system while it figures all this out.

Yes, during the network outage, A & C will serve all traffic. Yes, there will be some degradation (2 servers instead of 3). Yes there will be more degradation while patching B (since A or C is spending some effort to send updates to B).

It is not "perfect", but it is possibly the best available. I hope that A,B,C are geographically spread apart (think: wildfire, flood, earthquake, etc) so that no 2 are likely to be down at the same time. Of course, geographical spreading leads to added latency, which increases COMMIT time.

2
  • 1
    Further, your load balancer needs to gain awareness of the galera cluster state, with wsrep_notify_cmd triggering a loss of a backend node, or a backend node check on wsrep_local_state==4.
    – danblack
    Commented Aug 14, 2021 at 21:59
  • hah! Thanks a lot. Now I have a new question since you said the Galera use quorum to detemine whether a write will succ or not. What will happens if a write to A has succeed, but B is suffering a net partition from A and C. The result of write will not be seen from B since B may be think it's ok but in fact itself has been removed from cluster. There always exists a delay for consistency. In this situation, is the read from B is a stale read? Commented Aug 16, 2021 at 2:25
0

It's deponds on the time out setting. In Galera, a single node network interruption makes the others wait for it, and pending writes can be committed once either the connection is restored or the faulty node removed from cluster after the timeout

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.