When B becomes inaccessible from A and C,
A and C talk to each other and realize that they comprise more than 50% of the cluster, so they continue to accept writes. (I do not know whether the pending write you described will give up before the timeout.) Your code must check for errors after every COMMIT
and restart the transaction just in case. (2/3 is a quorum.)
B can't see anyone else, so it stops accepting writes and aborts (at COMMIT
) any transactions that have started on B. Again, the writers to B must catch this error and replay the transaction. (1/3 is not a quorum.)
After the network is fixed, the 3 nodes discuss the situation. Since B has been out of the cluster for some time, one of two things happens:
If the outage was brief enough, either A or C sends the missing updates to B. This is called IST.
If the outage has been "too long", either A or C sends the entire dataset to B. This is called SST.
More discussion: https://galeracluster.com/library/documentation/state-transfer.html
Note that COMMIT
is when each node must agree, disagree, or be found missing.
I don't have the details on the load balancer, but I assume that it eventually figures out that B is in trouble and stops sending writes to it. Similarly, when it is synced back up, it will get writes again.
Yes, there will be a few seconds of hiccup throughout the system while it figures all this out.
Yes, during the network outage, A & C will serve all traffic. Yes, there will be some degradation (2 servers instead of 3). Yes there will be more degradation while patching B (since A or C is spending some effort to send updates to B).
It is not "perfect", but it is possibly the best available. I hope that A,B,C are geographically spread apart (think: wildfire, flood, earthquake, etc) so that no 2 are likely to be down at the same time. Of course, geographical spreading leads to added latency, which increases COMMIT
time.