0

I'm administrating a galera cluster with 3 nodes, serving a Laravel application and HAProxy is the gateway to the cluster. We had an interesting issue, and I'm unable to figure out why was there an error message in the Laravel application. Here is what I know: At 13:25:36 there was a note on Node1, that ... connection to peer f3e3XXXX-XXX with addr tcp://X.X.X.X:4567 timed out, no messages seen in PT3S, ... This IP represents Node2. Roughly the same time 13:25:37, on Node2, the following message appeared in the logs ... WSREP: (f3e3XXXX-XXX, 'tcp://X.X.X.X:4567') turning message relay requesting on, nonlive peers: tcp://X.X.X.X:4567 ... This 2nd IP represent Node1. I'll post a more detailed log later in the post.

So my understanding is there could have been a network hickup between the clusters and Node2 (and 3) had to rejoin the cluster. Meanwhile, someone triggered the Larave application that made a query towards the cluster. But this request ended up in an error message saying: [2023-11-07 13:25:46] production.ERROR: SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use .....

If my understanding how a cluster should work is correct, I have no clue what happened. Isn't the galera cluster supposed to be able to manage if a node disappears from the cluster and use an existing node to handle the incoming query? Was this a HAProxy issue maybe and it could not detect that Node2 and 3 are unavailable and sent the query there eitherway?

This issue stayed for about 10 seconds and the cluster is working just fine ever since (I think... wsrep_cluster_size is 3 ATM so it must be OK).

Now I don't want to "spam" this post with all the logs, so I'll insert the logs from Node1 and 2 into pastebin and URLs are provided.

!!! Can anyone explain me what happened here in this situation exactly? Maybe a way to avoid this in the future? !!!

Thank you in advance !

MariaDB logs: https://pastebin.com/SpDbe7Bb

Laravel logs: [2023-11-07 13:25:46] production.ERROR: SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use (SQL: select * from .......) {"exception":"[object] (Illuminate\\Database\\QueryException(code: 08S01): SQLSTATE[08S01]: Communication link failure: 1047 WSREP has not yet prepared node for application use (SQL: select * from .......) at /var/www/html/myproject/releases/184/vendor/laravel/framework/src/Illuminate/Database/Connection.php:671)

1 Answer 1

1

If there are only 3 nodes and 2 of the nodes (2 & 3) were in an ERROR state then the remaining node will not accept any transaction since it does not know the current state of the cluster.

See https://galeracluster.com/library/documentation/crash-recovery.html "Two Nodes Disappear from the Cluster"

4
  • and according to the logs, my Node2 and 3 were gone for a few seconds, right? That is why the query was not executed. Nov 8, 2023 at 13:45
  • 1
    Correct, this is where a retry mechanism for short network glitches might be beneficial. Nov 8, 2023 at 22:24
  • Can you pont me to the right direction for the "retry mechanism"? Is that application specific code, or a MariaDB setting? Nov 13, 2023 at 9:23
  • 1
    it's application specific code. basically a try-catch method Nov 17, 2023 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.