1

I have a four node galera mariadb cluster based on mariadb 10.3 that has been running for about a year.

For unknown reasons a particular node tends to fall out of the cluster and is unable to rejoin automatically about once every 3 months.

Most attempts at recovery fail and I have taken to using the manual method described at Manual SST of Galera Cluster Node With Mariabackup to get it back online.

That method no longer works.

After manual recovery the node gets stuck reporting 'WSREP has not yet prepared node for application use' in response to any SELECT query.

Here is a log of an example recovery after performing the manual steps and then restarting the node:

2019-11-26  3:51:44 7 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-11-26  3:51:46 7 [Note] WSREP: Prepared SST request: mysqldump|10.4.44.86:3360
2019-11-26  3:51:46 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-11-26  3:51:46 7 [Note] WSREP: REPL Protocols: 9 (4, 2)
2019-11-26  3:51:46 7 [Note] WSREP: Assign initial position for certification: 29059601, protocol version: 4
2019-11-26  3:51:46 0 [Note] WSREP: Service thread queue flushed.
2019-11-26  3:51:46 7 [Note] WSREP: IST receiver addr using tcp://172.24.0.34:4568
2019-11-26  3:51:46 7 [Note] WSREP: Prepared IST receiver, listening at: tcp://172.24.0.34:4568
2019-11-26  3:51:46 0 [Note] WSREP: Member 0.0 (galera) requested state transfer from 'galera3'. Selected 3.0 (galera3)(SYNCED) as donor.
2019-11-26  3:51:46 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 29059601)
2019-11-26  3:51:46 7 [Note] WSREP: Requesting state transfer: success, donor: 3
2019-11-26  3:51:46 7 [Note] WSREP: GCache history reset: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:0 -> 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:29059601
2019-11-26  3:51:46 10 [Warning] IP address '10.4.44.82' could not be resolved: Name or service not known
2019-11-26  3:51:47 0 [Note] WSREP: (0f639e64, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-11-26  3:51:49 17 [Note] WSREP: SST received: 57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:29057993
2019-11-26  3:51:49 17 [Note] WSREP: wsrep_start_position set to '57b37aa2-d111-11e8-a015-ab6cf5f3b3ea:29057993'
2019-11-26  3:51:49 7 [Note] WSREP: Receiving IST: 1608 writesets, seqnos 29057993-29059601
2019-11-26  3:51:49 0 [Note] WSREP: 3.0 (galera3): State transfer to 0.0 (galera) complete.
2019-11-26  3:51:49 0 [Note] WSREP: Member 3.0 (galera3) synced with group.
2019-11-26  3:54:00 0 [Warning] WSREP: Protocol violation. JOIN message sender 3.0 (galera3) is not in state transfer (SYNCED). Message ignored.

SELECTs return ''WSREP has not yet prepared node for application use' and the system remains in the state permanently

If I keep an eye on the doner node it goes into the donor state for a while and then returns to the synced state.

The other 3 nodes, including the donor are all working correctly, at least as far as I can tell. The other 3 are willing to run SELECT queries.

Any ideas on how I might resolve this issue?

  • I can't see a bug report for this and there probably should be one. – danblack May 16 at 6:20
3

I met this issue too, and solved it by re-sync that node through SST. I use RSYNC as SST method to make sure the consistence between nodes.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.