0

We have a 5-node Galera cluster in a high-load application. One of the nodes freezes in production. Still the cluster-size=5 and Galera/WSREP writes to this node, but the node is not working. Then we have to kill the mysql-processes with kill -9 and empty te var/lib/mysql-dir and start mysql. One of the other nodes becomes a donor. But finally if al the data is transferred the node is not working. This is the last bit of the errorlog. Mysql cannot connect. Why? How can I debug? Nothing else has been changed on this server, the cluster is running for 2 years now. Anybody?

(...)
 protocols  = 0/8/3 (gcs/repl/appl),
        group UUID = eb3b8cba-8d84-11e9-aa79-a3e756a46963

2020-01-22 12:48:47 139999022409472 [Note] WSREP: Flow-control interval: [566, 572]
2020-01-22 12:48:47 139999022409472 [Note] WSREP: Trying to continue unpaused monitor
2020-01-22 12:48:47 139999022409472 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 138229140)
2020-01-22 12:48:47 139999492396800 [Note] WSREP: State transfer required:
        Group state: eb3b8cba-8d84-11e9-aa79-a3e756a46963:138229140
        Local state: eb3b8cba-8d84-11e9-aa79-a3e756a46963:129279566

2020-01-22 12:48:47 139999492396800 [Note] WSREP: New cluster view: global state: eb3b8cba-8d84-11e9-aa79-a3e756a46963:138229140, view# 95: Primary, number of nodes: 5, my index: 1, protocol version 3
2020-01-22 12:48:47 139999492396800 [Warning] WSREP: Gap in state sequence. Need state transfer.
2020-01-22 12:48:47 139999156627200 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '***.***.***.***' --datadir '/var/lib/mysql/'   --parent '7017'  '' '
2020-01-22 12:48:47 139999492396800 [Note] WSREP: Prepared SST request: rsync|***.***.***.***:4444/rsync_sst
ERROR 2002 (HY000): Can't connect to MySQL server on '127.0.0.1' (115)
2020-01-22 12:48:47 139999492396800 [ERROR] WSREP: Process completed with error: /usr/local/bin/wsrep_notify.sh --status Joiner --uuid eb3b8cba-8d84-11e9-aa79-a3e756a46963 --primary yes --index 1 --members 03414083-39c0-11ea-8d9e-5e3$

2020-01-22 12:48:47 139999492396800 [ERROR] WSREP: Notification command failed: 1 (Operation not permitted): "/usr/local/bin/wsrep_notify.sh --status Joiner --uuid eb3b8cba-8d84-11e9-aa79-a3e756a46963 --primary yes --index 1 --member$

2020-01-22 12:48:47 139999492396800 [Note] WSREP: REPL Protocols: 8 (3, 2)
2020-01-22 12:48:47 139999492396800 [Note] WSREP: Assign initial position for certification: 138229140, protocol version: 3
2020-01-22 12:48:47 139999408465664 [Note] WSREP: Service thread queue flushed.
2020-01-22 12:48:47 139999492396800 [Note] WSREP: IST receiver addr using tcp://***.***.***.***:4568
2020-01-22 12:48:47 139999492396800 [Note] WSREP: Prepared IST receiver, listening at: tcp://***.***.***.***:4568
2020-01-22 12:48:47 139999022409472 [Note] WSREP: Member 1.0 (server.************.local) requested state transfer from '*any*'. Selected 0.0 (server.********.local)(SYNCED) as donor.
2020-01-22 12:48:47 1399990224094 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 138229153)
2020-01-22 12:48:47 139999492396800 [Note] WSREP: Requesting state transfer: success, donor: 0
2020-01-22 12:48:47 139999492396800 [Note] WSREP: GCache history reset: eb3b8cba-8d84-11e9-aa79-a3e756a46963:0 -> eb3b8cba-8d84-11e9-aa79-a3e756a46963:138229140
2020-01-22 12:48:49 139999372793600 [Note] WSREP: (255212c2, 'tcp://0.0.0.0:4567') connection to peer 255212c2 with addr tcp://***.***.***.***:4567 timed out, no messages seen in PT3S
2020-01-22 12:48:49 139999372793600 [Note] WSREP: (255212c2, 'tcp://0.0.0.0:4567') turning message relay requesting off
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 7087 (20200122 12:53:15.808)
WSREP_SST: [INFO] Joiner cleanup done. (20200122 12:53:16.314)

2020-01-22 12:53:16 139999156627200 [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address '***.***.***.***' --datadir '/var/lib/mysql/'   --parent '7017'  '' : 3 (No such process)

2020-01-22 12:53:16 139999156627200 [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
2020-01-22 12:53:16 139999494010848 [ERROR] WSREP: SST failed: 3 (No such process)
2020-01-22 12:53:16 139999494010848 [ERROR] Aborting
2020-01-22 12:53:16 139999022409472 [Warning] WSREP: 0.0 (server.***********.local): State transfer to 1.0 (server.***********.local) failed: -255 (Unknown error 255)
2020-01-22 12:53:16 139999022409472 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
New contributor
user1030158 is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • 1. Do you have a file /usr/local/bin/wsrep_notify.sh? 2. Are SELinux or AppArmour enabled? (The command sestatus will tell you the status of SELinux.) If so, are they playing nice? 3. Are all the necessary ports open? 4. Are you running into some sort of timeout issue? I believe that can be the case with some versions of systemd. 5. Can you verify that the SST is triggering any kind of process on the donor node at all? – dbdemon Feb 14 at 16:36

Your Answer

user1030158 is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.