I have three nodes setup and running perfectly for the past months. Recently I want to add another node in a different location but somehow I keep on getting errors.

At first, I was just following this tutorial (where I setup the first time few months ago) https://www.howtoforge.com/tutorial/how-to-install-and-configure-galera-cluster-on-ubuntu-1604/ I did not start all the nodes again from the beginning, I just has to find the file of /mysql/conf.d/galera.cnf in the other three nodes I added the new nodes ip into the previous three. So for the forth one I had the /etc/mysql/conf.d/galera.cnf setup like...

# Galera Provider Configuration
# Galera Cluster Configuration

# Galera Synchronization Configuration

# Galera Node Configuration

somehow I am getting this HUGE error,

        Group state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
         Local state: 00000000-0000-0000-0000-000000000000:-1
  [Note] WSREP: New cluster view: global state: e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273, view# 122: Primary, number of nodes: 4, my
  [Warning] WSREP: Gap in state sequence. Need state transfer.
  [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/'   --parent '22828'  ''
 rsyncd version 3.1.1 starting, listening on port 4444
  [Note] WSREP: Prepared SST request: rsync|xxx.node.4.ip:4444/rsync_sst
  [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
  [Note] WSREP: REPL Protocols: 7 (3, 2)
  [Note] WSREP: Assign initial position for certification: 36273, protocol version: 3
  [Note] WSREP: Service thread queue flushed.
  [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not
          at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
  [Note] WSREP: Member 0.0 (Node4) requested state transfer from '*any*'. Selected 1.0 (Node1)(SYNCED) as donor.
  [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 36273)
  [Note] WSREP: Requesting state transfer: success, donor: 1
  [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> e3ade7e7-e682-11e7-8d16-be7d28cda90e:36273
 [Note] WSREP: (7642cf37, 'tcp://') connection to peer 7642cf37 with addr tcp://xxx.node.4.ip:4567 timed out, no messages
 [Note] WSREP: (7642cf37, 'tcp://') turning message relay requesting off
 mariadb.service: Start operation timed out. Terminating.
 WSREP_SST: [INFO] Joiner cleanup. rsync PID: 22875 
 sent 0 bytes  received 0 bytes  total size 0
 WSREP_SST: [INFO] Joiner cleanup done. 
  [ERROR] WSREP: Process was aborted.
  [ERROR] WSREP: Process completed with error: wsrep_sst_rsync --role 'joiner' --address 'xxx.node.4.ip' --datadir '/var/lib/mysql/'
  [ERROR] WSREP: Failed to read uuid:seqno and wsrep_gtid_domain_id from joiner script.
  [ERROR] WSREP: SST failed: 2 (No such file or directory)
  [ERROR] Aborting
 Error in my_thread_global_end(): 1 threads didn't exit
 mariadb.service: Main process exited, code=exited, status=1/FAILURE
 Failed to start MariaDB 10.1.33 database server.

P.S for the older 3 nodes maria db version is 10.1.29 and the new node is 10.1.33

Thanks in advance for any suggestions.


Sample log from donor

[Note] WSREP: (28d17e95, 'tcp://') turning message relay requesting on, nonlive peers: tcp://xxx.xxx.xxx.xxx:4567 
[Note] WSREP: declaring no1 at tcp://xx.xx.xx.xx:4567 stable
[Note] WSREP: declaring no2 at tcp://xx.xx.xx.xx:4567 stable
[Note] WSREP: forgetting no3 (tcp://xx.xx.xx.xx:4567)
[Note] WSREP: (28d17e95, 'tcp://') turning message relay requesting off
[Note] WSREP: Node no1 state prim
[Note] WSREP: view(view_id(PRIM,no1,49) memb {
} joined {
} left {
} partitioned {
[Note] WSREP: save pc into disk
[Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3
[Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
[Note] WSREP: forgetting no3 (tcp://xxx.xx.xx.xx:4567)
[Note] WSREP: STATE EXCHANGE: sent state msg: uuid-uuid-uuid-uuid
[Note] WSREP: STATE EXCHANGE: got state msg: uuid-uuid-uuid-uuid from 0 (Node3)
[Note] WSREP: STATE EXCHANGE: got state msg: uuid-uuid-uuid-uuid from 1 (Node1)
[Note] WSREP: STATE EXCHANGE: got state msg: uuid-uuid-uuid-uuid from 2 (Node2)
[Note] WSREP: Quorum results:
  • What does the error log on the donor node say (Node1 in this case)? – dbdemon May 27 '18 at 9:24
  • @dbdemon added donor log – Dora May 29 '18 at 22:35
  • Why are there no timestamps in your logs? Hard to verify that we're looking at the right part of the donor logs when we don't have any timestamps. Also, unless these are external IP addresses, I don't think you need to mask them? – dbdemon May 30 '18 at 21:07

Not much useful info in the log extracts provided, I'm afraid.

Anyway, with just the info you've provided, this looks a lot like a question I answered previously where the ownership and permissions of the datadir (/var/lib/mysql in your case) of the new node were not set correctly. The clue is the line [ERROR] WSREP: SST failed: 2 (No such file or directory). So:

  • Make sure you have the directory /var/lib/mysql/
  • Make sure the ownership and permissions are set correctly. You can run sudo chown mysql:mysql /var/lib/mysql to get the correct ownership on the datadir. The permissions should be 755 - sudo chmod 755 /var/lib/mysql.
  • is this config to where the node(4) for or is it for the donor(node1)? – Dora May 30 '18 at 21:12
  • @Dora This is for node(4). – dbdemon May 30 '18 at 21:13
  • thx thx I will give it a try :D – Dora May 30 '18 at 21:15
  • @Dora - any luck? – dbdemon May 31 '18 at 16:52
  • Just got the time to give it a try :( no luck :| – Dora Jun 12 '18 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.