0

I'm at my very first clustering experience. I need to create a MariaDB Cluster. Currenlty I have two servers but I planned to add 1 more in the future.

Now, I have:

maz-pihole (a debian 9 based OS with mariadb 10.4.6): ip address 192.168.250.111 - which I want to be the primary master node

dar-pihole (a debian 10 raspian ARM OS on raspberry pi with mariadb 10.3.15): ip address: 172.16.1.254

These are my galera.cnf settings for the first node, the second one joining is identical with exception for node address and node name.

[mysqld]

default_storage_engine=InnoDB
binlog_format=row
innodb_autoinc_lock_mode=2

# Galera cluster configuration
wsrep_on=ON
wsrep_provider=/usr/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.250.111,172.16.1.254"
wsrep_cluster_name="cluster01"
wsrep_sst_method=rsync

# Cluster node configuration
wsrep_node_address="192.168.250.111"
wsrep_node_name="maz-pihole"

Bootstrapping the first node was pretty easy, it went online immediately after I issued sudo galera_new_cluster.

But I can't manage to make the second one joining the cluster. I'll paste the logs here below from the second one joining. Can you find out something useful? I couldn't ... thanks

pi@dar-pihole:~ $ sudo mysqld -u root -p
2019-07-20 16:51:58 0 [Note] mysqld (mysqld 10.3.15-MariaDB-1) starting as process 1179 ...
2019-07-20 16:51:58 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2019-07-20 16:51:58 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
2019-07-20 16:51:58 0 [Note] WSREP: wsrep_load(): Galera 3.25(rddf9876) by Codership Oy <[email protected]> loaded successfully.
2019-07-20 16:51:58 0 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2019-07-20 16:51:58 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2019-07-20 16:51:58 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = 172.16.1.254; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 2147483647; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.check
2019-07-20 16:51:58 0 [Note] WSREP: GCache history reset: 35d8653e-aa4b-11e9-9465-fb45584e8b94:0 -> 00000000-0000-0000-0000-000000000000:-1
2019-07-20 16:51:58 0 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2019-07-20 16:51:58 0 [Note] WSREP: wsrep_sst_grab()
2019-07-20 16:51:58 0 [Note] WSREP: Start replication
2019-07-20 16:51:58 0 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2019-07-20 16:51:58 0 [Note] WSREP: protonet asio version 0
2019-07-20 16:51:58 0 [Note] WSREP: Using CRC-32C for message checksums.
2019-07-20 16:51:58 0 [Note] WSREP: backend: asio
2019-07-20 16:51:58 0 [Note] WSREP: gcomm thread scheduling priority set to other:0
2019-07-20 16:51:58 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2019-07-20 16:51:58 0 [Note] WSREP: restore pc from disk failed
2019-07-20 16:51:58 0 [Note] WSREP: GMCast version 0
2019-07-20 16:51:58 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
2019-07-20 16:51:58 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
2019-07-20 16:51:58 0 [Note] WSREP: EVS version 0
2019-07-20 16:51:58 0 [Note] WSREP: gcomm: connecting to group 'cluster01', peer '192.168.250.111:,172.16.1.254:'
2019-07-20 16:51:58 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') connection established to eca1a0c7 tcp://172.16.1.254:4567
2019-07-20 16:51:58 0 [Warning] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') address 'tcp://172.16.1.254:4567' points to own listening address, blacklisting
2019-07-20 16:51:58 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') connection established to 8dc5be6e tcp://192.168.250.111:4567
2019-07-20 16:51:58 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers:
2019-07-20 16:51:59 0 [Note] WSREP: declaring 8dc5be6e at tcp://192.168.250.111:4567 stable
2019-07-20 16:51:59 0 [Note] WSREP: Node 8dc5be6e state prim
2019-07-20 16:51:59 0 [Note] WSREP: view(view_id(PRIM,8dc5be6e,6) memb {
        8dc5be6e,0
        eca1a0c7,0
} joined {
} left {
} partitioned {
})
2019-07-20 16:51:59 0 [Note] WSREP: save pc into disk
2019-07-20 16:51:59 0 [Note] WSREP: gcomm: connected
2019-07-20 16:51:59 0 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
2019-07-20 16:51:59 0 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
2019-07-20 16:51:59 0 [Note] WSREP: Opened channel 'cluster01'
2019-07-20 16:51:59 0 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
2019-07-20 16:51:59 0 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
2019-07-20 16:51:59 0 [Note] WSREP: Waiting for SST to complete.
2019-07-20 16:51:59 0 [Note] WSREP: STATE EXCHANGE: sent state msg: ecf86567-aafd-11e9-9d16-2f7a7404d69d
2019-07-20 16:51:59 0 [Note] WSREP: STATE EXCHANGE: got state msg: ecf86567-aafd-11e9-9d16-2f7a7404d69d from 0 (maz-pihole)
2019-07-20 16:51:59 0 [Note] WSREP: STATE EXCHANGE: got state msg: ecf86567-aafd-11e9-9d16-2f7a7404d69d from 1 (dar-pihole)
2019-07-20 16:51:59 0 [Note] WSREP: Quorum results:
        version    = 4,
        component  = PRIMARY,
        conf_id    = 5,
        members    = 1/2 (joined/total),
        act_id     = 13,
        last_appl. = -1,
        protocols  = 0/9/3 (gcs/repl/appl),
        group UUID = 35d8653e-aa4b-11e9-9465-fb45584e8b94
2019-07-20 16:51:59 0 [Note] WSREP: Flow-control interval: [23, 23]
2019-07-20 16:51:59 0 [Note] WSREP: Trying to continue unpaused monitor
2019-07-20 16:51:59 0 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 13)
2019-07-20 16:51:59 2 [Note] WSREP: State transfer required:
        Group state: 35d8653e-aa4b-11e9-9465-fb45584e8b94:13
        Local state: 00000000-0000-0000-0000-000000000000:-1
2019-07-20 16:51:59 2 [Note] WSREP: New cluster view: global state: 35d8653e-aa4b-11e9-9465-fb45584e8b94:13, view# 6: Primary, number of nodes: 2, my index: 1, protocol version 3
2019-07-20 16:51:59 2 [Warning] WSREP: Gap in state sequence. Need state transfer.
2019-07-20 16:51:59 0 [Note] WSREP: Running: 'wsrep_sst_rsync --role 'joiner' --address '172.16.1.254' --datadir '/var/lib/mysql/'   --parent '1179'  ''  '''
2019-07-20 16:51:59 2 [Note] WSREP: Prepared SST request: rsync|172.16.1.254:4444/rsync_sst
2019-07-20 16:51:59 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2019-07-20 16:51:59 2 [Note] WSREP: REPL Protocols: 9 (4, 2)
2019-07-20 16:51:59 2 [Note] WSREP: Assign initial position for certification: 13, protocol version: 4
2019-07-20 16:51:59 0 [Note] WSREP: Service thread queue flushed.
2019-07-20 16:51:59 2 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (35d8653e-aa4b-11e9-9465-fb45584e8b94): 1 (Operation not permitted)
         at galera/src/replicator_str.cpp:prepare_for_IST():482. IST will be unavailable.
2019-07-20 16:51:59 0 [Note] WSREP: Member 1.0 (dar-pihole) requested state transfer from '*any*'. Selected 0.0 (maz-pihole)(SYNCED) as donor.
2019-07-20 16:51:59 0 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 13)
2019-07-20 16:51:59 2 [Note] WSREP: Requesting state transfer: success, donor: 0
2019-07-20 16:51:59 2 [Note] WSREP: GCache history reset: 00000000-0000-0000-0000-000000000000:0 -> 35d8653e-aa4b-11e9-9465-fb45584e8b94:13
2019-07-20 16:52:01 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') connection to peer eca1a0c7 with addr tcp://172.16.1.254:4567 timed out, no messages seen in PT3S
2019-07-20 16:52:02 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') turning message relay requesting off
2019-07-20 16:52:28 0 [Warning] WSREP: 0.0 (maz-pihole): State transfer to 1.0 (dar-pihole) failed: -255 (Unknown error 255)
2019-07-20 16:52:28 0 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():737: Will never receive state. Need to abort.
2019-07-20 16:52:28 0 [Note] WSREP: gcomm: terminating thread
2019-07-20 16:52:28 0 [Note] WSREP: gcomm: joining thread
2019-07-20 16:52:28 0 [Note] WSREP: gcomm: closing backend
2019-07-20 16:52:32 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') connection to peer 8dc5be6e with addr tcp://192.168.250.111:4567 timed out, no messages seen in PT3S
2019-07-20 16:52:32 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.250.111:4567
2019-07-20 16:52:33 0 [Note] WSREP: (eca1a0c7, 'tcp://0.0.0.0:4567') reconnecting to 8dc5be6e (tcp://192.168.250.111:4567), attempt 0
2019-07-20 16:52:34 0 [Note] WSREP: evs::proto(eca1a0c7, LEAVING, view_id(REG,8dc5be6e,6)) suspecting node: 8dc5be6e
2019-07-20 16:52:34 0 [Note] WSREP: evs::proto(eca1a0c7, LEAVING, view_id(REG,8dc5be6e,6)) suspected node without join message, declaring inactive
2019-07-20 16:52:34 0 [Note] WSREP: view(view_id(NON_PRIM,8dc5be6e,6) memb {
        eca1a0c7,0
} joined {
} left {
} partitioned {
        8dc5be6e,0
})
2019-07-20 16:52:34 0 [Note] WSREP: view((empty))
2019-07-20 16:52:34 0 [Note] WSREP: gcomm: closed
2019-07-20 16:52:34 0 [Note] WSREP: mysqld: Terminated.
Aborted
pi@dar-pihole:~ $ WSREP_SST: [ERROR] Parent mysqld process (PID:1179) terminated unexpectedly. (20190720 16:52:34.671)
/usr//bin/wsrep_sst_rsync: line 493: kill: (-1179) - No such process
WSREP_SST: [INFO] Joiner cleanup. rsync PID: 1228 (20190720 16:52:34.679)
WSREP_SST: [INFO] Joiner cleanup done. (20190720 16:52:35.206)
Improve this question
2
  • 1
    The first thing to do is use the same version of MariaDB on the two nodes. If the problem persists, check also for clues in the donor node's error log. Are all necessary ports open? BTW, Galera replication has no master/slave concept - all nodes are master, i.e. it's a multi-master architecture.
    – dbdemon
    Commented Jul 21, 2019 at 21:06
  • another server, with the same version of mariadb has joined the master (I say master the node which boostraps the cluster). So there's no way to make this node, with a different version, to join the cluster?
    – Dark345
    Commented Jul 25, 2019 at 19:57

1 Answer 1

Reset to default
0

I very much doubt it's recommended to run two different versions (10.3 and 10.4) in a cluster - imagine e.g. if you execute a write command on 10.4 that isn't supported on 10.3. However, if you can avoid doing that, then I think in principle it should be possible to make it work. See e.g. the documentation for doing a rolling upgrade from MariaDB Cluster 10.3 to 10.4 - it says that:

Galera 3 and Galera 4 should be compatible for the purposes of a rolling upgrade, as long as you are using Galera 26.4.2 or later".

(MariaDB 10.3 has Galera 3 (wsrep provider versions 25.3.x) and MariaDB 10.4 has Galera 4 (wsrep provider versions 26.4.x).)

But even if you could make your cluster work permanently with a mix of 10.3 and 10.4 nodes, I would recommend against doing it.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.