0

I have a 5 nodes cluster, 2 nodes per DC, and an arbiter on a 3rd DC.

The cluster is running fine in the sense that data is synced, DBs don't have any errors. The network is fine too.

However, in the logs, galera shows members leaving and rejoining the cluster every few hours. The same behaviour is observed on, it would be good to unuderstand.

2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
2022-10-10 11:33:39 2 [Note] WSREP: Server status change synced -> connected
2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-10-10 11:33:39 0 [Note] WSREP: view(view_id(NON_PRIM,599bc3d2-b9fe,13720) memb {
        599bc3d2-b9fe,0
} joined {
} left {
} partitioned {
        37d9a4cb-997b,0
        47b6cf8d-9d2a,0
        5de8a52e-aa58,0
        d5464472-acc7,0
})
2022-10-10 11:33:39 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2022-10-10 11:33:39 0 [Note] WSREP: Flow-control interval: [16, 16]
2022-10-10 11:33:39 0 [Note] WSREP: Received NON-PRIMARY.
2022-10-10 11:33:39 2 [Note] WSREP: ================================================
View:
  id: 2e8c4b3d-3d2c-11e9-ba33-9f9156834d83:50217575
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERE
D, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 599bc3d2-444c-11ed-b9fe-93ddbbf65238, p1mariadb1a
=================================================
2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
2022-10-10 11:33:39 2 [Note] WSREP: Server status change connected -> connected

5 nodes coming together

2022-10-10 11:33:40 0 [Note] WSREP: re-bootstrapping prim from partitioned components
2022-10-10 11:33:40 0 [Note] WSREP: view(view_id(PRIM,37d9a4cb-997b,13721) memb {
        37d9a4cb-997b,0
        47b6cf8d-9d2a,0
        599bc3d2-b9fe,0
        5de8a52e-aa58,0
        d5464472-acc7,0
} joined {
} left {
} partitioned {
})

2022-10-10 11:33:40 0 [Warning] WSREP: Quorum: No node with complete state: ....

cluster ok again:

2022-10-10 11:33:40 0 [Note] WSREP: Full re-merge of primary xx found: 5 of 5.
2022-10-10 11:33:40 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 7760,
        members    = 5/5 (joined/total),

sudo zgrep "members =" /var/log/mysql/error.log-202210*

/var/log/mysql/error.log-20221009.gz:   members    = 4/4 (joined/total),
/var/log/mysql/error.log-20221009.gz:   members    = 4/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 3/5 (joined/total),

show status where variable_name in ('wsrep_cluster_size','wsrep_cluster_status','wsrep_flow_control_paused','wsrep_ready','wsrep_connected','wsrep_local_state_comment');

| Variable_name             | Value      |
| wsrep_flow_control_paused | 0.00954013 |
| wsrep_local_state_comment | Synced     |
| wsrep_cluster_size        | 5          |
| wsrep_cluster_status      | Primary    |
| wsrep_connected           | ON         |
| wsrep_ready               | ON         |
4
  • Typically the kind of error hard to analyse without putting the hands on the context. I.E : Any jobs scheduled that might cause this effect
    – Hybris95
    Commented Oct 11, 2022 at 6:57
  • One possible explanation for this given this scenario with Galera nodes in multiple DCs is issues with the network and one of the timeout values in wsrep_provide_options being too small. See e.g. galeracluster.com/library/documentation/auto-eviction.html
    – dbdemon
    Commented Oct 11, 2022 at 9:29
  • Thanks for the tips, I don't think it is scheduled jobs, even happens on a quiet cluster. Now have tuned the following, which has quietened logs. Param, default, value gmcast.peer_timeout PT3S PT9S inactive_check_period PT1S PT10S evs.suspect_timeout PT5S PT30S evs.inactive_timeout PT15S PT60S Also setting wsrep_sst_donor, gmcast.segment,pc.weight too.
    – boran
    Commented Oct 11, 2022 at 13:57
  • Please show the config for each node.
    – Rick James
    Commented Oct 11, 2022 at 16:13

1 Answer 1

0

I think "gmcast.peer_timeout PT3S" was the key setting, this has calmed things over the last two weeks. It also needs to be set for the Arbiter.

gmcast.segment=1/2/3 is now also set for each site, which probably helps too.

Not convinced that touching evs.suspect_timeout or evs.inactive_timeout is a good idea. And donor is only used for initial state transfer, so apart from avoiding the primary master where traffic lands I suggest not touching

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.