mariadb galera : member count keeps changing between 5 and 3

Question

I have a 5 nodes cluster, 2 nodes per DC, and an arbiter on a 3rd DC.

The cluster is running fine in the sense that data is synced, DBs don't have any errors. The network is fine too.

However, in the logs, galera shows members leaving and rejoining the cluster every few hours. The same behaviour is observed on, it would be good to unuderstand.

2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
2022-10-10 11:33:39 2 [Note] WSREP: Server status change synced -> connected
2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-10-10 11:33:39 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-10-10 11:33:39 0 [Note] WSREP: view(view_id(NON_PRIM,599bc3d2-b9fe,13720) memb {
        599bc3d2-b9fe,0
} joined {
} left {
} partitioned {
        37d9a4cb-997b,0
        47b6cf8d-9d2a,0
        5de8a52e-aa58,0
        d5464472-acc7,0
})
2022-10-10 11:33:39 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2022-10-10 11:33:39 0 [Note] WSREP: Flow-control interval: [16, 16]
2022-10-10 11:33:39 0 [Note] WSREP: Received NON-PRIMARY.
2022-10-10 11:33:39 2 [Note] WSREP: ================================================
View:
  id: 2e8c4b3d-3d2c-11e9-ba33-9f9156834d83:50217575
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERE
D, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 599bc3d2-444c-11ed-b9fe-93ddbbf65238, p1mariadb1a
=================================================
2022-10-10 11:33:39 2 [Note] WSREP: Non-primary view
2022-10-10 11:33:39 2 [Note] WSREP: Server status change connected -> connected

5 nodes coming together

2022-10-10 11:33:40 0 [Note] WSREP: re-bootstrapping prim from partitioned components
2022-10-10 11:33:40 0 [Note] WSREP: view(view_id(PRIM,37d9a4cb-997b,13721) memb {
        37d9a4cb-997b,0
        47b6cf8d-9d2a,0
        599bc3d2-b9fe,0
        5de8a52e-aa58,0
        d5464472-acc7,0
} joined {
} left {
} partitioned {
})

2022-10-10 11:33:40 0 [Warning] WSREP: Quorum: No node with complete state: ....

cluster ok again:

2022-10-10 11:33:40 0 [Note] WSREP: Full re-merge of primary xx found: 5 of 5.
2022-10-10 11:33:40 0 [Note] WSREP: Quorum results:
        version    = 6,
        component  = PRIMARY,
        conf_id    = 7760,
        members    = 5/5 (joined/total),

sudo zgrep "members =" /var/log/mysql/error.log-202210*

/var/log/mysql/error.log-20221009.gz:   members    = 4/4 (joined/total),
/var/log/mysql/error.log-20221009.gz:   members    = 4/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 5/5 (joined/total),
/var/log/mysql/error.log-20221011.gz:   members    = 3/5 (joined/total),

show status where variable_name in ('wsrep_cluster_size','wsrep_cluster_status','wsrep_flow_control_paused','wsrep_ready','wsrep_connected','wsrep_local_state_comment');

| Variable_name             | Value      |
| wsrep_flow_control_paused | 0.00954013 |
| wsrep_local_state_comment | Synced     |
| wsrep_cluster_size        | 5          |
| wsrep_cluster_status      | Primary    |
| wsrep_connected           | ON         |
| wsrep_ready               | ON         |

Answer 1

I think "gmcast.peer_timeout PT3S" was the key setting, this has calmed things over the last two weeks. It also needs to be set for the Arbiter.

gmcast.segment=1/2/3 is now also set for each site, which probably helps too.

Not convinced that touching evs.suspect_timeout or evs.inactive_timeout is a good idea. And donor is only used for initial state transfer, so apart from avoiding the primary master where traffic lands I suggest not touching