0

Hello I'm currently having trouble with my galera cluster. It was working fine earlier but suddenly all of the Nodes crashed. I have 3 local nodes and 1 node on the cloud.

During the time it went down the logs shows the following:

2021-09-23  8:24:24 1 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(3):
        0: 077cec27-1b75-11ec-842c-5f218e28b692, Strike
        1: 25c6e10d-1b75-11ec-9ff7-de60adb87197, unspecified
        2: 4e87b761-1b96-11ec-b561-272e3101cb38, Duel
=================================================
2021-09-23  8:24:24 1 [Note] WSREP: Non-primary view
2021-09-23  8:24:24 1 [Note] WSREP: Server status change connected -> connected
2021-09-23  8:24:24 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:24:24 1 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:24:25 0 [Note] WSREP: (077cec27-842c, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.10:4567 timed out, no messages seen in PT3S, socket stats: rtt: 1166000 rttvar: 583000 rto: 3498000 lost: 0 last_data_recv: 1835 cwnd: 10 last_queued_since: 1835197309 last_delivered_since: 1835197309 send_queue_length: 0 send_queue_bytes: 0
           **This happened for ~90 retries**
2021-09-23  8:29:42 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2021-09-23  8:29:42 0 [Note] WSREP: Shutdown replication
2021-09-23  8:29:42 0 [Note] WSREP: Server status change connected -> disconnecting
2021-09-23  8:29:42 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:29:42 0 [Note] WSREP: Closing send monitor...
2021-09-23  8:29:42 0 [Note] WSREP: Closed send monitor.
2021-09-23  8:29:42 0 [Note] WSREP: gcomm: terminating thread
2021-09-23  8:29:42 0 [Note] WSREP: gcomm: joining thread
2021-09-23  8:29:42 0 [Note] WSREP: gcomm: closing backend
2021-09-23  8:29:42 0 [Note] WSREP: view(view_id(NON_PRIM,077cec27-842c,167) memb {
        077cec27-842c,0
} joined {
} left {
} partitioned {
        25c6e10d-9ff7,0
        4e87b761-b561,0
        ed270aba-aedd,0
})
2021-09-23  8:29:42 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2021-09-23  8:29:42 0 [Note] WSREP: view((empty))
2021-09-23  8:29:42 0 [Note] WSREP: Deferred close timer started for socket with remote endpoint: tcp://10.10.10.20:50904
2021-09-23  8:29:42 0 [Note] WSREP: gcomm: closed
2021-09-23  8:29:42 0 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
2021-09-23  8:29:42 0 [Note] WSREP: Flow-control interval: [16, 16]
2021-09-23  8:29:42 0 [Note] WSREP: Received NON-PRIMARY.
2021-09-23  8:29:42 0 [Note] WSREP: New SELF-LEAVE.
2021-09-23  8:29:42 0 [Note] WSREP: Flow-control interval: [0, 0]
2021-09-23  8:29:42 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2021-09-23  8:29:42 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 9842007)
2021-09-23  8:29:42 0 [Note] WSREP: RECV thread exiting 0: Success
2021-09-23  8:29:42 6 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 077cec27-1b75-11ec-842c-5f218e28b692, Strike
=================================================

2021-09-23  8:29:42 6 [Note] WSREP: Non-primary view
2021-09-23  8:29:42 6 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:29:42 0 [Note] WSREP: recv_thread() joined.
2021-09-23  8:29:42 0 [Note] WSREP: Closing replication queue.
2021-09-23  8:29:42 0 [Note] WSREP: Closing slave action queue.
2021-09-23  8:29:42 6 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================

Same happened to my 2nd Node:

2021-09-23  8:31:01 7 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 25c6e10d-1b75-11ec-9ff7-de60adb87197, Aegis
=================================================
2021-09-23  8:31:01 7 [Note] WSREP: Non-primary view
2021-09-23  8:31:01 7 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:31:01 0 [Note] WSREP: recv_thread() joined.
2021-09-23  8:31:01 0 [Note] WSREP: Closing replication queue.
2021-09-23  8:31:01 0 [Note] WSREP: Closing slave action queue.
2021-09-23  8:31:01 7 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================

And third Node:

2021-09-23  8:30:59 2 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(1):
        0: 4e87b761-1b96-11ec-b561-272e3101cb38, Duel
=================================================
2021-09-23  8:30:59 2 [Note] WSREP: Non-primary view
2021-09-23  8:30:59 2 [Note] WSREP: Server status change connected -> connected
2021-09-23  8:30:59 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:30:59 2 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:31:03 0 [Note] WSREP: (4e87b761-b561, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.10:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 2000000 lost: 1 last_data_recv: 385031380 cwnd: 1 last_queued_since: 385331379302597 last_delivered_since: 385331379302597 send_queue_length: 0 send_queue_bytes: 0

2021-09-23  8:31:04 0 [Note] WSREP: (4e87b761-b561, 'tcp://0.0.0.0:4567') reconnecting to ed270aba-aedd (tcp://10.10.10.10:4567), attempt 90
2021-09-23  8:31:05 0 [Note] WSREP:  cleaning up 25c6e10d-9ff7 (tcp://10.10.10.20:4567)
2021-09-23  8:31:08 0 [Note] WSREP: (4e87b761-b561, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.41:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 4000000 lost: 1 last_data_recv: 385036380 cwnd: 1 last_queued_since: 385336379962010 last_delivered_since: 385336379962010 send_queue_length: 0 send_queue_bytes: 0


2021-09-23  8:31:49 2 [Note] WSREP: ================================================
    View:
      id: d326832d-56e2-11eb-80c1-760504343273:9842007
      status: non-primary
      protocol_version: 4
      capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
      final: yes
      own_index: -1
      members(0):
    =================================================

As for my cloud node, this happened:

2021-09-23  8:29:07 0 [Note] WSREP: (ed270aba-aedd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.30:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 4000000 lost: 1 last_data_recv: 684592357 cwnd: 1 last_queued_since: 684892356898817 last_delivered_since: 684892356898817 send_queue_length: 0 send_queue_bytes: 0
2021-09-23  8:29:07 0 [Note] WSREP: (ed270aba-aedd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.40:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 4000000 lost: 1 last_data_recv: 684592358 cwnd: 1 last_queued_since: 684892357027577 last_delivered_since: 684892357027577 send_queue_length: 0 send_queue_bytes: 0
2021-09-23  8:29:09 0 [Note] WSREP: (ed270aba-aedd, 'tcp://0.0.0.0:4567') connection to peer 00000000-0000 with addr tcp://10.10.10.20:4567 timed out, no messages seen in PT3S, socket stats: rtt: 0 rttvar: 250000 rto: 4000000 lost: 1 last_data_recv: 684593857 cwnd: 1 last_queued_since: 684893856941066 last_delivered_since: 684893856941066 send_queue_length: 0 send_queue_bytes: 0
2021-09-23  8:32:28 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
2021-09-23  8:32:28 0 [Note] WSREP: Shutdown replication
2021-09-23  8:32:28 0 [Note] WSREP: Server status change connected -> disconnecting
2021-09-23  8:32:28 0 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2021-09-23  8:32:28 0 [Note] WSREP: Closing send monitor...
2021-09-23  8:32:28 0 [Note] WSREP: Closed send monitor.
2021-09-23  8:32:28 0 [Note] WSREP: gcomm: terminating thread
2021-09-23  8:32:28 0 [Note] WSREP: gcomm: joining thread
2021-09-23  8:32:28 0 [Note] WSREP: gcomm: closing backend
2021-09-23  8:32:28 0 [Note] WSREP: PC protocol downgrade 1 -> 0
2021-09-23  8:32:28 0 [Note] WSREP: view((empty))
2021-09-23  8:32:28 0 [Note] WSREP: gcomm: closed
2021-09-23  8:32:28 0 [Note] WSREP: New SELF-LEAVE.
2021-09-23  8:32:28 0 [Note] WSREP: Flow-control interval: [0, 0]
2021-09-23  8:32:28 0 [Note] WSREP: Received SELF-LEAVE. Closing connection.
2021-09-23  8:32:28 0 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 9842007)
2021-09-23  8:32:28 0 [Note] WSREP: RECV thread exiting 0: Success
2021-09-23  8:32:28 0 [Note] WSREP: recv_thread() joined.
2021-09-23  8:32:28 0 [Note] WSREP: Closing replication queue.
2021-09-23  8:32:28 0 [Note] WSREP: Closing slave action queue.
2021-09-23  8:32:28 2 [Note] WSREP: ================================================
View:
  id: d326832d-56e2-11eb-80c1-760504343273:9842007
  status: non-primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: yes
  own_index: -1
  members(0):
=================================================

I have the following configurations:

Node 1:

[galera] # Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera-4/libgalera_smm.so

#add your node ips here
wsrep_cluster_address="gcomm://strike,aegis,duel,clone"
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
#Cluster name
wsrep_cluster_name="cloud_test_cluster"
# Allow server to accept connections on all interfaces.

bind-address=0.0.0.0

# this server ip, change for each server
wsrep_node_address="strike"
# this server name, change for each server
wsrep_node_name="Strike"

wsrep_sst_method=rsync

wsrep_sst_donor="Aegis,Duel"

Node 2 is same as above but:

wsrep_sst_donor="Strike,Duel"

Node 3:

wsrep_sst_donor="Strike,Aegis"

Finally cloud:

wsrep_sst_donor="Duel,Aegis,Strike"

All of this happened almost at the same time. Did they just lose connection to each other? Did losing connection to 10.10.10.10 caused the crash? Why did the members count decrease from each other? This happened yesterday and today. I had this set up since last weekend but there were no problems during that time til yesterday...

Can someone explain to me what happened?

Improve this question

1 Answer 1

Reset to default
2

Your Strike node:

2021-09-23  8:29:42 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown

So it looks something it just sending a SIGTERM to shut the service down at this time.

The other nodes logs are after Strike shutdown.

Note all nodes have non-primary. If you are starting after a power failure, see the Restarting the Cluster documentation, you need to manually specify one of the nodes as the new primary.

This involves determining the latest node (singular), and running galera_recovery on that node only. Then start the other nodes.

Maybe the termination of Strike was a result of not getting to a primary status quick enough.

Improve this answer
1
  • Hello sir @danblack, Upon checking on nodes 2 and 3, I had the same message Node2: 2021-09-23 8:31:00 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown Node3 2021-09-23 8:31:49 0 [Note] /usr/sbin/mariadbd (initiated by: unknown): Normal shutdown
    – Glen G
    Commented Sep 24, 2021 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.