0

I have a 3 node AlwaysOn AG

Node 1 - Synchronous

Node 2 - Synchronous - Primary

Node 3 - Asynchronous

Quorum Configuration:

Node 1 - Vote

Node 2 - Vote

Node 3 - NO VOTE

File Share Witness - VOTE

Failover Mode: Manual

Upon restarting Node 1, the synchronous replica - The AG restarted.

What could be the reason why this happened? As far my understanding goes, this is not supposed to happen.

Some interesting log entries I found on Node 2:

Failed to update Replica status due to exception 41034.

The local availability replica of availability group 'MyAG' is in a failed state.  The replica failed to read or update the persisted configuration data (SQL Server error: 41034).  To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.

From Cluster Events:

The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk. 
Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

Why would restarting a secondary replica cause loss of Quorum?

From the cluster log:

death timer is already running, 1 seconds left.
One off quorum (1 3)
Death timer expired after 90 seconds (death timer started at 2023/10/19-20:14:29.186). Lost quorum.

More from the cluster log:

Proposed View is <ViewChanged joiners=() downers=(2) inExtendedGrace=() newView=5402(1 3) oldView=5302(1 2 3) joiner=false form=false/>

Any idea what the issue could be here?

Thanks

0

1 Answer 1

2

The whole issue, while we could dive into each specific error and write a novel, is that quorum was unable to be achieved and thus, was lost.

The Cluster service is shutting down because quorum was lost.

This partition of the cluster decided it did not have quorum and thus shut itself down as to not cause a potential split brain. This is normal behavior.

Why did the cluster lose quorum? We don't have the cluster logs so we don't know. However, from the excerpt given in the OP, 2 nodes have the potential for a vote and there is a witness which also has potential for a vote. In this case, the witness was unable to be arbitrated for its potential vote.

death timer is already running, 1 seconds left. One off quorum (1 3)

Given the OP already told us that one of the nodes has its voting disabled and that the only other node which had the potential for a vote was rebooted, one of the remaining nodes doesn't have the ability for a vote.

newView=5402(1 3)

This means it was (as shown above) 1 off quorum (3 votes, it has 1, needs 1 for quorum for this partition).

Why didn't it win arbitration for the witness? We don't know, we don't have the cluster log and other relevant windows logs.

Whether or not the current cluster configuration is proper for the expected results is also unknown as we don't know the full cluster config or expected results in events of various failures.

Once the regroup occurs and arbitration for the witness is attempted, that's where, in the cluster log, deep diving the issue should start.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.