0

Environment is Windows Server 2019 in Azure

I have a 3 node multi-region cluster. Server 1 & 2 are in the same region while server 3 is in a remote region.

Server 3 does not have a vote in the cluster quorum and there is a file share witness in place.

The way I understood it is that if I shutdown server 2, its vote will get be removed.

Cluster quorum settings:

enter image description here

I have shut down server 2 and this is the result from the same query:

enter image description here

Server 2 is down but still has a vote? Why is this?

I have confirmed that dynamic quorum is enabled:

enter image description here

The Witness dynamic weight also does not change if server 2 is up or down

enter image description here

What I thought would happen after shutting down server 2 is that the cluster would dynamically remove the vote for server 2 as well as the file share witness.

I have also shutdown the server that holds the file share witness, it's down in the cluster and neither the vote for the witness or server 2 was removed. The cluster went completely down in this case.

Further testing: Have removed the file share witness and granted server 3 a vote in the quorum. When shutting down nodes in this configuration, votes are removed as expected.

Further investigation: Found this in the cluster log:

WARN  [QUORUM] Node 1: weight adjustment not performed. Cannot go below weight count 3 in a hybrid configuration with 2+ nodes

I don't get why the votes are not being removed from nodes that go down.

Any advice?

Thanks

2
  • guess its simply by design ? Sep 7, 2022 at 11:59
  • This behavior is not mentioned anywhere in the MS documentation. Strange... Sep 7, 2022 at 12:27

1 Answer 1

1

Server 2 is down but still has a vote? Why is this?

The reason is because if you look at the possible votes in the cluster, changing this value will make no difference. Two possible voting nodes and one possible voting witness, for a total of 3 possible votes. Take away a node. Now you have a single node and a single witness. If you lose the last node, the cluster goes down. If you don't lose the node the cluster stays up. Now take away the dynamic weight from node 2. What's the result? The same. There's no need to take away the vote because there is no difference between what will and won't happen. Additionally, id node 2 were to have its vote taken away, then the witness would as well per the dynamic witness algorithm.

I don't get why the votes are not being removed from nodes that go down.

This depends on the setup and configuration of the cluster. In this case, with 2-3 nodes, this is expected. In larger clusters, more shifting of weights will occur, however when down to the last 2 or so nodes it is expected to behave this same way.

8
  • Thank you The problem I have is that if I take down node 2 as well as the witness - The entire cluster shuts down while node 1 is still up and could maintain the cluster. This behavior does not happen with a 3 node (all nodes vote) cluster and no witness, if I shut down 2 nodes, the remaining node still serves and keeps the cluster up. Sep 7, 2022 at 13:28
  • Yes, because you've lost the number of possible votes 1/3 is not a majority. Additionally, the whole point of the witness is to arbitrate in these situations, since it can't, you'll fail. If losing the witness is common, then it's already doomed to fail. You're asking for a last node standing solution where the last node has no idea if it's partitioned or not and thus shuts down. Additionally, the last node without witness only works because you're nicely shutting the system down - where the cluster knows a nice shutdown occurred. Pulling the network cable would induce a cluster failure. Sep 7, 2022 at 13:36
  • So in essence, even though the cluster is configured with dynamic quorum - It's not actually dynamic with this particular configuration Sep 7, 2022 at 13:47
  • " Additionally, id node 2 were to have its vote taken away, then the witness would as well per the dynamic witness algorithm." Now this is exactly what I expected to happen. Sep 7, 2022 at 13:51
  • It is dynamic with the configuration, to a point based on the inputs given (such as a witness and a nice shutdown). You're testing cases that the configuration specifically does not handle, and it calls that out. Sep 7, 2022 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.