Split brain scenario

Question

We have a 4 node availability group with 3 nodes at primary site and 1 node at the DR. Suppose the Primary site entirely goes down and we have to do a forced failover at the DR.

Now if there is no connectivity between the two sites and suddenly the primary site comes up and in this case since it has the majority hence it will form a quorom and now we have to writable primaries , one at the primary and one at the DR.

This is where I am confused. What are the step can I take to avoid this split brain scenario.

As soon as I failover should I inform my sysadmin/network team not to bring all the nodes up together and to make sure to resume the connectivity between data center first and then bring the nodes one at a time so that it can catch up and avoid split brain.

Answer 1

In this scenario you generally cannot prevent the split brain. You may have no knowledge about the internal state of the primary data center.

The failover and recovery process is documented here WSFC Disaster Recovery through Forced Quorum (SQL Server), in particular after a forced failover you should make the surviving nodes into a healty, self-sufficient cluster by, among other things, "dropping availability group replicas, evicting nodes from the cluster, or flattening and re-installing software on a node."