We have experienced a full cluster failure on our 3 node cluster running SQL Server 2008 R2 with 7 SQL Server instances. It started when we rebooted one of the nodes and it returned an error:
Node '%1' failed to establish a communication session while joining the cluster. The was due to an authorization failure. Please verify that the nodes are running compatible versions of the cluster service software.
The other two nodes were still online. Running a cluster validation test passed. After consulting with Microsoft we decided to take down all 3 nodes, bring one node online with force quorum option (so the cluster would come up on a single node), then bring the other nodes online one at a time. Unfortunately, after doing this, we were left with a single node forced online, because neither of the other nodes would join. The same error above was being reported on both the other nodes.
No resolution has been found for this after 2 weeks of investigation so we were presented with two options:
- Rebuild the cluster
- Evict the nodes and try to get them rejoined
My question is, this cluster has 7 SQL Server instances on it, and one File Server service. If the cluster is completely rebuilt, or a node is evicted, what are the steps to get SQL Server back working? Do we have to completely reinstall SQL Server or dDo we just have to add the instances back into the new cluster using Failover Cluster Manager without lengthy software installs?