We have a two-node cluster set up running SQL Server 2012 on Windows Server 2012 R2. The base cluster consists of only those two nodes. The quorum is set up as node + disk majority, with a shared disk sitting on an EqualLogic array connecting via iSCSI. Cluster configuration validates with no errors.
Earlier today, the two Availability Groups running on the cluster (both primary on Node #1) went into a resolving state on both nodes. Looking at the cluster events, there is nothing until it tries to restart the service. Everything underneath the Windows Failover cluster shows green (online and no warnings) - disks, network interfaces, nodes, etc.
Looking at the application log in Windows Event Viewer, the event showing that the Availability Groups are entering the Resolving state have been requested to do so by the cluster because a quorum could not be established. I cannot find anything else in any logs to support this, and the quorum passes during cluster validation.
The AGs do not recover. We found in the AlwaysOn log in SQL Server logs that the first node had a mirroring endpoint failure. This occurred immediately following a memory access violation with symptoms similar to a problem Microsoft has resolved with Cumulative Update 6, so we're going to try that next.
I was able to restart the SQL Server service on node 1 (the one with the memory access violations that the second note could not communicate with), and I was able to bring the availability group back online at that point. A reboot also brings everything back to 100%.
Do you have any insight as to why that wouldn't have failed node 1 and kept the database alive on node 2 when the mirroring endpoint failed? We're currently at AG level 3; would bringing that up to 4 trigger the failure on node 1 while leaving the service alive on node 2?