SQL Server 2017 AlwaysOn Failover Cluster issue

Question

I have 2 Windows 2012R2 VM's running on Vmware ESXi 7 with VMDK's as the SQL disks.

I have a SQL 2017 AlwaysOn Cluster running on Server 2012R2.

there is no any running backup (SNAPSHOT)

PROD : 10.10.22.1 and 10.10.22.2 HEARTBEAT : 192.168.200.2 and 192.168.200.3 file share witness already defined

CrossSubnetDelay          : 1000
CrossSubnetThreshold      : 20
PlumbAllCrossSubnetRoutes : 0
SameSubnetDelay           : 1000
SameSubnetThreshold       : 10

get-cluster |fl history

RouteHistoryLength : 20

LOG:

Cluster has missed two consecutive heartbeats for the local endpoint 10.10.22.1~3343~ connected to remote endpoint 10.10.22.2:~3343
Cluster has missed two consecutive heartbeats for the local endpoint 192.168.200.2:~3343~ connected to remote endpoint 192.168.200.3:~3343~.
Clustered role 'sqlcluster' is moving to cluster node 'host02'.

FailoverCluster Logs:

Cluster node 'host01' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
Cluster node 'host02' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
Cluster resource 'sqlcluster' of type 'SQL Server Availability Group' in clustered role 'sqlcluster' failed.
The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk. 
Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.
Cluster resource 'File Share Witness' of type 'File Share Witness' in clustered role 'Cluster Group' failed.
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it.  Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

System Log:

Unable to update the IP address on Isatap interface isatap.{1DF84235-46E7-44DE-BD8F-5A80FD1BD3BD}. Update Type: 0. Error Code: 0x57.

Answer 1

The errors are pretty descriptive. Usually this happens when network - whatever you put into this notion - physical hardware, or virtualized inside VMWare, or related drivers - are having issues and nodes are losing communication with each other. And/or fileshare witness is going down. This makes them skip the heartbeat, cluster loses quorum and goes offline.

To troubleshoot, you can try below:

• Run the Validate a Configuration wizard to check your network configuration
• check/diagnose the physical hardware
• update/patch the network drivers and firmware
• update/patch VMWare to latest versions
• run other network diagnostics software
• increase heartbeat settings / timeout interval in the Failover Cluster Managed, and in Availability group settings