Intermittently failover of my SQL Server resources on Windows Server 2016

Question

I have 2 Windows 2016 VM's running on Vmware ESXi VMware ESXi, 6.7.0, 17700523 with VMDK's as the SQL disks.

I have a SQL 2017 AlwaysOn Cluster running on Server 2016.

Basically everything is pointing to an issue with the network configuration but for the time being we're stuck without a solution.

Has anyone come across a similar issue which tends to failover the resources randomly?

SQL Server

First machine : SQLDB01 , 10.20.20.30 Second machine : SQLDB02 , 10.20.20.31 AG Name : SQLDBAG File share witness host : 10.20.20.40

we use VMXNET3 nic's

in the Failover Cluster Management – Cluster Event

[FTI][Follower] Ignoring duplicate connection: route to remote node found

[CHANNEL 10.20.20.30:~62034~] graceful close, status (of previous failure, may not indicate problem) (0)


[NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.20.20.31:3343 remote address 10.20.20.30:3343

[DCM] Force disconnect failed on DisconnectSmbInstance::CSV, status (c000000d)


[PULLER SQLDB01] ReadObject failed with GracefulClose(1226)' because of 'channel to remote endpoint fe80::a1b3:e30a:c6a:a379%9:~54878~ is closed'

[QUORUM] Node 2: One off quorum (2)

[DCM] UpdateClusDiskMembership: ctl 300224 nodeSet (2), status 87

[RCM] Moving orphaned group Cluster Group from downed node SQLDB01 to node SQLDB02.

[RES] SQL Server Availability Group <SQLDBAG>: [hadrag] Lease Thread terminated

Operational Log:

Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.

---

UPDATED-10/30/2021

1. does the backup network support heartbeat too ? No Also , there is no relationship between backup NIC and failover clustering configuration. Already checked "Do not allow cluster network communication on this network " for BACKUP NIC.

2)How many (virtual) NICs? We have 2 NICs (LAN and BACKUP) 3)Are the VMs on same host? No, different ESX host

No any intensive security scans and vMotions.

Only I am backing up boot disk.(C Volume image backup)

---

timeframes that these events occur:

10/27/2021, 1:00:44 AM
Task: Create virtual machine snapshot

10/27/2021, 1:14:21 AM  Backup successful

10/27/2021, 1:14:21 AM  
Task: Remove snapshot

10/27/2021, 1:15:38 AM  Virtual machine SQLDB01 disks consolidated successfully 

--  
10/28/2021 1:14:22 AM  --->>  Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.


10/28/2021 1:14:28 AM  ---->> Cluster has lost the UDP connection from local endpoint 10.20.20.30:~3343~ connected to remote endpoint 10.20.20.31:~3343~.


10/28/2021 1:15:35 AM   [CHANNEL 10.20.20.31:~3343~]/recv: Failed to retrieve the results of overlapped I/O: 10054

---

SQLDB02 events :

I am assuming , there is conflict between Veeam replication job and netbackup daily incremental backup job. then I am getting disk consolidation message. but it doesn't happen all the time.

 10/28/2021, 1:00:32 AMTask: Create virtual machine snapshot   (NETBACKUP)
 10/28/2021, 1:00:49 AM  User logged event: Source: Veeam Backup Action: Job "SQLDB02_Replication" Operation: Started Status 
 10/28/2021, 1:00:58 AMTask: Create virtual machine snapshot    (VEEAM)
 10/28/2021, 1:14:17 AM   NetBackup: Backup successful for SQLDB02
  10/28/2021, 1:14:18 AMTask: Remove snapshot 
 WARNING : 10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation is needed on ESX_IP   (NETBACKUP)
  10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation failed on ESX_IP  (NETBACKUP
 10/28/2021, 1:16:53 AM    NetBackup: Consolidate disk failed for SQLDB02. 

Answer 1

Basically everything is pointing to an issue with the network configuration but for the time being we're stuck without a solution.

Has anyone come across a similar issue which tends to failover the resources randomly?

I have to start by saying that these issues are generally not the same issue in each environment. Having said that, in this situation you've shown that the cluster (NETFT is in charge of this) is complaining that it's not getting healthchecks at all or in a timely manner. The initial reaction is that there is some sort of networking problem, and there very well could be, but there are many layers to this since it's a virtualized environment.

To level set, the healthcheck being done is a UDP packet over 3343 which is extremely tiny in size. The healthcheck has a timestamp in it and is considered lost if it is not received and processed within a specific threshold (configurable but typically 1 second).

One such issue with this can be availaility of scheduling and processing, in the VM itself but also on the host. This happens quite often when the host is over subscribed, generally from the expectation that hypethreaded cores are the same as full execution units. Thus the VM doesn't need to have high cpu in order to have issues receiving healthchecks in a timely manner. You'll want to check the various VMWare based counters for this (there are a few) to make sure that the VM isn't being stalled, staggered (held back in a multi-processor configuration), or otherwise unable to be scheduled in a timely manner (should be less than 1 ms).

Another common issue is that the network traffic is either discarded by the host or the host has other noisey neighbors. Inside of Windows this can be checked via the discarded packets, there are various items in the Host to which this can be checked. It would also make sense to run a circular network trace in the guest, on the host, and in the other nodes guests and hosts to make sure the traffic is indeed reaching its target in a timely manner.

Since guests can be stunned for things such as backups (which doesn't follow the prop use of VSS integration) and migrations (V-Motion for example), any backups, migrations, etc., should all be checked and plotted against times there are issues. Since other guests coming over to the host can take significant resources or starve other guests for resources based on internal company policies and configurations, this is an item that needs to be investigated.

This is not an all inclusive list, it is the most common that I tend to deal with on a daily basis, which is what was asked.