I have 2 Windows 2016 VM's running on Vmware ESXi VMware ESXi, 6.7.0, 17700523 with VMDK's as the SQL disks.

I have a SQL 2017 AlwaysOn Cluster running on Server 2016.

Basically everything is pointing to an issue with the network configuration but for the time being we're stuck without a solution.

Has anyone come across a similar issue which tends to failover the resources randomly?

SQL Server

First machine : SQLDB01 , 10.20.20.30 Second machine : SQLDB02 , 10.20.20.31 AG Name : SQLDBAG File share witness host : 10.20.20.40

we use VMXNET3 nic's

in the Failover Cluster Management – Cluster Event

[FTI][Follower] Ignoring duplicate connection: route to remote node found

[CHANNEL 10.20.20.30:~62034~] graceful close, status (of previous failure, may not indicate problem) (0)


[NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.20.20.31:3343 remote address 10.20.20.30:3343

[DCM] Force disconnect failed on DisconnectSmbInstance::CSV, status (c000000d)


[PULLER SQLDB01] ReadObject failed with GracefulClose(1226)' because of 'channel to remote endpoint fe80::a1b3:e30a:c6a:a379%9:~54878~ is closed'

[QUORUM] Node 2: One off quorum (2)

[DCM] UpdateClusDiskMembership: ctl 300224 nodeSet (2), status 87

[RCM] Moving orphaned group Cluster Group from downed node SQLDB01 to node SQLDB02.

[RES] SQL Server Availability Group <SQLDBAG>: [hadrag] Lease Thread terminated

Operational Log:

Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.

UPDATED-10/30/2021

1. does the backup network support heartbeat too ? No Also , there is no relationship between backup NIC and failover clustering configuration. Already checked "Do not allow cluster network communication on this network " for BACKUP NIC.

2)How many (virtual) NICs? We have 2 NICs (LAN and BACKUP) 3)Are the VMs on same host? No, different ESX host

No any intensive security scans and vMotions.

Only I am backing up boot disk.(C Volume image backup)

timeframes that these events occur:

10/27/2021, 1:00:44 AM
Task: Create virtual machine snapshot

10/27/2021, 1:14:21 AM  Backup successful

10/27/2021, 1:14:21 AM  
Task: Remove snapshot

10/27/2021, 1:15:38 AM  Virtual machine SQLDB01 disks consolidated successfully 

--  
10/28/2021 1:14:22 AM  --->>  Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.


10/28/2021 1:14:28 AM  ---->> Cluster has lost the UDP connection from local endpoint 10.20.20.30:~3343~ connected to remote endpoint 10.20.20.31:~3343~.


10/28/2021 1:15:35 AM   [CHANNEL 10.20.20.31:~3343~]/recv: Failed to retrieve the results of overlapped I/O: 10054