Skip to main content
Database Administrators

Return to Question

added 1007 characters in body
Source Link
Cell-o
Cell-o
  • 1.1k
  • 8
  • 21
  • 40

SQLDB02 events :

I am assuming , there is conflict between Veeam replication job and netbackup daily incremental backup job. then I am getting disk consolidation message. but it doesn't happen all the time.

 10/28/2021, 1:00:32 AMTask: Create virtual machine snapshot   (NETBACKUP)
 10/28/2021, 1:00:49 AM  User logged event: Source: Veeam Backup Action: Job "SQLDB02_Replication" Operation: Started Status 
 10/28/2021, 1:00:58 AMTask: Create virtual machine snapshot    (VEEAM)
 10/28/2021, 1:14:17 AM   NetBackup: Backup successful for SQLDB02
  10/28/2021, 1:14:18 AMTask: Remove snapshot 
 WARNING : 10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation is needed on ESX_IP   (NETBACKUP)
  10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation failed on ESX_IP  (NETBACKUP
 10/28/2021, 1:16:53 AM    NetBackup: Consolidate disk failed for SQLDB02.

SQLDB02 events :

I am assuming , there is conflict between Veeam replication job and netbackup daily incremental backup job. then I am getting disk consolidation message. but it doesn't happen all the time.

 10/28/2021, 1:00:32 AMTask: Create virtual machine snapshot   (NETBACKUP)
 10/28/2021, 1:00:49 AM  User logged event: Source: Veeam Backup Action: Job "SQLDB02_Replication" Operation: Started Status 
 10/28/2021, 1:00:58 AMTask: Create virtual machine snapshot    (VEEAM)
 10/28/2021, 1:14:17 AM   NetBackup: Backup successful for SQLDB02
  10/28/2021, 1:14:18 AMTask: Remove snapshot 
 WARNING : 10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation is needed on ESX_IP   (NETBACKUP)
  10/28/2021, 1:15:35 AM   Virtual machine SQLDB02 disks consolidation failed on ESX_IP  (NETBACKUP
 10/28/2021, 1:16:53 AM    NetBackup: Consolidate disk failed for SQLDB02.
added 1425 characters in body
Source Link
Cell-o
Cell-o
  • 1.1k
  • 8
  • 21
  • 40

UPDATED-10/30/2021

  1. does the backup network support heartbeat too ? No Also , there is no relationship between backup NIC and failover clustering configuration. Already checked "Do not allow cluster network communication on this network " for BACKUP NIC.

2)How many (virtual) NICs? We have 2 NICs (LAN and BACKUP) 3)Are the VMs on same host? No, different ESX host

No any intensive security scans and vMotions.

Only I am backing up boot disk.(C Volume image backup)

timeframes that these events occur:

10/27/2021, 1:00:44 AM
Task: Create virtual machine snapshot

10/27/2021, 1:14:21 AM  Backup successful

10/27/2021, 1:14:21 AM  
Task: Remove snapshot

10/27/2021, 1:15:38 AM  Virtual machine SQLDB01 disks consolidated successfully 

--  
10/28/2021 1:14:22 AM  --->>  Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.


10/28/2021 1:14:28 AM  ---->> Cluster has lost the UDP connection from local endpoint 10.20.20.30:~3343~ connected to remote endpoint 10.20.20.31:~3343~.


10/28/2021 1:15:35 AM   [CHANNEL 10.20.20.31:~3343~]/recv: Failed to retrieve the results of overlapped I/O: 10054

UPDATED-10/30/2021

  1. does the backup network support heartbeat too ? No Also , there is no relationship between backup NIC and failover clustering configuration. Already checked "Do not allow cluster network communication on this network " for BACKUP NIC.

2)How many (virtual) NICs? We have 2 NICs (LAN and BACKUP) 3)Are the VMs on same host? No, different ESX host

No any intensive security scans and vMotions.

Only I am backing up boot disk.(C Volume image backup)

timeframes that these events occur:

10/27/2021, 1:00:44 AM
Task: Create virtual machine snapshot

10/27/2021, 1:14:21 AM  Backup successful

10/27/2021, 1:14:21 AM  
Task: Remove snapshot

10/27/2021, 1:15:38 AM  Virtual machine SQLDB01 disks consolidated successfully 

--  
10/28/2021 1:14:22 AM  --->>  Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.


10/28/2021 1:14:28 AM  ---->> Cluster has lost the UDP connection from local endpoint 10.20.20.30:~3343~ connected to remote endpoint 10.20.20.31:~3343~.


10/28/2021 1:15:35 AM   [CHANNEL 10.20.20.31:~3343~]/recv: Failed to retrieve the results of overlapped I/O: 10054
Source Link
Cell-o
Cell-o
  • 1.1k
  • 8
  • 21
  • 40

Intermittently failover of my SQL Server resources on Windows Server 2016

I have 2 Windows 2016 VM's running on Vmware ESXi VMware ESXi, 6.7.0, 17700523 with VMDK's as the SQL disks.

I have a SQL 2017 AlwaysOn Cluster running on Server 2016.

Basically everything is pointing to an issue with the network configuration but for the time being we're stuck without a solution.

Has anyone come across a similar issue which tends to failover the resources randomly?

SQL Server

First machine : SQLDB01 , 10.20.20.30 Second machine : SQLDB02 , 10.20.20.31 AG Name : SQLDBAG File share witness host : 10.20.20.40

we use VMXNET3 nic's

in the Failover Cluster Management – Cluster Event

[FTI][Follower] Ignoring duplicate connection: route to remote node found

[CHANNEL 10.20.20.30:~62034~] graceful close, status (of previous failure, may not indicate problem) (0)


[NETFTAPI] Signaled NetftRemoteUnreachable event, local address 10.20.20.31:3343 remote address 10.20.20.30:3343

[DCM] Force disconnect failed on DisconnectSmbInstance::CSV, status (c000000d)


[PULLER SQLDB01] ReadObject failed with GracefulClose(1226)' because of 'channel to remote endpoint fe80::a1b3:e30a:c6a:a379%9:~54878~ is closed'

[QUORUM] Node 2: One off quorum (2)

[DCM] UpdateClusDiskMembership: ctl 300224 nodeSet (2), status 87

[RCM] Moving orphaned group Cluster Group from downed node SQLDB01 to node SQLDB02.

[RES] SQL Server Availability Group <SQLDBAG>: [hadrag] Lease Thread terminated

Operational Log:

Microsoft Failover Cluster Virtual Adapter (NetFT) has missed more than 40 percent of consecutive heartbeats.