1

We have a 2012 Windows failover Cluster set up for SQL Server 2014 on VMWare ESXi, 6.0.0, 3620759. A couple weekends ago, our Server Admin made some updates to the the UCS Manager software, which in turn updated the Fabric Interconnects < one at a time >, so path B went down then Path A when it was completed. Per our Server Admin at no point were the VM’s disconnected or put in a disconnected state but this impacted our Windows cluster. The Cluster Manager lost connection to the nodes, quorum, Resource Hosting Subsystem and disk resources resulted in role fail overs for a couple of our SQL Instances (we have 4 SQL Server Instance roles total). Does anyone have any explanation on why this happened and should we bother to update the resource restart/dependency settings in the Cluster Failover Manager to mitigate this issue? See below.

7/14/18 3:36:19PM

Cluster node 'xxxxxxxxx' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

7/14/18 3:36:40PM

The Cluster service is shutting down because quorum was lost. This could be due to the loss of network connectivity between some or all nodes in the cluster, or a failover of the witness disk.

Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapter. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

7/14/18 3:36:48PM

The cluster Resource Hosting Subsystem (RHS) process was terminated and will be restarted. This is typically associated with cluster health detection and recovery of a resource. Refer to the System event log to determine which resource and resource DLL is causing the issue.

7/14/18 3:36:49PM

Cluster node 'xxxxxxxxx' was removed from the active failover cluster membership. The Cluster service on this node may have stopped. This could also be due to the node having lost communication with other active nodes in the failover cluster. Run the Validate a Configuration wizard to check your network configuration. If the condition persists, check for hardware or software errors related to the network adapters on this node. Also check for failures in any other network components to which the node is connected such as hubs, switches, or bridges.

7/14/18 3:36:50PM

Cluster resource 'Instance4Backup' of type 'Physical Disk' in clustered role 'SQL Server (SQL14SINST04)' failed.

Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet.

7/14/18 3:36:50PM

Ownership of cluster disk 'Instance4Backup' has been unexpectedly lost by this node. Run the Validate a Configuration wizard to check your storage configuration.

7/14/18 4:04:51PM

Ownership of cluster disk 'Instance2Backup' has been unexpectedly lost by this node. Run the Validate a Configuration wizard to check your storage configuration.

7/14/18 4:04:51PM

Cluster resource 'Instance2Backup' of type 'Physical Disk' in clustered role 'SQL Server (SQL14SINST02)' failed.

Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failed...

  • "Per our Server Admin at no point were the VM’s disconnected or put in a disconnected state but this impacted our Windows cluster." VMWare is well known for silently retrying until it succeeds and not telling you it's failing. Obviously it was failing or you wouldn't have lost access to these objects :) – Sean Gallardy Aug 2 '18 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.