0

The following error was occurred in cluster events and the availability group was failed which resulted the databases in non-synchronizing state.

A component on the server did not respond in a timely fashion. This caused the cluster resource 'AG' (resource type 'SQL Server Availability Group', DLL 'hadrres.dll') to exceed its time-out threshold. As part of cluster health detection, recovery actions will be taken. The cluster will try to automatically recover by terminating and restarting the Resource Hosting Subsystem (RHS) process that is running this resource.

Please help me to find the root cause (A component on the server did not respond in a timely fashion).

2 Answers 2

2

That error means the AG failed one of the health detection timeouts (lease, session, or health check) summarized in this table from the docs:

Mechanics and guidelines of lease, cluster, and health check timeouts for Always On availability groups - Summary of Timeout Guidelines

Your first stop should be to review the SQL Server error log on each node in the AG, to see if there was something going on that might have caused the instance to stop responding to the cluster AG resource. For instance, you might have crash dumps, deadlocked schedulers, etc.

To dig into the cluster-related details, you'll need to call the PowerShell Get-ClusterLog cmdlet (see here) to get the cluster log file from each node in the AG. Then find the time associated with the failure error message you mentioned in the question. Review the cluster log files around that time period for errors (search for "ERR") or any messages of interest from the AG resource DLL itself (which will include "[RES]" or "[hadrag]").

I would expect you'll find messages that include "timed out" or "IsAlive" showing the timing of the failures. At this point, you should be able to figure out which server was not responding, and investigate that directly.

Figuring these things can be difficult. Microsoft support has released a tool that can be used to categorize some types of failovers, which might be useful to you:

Failover Detection Utility - Availability Group Failover Analysis Made Easy

6
  • 1
    "Failover Detection Utility - Availability Group Failover Analysis Made Easy" cool, certainly missed something like this couple of years ago :D Commented Nov 6, 2020 at 18:13
  • @Josh Darnell Thanks for the information. From Sql logs found that the "SQL Server hosting availability group 'AG' did not receive a process event signal from the Windows Server Failover Cluster within the lease timeout period." which resulted lease time out. And I didn't find any issue in the log, prior to the lease timeout. In Cluster Log found that the "Resource 'AG' has not responded to the call LOOKSALIVE:0". Also. found the cluster resource(AG role) in failed state. Not able to figure out why the cluster resource is in failed state.
    – Sree Lekha
    Commented Nov 7, 2020 at 14:10
  • @Josh Darnell If one of the cluster networks is down, the cluster resource goes to failed state?
    – Sree Lekha
    Commented Nov 7, 2020 at 14:17
  • @SreeLekha Yeah, if you mean the thing in this screenshot is down, then yes - that will bring down the AG. That network is used by the cluster to communicate heartbeats and health checks. Commented Nov 9, 2020 at 13:09
  • @Josh Darnell Thanks! Please help me here . I can see the windows failed to install the update KB4052623 two minutes before the AG offline. Does this failure update installation have any effect on the cluster or AG?
    – Sree Lekha
    Commented Nov 9, 2020 at 16:51
-1

Unfortunately I couldn't find much information out there but you might want to review this article on Failover Clustering by Microsoft (it's seemed to help other people with similar issues, and goes pretty in depth): https://techcommunity.microsoft.com/t5/failover-clustering/failover-clustering-networking-basics-and-fundamentals/ba-p/1706005

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.