I'm attempting to determine root cause for an issue that caused an outage for my company.
Essentially the site connections to our database server were timing out after not receiving a response for 5 min. A check of perfmon counters showed that the database server was using about 10% of processor, memory was under utilized, and our hard disks were not being stressed.
A check of long running queries using the following statement showed that there were a lot of queries queued up with a status of suspended the longest of which had been running for 50 minutes.
SELECT sqltext.TEXT, req.session_id, req.status, req.command, req.cpu_time, req.total_elapsed_time FROM sys.dm_exec_requests req CROSS APPLY sys.dm_exec_sql_text(sql_handle) AS sqltext
There were also several queries with a status of running but none of these had been running for a significant amount of time.
SQL Server threw no alerts during the time frame and nothing looked out of the ordinary in the logs. There were also no alerts for Blocking processes at this time. Finally the decision was made to fail over the SQL Server cluster.
This cleared up the issue.
I've exhausted the places I can think of to look to come up with an explanation for the outage. Has anyone experienced similar behavior? Is there something I should be checking that I haven't mentioned?