Scenario: two Windows Server 2012 Standard boxes, housing 4 SQL Server 2012 Standard instances (2 active on each node). Housed in Cisco UCS B200 blades and on a Hitachi HUS 110 SAN (all cluster drives in a raid 10 pool, hooked up by 4x8 gig FC paths)
Friday one instance restarted twice in the morning, and another instance restarted later in the day that I have no logs from. The instance that restarted twice, I have the extended events sessions and cluster logs for, below is an portion of the cluster logs. The logs are word wrapped and best shown in notepad unwrapped.
I realize that the
isalive returns false is one of the cluster diagnostic checks and when it fails the Windows Cluster will restart the instance affected. I am having a hard time pinpointing where and what in the cluster logs I should be reading to figure out my exact root causes.
As far as I can tell the two root causes are as follows, from the first run "Query timeout expired " and the second run "TCP Provider: The specified network name is no longer available"
Now am I posting this here because what I want to know is if I am right in pointing to these lines as the reason for the restart. The "why did they happen and how do I fix them" is questions I'll have, assuming I've pointed out the two right items, but I need to know what I'm trying to fix to attempt to learn to fix it. I would ask other people at work but I'm the person who built this stuff and sadly the only DBA and the only one with half a clue how it works. I've been reading quite a bit on the cluster log since the crash but I'm spinning my head in circles and am hopeful someone out here in magic internet land can help me out.
I'm trying to avoid posting the whole bunch of logs, and I realize I may need to get a better look into things it's just that they don't look nice here and I'm not sure where else to throw them to point them to someone to read to give me a hand. If someone wants to see more and has an idea of where to throw them just let me know.
Thanks for your time.
First time it happened:
000010f4.00000b5c::2015/03/13-03:53:59.474 WARN [RES] SQL Server : [sqsrvres] Failed to retrieve data column. Return code -1 000010f4.00000b34::2015/03/13-03:54:01.709 ERR [RES] SQL Server : [sqsrvres] Failure detected, diagnostics heartbeat is lost 000010f4.00000b34::2015/03/13-03:54:01.709 INFO [RES] SQL Server : [sqsrvres] IsAlive returns FALSE .. 000010f4.00000b5c::2015/03/13-03:56:59.509 ERR [RES] SQL Server : [sqsrvres] ODBC Error: [HYT00] [Microsoft][SQL Server Native Client 11.0]Query timeout expired (0)
Second time it happened:
000002e4.00001f7c::2015/03/13-07:28:36.827 ERR [RES] SQL Server : [sqsrvres] Failure detected, diagnostics heartbeat is lost 000002e4.00001f7c::2015/03/13-07:28:36.827 INFO [RES] SQL Server : [sqsrvres] IsAlive returns FALSE 000002e4.00001f7c::2015/03/13-07:28:36.827 WARN [RHS] Resource SQL Server (RPT1) IsAlive has indicated failure. 000002e4.000015f4::2015/03/13-07:29:02.062 ERR [RES] SQL Server : [sqsrvres] ODBC Error: [08S01] [Microsoft][SQL Server Native Client 11.0]TCP Provider: The specified network name is no longer available. (64) and 000002e4.000015f4::2015/03/13-07:29:02.062 ERR [RES] SQL Server : [sqsrvres] ODBC Error: [08S01] [Microsoft][SQL Server Native Client 11.0]Communication link failure (64)