Error 823 and no connection

Question

One of our servers tells us it has Error 823 issues:

TITLE: Connect to Server

Cannot connect to ServerName. ADDITIONAL INFORMATION: Warning: Fatal error 823 occurred at Mar 1 2016 > 10:45AM. Note the error and time, and contact your system administrator. (Microsoft SQL Server, Error: 21)

The windows log is floated with the same message.

No connection to the sql server is possible. So DBCC CHECKDB cannot be used / executed.

I ran SQLIOSIM and it raised couple of errors that requests are outstanding for more than n sec on a specific drive.

A Similar issue happened last night on another system - and it looks like it was workarounded by a failover.

I don't know where to look at next since I cant even connect to the Sql Server.

Answer 1

SQLIOSIM has identified at least one disk that is failing. You could look in the system event log for errors or warnings that concern the same disk subsystem's IO. The active SQL Server errorlog (C:\Program Files\Microsoft SQL Server\MSSQLx\MSSQL\Log, which is a text file), will also contain the 823 error. The 823 error being raised in the system event log could concern a driver. If SQL Server is failing to start, one of the system databases' disk(s) is likely to have failed.

Failing over SQL Server resets the driver and sets up a new IO path, but failover clustering is designed for high availability, not fault tolerance. Each time an IO failure occurs, a database can become corrupted, which database backups can backup. You should restore your backups on a DR server, run checkdb upon the restored backups. In the meanwhile, ask your system administrator to resolve the disk IO failures. The longer you wait, the higher the risk of corruption being backed up.