2

First, a bit about our environment.

  • We run two instances of SQL Server 2016 Enterprise (13.0.4435.0) on Windows 2016 Servers.
  • To save money, we run SSRS 2016 alongside SQL Server on both servers
  • We have SSRS configured for Scale-out Deployment
  • We have the SSRS databases setup with database mirroring
  • Our principal sql server is named "CARL1" and our mirror sql server is named "CARL2"
  • CARL2's SSRS is configured to connect to CARL1 for the SSRS databases and uses the login "CARL2-ADMIN"
  • CARL1's SSRS is configured to connect to CARL1 for the SSRS databases and uses the login "CARL1-ADMIN"
  • SSRS on both CARL1 and CARL2 is working perfectly fine outside of the errors mentioned below
  • Both CARL1 and CARL2 have plenty of disk space

Almost every morning around 1:15am EST, we receive one or all of the following errors on our Principal SQL Server (CARL1):

The client was unable to reuse a session with SPID ###, which had been reset for connection pooling. The failure ID is 51. This error may have been caused by an earlier operation failing. Check the error logs for failed operations immediately before this error message.

Login failed for user 'STN\CARL1-Admin'. Reason: Failed to send an environment change notification to a log shipping partner node while revalidating the login. [CLIENT: ]

Login failed for user 'STN\CARL2-Admin'. Reason: Failed to send an environment change notification to a log shipping partner node while revalidating the login. [CLIENT: ]

Upon further investigation, I found this in the CARL2 SSRS 2016 error log files (the ones that begin with ReportServerService_MM_DD_YYYY_HH_MM_SS.log file name and are located in the C:\Program Files\Microsoft SQL Server\MSRS13.MSSQLSERVER\Reporting Services\LogFiles folder):

library!WindowsService_41!4a8!08/08/2017-01:21:11:: e ERROR: Throwing Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerStorageException: , An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database.; dbpolling!WindowsService_41!4a8!08/08/2017-01:21:11:: e ERROR: PollingFunction: Microsoft.ReportingServices.Diagnostics.Utilities.ReportServerStorageException: An error occurred within the report server database. This may be due to a connection failure, timeout or low disk condition within the database. ---> System.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception: The wait operation timed out.

dbpolling!WindowsService_41!4a8!08/08/2017-01:21:12:: i INFO: PollingMaintenance: Polling cycle completed.

library!WindowsService_41!19c4!08/08/2017-01:21:15:: e ERROR: Throwing Microsoft.ReportingServices.Library.ReportServerDatabaseUnavailableException: , Microsoft.ReportingServices.Library.ReportServerDatabaseUnavailableException: The report server cannot open a connection to the report server database. A connection to the database is required for all requests and processing. ---> System.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Login failed for user 'stn\CARL2-admin'.

Cannot continue the execution because the session is in the kill state. A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception: The wait operation timed out

--- End of inner exception stack trace ---

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)

at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()

at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()

at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()

at System.Data.SqlClient.TdsParserStateObject.TryReadByte(Byte& value)

at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()

at System.Data.SqlClient.SqlDataReader.get_MetaData()

at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)

at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)

at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)

at System.Data.SqlClient.SqlCommand.ExecuteReader()

at Microsoft.ReportingServices.Library.RunningJobsDb.GetMyRunni runningjobs!WindowsService_41!19c4!08/08/2017-01:21:15:: e ERROR: Error in timer Running Requests DB : Microsoft.ReportingServices.Library.ReportServerDatabaseUnavailableException: The report server cannot open a connection to the report server database. A connection to the database is required for all requests and processing. ---> System.Data.SqlClient.SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

Login failed for user 'stn\CARL2-admin'.

Cannot continue the execution because the session is in the kill state. A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.) ---> System.ComponentModel.Win32Exception: The wait operation timed out

Curious if anyone has encountered this trouble before and how we can solve it?

Both CARL1 and CARL2 are generally not under any load around 1:15am, so this error is very perplexing to us!

Our servers are running at Amazon Web Services within a private VPC. We do not take any EC2 backups/snapshots of the drive as they are practically worthless (we instead take backups of our databases and write them to Amazon S3). Also, we have multiple domain controllers and in reviewing logs, neither are under any load at that point of the day.

  • Have you read dba.stackexchange.com/questions/81077/… ? – Nic Aug 15 '17 at 21:33
  • Yep, we reviewed this post - however you cannot specify a Failover Partner in the SSRS database connection. Plus we are only encountering this error once a day...not every 5 seconds. – ChrisK Aug 16 '17 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.