0

We have a 3 replica SQL Server 2016 availability group. When the secondary synchronous goes unhealthy the database that resides on the primary becomes unreadable / unwritable, at this time the primary still shows healthy.

The cause behind the unhealthy secondary was upgrading our storage firmware. When removing the synchronous secondary from the availability group the primary then becomes usable. After we are able to use 'join' on the synchronous secondary to add back to the AG. So there doesn't seem to be any synchronization issues.

Why would the primary become unusable in this situation, does not seem like our AG is working as expected.

Additional Notes: - The 3rd SQL instance is asynchronous - We have tried to change all replicas to manual failover to try to negate the issue

From the logs it looks like the WSFC service is stopped, after it is restarted the SQL Server Availability group does not seem to recover on it's own. Making the primary database unusable. In order to resolve we remove the synchronous secondary and the primary server database is usable again.

SECONDARY LOG

2019-03-04 00:00:13.15 spid15s     This instance of SQL Server has been using a process ID of 2124 since 2/24/2019 10:48:39 PM (local) 2/25/2019 4:48:39 AM (UTC). This is an informational message only; no user action is required.  
2019-03-04 20:15:12.36 spid33s     SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [Z:\Microsoft SQL Server\MSSQL12\MSSQL\TempDB\templog.ldf] in database id 2.  The OS file handle is 0x0000000000000D28.  The offset of the latest long I/O is: 0x00000050e7b800  
2019-03-04 20:15:22.45 spid33s     SQL Server has encountered 1 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [F:\Microsoft SQL Server\MSSQL12\MSSQL\DATA\MSDBLog.ldf] in database id 4.  The OS file handle is 0x0000000000000D14.  The offset of the latest long I/O is: 0x000000483ade00  
2019-03-04 20:15:46.89 spid50s     A connection timeout has occurred on a previously established connection to availability replica 'SERVER\INSTANCE' with id [SECONDARY_GUI].  Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.  
2019-03-04 20:16:02.84 spid50s     A connection timeout has occurred while attempting to establish a connection to availability replica 'SERVER\INSTANCE' with id [SECONDARY_GUI]. Either a networking or firewall issue exists, or the endpoint address provided for the replica is not the database mirroring endpoint of the host server instance.  
2019-03-04 20:16:02.84 spid29s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 Server      Always On Availability Groups: Local Windows Server Failover Clustering node is no longer online. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid52s     Always On: The availability replica manager is going offline because the local Windows Server Failover Clustering (WSFC) node has lost quorum. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid52s     Always On: The local replica of availability group 'AG_Name' is stopping. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid52s     The state of the local availability replica in availability group 'AG_Name' has changed from 'SECONDARY_NORMAL' to 'RESOLVING_NORMAL'.  The state changed because the local instance of SQL Server is shutting down.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 20:16:10.26 spid50s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid22s     The availability group database "DB" is changing roles from "SECONDARY" to "RESOLVING" because the mirroring session or availability group failed over due to role synchronization. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid22s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(12048:90279:33)'    Commit Time: 'Mar  4 2019  8:11PM'  
2019-03-04 20:16:10.26 spid52s     Always On: The availability replica manager is starting. This is an informational message only. No user action is required.  
2019-03-04 20:16:10.26 spid52s     Always On Availability Groups: Waiting for local Windows Server Failover Clustering service to start. This is an informational message only. No user action is required.  
2019-03-04 20:16:16.75 Backup      Error: 3041, Severity: 16, State: 1.  
2019-03-04 20:16:16.75 Backup      BACKUP failed to complete the command BACKUP LOG DB. Check the backup application log for detailed messages.  
2019-03-04 20:17:15.27 spid52s     Always On Availability Groups: Local Windows Server Failover Clustering service started. This is an informational message only. No user action is required.  
2019-03-04 20:17:15.27 spid52s     Always On Availability Groups: Waiting for local Windows Server Failover Clustering node to start. This is an informational message only. No user action is required.  
2019-03-04 20:17:15.27 spid52s     Always On Availability Groups: Local Windows Server Failover Clustering node started. This is an informational message only. No user action is required.  
2019-03-04 20:17:15.27 spid52s     Always On Availability Groups: Waiting for local Windows Server Failover Clustering node to come online. This is an informational message only. No user action is required.  
2019-03-04 20:17:20.27 spid52s     Always On Availability Groups: Local Windows Server Failover Clustering node is online. This is an informational message only. No user action is required.  
2019-03-04 20:17:20.28 spid52s     Always On: The availability replica manager is waiting for the instance of SQL Server to allow client connections. This is an informational message only. No user action is required.  
2019-03-04 20:17:20.28 spid52s     Always On: The local replica of availability group 'AG_Name' is starting. This is an informational message only. No user action is required.  
2019-03-04 20:17:44.63 spid31s     The state of the local availability replica in availability group 'AG_Name' has changed from 'RESOLVING_NORMAL' to 'SECONDARY_NORMAL'.  The state changed because the availability group state has changed in Windows Server Failover Clustering (WSFC).  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 20:17:44.63 spid22s     The availability group database "DB" is changing roles from "RESOLVING" to "SECONDARY" because the mirroring session or availability group failed over due to role synchronization. This is an informational message only. No user action is required.  
2019-03-04 20:17:44.63 spid22s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(12048:90279:33)'    Commit Time: 'Mar  4 2019  8:11PM'  
2019-03-04 20:17:44.63 spid62s     A connection for availability group 'AG_Name' from availability replica 'PRIMARYSERVER\INSTANCE' with id  [PRIMARY_GUI] to 'SERVER\INSTANCE' with id [SECONDARY_GUI] has been successfully established.  This is an informational message only. No user action is required.  
2019-03-04 20:30:00.43 Backup      Error: 3041, Severity: 16, State: 1.  
2019-03-04 20:30:00.43 Backup      BACKUP failed to complete the command BACKUP LOG DB. Check the backup application log for detailed messages.  
2019-03-04 20:45:00.45 Backup      Error: 3041, Severity: 16, State: 1.  
2019-03-04 20:45:00.45 Backup      BACKUP failed to complete the command BACKUP LOG DB. Check the backup application log for detailed messages.  
2019-03-04 21:00:00.43 Backup      Error: 3041, Severity: 16, State: 1.  
2019-03-04 21:00:00.43 Backup      BACKUP failed to complete the command BACKUP LOG DB. Check the backup application log for detailed messages.  
2019-03-04 21:15:00.34 Backup      Error: 3041, Severity: 16, State: 1.  
2019-03-04 21:15:00.34 Backup      BACKUP failed to complete the command BACKUP LOG DB. Check the backup application log for detailed messages.  
2019-03-04 21:29:01.14 spid31s     Error: 19435, Severity: 16, State: 1.  
2019-03-04 21:29:01.14 spid31s     Always On: WSFC AG integrity check failed for AG 'AG_Name' with error 41041, severity 16, state 0.  
2019-03-04 21:29:01.14 spid31s     Always On: The local replica of availability group 'AG_Name' is being removed. The instance of SQL Server failed to validate the integrity of the availability group configuration in the Windows Server Failover Clustering (WSFC) store.  This is expected if the availability group has been removed from another instance of SQL Server. This is an informational message only. No user action is required.  
2019-03-04 21:29:01.14 spid31s     The state of the local availability replica in availability group 'AG_Name' has changed from 'SECONDARY_NORMAL' to 'NOT_AVAILABLE'.  The state changed because the associated availability group has been deleted.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 21:29:10.20 spid34s     A connection timeout has occurred on a previously established connection to availability replica 'SERVER\INSTANCE' with id [SECONDARY_GUI].  Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.  
2019-03-04 21:29:10.20 spid34s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:29:20.20 spid21s     A connection timeout has occurred while attempting to establish a connection to availability replica 'SERVER\INSTANCE' with id [SECONDARY_GUI]. Either a networking or firewall issue exists, or the endpoint address provided for the replica is not the database mirroring endpoint of the host server instance.  
2019-03-04 21:29:34.30 spid66s     Nonqualified transactions are being rolled back in database DB for an Always On Availability Groups state change. Estimated rollback completion: 100%. This is an informational message only. No user action is required.  
2019-03-04 21:29:34.30 spid58s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:29:34.44 spid66s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(12048:90279:33)'    Commit Time: 'Mar  4 2019  8:11PM'  
2019-03-04 21:29:34.51 spid66s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(12048:90279:33)'    Commit Time: 'Mar  4 2019  8:11PM'  
2019-03-04 21:29:34.52 spid66s     Starting up database 'DB'.  
2019-03-04 21:29:34.53 spid66s     The database 'DB' is marked RESTORING and is in a state that does not allow recovery to be run.  
2019-03-04 21:33:51.05 spid58      The state of the local availability replica in availability group 'AG_Name' has changed from 'NOT_AVAILABLE' to 'RESOLVING_NORMAL'.  The state changed because the local availability replica is joining the availability group.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 21:33:51.10 spid23s     The state of the local availability replica in availability group 'AG_Name' has changed from 'RESOLVING_NORMAL' to 'SECONDARY_NORMAL'.  The state changed because the availability group state has changed in Windows Server Failover Clustering (WSFC).  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 21:33:51.11 spid30s     A connection for availability group 'AG_Name' from availability replica 'PRIMARYSERVER\INSTANCE' with id  [PRIMARY_ID] to 'SERVER\INSTANCE' with id [SECONDARY_GUI] has been successfully established.  This is an informational message only. No user action is required.  
2019-03-04 21:33:51.32 spid58      ALTER DB DB with AGNAME:AG_Name  
2019-03-04 21:33:51.32 spid58      ALTER DB param option: SET  
2019-03-04 21:33:51.43 spid58      Always On Availability Groups connection with primary database established for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.45 spid30s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(0:0:0)'    Commit Time: 'Jan  1 1900 12:00AM'  
2019-03-04 21:33:51.45 spid34s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.47 spid30s     State information for database 'DB' - Hardened Lsn: '(12048:90290:1)'    Commit LSN: '(0:0:0)'    Commit Time: 'Jan  1 1900 12:00AM'  
2019-03-04 21:33:51.47 spid30s     Starting up database 'DB'.  
2019-03-04 21:33:51.61 spid30s     Always On Availability Groups connection with primary database established for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.61 spid30s     The recovery LSN (12048:90290:1) was identified for the database with ID 5. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.61 spid30s     175 transactions rolled forward in database 'DB' (5:0). This is an informational message only. No user action is required.  
2019-03-04 21:33:51.62 spid34s     Always On Availability Groups connection with primary database established for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.62 spid34s     The recovery LSN (12048:90290:1) was identified for the database with ID 5. This is an informational message only. No user action is required.  
2019-03-04 21:33:51.63 spid30s     CHECKDB for database 'DB' finished without errors on 2019-03-02 23:10:00.350 (local time). This is an informational message only; no user action is required.  
2019-03-04 21:33:51.63 spid41s     Always On Availability Groups connection with primary database established for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 21:33:53.63 spid41s     The recovery LSN (12048:90290:1) was identified for the database with ID 5. This is an informational message only. No user action is required.  
2019-03-04 22:24:08.24 spid28s     Always On Availability Groups connection with primary database terminated for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 22:24:08.26 spid67s     A connection for availability group 'AG_Name' from availability replica 'PRIMARYSERVER\INSTANCE' with id  [PRIMARY_ID] to 'SERVER\INSTANCE' with id [SECONDARY_GUI] has been successfully established.  This is an informational message only. No user action is required.  
2019-03-04 22:24:16.09 spid67s     Always On Availability Groups connection with primary database established for secondary database 'DB' on the availability replica 'SERVER\INSTANCE' with Replica ID: {SECONDARY_GUI}. This is an informational message only. No user action is required.  
2019-03-04 22:24:16.10 spid67s     The recovery LSN (12048:90704:1) was identified for the database with ID 5. This is an informational message only. No user action is required.  
2019-03-04 22:24:24.36 spid33s     SQL Server has encountered 9 occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [Z:\Microsoft SQL Server\MSSQL12\MSSQL\TempDB\templog.ldf] in database id 2.  The OS file handle is 0x0000000000000D28.  The offset of the latest long I/O is: 0x00000054d6c600  
2019-03-04 23:00:34.65 spid54      Attempting to cycle error log. This is an informational message only; no user action is required. 

PRIMARY LOG

2019-03-04 20:15:45.97 spid24s     A connection timeout has occurred on a previously established connection to availability replica 'SYNCSERVER\INSTANCE' with id [741665CF-9601-4FE3-9ABF-9E1A24F61333].  Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.  
2019-03-04 20:15:45.97 spid15s     DbMgrPartnerCommitPolicy::SetSyncAndRecoveryPoint: 741665CF-9601-4FE3-9ABF-9E1A24F61333:4  
2019-03-04 20:15:45.97 spid24s     Always On Availability Groups connection with secondary database terminated for primary database 'DB' on the availability replica 'SYNCSERVER\INSTANCE' with Replica ID: {741665cf-9601-4fe3-9abf-9e1a24f61333}. This is an informational message only. No user action is required.  
2019-03-04 20:15:46.15 spid24s     A connection timeout has occurred on a previously established connection to availability replica 'ASYNCSERVER\INSTANCE' with id [1BE5FDBB-E0D1-4D9D-9456-669203FC4D5B].  Either a networking or a firewall issue exists or the availability replica has transitioned to the resolving role.  
2019-03-04 20:15:46.15 spid24s     Always On Availability Groups connection with secondary database terminated for primary database 'DB' on the availability replica 'ASYNCSERVER\INSTANCE' with Replica ID: {1be5fdbb-e0d1-4d9d-9456-669203fc4d5b}. This is an informational message only. No user action is required.  
2019-03-04 20:16:01.46 Server      Error: 19421, Severity: 16, State: 1.  
2019-03-04 20:16:01.46 Server      SQL Server hosting availability group 'AG_Name' did not receive a process event signal from the Windows Server Failover Cluster within the lease timeout period.  
2019-03-04 20:16:01.51 Server      Error: 19407, Severity: 16, State: 1.  
2019-03-04 20:16:01.51 Server      The lease between availability group 'AG_Name' and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.  
2019-03-04 20:16:01.52 Server      Always On: The local replica of availability group 'AG_Name' is going offline because either the lease expired or lease renewal failed. This is an informational message only. No user action is required.  
2019-03-04 20:16:01.52 Server      The state of the local availability replica in availability group 'AG_Name' has changed from 'PRIMARY_NORMAL' to 'RESOLVING_NORMAL'.  The state changed because the lease between the local availability replica and Windows Server Failover Clustering (WSFC) has expired.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.
2019-03-04 20:16:01.52 spid56s     Always On Availability Groups connection with secondary database terminated for primary database 'DB' on the availability replica 'SYNCSERVER\INSTANCE' with Replica ID: {741665cf-9601-4fe3-9abf-9e1a24f61333}. This is an informational message only. No user action is required.  
2019-03-04 20:16:01.52 spid61s     Always On Availability Groups connection with secondary database terminated for primary database 'DB' on the availability replica 'ASYNCSERVER\INSTANCE' with Replica ID: {1be5fdbb-e0d1-4d9d-9456-669203fc4d5b}. This is an informational message only. No user action is required.  
2019-03-04 20:16:04.82 spid44s     Failed to update Replica status within the local Windows Server Failover Clustering (WSFC) due to exception 41034.  
2019-03-04 20:16:04.86 spid44s     Failed to update Replica status within the local Windows Server Failover Clustering (WSFC) due to exception 41005.  
2019-03-04 20:16:05.09 spid15s     Error: 41144, Severity: 16, State: 8.  
2019-03-04 20:16:05.09 spid15s     The local availability replica of availability group 'AG_Name' is in a failed state.  The replica failed to read or update the persisted configuration data (SQL Server error: 41034).  To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.  
2019-03-04 20:16:05.21 spid45s     Error: 41143, Severity: 16, State: 2.  
2019-03-04 20:16:05.21 spid45s     Cannot process the operation.  The local replica of availability Group 'AG_Name' is in a failed state.  A previous operation to read or update persisted configuration data for the availability group has failed.  To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.  
2019-03-04 20:16:05.26 spid44s     Error: 41143, Severity: 16, State: 2.  
2019-03-04 20:16:05.26 spid44s     Cannot process the operation.  The local replica of availability Group 'AG_Name' is in a failed state.  A previous operation to read or update persisted configuration data for the availability group has failed.  To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.  
2019-03-04 20:16:05.26 spid24s     Error: 41143, Severity: 16, State: 2.  
2019-03-04 20:16:05.26 spid24s     Cannot process the operation.  The local replica of availability Group 'AG_Name' is in a failed state.  A previous operation to read or update persisted configuration data for the availability group has failed.  To recover from this failure, either restart the local Windows Server Failover Clustering (WSFC) service or restart the local instance of SQL Server.  
2019-03-04 20:16:05.27 Server      Stopped listening on virtual network name 'Listener'. No user action is required.  
2019-03-04 20:16:05.32 Server      Always On Availability Groups: Local Windows Server Failover Clustering node is no longer online. This is an informational message only. No user action is required.  
2019-03-04 20:16:05.34 spid44s     Always On: The availability replica manager is going offline because the local Windows Server Failover Clustering (WSFC) node has lost quorum. This is an informational message only. No user action is required.  
2019-03-04 20:16:06.26 spid44s     Always On: The local replica of availability group 'AG_Name' is stopping. This is an informational message only. No user action is required.  
to 2019-03-04 20:17:21.27 spid44s  
2019-03-04 20:17:26.89 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:17:26.89 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.2]  
2019-03-04 20:17:27.42 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:17:27.42 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.1]  
2019-03-04 20:17:37.43 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:17:37.43 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.1]  
2019-03-04 20:17:44.53 spid61      Always On: The local replica of availability group 'AG_Name' is preparing to transition to the primary role in response to a request from the Windows Server Failover Clustering (WSFC) cluster. This is an informational message only. No user action is required.  
2019-03-04 20:17:44.57 spid61      The state of the local availability replica in availability group 'AG_Name' has changed from 'RESOLVING_NORMAL' to 'PRIMARY_PENDING'.  The state changed because the availability group is coming online.  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 20:17:44.57 Server      The Service Broker endpoint is in disabled or stopped state.  
2019-03-04 20:17:44.57 Server      Started listening on virtual network name 'AG_Listener'. No user action is required.  
2019-03-04 20:17:44.57 Server      The lease worker of availability group 'AG_Name' is now sleeping the excess lease time (163860 ms) supplied during online. This is an informational message only. No user action is required.  
2019-03-04 20:17:44.59 Server      The state of the local availability replica in availability group 'AG_Name' has changed from 'PRIMARY_PENDING' to 'PRIMARY_NORMAL'.  The state changed because the local replica has completed processing Online command from Windows Server Failover Clustering (WSFC).  For more information, see the SQL Server error log, Windows Server Failover Clustering (WSFC) management console, or WSFC log.  
2019-03-04 20:17:44.59 Server      The Service Broker endpoint is in disabled or stopped state.  
2019-03-04 20:17:44.62 spid15s     A connection for availability group 'AG_Name' from availability replica 'PRIMARYSERVER\INSTANCE' with id  [4F902163-A93E-4AD9-AE6A-429741623C31] to 'SYNCSERVER\INSTANCE' with id [741665CF-9601-4FE3-9ABF-9E1A24F61333] has been successfully established.  This is an informational message only. No user action is required.  
2019-03-04 20:17:44.62 spid15s     A connection for availability group 'AG_Name' from availability replica 'PRIMARYSERVER\INSTANCE' with id  [4F902163-A93E-4AD9-AE6A-429741623C31] to 'ASYNCSERVER\INSTANCE' with id [1BE5FDBB-E0D1-4D9D-9456-669203FC4D5B] has been successfully established.  This is an informational message only. No user action is required.  
2019-03-04 20:17:45.76 spid73s     Remote harden of transaction 'QDS base transaction' (ID 0x000000000248b6f5 0000:01d5cb41) started at Mar  4 2019  8:17PM in database 'DB' at LSN (12048:90290:5) failed.  
2019-03-04 20:17:46.29 spid73s     Error: 12412, Severity: 16, State: 1.  
2019-03-04 20:17:46.29 spid73s     Internal table access error: failed to access the Query Store internal table with HRESULT: 0x80004005.  
2019-03-04 20:17:46.65 spid73s     Remote harden of transaction 'QDS nested transaction' (ID 0x000000000248b6f3 0000:01d5cb40) started at Mar  4 2019  8:17PM in database 'DB' at LSN (12048:90292:1) failed.  
2019-03-04 20:18:08.87 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:18:08.87 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.2]  
2019-03-04 20:18:18.88 Logon       Error: 18456, Severity: 14, State: 38.  
to 2019-03-04 20:25:40.  
2019-03-04 20:25:50.74 spid58s     Remote harden of transaction 'SyscommitTabTblDelete' (ID 0x000000000249609b 0000:01d5cb42) started at Mar  4 2019  8:25PM in database 'DB' at LSN (12048:90293:3) failed.  
2019-03-04 20:25:50.82 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:25:50.82 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.2]  
2019-03-04 20:26:44.61 Logon       Error: 18456, Severity: 14, State: 38.  
to 2019-03-04 20:55:22.75  
2019-03-04 20:55:50.88 spid40s     Remote harden of transaction 'SyscommitTabTblDelete' (ID 0x00000000024bc1ef 0000:01d5cb43) started at Mar  4 2019  8:55PM in database 'DB' at LSN (12048:90294:3) failed.  
2019-03-04 20:56:03.99 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 20:56:03.99 Logon       Login failed for user 'SQL_Login'. Reason: Failed to open the explicitly specified database 'DB'. [CLIENT: 192.168.1.1]  
2019-03-04 20:56:14.01 Logon       Error: 18456, Severity: 14, State: 38.  
2019-03-04 21:25:08.73  
  • Could you add the error message(s) you receive when trying to read or write? The entries in the sql server event log from when this is happening could also be of use. – Randi Vertongen Mar 5 at 15:01
  • What does the error in sql server event log say when this happens? Also, can you attach a screenshot of what failover cluster manager looks like? – John-Henry Lochbaum Mar 5 at 15:32
  • 1
    It seems to be a Quorum problem. How is it setup? Please read more about it, here: docs.microsoft.com/en-us/windows-server/failover-clustering/… – Dan Mar 5 at 16:13
  • Your secondary had an issue at 2019-03-04 20:16:10.26 but your primary log you uploaded starts at 2019-03-04 21:25:11.50 or a little over an hour later. Any chance we can see it from the time the issue occurred. – Sean Gallardy Mar 6 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.