I've just bumped into a new situation regarding alwaysOn availability. We have a setup with a large database (terabyte range) with one primary and one secondary replica. Geographically the servers are located a small distance apart. Log and full backups are taken from the secondary replica, and index maintenance is obviously run on primary. Sometimes these schedules overlap, if the full backup takes particularly long to complete.
For several years now, this setup has worked perfectly, until recently. Now we have had several nights where at a certain, specific time the log backups on secondary fail, giving an error on primary having a more recent backup than secondary (when there are no backups taken from primary at all). At the same point, the synchronization state is "synchronized and healthy", but the dm_hadr_database_replica_states system view shows that last_received_time is stuck at the same time as the failure of log backup on secondary, and the redo queue is ever growing. As a result, the log backup jobs fail and the log keeps increasing indefinitely. Once suspending and resuming the replication helped, but otherwise we've had to boot the entire secondary replica server (booting the service just got it stuck).
At first we thought it was a combination of some network error at a specific time, and some hitherto unknown bug in alwaysOn not resuming or reporting its state correctly. But then we reproduced this during the day: We were taking a full backup and reading a high volume of data for an ETL package, from the secondary replica, during which the log backups started failing. As we aborted the backup, after a while the replication caught up and started working again.
So the question: What can cause alwaysOn to behave this way? It seems extremely dangerous to me that a high availability feature can get into a state where the alarms don't work, because the synchronization state is on and supposedly healthy, while in reality it's not. And that it can't restore itself even during no additional load, but just gets stuck until rebooted.