looking for ideas for diagnosing the root cause of the following issue. We have 3 availability nodes Primary,Synchronous replica, Asynchronous replica. no load other than T-Log backups exists on the Synchronous replica. At a certain time all replicas start lagging behind. The lag increases over a period of 6 hours. Data is still being written but very slowly. Suddenly the replicas start catching up and are back up to date within a period of around 30 minutes. No errors in any of the SQL Server Error logs. No blocking of the DB STARTUP process or any of the redo processes on either of the replicas (one has no users on it) HADR_DATABASE_FLOW_CONTROL has moved into the top waits for the Primary On Secondary top waits are normally PARALLEL_REDO_WORKER_WAIT REDO_THREAD_PENDING_WORK PARALLEL_REDO_TRAN_TURN
During the issue this stayed much the same other than PARALLEL_REDO_FLOW_CONTROL existed.
No evidence of a long running large transaction on the primary. Given that all replicas were affected it points to a problem with the primary. No index maintenance type jobs running. Can anyone think of any ddl,dml operations or other task that could cause this type of behavior? thanks
Environment is all SQL2016 CU9, disks are local on all machines