I have an agent job that runs a very simple "select max(date) from table" against a linked sql server. The linked server is part of an AG and the link from remote uses a listener to get to primary. Most occasions, the job runs without issue but, with increased frequency, it times out at the 10min mark (default remote connection timeout). If it fails, I can run the underlying sql directly on the linked server and it returns within 5 seconds. Here is a screen grab of my activity monitor showing what the process looks like on the linked server when executing from remote and hanging; PAGEIOLATCH_SH and CXPACKET. While this is occurring, running the "select" locally produces results as expected. I cannot find anything, for the life of me, that gives a viable solution or a reason as to why this fails to begin with. Any help would be greatly appreciated.
So far, I have looked in logs but did not find anything. Increasing the default remote cnxn timeout to 20min decreased the fail frequency but has not removed it entirely. There are not other significant loads against the table, database, or server as can be proven when running locally produces sub-minute results as expected.
The last execution was successful, albeit longer than anticipated... Adding the Results, Stats, etc. images from Server1 (execution source).