I am attempting to create a distributed AG on SQL 2016 SP1, but keep getting the following errors. I have tried this in a VirtualBox (local desktop) and actual VMWare virtual environment but getting the same errors each time.
The AGs creates successfully on both WFCs, but when I try to "ALTER AVAILABILITY GROUP [Dist AG Name] JOIN" on the primary replica of the 'remote' AG, the command completes successfully, but I immediately get this error in the log:
A connection timeout has occurred while attempting to establish a connection to availability replica 'AG1' with id [FCAA4083-D6B8-1BAC-9E8C-6AECE34513E6]. Either a networking or firewall issue exists, or the endpoint address provided for the replica is not the database mirroring endpoint of the host server instance.
On the primary replica of the 'local' AG, I start getting these errors in the log repeatedly:
Length specified in network packet payload did not match number of bytes read; the connection has been closed. Please contact the vendor of the client library. [CLIENT: 192.168.56.104]
These are the bytes that show up in Event Viewer event:
0000: AC 45 00 00 14 00 00 00 ¬E...... 0008: 14 00 00 00 41 00 47 00 ....A.G. 0010: 31 00 53 00 45 00 52 00 1.S.E.R. 0018: 56 00 45 00 52 00 31 00 V.E.R.1. 0020: 5C 00 41 00 47 00 31 00 \.A.G.1. 0028: 49 00 4E 00 53 00 54 00 I.N.S.T. 0030: 31 00 00 00 00 00 00 00 1.......
In both environments, I am only dealing with a single subnet and a single domain. I've also tried removing all special characters from VM hostnames, instance names, AG names, and listener names during troubleshooting.
Wondering if anyone has ran into this before. I don't see anything in the SP2 release notes that suggests its a bug, but I'm willing to try an SP2 upgrade if anyone thinks its worthwhile. (Unfortunately I am not in a position to be able to upgrade my prod environment for some time which is why I haven't tried it yet.)
I can also post full endpoint/AG/listener creation scripts if anyone is willing to try to reproduce or provide troubleshooting tips.