We are using an AlwaysOn Availability Group in SQL 2017 on Windows Server 2016 which consists of two servers on our primary site and one DR server on a remote site.
We recently did an internal IT DR test as proof-of-concept during which the rest of the company continued using the primary site. We isolated our remote DR network to simulate a catastrophic failure of our main site and tested the DR site to make sure we could bring it up and use it successfully. The test went very well up to this point, but we had a major issue when setting things back to normal.
Because the main site was still in use during the test, my main concern was to ensure that the live database would not be corrupted by the DR database which had also been used during the test, so I dropped the DR database before we opened the VPN and I expected the main site to start automatic seeding back to the DR site. But, because I had to force failover on the DR site, as soon as the network link was re-established, the primary site detected that the DR server had been started with the Force Quorum switch ( /fq) and it basically shut down the two live servers. I could not force failover back to the Live servers and had to rebuild the cluster.
Question: How do I prevent the Live servers from detecting that a forced failover was performed on the DR site?
I did consider shutting down the DR server before opening the network link, but am still not sure what would happen when it’s brought online. In any case, the remote database would need reseeding from Live.