1

I have setup SQL AlwaysOn between primary and DR data centers. Here is the setup:

Primary data center: Server1 (Primary), Server2 (Sync Commit Secondary), Server3 (ASync Commit Secondary)

DR data center Server4 (ASync Commit Secondary)

Data synchronization and manual failover works fine. But, sometimes, the AlwaysOn cluster automatically fails over to Sync Commit Secondary on Primary data center. Here is the error message from Failover Cluster Manager->Cluster Events:

"Cluster has missed two consecutive heartbeats for the local endpoint xx.xx.xx.yy:~3343~ connected to remote endpoint xx.xx.xx.zz:~3343~"

"Cluster has lost the UDP connection from local endpoint xx.xx.xx.yy:~3343~ connected to remote endpoint xx.xx.xx.zz:~3343~"

I had our network engineer check all connections multiple times and he confirmed everything is fine. But he was also able to confirm (using monitoring tools) that right at the time of a failover, there is almost 2GB worth of traffic going from Primary Server to DR server. That happens every time. I had checked the times of all failovers and there is no job or process occuring that will produce 2GB worth of data. Also, this happens regardless of which server is primary.

I'd really appreciate if someone can provide some guidance. Eventhough the failover works fine, this unexpected automatic failover due to missed heartbeats are occurring often (2-3 times a month).

Thanks in advance. Have a nice day.

--Update

Here is the list of errors from the Cluster Validation Report:

Under Network Section, I see the following error messages in Red:

Validate Network Communication

Network interfaces Server4 (DR) - SAN_Team and Server1 (Primary) - SAN_Team - VLAN 20 are on the same cluster network, yet address xx.xx.xx.pp is not reachable from xx.xx.xx.yy using UDP on port 3343.

Network interfaces Server4 (DR) - SAN_Team and Server2 (Secondary) - SAN_Team - VLAN 20 are on the same cluster network, yet address xx.xx.xx.qq is not reachable from xx.xx.xx.yy using UDP on port 3343.

Network interfaces Server4 (DR) - SAN_Team and Server3 (Secondary) - SAN_Team - VLAN 20 are on the same cluster network, yet address xx.xx.xx.zz is not reachable from xx.xx.xx.yy using UDP on port 3343.

Network interfaces Server1 (Primary) - SAN_Team - VLAN 20 and Server4 (DR) - SAN_Team are on the same cluster network, yet address xx.xx.xx.yy is not reachable from xx.xx.xx.pp using UDP on port 3343.

Network interfaces Server2 (Secondary) - SAN_Team - VLAN 20 and Server4 (DR) - SAN_Team are on the same cluster network, yet address xx.xx.xx.yy is not reachable from xx.xx.xx.qq using UDP on port 3343.

Network interfaces Server3 (Secondary) - SAN_Team - VLAN 20 and Server4 (DR) - SAN_Team are on the same cluster network, yet address xx.xx.xx.yy is not reachable from xx.xx.xx.zz using UDP on port 3343.

--

Under System COnfiguration, I see the following error messages in Red:

Validate Active Directory Configuration

The servers are not all in the same organizational unit. Connectivity to a writable domain controller from node Server4 (DR) could not be determined because of this error: Could not get domain controller name from machine Server4 (DR).

Connectivity to a writable domain controller from node Server1 (Primary) could not be determined because of this error: Could not get domain controller name from machine Server1 (Primary).

Connectivity to a writable domain controller from node Server2 (Secondary) could not be determined because of this error: Could not get domain controller name from machine Server2 (Secondary).

Connectivity to a writable domain controller from node Server3 (Secondary) could not be determined because of this error: Could not get domain controller name from machine Server3 (Secondary).

Node(s) Server4 (DR) Server1 (Primary) Server2 (Secondary) Server3 (Secondary) cannot reach a writable domain controller. Please check connectivity of these nodes to the domain controllers.

  • Is there a full cluster validation report that shows a good configuration? – Shawn Melton Dec 11 '14 at 0:34
  • Hi Shawn, thanks for the reply. I have updated the post with list of errors from Cluster Validation Report. Everything else looks fine. – Raj Dec 11 '14 at 1:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.