1

We are trying to build a multi-subnet cluster.

VM1 - 172.16.0.11 and 172.16.0.12
VM2 - 172.16.0.22 and 172.16.0.23
VM3 - 192.168.0.111 and 192.168.0.112

Windows Failover Cluster - WINCLUSTER - 172.16.0.112 and 192.168.0.113
AG Listener - AGLSNR - 172.16.0.213 and 192.168.0.213

We were able to create cluster between VM1 and VM2, however we are not able to add the 3rd node(VM3) from another subnet, as the cluster log fails like this (it's a lengthy log file, not sure what to paste here, posted which I see as alarming)

[=== Microsoft-Windows-FailoverClustering/Operational logs ===]
INFO Cluster has lost the UDP connection from local endpoint 172.16.0.11:~3343~ connected to remote endpoint 192.168.0.112:~3343~.
INFO  Cluster network 'Cluster Network 1' was added to the failover cluster.
INFO  Cluster network interface 'VM1 - Ethernet' was added for node 'VM1' on network 'Cluster Network 1'.
INFO  Cluster network interface 'VM1 - Ethernet 2' was added for node 'VM1' on network 'Cluster Network 1'.
INFO  Cluster has established a UDP connection from local endpoint 172.16.0.11:~3343~ connected to remote endpoint 172.16.0.23:~3343~.
INFO  Cluster has established a UDP connection from local endpoint 172.16.0.12:~3343~ connected to remote endpoint 172.16.0.23:~3343~.
INFO  Cluster has established a UDP connection from local endpoint 172.16.0.12:~3343~ connected to remote endpoint 172.16.0.22:~3343~.
INFO  Node 'VM1' successfully became part of a failover cluster and detected that its cluster service software has been updated.
INFO  Cluster network interface 'VM1 - Ethernet' for cluster node 'VM1' on network 'Cluster Network 1' is operational (up). The node can communicate with all other available failover cluster nodes on the network.
INFO  Cluster network interface 'VM1 - Ethernet 2' for cluster node 'VM1' on network 'Cluster Network 1' is operational (up). The node can communicate with all other available failover cluster nodes on the network.
INFO  This node has successfully joined the failover cluster 'WINCLUSTER'.
INFO  A new node, VM3, has been added to the failover cluster.
INFO  Cluster has lost the UDP connection from local endpoint 172.16.0.11:~3343~ connected to remote endpoint 192.168.0.112:~3343~.

mscs::ListenerWorker::operator (): (1460)' because of '[FTI][Initiator] Aborting connection because NetFT route to node VM3 on virtual IP fe80::d9b0:983b:dd60:a635:~9547~ has failed to come up.'

I was able to ping the VM3 from VM1 and VM2. Is the cluster trying to connect over IPv6? It's not using IPv4? Please let me know, what else I can provide from the log?

Improve this question
6
  • Unfortunately that's not the problem, just one of many errors I'm sure exist in the cluster log. Internally with clustering, items such as netft do use ipv6 and tunnel over ipv4. In any case you'll need to add more log than that, and yes I know it can be quite large. Might make sense to put it somewhere we can see it. Make sure to replace names or other pii before doing so.
    – Sean Gallardy
    Commented Mar 1, 2023 at 1:10
  • Thanks Sean...added more log
    – Kris
    Commented Mar 1, 2023 at 3:58
  • Unfortunately that's not really enough to help. You can see that VM3 apparently is added but we have no idea what steps it went through. Since each node has a copy of its local log, you'll need to go between the logs to see where it's having an issue. Setting the log to debug (level 5) can help but will add to the size.
    – Sean Gallardy
    Commented Mar 1, 2023 at 14:20
  • @SeanGallardy Thanks for your inputs. We tried some thing yesterday night, changed the IP addresses to be in same subnet as like VM1 and VM2 and was able to add VM3. However when we change adapter IP's back to different subnet, that node went down (obviously) and we can't add it back. Our n/w team says as IPv6 traffic is not allowed across subnets, hence the reason it is failing. Also, how and where to upload the log file. Thanks again.
    – Kris
    Commented Mar 1, 2023 at 18:23
  • If IPv6 isn't allowed (which is weird, tbh) then disable it on those interfaces, should tunnel over IPv4. The logs, no clue where you'd want to put them, sorry. something like GitHub or anywhere you can host text files I guess - the hard part being anyone on the internet can see it... if you do this make sure to obfuscate sensitive data.
    – Sean Gallardy
    Commented Mar 2, 2023 at 15:14

1 Answer 1

Reset to default
1

Their analysis - Indicated multiple Virtual Machine (VM) Switches that are not using VLAN tagging. Each VM Switch represents a different subnet, your company is using VMs on the same host and changing the NICs to use a different VM Switch to have each node in a different subnet. And machine seems trying to use IPv6 in the cluster despite it being disabled and the UDP not being routable in this environment.

After hours and hours of troubleshooting, providing n/w traces, capturing Hyper-V data, wireshark and cluster logs - with no results. So, we wanted to create the VM's on VMWare and it worked. They still did not know, why it failed in HyperV.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.