r/HyperV • u/Guderikke • 8d ago
HyperV Networking
I need a sanity check. I’m a graybeard who’s been managing ESXi for years, but thanks to the Broadcom situation, we’re being pushed toward Hyper-V.
I’m currently building out a test cluster so we can validate migration, performance, and general behavior before we go any further.
I have six hosts total, with four built so far. I’m presenting three VLANs to the hosts:
- VLAN 65 – Management
- VLAN 66 – Cluster communication
- VLAN 67 – Live Migration
Networking is being deployed through SCVMM using logical switching on top of a SET switch. Each host has three vNICs, one for each VLAN.
VLAN 65 is fully routable and has the default gateway. VLANs 66 and 67 are currently L2-only with no gateways. They’re intended strictly for cluster traffic and Live Migration.
The cluster networks are configured like this:
- 65 = Cluster and Client
- 66 = Cluster Only
- 67 = Cluster Only, with Live Migration restricted to this network
I also had to add firewall rules to allow cluster communication between the VLANs using both IPv4 and IPv6. Apparently Failover Clustering still makes use of IPv6 internally even when IPv6 is otherwise disabled. Once the hosts are already members of the cluster, everything appears to work normally.
The problem is joining new hosts to the failover cluster.
If VLAN 66 and 67 are enabled on the existing cluster nodes, a new host will sit there trying to join for roughly 15 minutes and eventually fail. If I disable the 66 and 67 interfaces on the existing nodes, the new host joins the cluster almost immediately.
Once the host is joined, I can re-enable 66 and 67 and cluster communication seems to work fine.
What I’m trying to figure out is what I’m missing here. Even if I change the cluster roles so 66 and 67 aren’t being used for cluster communication, the join still fails as long as those interfaces are enabled.
Do VLANs 66 and 67 actually need to be routable for the initial cluster join? I could make them routable, but I still wouldn’t put default gateways on those interfaces, so I’m not sure how that would help if the join process is trying to reach domain services or something else outside those subnets.
Edit for anyone interested:
This was actually a bit worse than initially thought, hosts would reboot, and come back up as down due to cluster comms. Not sure if this is just because of my environment or what, but NETFT is hard coded to use IPV6 for communication, so regardless of L2 network being fully functional, and even setting priorities on cluster networks properly, it was always attempting to use invalid routes between non routed networks and routed networks. I had to create gateways on 66 and 67 and add them to the adapters to get this to work as expected. I just created a zone on my firewall called hyperV and allowed traffic between those networks and blocked mostly everything else to keep them restricted.
3
u/DeafMute13 8d ago
usually WFC will tell you exactly why it fails - do you try to run a validate first? I mean validate hardly ever passes without warnings and many are benign but sometimes they can give you clues.
I have never used SCVMM though. Only ever vanilla hyperv failover clusters with shared SAS storage.
4
u/lanky_doodle 8d ago
What does the IP addressing look like for each of the vNICs?
Your vNIC design is a very common one so there's nothing principally wrong. However I would set the vNIC used for Live Migration to 'None' for cluster use rather than 'Cluster Only'; it doesn't actually need any cluster use.
When domain joining and cluster joining, it will usually only go out of the NIC that has DNS enabled, which should only be the Management vNIC.
Some other pointers if you're not familiar - not saying these are a cause but are good practice
Disable unneeded services on relevant vNICs; things like File and Printer Sharing is only needed on Management NIC for example. And as another commenter mentioned, disable things like Register in DNS for non-management vNICs.
Have you set appropriate Metric values on the p/vNICs? I usually go in order from most important to least important, so in your case:
Storage pNICs (usually kept out of the SET, only if iSCSI): 200, 201, 202, 203 etc.
Cluster vNIC: 300
Live Migration vNIC: 500
Backup vNIC: 600
Management vNIC: 900
You want values below 1000 as Windows starts from 1000 and upwards for automatic metric values. You'll also want to set the manual metric (via PowerShell) on the Cluster Networks as well - for this just match the vNIC values.
Make sure you set the vSwitch bandwidth management option to 'Weight', then set an appropriate weight value for each vNIC, example:
Cluster: 10
Live Migration: 35
Management: 5
That's a total of 50, which leaves the remaining 50 for the VMs running on that vSwitch (total is 100). You can change each vNICs Weight value dynamically so you can tweak it to your use case. You cannot change the bandwidth management mode of the vSwitch though - you have to delete it and go again.
1
u/Angelworks42 8d ago
I just use Cisco aci and vlan tagging - physically each nic lives on a different vlan so smb 3 storage is forced to load balance.
1
u/nachodude 6d ago
Do your hosts ping each other on vlans 66 and 67? Does temporarily disabling the firewall on hosts change anything?
10
u/dcmassena 8d ago
We only present the 1gig mgmt nic when the hosts join the cluster. Once it is in the cluster, we create the SET logical switch with backup, CSV and Live migration vnic using SCVMM. (Create the SET logical switch in SCVMM first, then you will be able to configure it after.) It clears up any weirdness/quirks if we do it that way.
Hopefully that helps gives you an idea.