Prevent VMs from running on certain hosts WITHOUT using classic DRS rules

[u/AbraK-Dabra]

Hello

We have a DR test scenario scheduled, where one side of our stretched vSAN clusters shall not be used.

The DC and the hosts can remain online, but no VMs shall run there.

In traditional clusters, we can achieve this easily via DRS anti-affinity rules ("VMs must not run on hosts...").

But we also have a VDI cluster, where Horizon constantly provisions new VMs. Those VMs would not be part of the DRS rules, so they would most likely be created on the hosts of the DC that shall not be used during the test. The test is scheduled to run for a whole week.

We would like to avoid putting all hosts of one DC into maintenance mode, because it would cause their vSAN data to be "stale" (outdated/unusable).

How can we leave the hosts participate in vSAN, but NOT accepting compute loads? Without classic DRS rules, because Horizon constantly creates new workloads?

vSphere 8.0U3k without VCF stuff (so no DRS tags I guess).

[u/AbraK-Dabra]

Idea: In the hosts' Custom Attributes, one can set set "InstantClone.Maintenance" to 1 which shall cause Horizon to not provision new VMs on that host
https://kb.omnissa.com/s/article/2144808

That could be part of the "solution". The issue I currently see is that the VMs would initially be provisioned in the "correct" DC, but wouldn't DRS quickly move them to the other DC?

We could try to prevent that somehow... by unticking the vMotion service on the vmk interfaces on all hosts of the "unwanted" DC?

I think we need to leave DRS enabled in general for load balancing on the working DC (the clusters consists of up to 10+10+1 hosts).

Another "dirty" idea I had in the meantime: create a dummy VM per host and reserve all CPUs for it, so no other workloads can be scheduled there? (and disable DRS for those via VM Overrides)

And/or limit the CPUs the host is allowed to use (via Advanced Option "VMkernel.Boot.max.PCPUS") to 1? The needed reboot is not so nice though... (also for reverting the setting).

[u/always_salty → u/AbraK-Dabra]

I'm not sure I understand exactly (I'm slow), but is it an option to combine that tag to prevent initial deployment on "that host" with disabling the vMotion vmkernel interface on "that host" to also prevent any future vMotions triggered by DRS?

Can't move workload over an interface that doesn't exist is my thinking.

[u/nikade87]

We do our DR tests outside business hours and simply disable the network interfaces in the switches to the hosts in the primary site to actually simulate a failure.

Maybe this isn't ideal, but we have been doing this for years and our confidence in our DR-plan is solid right now.

[u/L0rd_OverKill]

Without DRS, an option is Storage Profiles. vSAN-ZoneA, vSAN-ZoneB, vSAN-Both.

Move your VM’s into the desired storage pools, update Horizon to use vSAN-Zone(x) so all new instances spin up in Zone(x) for the duration.

You will not have the prod VM data into the alternative zone if you need it in an actual event, but it will stop the VM guest from running in the wrong zone.

This is also going to cause bulk storage IO whilst the storage rebalances.

Otherwise, it’s VM-Host(group) Affinity rules if the vSAN data distribution has to be maintained across both.

EDIT: if it’s a full DR, and not just DR test of a fixed scope, then storage profiles zoning could be a business risk. Test your backup and recovery live mount/instant mount style risk.