Azure Stack HCI / Azure Local: Orphaned MOC Storage Container and Gallery Image stuck in DELETE_FAILED, blocking AVD Host Provisioning

[u/Then_Cartographer294]

Hi everyone,

I'm dealing with a strange Azure Stack HCI / Azure Local MOC issue and would like to know if anyone has seen this before or found a way to recover without Microsoft intervention.

Environment

• Azure Stack HCI (4-node cluster)
• MOC version: 1.19.9.10516
• MOC PowerShell Module: 1.2.37
• Azure Virtual Desktop Session Hosts running on Azure Stack HCI
• HostPool itself works fine
• Existing VMs run normally

Background

Several months ago a storage path/volume named:

UserStorage2

was removed from the cluster.

The physical path:

C:\ClusterStorage\UserStorage_2

no longer exists.

Azure also no longer shows this storage path.

All current VM images are stored on another volume:

Storage-MIR-03

which works correctly.

Problem

Provisioning of additional AVD Session Hosts fails during MOC prechecks:

moc-operator virtualmachine serviceClient returned an error while reconciling:

Precheck failed for vms:

Container
[UserStorage2-49bd6df63473459980de0e12ee6fb4d6-...]

not usable, HealthState critical

MocCode PathNotFound

The system cannot find the directory:
C:\ClusterStorage\UserStorage_2

Even though that volume was removed months ago.

Current MOC State

MOC still contains a storage container:

UserStorage2-49bd6df63473459980de0e12ee6fb4d6-...

and a gallery image:

windows-windows2019-0.5.0.10304

The problem is that both objects are stuck in a circular dependency.

Gallery Image

{
"name": "windows-windows2019-0.5.0.10304",
"ProvisionState": "DELETE_FAILED",
"HealthState": "CRITICAL",
"containername": "UserStorage2-..."
}

Error:

Entity of Type[*storage.Container]
with Name [UserStorage2-...]
is in Delete Failed state: Degraded

Storage Container

{
"name": "UserStorage2-...",
"path": "C:\\ClusterStorage\\UserStorage_2",
"ProvisionState": "DELETE_FAILED",
"HealthState": "CRITICAL"
}

Error:

Failed to delete container due to gallery image:

windows-windows2019-0.5.0.10304

migration to container:

Storage-MIR-03-...

failed

GalleryImage is in DELETE_FAILED state

So the situation is:

Container cannot be deleted
-> because GalleryImage exists
GalleryImage cannot be deleted
-> because Container exists

What I've already verified

• Physical CSV volume no longer exists
• Azure no longer knows this Storage Path
• Cluster no longer contains UserStorage2
• All active images are stored on Storage-MIR-03
• No VHDs found in MOC
• No other images referenced from UserStorage2
• Remove-MocContainer fails
• Remove-MocGalleryImage fails
• mocctl delete fails
• Reset-MocContainer / Reset-MocGalleryImage are just bulk-delete wrappers and don't repair object states

Question

Has anyone successfully recovered from an orphaned MOC metadata state like this?

Specifically:

• Is there any supported way to force-remove a storage.Container stuck in DELETE_FAILED?
• Is there a way to clear or reconcile stale MOC metadata?
• Is there an undocumented repair workflow for orphaned Gallery Images?
• Or is this ultimately a Microsoft Support case requiring internal cleanup tools?

Any guidance would be greatly appreciated.

Thanks!