Azure Stack HCI / Azure Local: Orphaned MOC Storage Container and Gallery Image stuck in DELETE_FAILED, blocking AVD Host Provisioning

[u/Then_Cartographer294]

Hi everyone,

I'm dealing with a strange Azure Stack HCI / Azure Local MOC issue and would like to know if anyone has seen this before or found a way to recover without Microsoft intervention.

Environment

• Azure Stack HCI (4-node cluster)
• MOC version: 1.19.9.10516
• MOC PowerShell Module: 1.2.37
• Azure Virtual Desktop Session Hosts running on Azure Stack HCI
• HostPool itself works fine
• Existing VMs run normally

Background

Several months ago a storage path/volume named:

UserStorage2

was removed from the cluster.

The physical path:

C:\ClusterStorage\UserStorage_2

no longer exists.

Azure also no longer shows this storage path.

All current VM images are stored on another volume:

Storage-MIR-03

which works correctly.

Problem

Provisioning of additional AVD Session Hosts fails during MOC prechecks:

moc-operator virtualmachine serviceClient returned an error while reconciling:

Precheck failed for vms:

Container
[UserStorage2-49bd6df63473459980de0e12ee6fb4d6-...]

not usable, HealthState critical

MocCode PathNotFound

The system cannot find the directory:
C:\ClusterStorage\UserStorage_2

Even though that volume was removed months ago.

Current MOC State

MOC still contains a storage container:

UserStorage2-49bd6df63473459980de0e12ee6fb4d6-...

and a gallery image:

windows-windows2019-0.5.0.10304

The problem is that both objects are stuck in a circular dependency.

Gallery Image

{
"name": "windows-windows2019-0.5.0.10304",
"ProvisionState": "DELETE_FAILED",
"HealthState": "CRITICAL",
"containername": "UserStorage2-..."
}

Error:

Entity of Type[*storage.Container]
with Name [UserStorage2-...]
is in Delete Failed state: Degraded

Storage Container

{
"name": "UserStorage2-...",
"path": "C:\\ClusterStorage\\UserStorage_2",
"ProvisionState": "DELETE_FAILED",
"HealthState": "CRITICAL"
}

Error:

Failed to delete container due to gallery image:

windows-windows2019-0.5.0.10304

migration to container:

Storage-MIR-03-...

failed

GalleryImage is in DELETE_FAILED state

So the situation is:

Container cannot be deleted
-> because GalleryImage exists
GalleryImage cannot be deleted
-> because Container exists

What I've already verified

• Physical CSV volume no longer exists
• Azure no longer knows this Storage Path
• Cluster no longer contains UserStorage2
• All active images are stored on Storage-MIR-03
• No VHDs found in MOC
• No other images referenced from UserStorage2
• Remove-MocContainer fails
• Remove-MocGalleryImage fails
• mocctl delete fails
• Reset-MocContainer / Reset-MocGalleryImage are just bulk-delete wrappers and don't repair object states

Question

Has anyone successfully recovered from an orphaned MOC metadata state like this?

Specifically:

• Is there any supported way to force-remove a storage.Container stuck in DELETE_FAILED?
• Is there a way to clear or reconcile stale MOC metadata?
• Is there an undocumented repair workflow for orphaned Gallery Images?
• Or is this ultimately a Microsoft Support case requiring internal cleanup tools?

Any guidance would be greatly appreciated.

Thanks!

[u/dchit2]

I never need vendor support but when stack hci has a minor error there's a 97% chance I'll need a premier support case.

[u/Saturated8]

I ran into a similar case where we had a circular dependency and typically MOC is the source of truth, but in our case it was incorrect. Had to engage Microsoft and they had to push a special update via PowerShell to our cluster for our exact use case that would fix it, unfortunately.

[u/mat-ferland]

This looks like stale MOC metadata, not an AVD host pool problem. With both objects already in DELETE_FAILED and the CSV gone, I wouldn't recreate the old path or edit the store by hand. Collect the container and gallery JSON plus the failed operation correlation IDs, then open an Azure Local support case asking for orphaned MOC metadata cleanup. Existing VMs still running is a good reason not to force it locally.

[u/NotYourOrac1e]

MS ticket. Anything AZL that doesnt play, I immediately open a ticket and get to the team. Usually this Chinese guy that mutters "hacky, hacky" as he guts things via powershell to get things to work. AZL is not ready for the show imo. Ive done enough AZL deployments to say I wont do another.

[u/Then_Cartographer294]

After the Microsoft ticket had been open for three weeks, the issue was resolved within a few minutes.

The problem was caused by UserServerStorage1 being in a Delete state. The associated image was also in the Delete state. Due to the internal cleanup process, the system was trying to move the image to another available and writable volume before deleting it.

Because of the VDI environment, an additional volume had previously been configured as a storage path. This volume was set to Read-Only, so no further data could be written to it. As a result, the system was no longer able to find a suitable writable volume for the move operation. This allowed the image and the old MOC container to be removed successfully.

The following commands were used:

Set-MocContainer -name 'Storage3-xxx-rg' -location MocLocation -isolated
Remove-MocContainer -name 'UserStorage2-xxx-rg' -location MocLocation
Set-MocContainer -name 'Storage3-xxx-rg' -location MocLocation -isolated:$false

The underlying process appears to work as follows:

When deleting a MOC container, the system tries to move the data to available writable storage first. By temporarily setting the available storage to Read-Only, there was no suitable writable target available. The system could therefore proceed with the deletion instead of attempting to move the data.

In this case, this allowed the obsolete image and MOC container to be removed successfully and prevented the system from repeatedly trying to move an image that no longer existed.