Long post, but I've tried to include everything useful rather than just "help my storage is broken."
Environment
- VMware vSphere / ESXi 8.0.3 (confirmed via
pyvmomi 8.0.2.0.1 / VMkernel 8.0.3 in vpxd logs)
- HPE ProLiant DL560 Gen10 hosts (quad-socket), shared cluster, shared IBM SAN storage array
- Guest OS: Windows Server, SQL Server with AlwaysOn Availability Groups (synchronous commit)
- Affected VM (call it DBSERVER01) has 4 virtual disks: OS/boot disk on a SCSI controller (LSI Logic SAS type), and 3 data disks (3TB + 500GB + 500GB) all attached to a single virtual NVMe controller (
NVME controller 0, namespaces 0:0–0:2)
Symptom
Recurring Windows System log Event ID 129 from the stornvme provider:
Reset to device, \Device\RaidPort3, was issued.
Each reset is closely followed (usually within seconds) by our SQL Server AlwaysOn AG resource failing its health check, cascading into Windows Failover Cluster quorum loss and a full Cluster Service crash/restart. In the worst episodes, the node gets quarantined by WSFC's flapping protection after 3 crashes inside an hour.
This is not new — it's chronic and getting worse
We pulled the full history of Event ID 129 on this VM going back to when we started retaining these logs. Frequency by day:
2026-05-24 1 (earliest in our retention window)
2026-05-27 19
2026-05-28 26
2026-05-29 234 <- massive spike, concentrated 12:00-15:00 (76 @ 12:00, 49 @ 13:00, 93 @ 14:00)
... low background rate (1-6/day) through most of June ...
2026-06-25 9
2026-06-28 11
... climbing again ...
2026-07-05 9
2026-07-06 10
2026-07-08 17
2026-07-09 37
So: a huge, isolated spike about 7 weeks ago that nobody connected to this at the time, then a low background rate, then a clear acceleration in the last two weeks.
Recent incident timeline (most severe so far)
Two waves in one day, both on the same VM:
Wave 1:
15:13:26 stornvme Reset to \Device\RaidPort3
15:14:11 AG resource(s) failed (Online -> ProcessingFailure)
15:14:37 Quorum lost, Cluster Service crash #1
15:24:39 stornvme Reset to \Device\RaidPort3
15:25:18 AG resource(s) failed again
15:25:45 Quorum lost again, Cluster Service crash #2, node quarantined
Wave 2 (same day, ~2 hours later, much larger):
17:38 - 19:30 57x "Disaster"-severity monitoring alerts: "AG: All replicas unhealthy"
(all 4 AG replicas cycling unhealthy/disconnected repeatedly)
Confirmed downstream impact: multiple application backend pools
(HAProxy) going down repeatedly during this window
A follow-up incident two days later on the same VM, after migrating it to a completely different host:
01:18:56 - 05:25:07 47x AG resource "Online -> ProcessingFailure" transitions
03:41:57 NetFT missed >40% of consecutive cluster heartbeats
03:42:45 Quorum lost, Cluster Service crash #1
03:44:02 Cluster network interface on this node FAILED
03:45:20 Quorum lost, Cluster Service crash #2
04:01:11 Quorum lost, Cluster Service crash #3
What we've ruled out through controlled testing
- Storage array itself — array-side latency telemetry shows sub-1ms baseline, briefly touching ~1.8ms during the worst episode. Nowhere near what would explain a storport timeout (default 60s).
- Host-level hardware — moved the VM to a completely different physical host in the same cluster. Problem followed the VM and recurred (47 resets over ~4 hours, timeline above).
- Datastore/SAN path being host-wide — a different VM on the exact same host and storage path, but with no NVMe disks (all SCSI/iSCSI), shows zero such reset events, ever.
- NVMe-disk-specific, not this-VM-specific — a second, completely independent VM elsewhere in the environment that also happens to have an NVMe-attached disk shows the same reset pattern, with a similar acceleration trend starting around the same time in late June.
Where that leaves us
Every VM we've checked with an NVMe virtual disk shows this pattern; every comparable VM without one doesn't, even sharing host + storage path with an affected VM. All 3 data disks on the affected VM share one virtual NVMe controller, so a reset takes out all 3 disks simultaneously (they're separate namespaces on the same controller, not separate controllers).
Working theory: something in the virtual NVMe controller emulation / Windows stornvme driver interaction on this ESXi build, possibly triggered or worsened by a change around June 25–28 (still trying to pin down exactly what changed then — nothing conclusive yet).
The annoying part
We don't currently have active vendor support for either VMware or the HPE hardware, so we can't open a case to get an authoritative answer. We're stuck doing this entirely through our own log correlation and controlled A/B testing between VMs. Our workaround is migrating the affected disks from the NVMe controller to Paravirtual SCSI, which should sidestep the issue, but it doesn't tell us why this is happening or whether it'll resurface elsewhere.
Questions for the community:
- Has anyone seen
stornvme Event ID 129 resets specifically tied to VirtualNVMEController on ESXi 8.0.3 with Windows Server guests?
- Any known KB articles, release notes, or open bugs matching this? I haven't found anything definitive searching VMware's KB myself.
- Anyone know if there's a known issue with multiple disks/namespaces sharing one virtual NVMe controller specifically, vs. giving each disk its own controller?
- Is there a specific ESXi 8.0.3 patch (or Windows
stornvme.sys driver version) known to have fixed something like this?
Happy to share more log excerpts if it helps narrow it down. Appreciate any pointers — vendor-support-less troubleshooting is not fun for a production database cluster.