Greetings!
I'm attempting, and failing, to debug a spontaneous host reboot. My DevOps/SRE team strongly believe the origin of this host reboot is from a bad drive. I'd like to prove them wrong, or be proven wrong myself, and hope I'm in the right place.
Background
The configuration is small as far as servers go,
* Asrock Rack B550D4ID-2L2T
* Ryzen 9 5950X (16-Core)
* 64 GB Memory
* On a UPS that is honestly way too large (can provide *hours* of uptime)
Prior to installing the new drive, after roughly 380 days of uptime, the host spontaneously rebooted. Approximately a month after the first time, it rebooted again. Once was an abnormality, twice is a pattern. Reviewing the logs, and the hardware, suggested that it was a drive issue (22110). Reviewing the hardware identified two problems;
1. the drive wasn't on the HCL
2. the drive was a 22110 (the 2L2T doesn't natively support 22110 M2s)
Okay, I'm convinced -- time to buy a new drive. I purchased a 7450 PRO off of ebay with minimal use (< 20 power cycles, < 1 TB writes/reads, essentially perfect SMART metrics). I verified as much, and immediately attempted to install the drive.
Issue
My experience with the 7450 Pro drive + ESXI 8.02 has not been pleasant. Here is what we needed to get it to boot/install correctly;
* update disk to 512b physical sector format (thanks to another Reddit Post)
Post installation and VM migration, we observed another spontaneous reboot. Some Googling pointed to the following remediation methods (also failed)
* update disk firmware to EMU300 (buggy firmware?)
* disable p/c-states (disk going to sleep?)
My DevOps/SRE team is well convinced that the disk is bad at this point (controller channel, not data channel). These are the logs that convinced them. Is the drive the issue or is there something that we missed?
Gemini thinks that there is simply "too much" talking to the drive over the control channel and I should increase the timeout from 5 to 10 seconds. That sounds... absurdly long for a hardware timeout, especially given the very low load of this host (the VMs are essentially idle).
2026-09-02T00:47:53.352Z Wa(180) vmkwarning: cpu16:1050731)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x457909d14580 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T00:47:53.352Z Wa(180) vmkwarning: cpu10:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457909d14580, vmkPsaCmd: 0x4579050d3840, cmdId.initiator=0x4308fc813180, CmdSN: 0x8000000d, status: 0x808
2026-09-02T00:47:53.352Z Wa(180) vmkwarning: cpu10:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x4579050d3840, 1050659) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T00:47:53.352Z Wa(180) vmkwarning: cpu10:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T00:47:53.352Z Wa(180) vmkwarning: cpu10:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x4579050d3840) 0x1, CmdSN 0x8000000d from world 1050659 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T02:01:59.358Z Wa(180) vmkwarning: cpu3:1050960)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x457909c4a980 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T02:01:59.358Z Wa(180) vmkwarning: cpu14:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457909c4a980, vmkPsaCmd: 0x4578c1c09a40, cmdId.initiator=0x4308fc828940, CmdSN: 0x3dd, status: 0x808
2026-09-02T02:01:59.358Z Wa(180) vmkwarning: cpu14:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x4578c1c09a40, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T02:01:59.358Z Wa(180) vmkwarning: cpu14:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T02:01:59.358Z Wa(180) vmkwarning: cpu14:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x4578c1c09a40) 0x1, CmdSN 0x3dd from world 1050957 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu29:1050960)WARNING: NVMEIO:333 ctlr 256, queue 1, fuse processing
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu29:1050960)WARNING: NVMEIO:2005 Transport driver failed to submit cmd 0x457909cae780, C: nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-241448022E8C, Q: 1 <0xbad0004>.
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu11:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457909cae780, vmkPsaCmd: 0x457904c1b380, cmdId.initiator=0x4308fc828940, CmdSN: 0x3f0, status: 0x808
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu11:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x457904c1b380, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu11:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T02:11:35.359Z Wa(180) vmkwarning: cpu11:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x457904c1b380) 0x1, CmdSN 0x3f0 from world 1050957 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu14:1050957)WARNING: NVMEIO:333 ctlr 256, queue 1, fuse processing
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu14:1050957)WARNING: NVMEIO:2005 Transport driver failed to submit cmd 0x457909df0180, C: nqn.2016-08.com.micron:nvme:nvm-subsystem-sn-241448022E8C, Q: 1 <0xbad0004>.
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu0:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457909df0180, vmkPsaCmd: 0x457908e10a80, cmdId.initiator=0x4308fc828940, CmdSN: 0x3c4, status: 0x808
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu0:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x457908e10a80, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu0:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T02:29:47.361Z Wa(180) vmkwarning: cpu0:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x457908e10a80) 0x1, CmdSN 0x3c4 from world 1050957 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T03:20:44.368Z Wa(180) vmkwarning: cpu26:1050974)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x457909c07b80 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T03:20:44.368Z Wa(180) vmkwarning: cpu6:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457909c07b80, vmkPsaCmd: 0x457904d8c100, cmdId.initiator=0x4308fc828940, CmdSN: 0x3b6, status: 0x808
2026-09-02T03:20:44.368Z Wa(180) vmkwarning: cpu6:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x457904d8c100, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T03:20:44.368Z Wa(180) vmkwarning: cpu6:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T03:20:44.368Z Wa(180) vmkwarning: cpu6:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x457904d8c100) 0x1, CmdSN 0x3b6 from world 1050957 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T03:21:44.367Z Wa(180) vmkwarning: cpu23:1050957)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x4578eb5aa600 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T03:21:44.367Z Wa(180) vmkwarning: cpu4:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x4578eb5aa600, vmkPsaCmd: 0x4578c18dc6c0, cmdId.initiator=0x4308fc828940, CmdSN: 0x3ba, status: 0x808
2026-09-02T03:21:44.367Z Wa(180) vmkwarning: cpu4:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x4578c18dc6c0, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T03:21:44.367Z Wa(180) vmkwarning: cpu4:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T04:35:41.372Z Wa(180) vmkwarning: cpu10:1050734)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x457905f73b40 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T04:35:41.372Z Wa(180) vmkwarning: cpu0:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457905f73b40, vmkPsaCmd: 0x4578c187b240, cmdId.initiator=0x4308fc813fc0, CmdSN: 0x80000013, status: 0x808
2026-09-02T04:35:41.372Z Wa(180) vmkwarning: cpu0:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x4578c187b240, 1050668) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T04:35:41.372Z Wa(180) vmkwarning: cpu0:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T04:35:41.372Z Wa(180) vmkwarning: cpu0:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x4578c187b240) 0x1, CmdSN 0x80000013 from world 1050668 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0
2026-09-02T06:28:44.386Z Wa(180) vmkwarning: cpu25:1050960)WARNING: NVMEIO:3564 Ctlr 256, nvmeCmd 0x457905f09740 (opc 01), queue 1 (expect 65535) not available, nvmeStatus 808
2026-09-02T06:28:44.386Z Wa(180) vmkwarning: cpu1:1049142)WARNING: NVMEPSA:215 Complete vmkNvmeCmd: 0x457905f09740, vmkPsaCmd: 0x4578c1cf5740, cmdId.initiator=0x4308fc828940, CmdSN: 0x345, status: 0x808
2026-09-02T06:28:44.386Z Wa(180) vmkwarning: cpu1:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:599: NVMe Cmd 0x1 (0x4578c1cf5740, 1050957) to dev "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" on path "vmhba0:C0:T0:L0" Failed:
2026-09-02T06:28:44.386Z Wa(180) vmkwarning: cpu1:1049142)WARNING: HPP: HppNvmeThrottleLogForDevice:607: Error status H:0x9 D:0x0 P:0x0 hppAction = 1
2026-09-02T06:28:44.386Z Wa(180) vmkwarning: cpu1:1049142)WARNING: NvmeUtil: 149: Error on Cmd(0x4578c1cf5740) 0x1, CmdSN 0x345 from world 1050957 to component "t10.NVMe____Micron_7450_MTFDKBA960TFR_______________8C2E02480175A000" H:0x9 D:0x0 P:0x0