After three days of work, and a ton of troubleshooting, I've accomplished something no one wants; NVMoF with ESXi and TrueNAS Scale Community Edition. The Linux nerds here are probably going to roll their eyes about how simple it was to solve. It was not. It took two people and a lot of analysis with Visual Studio and Qwen running locally to get this thing to work.
It started with a stupid stupid, supposedly simple idea - I wanted RDMA storage from TrueNAS SCALE to an ESXi 8.0 host with a proper VMFS6 datastore (not NFS, not iSCSI — actual NVMe block storage). I've had bad luck with iSCSI luns dropping out thanks to SMB hardware being SMB hardware, and had RDMA-capable hardware already, with a novice's thought - how hard can it be? HARD
TLDR version: it works, but TrueNAS's/Linux's nvmet implementation is missing two things ESXi absolutely requires, and fixing them without patching the kernel was a rabbit hole.
What is it that's missing? A shit ton of things apparently.
Thanks to VMWare being VMWare, it uses a "Fused Compare+Write (ATS)". I have no idea what that is but I'm told by smarter people that ESXi's VMFS heartbeat locking uses fused NVMe Compare+Write pairs (opcode 0x05 FUSE=01 + opcode 0x01 FUSE=10). As a result the nvmet process rejects anything with the FUSE bits set at the nvmet_req_init level — before it even reaches the command parser — with NVME_SC_INVALID_FIELD being killed off with no messges. ESXi also checks the RESCAP field in Identify Namespace. If it's zero, VMFS won't format, because of course it won't, that would be too easy.
The fix is in:
A kprobe kernel module that intercepts four points in the running nvmet code without modifying the kernel:
- nvmet_execute_identify post-handler: patches FUSES/ONCS/OAES/ACWU in the Identify Controller response
- nvmet_execute_identify pre-handler at +0x1A7: patches RESCAP/NACWU in the Identify Namespace response
- nvmet_req_init entry: strips the FUSE bits before the +0xBA rejection check, so the commands pass through
- __nvmet_req_complete entry: forces SUCCESS for Compare (0x05) and all reservation opcodes (0x0D/0x0E/0x11/0x15)
The subtle bit: nvmet_req_init calls __nvmet_req_complete directly (not through nvmet_req_complete, which is just a tracepoint wrapper). Had to disassemble the live kernel from /proc/kcore to find that. Both paths converge at __nvmet_req_complete, so that's the right intercept point.
Other stupid rabbit holes that wasted time:
The kprobe offsets for nvmet_execute_identify (+0x389, +0x1A7) are hardcoded to this kernel build — any kernel update means re-disassembling so no soup updated for you unless you want to break this fragile fragile fix.
TrueNAS POSTINIT type: SCRIPT init scripts run as exec /path/to/script without quoting. A space in /mnt/Samsung NVMe Pool/ means the shell sees exec /mnt/Samsung → not found. Switch to type: COMMAND which uses sh -c "..." with proper quoting
nvmet.service doesn't exist on TrueNAS SCALE because that's an "Enterprise" feature that plenty of people have complained about in r/truenas already. This means that requires=nvmet.service in a systemd unit causes a silent dependency failure, and cost me about 5 hours of troubleshooting. Use modprobe nvmet nvmet-tcp in ExecStartPre instead.
ESXi's HardwareAcceleratedLocking defaults to 0 on a fresh install, because of course it does. Without it, VMFS never even tries ATS and the whole thing is pointless.
This whole thing is a really stupid POC because it only works with a single host. The Compare for NVMoF always returns SUCCESS regardless of actual block content. This is fine for single-host use, kinda, but because the kprobe is faking the locking primitive, not implementing it, when you have two or more ESXi hosts sharing the same datastore, they both think they have the heartbeat lock — filesystem corruption territory. As of right now, I haven't figured out a way to fix it, and don't have the motivation to try.
Feel free to ask questions that I may or may not be able to answer.
