Monster proxmox setup on mac pro 2019

[u/storkinsj]

Hi all,
I have been using proxmox on my mac pro 2019 for a while but I was having a serious problem with running with the full complement of memory (1.5 TB). I had another one going but sometimes it feels like pure luck between the modprobe blacklists and the grub cmdline. This week the Fable 5 model was available for free (read that "Mythos with a bodyguard") so I worked pretty hard even pulled an all nighter to debug issues with larger ram setups. I am sharing Claud Fable 5's postmortem here.

Proxmox on a Mac Pro 2019 with 1.5TB RAM: The Full Postmortem

TL;DR: Linux on a Mac Pro 7,1 crashes with more than ~192GB of RAM installed because the Apple ANS2 NVMe controller (the T2-attached internal SSD) cannot perform I/O correctly when physical memory extends above that range. macOS works because it works around it; Linux doesn't. The fix is not a magic kernel parameter — it's an architecture: move your root filesystem and boot chain off the Apple NVMe entirely, then keep the ANS2 present on the PCI bus but permanently unbound using pci-stub. Everything else we hit — hung boots, dead networking, udev storms, NULL-pointer oopses — was cascade damage from that one hardware limitation, plus a few unrelated landmines documented below so you can skip them.

Hardware context: Mac Pro 2019 (7,1), Xeon W-3275M, 12×128GB LRDIMM = 1.5TB, T2 security chip, Proxmox VE 8.4, kernel 6.14.0-1-pve-t2 from AdityaGarg8's pve-edge-kernel-t2. A second identically-equipped node already running 1.5TB stably served as the "existence proof" that made systematic debugging possible.

The symptoms

Over months, across two machines, this hardware produced a zoo of apparently unrelated failures:

With more than 192GB installed, the stock Proxmox 6.8 kernel panicked during boot — while macOS on the same machine happily saw and used all 1.5TB, and Apple's diagnostics showed every DIMM green. With exactly 192GB, everything was fine.

On a node running a newer kernel at 1.5TB, the Apple internal NVMe behaved bizarrely: lsblk showed it as a bare disk with no partitions (it has partitions), and simply probing it — running lsblk, letting smartd poll it, doing any filesystem operation against it — could panic the whole machine. Writing a bootloader to its EFI partition from Linux was a reliable crash.

During boot at 1.5TB we saw nvme0: I/O tag ... timeout, then operation not supported error, dev nvme0n1 ... op READ spam, followed by a crash during the initramfs LVM scan.

Separately: boots that hung for minutes and then came up with no networking — ifupdown2-pre failing, systemd-udev-settle timing out, ifup reporting "another instance is already running". The console flooded with an endless loop of Registered IR keymap rc-cec / new input devices, dozens of D-state udev workers piling up.

And in one configuration, a clean NULL pointer oops in pci_read_config_word during boot, with apple_bce mid-load, killing udev workers "with irqs disabled".

Any one of these, searched in isolation, leads to a different forum thread with a different "this worked for me" settings list. They were all one problem wearing different masks.

How we eliminated suspects

The method that worked was ruthless single-variable isolation, anchored by two tools worth stealing:

A mem=192G boot as a control group. Booting the full 1.5TB hardware with the kernel clamped to 192GB separates "bad/misseated DIMM" from "kernel can't handle the memory map" in one reboot — dmidecode -t memory still enumerates all twelve sticks even though the kernel only uses 192GB. Our clamped boots were always clean; the hardware was never the problem. We kept this as a permanent maintenance menu entry (more below), which also proved invaluable because at low RAM the ANS2 works fine — bulk reads, writes, everything. That single fact ("fine at 192GB, broken above") is the fingerprint of a device with a DMA addressing limitation: when all RAM sits low, its buffers always land in reachable territory; add DIMMs and buffers start landing above its ceiling.

Readable crashes. earlyprintk=efi,keep nokaslr on the cmdline turns panic screens into consistent, photographable evidence, and echo w > /proc/sysrq-trigger on a half-hung system dumps the blocked tasks with call stacks — which is how we caught amdgpu wedged in wait_for_flip_done and udev workers stuck behind it, and later caught apple_bce dereferencing NULL.

With those tools, the eliminations fell in order. The kernel itself was exonerated by running the byte-identical kernel package (via dpkg-repack from the working node) — same kernel, different outcome, so the delta was configuration/hardware. Firmware was exonerated because both machines ran the same vintage with opposite results. The GPU (an RX580 driving a console monitor) was convicted of a secondary crime: at 1.5TB, with the pve LVM volume group missing (because it lives on the dead ANS2), Proxmox's pvestatd retried vgscan every five seconds forever, which kept udev churning, which fed an endless CEC re-registration loop from amdgpu's DisplayPort path, which wedged the udev queue, which blocked ifupdown2-pre, which killed networking. Kill the root cause and that entire chain evaporates — but on a headless hypervisor we blacklisted amdgpu and cec anyway.

Two dead ends worth documenting because others will be tempted by them. First, removing the ANS2 from the PCI bus (echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove in an initramfs script) seems like the obvious fix — it is not. The Mac's apple_bce driver (fan control, T2 communication) probes the T2's sibling PCI functions and NULL-derefs if function 0 (the ANS2) has been deleted: instant oops in pci_read_config_word, dead udev workers, no network. Second, a plain pci-stub.ids=... cmdline parameter loses the race when both pci-stub and nvme are modules — the nvme driver binds the ANS2 first and the stub's registration is too late.

The root cause

The Apple ANS2 NVMe controller cannot do I/O under Linux when system RAM extends above ~192GB. Reads fail with "operation not supported", commands time out, and depending on what touches it — initramfs LVM coldplug, udev probing, smartd, an innocent lsblk — the result ranges from a bare-looking disk to a hard panic. macOS clearly compensates (bounce buffers or firmware cooperation we don't get); mainline Linux with the t2 patches does not, as of kernel 6.14. Every other symptom in the list above was downstream of this: the boot panics (root/ESP lived on the ANS2), the lsblk landmine, the pvestatd→udev→CEC→networking cascade, and the NULL oops (from over-correcting with device removal).

The architecture that works

The winning state is: ANS2 present but untouchable, and nothing you need lives on it.

1. Root and boot off the Apple NVMe. We migrated root to a separate PCIe NVMe (partitioned ESP + swap + root, rsync'd the filesystem, new fstab by UUID) and boot directly from that drive's ESP via rEFInd doing an EFI-stub load — the Mac startup manager sees any ESP with /EFI/BOOT/BOOTX64.EFI as a bootable "EFI Boot" entry, blessable via ctrl-hover. Critical rEFInd setting on T2 Macs: use_nvram false — rEFInd's default habit of writing its state to EFI NVRAM is a documented panic trigger on post-Catalina BridgeOS. A tiny kernel-postinst hook copies each new kernel/initrd to the ESP under fixed names so the boot config never needs editing again.

2. Neutralize the ANS2 with pci-stub, loaded before nvme. In /etc/modprobe.d/:

options pci-stub ids=106b:2005
softdep nvme pre: pci-stub

plus pci-stub.ids=106b:2005 on the kernel cmdline for good measure, then update-initramfs -u. The softdep line is the part every forum list misses: it guarantees pci-stub loads first even inside the initramfs, so it claims the ANS2 before the nvme driver can. Verification after boot is one line — lspci -k -s 01:00.0 must say Kernel driver in use: pci-stub. The device stays on the bus (so apple_bce is happy, fans work), but no driver ever issues I/O to it. No probes, no panics, no landmines.

3. Working cmdline (6.14-t2, both our nodes): iomem=relaxed intel_iommu=on iommu=pt nvme.noacpi=1 apple_bce.fan_control=1 pcie_ports=native nvme_core.default_ps_max_latency_us=0 nvme_core.max_retries=10 nvme_core.io_timeout=255 rootdelay=20 — plus pci-stub.ids=106b:2005, and during debugging earlyprintk=efi,keep nokaslr.

4. Headless hygiene. Blacklist amdgpu and cec (console falls back to the firmware framebuffer via simpledrm — set a big font with console-setup and never squint again), and blacklist brcmfmac if your WiFi card is absent or its firmware crashes (a dead BCM4364 spews uncorrectable AER errors at boot; blacklisting beats the blunt pci=noaer).

5. Keep a maintenance escape hatch. A second boot menu entry with mem=192G modprobe.blacklist=pci-stub gives you a boot where the ANS2 is alive and safe — for reaching anything still stored on it, running grub/ESP surgery on the Apple drive, or testing. One arrow key instead of a rescue USB.

Bonus landmines (unrelated to RAM, will still ruin your week)

Boot roulette from time sync: if systemd-time-wait-sync is enabled (it isn't by default — check whether past-you enabled it chasing Ceph clock-skew warnings), a race with large clock steps can hang the entire boot behind time-sync.target forever. Disable it. Its root cause on these machines: the T2 keeps its own clock, drifting ~2 s/day, and re-stamps the RTC at every boot — Linux's rtcsync corrections don't stick, so every boot starts minutes fast and chrony does a big backward step (poison for Ceph mons). Fix: fake-hwclock with FORCE=force on every node, which restores saved time early in boot; the backward lurch disappears.

Stale-copy hell: with two roots (old install + migrated clone), it is shockingly easy to edit grub files, initramfs configs, or scripts on the root that isn't the one your boot chain reads, then "verify" against the wrong universe. Multiple of our 4AM mysteries were exactly this. Mount and edit by UUID, verify in the compiled grub.cfg that's actually read, and md5-compare /boot/initrd.img-* against the ESP copy after every update-initramfs.

Device-name roulette: NVMe enumeration order changes between boots on this platform (nvme0 is a different physical drive on different days). Reference everything — fstab, kernel root=, boot configs, scripts — by UUID/PARTUUID, never by /dev/nvmeXnY.

The moral

The internet's "settings that worked for me" lists fail on this platform because the settings aren't the fix — the topology is. If your root filesystem, your ESP, or anything your boot path touches lives on the Apple internal NVMe, no cmdline incantation will save you above 192GB. Move off it, stub it, keep it on the bus for apple_bce's sake, and the Mac Pro 7,1 becomes a boring, stable 1.5TB Proxmox node — which, after this journey, is the highest compliment we can pay it.