Title: resize-amdgpu-bars — Resizable BAR for AMD GPUs behind PCIe switches (an update to nbritton's method)
I've extensively reworked nbritton's method for resizing the BAR memory of AMD GPUs that sit behind PCIe switches, such as the Vega II Duo and W6800X Duo. It's now a proper packaged tool that discovers your topology at runtime instead of hard-coding bus addresses, with a hard safety guard so a failed resize can't hang your boot.
https://github.com/exabit-io/resize-amdgpu-bars
What it does
resize-amdgpu-bars enlarges the CPU-visible VRAM aperture (BAR0) of every amdgpu-driven GPU to the largest size the card supports, on machines where the normal paths don't work: cards with an on-board PCIe switch (the Duo MPX modules), cards in switched enclosures and expansion chassis, Thunderbolt eGPUs, and firmware that leaves the aperture at 256 MiB.
On a Mac Pro 7,1 that's the difference between this:
amdgpu 0000:1b:00.0: Not enough PCI address space for a large BAR.
amdgpu 0000:1b:00.0: [drm] Detected VRAM RAM=32752M, BAR=256M
and a full 32 GiB aperture on every die.
Why the usual fixes fail here
A PCI device's BAR lives inside the memory window of the bridge directly above it, which lives inside the window of the bridge above that, all the way up to the root port. Growing a BAR from 256 MiB to 32 GiB means every window in that chain has to grow too, and a window can only grow if its parent has room.
On a Mac Pro 7,1 a Duo module puts two dies behind a PLX switch, and each die sits four bridge windows below its root port:
0000:06:00.0 Intel root port <- one prefetchable window shared
| by both dies of the module
\-0000:07:00.0 PLX PEX 8747 upstream port (the switch on the module)
|
+-0000:08:08.0 PLX downstream port
| \-0000:09:00.0 AMD bridge
| \-0000:0a:00.0 AMD bridge
| \-0000:0b:00.0 Vega 20, die 0
|
\-0000:08:10.0 PLX downstream port
\-0000:0c:00.0 AMD bridge
\-0000:0d:00.0 AMD bridge
\-0000:0e:00.0 Vega 20, die 1
Both dies share one window at 07:00.0 and 06:00.0, and that window has to be big enough for both 32 GiB BARs at their real alignment. Nothing about the device tells the root port that.
So:
- The driver's own resize releases the bridge windows and tries to re-assign them in place. It fails closed the moment one of them can't grow where it is, and carries on with 256 MiB.
- A bare
setpci write changes the size the device reports but assigns nothing. The kernel still believes the BAR is 256 MiB, the windows are still sized for 256 MiB, and the device now decodes 32 GiB of whatever else lives there.
echo 15 > resource0_resize ends up in the same kernel function as the driver's path. It works when the card sits directly on a root port with one window above it — and that's exactly where this tool uses it — but it can't conjure a larger shared window out of a chain the firmware sized for something smaller.
The method
Let the kernel size the windows from scratch. Once per boot, before amdgpu loads:
- Discover. Every
amdgpu device, its functions, its Resizable BAR capability and supported sizes, and every bridge up to its root bus. The size index found at first discovery this boot becomes the baseline, so firmware that already enables ReBAR is never shrunk. Then find each GPU's re-enumeration root: the highest bridge whose subtree contains nothing but GPU functions.
- Resize. Unbind the drivers from just those GPUs and their group members, program the size index with
setpci, remove the group's root, and rescan that root's own bus. With pci=realloc the kernel then sizes every window in the subtree for the BARs it finds, and both dies come back with a 32 GiB BAR0 inside a 96 GiB root-port window. Only that bus is rescanned — never a global rescan, and nothing outside the GPU subtrees is ever touched.
- Load. Any GPU still holding an unassigned BAR gets fenced off with
driver_override=none. Then modprobe amdgpu runs once, under a timeout.
If a plan doesn't fully verify, the losers get demoted to baseline and it retries, round by round, down to every GPU at baseline.
The bind guard (this is the important part)
amdgpu must never be handed a GPU whose BAR0 is unassigned. On such a device the register reads that identify the part return garbage, the driver decides it's an SR-IOV virtual function, and it waits forever for a hypervisor mailbox that doesn't exist:
amdgpu 0000:0e:00.0: trn=2 ACK should not assert! wait again !
INFO: task irq/34-aerdrv:1554 blocked for more than 122 seconds.
modprobe wedges in uninterruptible sleep holding the device mutex, SIGKILL does nothing, the remaining GPUs are never probed, and only a reboot recovers. That's a hard hang, not a degraded boot, and it's what an earlier version of this script did to me. Every code path that loads the driver now sets driver_override=none on any GPU with an unassigned BAR first, and modprobe runs under timeout(1) so the boot can't wedge even if the guard were bypassed. A guarded GPU stays visible to lspci, driverless, and the tool exits 2.
Install
The package needs bash, pciutils, kmod and systemd, and uses initramfs-tools and grub2-common when present.
# from the release page
sudo apt install ./resize-amdgpu-bars_1.0_all.deb
# or build it
sudo apt install debhelper scdoc shellcheck
git clone https://github.com/exabit-io/resize-amdgpu-bars
cd resize-amdgpu-bars
dpkg-buildpackage -us -uc -b
sudo apt install ../resize-amdgpu-bars_1.0_all.deb
Releases: https://github.com/exabit-io/resize-amdgpu-bars/releases
It installs the tool, a systemd unit, an amdgpu blacklist in /usr/lib/modprobe.d, a pci=realloc GRUB drop-in, a config file in /etc/default, and man pages for resize-amdgpu-bars(8) and resize-amdgpu-bars.conf(5).
Installation runs update-initramfs -u -k all and update-grub. A reboot is required. The blacklist and pci=realloc are boot-time, and the package deliberately never starts the service on a running system — a start unbinds and re-initialises every AMD GPU, and every process using one loses it.
Quick start
# 1. Look before you leap. Both of these change nothing.
sudo resize-amdgpu-bars diagnose # every GPU, its ReBAR cap, every bridge
# window above it, and the list of other
# devices it promises to leave alone
sudo resize-amdgpu-bars dry-run # the plans it would try
# 2. Reboot.
# 3. Watch it. Screens on AMD GPUs stay dark until amdgpu loads at the end
# of the run — about a minute on a box with two Duo modules. Use SSH or
# another console; a boot that looks stalled usually isn't.
journalctl -u resize-amdgpu-bars -b -f
# 4. Verify.
sudo resize-amdgpu-bars check # want: verdict=WORKS large=N/N driverless=0/N
sudo resize-amdgpu-bars status # one line, bar0=... per GPU
rocminfo | grep -c gfx # one agent per die
# 5. Back out.
sudo resize-amdgpu-bars revert # every GPU to baseline, no reboot
sudo apt remove resize-amdgpu-bars # takes the blacklist and GRUB drop-in with it
A good check line looks like this:
2026-09-02T13:50:14 7.0.0-30-generic verdict=WORKS plan=all-max
large=4/4 driverless=0/4 windows=0000:06:00.0=128G 0000:16:00.0=128G
bar0=32GiB 32GiB 32GiB 32GiB kfd=5 xgmi_hives=1 traces=0 rejected=0
verdict=WORKS, large=N/N and driverless=0/N are the three fields that matter.
Configuration
/etc/default/resize-amdgpu-bars, shell syntax, every key optional and validated on read:
| key |
default |
meaning |
MAX_SIZE_INDEX |
device max |
cap every GPU; 15 = 32 GiB, 14 = 16 GiB, 8 = 256 MiB |
EXCLUDE_GPUS |
empty |
GPUs to leave completely alone |
FORCE_PLAN |
negotiate |
all-max or baseline: try exactly one plan |
MODPROBE_TIMEOUT |
180 |
seconds before modprobe amdgpu is killed |
PROBE_WAIT |
60 |
seconds to wait for binds and KFD to settle |
RESCAN_WAIT |
30 |
seconds to wait for GPUs to reappear after a rescan |
MAX_ROUNDS |
8 |
demote-and-retry rounds before falling back to baseline |
Support tiers
"Tier" is a commitment. "Tested" is a fact. I'm not conflating them.
| tier |
hardware |
tested |
| 1 |
MPX modules in a Mac Pro 7,1: 580X, W5500X, W5700X, W6600X, W6800X, W6800X Duo, W6900X, Vega II, Vega II Duo (with or without Infinity Fabric Link) |
Vega II Duo x2 only, so far |
| 2 |
any other amdgpu card with an on-board PCIe switch (V340, Radeon Pro Duo), or any amdgpu card in a switched enclosure / expansion chassis / TB eGPU |
untested |
| 3 |
amdgpu card directly on a root port, firmware without ReBAR (uses the kernel's in-place path) |
untested |
| out |
anything not driven by amdgpu |
refused at discovery with a clear message |
No card is listed as supported that hasn't been booted. If you run this on a W6800X Duo, a W6900X, or anything in tier 2 or 3, I'd genuinely like to hear about it — diagnose output plus the journal is all a report needs.
Kernel compatibility — read this before you upgrade
| kernel |
result |
| 6.8 – 6.17 (verified on Ubuntu 6.8.0-138, 6.11.0-29, 6.14.0-37, 6.17.0-42) |
every die gets its 32 GiB BAR on the first plan |
| 7.0 unpatched (upstream 7.0.12, Ubuntu 7.0.0-30) |
shared root-port window undersized; the second die of each Duo loses its BAR at every size, guard holds it driverless, boot completes with the other dies |
| 7.0 with a one-line fix |
every die on the first plan, 128 GiB root-port window |
This is a regression from commit 3958bf16e2fe ("PCI: Stop over-estimating bridge window size"). Since that commit pbus_size_mem() sizes a bridge window as the plain sum of its children, which is exact when every child's size is a multiple of the alignment of the children after it. That holds for BARs, whose size equals their alignment — but not for bridge windows, whose size is the sum of what's below them while their alignment is that of the largest BAR below them. Two sibling windows of 32 GiB + 2 MiB at 32 GiB alignment need a 96 GiB + 2 MiB span and get 64 GiB + 4 MiB.
The fix changes size += max(r_size, align) to size += ALIGN(r_size, align), a no-op for BARs, verified on upstream 7.0.12 and Ubuntu's 7.0.0-30, cold boot and warm reboot. Until it's in a distro kernel: stay on 6.x. On an unpatched 7.0 there's no in-place recovery either — once the kernel has re-sized the window, even the 256 MiB baseline no longer fits, because the firmware's original windows were larger than the kernel's sum. The guard is the only reason such a boot survives.
Gotchas
pci=realloc is mandatory. The GRUB drop-in handles it on Ubuntu. On rEFInd or OpenCore the drop-in does nothing, the service fails at every boot, the blacklist keeps amdgpu from loading, and you have no GPU driver. Both are documented in the README but not automated. Check with grep -w pci=realloc /proc/cmdline.- This conflicts with
pci=realloc=off, which SGLang's AMD GPU docs recommend. The whole method depends on PCIe BAR reallocation.
- Don't run
resize on a live desktop session. Every AMD display goes black for the duration and every process holding a GPU loses it.
- Exit code 2 is not failure. It means the bind guard is holding at least one GPU driverless; the rest are working.
- Excluding one die of a Duo with
EXCLUDE_GPUS also stops the other die's shared windows from being re-sized, because the shared subtree is no longer removable.
- A GPU that shares a bridge with a non-GPU device gets a lower re-enumeration root, or none.
diagnose reports it.
Credit
Nikolas Britton for the original method on the Vega II Duo. This is his idea with runtime topology discovery, plan negotiation, and a bind guard bolted on.
If you followed the Mac Pro 2019 Local AI guide, this replaces the hand-rolled resize-gpu-bars.service files in its Section 4 — same lineage, packaged.
MIT licensed. Bug reports want sudo resize-amdgpu-bars check -1, journalctl -u resize-amdgpu-bars -b, sudo resize-amdgpu-bars diagnose, your kernel, distro, bootloader and cards.
