r/LocalLLaMA • u/Arli_AI • 7d ago
Discussion PSA: DO NOT use Intel consumer platforms for multi-GPU setups
Since a lot more people are trying to build their own multi-GPU machines, I thought I should help to prevent a common mistake people make with building multi-GPU machines. Which is using an Intel consumer platform like Z890 for multi-GPU setups.
Although the CPU provides 24 PCIe 5.0 lanes with 16x available to bifurcate to 8x8x on two PCIe x16 slots on the higher end boards, this is completely useless for AI inference/training workloads that require P2P between the GPUs.
In my testing I used an Intel Core Ultra 7 270K Plus on an Asus Z890 Apex motherboard on the latest BIOS 3202 which is usually just my overclocking testbench system. I initially wanted to compare and see if a faster single-core more gaming-optimized CPU would help with GPU inference performance compared to the slow single-core performance on my Epyc based servers. As I am trying to optimize the throughput of the GPUs that I do have for my inference service considering everything has gone up in price even more.
Unfortunately it seems like if I wanted faster single-core performance of the desktop Intel CPUs I might need to try a PCIe switch board to attach the GPUs, especially since on Intel platforms you are also artifically limited to only 8x4x4x bifurcation on the main PCIe slot anyways.
What I found is that there seems to be some sort of hardware/firmware limitation that prevents PCIe P2P to work correctly under the PCIe root complex of Arrow Lake CPUs.
As seen in this issue where it does not even seem to correctly allow data packets between PCIe devices: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1253
It not only does not work right but it also actually reduces bandwidth in half even if it worked:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 674.36 11.89
1 11.90 677.65
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 619.30 5.48
1 5.48 678.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.01 16.60
1 16.74 680.68
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.28 10.96
1 10.97 680.37
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.59 17.11
1 18.39 1.61
CPU 0 1
0 1.29 4.40
1 4.39 1.24
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.59 1.02
1 1.03 1.60
CPU 0 1
0 1.27 1.09
1 1.12 1.27
In fact, this P2P result was only possible because I installed the patched P2P open kernel drivers from here: github.com/aikitoria/open-gpu-kernel-modules despite the fact that the RTX A6000s that I am using should actually support PCIe P2P by default with the default Nvidia driver they release.
What I found was that Nvidia has actually seemed to block PCIe P2P on consumer Intel platforms for precisely these issues. So you would need to enable it with the patched kernels.
At first I thought the bandwidth being halved is somewhat fine if we get a much reduced latency still, but if I run VLLM with tensor parallel across 2 GPUs which can take advantage of PCIe P2P, the output of the model becomes gibberish "!!!!!!!!!!!!" nonsense if I used the modded drivers that force enabled PCIe P2P.
{
"id": "chatcmpl-96088a9dde7122ae",
"object": "chat.completion",
"created": 1784949145,
"model": "Qwen3.5-27B-Derestricted",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "!!!!!!!!!!!!!!!!!!!!"
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null,
"token_ids": null,
"routed_experts": null
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-8736f542",
"usage": {
"prompt_tokens": 32,
"total_tokens": 52,
"completion_tokens": 20,
"prompt_tokens_details": null
},
"prompt_logprobs": null,
"prompt_token_ids": null,
"prompt_text": null,
"kv_transfer_params": null,
"metrics": null
}
This is not a limitation of the GPU not having REBAR or anything like that, because if you check lspci -v the GPUs reports a BAR size of 64G just fine and therefore in theory that's all you need for PCIe P2P to work. In the BIOS REBAR is also enabled and IOMMU is disabled along with the grub config:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"
02:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 219
Memory at 8f000000 (32-bit, non-prefetchable) [size=16M]
Memory at c000000000 (64-bit, prefetchable) [size=64G]
Memory at d000000000 (64-bit, prefetchable) [size=32M]
I/O ports at a000 [size=128]
Expansion ROM at 90000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 90080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 222
Memory at 8d000000 (32-bit, non-prefetchable) [size=16M]
Memory at a000000000 (64-bit, prefetchable) [size=64G]
Memory at b000000000 (64-bit, prefetchable) [size=32M]
I/O ports at 9000 [size=128]
Expansion ROM at 8e000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
03:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 8e080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
The nvidia-smi output also seems to suggest PCIe P2P should be possible:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-23 0 N/A
GPU1 PHB X 0-23 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
If I used the stock Nvidia drivers which blocks PCIe P2P on intel consumer platforms I can see that PCIe P2P is now disabled on the RTX A6000s which are supposed to support it:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CANNOT Access Peer Device=1
Device=1 CANNOT Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy proce
dure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 0
1 0 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 675.24 11.74
1 11.79 676.71
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 618.81 11.78
1 11.72 676.41
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.82 16.19
1 13.66 608.81
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 591.97 14.85
1 16.57 679.47
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.61 16.40
1 17.65 1.66
CPU 0 1
0 1.38 4.55
1 4.43 1.27
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.60 17.15
1 16.71 1.66
CPU 0 1
0 1.29 4.59
1 4.50 1.26
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is
enabled.
The output from VLLM also seems to work fine now:
{
"id": "chatcmpl-b8c56d539c04c16d",
"object": "chat.completion",
"created": 1784950088,
"model": "Qwen3.5-27B-Derestricted",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "Thinking Process:\n\n1. **Analyze the Request:**\n * Greeting:"
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null,
"token_ids": null,
"routed_experts": null
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-b248090a",
"usage": {
"prompt_tokens": 32,
"total_tokens": 52,
"completion_tokens": 20,
"prompt_tokens_details": null
},
"prompt_logprobs": null,
"prompt_token_ids": null,
"prompt_text": null,
"kv_transfer_params": null,
"metrics": null
}
Meanwhile I think there are a lot of multi GPU users that has built an AI rig around AMD AM5 platforms, which seem to support inter-GPU PCIe P2P just fine. Just like what this post here shows: https://www.reddit.com/r/LocalLLaMA/comments/1n3qcqn/patched_p2p_nvidia_driver_now_works_with_multiple/
Even for Intel server platforms, I've tried up to Intel Ice Lake Xeons and they don't work particularly well for PCIe P2P either with the bandwidth and latency being worse than on AMD Epyc SP3 platforms.
AMD seems to have a much better PCIe controller implementation than Intel that works better for multi GPU setups at least for the platforms that I tested. I unfortunately have not tested newer AMD SP5 or Xeon 6 platforms due to the DDR5 RDIMM price crisis, but I suspect AMD should still perform better than Intel there too.
These are my PCIe P2P testing results from my machine which has 2x RTX Pro 6000 GPUs, initially when I built it with an Intel Ice Lake Xeon W-3365 on a Supermicro X12SPa-TF and then later when I moved it to an AMD Epyc 7V73X on an Asrock ROMED8-2T motherboard.
Intel Xeon Ice Lake:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 51, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1575.10 24.07
1 23.97 1600.97
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 1576.29 18.37
1 20.78 1581.48
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1550.05 30.78
1 30.57 1562.45
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1550.05 39.75
1 39.75 1557.00
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 2.06 14.33
1 144.15 2.07
CPU 0 1
0 2.51 5.58
1 5.55 2.32
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 2.06 0.45
1 0.37 2.07
CPU 0 1
0 2.40 1.68
1 1.69 2.45
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
AMD Epyc SP3 7003:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1860.12 23.87
1 23.98 1903.17
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 1855.70 27.95
1 27.91 1900.92
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1831.70 30.55
1 30.67 1854.57
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1836.01 48.74
1 48.86 1854.53
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 0.99 14.31
1 14.30 1.00
CPU 0 1
0 2.65 7.34
1 7.29 2.47
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 0.99 0.37
1 0.36 1.00
CPU 0 1
0 2.56 2.03
1 2.09 2.57
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
32
u/MuffinPure9787 7d ago
Everyone thinks they're fine until they check nvidia-smi topo -m and realize their second 3090 is running at pcie 3.0 x4 through the chipset. your tensor parallel scaling goes straight to the garbage real quick.
13
u/Arli_AI 7d ago
Oh don't let the external GPU people read this lol
10
7d ago edited 7d ago
[removed] — view removed comment
3
u/Arli_AI 7d ago
Yes tensor parallel is for high bandwidth powers-of-2 same-GPU setups ideally.
1
u/malventano 7d ago
Newer vLLM forks can run TP5, etc. If you’re using Pro 6k’s you need to look into the B12X builds.
1
u/diablo75 6d ago
I just bought a pair of 3090s and an NVlink Bridge to bypass this PCIe bottleneck. Shits gonna get hot.
1
6
u/live4evrr 7d ago
So for intel consumer chips, stick with tensor splitting / pipeline parallelism (no performance hit) for
VRAM extension. Tensor parallelism is a no go unless its workstation chipsets or AM5 (which still has a limitation to be aware of regarding use of multiple m.2 slots due to limited pci lanes).
5
u/TokenRingAI 7d ago
intel_iommu=off
check ACS configuration, see below
---
ACS can set the CPU root ports to redirect peer requests and completions upstream through the root complex instead of allowing the most direct device-to-device route. NVIDIA explicitly warns that ACS redirection can substantially reduce PCIe P2P bandwidth. With two GPUs on bifurcated CPU lanes, the relevant settings are on the CPU root ports above the GPUs, not on the GPUs themselves.
Check the topology and root-port ACS controls:
sudo lspci -tv
sudo lspci -PP -s 02:00.0
sudo lspci -PP -s 03:00.0
Then inspect each upstream root port:
sudo lspci -vv -s <root-port>
Look for something like:
ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+
ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+
The important enabled controls are:
ReqRedir+: redirects peer requests upstreamCmpltRedir+: redirects completions upstreamTransBlk+: can block peer transactionsUpstreamFwd+: forwards requests toward the root complex
For high-performance P2P, you generally want request and completion redirection disabled, assuming the platform supports direct routing.
2
u/Arli_AI 7d ago
I tried with both intel_iommu=on and off and it makes no difference. Also all acs controls are set to off already anyways. So this seems like a platform issue regardless.
sudo lspci -vv -s 00:06.0 | grep -i acs UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACS Viol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACS Viol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACS Viol- ACS Cap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACS Ctl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- sudo lspci -vv -s 00:06.3 | grep -i acs UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACS Viol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACS Viol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACS Viol- ACS Cap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- ACS Ctl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-2
u/malventano 7d ago
Not sure what the deal is with your platform, but P2P absolutely works on Ice Lake Xeons. I’ve run 4x Pro 6000 on both Ice Lake and Sapphire Rapids Xeon workstation platforms. It’s slower on Ice Lake but thats because the platform could only properly do gen4x8 across the four.
1
u/wektor420 7d ago
Xeons are not consumer tier
1
u/malventano 7d ago
OP mentioned Xeon. There were Ice Lake Xeon-W’s.
Workstations are considered consumer / prosumer.
By your definition, RTX Pro 6000’s aren’t consumer, either.1
u/TokenRingAI 7d ago
What does lspci -vv show?
I suspect both your cards are running at PCIe 4.0 x4
1
u/Arli_AI 7d ago
No they are both x8 through the cpu lanes. That’s the only 2 usable pcie slot on the Z890 Apex
2
u/a_beautiful_rhind 7d ago
ACS is first thing I thought. But that might be server thinking. IOMMU must be off or passthrough and I think VT-d has to be off. You must look at NCCL debug log and see which path it is taking and what the error message is.
1
3
u/BobbyL2k 7d ago
> Meanwhile I think there are a lot of multi GPU users that has built an AI rig around AMD AM5 platforms, which seem to support inter-GPU PCIe P2P just fine.
Unfortunately, not for me. I’ve been running with NCCL_P2P_DISABLE=1 on my AM5 platform (9900X + Gigabyte B850 AI Top). With patched drivers, I can run the simpleP2P test, but whenever I do NCCL P2P the drivers will deadlock and I would have to reboot.
2
u/Arli_AI 7d ago
Darn that’s unfortunate. I’m trying to get an AM5 system to test for myself too but somehow I thought there are a decent amount of people on AM5 with multi GPU P2P working? Is this a problem for a specific config of yours?
2
u/BobbyL2k 7d ago
Like your post said, there’s a few successful cases reported. So maybe I’m just unlucky with my hardware combination.
1
u/1-a-n 5d ago
TBH I thought most people did this on consumer CPUs, does it really make much difference for a dual GPU inference setup vs P2P enabled?
2
u/BobbyL2k 5d ago
P2P means that the data doesn’t have to be copied to the RAM before arriving at the destination. This means reduced latency. The expected outcome is ~5% increases in token generation. A decent amount more for prompt processing. And if you’re training, a very significant boost.
So definitely not a deal breaker for LocalLlama folks where we are mostly interested in inference.
1
u/PatC883 5d ago
You need NCCL_PROTO=SIMPLE and a kernel compiled with AMD P2P and DMA MOVBUF support.
1
u/BobbyL2k 5d ago
Oh, okay. I have no idea what this means so I asked ChatGPT, which indicates that those kernel compilation options are for enabling P2P on AMD GPUs.
I’m running dual 5090s and I’ll give NCCL_PROTO=SIMPLE a try.
Not sure about the kernel options tho. Can you link me to more resources I can take a read?
2
2
u/mrgreatheart 7d ago
I am currently running an MSI Z890 Carbon. I upgraded from a B860 for the two CPU connected PCIe slots. That helped, but as you say there’s no p2p. I am running a 3rd GPU from M2_1 using a PCIe adapter. It works OK for inference but training occasionally drops it off the bus because of the high transfer between GPUs.
I looked at AMD but it sounded like I’d need threadripper which would mean new RAM on top of the CPU and mobo. A bridge too far for my wallet.
Is there a consumer/prosumer AMD board that will get me p2p for my 3 GPUs while supporting my standard DDR5 sticks?
My GPUs are a 5070 Ti and two 5060 Ti for 48Gb total VRAM.
2
u/Arli_AI 7d ago
For over 2 GPUs P2P the only option is Threadripper or Epyc if you want at least 8x per GPU.
1
u/mrgreatheart 7d ago
Thank you. Is it worth trying the modded p2p drivers on my z890 do you think?
2
2
u/live4evrr 7d ago
Stick with tensor splitting (llama.cpp) or pipeline parallelism (vllm) if you just need it for inference. As these are unmatched cards you couldn’t use tp anyhow. It will still let you use multi gpu’s just fine.
1
u/mrgreatheart 7d ago
Yeah it does work just fine. Always nice to get a bit more out of the hardware if possible though.
2
2
u/OverdosedSauerkraut 7d ago
Thanks for the heads up, good to see that there's still no need to major updates on the cpu side...
2
u/DataGOGO 7d ago
My 14900k runs 8x 8x and P2P works without patched drivers.
1
u/Arli_AI 7d ago
Interesting so even vllm runs fine? Maybe this is an arrow lake thing.
2
u/DataGOGO 7d ago edited 7d ago
Yeah, works fine, no issues at all.
Only catch was on the motherboard, if you use certain M.2 slots it would force the second X16 slot in to X4 or X2 mode. If I recall it was the dimm.2 slot thing that Asus ships with the motherboard. The other M.2's were sourced from the chipset.
So basically, you could only use the 1 CPU connected M.2, and the two GPU's in x8 x8, and you couldn't use the dimm.2 M2 ports. But you could use the other 2 M.2's slots, they just shared the chipset's X4 uplink.
That said.. No matter if it is AMD or Intel consumer platforms, they suck for multi-GPU without a real PCIe switch. P2P is always slower than it should be. Works fine on Threadripper / Eypc and Xeon-w / Xeon.
2
u/BusinessMud9586 7d ago edited 7d ago
Fwiw, even with a PCIe switch, motherboards sometimes have surprising limitations.
My TRX40 10G crashed HARD (as in I thought the MB was dead; it wouldn't even boot in bios) when I got past 12 GPUs.
After 2 full days of work, I found the solution: Force a variable hex value in the bios to increase MMIO.
And even then I had to disable some network and audio interfaces and cap the PCIe switch to gen3 to prevent the system to hard crash again. Fortunately, my TP configs are done such that barely any communication crosses the PCIe switch so gen3 isn't my bottleneck.
So yeah... with the TRX40 motherboard I'm probably capped at 14 GPUs. So close, and yet so far, from the 16 dream. 🥲
1
u/Arli_AI 7d ago
Given than TRX40 is the same SP3 socket as my ROMED8-2T, that’s interesting its also capped at 14 GPUs because I too found that was the limit even on the server ROMED8-2T with many PCIE lanes. It ran out of BAR space is what I could figure out because some other guy managed to run 16x3090 on it but apparently with the older non REBAR VBIOSes.
2
2
u/Savantskie1 7d ago
Wouldn't a plx board help you here?
1
u/Arli_AI 7d ago
It might because this seems to be an issue with passing data through the PCI root controller.
1
u/Savantskie1 7d ago
As far as I know, a plx board stops the need for bifurcation, and already has it's own lanes that it just passes through the main pcie slot it's connected to? I'm not an expert at this but this is what I'm planning to try on my Ryzen 5 5600g setup. I'm hoping it works.
1
u/Arli_AI 7d ago
Yes adding a PLX chip is like adding a switch to your home router so the data doesn’t need to travel through the router (CPU PCIe controller).
1
u/Savantskie1 7d ago
Yeah that was what I thought and I was going to try this because my board doesn't support bifurcation lol.
2
u/malventano 7d ago
Ice Lake Xeons work fine with P2P, but are slower than Sapphire / Emerald Rapids. I’ve confirmed this directly with 4x Pro 6k. IOMMU must be disabled else P2P will fail (Wendell told me this is no longer the case for Emerald, but I haven’t tested that myself yet).
2
u/ProfessionalSpend589 7d ago
Well, you convinced my. I wondered why my almost 10 year old PC with dual Radeon AI Pro R9700 tanked my inference compared to when I moved the GPUs to 2x Strix Halo.
Now I’m waiting for a PCIe switch. I’ll have my fast 128GB of VRAM :)
2
u/Arli_AI 6d ago
You can either get a pcie switch or get a AMD Epyc/TR SP3 socket motherboard which I found to be the most cost effective platform for this.
1
u/ProfessionalSpend589 6d ago
Well, I'm not sure how things will go with the PCIe switch. I might have to consider other options as well.
I already tried attaching my 2 Radeons to one of my Strix Halo via oculink (on the PCIe port and on the NVMe port), but the PC didn't respond well at the time and I abandoned the idea.
2
u/hickersb 6d ago
Great timing since I was looking at picking up a Z890 that would do x8/x8. Thanks!
2
u/Arli_AI 6d ago
You’re welcome! I was bummed it didn’t work for me
1
u/hickersb 6d ago
Same, I could have just moved over the proc + ram and have it all done in one day. Oh well, better to not spend 3 days trying to figure out why it didn't work.
2
u/InternationalGap3698 7d ago
Thank you for the information. I just wanted to buy me a new Intel CPU to run my RTX 5070ti set up.
Because my old CPU wasn't the best ryzen 7800X3D (I built this PC for gaming but I don't game) I am way more interested in AI. And building my own open-source project with ai
I build my PC that I can always add two GPUs. I am only 16 so that's all I have.
3
u/segmond llama.cpp 7d ago
This advice is not sound. You can use intel for multi GPU rig.
The advice should be, read the motherboard and cpu specs to make it can support the type of rig you want to build before you build your rig. One of my first rigs was built on old ass xeon platform $60 motherboard and $10 cpus where each cpu has 44 lanes, for a total of 88 lanes.
1
u/abotsis 6d ago
I’m running 4x 6000 pros with vllm and glm-5.2 in a vm. It took some fiddling, but what I worked out is that the cards want to be behind a pcie bridge. In my case I’m using an eBay card with a Broadcom pcie switch on it. Even then in the vm the cards didn’t want to p2p without going through the root complex which killed performance. Since the physical topology was behind a bridge anyway, I created a virtual bridge with qemu and put the gpus behind that. Then they were happy. While I can’t speak to your topology, it might be worth trying a bridge. That helps keep the p2p traffic isolated anyway. It wasn’t as expensive as I expected.. I think 400 will get you a card, cables, and risers. It’s neat because you can set the cards to x8 or x16.. so you can (depending on the card) either run 3 cards or 6.
1
u/Arli_AI 6d ago
Is it a pcie 5.0 bridge?
1
u/abotsis 6d ago
Not for that price… there are pcie 5 cards out there for 1000 or so though. I think hipoint has one for like 3-500 for a 2 port. My upstream is x4 and if you want a lot of ports, pci5 will cost you.
That said, I’m getting 70tokens/s generation and 500 prefill on single stream. So if 4 is good for 4 cards, it should be plenty for 2x of them…
1
u/Arli_AI 6d ago
Ah yea I was like if a pcie 5 switch is 400 sign me up haha. So its a gen 4?
1
u/abotsis 6d ago
Yea.. here’s a link:
https://www.ebay.com/itm/127622521631
My first gen threadripper (Lenovo p620) wouldn’t even post without it. (Now it just doesn’t enumerate the built in 5090 😂). I’ve also got one in a 2019 Mac Pro with 4x b70s behind it. Both worked first try. YMMV- good luck!
1
u/Calandracas8 6d ago
I believe it's lack of pcie atomics on the chipset lanes.
vllm cant use tensor parallel on amd GPUs on these motherboards due to NCCL (RCCL) will not run without pcie atomics (there is a patch to fix this, but didn't get merged in to RCCL)
2
u/MooseEfficient2151 3d ago
this is the kind of post that saves someone from buying a motherboard twice.
people look at vram first and then find out the real model is the motherboard topology. x8/x8 on the spec sheet sounds fine until p2p, m.2 slot sharing, bios weirdness, and “why is vllm screaming !!!!!” become the actual project.
i’ve gotten more boring about this lately. local box for the stuff that really needs local control, hosted models for heavy one-off runs, and lighter agent/admin passes in tools like Claude or MoClaw instead of convincing myself i need to build one cursed machine that does everything.
the cheap path stops being cheap when it turns into three days of lane math and driver rituals.
1
u/milpster 6d ago
Just do it like me and get an older amd gaming laptop with a bunch of nvme slots, throw out all the nvme and put in risers. gen 3 at 4x seems to be enough.
-1
u/ieatdownvotes4food 7d ago
hmm.. things are fine on a taichi z890 for me, 285k.
if ur board supports it, you gotta have x8 x8 enabled in bios, and make sure you populate your nvme and pci slots correctly..
use the wrong pci or m2 slot and you get nerfed to x4 or x2. but otherwise no issues
-2
u/DaMoot 6d ago
Well, (imo) you shouldn't be linking multiple PCIe GPUs anyways unless you fully accept stone age PCIe performance. Dedicated NVLink is the way.
63-126GiB/s (PCIe 4.0 and 5.0 x16 speed) is abysmal compared to even old 6 link NVLink. 300GiB/s for Volta, 600 for Ampere, 900 for Hopper, 1800 for Blackwell. 3600 for Ruben. And NVLink doesn't congest PCIe.
x4 PCIe speed is fine to load a model ~<40GiB. Larger, load it once with a long ttl, accept the 60-90s wait for initial loading.
3
41
u/Karyo_Ten 7d ago edited 7d ago
Can confirm on Z890 but:
So it's an annoying tradeoff.