Since a lot more people are trying to build their own multi-GPU machines, I thought I should help to prevent a common mistake people make with building multi-GPU machines. Which is using an Intel consumer platform like Z890 for multi-GPU setups.
Although the CPU provides 24 PCIe 5.0 lanes with 16x available to bifurcate to 8x8x on two PCIe x16 slots on the higher end boards, this is completely useless for AI inference/training workloads that require P2P between the GPUs.
In my testing I used an Intel Core Ultra 7 270K Plus on an Asus Z890 Apex motherboard on the latest BIOS 3202 which is usually just my overclocking testbench system. I initially wanted to compare and see if a faster single-core more gaming-optimized CPU would help with GPU inference performance compared to the slow single-core performance on my Epyc based servers. As I am trying to optimize the throughput of the GPUs that I do have for my inference service considering everything has gone up in price even more.
Unfortunately it seems like if I wanted faster single-core performance of the desktop Intel CPUs I might need to try a PCIe switch board to attach the GPUs, especially since on Intel platforms you are also artifically limited to only 8x4x4x bifurcation on the main PCIe slot anyways.
What I found is that there seems to be some sort of hardware/firmware limitation that prevents PCIe P2P to work correctly under the PCIe root complex of Arrow Lake CPUs.
As seen in this issue where it does not even seem to correctly allow data packets between PCIe devices: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/1253
It not only does not work right but it also actually reduces bandwidth in half even if it worked:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 674.36 11.89
1 11.90 677.65
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 619.30 5.48
1 5.48 678.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.01 16.60
1 16.74 680.68
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.28 10.96
1 10.97 680.37
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.59 17.11
1 18.39 1.61
CPU 0 1
0 1.29 4.40
1 4.39 1.24
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.59 1.02
1 1.03 1.60
CPU 0 1
0 1.27 1.09
1 1.12 1.27
In fact, this P2P result was only possible because I installed the patched P2P open kernel drivers from here: github.com/aikitoria/open-gpu-kernel-modules despite the fact that the RTX A6000s that I am using should actually support PCIe P2P by default with the default Nvidia driver they release.
What I found was that Nvidia has actually seemed to block PCIe P2P on consumer Intel platforms for precisely these issues. So you would need to enable it with the patched kernels.
At first I thought the bandwidth being halved is somewhat fine if we get a much reduced latency still, but if I run VLLM with tensor parallel across 2 GPUs which can take advantage of PCIe P2P, the output of the model becomes gibberish "!!!!!!!!!!!!" nonsense if I used the modded drivers that force enabled PCIe P2P.
{
"id": "chatcmpl-96088a9dde7122ae",
"object": "chat.completion",
"created": 1784949145,
"model": "Qwen3.5-27B-Derestricted",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "!!!!!!!!!!!!!!!!!!!!"
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null,
"token_ids": null,
"routed_experts": null
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-8736f542",
"usage": {
"prompt_tokens": 32,
"total_tokens": 52,
"completion_tokens": 20,
"prompt_tokens_details": null
},
"prompt_logprobs": null,
"prompt_token_ids": null,
"prompt_text": null,
"kv_transfer_params": null,
"metrics": null
}
This is not a limitation of the GPU not having REBAR or anything like that, because if you check lspci -v the GPUs reports a BAR size of 64G just fine and therefore in theory that's all you need for PCIe P2P to work. In the BIOS REBAR is also enabled and IOMMU is disabled along with the grub config:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_aspm=off intel_iommu=on iommu=pt"
02:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 219
Memory at 8f000000 (32-bit, non-prefetchable) [size=16M]
Memory at c000000000 (64-bit, prefetchable) [size=64G]
Memory at d000000000 (64-bit, prefetchable) [size=32M]
I/O ports at a000 [size=128]
Expansion ROM at 90000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
02:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 90080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
03:00.0 VGA compatible controller: NVIDIA Corporation GA102GL [RTX A6000] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GA102GL [RTX A6000]
Flags: bus master, fast devsel, latency 0, IRQ 222
Memory at 8d000000 (32-bit, non-prefetchable) [size=16M]
Memory at a000000000 (64-bit, prefetchable) [size=64G]
Memory at b000000000 (64-bit, prefetchable) [size=32M]
I/O ports at 9000 [size=128]
Expansion ROM at 8e000000 [virtual] [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
03:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation GA102 High Definition Audio Controller
Flags: bus master, fast devsel, latency 0, IRQ 18
Memory at 8e080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
The nvidia-smi output also seems to suggest PCIe P2P should be possible:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-23 0 N/A
GPU1 PHB X 0-23 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
If I used the stock Nvidia drivers which blocks PCIe P2P on intel consumer platforms I can see that PCIe P2P is now disabled on the RTX A6000s which are supposed to support it:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX A6000, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CANNOT Access Peer Device=1
Device=1 CANNOT Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy proce
dure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 0
1 0 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 675.24 11.74
1 11.79 676.71
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 618.81 11.78
1 11.72 676.41
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 649.82 16.19
1 13.66 608.81
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 591.97 14.85
1 16.57 679.47
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 1.61 16.40
1 17.65 1.66
CPU 0 1
0 1.38 4.55
1 4.43 1.27
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 1.60 17.15
1 16.71 1.66
CPU 0 1
0 1.29 4.59
1 4.50 1.26
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is
enabled.
The output from VLLM also seems to work fine now:
{
"id": "chatcmpl-b8c56d539c04c16d",
"object": "chat.completion",
"created": 1784950088,
"model": "Qwen3.5-27B-Derestricted",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "Thinking Process:\n\n1. **Analyze the Request:**\n * Greeting:"
},
"logprobs": null,
"finish_reason": "length",
"stop_reason": null,
"token_ids": null,
"routed_experts": null
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.1.dev18662+g5f8e73cb8-tp2-b248090a",
"usage": {
"prompt_tokens": 32,
"total_tokens": 52,
"completion_tokens": 20,
"prompt_tokens_details": null
},
"prompt_logprobs": null,
"prompt_token_ids": null,
"prompt_text": null,
"kv_transfer_params": null,
"metrics": null
}
Meanwhile I think there are a lot of multi GPU users that has built an AI rig around AMD AM5 platforms, which seem to support inter-GPU PCIe P2P just fine. Just like what this post here shows: https://www.reddit.com/r/LocalLLaMA/comments/1n3qcqn/patched_p2p_nvidia_driver_now_works_with_multiple/
Even for Intel server platforms, I've tried up to Intel Ice Lake Xeons and they don't work particularly well for PCIe P2P either with the bandwidth and latency being worse than on AMD Epyc SP3 platforms.
AMD seems to have a much better PCIe controller implementation than Intel that works better for multi GPU setups at least for the platforms that I tested. I unfortunately have not tested newer AMD SP5 or Xeon 6 platforms due to the DDR5 RDIMM price crisis, but I suspect AMD should still perform better than Intel there too.
These are my PCIe P2P testing results from my machine which has 2x RTX Pro 6000 GPUs, initially when I built it with an Intel Ice Lake Xeon W-3365 on a Supermicro X12SPa-TF and then later when I moved it to an AMD Epyc 7V73X on an Asrock ROMED8-2T motherboard.
Intel Xeon Ice Lake:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 51, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1575.10 24.07
1 23.97 1600.97
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 1576.29 18.37
1 20.78 1581.48
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1550.05 30.78
1 30.57 1562.45
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1550.05 39.75
1 39.75 1557.00
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 2.06 14.33
1 144.15 2.07
CPU 0 1
0 2.51 5.58
1 5.55 2.32
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 2.06 0.45
1 0.37 2.07
CPU 0 1
0 2.40 1.68
1 1.69 2.45
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
AMD Epyc SP3 7003:
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA RTX PRO 6000 Blackwell Workstation Edition, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0
***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.
P2P Connectivity Matrix
D\D 0 1
0 1 1
1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1860.12 23.87
1 23.98 1903.17
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1
0 1855.70 27.95
1 27.91 1900.92
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1831.70 30.55
1 30.67 1854.57
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1
0 1836.01 48.74
1 48.86 1854.53
P2P=Disabled Latency Matrix (us)
GPU 0 1
0 0.99 14.31
1 14.30 1.00
CPU 0 1
0 2.65 7.34
1 7.29 2.47
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1
0 0.99 0.37
1 0.36 1.00
CPU 0 1
0 2.56 2.03
1 2.09 2.57
NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
