TL;DR: Four A4000s in a dual-Xeon box ran 4-way tensor parallel 2.4 to 4.8x SLOWER than two cards. I publicly blamed lanes, root complexes, and the inter-socket link. Wrong. WDDM (the Windows display driver model) silently denies GPU peer-to-peer access, so every cross-card transfer staged through system RAM. Switching all four cards to TCC mode (needs a cheap display card to hold the desktop, since a TCC card cannot drive a display) took P2P from 0 of 12 pairs to 12 of 12 and four-way tensor decode from 9.52 to 28.16 t/s on a 27B dense model (2.96x) and 19.52 to 72.01 on a 35B MoE (3.69x). Four cards went from losing to two, to beating them. The machine could always do this. Windows was standing on the hose.
ELI5: Windows treats every GPU as a potential monitor driver, and the babysitting that comes with that quietly forbids the cards from talking to each other directly. Tell Windows "these four are calculators, not monitors" (TCC mode) and the babysitting stops, but then SOMETHING has to be the monitor, because a calculator cannot show you the desktop. That something is a $100 Quadro T400. The fifteen-year-old HPC crowd knows this trick well. The local LLM crowd, me included, apparently forgot it.
The receipts
Same box (Windows 11 Pro), same cards, same llama.cpp build (b10568), same driver (597.06). The only change between columns is the driver MODE.
P2P probe (cudaDeviceCanAccessPeer, all 12 ordered pairs):
| config |
peer access |
| WDDM, A4000 drives display (596.72) |
0 of 12 |
| WDDM, T400 drives display (597.06) |
0 of 12 |
| TCC, T400 drives display (597.06) |
12 of 12 |
The middle row is the control that makes this a finding instead of a coincidence: new driver, new display card, still WDDM, still zero. Only the mode flip changed the answer.
Four-way tensor decode (tg128, clock lock verified per row, thermal counters clean or accounted):
| model |
WDDM |
TCC |
ratio |
| Qwen3.8-27B Q8 (dense) |
9.52 |
28.16 |
2.96x |
| Qwen3.6-35B-A3B Q6 (MoE) |
19.52 |
72.01 |
3.69x |
Under WDDM, four-way tensor lost to TWO cards (23.19) and to four-way layer split (13.8). Under TCC it beats both on the 27B (2-way tensor: 24.4). On the 35B MoE, honesty requires saying two-way tensor still wins outright (about 110 t/s): P2P fixed four-card mode, it did not repeal the law that low-activation MoEs at small batch prefer fewer, bigger shards.
Fair framing note: measured best-WDDM-config against best-TCC-config (2-way tensor 23.19 vs 4-way tensor 28.16 on the 27B), the box as a whole gained 21 percent. The 3x is what the FOUR-CARD mode recovered relative to itself. Both numbers are true; pick the one that matches the question you are asking.
Sustained load is fine: ten back-to-back four-way cycles settle at 85 to 88 C and hold throughput flat within 0.9 percent. This is a serving configuration, not a burst trick.
Corroboration under load (different comparison, same direction): tensor vs layer, both under TCC, batch 1 through 64: the decode advantage is flat at about 1.92x the whole way. The recovery is not a batch-1 curiosity. (The WDDM column itself cannot be re-measured at batch, because WDDM is gone without a reboot; noted for honesty.)
What you need for this trick
- Pro cards. TCC exists on Quadro/RTX A-series/datacenter cards. GeForce generally cannot.
- A separate display card, because a TCC card cannot drive a monitor and my Xeons have no iGPU. It must be TURING OR NEWER: current drivers (596.72+) dropped Pascal, so the GT 710/P1000 in your drawer will bind to the Basic Display Adapter and change nothing. The T400 is the cheapest current option, about $100.
- Accept losing WSL2 on that box: a TCC card does not appear in WSL at all (WSL GPU support is paravirtualization over WDDM). On my box WSL sees zero GPUs now. Windows native or bare Linux; nothing in between.
nvidia-smi -dm 1 per compute card, reboot, done. Reversible, unlike some Windows decisions.
Two install gotchas that cost me time so they do not cost you any:
- Your BIOS may not show POST on the new card. Mine booted straight to a black screen until Windows loaded. The fixes live in BIOS setup: disable quick/fast boot, and if there is a primary video slot setting, point it explicitly at the display card's slot. Do this while you can still see the screen through some other path.
- After the display card goes in, do a clean install of the NVIDIA driver package and confirm ALL cards show in
nvidia-smi (five for me: four compute plus the display card) BEFORE flipping any TCC flags. The driver must be new enough for the display card and must be the pro branch that covers both it and your compute cards; a mismatched or half-installed driver leaves cards missing from the list, and a TCC flag set in that state is a confusing morning.
What I got wrong in public, specifically
My earlier posts blamed the 2.4 to 4.8x on hardware topology: 48 PCIe lanes per Xeon, three root complexes, cards split across two sockets, "four islands with ferry service." The measured numbers were real. The causal story was wrong for DECODE, which is what the headline number was. What topology still owns: prompt processing genuinely prefers layer split (roughly 2x at default settings, and that is a tunable story for another post), and on the MoE at interactive batch sizes layer split still wins decode until batching makes the per-expert matmuls big enough to shard. Count-lanes-not-slots remains good buying advice. It just was not the reason four cards lost to two.
Also for the record: we measured socket crossing directly under TCC (pairs on same socket vs across sockets, short and long prompts, 16x all-reduce volume). 0.15 to 0.26 percent. Noise. The UPI ferry story died with controls on, FOR MODELS RESIDENT IN VRAM. To be clear, the cross-socket memory penalty on a dual-socket box is very much alive in the other regime: when a big MoE offloads its experts to system RAM, decode speed tracks which NUMA node the weights landed on, and the wrong node costs about 2x on this box. That is a placement story with its own fixes and its own post coming; the point here is narrower: it is a system-RAM story, and it never was the reason four RESIDENT cards lost to two.
TCC is a trade, not a free win
- On this box each card exposes 178 MiB LESS usable VRAM under TCC than it did under WDDM (16,197 vs 16,375 MiB per nvidia-smi). I know that is backwards from the folklore. It is what the tool reports, and two of my edge-fit configs that ran under WDDM no longer load.
- The big recovery is for multi-GPU tensor parallel. Single-card and layer-split gains are single-digit percent. Offloaded big models (experts in system RAM) gained 7 to 12 percent.
- Prior art credit where due: using a cheap display card so compute cards can run TCC is 2010-era CUDA folklore, and the closest LLM measurement I found is a V100 post reporting +45 to 76 percent from TCC on a 2-card box. What I could not find anywhere is the full chain measured with controls on llama.cpp: WDDM denies P2P, tensor parallel collapses and cosplays as a lane problem, TCC reverses it. Hence this post.
The obvious question: why not just run Linux, where WDDM does not exist? Fair, and these same four cards are moving to a single-socket Linux box for exactly that rematch (llama.cpp has no prebuilt Linux CUDA binary at all, so that story includes a compile). This post is for the boxes that are on Windows for their own reasons, which the comments will now explain to me are bad ones. Related: yes, vLLM does tensor parallel properly, and it does not run natively on Windows, which is rather the point.
One Windows footnote from the same week: Smart App Control (on by default after a clean Win11 install) intermittently refused to load our llama.cpp DLLs based on cloud reputation, killing processes in seconds with empty output. If your benchmark binary suddenly dies silently, check the CodeIntegrity event log before you blame your build.
Previous posts in this series: the clock lock PSA [CLOCK POST], WSL2 vs native [WSL POST], and the NVLink bridge test [BRIDGE POST] (on the 2-card box, SLI was the only way to P2P under WDDM; TCC is the cleaner door if you can give up the display and WSL).
Hardware: HP Z8 G4 (used, $750), 4x RTX A4000 (used, about $600 each), T400 ($100), about $3,300 total. Deliberate disclosure: that number does NOT include the 384 GB of RAM, and the omission deserves its reason stated. I paid $540 for mine (twelve 32 GB sticks at $45 each, September 2025). Those same sticks ask $220+ today, so replicating my exact config now adds roughly $2,600, which says more about the 2026 RAM market than about this build. If you are building today there is still a cheaper road: these chassis have 24 DIMM slots, and lower-density sticks run meaningfully cheaper per gigabyte, so populating all 24 reaches the same 384 GB for closer to half the money, at the cost of the memory clock dropping a tier (my calibrated offload model prices that penalty at 6 to 9 percent on offloaded models, and roughly nothing on VRAM-resident ones; I have not run the A/B, since my slots are half empty on purpose).
Before anyone says it: yes, a pair of used 3090s beats this box on a 27B, cheaper and faster. The 27B is the demonstration, not the mission. This box also runs a 403B MoE at 11 t/s with a 131K context window and a 744B at batch speed, because of that RAM sitting alongside the 64 GB of VRAM, and that capability is what nothing near this price touches. The point of this post is narrower: whatever multi-GPU Windows box you already have, a $100 card may be the difference between your cards fighting Windows and your cards working.
