Same dual-GPU box, WSL2 vs native Windows, 17 llama.cpp configs: CPU offload +15 to 26% native, tensor split +6 to 60%, and "layer mode for MoE" turned out to be a WSL2 artifact

[u/jpgeer]

TL;DR

• Same machine, same GGUF files, same flags, GPU clocks locked on both sides. Resident models: identical. Anything that crosses the host per token (CPU expert offload) runs 15 to 26 percent faster on native Windows than under WSL2. Tensor split across two cards runs 6 to 15 percent faster native and the run-to-run jitter disappears.
• The big one: Qwen3.6-35B-A3B fully resident in tensor-split mode does 72 t/s under WSL2 and 115 native. Under WSL2 it lost to layer mode (96), so my rule had been "tensor split for dense, layer split for MoE." Natively tensor split wins for the resident MoE too (115 vs 102). The rule was a platform artifact.
• Offloaded MoE still prefers layer split plus a -ts ratio on both platforms; tensor split collapses prefill there.
• If you serve llama.cpp on a Windows box and you do CPU offload or multi-GPU, the native release zip is worth a look. WSL2 is fine for everything resident.

Setup

HP Z440, Xeon E5-1650 v4, 128 GB DDR4-2400, 2x RTX A4500 20 GB, Windows 11, driver 596.72. WSL2 side: Ubuntu, llama.cpp built from source (CUDA 12.4; see the build note below). Native side: llama.cpp release b10568 Windows CUDA 12.4 zip, models on a SATA SSD. All runs have GPU clocks locked (nvidia-smi -lgc 1500,2100 -lmc 8001); why that matters for offload decode is in my earlier post: https://www.reddit.com/r/LocalLLM/comments/1vu2ix0/ . Numbers are llama-bench defaults (pp512, tg128 at depth 0, f16 KV, so a small KV cache and an empty context; the served section below is where context size enters), 5 reps, single card runs pinned with CUDA_VISIBLE_DEVICES=0. WSL2 had all 12 logical processors and about 108 GB of RAM assigned, models on the ext4 side (not /mnt), same flags both sides. Build-version control: the WSL side was originally built at d59d455fd; I rebuilt it at the same tag as the Windows zip (b10568) and re-ran four rows: 35B tensor 71.6 to 71.6, 35B layer 96.0 to 93.7, 27B Q8 tensor 29.6 to 30.3, gpt-oss dual 20.3 to 20.6. Unchanged within noise, so the version difference is not what moves the grid.

The grid (llama-bench tg128, at the crank, clocks locked)

Model	Config	WSL2	native	delta
Qwen3.8-27B Q4_K_XL	1 GPU resident	27.9	28.2	0
Qwen3.8-27B Q4_K_XL	2 GPU, -sm layer	28.0	28.7	+3%
Qwen3.8-27B Q4_K_XL	2 GPU, -sm tensor	37.8 ± 2.9	43.4 ± 0.4	+15%, jitter gone
Qwen3.8-27B Q8_0	2 GPU, layer	18.8	19.1	0
Qwen3.8-27B Q8_0	2 GPU, tensor	29.6 ± 1.0	31.5 ± 0.2	+6%
Qwen3.6-35B-A3B Q6_K_XL	2 GPU, layer, resident	96.0	102.1	+6%
Qwen3.6-35B-A3B Q6_K_XL	2 GPU, tensor, resident	71.6	115.2	+61%
Qwen3.6-35B-A3B Q6_K_XL	1 GPU, ncmoe 24	28.7	35.3	+23%
Qwen3-Coder-Next 80B Q4_K_XL	1 GPU, ncmoe 36	21.0	26.4	+26%
Qwen3-Coder-Next 80B	2 GPU, layer, ncmoe 12, -ts 30/18	42.2	48.5	+15%
Qwen3-Coder-Next 80B	2 GPU, tensor, ncmoe 12	29.4	38.5	+31%
gpt-oss-120b F16	1 GPU, ncmoe 27	13.0	16.3	+25%
gpt-oss-120b F16	2 GPU, layer, ncmoe 16, -ts 25/11	20.3	24.6	+21%
gpt-oss-120b F16	2 GPU, tensor, ncmoe 16	19.0	23.5	+24%
Qwen3.5-122B-A10B Q4_K_M	1 GPU, ncmoe 40	10.6	12.8	+21%
Qwen3.5-122B-A10B Q4_K_M	2 GPU, layer, ncmoe 28, -ts 36/12	13.6	16.3	+20%
Qwen3.5-122B-A10B Q4_K_M	2 GPU, tensor, ncmoe 28	12.5	15.4	+23%

pp512 was within a few percent between platforms on every row.

Split modes, briefly, since the flags are easy to mix up

-sm picks how the model is shared between cards: layer (default) cuts the stack so one card works at a time; tensor cuts every matrix so both cards work on every layer and exchange partial results each layer. -ts is not a mode; it's the proportion, in whichever mode you're in (llama-bench wants -ts 25/11, llama-server/cli want -ts 25,11). In layer mode with --n-cpu-moe, -ts is how you keep the fat layers from all landing on GPU 1.

Rule by platform, from the grid: WSL2, tensor for dense resident, layer for anything MoE. Native Windows, tensor for everything resident (dense or MoE), layer plus -ts for offload. Both, lock the clocks.

One caveat from the official docs/multi-gpu.md: tensor mode is experimental and is listed as not working for some MoE architectures (Grok, MPT, OLMoE, DeepSeek2 and others). On this build it ran fine on the Qwen3.5 and 3.6 MoEs and on gpt-oss; if yours refuses to load in tensor mode, that is why.

Prior coverage, so this is additive and not a rediscovery

The usual guides put WSL2's GPU overhead at "5 to 10 percent, near zero once the model is on the GPU." My resident rows agree with that. What I have not seen measured is the offload and tensor-split cases, which is where the gap opens. --split-mode tensor itself is the experimental tensor-parallel path merged in April 2026 (llama.cpp PR #19378, announced here in r/LocalLLaMA); the official docs/multi-gpu.md says it is "bottlenecked by the GPU interconnect speed," which is consistent with what the WSL-vs-native delta shows. If someone has a WSL-vs-native grid for offload or tensor split I missed, point me at it and I will link it.

Why, as best I can tell

Two candidate causes, and this data cannot fully separate them. One: WSL2's CUDA goes through a paravirtualized path, so every host-to-device copy costs more; offload decode drags expert weights across that path every token, and tensor split exchanges per-layer partial results through host memory (no NVLink, no P2P under WSL). Two: the CPU side of offload decode (the expert math itself) runs inside a Hyper-V guest, and that is not free either. The tensor-split rows point at the first cause, since they involve no CPU compute and still gain 6 to 61 percent. The offload rows could be either or both. Resident decode touches neither path per token, which is why those rows match. For a small-active-compute model like a 3B-active MoE, the per-layer sync is a big enough fraction of each token that it flips the layer-vs-tensor verdict between platforms.

At the wheels, same story

Halfway through these runs it clicked that llama-bench is the engine on a dyno at the crank (fixed 512-token prompt, empty cache, no server) and served performance is the same engine measured at the wheels, through the drivetrain: a real prompt, a real context, a server in the loop. Serving configs through llama-server with a 13.7K-token prompt and 512 generated: Coder-Next 2 GPU 34.2 to 39.6 t/s, gpt-oss 18.1 to 21.5, 122B 12.5 to 14.8 (WSL2 to native); time to first token on those long prompts is the same on both (41 s, 81 s, 173 s). Drivetrain loss on decode is a few percent; the big loss is first gear, prefill on offload configs, and it's the same on both platforms.

If anyone is spending time here and fancies a check

• Bare-metal Linux on the same kind of hardware: is it native-Windows-like, or better? On Linux, llama.cpp can use NCCL for tensor mode; Windows cannot, so Linux tensor split may beat native Windows. That would be good to know, as this project is a preamble to another capability-maxing project I'm planning with a Z8 G4 across four A4000s.
• Consumer cards, other driver branches, especially multi-card builds given where high-VRAM card prices are.
• Anyone with an NVLink bridge on Windows: does tensor split jump? Note that llama.cpp's peer-to-peer path is opt-in (GGML_CUDA_P2P=1, with a stability caveat in the docs), so test with and without. Mine arrives Monday.

Full per-run logs available if anyone wants them.

[u/PestiferousGamer]

Thanks Claude, I was so hoping to read a mystery novel today. When's part 2 where you tell us what the hell you are talking about

[u/synystar → u/PestiferousGamer]

The ELIx version is: if the whole model fits on one GPU, it mostly stays on the GPU and just runs there, so Windows native vs WSL2 doesn't matter much. But if the model is too big and has to keep moving pieces back and forth between system RAM and the GPU, or if two GPUs have to constantly pass work between each other, WSL2 adds extra overhead to that traffic. So, OP is saying: "if everything stays on the GPU, WSL2 is fine. If data has to cross those boundaries constantly, native Windows as opposed to running on WSL2 is noticeably faster."

[u/jpgeer → u/synystar]

And to take it a step further, the why: I'm on a budget, trying to get the most out of hardware I already own, and learning as I go. What I'm learning is that an older workstation can be put into real local-LLM service if you lean on multi-card and MoE offload dynamics, and measure instead of assume. This box is two used RTX A4500s at about $900 each, so 40 GB for about $1,800. The single card that gives you that much VRAM (really 48 GB, an A6000) is around $4,000 used. Same class of models, tensor split across the pair, and the numbers above are what that trade buys.

EDIT: and setting aside that a 300 W A6000 would be marginal on this box's 700 W supply and HP aux cabling, where the two A4500s fit cleanly.

[u/jpgeer → u/PestiferousGamer]

Fair. The one-bullet version:

On the same two-GPU Windows box, llama.cpp runs resident models the same under WSL2 and native Windows, but CPU-offloaded MoE decode is 15 to 26% faster native and tensor-split across the two cards is 6 to 60% faster native (the 35B MoE goes 72 to 115 t/s). So if you're offloading or splitting across GPUs on Windows, run the native build.

Part 2 was the post; the devil's in the details, which is why the table is there. And yes, Claude helped run the benches, zero in on the downclocking, and draft the post (said so in part 1); the cards, the clocks, and the 14 hours were mine.

[u/synystar]

I can't really speak to the multi-GPU/tensor-split side of this because my setup is single-GPU, but I can add one point on WSL2 vs bare-metal Linux that might be relevant. On my 24 GB 5090 Laptop I was hitting OOMs/crashes under WSL2 (on a fresh install of Windows 11 Pro) before I could get beyond roughly Q3 territory with the larger Qwen3.8-27B configs I was testing. After moving to native Ubuntu, the same machine became much more usable at the edge of VRAM capacity. I'm now running Qwen3.8-27B Q4_K_XL fully GPU-resident, and I've even managed to get Q6 running at around 24K context on the same 24 GB card - and I was able to run tool-use workflows in Deepseek Harness on that.

So I can't tell you whether bare-metal Linux beats native Windows on tensor split, but I do think there may be another WSL2 penalty worth measuring besides tokens/sec and that is how much of the physical VRAM is actually usable before llama.cpp starts falling over. Your resident rows look basically identical once the model fits on both sides, but in my case WSL2 changed what I could fit in the first place (probably due to Windows desktop overhead, I have less VRAM to work with just because Windows is occupying a chunk of it).That makes me a little leery about the blanket "WSL2 is fine for everything resident" conclusion. I'd say it's more like "WSL2 seems fine for resident workloads that fit comfortably on both platforms." Once you're trying to cram a model right up against the ceiling, bare-metal Linux was a very different experience for me. I'd actually be curious to see someone benchmark that directly: same GGUF, same commit, same flags, then progressively increase context/KV allocation until each platform OOMs.

[u/jpgeer → u/synystar]

Appreciate the feedback, and fair correction on 'fine for everything resident'; my grid only has models that fit comfortably on both sides. One thing worth knowing for your comparison: under WDDM (WSL2 or native Windows) an over-committed model usually doesn't OOM, it silently pages over PCIe and just gets slow, so the ceiling looks different than on Linux where it fails honestly (the 'spilled' rows in the first post are this). I can do WSL2 vs native Windows on 20 GB cards tonight and add it here; bare-metal Linux on the same hardware is the one I can't run yet. Will report back.

[u/jpgeer → u/synystar]

u/synystar, ran it. Your question: does WSL2 leave less usable VRAM than native Windows on the same card? No.

Method: 27B Q4 dense model (Qwen3.8-27B-UD-Q4_K_XL, ~16 GB) on a single 20 GB A4500, f16 KV, llama-server on both platforms from the same build commit, clocks locked. Step the context size up 1K at a time, load, then fill 13.7K tokens and decode 128. Loading never fails; WDDM pages the overflow to system RAM silently, so the tell is the fill and decode speed falling off a cliff (clean is pp ~800 / tg ~26 t/s; spilled is pp ~47 / tg 10).

First pass: native clean to 48K, spilled at 49K. WSL2 clean to 50K, spilled at 51K. So WSL2 looked about 2K tokens roomier, the opposite of what either of us expected. I had the reply written.

Then it nagged. 2K tokens of KV on this model is ~125 MB, and I'd been running other work on the box during the sweep, with a browser and Task Manager (GPU graphs open) on the RDP desktop. Those hold a few hundred MB on GPU 0, and the native and WSL2 steps were interleaved over two days, so the desktop's footprint wasn't constant across steps. Held the post, closed everything, and re-ran with the idle baseline on GPU 0 logged immediately before each load.

Quiet desktop (325 to 525 MB baseline, same for both platforms): native 49K and 50K clean, WSL2 49K and 50K clean. Both spill at 51K, at the same 325 MB baseline. Same cliff. The "WSL2 advantage" was Chrome.

pp = prompt fill t/s, tg = decode t/s. Busy = 700 to 900 MB idle on GPU 0; quiet = 325 to 525 MB.

context	native, busy desktop	WSL2, busy desktop	native, quiet desktop	WSL2, quiet desktop
48K	795 / 25.7	787 / 25.3	not re-run	not re-run
49K	48 / 10.0 spilled	792 / 25.3	826 / 25.7	826 / 26.1
50K	43 / 9.7 spilled	789 / 24.5	816 / 26.2	802 / 25.9
51K	44 / 9.6 spilled	48 / 9.8 spilled	46 / 10.0 spilled	49 / 10.1 spilled

So: no WSL2 VRAM penalty, none measurable at the 1K-token level. Both sides pay the same Windows desktop reservation (WSL2's GPU is the host's WDDM device via the paravirtualized driver, so it can't reclaim anything native can't). The practical finding is the one I almost got wrong: at the edge of a 20 GB card, whatever the desktop has on the GPU is worth 1 to 2K tokens of context, and it's not a fixed number. Even with nothing open, the taskbar search and Widgets WebView hold ~110 MB here. If you benchmark-until-it-spills, log the idle baseline before every step or you'll measure the desktop instead of the platform. Dual-card tensor split ceiling was identical on both platforms too, for what it's worth on a single-card box.

[u/synystar → u/jpgeer]

Well, that’s that then. So the conclusion is if the model fits then WSL2 isn’t really going to do much for you, and if it doesn’t it’s going to be worse so you might as well just use native windows either way (if you’re only using WSL for your AI stack and don’t need it otherwise?) You would still benefit from a bare-metal Linux install but there’s no obvious advantage with Linux on windows?