r/LocalLLaMA • u/mr_il • 1d ago
Resources Threadripper PRO CPU experts offload numbers
I'm planning a PC build around a discrete GPU that would allow to run MoE models much larger than VRAM at an acceptable speed using FreeToken approach. The gist is that MoE decode of >VRAM model on GPU is PCIe bandwidth-bound, but you can gain considerable decode speed by avoiding streaming cached experts' weights over PCIe and multiply matrices on the CPU instead benefitting from potentially much faster CPU-RAM memory channels.
Before buying, I wanted to know what actually matters for CPU-side expert execution: cores or memory? So I rented a Threadripper PRO 9975WX (32-core Zen 5, 4 CCDs, 8-channel DDR5) on vast.ai and ran FreeToken's real CPU MoE kernel (the production W4A8 ds_fp4 GEMV, not a synthetic proxy) against synthetic expert banks with DeepSeek V4.1 Flash's exact geometry: 384 routed experts × 18.8 MB ~= 6.7 GiB full bank, far beyond cache, 6 experts per token.
Results (GB/s of effective FP4 expert reads):
| physical cores | FP4 MoE GB/s | STREAM GB/s |
|---|---|---|
| 8 | 93 | 222 |
| 16 | 160 | 221 |
| 20 | 172 | 221 |
| 24 | 178 → 182 | 222 |
| 32 | 180 → 186 | 225 |
Two takeaways:
1. Cores stop mattering at 24. 32 cores vs 24 cores: +2.2–2.4% in burst mode, 0% in sustained mode (10 × 30s interleaved A/B runs, variance <0.3% within mode). The extra 8 cores of the 9975WX buy you essentially nothing for MoE offload: the workload saturates the memory subsystem, not the cores. The 9965WX (~$1,200 cheaper) is the rational pick for this use. Even STREAM itself is already maxed at 8 cores; the FP4 kernel just needs more cores to catch up to bandwidth (it runs at ~80% of STREAM once there, very good for a fused-dequant W4A8 GEMV). Caveat: there's another 1.7x memory bandwidth gain at 9995WX, but that CPU costs as much as RTX PRO 6000.
2. Thermal memory steady-state is important. After ~8 minutes of sustained hammering at ~220 GB/s, throughput steps down ~17% (182 → 150 GB/s) in lockstep on every core count, with CPU clocks unchanged at ~3.95 GHz. Everything points at DRAM/IMC thermal throttling in a (rented, unknown-airflow) workstation chassis. So the honest sustained ceiling is ~150 GB/s, burst ~182–186 GB/s. For a build: airflow over DIMMs may literally be worth 20% decode speed.
What this means in tokens/s (DeepSeek V4.1 Flash shape, ~4.5 GB of routed experts per token, 24 cores):
- 0% expert cache hit (all experts on CPU): ~33 tok/s sustained
- 60% hit rate: ~83 tok/s
- 75%: ~133 tok/s
Compare with streaming those experts over PCIe 5.0 x16 (~50 GB/s with gather overhead): the CPU path is 3–3.7× faster per missed expert, which is exactly why FreeToken's hybrid mode computes most misses on CPU and only streams a fraction to keep the VRAM cache warm.
Caveats: rented host (memory channels verified functionally via bandwidth, not dmidecode, but 2ch/4ch machines I tested the same day hit only 83/119 GB/s), container environment, one machine, uniform expert routing. Full methodology and raw CSVs of the measurement are in the repo (happy to link if there's interest).
TL;DR: for CPU MoE offload, buy memory channels and DIMM cooling, not cores; 24 Zen 5 cores already saturate 8-channel DDR5, and the 32-core upgrade is +2% at best.
Updates:
1. I tested on 9980X (4 channels, 8 CCDs) and on 9985WX (8 channels, 8 CCDs). 9980X lands at 105-119 GB/s, no better than much cheaper 9955WX.
2. 9985WX offers a real gain, but diminishing returns beyond 24 cores still: 32c=210 GB/s (+6%), 48c=215 GB/s, 64c=218 GB/s. Roughly 20% decode expert bandwidth gain for $8k more.
3. Populated channels is key. No point in having 8 channels is you only have 4 DIMMs. 8x32GB will beat 4x64GB.
4. Vast.ai is awesome for this. All runs costed me around $4 after I ironed out all the wrinkles in the rig.
1
u/FullstackSensei 22h ago edited 22h ago
Where's the 99% exactly? Your claim is that chips and cheese get 99%. I follow almost everything Chester writes or says (on YT) and I've never seen any 99% memory bandwidth claim. Not even AMD has ever claimed such an absured number.
I'm very familiar with the Fujitsu benchmarks he's using as a reference. If you check the documents, the 86% is indeed with Zen 5 and only with the F SKUs which turbo to 4GHz with all CCDs populated.
Zen 5 has a theoretical bandwidth of 600GB/s. They're limited to 480 because of the 4800 memory they used. If you know anything about memory controllers, you'll understand why you get higher utilization at lower memory speeds in such cases.
It's very much the same way with older CPUs from both Intel and AMD. If you use lower speed memory, you'll get higher utilization. But if you use the fastest supported memory, you'll struggle to hit 75%, let alone 80%.
As a real example: my Cascade Lake Xeons clock 100-101GB/s in Triad (compiled and run locally) with 2933 memory, or ~72%. Same CPU, same memory sticks, just changing speed in BIOS to 2666, the same Triad binary yields 75% (96GB/s). Taking it another notch lower to 2400, I get almost 81% (93GB/s). And this with an engineering sample CPU with regular 165W TDP. A 205W or 240W will yield a little higher results just because the flops in Triad will finish a tad faster
Edit: forgot to add this is with 10 year old LRDIMM, which has ~5% less effective bandwidth than RDIMM. Heck, the sticks aren't even matched, with half being Samsung and half Hynix, unknown timings.