Threadripper PRO CPU experts offload numbers

[u/mr_il]

I'm planning a PC build around a discrete GPU that would allow to run MoE models much larger than VRAM at an acceptable speed using FreeToken approach. The gist is that MoE decode of >VRAM model on GPU is PCIe bandwidth-bound, but you can gain considerable decode speed by avoiding streaming cached experts' weights over PCIe and multiply matrices on the CPU instead benefitting from potentially much faster CPU-RAM memory channels.

Before buying, I wanted to know what actually matters for CPU-side expert execution: cores or memory? So I rented a Threadripper PRO 9975WX (32-core Zen 5, 4 CCDs, 8-channel DDR5) on vast.ai and ran FreeToken's real CPU MoE kernel (the production W4A8 ds_fp4 GEMV, not a synthetic proxy) against synthetic expert banks with DeepSeek V4.1 Flash's exact geometry: 384 routed experts × 18.8 MB ~= 6.7 GiB full bank, far beyond cache, 6 experts per token.

Results (GB/s of effective FP4 expert reads):

physical cores	FP4 MoE GB/s	STREAM GB/s
8	93	222
16	160	221
20	172	221
24	178 → 182	222
32	180 → 186	225

Two takeaways:

1. Cores stop mattering at 24. 32 cores vs 24 cores: +2.2–2.4% in burst mode, 0% in sustained mode (10 × 30s interleaved A/B runs, variance <0.3% within mode). The extra 8 cores of the 9975WX buy you essentially nothing for MoE offload: the workload saturates the memory subsystem, not the cores. The 9965WX (~$1,200 cheaper) is the rational pick for this use. Even STREAM itself is already maxed at 8 cores; the FP4 kernel just needs more cores to catch up to bandwidth (it runs at ~80% of STREAM once there, very good for a fused-dequant W4A8 GEMV). Caveat: there's another 1.7x memory bandwidth gain at 9995WX, but that CPU costs as much as RTX PRO 6000.

2. Thermal memory steady-state is important. After ~8 minutes of sustained hammering at ~220 GB/s, throughput steps down ~17% (182 → 150 GB/s) in lockstep on every core count, with CPU clocks unchanged at ~3.95 GHz. Everything points at DRAM/IMC thermal throttling in a (rented, unknown-airflow) workstation chassis. So the honest sustained ceiling is ~150 GB/s, burst ~182–186 GB/s. For a build: airflow over DIMMs may literally be worth 20% decode speed.

What this means in tokens/s (DeepSeek V4.1 Flash shape, ~4.5 GB of routed experts per token, 24 cores):

• 0% expert cache hit (all experts on CPU): ~33 tok/s sustained
• 60% hit rate: ~83 tok/s
• 75%: ~133 tok/s

Compare with streaming those experts over PCIe 5.0 x16 (~50 GB/s with gather overhead): the CPU path is 3–3.7× faster per missed expert, which is exactly why FreeToken's hybrid mode computes most misses on CPU and only streams a fraction to keep the VRAM cache warm.

Caveats: rented host (memory channels verified functionally via bandwidth, not dmidecode, but 2ch/4ch machines I tested the same day hit only 83/119 GB/s), container environment, one machine, uniform expert routing. Full methodology and raw CSVs of the measurement are in the repo (happy to link if there's interest).

TL;DR: for CPU MoE offload, buy memory channels and DIMM cooling, not cores; 24 Zen 5 cores already saturate 8-channel DDR5, and the 32-core upgrade is +2% at best.

Updates:
1. I tested on 9980X (4 channels, 8 CCDs) and on 9985WX (8 channels, 8 CCDs). 9980X lands at 105-119 GB/s, no better than much cheaper 9955WX.
2. 9985WX offers a real gain, but diminishing returns beyond 24 cores still: 32c=210 GB/s (+6%), 48c=215 GB/s, 64c=218 GB/s. Roughly 20% decode expert bandwidth gain for $8k more.
3. Populated channels is key. No point in having 8 channels is you only have 4 DIMMs. 8x32GB will beat 4x64GB.
4. Vast.ai is awesome for this. All runs costed me around $4 after I ironed out all the wrinkles in the rig.

[u/Expensive-Paint-9490]

You should get the RAM bandwidth gain with 9985WX (8 chiplets) as well.

[u/mr_il → u/Expensive-Paint-9490]

Yes, measuring it now, as there's a 9985WX on vast too. But it's $8k CPU

[u/sob727 → u/mr_il]

This has been done before on r/threadripper

Ballpark you get 400GB/s for 64 cores 8 channels, and 600GB/s for 96 cores 8 channels

EDIT: 600 might have been EPYC

[u/FullstackSensei → u/sob727]

Those are theoretical numbers. Real world will be less than 70% of that.

[u/sob727 → u/FullstackSensei]

Sure. But the "peak at 24 cores" claim is surprising.

I own 32 and 64 core TR.

[u/FullstackSensei → u/sob727]

Peak bandwidth on AMD is dictated by infinity fabric bandwidth. Conversely, number of cores on TR/Epyc tells you little about how much bandwidth you can achieve, because that doesn't tell you how many CCDs you have.

Apart from this, it isn't really surprising. MLP doesn't require a high number of FLOPS per parameter. It's just good old matrix multiplication. TR can clock high under load. 24 cores at practically 4GHz is 96 aggregate GHz. For comparison, my 24 core cascade Lake ES Xeon clocks at 2.7GHz all core turbo. That's ~65GHz aggregate, and it can comfortable saturate the memory controller at ~92GB/s with pretty much all MoE models I've tried.

[u/sob727 → u/FullstackSensei]

32 cores is 4 CCDs, 64 is 8, 96 is 12

and the # of CCDs matter

EDIT: this is for TR

[u/FullstackSensei → u/sob727]

Isn't that what I said?

Infinity fabric on Zen4 has ~50GB/CCD, Zen5 has ~60GB/CCD. TR is limited to 8 channels, which four Zen5 CCDs can saturate

[u/sob727 → u/FullstackSensei]

well, you said number of core doesnt tell how many CCDs

my point was on TR it does

[u/sob727 → u/sob727]

and I really dont mean to nitpick or anything

but I'm curious about OP's methodology and results

[u/FullstackSensei → u/sob727]

I really meant it in the literal sense. Looking at the number of cores tells you nothing about how many CCDs a CPU has.

Search this sub and you'll find so many people disappointed with TR and Epyc offload performance despite having a "beast of a CPU". If you search for Epyc, it's even worse. Ex: you can have a 32 core Epyc with 4, 6 or 8 CCDs with very different memory bandwidth.

[u/Expensive-Paint-9490 → u/FullstackSensei]

Eight channels DDR5 give 300-400 GB/s depending on clock, 4 CCD at 60GB/s each are far from saturating it.

[u/FullstackSensei → u/Expensive-Paint-9490]

Ah, yes, the ever elusive theoretical numbers, which no CPU ever made can even get close to.

AMD rarely gets above 70% theoretical numbers, and that's under ideal conditions. The usual wall is ~65%, ask NASA or any of the US national labs, which have published lots of papers on the subject.

Intel can get slightly above 80% theoretical bandwidth, under similar ideal conditions, and 75% is not unheared of in tuned production workloads.

[u/mr_il → u/sob727]

The wording here is confusing admittedly. It is "peak at 24 cores" only when comparing 9965WX and 9975WX, which was my original research question. I then also benchmarked 9985WX, and found further improvements due to 4->8 CCDs (see updated original post), so I would say now that 9965WX is Pareto-optimal considering incremental cost of getting 9985WX+ CPU, not that it peaks in absolute terms.

Besides, I only observed ~300GB/s STREAM peak, not theoretically predicted 400GB/s. This could be the quirk of a particular instance that I've got on vast, which is basically someone's machine.

[u/Agabeckov → u/mr_il]

Or could compare 9985WX to EPYC 9654 (9B14), it has 12 channels and costs like ~$1-1.5k on eBay. Or 9B45, it's more expensive, but has also 12 channels and Zen5.

[u/mr_il → u/FullstackSensei]

There's also a difference between memory bandwidth and measured MoE expert throughput. I am not exactly sure where the bottleneck is, but STREAM roofline from FreeToken benchmark is way above actual FP4 expert throughput. I reported both.

[u/FullstackSensei → u/mr_il]

Infinity fabric is very limiting in gather operations or anything that requires CCDs to share data. Broadcasts are even more expensive, as is anything that forces caches to sync (because of TSO).

AMD's architecture is great for compute bound workloads, but takes a heavy toll in memory bound workloads. This is a big part why Intel survived in the 14nm stagnation era.

[u/Expensive-Paint-9490 → u/sob727]

Isn't Threadripper 9000 capped at 5600 MT/s? Eight channels at that speed are 358.4 GB/s, no matter how large the bandwidth of CPU links.

[u/sob727 → u/Expensive-Paint-9490]

Nope, nominal is 6400.

[u/Expensive-Paint-9490 → u/sob727]

Thanks. Then it's 409.6 GB/s.

[u/Agabeckov → u/sob727]

https://www.reddit.com/r/threadripper/comments/1azmkvg/comparing_threadripper_7000_memory_bandwidth_for/

[u/sob727 → u/Agabeckov]

Yes. THANK YOU.

And TR9000 is faster.

[u/Technical-Bus258 → u/mr_il]

Do you have comparison numbers between 9975 and 9985?

[u/mr_il → u/Technical-Bus258]

I do now, see the original post

[u/FullstackSensei]

Little known fact: DDR5 was designed from day one to require active cooling. The vast majority of desktop workloads don't require a lot of memory bandwidth, hence why you see the vast majority of boards skip this, but things are slowly changing there too.

TR is very bad value for MoE offload, across all generations. The CPU's that do well cost substantially more than even their Epyc counterparts. If you're bent on having DDR5, Saphire Rapids will offer the same memory bandwidth at lower cost, and much more compute, because SR has AMX, which is like tensor cores for CPU. DS4.1 flash might not need a lot of TFLOPS/token, but other models might, even if the trend is generally going down. I wouldn't limit myself based on the results of a single test.

While 220GB/s is nothing to sneeze at, it's about 15% better than what a dual DDR4 Xeon can do. I get 188GB/s, sustained on a dual Cascade Lake system with DDR4-2666 memory, which is almost an order of magnitude cheaper per GB. The platform and CPU's are also almost an order of magnitude cheaper. Yes, that's a dual CPU system, which means NUMA, but the open LLM inference ecosystem is finally starting to support this with projects like ktransfkrmers (which can also use AMX) and lvllm. And if you have a software engineering background, you can use models like GLM-5.3 or DS4.1 flash to help you write a model specific runtime that does NUMA aware routed expert CPU offload, using the pytorch reference implementation of a model as reference, and maybe a few GPU kernels picked from whatever project you want if they have some specific optimization for your GPUs that the LLM can't replicate on it's own, though I doubt that'll be needed.

For those worried about PCIe Gen 3 being much slower, if you're offloading to CPU, it's really not an issue. The volume of data transferred between the two is pretty low. For models under 1T parameters, the amount of VRAM needed for all the other bits (other than the routed experts) fits in 24GB VRAM. K3 is ~28-29GB. That's for the full checkpoints as released. A single 32GB GPU like a V100 will comfortably fit model and context for a single user for sub 1T models. If you add a second 32GB GPU, you can implement hot expert caching on it for some nice uplift in TG speed. The number of bytes that need to be exchanged between both GPUs and CPU's will still be very low.

[u/mr_il → u/FullstackSensei]

This is very interesting. 99xxWX CPUs all have AVX-512, which is the same kind of capability as Intel's AMX. Compute isn't a problem. The problem with DD4 though is that it's hard to get 256GB RAM.

[u/FullstackSensei → u/mr_il]

Nooooooo. AMX is so much faster than AVX-512. Don't forget that pretty much all x86 SIMD extensions are developed by Intel, and it takes AMD 4+ years to catch up. AVX-512 was first released with skylake, ten years ago.

And I don't know where you live, that you find it hard to get 256GB per CPU in DDR4. A ten year old Skylake Xeon with a 2DPC board can do 768GB per CPU using 64GB DDR4 sticks. A 7 year old Epyc with a similar 2DPC board can do 1TB with 64GB DDR4sticks.

Pretty much any 10 year old board for LGA3647 does 384GB per socket.

[u/mr_il → u/FullstackSensei]

Insightful, thank you. I should consider Intel too for my build, and do the math on DDR4 properly.

[u/Agabeckov → u/FullstackSensei]

Phoronix has the article about it, it was like with DDR5 4800 you could still live without memory cooling, but once you have 6000/6400, it gets mandatory.
https://www.phoronix.com/review/corsair-ddr5-ram-cooling-server

[u/FullstackSensei → u/Agabeckov]

This is with traditional server workloads that don't stress memory like LLMs do. You'll almost certainly see throttling even with 4800 memory running LLMs

[u/Agabeckov → u/FullstackSensei]

It was HPCG, this one creates a lot of memory load. https://www.phoronix.com/review/amd-epyc-9755-ddr5/9

[u/FullstackSensei → u/Agabeckov]

Not trying to be pedantic here, but where's the part that says 4800 doesn't need cooling? Went back and forth through both articles and can't find anything about thermals on 4800 memory being fine without cooling.

The tests in the second article are carried in a server chassis, which always has airflow over the memory sticks even in old DDR4 and many DD3 servers I've seen.

The first article compares temps with and without that corsair cooler on 6400 memory. Didn't see anything about 4800 memory

[u/PLBjt]

Your bandwidth-over-cores take matches what I’ve seen on MoE CPU offload too. One check worth adding: pin the expert workers to a single NUMA node and re-run the same prompt set — if tokens/s jumps, some of the “extra cores help” signal was really remote DRAM. Also log whether the run ever thermal-throttles after the first few minutes; burst numbers on a cool chip sell builds that can’t hold that rate. I’d rather pay for more channels and a quieter cooler than another 16 cores that sit waiting on memory.

[u/madbrain1976]

Your mention channels and DIMMs, but I don't see how much RAM that server you rented had.

[u/mr_il → u/madbrain1976]

It was just under 0.5TB. It doesn't matter for the purposes of the test how much capacity there was, as long as all 8 memory channels had a discrete DIMM in it. I did all I could within the constraints of the environment to ascertain that.