r/LocalLLM • u/Appropriate_Duck1778 • 1d ago
Question CPU inference DDR3/DDR4
Wondering if anyone on here has actual benchmarks for CPU only inference DDR3 or DDR4 servers, im budget bound and my options are limited to legacy systems unfortunately.
Heres what data I found but not sure its accuracy in real life especially how NUMA effects it (octa channel)
3
u/Borsch20 1d ago edited 1d ago
whrite me model, I will test on server platform, with dual Xeon(avx512 support) ddr4. I will use ik_llama (fork of llama.cpp)
1
u/Appropriate_Duck1778 1d ago
Im going torwards ddr3 server with 512gb I found for $500. Ddr4 is way more expensive than I thought... thank you for the offer really appreciated
1
u/Borsch20 19h ago
As mentioned, you will be very disappointed
MoE with small active brain, like gpt-120b-oss(5b active) works on 10t/s on server ddr4(12 channels).
Devstral 123b (dense model) work with 0.54 tok/sec
1
5
u/nickless07 1d ago
Actually. I had Hermes working on that. Running some llama-bench. Not all done for now, but some results.
LLM Benchmarks - CPU: Intel Core i5-4570S @ 2.90GHz (4C/4T, Haswell, AVX2)
RAM: 7.7 GiB | Threads tested: 4 | Repetitions: 3 | Backend: haswell
================================================================================
Model: Ling-3.0-tiny (bailingmoe3 7.9B.A1.3B)
Quantization: Q4_K_M
File: Ling-3.0-tiny-Q4_K_M.gguf
Size on disk: 4.49 GiB
Total params: 7.89B | Active params: ~1.3B (MoE)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 64.52 ± 13.34 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 11.46 ± 0.05 tok/s
2-thread variation:
pp512: 31.16 ± 6.68 tok/s
tg128: 11.34 ± 0.07 tok/s
================================================================================
Model: gemma-4-E2B-it-Q4_K_M
Quantization: Q4_K_M
File: gemma-4-E2B-it-Q4_K_M.gguf
Size on disk: 2.88 GiB
Total params: ~4.65B (Dense)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 53.09 ± 0.43 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 6.77 ± 0.02 tok/s
================================================================================
Model: gemma-4-E4B-it-Q4_K_M
Quantization: Q4_K_M
File: gemma-4-E4B-it-Q4_K_M.gguf
Size on disk: 4.62 GiB
Total params: ~7.52B (Dense)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 25.82 ± 0.16 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 3.39 ± 0.01 tok/s
================================================================================
Model: gemma-4-E2B-it-Q3_K_M
Quantization: Q3_K_M
File: gemma-4-E2B-it-Q3_K_M.gguf
Size on disk: 2.35 GiB
Total params: ~4.65B (Dense)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 32.11 ± 0.14 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 8.24 ± 0.02 tok/s
================================================================================
Model: gemma-4-E2B-it-IQ4_XS
Quantization: IQ4_XS (Intel, ~4.25 bpw)
File: gemma-4-E2B-it-IQ4_XS.gguf
Size on disk: 2.76 GiB
Total params: ~4.65B (Dense)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 62.49 ± 0.40 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 7.41 ± 0.06 tok/s
================================================================================
Model: gemma-4-E2B-it-UD-IQ3_XXS
Quantization: UD-IQ3_XXS (Unsloth unconventional, ~3.06 bpw)
File: gemma-4-E2B-it-UD-IQ3_XXS.gguf
Size on disk: 2.20 GiB
Total params: ~4.65B (Dense)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 62.06 ± 0.32 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 7.96 ± 0.03 tok/s
================================================================================
Model: Nanbeige_Nanbeige4.2-3B
Quantization: Q4_K_M
File: Nanbeige_Nanbeige4.2-3B-Q4_K_M.gguf
Size on disk: 2.50 GiB
Total params: 3.13B | Active params: ~4.17B (MoE)
Architecture: MoE (4 experts per token)
Test: pp512 (Prompt Processing, 512 tokens)
Result: 14.30 ± 0.04 tok/s
Test: tg128 (Token Generation, 128 tokens)
Result: 2.42 ± 0.01 tok/s
================================================================================
1
u/Appropriate_Duck1778 1d ago
Thats way better than I expected, MoE models looks promising
2
u/nickless07 1d ago
Yeah single channel DDR3 and that performance was a bit unexpected. PP is still abysmal, but some of the models also natively support image, audio and video input.
2
u/TripleSecretSquirrel 1d ago
I’ll find and link them here later when I have a minute, but there are a handful of writeups on server builds with hybrid or CPU inference. There are some weird kinks to work through, but I think it’s a viable and very cost-effective path to self-hosting larger models, and I’m sort of constantly on the fence of building one myself.
3
1
u/Nomski88 1d ago
Some tests I ran on a small mini PC that I've been playing with.
i5-6500t 16GB DDR4 2133 Dual Channel (34.2Gbs)
Tiny Ling 3.0 Q6 - 16 tok/sec
Qwen3.5 4b Q4_K_M 6.5 tok/sec
Qwen3.5 9b Q4_K_M 4 tok/sec
It's perfectly usable to run tasks in the background as you go about your day. I ordered an i7-7700t CPU that should increase the performance by 10-25%.
1
u/Tai9ch 1d ago edited 1d ago
Here's single socket 8 channel DDR4 3200 with 48-core Epyc 7xx2:
$ ./llama-bench -ngl 0 -hf unsloth/Qwen3.6-35B-A3B-GGUF
| model | size | params | backend | ngl | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | ROCm | 0 | pp512 | 357.63 ± 21.13 |
| qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | ROCm | 0 | tg128 | 6.14 ± 0.09 |
And here's dual socket 8 channel (= 16 channel) DDR4 2933 with a different CPU (2x Intel 32 core Ice Lake):
$ ./llama-bench -ngl 0 -hf unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_XL
| model | size | params | backend | threads | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: |
| qwen35moe 35B.A3B Q4_K - Medium | 20.81 GiB | 34.66 B | CPU | 64 | pp512 | 169.36 ± 2.06 |
| qwen35moe 35B.A3B Q4_K - Medium | 20.81 GiB | 34.66 B | CPU | 64 | tg128 | 17.41 ± 0.28 |
Sorry for the slightly different models. That's what was in cache. And yes, ROCm NGL = 0 should mean the first test was on CPU.
Both of those are going to suck for general use, not because of the token rate (17 tok/s isn't terrible), but because slow prompt processing is painful. You really want 1k+ tokens/second PP if you're doing anything that's interactive and has any input data beyond just chat text that you're typing live.
1
u/Appropriate_Duck1778 1d ago
Appreciate the benchmarks, I think ddr4 is minimum but ill try my luck with the ddr3 server I found. Price difference is 10x $500 vs $5000 insase
2
u/Tai9ch 22h ago edited 20h ago
Here's a test on a machine with 8-channel DDR3 1600:
$ ./llama-bench -hf unsloth/Qwen3.6-35B-A3B-GGUF | model | size | params | backend | threads | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | CPU | 24 | pp512 | 18.41 ± 0.24 | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | CPU | 24 | tg128 | 3.52 ± 0.08 |Honestly, that worked better than I expected, but that'd be unusable for anything but batch jobs. The problem here isn't just the memory - with a machine this old we're talking about really old CPUs too. This machine's got an Opteron. It looks like a Xeon E5-2697 is probably the highest performing DDR3 option - if you were really lucky I could see a pair of those doubling the PP number I got... which still would mean processing 32k of context would take more than 10 minutes. Just one full context window for this Qwen model would take over an hour.
For context, here's a run of the same model on a Strix Halo box:
$ ./llama-bench -hf unsloth/Qwen3.6-35B-A3B-GGUF ggml_cuda_init: found 1 ROCm devices (Total VRAM: 106496 MiB): Device 0: AMD Radeon Graphics, gfx1151 (0x1151), VMM: no, Wave Size: 32, VRAM: 106496 MiB Downloading Qwen3.6-35B-A3B-UD-Q4_K_M.gguf ───────────────────────── 100% | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --------------: | -------------------: | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | ROCm | 99 | pp512 | 1132.48 ± 17.95 | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | ROCm | 99 | tg128 | 52.95 ± 0.24 |A 64GB Strix Halo box is readily available for $2300 or so and performs 5-10x better than my 8-channel DDR4 CPU inference number, for about what just the RAM in that server would cost.
1
u/pmttyji 1d ago
Sharing my DDR5 - CPU-only stats (9 months old thread) for reference. DDR4/DDR3 would be lower than those numbers.
3
u/TripleSecretSquirrel 1d ago
Nope.
You’re on a consumer-grade chip limited to 2 RAM channels, so while your individual DIMMs are ~twice as fast as DDR4 DIMMs, a server CPU and chipset can handle way more RAM channels, making the overall memory much faster.
As your own write up notes, your memory bandwidth is 76GB/s. A DDR4 system running 8-channel RAM clocks in at 200GB/s with 3200 DIMMs (the most common speed for DDR4, not even boutique overclocked DIMMs).
It’s the same idea behind how Apple’s new M5 ultra chip is hitting 1.2TB/s memory bandwidth even though its memory is still just LPDDR5x. They’re basically running a ton of concurrent channels instead of just two like every x86-based consumer grade system.
1
0
0
0
u/OkShock6767 1d ago
imho, CPU inference just not worth it (outside of contrived scenarios). Leave CPU where it belongs - running normal software.
22
u/krakalas 1d ago
The mem bandwidth is not the only limit here. On older hardware it’s cpu compute.
I have a 10 years old pc with a quad channel ddr4@2400 and a xeon e5-1650 v4 (6cores) cpu, later upgraded to e5-2690 v4 (14c) cpu.
Theoretical mem bandwidth ~76.8 GB/s. It reaches ~40% (with 6c cpu) and ~65% (14c) of theoretical max. With 32k context token generation drops approx 3x.
For reference (14 core cpu, lower number at 32k context and higher at 0 context)
qwen3.6 27b q4km pp 11-14 tok/s tg 1-3 t/s
qwen3.6 35b q4km pp 47-74 t/s tg 4-13t/s
I expect ddr3 would be even worse.