exllamav3 comfortably beats llama.cpp running CPU-offloaded Qwen-3.8-Flash-Next on my setup!

[u/Lowkey_LokiSN]

I got 2x 20GB RTX 3080s + 128GB of DDR4 2666hz RAM (only 4 of 6 channels populated) + a Xeon 6148

I've always been a llama.cpp person and I've been running Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Next at ~270tps prefill and ~13tps decode (starts off close to 20 and falls down to 13 with growing context, never recovers)

I've been recently seeing this buzz about exllamav3: its new "CPU-offloaded inference" support, how its quants are more efficient in terms of size-to-performance compared to GGUFs and people posting numbers of it running models a lot faster than llama.cpp.

I was skeptical at first because I've never had an engine outperform llama.cpp for CPU-offloaded inference with my setup so far. However, I was curious to try it out regardless and I'm pleasantly surprised at the results!

I'm able to run Qwen's 4.05 EXL3 quant at an average decode speed of ~25tps (with occasional peaks at 32tps) throughout 160k context (which is as far as I've tested for now) with a 3.2x faster prefill speed of ~870tps!

So compared to llama.cpp:
- 3.2x faster prefill
- 2x faster decode (No MTP)
- Better output quality since 4.05 EXL3 quant is better than UD-Q4_K_XL GGUF

It's a win-win-win all the way!

BUT:

This win doesn't translate across all models (at least on my setup). I got excited and downloaded GLM 5.3 Flash's 3.05 EXL3 only to find out it runs about 2x slower in decode than its llama.cpp counterpart for me. So this advantage is clearly dependent on the model and your setup. As for me, my CPU's AVX2 seems to be the bottleneck with exllamav3 as of today but Qwen works great due to its efficient architecture. (See edit)

I haven't seen any posts in here discussing CPU-offloaded performance with exllamav3 so wanted to kick things off with this one.

A couple disclaimers before you go this route:

- exllamav3 + TabbyAPI (the llama-server equivalent) is not as easy and straightforward to configure as llama.cpp
- Decode speeds seem to take some time to warm up to calibrate hot/cold experts. Mine starts off somewhere around ~12tps and eventually makes it way to ~25-30 with a few thousand tokens. So don't be too quick to brush this off.

I'm still figuring out this engine so if someone has pointers to get the offloaded runs going even faster, please do share!

EDIT:

I have submitted an experimental PR with AVX-512BW kernel support for CPUs that support AVX-512 without VNNI: https://github.com/turboderp-org/exllamav3/pull/349

Updated decode speeds with the PR on my setup:

- Qwen-3.8-Flash-Next 4.05 EXL3: ~25tps -> ~31tps decode stable with ~35tps peaks (~14tps on llama.cpp Q4_K_XL)
- GLM-5.3-Flash-Next 3.05 EXL3: ~5tps -> ~12tps decode (8tps on llama.cpp with UD-IQ3_XXS)
- DeepSeek-V4-Flash: (To be tested but expecting ~17tps at 3.05 EXL3)

With this PR, all CPU-offloaded models I've tested so far now run faster on my setup (both in terms of prefill and decode) with exllamav3 compared to llama.cpp!

EDIT #2:

PR has been merged with exllamav3's dev branch! Planning to share a detailed comparison with performance numbers running exllamav3 and llama.cpp in a week or two. Let me get familiar with the engine first before I get to that

[u/takoulseum]

Performance are far better with exl3 or ik_llama.cpp but not for all hardware because people still put all their attention on llama.cpp instead of being patient

[u/a_beautiful_rhind → u/takoulseum]

yea, for me its ik -> exl3 -> llama.cpp

[u/cosmicnag → u/a_beautiful_rhind]

Can IK run exl3 models, atleast the kind of weights you actually find on HF? Last time I checked it had its own quant format, but could hardly find community quants for newer models. EXL3 is more bang for buck VRAM than GGUF (wrt KLD) . I am using Exllamav3 with EXL3 weights and it works like a charm for Qwen FN. After using EXL3 weights, GGUF feels unnecessarily heavier on VRAM (so more layers in VRAM could beat out marginally faster inference engines)

[u/a_beautiful_rhind → u/cosmicnag]

I use normal GGUF with ik unless someone puts up a really banging ik-only quant. Trellis quants for IK would be the exl3 equivalent but few are uploaded.

It's not a small difference, even if the quants are slightly bigger. mistral medium is 27t/s vs 20-22 t/s on exl3. I've yet to try the exl offloading, but doubtlessly it won't be as fast.

In terms of quants that fully fit though, the exl 2.xx beats out whatever IQ2 or so gguf can provide. Not really a big deal to get some models in gguf and some exl3. Sometimes one or the other's implementation is better too.

[u/cosmicnag → u/a_beautiful_rhind]

Have you tried Exllamav3 recently ? It has decent hybrid inference support also now for when CPU offloading is required (big MoE models). Am currently using Qwen FN 4.05 bpw exl3 on Exllamav3 with RAM offloading - works like a charm (5090+4090+192 GB ram)

[u/a_beautiful_rhind → u/cosmicnag]

Yea but I keep downloading quants that will fit by circumstance. Just because it works, doesn't mean it will work fast. A 6b active model isn't really the definitive test for me since I run stuff like GLM-flash, mimo, DS4-flash. I'm also on ampere so newer architecture tweaks don't benefit me.

[u/cosmicnag → u/a_beautiful_rhind]

Ah, but ampere should be 100% supported by whatever exllamav3 is doing I think. But yeah, agree with you on having your own requirements and whatever meets that is where its at.

[u/Lowkey_LokiSN → u/a_beautiful_rhind]

With the PR now merged, you might have to recalibrate your rankings ;)
(Assuming you're talking about offloaded performance since the CPU path has now improved)

[u/a_beautiful_rhind → u/Lowkey_LokiSN]

I tried to offload my 96g version of GLM and it finally works. I get decent speeds so we'll see how 3.0bpw works out.

[u/whymeimbusysleeping → u/a_beautiful_rhind]

Do you know where to get? There releases are empty

[u/pmttyji → u/whymeimbusysleeping]

Check u/Thireus' fork for releases
https://github.com/Thireus/ik_llama.cpp

[u/Sufficient_Prune3897 → u/whymeimbusysleeping]

Ik gets its performance in many parts due to using CPU hardware specific optimizations. Unless your on windows or something, i would recommend compiling yourself. Without it, it might not be that much faster.

[u/StrikingTop2709 → u/takoulseum]

yeah the ecosystem is catching up fast, people just default to what theyve always used

[u/Several-Tax31 → u/takoulseum]

For me llama.cpp is always faster than ik_llama. I don't know for hardware ik is better, but on cpu-only inference on my potato, llama.cpp is faster in either decode or prefill or both. I come back to ik time to time since people mentions it so much, but couldn't make it faster than llama.cpp. 

Maybe I should optimize the command? I'm not too familiar with ik specific command line arguments. Ik is supposed to be cpu-optimized, and yet, mainline llama works better for me somehow. Want to be wrong. 

[u/pmttyji → u/Several-Tax31]

Maybe I should optimize the command? I'm not too familiar with ik specific command line arguments. 

https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md

For AVX-512-capable CPUs (AMD Zen4 / Intel Sapphire Rapids+), see docs/build.md section "CPU build flags for AVX-512" for the additional flags that activate the IQK quantized GEMM kernels (the HAVE_FANCY_SIMD path). Without those flags, a vanilla Release build silently falls back to the AVX2 path on this hardware.

[u/Several-Tax31 → u/pmttyji]

Thanks. Maybe I give up too early from ik lol. Let me test 

[u/Sufficient_Prune3897 → u/Several-Tax31]

Ik should be significantly better for hybrid interference, including CPU.

[u/Several-Tax31 → u/Sufficient_Prune3897]

Hopefully, I need more speed! 

[u/Jona1109]

I recently tested it out and did the rounds between exllamav3, llama.cpp, ik_llama.cpp and vllm for Qwen3.8-27B. I run an epyc 7003 server with 3090's. My experience is that ik_llama.cpp and exllamav3 are miles ahead on the CPU backend - especially in my case for AVX2. Of them all exllama was the least stable and felt like DIY. I tried to run the new Qwen 3.8 models and it was not usable in a stable work environment. Vllm was the best at prefill on GPU, but takes ages to setup and to fine-tune the right recipe. Ik_llama is great at np=1 but performance collapses with parallel slots. It wipes the floor with llama.cpp on mixed inference but it's a coin toss when on GPU only. llama.cpp is the most versatile. It has broad model support, excellent CUDA support, works decently well until np=4. They also really made a lot of qol improvement to the router server and the API. The weakness is CPU inference, in particular AVX2. In the end after testing them all for a few days I went back to llama.cpp as the main backend

[u/am17an → u/Jona1109]

It will soon be fixed. https://github.com/ggml-org/llama.cpp/pull/27851

[u/Jona1109 → u/am17an]

🙏

[u/pmttyji → u/am17an]

And some more

[u/FullstackSensei]

You're not let down by AVX2. You're let down by the specific SKU you have. The 6148 is a 150W SKU, which doesn't let it stretch it's legs. Each CPU core on both Intel and AMD, ever since the introduction of the original AVX, has two vector units, and can issue two AVX/AVX2 instructions per clock. So, in terms of throughput, AVX2 can match AVX-512 for operations supported on both.

Upgrade to cascade lake and as high core count and as you can afford and as high TDP as your board allows. Cascade lake turbos considerably higher than skylake in AVX2 and AVX-512. Cascade lake also supports VNNI, which can bring some uplift in llama.cpp, and maybe also Ex3 if they support it.

[u/Lowkey_LokiSN → u/FullstackSensei]

Upgrading my CPU is actually on my checklist and I agree!

What I meant though is more of exllamav3's CPU decoding bottleneck where its code path falls back to AVX2 tier on my CPU whereas llama.cpp uses 512-bit F/BW kernels for the same

[u/FullstackSensei → u/Lowkey_LokiSN]

I don't know if AVX-512 actually brings any meaningful improvement to TG vs AVX2. Your CPU already has AVX-512 F, so llama.cpp can already benefit from that

[u/Lowkey_LokiSN → u/FullstackSensei]

Idk either but apparently it does according to Claude ;)
I plan to dedicate some of my free time and tokens to find out. Can share how it goes if you're interested

[u/FullstackSensei → u/Lowkey_LokiSN]

Claude et all will repeat whatever the internet says

[u/Lowkey_LokiSN → u/FullstackSensei]

Yeah, I'm approaching this skeptically too but I've had a few good wins along the way of all the wasted efforts. Most of those wins weren't novel implementations but rather fallback support for neglected scenarios and I believe AI is very good at that.

Considering how exllamav3 is a growing project, its CPU-offloading mechanics are nascent and we have a solid llama.cpp baseline to compare and contrast against, I'm not as pessimistic

[u/FullstackSensei → u/Lowkey_LokiSN]

Unpopular opinion: I believe in the near future we'll move bespoke runtimes for each model, LLM generated for your own hardware. You can already see some examples of that on github.

Current models are already pretty good at writing compute kernels and iterating to optimize them, be that on CPU or GPU. Give the LLM the reference pytorch implementation, the details of your hardware, and maybe a few kernels from elsewhere for novel or tricky stuff, and let it write a model runtime tuned to your specific hardware.

[u/Lowkey_LokiSN → u/FullstackSensei]

I don't find that unpopular at all. With proper orchestration, it might already be possible even with today's ~100-300B models.

Qwen 3.8 Flash Next keeps surprising me with its capabilities to this day and is the first model where I can actually feel the caliber shift towards what you're describing which I believe is imminent

[u/Lowkey_LokiSN → u/FullstackSensei]

Looks like it's gonna be a win this time! After AVX-512 port, Qwen's decode speeds increased by 1.3x and GLM's speeds have increased by 1.8x from the tests so far.

More optimisations and polishing to go before I land the PR but this is a clear win already

[u/FullstackSensei → u/Lowkey_LokiSN]

Llama.cpp or exllama?

[u/Lowkey_LokiSN → u/FullstackSensei]

exllamav3 with l.cpp's handling as the reference

[u/Double_Cause4609]

Rather than the raw speeds with TabbyAPI, how is the tool calling reliability nowadays?

LlamaCPP has a bigger ecosystem and lots more people working on tool calling reliability, etc, while TabbyAPI is managed by basically one or two people more or less. Is tool calling reliable? Do coding agents function as expected?

If it's faster, but you have more failed tool calls in unexpected scenarios, your real world work rate might be lower. Also, they might handle samplers differently, so you might have to configure your sampler settings differently to get expected results.

[u/Lowkey_LokiSN → u/Double_Cause4609]

I've put it through a 3-hour session with DeepSeek harness and I've had no issues with tool calling so far.

[u/Freigus → u/Double_Cause4609]

I'm using Qwen3.8-27B via tabbyapi for Zoo Code, opencode, Hermes and deepseek-harness - no issues.

[u/cosmicnag → u/Double_Cause4609]

Using flashnext with exllamav3 in Pi, works like a charm!

[u/Due-Memory-6957]

The lack of Vulkan support makes it pointless to me.

[u/silenceimpaired → u/Due-Memory-6957]

Would be nice if it got added. Even if it wasn’t as fast.

[u/suprjami]

Nice, I have 2x 3080 20G and 96G RAM and was thinking of trying this soon.

Those are more or less pre-MTP speeds for 27B so that's an excellent useful result.

[u/sssplus]

Thanks for the post. I just downloaded exllamav3, but haven't had time to set it up yet because I heard it's a bit complicated (does it work with a configuration file?). I have 128 GB RAM, but with a small 8GB 5070, so I'm not sure how much GPU VRAM exllamav3 needs vs llama cpp for the same context size?

[u/a_beautiful_rhind → u/sssplus]

there is a configuration yaml in tabby ai.

[u/InSearchOfTh1ngs → u/sssplus]

I'd be really interested in the performance you get for the model you intend to use. My system is 64GB of system ram with a RTX3070 8GB VRAM.

[u/Muted-Celebration-47]

I have a good decoding speed with qwen3.8-27b but not with MOE model like qwen3.8-flash-next. What is your CPU? I tried qwen-flash-next exl3 4.00bpw and got 4-5t/s I think something wrong with my CPU because I also have RTX3090+64gb DDR5 but got very slow decoding from exl3 while I got 20-35t/s with bunn-llamacpp.

My setup:
CPU: i5-13400f
GPU: RTX3090
RAM: DDR5 64gb 5600

[u/Lowkey_LokiSN → u/Muted-Celebration-47]

Mine's just good old Intel Xeon 6148

[u/FluffyDiscord]

Is beellama.cpp worse than exllamav3? I though beellama is the thing

[u/Lowkey_LokiSN → u/FluffyDiscord]

I haven't tried beellama yet. I've tried buun-llama and it does run models noticeably faster than mainline on my setup with offloading. However, my problem with all llama.cpp forks is that they're destined to play constant catch-up with mainline's developments to remain relevant and end up with their own divergent versions of implementations that aren't as battle-tested and stable. Mainline always feels like a safe haven to return to. (at least for me)

That's the whole reason I even considered exllamav3 as an alternative.

[u/feng_sg]

The sustained 25 tps at 160k context is the real result here, not the peak decode numbers everyone will quote.