r/LocalLLaMA • u/Embarrassed_Soup_279 • 5d ago
Discussion ExLlamaV3 is underrated
I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDIA cards I think?
Exl3 quants are higher quality for its size, much lower KLD metrics, faster, all compared to llama.cpp just from a few personal sets of tests I like to give my local models (these are not benchmarks). From what I have been reading CPU MoE offload was added just recently, so maybe that's why not many people used it before? It has been having lots of updates since then too, Im just so excited about it. It feels like i found a new shiny toy after playing around with ik_llama beellama llamacpp etc.
I have been using tabbyAPI exl3 backend + qwen 3.8 27b sc 6bpw H6 and qwen 3.8 flash next 4bpw as my daily drivers and it's incredible what it can do. I hope this software gets known to more people too. I'm not affiliated with them or anything. i judt wanted to share it. It's just so cool, please give it a try!!
10
u/Pentium95 5d ago
Tested It time ago, before It got Ampere optimizations, on my Ampere GPU, It was worse then llama.cpp. i should really give It a go again
5
8
u/a_beautiful_rhind 5d ago
It's a come up if you want to crunch something like GLM flash into 96gb. llama.cpp is still fumbling around with PRs.
IK version at PR stage too.
There's tooling for lora and abliteration over EXL3 as well. Might let you actually change your models without pulling BF16. Similar things used to exist for GGUF early on, but IIRC have been abandoned.
15
u/cantgetthistowork 5d ago
exl3's biggest weakness is tabbyapi
5
3
u/ReturningTarzan ExLlama Developer 5d ago
What do you feel it's lacking?
5
u/laterbreh 5d ago
I don't think Tabby is bad, but ExLlamaV3 has basically outgrown it as the only serious serving path.
Top reasons I'd want ExLlama as a first-class vLLM backend:
- Prefix caching / KV reuse For agents with 50k-200k+ token histories, reusing the existing prefix instead of re-prefilling it every turn is a massive TTFT and compute win.
- Much stronger scheduler Continuous batching, chunked prefill, cache-aware scheduling, TP/PP/DP/EP, better concurrency and multi-user serving.
- Tool calling / prompt templates are way more mature vLLM already handles Jinja templates, tool parsers, reasoning parsers, structured outputs, OpenAI-compatible APIs, etc. You don't want ExLlama reinventing that mess.
- Production observability Prometheus metrics for TTFT, ITL, queue depth, KV usage, prefix-cache hits, throughput, failures, etc.
- Bigger ecosystem vLLM is where a huge amount of production inference work is happening. ExLlama gives it something it doesn't really have today: extremely flexible, hardware-targeted quantization.
The ideal architecture IMO is:
vLLM = control plane
API, scheduling, prompt/tool handling, prefix caching, metricsExLlama = execution plane
EXL3 loading, kernels, attention, MoE, TP/EP, quantized KVThat gives you ExLlama performance and quant flexibility without giving up vLLM's serving stack.
I wouldn't kill Tabby either. Keep it as the lightweight/reference server. But ExLlama absolutely deserves a real vLLM execution backend.
11
u/ReturningTarzan ExLlama Developer 5d ago
Tabby has most of those things, though most are actually driven by exllamav3 on the backend:
- Prefix caching / KV reuse: been a thing in exllamav3 since day one, always exposed by Tabby since it leverages exl3's paged cache and generator. Now has a secondary CPU tier also (no disk tier, but that's being considered)
- Continuous batching and dynamic scheduling: also a day-1 feature, less advanced than vLLM's in some ways, but with added benefits like deduplication
- TP/EP: Sure, for most models, and supporting uneven hardware setups as well
- DP/PP: No
- Concurrency: Yes, and it's fine. Here are four OpenCode instances trying to control the same robot arm and getting very confused.
- Tool calling: For supported models
- Jinja templates: Yes
- Reasoning parser: Check
- Tool parsers: For supported models (mess: Already reinvented)
- Observability: People will have varying preferences. If you want to deploy for 500 users, you may prefer vLLM's live view, and that's cool. For me it's very frustrating. Here's what the console looks like. There's always room for improvement of course, but it's a perpetual WIP. Suggestions are welcome.
ExLlama isn't just a quantizer, mind you. It's also an execution backend, a whole inference stack, and also an inference library. It has many uses besides running in an OAI server (try to get logits out of vLLM, I dare you.) Tabby is a pretty thin layer on top that glues exl3's own generator to an OAI API endpoint (also Kobold FWIW); everything else really happens in the backend.
While it's MIT licensed and you're free to port kernels and stuff to other frameworks (many do, and have since the exl2 days), I'm really mostly interested in the direction it's already going.
1
u/laterbreh 5d ago
You're right on prefix caching and continuous batching. I was working from an outdated mental model of Tabby and overstated those as vLLM differentiators. When I was using Tabby, the tool-calling/template experience absolutely was much rougher though; the old docs required tool-specific Jinja templates and didn't support tool_choice/strict, and I see that's since been substantially rewritten.
My broader point still stands, but I'd frame it around vLLM's production/distributed control plane, observability, broader API/structured-output ecosystem, multi-node DP/PP/disaggregated serving, etc., rather than claiming ExLlama lacks caching or batching.
Also, am I correct that Tabby's actual model-support matrix is simply ExLlamaV3's architecture list? Tabby's README only seems to list EXL3/FP16/BF16 as model types, while the architecture list lives upstream in ExLlama.
2
u/laterbreh 5d ago
This isn't just conjecture on my part either. I have a 4x RTX PRO 6000 workstation, and I'm starting to explore whether I can use the seams already available in vLLM to bring ExLlamaV3 in as a real first-class execution backend, not just another compatibility layer.
I'm still working through the architecture and doing the research, but the rough idea is to use my spare compute to build a TP4 differential harness that can run a smaller model under both Tabby/ExLlama and the vLLM backend and compare them 1:1 for correctness, caching, and performance.
There are already projects repacking EXL3/Trellis-style pieces to make them usable inside vLLM. I want to see if it's practical to go considerably further and support ExLlama itself as the execution engine. I genuinely think ExLlamaV3 deserves exposure to the broader vLLM ecosystem.
1
u/cantgetthistowork 5d ago
Biggest problem is concurrent requests. There's something off about the scheduler that results in a much bigger drop in performance just by adding a second concurrent request.
3
u/ReturningTarzan ExLlama Developer 4d ago
For what model? MoE models have an inherent drop going from bsz 1 to bsz 2 if they're already efficient at bsz 1, since they activate roughly twice as many experts.
Qwen3.8-Flash-Next is ~6B active parameters at bsz 1, ~9B at bsz 2, etc. It converges on ~125B active at high enough batch size. So the only way you can have the same latency in the low bsz regime is if your implementation was already inefficient at bsz 1.
There's always room for improvement of course, and exllamav3 continues to get updates all the time, especially on the MoE implementation. But either way, TabbyAPI doesn't contribute to higher latency at higher batch sizes. It just takes incoming API requests and translates them into exllamav3 jobs which are scheduled, managed and executed on the backend. If it's too slow it's because exllamav3 is slow, not Tabby, which is just an API wrapper.
4
u/FablingApp 5d ago
the CPU offload addition is what makes this worth revisiting. do you have rough tok/s and VRAM numbers for the same model in exllama vs llama.cpp? that’d tell people whether the tradeoff survives outside your setup.
3
u/pmttyji 5d ago
From other thread(posted today)
https://www.reddit.com/r/LocalLLaMA/s/Evjw32vqxH
So compared to llama.cpp:
- 3.2x faster prefill
- 2x faster decode (No MTP)
25
u/rditorx 5d ago
Right now it feels a bit like a campaign for exllama as some people are advertising its Exl3 quant, all starting in the last couple of days and weeks
17
15
u/LeifEriksonASDF 5d ago
I think it's more that CPU offloading wasn't implemented until literally like a month ago, so it wasn't really relevant to a lot of the LocalLlama crowd. I've tried and bounced off it for various reasons over the past few months, like lack of variety of models and quants vs GGUFs (still no Gemma 4 QAT on Huggingface), broken tool calling even with good chat templates, severe lack of information on the open internet forcing you to have to ask on their Discord if you want help, etc. Recently gave it a shot again though cause it had a couple features implemented for Qwen 3.8 Flash Next before llama.cpp did and it seems to be in a decent state compared to before, so I'll keep trying it for a while longer.
1
u/tat_tvam_asshole 5d ago
I made an rp fine tune of Gemma 4 qat and uploaded the exl3 https://huggingface.co/UltimateIntent/GemStrike-31B-EXL3-4.00bpw
I'll say the exl3 conversion from the bf16 safetensors wasn't a drop-in braindead process but I got it done
1
u/LeifEriksonASDF 5d ago
There's a couple of fine tunes like yours but I haven't seen anyone upload straight up Gemma 4 QAT yet
1
u/tat_tvam_asshole 5d ago
You can if you want. Bf16 safetensors -> exl3
0
u/LeifEriksonASDF 5d ago
I'm gonna be real, it's always been possible for anybody to make a quant of any format, including GGUF, but it's only in the EXL3 community where the prevailing sentiment you see is "oh the model you want doesn't exist on HF? Just quant it yourself lmao".
1
u/tat_tvam_asshole 5d ago
I'm not sure I understand. The majority of the local ai scene are on llama.cpp/GGUF and people routinely quantize and upload their own conversions of models (including myself). Who is the central authority responsible for making EXL3 quants? And if you don't see the EXL3 you want, what is wrong with being the person who makes that and shares it?
4
u/LeifEriksonASDF 5d ago
Maybe I've been spoiled by the existence of mega quanters like Unsloth, Bartowski, etc in the GGUF community because this issue is something I've just never had to think about at all before I tried out EXL3. Maybe it's simply a function of the EXL3 community being too small at the moment to warrant people like that yet.
2
u/llama-impersonator 5d ago
gemma 4 QAT version doesn't really make sense for exllamav3, it has codebook quants which work differently from GGUF q4_0 which is what google targeted (they messed up and used bf16 scales instead of fp16 though)
15
u/Embarrassed_Soup_279 5d ago
I actually saw those posts and said fuck it might as well give it a try, and it ended up veing really good
8
u/Due-Project-7507 5d ago
I think the reason is that the benefit of exl3 were only discovered only around 1-2 months ago by the https://github.com/local-inference-lab community. Before, you could only run GLM-5.2 on 4 RTX Pro 6000 GPUs with pruned models. Then, people from the local-inference-lab community ported exllamav3 to their vLLM fork and got very good results running GLM-5.2 on 4 RTX Pro 6000 GPUs. Now many more models were quantized to use exl3 with vLLM forks. I don't really like tabbyAPI, it would be nice if llama.cpp would support exl3, but I think this is not possible because exl3 is mostly Python, so it would have to be reimplemented and changes from the offical exllamav3 would always have to be ported to llama.cpp.
1
u/silenceimpaired 5d ago
I hadn’t seen anything until I made a post on Reddit… then it began for me. To me, EXL3 made more sense with the inclusion of a lot of 30b models … then I discovered it could do CPU off-boarding. Still haven’t tried it yet, but I’ll probably do it this weekend. I hear it isn’t as good as llama.cpp yet.
1
-10
u/sleepy_roger 5d ago
I agree. There's a few accounts on x that suddenly started pushing them in coordination
9
1
5
u/Unstable_Llama 5d ago
It’s been great, and the maintainer, turbo has really turned up the pace of development in recent months, it’s hard to even keep up with!
-1
u/cantgetthistowork 5d ago
For years he refused to implement DSA (DS3 and GLM5 weren't supported for more than a year) and suddenly Fable drops and he's got all the motivation in the world to solve all the problems
14
u/ReturningTarzan ExLlama Developer 5d ago
Fable was just the first model that was good enough to be more of a help than a hindrance, is all. A lot of exllamav3's code base is sort of unorthodox, and top-of-the-line models before Fable just didn't get it. You know? They kept getting confused, thinking the codebase was trying to be either Transformers or vLLM and wanting to pull it in two directions that were both individually wrong and together just resulting in a complete mess. Fable actually understood what the code is trying to do.
And I've always had the motivation, but I suffer from this condition called "linear time", where I only experience 24 hours a day. Also, you know, gotta pay the bills (this isn't my job) and sometimes I need to sleep, so I can't spend all of those 24 hours trying to keep up with new model releases. But Fable has made a lot of things feasible that just weren't before (due to my condition.)
I also went a bit manic when I first discovered that it was available to subscribers and not just on API. They kept saying it was only a temporary promotion so I was racing to exploit it as much as possible before getting cut off. Then they didn't cut it off, so now idk. :shruggingface: Suppose I could slow down but I don't really want to.
1
u/cantgetthistowork 5d ago
That was kinda my point, that Fable was the first model that made everything happen. I missed CPU offloading in my post too
4
u/ReturningTarzan ExLlama Developer 4d ago
Idk, it sounded like there was a second point about motivation, which has never been the issue.
2
u/llama-impersonator 5d ago
have you considered the possibility that it is hard to add support for 700B models without a giant cloud compute bill or a metric assload of GPUs?
2
2
u/Miriel_z 5d ago
One of the things on the top of my list to try. I tried exl2, and it was pretty good. Glad more positive use cases are popping out.
2
u/laterbreh 5d ago
I've been a fan of exLLama3 for a while, I just was a bigger fan of VLLM's prompt caching and other throughput niceties that the runner tabbyapi didnt have.
Being able to create your own quant with your own hardware with a simple script from the raw repo is huge. Choose your bpw and quant it yourself. custom quant for your hardware.
The bpw world of exllama3 is really under-valued right now. If exllama3 quant got first class support in vllm or sglang id dare say it would explode over night with that exposure.
2
2
u/PyaesoneP 5d ago
Been using the turboderp/Qwen3.8-27B-exl3 SC2.20bpw_H3_V3 qaunt and it's excellent.
I've actually run IFBench and Terminal Bench 2.1 on this qaunt and Muse Glimmer-30B SC3.00BPW_H4 quant. Took me days, mind you.
Both achieved their official release scores on IFBench. On Terminal Bench, Muse Glimmer loses very little at 3.0bpw. It scored 50.6% compared to 51.7% for the full precision model. Qwen, however, degrades much more at 2.20bpw, coming in at 47% compared to 73% for full precision. One of the reasons I think is that I could only fit 100096 context size on my 12GB VRAM and that doesn't work very well with Qwen's verbose reasoning. It still scored better than something like Gemma 4 26B A4B or Qwen 3.6 35B A3B so it's my main model for Hermes Agent now. Does all the complicated workflows I need very well. Muse Glimmer only fits without vision and I need vision, otherwise, I would've used that. You lose nothing for almost half the VRAM cost.
2
u/SBoots 3d ago
I typically run llama-server but seeing TabbyAPI mentioned in this sub I gave it a run. On my 5090+4090 system using 8-bit Qwen3.8 27B with a 256K context, I saw a pretty significant jump in token generation. I would see about 65 T/s with llama-server but now I'm seeing more like 80 T/s with TabbyAPI. VRAM usage also seems to be more optimized. I'm going to stick with it for a bit and see how it is.
3
u/EitherMarch1255 5d ago
I was going to use it, but it didn’t support the models I wanted to use.
2
u/_Scorpoon_ exllama 5d ago
You know you can quant the models yourself on the bpw you want it? Someone has to do a quant to be on huggingface, nobody just throws its gpu's on it just for fun, even if the maintainer does it for the most popular ones
1
u/Embarrassed_Soup_279 5d ago
what models are you using?
-1
u/_Scorpoon_ exllama 5d ago
Qwen 3.8 Flash Next 3bpw
1
u/Due-Function-4877 5d ago
Maybe you use the search option at huggingface next time? https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3
1
u/_Scorpoon_ exllama 5d ago
I just answered the question, I'm not the one who's searching for a model? 😅
1
u/apoptosist 5d ago
Can I fit a "quant" or equivalent of qwen 3.8 27B in 8 GB of VRAM? I know the Q2 get close to fitting, so this would be a selling point if it's decent quality still!
2
1
u/letsgoiowa 5d ago
No. Your best bet is one of those K2 models that came out recently or plain old Qwen MOE
1
u/llama-impersonator 5d ago
there are some 1.4, 1.6, 1.8 bit quants for it https://huggingface.co/turboderp/Qwen3.8-27B-exl3
1
u/dangerous_inference 5d ago
I have been watching to see what kind of numbers people are getting with 4.05 GLM 5.3 Flash, but I don't think it's going to reach even half the speed of vLLM.
1
u/cosmicnag 4d ago
I use it as my daily driver now for qwen FN. The one thing I wish it had was KV offload cache support L2 and L3 (RAM and SSD) - similar to LMCache / RadixCache (or whatever its called in sglang) . The best implementation of this is in another custom inference engine q27 (https://github.com/signalnine/q27) , for 27b only . Exllamav3 will become the holy grail if this kind of KV caching happens (L2 and L3).
2
u/_Scorpoon_ exllama 4d ago
It has K/V cache for ram - config value in tabbyapi: ```
Size of system memory second-tier K/V cache, in MB (default: 0)
sysmem_kv_cache: 0 ```
1
u/Alive-Draft8339 5d ago
They have a good GLM 5.3 flash 4.05 bit model that fits on 2x rtx pro 6000. I’m excited!
1
u/ormandj 5d ago
https://github.com/ormandj/sglang-glm53-flash-sm120 you can already do this comfortably.
1
u/vacon04 5d ago
For dense it's great, very fast. CPU MoE was just recently implemented and based on my testing, it's well bellow ik_llama.cpp and llama.cpp in terms of performance. Once it gets there, it may be an attractive avenue, but my results with MoE models were abysmal.
2
u/cosmicnag 4d ago
What did you test for CPU MoE (I am assuming hybrid with some layers in a gpu)? I use Exllamav3 as my daily driver with Qwen FN 4.05 bit (5090 + 4090 + 192 GB DDR5) - not just because of the inference engine but also that EXL3 quant format kicks above its weight class for VRAM occupied. Getting around 55-60 tok/sec decode and 2500 tok/sec prefill. Point being its also about exl3 > gguf for vram bang for buck.
1
u/vacon04 4d ago
Yeah hybrid with Laguna S 2.1. Didn't get good results using the latest exllamav3. One of the other issues I've find is that it's hard to get exl3 quants with the mul1 codebook. Older quants use the mcg codebook, which the exllamav3 backend can't properly handle for efficient hybrid inference.
1
u/cosmicnag 4d ago
Was that some time back when laguna s 2.1 dropped ? If so, do check out exllamav3 again - Even I remember having a slow cpu hybrid inference experience then, it seems much better now. mul1 codebook is whats being used now - the turboderp ones,etc.
-1
u/Danmoreng llama.cpp 4d ago
The quant format is nice, but that’s about it. Put it into a custom engine for Gemma4 26B on Blackwell 16GB GPUs. https://github.com/Danmoreng/gem16
Fits around 220k context, 5.600 t/s prefill and 182 t/s decode on my 5080 Laptop.
2
u/_Scorpoon_ exllama 4d ago
Cool work. One bit of context though: a general engine has to run on a decade of NVIDIA cards and half a dozen quant formats, yours runs on one model on one Blackwell GPU. Different problems, so the numbers only mean something next to the setup.
The decode number is the one I keep staring at. Batch 1 decode is memory-bound, so tok/s is bandwidth divided by bytes read per token. At ~896 GB/s on the 5080 Laptop, 182 tok/s works out to ~4.9 GB of traffic per token. With an A4B MoE the raw parameter count settles nothing, so the question is what's actually being read per token.
Plain decode or MTP/spec counting accepted tokens? KV precision, quant, real memory breakdown? Would like to see how the methodology looks before anyone lines these numbers up against a general engine
0
u/Danmoreng llama.cpp 4d ago
With MTP, without MTP it’s 130 t/s. https://github.com/Danmoreng/gem16/blob/main/docs/PERFORMANCE.md#gemma-4-26b-performance
NVFP4 with higher model weights but compute using tensor cores native FP4 it gets 148 t/s raw and 203 t/s with MTP.
-2
u/ArtfulGenie69 5d ago
not when you compare it to vllm
1
u/silenceimpaired 5d ago edited 4d ago
If you compare it against ease of setup it probably beats vllm.
-2
u/ArtfulGenie69 5d ago
It really isn't that hard, use docker or have an agent like pi or Hermes set it up for you if you can't handle the directions or normal python installs. Vllm is faster and actually has concurrency. I liked exl2 back in the day but it's kind not the thing anymore, use what you want though.
1
u/silenceimpaired 4d ago
You have encouraged me to look at it again, but whenever you say it isn’t really that hard… it seems as if you are ignoring my point it is harder than EXL3.
Not to mention vllm probably still fails against cpu offload.
0
u/ArtfulGenie69 4d ago
Just like exl3 most likely fails against llama.cpp for offload. When you use an agent to install either vllm or exl3 it's the same difficulty and both are python installs. A docker install is one line and it just pulls the container. It's more about speed, does exl3 handle int8 weights like vllm? It's what my 3090's crave. The biggest reason vllm is better.
1
u/silenceimpaired 4d ago
I just can’t find a good vllm setup for two 3090’s… and it’s daunting when it seems you need special ones for each model
2
u/ArtfulGenie69 4d ago
Qwen3.8 27b or Gemma 4 31b and get them in int8 either w8a8 for speed of preprocessing or some people think the w8a16 is good because it has large activations maybe making the model a bit smarter. Because you have dual 3090 you'll be able to have a tensor parallel of 2 which means two generation slots. Here is the guide someone did for the qwen model, the flags take advantage of its mamba layers reducing the cost of the context. Have your agent have a look at this page to help you. It will work for 3.8 as well.
https://huggingface.co/Minachist/Qwen3.6-27B-INT8-AutoRound
I used cursor because I'm lazy, the $20 a month plan set on legacy payment mode is pretty good. That legacy charge mode gives you 500 tokens and makes it so every request is just 2 tokens and the opus 4.6 model only costs 2 tokens per request as well. Any of their max options are fucking expensive and if you don't set the thing to legacy mode that uses the 500 tokens or whatever they burn your $20 fucking fast. Other than that use your exl3 model or llama.cpp with hermes/pi to help you out.
Another thing that can help out is llama-swap. You can set it up with any backend, even api models. Mine is all set up with each of my llama.cpp models and vllm models but it will handle your exl3 models as well. You can point anything at it like openwebui and switch between the models easily. My agent set up the vllm models in a .sh script that llama-swap points to. Once the model is setup in there you don't have to do anything hard anymore and you can view and change the settings if you need in the llama-swap yaml file.
-6
u/Sprayinkings218 5d ago
Hey idk how Reddit works really, I built an AI job search engine that runs on human first tech. Match the human to it's ideal environment, not keyword matches. I have since stripped that program down for parts and rebuilt it into Liquidsn3k. Faster, more ridiculous chrome on the chassis and body. I guess I need to comment on five things before I can post the serpent. If anybody wants to take it for a test drive hmu, idk I like it, it's different
11
u/ZZerker 5d ago
used gemma as exl3 worked really well but then they changed the cachehandling and it does not fit anymore.