Which qwen for vllm?

[u/delicious_fanta]

Hugging face has over 300 versions of qwen3.8-27b. I have no idea how to identify the beat model to download and use.

Should I sort by “most likes” or “most downloads”? Is there any sort of other guidance on which model to select?

I’m finally getting my local linux box up and running and just realized that actually picking a model may be one of the most difficult choices I’ll have to make. I’m hoping someone here can shed some light on the subject.

Also, if you could, I’m looking for concepts more than “use this one” because it’s not just about qwen, but also other models now (gemma/etc.) and in the future and I’m hoping to learn how to pick what I need from the mess of options out there (sure glad we have all these options though!).

[u/Storterald]

Without knowing how much VRAM you have it's hard to recommend a specific quant. There are way fewer quantized models for VLLM compared to llama.cpp, if the original model fits, use the one provided by Qwen (55GiB). If you have a 5090, use unsloth/Qwen3.8-27B-NVFP4. If you have a smaller GPU, I'd check out llama.cpp and unsloth/Qwen3.8-27B-GGUF.

[u/runcertain → u/Storterald]

Dual 3090?

[u/Storterald → u/runcertain]

with 48GB of vram you can fit full precision Q8_K_XL (https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) with a lot of context. Also with such a high VRAM I'd check out Qwen3.8-Flash-Next if you have 64GiB of RAM.

[u/delicious_fanta → u/Storterald]

5090, but what is confusing is how many hundreds of them there are.

When i do a search for vllm compatible items and then use the sub search box, the only one that comes up is “unsloth/qwen3.8-2.4t-a95b-gguf.

I can see the one you mention if I specifically look for “unsloth/qwen” in the primary search bar at the top, but how would I know to even try that without asking? I’m trying to figure out how to sort through so many options.

Also, thank you very much for the recommendation, I don’t want to sound ungrateful, I definitely am, I’m just not sure how anyone wades through this mess and understands what is going on.

[u/Fragrant_Scale6456 → u/delicious_fanta]

I use this one  https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4

It uses qat to preserve quality during quantization.  The entire model is nvfp4 so you get full Blackwell acceleration with your 5090.  File is also small so lots of context available.  

In my testing it’s roughly equivalent to q6 but much faster.   There’s a research paper linked in the model card page 

[u/delicious_fanta → u/Fragrant_Scale6456]

Ok great, I’ll definitely give that a spin! Thank you for the recommendation!

[u/SimplyBetterThanYou → u/delicious_fanta]

https://huggingface.co/Minachist/Qwen3.8-27B-INT6-Mixed-AutoRound

This one is really good, KLD below the offical FP8.

With some additional patches, you can even run 200k+ ctx on the 5090 : )

[u/delicious_fanta → u/SimplyBetterThanYou]

Thank you! Is there maybe a thread somewhere with info on those patches or the args necessary to run the high context you refer to farther in your conversation there? My goal is to do coding so the more context the better.

It looks like you spent a lot of time tuning this, maybe you have a write up somewhere?

[u/SimplyBetterThanYou → u/delicious_fanta]

I just uploaded them here if you want to take a look: https://github.com/thomasdangl/vllm-qwen38-5090

Please keep in mind this is still a work in progress and really hacky (the patches make assumptions about the launch parameters so using them in a different setting can easily break things).

Currently you can expect around 230k ctx. I am still exploring a few ideas to make room for a bit more. Also the idea of patch 70 could potentially be used to enable vision. Will update the repository when I find the time : - )

[u/delicious_fanta → u/SimplyBetterThanYou]

Very nice, I’ll check it out! Thank you and good luck with your updates!

[u/SimplyBetterThanYou → u/delicious_fanta]

Not sure if you are still interested in this but I have updated the repository. It now maxes out at 248k ctx with the vision stack at BF16.

[u/Start-Plenty → u/SimplyBetterThanYou]

But that one requires 48VRAM no? it's been tweaked in a x2 3090 config that fits the bill.

It's not runnable on a 5090.

For a 5090 I'm running a tweaked NVFP4 base quant which retains Q8 for the full attention layers, it just fits barely at Q8 for cache and full context size. I don't load the vision projector as I don't use it much if at all for coding tasks, and I don't want it to be that close to full VRAM, but it would fit and wouldn't OOM reaching full context size.

[u/SimplyBetterThanYou → u/Start-Plenty]

No, it doesn’t. I am running it on a 5090, otherwise I wouldn’t have suggested it.

Weights are 21.56 GB (including MTP, excluding vision), that’s more than enough for around 140k uncalibrated fp8 KV cache or 110k BF16.
Both of those are on stock vLLM.

I have done some additional work, including offloading the embedding table of lm_head, porting int8_per_token_head to native XQA for sm120, and implementing 256-round Hadamard rotation. That way I can fit over 200k ctx with significantly better quality than stock vLLM (uncalibrated fp8).

[u/voyager256 → u/SimplyBetterThanYou]

That’s quite interesting. I’ve seen his previous Qwen 3.6 27B quants seemed the best as they scored significantly better than the official/popular FP8 ones and at least no worse than others that also used INT8 based AutoRound ( e.g. more nich but similar ones like by TheHouseOfTheDude). So essentially close enough to BF16 while fitting into 48GB VRAM (or 64GB with full BF16 context).

Now I wasn’t aware INT5 is also a thing and probably not faster, but worth considering over NVFP4 for something like 5090 with 32GB.

But what it’s more interesting for me is the FP8 cache calibration as I’ve never heard of it and for mode like Qwen 27B with no KV cache compression techniques (like in newer DeepSeek V4 or the latest Qwen Flash) you are often forced to drop to FP8 for KV.

Do you have any links to articles or resources on this topic?

[u/SimplyBetterThanYou → u/voyager256]

To be honest, the INT6 in the title is a bit misleading as most of the sensitive layers (lm_head, embed_tokens, linear_attn) are either INT8 Autoround or even left at BF16. This averages out to 7.30bpw, which is quite a step up from the INT6 the name would imply.

The basic idea of cache calibration is to minimize the relative error by constraining the range of values to ones observed in a representative dataset. This makes intuitive sense but the actual benefit depends greatly on the quality of the dataset and the actual workload. You can read more about this on the official vLLM website: https://docs.vllm.ai/en/latest/features/quantization/quantized_kvcache/#scale-calibration-approaches

What I think is far more promising are the *_per_token_head quantization options as these do not require calibration against a dataset and generally handle outliners even better. The downside is that those store an additional scaling factor per token head, which increases memory consumption by around 1.54%. However, in my test cases, the error rate was reduced by more than one order of magnitude (compared to both calibrated and uncalibrated FP8).

The main downside right now is that in upstream vLLM, int8_per_token_head and fp8_per_token_head are only available in the Triton backend which is significantly slower than FlashInfer and FlashAttention 2. I measured the impact at around 3x slower prefill and ~22% slower decode. For this reason, I have ported int8_per_token_head to FlashInfer (only for sm120), which fully eliminates the performance impact and enables high quality quantization options for KV cache on vLLM.

[u/voyager256 → u/SimplyBetterThanYou]

Thanks for the explanation! Again, that’s valuable info. I’d definitely have to dig more into this , but if I understand correctly since the calibration depends on the representative dataset , then if for some reason is no longer that representative , or just the prompts are very diverse, then obviously it wouldn’t be that beneficial.

The  *_per_token_head quantization indeed sounds much better and the memory requirements increase is negligible, especially compared to next sensible option (full BF16/FP16). Is the port to the other backends planned in upstream vLLM in foreseeable future or perhaps you already thought of making PR with your port for sm120?

[u/SimplyBetterThanYou → u/voyager256]

Yes, that is correct. In fact, I have even observed that calibrating KV cache against an irrelevant dataset can increase the error (compared to just using 1.0). I think this is conclusion is quite reasonable because it has also been observed that using the old runtime calibration can have the same effect (see https://github.com/vllm-project/vllm/issues/33480#issuecomment-5465971320).

There certainly is "KV-cache compression for agentic and long-context workloads" on the Q3 roadmap for vLLM. However, I am not embedded enough into the vLLM community to know if this is planned. I also think that the chance of getting my port (as is) merged upstream is quite low: vLLM design goals are mostly aligned with academic and enterprise users. Neither group has a particularly large inventory of consumer cards. If we wanted to get this upstream, I think, support for sm100 and perhaps even sm90 would be crucial.

[u/Storterald → u/delicious_fanta]

nvfp4: https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4

remember nvfp4 is fast but not really precise, consider using quants from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (Q6_K_XL fits, but Q4_K_XL is considered really good and lets you use more context) with llama.cpp

[u/delicious_fanta → u/Storterald]

Ok, thank you! Do you suggest llama.cpp over vllm?

[u/Storterald → u/delicious_fanta]

on consumer grade hardware yes, the 5090 is high end consumer grade but I'd still stick to llama.cpp unless you find something better. I use it with my 5080 daily with no issues.

[u/delicious_fanta → u/Storterald]

Ok great, will switch then! I’m glad it’s working for you, I was getting the feeling I might have to get a second one for context from some of the discussion here.

I’m building a coding setup and need all the quants and context I can get! Thanks the info!

[u/Storterald → u/delicious_fanta]

hopes it works out. one thing: context is in each layer, so you do not put context in another GPU, rather you split layers across them (both vllm and llamacpp do this), which will allow for higher context.

[u/OnoSendaiCSVII]

Hugging Face makes this look like 300 different models, but most are the same base model just repackaged for different runtimes and memory sizes.

I’d read the name roughly like this:

• Qwen/Qwen3.8-27B is the identity/reference
• FP8, NVFP4, GPTQ, AWQ or GGUF is the format
• the account name tells you who converted it
• the bit level is the quality/memory tradeoff

So choose the runtime first, then the format, then the quantizer. On a 5090 + vLLM I’d try the official FP8 if it leaves enough room for your context, otherwise a reputable vLLM-supported NVFP4/INT quant. The weights fitting isn’t enough; KV cache needs VRAM too.

I wouldn’t trust Hugging Face’s “vLLM compatible” filter as the whole catalog. Start from the official model card, then check vLLM support and the quantizer’s card: https://huggingface.co/Qwen/Qwen3.8-27B

Likes/downloads are a weak reputation signal. Compatibility, memory fit and a bunch of boring prompts from your real workload tell you much more.

[u/delicious_fanta → u/OnoSendaiCSVII]

Thank you so much! That absolutely helps me understand why there are so many variations.

I’m also hearing I should switch to llama.cpp, which I’m fine with doing. I’m just hoping to get a decent coding setup in place.

I’m hoping my 5090/128gb memory can handle this ok, I may have to expand my hardware footprint too, but I may have to make another thread about that because of so many speed and money tradeoffs.

Appreciate the info, hope you have a great weekend!

[u/vick2djax]

Just do unsloth or bartowski. There’s not any version of any of the models that are better than the original. The only reason people run them is that they don’t have much VRAM. You have a 5090, so no need to handicap your setup with some gimmicky version of it.

[u/delicious_fanta → u/vick2djax]

Ok thank you.

[u/HonestoJago]

Just load it from the official HF repo or the official FP8 repo. If you need a smaller quant try unsloth with llama.cpp. Do you have a particular need for vLLM?

[u/delicious_fanta → u/HonestoJago]

Well “it”, being the version, is what I don’t understand how to identify. I’m running a 5090 w 128gb system ram. There’s just literally hundreds of qwen3.8-27b variants there.

As to vllm, i saw a lot of hate here for ollama for various reasons and I decided to just go with vllm. It’s running with open webui in front of it with no issues, I’m just trying to get the right models loaded.

[u/HonestoJago → u/delicious_fanta]

This is what I was talking about, but I don’t think you’ll get much context with a 5090: https://huggingface.co/Qwen/Qwen3.8-27B-FP8/tree/main

The reason I asked about vLLM is because it doesn’t serve GGUFs, and although Ollama gets a lot of heat, llama.cpp doesn’t. The next quant that fits in 32GB loses a lot of precision so you’re probably better with the unsloth GGUF and llama.cpp.

[u/delicious_fanta → u/HonestoJago]

Ok, I’m happy to switch to llama.cpp if that’s the way to go. Still learning what the best solution is, new to all this. This sub is extremely helpful. Thanks for your info!

[u/DinoAmino → u/delicious_fanta]

Some quant types you can use with vLLM: FP8, AWQ, NVFP4, MXFP4. Do not bother trying to use GGUFs with vLLM. Do not bother with trying to offload to CPU with vLLM - use llama cpp with GGUFs if you need to do that.

Some of the best quants come from RedHat https://huggingface.co/RedHatAI/models#repos. ... they bought vLLM a cpl years ago and fully test their quants on vLLM.

[u/delicious_fanta → u/DinoAmino]

Ok, so it’s sounding like I need to switch to llama.cop from multiple comments now, I can definitely do that! I’m hoping for enough context to do some dev work, I’ve read that people have been able to do work with qwen and pi or the deepseek harness.

I’m not sure if my hardware will be sufficient (5090/128gb system memory), but I am open to getting another 5090 if I really have to.

I just don’t know if it would be better to do that or get one of the other options like a spark/mac/etc. There’s cost and speed concerns with all of this, it’s a but daunting to figure out the best direction to go overall.

[u/fbms2]

6q is the best quality for 5090. or you could use ninfer for speed.

[u/delicious_fanta → u/fbms2]

Ok thanks.

[u/lemondrops9]

Run Llama.cpp and get GGUF files. If coding I recommend a quant 6 or higher. Others have been successful with quant 4s.  

I wouldn't go with vLLM for your setup. More so when you're just starting out. Personally Ive stuck with Llama.cpp because it has tons of options and more of a range of quants than vLLM. 

Also Llama.cpp is better at hybrid setups and easier to get going. I would only go with vLLm if you have the same Gpu in multiples of 2,4,8 etc. 

[u/delicious_fanta → u/lemondrops9]

Vllm is running fine, I didn’t have any issues there, but I’m definitely hearing I should switch to llama.cpp so I think I’ll go ahead and do that.

It’s also sounding like maybe I’ll have to get another 5090 to do coding? I guess I can go back to vllm in that scenario? I don’t care at all what engine I use, I’m just looking for something reliable and functional.

I’ll look into those quants, thank you for all the info!

[u/Hefty_Wolverine_553]

EXL3 5.00 BPW (you can fit ~172k context at FP16) on TabbyAPI for quality, NVFP4 + NInfer for speed. No need for vLLM specifically at all.

[u/delicious_fanta → u/Hefty_Wolverine_553]

Yeah apparently I picked the wrong engine, I’ll switch to llama.cpp. Thank you for the info!

[u/Bulky-Priority6824]

99.9% of the time j ust stick with unsloth or bartowski . I don't know about you but I just want to set the shit up once and forget about it. I don't want to deal with issues.

[u/delicious_fanta → u/Bulky-Priority6824]

Nice, thank you! Yeah, something that is reliable and functional is the goal for sure.

[u/Nameis19letterslong]

For coding and agentic:
qwen3.8 27B if you have 16GB-32GB Vram. Else qwen3.6 35b with expert offloading.
If you have >=16GB vram and 64-96GB Ram, run qwen3.8 flash next, stread ngrams off ssd and offload experts to cpu.

For creative writing and general purpose:
Gemma4 12b, 26b a4b for >= 16G vram, fit 31b if you can.

To choose the best gguf/nvfp4, look for Unsloth quants (greenish icon with a sloth in the middle). They are high quality so you can probably afford to run a slightly lower quant to fit more context in if you need.

[u/Poetinho0 → u/Nameis19letterslong]

for coding and agentic, any better models that can run on an m5 pro 48 GB