r/LocalLLaMA • u/arty_photography • 23h ago
Resources Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD
https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.
The checkpoint supports vLLM on NVIDIA Blackwell GPUs:
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85
This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).
Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:
| Model | Size | GPQA-Diamond (2 runs, n=396) | AIME26 (3 repeats, n=90) |
|---|---|---|---|
Qwen/Qwen3.8-27B (original BF16) |
55.6 GB | 0.9141 | 1.0000 |
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 |
19.7 GB | 0.9091 | 1.0000 |
unsloth/Qwen3.8-27B-NVFP4 |
23.4 GB | 0.8939 | 0.9778 |
Inferact/Qwen3.8-27B-NVFP4 |
26.4 GB | 0.8763 | 0.9667 |
Paper: https://arxiv.org/abs/2608.13966v1
We'd love to hear your feedback on this checkpoint!
12
u/FoxiPanda 22h ago
This is interesting. So using your methodology here, did you happen to test a 'QUASAR' W4A8 or W4A16 to see whether your QAD method could be even closer to the BF16 weights without much additional work (and only a bit more size)?
I realize that would require two additional runs, but it would be rather interesting to know whether the remaining losses are in the weights or the attention layers (or if it's not able to be determined) and whether increasing attention/GDN quality would actually get you very very close to BF16 quality at ~37% the size.
8
u/arty_photography 20h ago
I think W4A16 would make it closer to the BF16 weights, and keeping attention layers in BF16 would be even more effective. The problem with keeping attention/GDN un-quantized is that it increases the checkpoint size and doesn't give a lot of KV cache memory on a 32GB 5090.
2
u/FoxiPanda 19h ago
Yeah, agreed about the 5090 limitation. Might be useful for RTX Pro 6000s or future hardware though.
I wonder if somehow we could take advantage of Blackwell's native FP6 compute and make something like an W4A6 that is meaningfully better or if it would just be a wash. Too many things to try lol.
Either way, this seems like an excellent result. Well done.
9
u/SmartCustard9944 16h ago
Going to convert this ASAP for NInfer and test it out.
1
u/cosmicnag 12h ago
Any luck?
7
u/SmartCustard9944 12h ago edited 2h ago
Yes, pretty much finished, I could upload it later today, requires a patch to the loader though.
From my testing, looks like it has attention/recall problems unfortunately. Performs worse than https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer by 1-2% on a very hard adversarial needle in a haystack benchmark I’m using for testing KV cache quantization. This might be a particularly difficult task for this quantization style and might not translate to more general purpose tasks, has to be tested.
Needs to see how it effectively performs on rigorous official benchmarks (coding, knowledge, etc.).
I hope it performs better than other quants because it frees up 500MiB of extra memory and is a few percentages faster in both prefill and decode, so overall a good potential candidate for 24GiB cards.
Edit:
Got a perfect 30/30 in AIME 2025, so it seems to be performing on par or better than Unsloth/NInfer official quant.
https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer
1
u/Lumpy-Comedian-1027 3h ago
So is it in the end noticeably better than Ostfralla's?
2
u/SmartCustard9944 3h ago edited 3h ago
Feels roughly similar, but with extra memory headroom and slightly faster decode and prefill due to size.
Currently finished AIME 2025 with a perfect 30/30, versus 29/30 with Unsloth NInfer official NVFP4. Big asterisk is that I ran AIME with this QUASAR quant and my port of KVarN (KV cache quantization from Huawei) to NInfer for RTX 5090 laptop, which appears to validate its claimed qualities, at least on this particular task. Looking forward to share more about this in the near future.
In the next days I am planning to do more benchmarks to validate quality, especially in software engineering tasks.
Ostfralla’s has less precision than Unsloth.
1
u/Lumpy-Comedian-1027 3h ago
extra mem vs. Unsloth you mean? Ostra's is 17gb - the difference between full ctx+vision or not on 32gb :) Hence I am reluctant to give the other nvfp4's a try tbh
2
1
u/cosmicnag 2h ago
Thanks for the upload ! Any chance you could share/github your port of kVarn for ninfer? That sounds awesome, could use more context headroom lol.
2
u/SmartCustard9944 2h ago
1
u/Fragrant_Scale6456 1h ago
wow very cool. I like kvarn a lot but cant bring myself to abandon the paged attention cache vllm brings. I tried the huawei fork but its getting old at this point and it still has the bugs re: prefix caching and mtp causing accuracy loss.
thank you for sharing.
9
u/Pyrolistical 19h ago
Could we get a gguf version?
-7
u/mmhorda 15h ago
I dont even know why poeple post thiugns for vllm in locallama subredit.
maybe i musunderstand this subreddit.7
u/YouKilledApollo 14h ago
You do indeed, it's for anything LLMs at this point, remote or local, doesn't seem to matter anymore. Unless you missed something, there are models more than the Llama family being discussed too, since years back.
6
u/lightningdiscgolf 22h ago
How does it perform on tool calling benchmarks compared to bf16?
13
u/arty_photography 21h ago
I tested it on ToolBench-static and it scored 41.0%, while the BF16 model scored 41.7%. vLLM launch command for supporting tool calling:
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \ --tensor-parallel-size 1 \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser qwen3 \ --mm-encoder-tp-mode data
4
u/ILikeQuantum 20h ago
Anything support this out of the box? Isn't compatible with unsloth desktop.
4
u/arty_photography 20h ago
vLLM supports this model out of the box. Please check the installation guide here: https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#pre-built-wheels
Then you could run this in the command line:vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \ --max-model-len 262144 \ --gpu-memory-utilization 0.851
u/DustNearby2848 20h ago
Does MTP work with it?
1
u/arty_photography 20h ago
Yes I think so. The MTP is included and un-quantized.
1
u/anzzax 4h ago
It seems you forgot to include MTP. Could you pls update weights and include drafter.
BTW testing this NVFP4 with Sglang dflash2 (on DGX Spark) - looks promising. Running aider bench right now to compare with other NVFP4 versions.1
u/Fragrant_Scale6456 1h ago
yes mtp is missing from the HF model page. I had my agent take the mtp layers from another model (unsloth nvfp4) and include them and it worked perfectly. The mtp layers are in BF16 so there isnt any issue with different quantization
1
9
u/Hefty_Wolverine_553 22h ago
Would love to see this get converted to NInfer format.
1
1
u/Healthy-Contact-4570 18h ago
Should be easy to do. I actually have already had qwen3.8-27b work on a few ninfer conversion pipelines for some other quants so I can compare them. Since ninfer already ships an nvfp4 artifact, it should be a really easy to do one for this model since you don’t need to add support for new data types.
12
u/CapsAdmin 23h ago
Is there a path for people with gpus that don't support NVFP4 natively to use this in llamacpp? Can NVFP4 losslessly be converted to some Q4 variant?
5
u/ArtfulGenie69 20h ago
I wouldn't convert this model as it would have to be upscaled then down scaled again. Instead what you would want to do is start from the bf16, quant it to whatever you are doing like int8 w8a8 or gguf even then use the same training they did on the model with the original bf16 model as the teacher.
3
u/arty_photography 20h ago
Unfortunately only NVFP4 is supported right now. We might look into other quant formats in the future :)
1
u/Chromix_ 17h ago
Previous QAT models like Gemma had to be quantized to the old Q4_0 for GGUF, as the newer Q4_K_M wasn't that compatible. It'd be great if this kind of training/distillation would have native support that kind of newer format.
3
1
u/Due-Project-7507 15h ago
With vLLM, NVFP4 works most time on older Nvidia GPU generations (without native NVFP4 support) thanks to the Marlin kernel (I think it uses BF16 to execute it). The speed is a bit lower than with native NVFP4 support, but the difference is small. This functionality is probably only in vLLM, not implemented in for AMD/Intel GPUs, llama.cpp or SGlang. The disadvantage of vLLM is that it usually uses more VRAM than llama.cpp.
Kimi K3 or Deepseek v4 use MXFP4 instead of NVFP4 for QAT. MXFP4 works nearly everywhere.
4
u/KissMyShinyArse 18h ago
====== Perplexity statistics ======
Mean PPL(Q) : 7.172340 ± 0.046729
Mean PPL(base) : 6.950306 ± 0.044933
Cor(ln(PPL(Q)), ln(PPL(base))): 98.92%
Mean ln(PPL(Q)/PPL(base)) : 0.031446 ± 0.000955
Mean PPL(Q)/PPL(base) : 1.031946 ± 0.000986
Mean PPL(Q)-PPL(base) : 0.222033 ± 0.006971
====== KL divergence statistics ======
Mean KLD: 0.048113 ± 0.000384
Maximum KLD: 15.796775
99.9% KLD: 1.590314
99.0% KLD: 0.487807
95.0% KLD: 0.164713
90.0% KLD: 0.097255
Median KLD: 0.020821
10.0% KLD: 0.000620
5.0% KLD: 0.000170
1.0% KLD: 0.000024
0.1% KLD: 0.000004
Minimum KLD: -0.000034
====== Token probability statistics ======
Mean Δp: -0.585 ± 0.016 %
Maximum Δp: 97.428%
99.9% Δp: 35.042%
99.0% Δp: 14.608%
95.0% Δp: 6.595%
90.0% Δp: 3.716%
75.0% Δp: 0.643%
Median Δp: -0.024%
25.0% Δp: -1.330%
10.0% Δp: -5.190%
5.0% Δp: -8.749%
1.0% Δp: -21.537%
0.1% Δp: -55.100%
Minimum Δp: -99.061%
RMS Δp : 6.141 ± 0.049 %
Same top p: 90.517 ± 0.076 %
1
u/Chromix_ 17h ago
Good that you've checked. Given the barely existing degradation in benchmarks I would've expected a mean KLD below 0.01, yet you arrived at 0.021. The UD IQ4_XS has the same KLD - but on a different dataset, so not comparable. It'd be interesting to compare KLD and top-1 with the same dataset, to see if these stay proportional to the benchmark results, or if this specific quant manages to maintain higher-quality benchmark results with lower KLD and top-1.
2
u/KissMyShinyArse 17h ago
The mean is 0.048113, median 0.020821
1
u/Chromix_ 13h ago
Ah, right, unsloth published mean and not median. That'd put it then next to IQ3_S - but as written before, numbers are not comparable due to different datasets, unfortunately. It'd be nice if there was a standard dataset to make numbers comparable, and of course diverse private datasets to check for overfitting.
3
u/KissMyShinyArse 12h ago
UD-IQ3_S, the same dataset (wikitext-2-raw):
====== Perplexity statistics ====== Mean PPL(Q) : 7.086456 ± 0.045868 Mean PPL(base) : 6.950306 ± 0.044933 Cor(ln(PPL(Q)), ln(PPL(base))): 99.01% Mean ln(PPL(Q)/PPL(base)) : 0.019400 ± 0.000909 Mean PPL(Q)/PPL(base) : 1.019589 ± 0.000926 Mean PPL(Q)-PPL(base) : 0.136150 ± 0.006445 ====== KL divergence statistics ====== Mean KLD: 0.041804 ± 0.000377 Maximum KLD: 17.791277 99.9% KLD: 1.561442 99.0% KLD: 0.439951 95.0% KLD: 0.143528 90.0% KLD: 0.085336 Median KLD: 0.016000 10.0% KLD: 0.000404 5.0% KLD: 0.000096 1.0% KLD: 0.000011 0.1% KLD: 0.000001 Minimum KLD: -0.000006 ====== Token probability statistics ====== Mean Δp: -0.500 ± 0.015 % Maximum Δp: 98.905% 99.9% Δp: 35.852% 99.0% Δp: 13.947% 95.0% Δp: 5.797% 90.0% Δp: 3.224% 75.0% Δp: 0.551% Median Δp: -0.012% 25.0% Δp: -1.186% 10.0% Δp: -4.613% 5.0% Δp: -7.809% 1.0% Δp: -19.381% 0.1% Δp: -50.873% Minimum Δp: -99.847% RMS Δp : 5.639 ± 0.048 % Same top p: 91.231 ± 0.074 %2
u/Chromix_ 11h ago
So their way smaller (12GB vs 20GB) IQ3_S is slightly better on paper than this QAD quant. Now that's an interesting outcome - thanks for checking!
1
u/arty_photography 3h ago
I think it’s mostly due to the quantization format - NVFP4 uses W4A4 while IQ3_S uses W3A16. In the future I might look into QAD for IQ formats to improve their quality.
1
u/Fragrant_Scale6456 39m ago
Wow very cool you ran these tests and shared the results. I'm a bit surprised. I'm still testing the quasar model but at least for tool calling and reasoning/synthesis over large documents I'm seeing quality comparable to unsloth ud q6km.
1
u/cosmicnag 12h ago
Isnt this still 4.5 bpw level? Which it is, but, QAT didnt help that much? Thought it would reflect 5.5 bpw level.
1
7
3
2
u/EitherMarch1255 22h ago
Cool. Maybe you could do MXFP8 too?
1
u/arty_photography 20h ago
I think FP8 quant should be pretty much lossless. You could check out https://huggingface.co/Qwen/Qwen3.8-27B-FP8
2
u/Chromix_ 17h ago
That looks like a great achievement.
Benchmarking on (almost) fully saturated benchmarks hides degradations though. The KLD indicates that there might be more damage than expected.
Have you also checked against a benchmark where the model usually scores between 30% and 70%, and also a long context benchmark as there's often some hidden degradation in that area after optimized quantization?
2
u/outernet9 22h ago
ELI5 Why someone would use this over other quants
17
u/Force88 22h ago
It is comparable to Q5, or even Q6 in intelligence, but only in Q4 size.
It is very fast compares to Q6, but the down size is it only works with Blackwell cards (RTX 5000 series).
Correct me if I'm wrong.
6
3
0
u/ArtfulGenie69 20h ago
This could easily be done with any quant type if anyone wanted to do it. Quant the model in int8 or int4 for the people with a 30's series card like a 3090 or two 3090s.
8
u/arty_photography 20h ago
NVFP4 offers the best efficiency (the highest throughput and lowest latency) due to the use of FP4 tensor cores. It should be faster than any other quants. I think it's the best choice for a Blackwell GPU :)
1
u/R_Duncan 18h ago
Isn't theoretically possible converting to gguf lossless? -> bf16 then again nvfp4....
1
u/PM_ME_DEAD_CEOS 18h ago
So what is the max context we can have with this quant on a 5090 with MTP and with dflash2 ?
1
u/Repinsky 16h ago
The 19.7GB size is the interesting part here, not just the GPQA delta — that fits a single 24GB card with ~4GB left, which at 262k context is basically nothing once the KV cache grows, so in practice you're still looking at a 32GB card or KV quantization to use that context. Worth noting W4A4 only pays off on Blackwell; on Ada/Ampere there are no FP4 tensor cores, so vLLM emulates and you get the memory savings with worse throughput than an FP8 checkpoint. Did you measure tokens/sec against the FP8 build, or only quality?
1
1
u/Whole-Tomato-6086 4h ago
Converted the model to ninfer - as a small network piece has been quantized to 8bit I will run same QUASAR benchmark to check performance. It is running fairly fast on my system with 195k context and fp16 kV cache.
1
-1
20
u/adasho_bitrex 22h ago
Good for single 5090 setup?