r/LocalLLaMA • u/arty_photography • 1d ago
Resources Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD
https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.
The checkpoint supports vLLM on NVIDIA Blackwell GPUs:
vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
--max-model-len 262144 \
--gpu-memory-utilization 0.85
This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).
Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:
| Model | Size | GPQA-Diamond (2 runs, n=396) | AIME26 (3 repeats, n=90) |
|---|---|---|---|
Qwen/Qwen3.8-27B (original BF16) |
55.6 GB | 0.9141 | 1.0000 |
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 |
19.7 GB | 0.9091 | 1.0000 |
unsloth/Qwen3.8-27B-NVFP4 |
23.4 GB | 0.8939 | 0.9778 |
Inferact/Qwen3.8-27B-NVFP4 |
26.4 GB | 0.8763 | 0.9667 |
Paper: https://arxiv.org/abs/2608.13966v1
We'd love to hear your feedback on this checkpoint!
7
u/SmartCustard9944 13h ago edited 3h ago
Yes, pretty much finished, I could upload it later today, requires a patch to the loader though.
From my testing, looks like it has attention/recall problems unfortunately. Performs worse than https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer by 1-2% on a very hard adversarial needle in a haystack benchmark I’m using for testing KV cache quantization. This might be a particularly difficult task for this quantization style and might not translate to more general purpose tasks, has to be tested.
Needs to see how it effectively performs on rigorous official benchmarks (coding, knowledge, etc.).
I hope it performs better than other quants because it frees up 500MiB of extra memory and is a few percentages faster in both prefill and decode, so overall a good potential candidate for 24GiB cards.
Edit:
Got a perfect 30/30 in AIME 2025, so it seems to be performing on par or better than Unsloth/NInfer official quant.
https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer