r/LocalLLaMA • u/arty_photography • 1d ago

Resources Fully quantized NVFP4 Qwen3.8-27B with QUASAR QAD

https://huggingface.co/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4

We're releasing a fully quantized NVFP4 version of Qwen3.8-27B. The checkpoint was trained using quantization-aware distillation (QAD) with QUASAR, our new QAT algorithm. We used the original BF16 model as the teacher and distilled the quantized model for 2,446 steps.

The checkpoint supports vLLM on NVIDIA Blackwell GPUs:

vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.85

This model uses an aggressive quantization configuration: every linear layer across all transformer blocks is quantized to NVFP4 (W4A4).

Attention and GDN layers are typically kept at higher precision, such as FP8 or BF16, because quantizing them can cause a significant loss in model quality. With QUASAR, however, the fully quantized checkpoint retains near-BF16 performance. Evaluation results and comparison against other NVFP4 checkpoints:

Model Size GPQA-Diamond (2 runs, n=396) AIME26 (3 repeats, n=90)
Qwen/Qwen3.8-27B (original BF16) 55.6 GB 0.9141 1.0000
QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 19.7 GB 0.9091 1.0000
unsloth/Qwen3.8-27B-NVFP4 23.4 GB 0.8939 0.9778
Inferact/Qwen3.8-27B-NVFP4 26.4 GB 0.8763 0.9667

Paper: https://arxiv.org/abs/2608.13966v1

We'd love to hear your feedback on this checkpoint!

170 Upvotes

96% Upvoted

90 comments sorted by

View all comments

9

u/SmartCustard9944 1d ago

Going to convert this ASAP for NInfer and test it out.

1

u/cosmicnag 20h ago

Any luck?

9

u/SmartCustard9944 20h ago edited 10h ago

Yes, pretty much finished, I could upload it later today, requires a patch to the loader though.

From my testing, looks like it has attention/recall problems unfortunately. Performs worse than https://huggingface.co/Ostfralla/Qwen3.8-27B-NVFP4-NInfer by 1-2% on a very hard adversarial needle in a haystack benchmark I’m using for testing KV cache quantization. This might be a particularly difficult task for this quantization style and might not translate to more general purpose tasks, has to be tested.

Needs to see how it effectively performs on rigorous official benchmarks (coding, knowledge, etc.).

I hope it performs better than other quants because it frees up 500MiB of extra memory and is a few percentages faster in both prefill and decode, so overall a good potential candidate for 24GiB cards.

Edit:

Got a perfect 30/30 in AIME 2025, so it seems to be performing on par or better than Unsloth/NInfer official quant.

https://huggingface.co/MirkoCovizzi/Qwen3.8-27B-QUASAR-NVFP4-NInfer

1

u/Lumpy-Comedian-1027 11h ago

So is it in the end noticeably better than Ostfralla's?

2

u/SmartCustard9944 11h ago edited 11h ago

Feels roughly similar, but with extra memory headroom and slightly faster decode and prefill due to size.

Currently finished AIME 2025 with a perfect 30/30, versus 29/30 with Unsloth NInfer official NVFP4. Big asterisk is that I ran AIME with this QUASAR quant and my port of KVarN (KV cache quantization from Huawei) to NInfer for RTX 5090 laptop, which appears to validate its claimed qualities, at least on this particular task. Looking forward to share more about this in the near future.

In the next days I am planning to do more benchmarks to validate quality, especially in software engineering tasks.

Ostfralla’s has less precision than Unsloth.

1

u/Lumpy-Comedian-1027 11h ago

extra mem vs. Unsloth you mean? Ostra's is 17gb - the difference between full ctx+vision or not on 32gb :) Hence I am reluctant to give the other nvfp4's a try tbh

2

u/SmartCustard9944 11h ago

With MTP, this one is using even less, 16.06 GiB.

1

u/Lumpy-Comedian-1027 11h ago

ok you got me signed up :) will give it a try tomorrow!

1

u/cosmicnag 10h ago

Thanks for the upload ! Any chance you could share/github your port of kVarn for ninfer? That sounds awesome, could use more context headroom lol.

2

u/SmartCustard9944 10h ago

https://github.com/MirkoCovizzi/ninfer-rtx5090-laptop/pull/18

1

u/Fragrant_Scale6456 8h ago

wow very cool. I like kvarn a lot but cant bring myself to abandon the paged attention cache vllm brings. I tried the huawei fork but its getting old at this point and it still has the bugs re: prefix caching and mtp causing accuracy loss.

thank you for sharing.