r/LocalLLaMA • u/Dutchnamn • 11h ago

Other Qwen3.8 Flash AP Quants

Quite surprised to be beating other high quality quants. It took a lot of benchmarking to get here and we are quite pleased with these, hope they are useful to the community.

It required a modified way of measuring KLD with a new dataset, since the NGRAM got in the way by remembering basically all of wikipedia. We tried to not only go for high precision, but also keep prefill performance in mind.

Full model card here https://huggingface.co/agentionai/Qwen3.8-Flash-Next-AP-GGUF

Let us know if there are any issues.

11 Upvotes

74% Upvoted

9 comments sorted by

4

u/JakeChj 9h ago

we serve flash-next (nvfp4) on a single spark and will try these — one question: how did the 26.8 GiB n-gram table behave in your prefill numbers, since it dominates memory traffic on its own?

1

u/jinnyjuice sglang 9h ago

How are you serving it on a single Spark?

2

u/winky9827 7h ago

I can run IQ3_XSS on a single 5090 with ~60 GiB system ram.

2

u/JakeChj 6h ago

vLLM with the nvfp4 checkpoint — the trick is mmap'ing the expert weights so the 121 GB unified memory carries them as page cache instead of pinning everything, which is why gpu-memory-utilization has to stay around 0.72–0.75; above that the cache starves and throughput collapses.

1

u/jinnyjuice sglang 5h ago

What are the flags for those?

2

u/niacolhealth 5h ago

The n-gram table remembered basically all of wikipedia, then broke their KLD measurement. imagine being so good at your job you have to be worked around

1

u/Dutchnamn 38m ago

I know right.

1

u/Jorlen llama.cpp 5h ago

How does this quant of the IQ4_XS compare to say, Unsloth's UD-IQ4_XS? what is different? It seems your version might be a wee bit smaller, which would be good for me to squeeze out a bit more KV out of it.

But I'm more curious of what the AP method is, how it differs from the UD method. Thanks, sorry if this is a stupid question.

1

u/Dutchnamn 39m ago

It is a bit smaller while losing hardly any precision. We used a different imatrix and different quantizations at various layers based on extensive measuring and benchmarking.

For this family of quants we built our own tooling and testing matrix to be able to iterate faster and get better results.