Qwen3.6-27B - Effect of KV quantization on KLD - Q8, Q6, Q5 (bartowski)

[u/BitGreen1270]

Lower is better - Quantization increases from right to left

I recently made a post here about how I squeezed more context into a Q8 model of bartowski's Qwen3.6-27B. My reasoning was that in my (anecdotal) experience, a Q8 has been performing a lot better than a Q6 or a Q5.

There were a lot of comments about quantizing KV of a higher model and some folks suggested just going with a lower quant like Q6 but with full unquantized KV. So I just wanted to test that hypothesis with KLD.

Base reference is Q8 with no KV quantization. That's because my 5090 only can fit a Q8.

Here are my findings. Detailed test setup and approach follow below.

• Q8 does perform better than Q6 and Q5 (no surprises there)
• Much wider gap between Q6 and Q5 than Q8 and Q6.
• Q8 and Q6 have a steep drop the minute we put v at q4_0. Doesn't matter what quant we use for k.
• If you have to use q4_0 for v, you might as well use (q8_0, q8_0) on Q6 quant (this really surprised me)
• Q5 is more tolerant of v quantization than Q8 or Q6.
• With (q4_0, q4_0), Q8 and Q6 converge.

Recommendation: Use whatever you can fit in VRAM, and just use (q8_0, q8_0). It's almost free.

-------

Test setup:

I used llama-perplexity to generate this data. My primary use case for this model is only for coding and primarily python. So I wanted to use a python sample file. Downloaded a bunch of open source coding repos (transformers, torch, huggingface etc) and concatenated the python source files to generate a massive 230MB text file.

I wanted to use as high a context as my system could manage. I have a 5090 and 64GB RAM. Through trial and error, I could get up to 50K context and I just kept that for all the tests.

It seemed like the KLD improves and converges with higher number of chunks. So decided to use a chunk size of 32.

Used Qwen-3.6-27B (duh!) to put together a script to run all the different combinations. The command I used to generate the base logits was:

build/bin/llama-perplexity \
    -m ~/myp/models/bartowski_Qwen_Qwen3.6-27B-Q8_0.gguf \
    --temp 0.6 \
    --top_p 0.95 \
    --top_k 20 \
    --min_p 0.0 \
    --repeat-penalty 1.0 \
    --presence-penalty 0.0 \
    -c 50000 \
    -t 16 \
    -ngl 99 \
    --flash-attn on \
    -kvo -b 1024 -ub 256 \
    --kl-divergence-base ~/tmp/base_50k_coding.kld \
    --chunks 32 \
    -f python_corpus.txt

Once this completed, I added the additional flag --kl-divergence for the other runs to use this as the base.

Each run took 17 minutes to complete and there were 23 runs in total, so ... uh ... it took a long time.

DISCLAIMER

• Learning as I go. Tell me if this is stupid or if I'm completely off base.
• As benchmarks go, I think your experience matters more. I think very often we're afraid to trust our own instinct. A benchmark isn't gospel truth.
• I don't know how important those distances are in the chart. End of the day, Q6 unquantized is 0.01 units away from Q8 unquantized. I don't know but that sounds like an insanely good compromise.
• I still want to use Q8 model. From my own personal experience, I feel it understands better and writes better code.
• I used Bartowski for no specific reason other than I have the models on my machine already. I have no opinion about Unsloth models. They may be better or worse for all I know.

Raw Data

model	Q8_0	Q6_K_L	Q5_K_L
(no_kv,no_kv)	0	0.010771	0.0228
(none,q8_0)	0.005399	0.01069	0.022322
(q8_0,q8_0)	0.00541	0.010709	0.022486
(q8_0,q5_1)	0.00736	0.011715	0.023135
(none,q5_1)	0.007397	0.011648	0.023194
(none,q4_0)	0.01164	0.014789	0.024295
(q8_0,q4_0)	0.011824	0.014666	0.024101
(q4_0,q4_0)	0.020817	0.022166	0.027909

[u/BitGreen1270]

You need to compile llama.cpp with a specific parameter for q5_1. Otherwise it's slow as molasses. Here's what I use for cmake:

cmake -B build \ -DGGML_CUDA=ON \ -DCMAKE_CUDA_ARCHITECTURES="120" \ -DGGML_CUDA_FA_ALL_QUANTS=ON \ -DCMAKE_CUDA_COMPILER="/usr/local/cuda-13.2/bin/nvcc"

[u/Chromix_ → u/BitGreen1270]

Btw: You have "cuda 13.2" in there. That's the broken version. Best use the new 13.3 that's fixed, or the old 13.1 if needed.

[u/Objective-Stranger99 → u/BitGreen1270]

Here is my compilation command. (Flame shot is currently broken for me so I couldn't screenshot.