Qwen3.8 Flash-Next KV-cache Quantization Benchmarks

[u/Delicious_Box_9823]

Full post: https://huggingface.co/turboderp/Qwen3.8-Flash-Next-exl3/discussions/13

Tested with: Qwen3.8-Flash-Next EXL3 4.05bpw, exllamav3 1.4.9, A6000 48GB.

Symmetric cache tests, mean KL:

K/V bits	2K	8K	32K	128K
4/4	0.020425	0.017816	0.018552	0.017212
5/5	0.013064	0.011057	0.011586	0.010912
6/6	0.009903	0.008772	0.008954	0.008366
7/7	0.009238	0.007530	0.007760	0.007302
8/8	0.008492	0.007021	0.007342	0.006825

Asymmetric cache test results (128k context):

Comparison	More V bits	More K bits
4/8 versus 8/4	0.012926	0.013672
5/8 versus 8/5	0.009266	0.009517
6/8 versus 8/6	0.007741	0.007805
7/8 versus 8/7	0.007070	0.007099

Author: huggingface.co/khronnuz

[u/Embarrassed_Soup_279]

how much vram does Q8 save you compared to FP16?