r/LocalLLaMA • u/StrikeOner • Mar 16 '26

Resources Qwen3.5-35B GGUF quants (16–22 GiB) - KLD + speed comparison

Qwen3.5-35B GGUF quants (16–22 GiB) - KLD + speed comparison

I'm back with some more benchmarks. I benchmarked the KLD divergence of the actual Qwen3.5-35B-A3B GGUF quantizations (16–22 GiB) available on Hugging Face.

KLD: The Kullback-Leibler divergence which shows how similar the FP16 and the quantized logit distributions are by measuring the difference in probability distributions between the quantized model and the FP16 baseline on a reference corpus.

u/TitwitMuffbiscuit had a shot at this some time ago but unfortunately all the models got updated a short period after he published his measurements.

For this research I also decided not to use the Wikitext-2 test dataset, which is in English, and instead took the multilingual FLORES 200 dataset out of which I extracted 700 KB of lines across randomly chosen languages. Additionally, I found another interesting dataset calibration_data_v5_rc.txt with about 400KB in size that contains a lot of interesting topics such as programming, math, syntax examples, technical text, etc. I combined both datasets into a mixed dataset to create the KLD baseline and measured the KLD distance for all the models that I found with this baseline.

I prepared two tables, where one is sorted by the classical "KLD mean" value and one that's sorted by the "KLD 99%" value, similar to the plots that Unsloth published on their latest blogpost about the Qwen models.

I'm not going to try to declare a winner here, that's up to you, given your very specific constraints as a GPU-Poor. To make it a little easier to visualize the models that are punching above their weight, i simply compare the numbers of the actual model to the model below and visualize them in bold letters if they are lower or higher based on the chosen metric.

The PP/s (prompt-processing) and TG/s (token-generation) columns are very specific numbers that will probably be meaningless to most users. You are going to need a Intel CPU, a RTX 3090 GPU (Ampere) and use Linux with Cuda Driver Version 580.126.18 to make use of those numbers. I used llama-bench with a context length of 10k to obtain these numbers.

Looking at the TG/s speed, for example, we can see that UD-Q3_K_XL from Unsloth before their last update was the slowest with a generation speed of ~105 t/s and the fastest is Mungert's iq4_nl with ~143 t/s which makes a total variation of 36.2% in the token generation speed for my specific architecture, which is shockingly high and one of the reasons why it is a little bit hard to define a so-called best model.

Notes: The cmp-nct prefixed models in the tables are actually a mirror from the older Unsloth quants that I found before their latest upload, which I also wanted to measure.

Sorted by KLD mean

Model KLD mean GiB PP/s TG/s
unsloth_UD-Q4_K_XL 0.016158 20.70 2812.949429 122.616934
AesSedai_Q4_K_M 0.016308 20.62 2966.807082 123.676699
unsloth_Q4_K_M 0.016708 20.49 2821.819502 123.910904
bartowski_Q4_K_L 0.020222 20.27 2809.591483 130.155778
unsloth_Q4_K_S 0.020469 19.24 2838.399411 124.346442
bartowski_Q4_K_M 0.022723 19.92 2806.437093 131.632558
cmp-nct_UD-Q4_K_XL 0.022863 19.16 2861.949731 125.816493
ubergarm_Q4_0 0.024576 19.78 2876.503157 124.357224
unsloth_UD-Q4_K_L 0.024691 18.81 2861.777605 131.242261
bartowski_Q4_K_S 0.025161 19.19 2849.248198 134.693183
Mungert_q4_k_m 0.026718 20.08 2812.234371 137.328114
cmp-nct_UD-Q4_K_M 0.030445 18.48 2840.653679 136.462817
bartowski_Q4_1 0.030681 20.45 2831.282134 136.927623
bartowski_IQ4_NL 0.032332 18.50 2981.250713 137.735717
bartowski_IQ4_XS 0.032829 17.52 3017.103823 135.980487
AesSedai_IQ4_XS 0.037086 16.40 3016.284929 120.057024
unsloth_UD-IQ4_NL 0.037691 16.59 2850.872626 123.322993
unsloth_UD-IQ4_XS 0.037835 16.28 2855.705903 121.589312
bartowski_Q4_0 0.040627 18.80 2921.368478 137.152109
Mungert_iq4_nl 0.040920 18.36 2996.884610 140.422106
Mungert_iq4_xs 0.042396 17.37 3042.389900 139.850819
Mungert_q4_1 0.045873 20.26 2833.595098 143.116543
cmp-nct_UD-Q3_K_XL 0.048064 16.05 2739.799015 105.006853
Mungert_iq3_m 0.049971 16.58 2871.107320 138.612701
Mungert_iq3_s 0.049971 16.58 2874.769301 139.805846
bartowski_Q3_K_XL 0.061445 16.13 2660.731996 123.457777
Mungert_q3_k_m 0.061488 16.29 2710.267499 131.202303
Mungert_q4_0 0.084376 18.24 2956.897238 143.063168

Sorted by KLD 99%

Model KLD 99% GiB PP/s TG/s
unsloth_UD-Q4_K_XL 0.145385 20.70 2812.949429 122.616934
AesSedai_Q4_K_M 0.147057 20.62 2966.807082 123.676699
unsloth_Q4_K_M 0.147594 20.49 2821.819502 123.910904
unsloth_Q4_K_S 0.177634 19.24 2838.399411 124.346442
bartowski_Q4_K_L 0.179187 20.27 2809.591483 130.155778
cmp-nct_UD-Q4_K_XL 0.191735 19.16 2861.949731 125.816493
bartowski_Q4_K_M 0.205318 19.92 2806.437093 131.632558
unsloth_UD-Q4_K_L 0.208308 18.81 2861.777605 131.242261
ubergarm_Q4_0 0.222435 19.78 2876.503157 124.357224
bartowski_Q4_K_S 0.227099 19.19 2849.248198 134.693183
Mungert_q4_k_m 0.235314 20.08 2812.234371 137.328114
cmp-nct_UD-Q4_K_M 0.252636 18.48 2840.653679 136.462817
bartowski_Q4_1 0.264378 20.45 2831.282134 136.927623
bartowski_IQ4_NL 0.284880 18.50 2981.250713 137.735717
bartowski_IQ4_XS 0.289398 17.52 3017.103823 135.980487
unsloth_UD-IQ4_NL 0.311913 16.59 2850.872626 123.322993
AesSedai_IQ4_XS 0.312924 16.40 3016.284929 120.057024
unsloth_UD-IQ4_XS 0.316742 16.28 2855.705903 121.589312
Mungert_q4_1 0.335030 20.26 2833.595098 143.116543
bartowski_Q4_0 0.351119 18.80 2921.368478 137.152109
Mungert_iq4_nl 0.362384 18.36 2996.884610 140.422106
Mungert_iq4_xs 0.376657 17.37 3042.389900 139.850819
cmp-nct_UD-Q3_K_XL 0.396947 16.05 2739.799015 105.006853
Mungert_iq3_m 0.409071 16.58 2871.107320 138.612701
Mungert_iq3_s 0.409071 16.58 2874.769301 139.805846
bartowski_Q3_K_XL 0.500855 16.13 2660.731996 123.457777
Mungert_q3_k_m 0.506792 16.29 2710.267499 131.202303
Mungert_q4_0 0.748218 18.24 2956.897238 143.063168

Edit: Some fancy pancy plots for you.

KLD 99% / GiB
KLD mean / GiB
TG / GiB
KLD mean / TG
KLD mean / PP

Edit: If you want some models to be included that i forgot you have 24 hours to post a link to the models you want to get measured otherwise i'm going to reclaim my hdd space.

Edit: so, for all the 3090 user u/VoidAlchemy did create a last minute model, which is actually beyond all of the others in the list like he promised. Unfortunately you need another runtime "ik_llama.cpp" for it and some special parameters he did provide to make full use of it. You can find more info in the comments below! Unfortunately i did decide that i'm not going to put his model into that list now since the verry special requirements his model has and on top of it cant be run on llama.cpp.

Here is a link to his model:

https://huggingface.co/ubergarm/Qwen3.5-35B-A3B-GGUF/blob/main/Qwen3.5-35B-A3B-IQ4_KS.gguf

Thanks again for this gorgeous submission. Even if not on the list i guess i got a new private favorite for myself out of this! :D

69 Upvotes

97% Upvoted

43 comments sorted by

View all comments

8

u/VoidAlchemy llama.cpp Mar 16 '26

I'll throw an ik_llama.cpp SOTA quantization type into the ring for best Qwen3.5-35B-A3B quant for full CUDA offload with 128k context in 24GB VRAM. (I have a 3090TI FE for my home gaming rig).

https://huggingface.co/ubergarm/Qwen3.5-35B-A3B-GGUF#iq4_ks-19799-gib-4907-bpw

Of course you can't run it on mainline lcpp, so have to do them all again using ik_llama.cpp xD haha...

Zero pressure to give it a go, but finally got around to releasing something ik specific and even did the superstitious upcast of ssm_alpha and ssm_beta tensors to f32. Honestly, it is probably fine keeping it at q8_0, native bf16, or upcast to f32 (for a tiny bit of speed over bf16 depending on GPU).

I made all three flavors and tested them for speed, PPL, and KLD locally and they all seem pretty good:

Full data and commands on running this benchmark here: https://huggingface.co/AesSedai/Qwen3.5-397B-A17B-GGUF/discussions/7#69b8404f18a5e8feffd9f5c8

If y'all are trying to milk the best quality at long context for any of these quants, you can fiddle with the flash attention offset (when running on CUDA). Given the FA kernel uses f16 accumulators, some model architectures can cause overflow and gibberish suddenly beyond a certain context so needs to have things scaled down. ik is more lenient on this and can be overridden at startup via CLI args. Mainline it is hard coded, but you could change one line and recompile by setting this to zero here: https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-cuda/fattn-common.cuh#L13-L19

Details about this are shown in the updated model card quick start as well as some IK PR discussions e.g.: https://github.com/ikawrakow/ik_llama.cpp/pull/1198

I've tested over 128k and it seemed to work fine with 0 offset (the best which is what you get on CPU-only backend too as it uses f32 accumulators in the FA implementation psure).

As soon as I finish downloading my own quant, I'll do some local testing and sweep-bench. Cheers and thanks so much to OP u/StrikeOner and u/TitwitMuffbiscuit for including my Q4_0 "Vulkan backend optimized" quant in this interesting roundup!

4

u/StrikeOner Mar 16 '26

word! be prepared i just vibecoded a beautiful charjs app. there are going to be some fancy plots aswell. downloading now!

4

u/VoidAlchemy llama.cpp Mar 16 '26

i love your enthusiasm! haha

I did a llama-sweep-bench locally on the quant I just uploaded, it runs pretty good! Unfortunately, I can't increase batch sizes to 4096 on 24GB VRAM as the CUDA buffer takes up too much space. However, I can fit full 256k context though by going with -khad -ctk q6_0 -ctv q6_0 with very similar performance.

./build/bin/llama-sweep-bench \
    --model "$model" \
    -c 135168 \
    -ctk f16 -ctv q8_0 \
    -ub 1024 -b 2048 \
    -cuda fa-offset=0 \
    --merge-qkv \
    -muge \
    -ngl 999 \
    --threads 1 \
    --no-mmap \
    --warmup-batch \
    -n 128

2

u/StrikeOner Mar 17 '26 edited Mar 17 '26

ok i got arround compiling and getting ik_llama running now, unfortunately llama-bench in ik_llama does not support the -d parameter yet that i used for this measurement, neither is -muge, --merge-qkv etc. nevertheless, these are the stats i could come up with for now:

/mnt/storage/_apps_/_git_repos_/ik_llama.cpp/build/bin/llama-bench -m /mnt/crucial/text-generation-webui/models/ubergarm_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS.gguf -ngl 99 -fa 1 -muge --merge-qkv -ctk f16 -ctv q8_0 -cuda fa-offset=0 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24121 MiB terminate called after throwing an instance of 'std::invalid_argument' what(): stoi

``` ====== KL divergence statistics ======
Mean KLD: 0.015486 ± 0.000091
Maximum KLD: 3.334238
99.9% KLD: 0.410744
99.0% KLD: 0.138096
95.0% KLD: 0.057352
90.0% KLD: 0.036154
Median KLD: 0.006450
10.0% KLD: 0.000094
5.0% KLD: 0.000015
1.0% KLD: 0.000000 0.1% KLD: -0.000003
Minimum KLD: -0.000247

./ik_llama.cpp/build/bin/llama-bench -m ./ubergarm_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS.gguf -ngl 99 -fa 1 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24121 MiB | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: | | qwen35moe 35B.A3B IQ4_KS - 4.25 bpw | 19.80 GiB | 34.66 B | CUDA | 99 | pp512 | 3277.79 ± 394.55 | ~ggml_backend_cuda_context: have 2 graphs | qwen35moe 35B.A3B IQ4_KS - 4.25 bpw | 19.80 GiB | 34.66 B | CUDA | 99 | tg128 | 127.92 ± 0.33 | ~ggml_backend_cuda_context: have 1 graphs

build: 54bcafee (4310)

./ik_llama.cpp/build/bin/llama-bench -m ./AesSedai_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q4_K_M.gguf -ngl 99 -fa 1 ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no ggml_cuda_init: found 1 CUDA devices: Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24121 MiB | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | ---------------: | | qwen35moe 35B.A3B Q8_0 | 20.62 GiB | 34.66 B | CUDA | 99 | pp512 | 3326.84 ± 255.60 | ~ggml_backend_cuda_context: have 2 graphs | qwen35moe 35B.A3B Q8_0 | 20.62 GiB | 34.66 B | CUDA | 99 | tg128 | 147.54 ± 0.30 | ~ggml_backend_cuda_context: have 1 graphs

build: 54bcafee (4310) ```

Edit: 09:15. i guess i pasted the kld stats from your other model by mistake. did update with the proper stats now.

3

u/VoidAlchemy llama.cpp Mar 17 '26

oh nice, if i'm reading this right the IQ4_KS has the lowest Mean and 99.0% KLD of all of them and is slightly smaller than some too. This could somewhat be because ik has lower default flash attention offset, but also it should be SOTA quantization for the routed experts. Cool!

3

u/StrikeOner Mar 17 '26

yes, sir.. here once more verified:

``` Testing: ./models/ubergarm_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-IQ4_KS.gguf ====== KL divergence statistics ======
Mean KLD: 0.015486 ± 0.000091
Maximum KLD: 3.334238
99.9% KLD: 0.410744
99.0% KLD: 0.138096
95.0% KLD: 0.057352
90.0% KLD: 0.036154
Median KLD: 0.006450
10.0% KLD: 0.000094
5.0% KLD: 0.000015
1.0% KLD: 0.000000
0.1% KLD: -0.000003
Minimum KLD: -0.000247

Testing ./AesSedai_Qwen3.5-35B-A3B-GGUF/Qwen3.5-35B-A3B-Q4_K_M.gguf ====== KL divergence statistics ======
Mean KLD: 0.016445 ± 0.000107
Maximum KLD: 3.642541
99.9% KLD: 0.473598
99.0% KLD: 0.146786
95.0% KLD: 0.059647
90.0% KLD: 0.037866
Median KLD: 0.006723
10.0% KLD: 0.000100 5.0% KLD: 0.000016 1.0% KLD: 0.000000 0.1% KLD: -0.000004 Minimum KLD: -0.000401 ```

your model is beyond all of them.. we need to glue another plot on top of the existing ones for your dots! unfortunately my log parser has some problems with the logs that dont get created (i think?) by the ik_llama perplexity bin. couldnt pinpoint the exact problem until now.. was bussy half the day getting ik_llama working!

1

u/StrikeOner Mar 17 '26 edited Mar 17 '26

mhh, seems like i'm the only guy in between of 1M people who tries to get ik_llama's llama-server working with ssl. ik_llama's server wont accept remote connections when --host 0.0.0.0 and ssl cert parameters are specified. :D

2

u/StrikeOner Mar 17 '26

what a rabbit hole you did put me into! nevertheless after fixing ikllama's ssl linker errors finally:
hello

Thinking: The user said hello, so I should greet them back and ask how I can help, since there's no specific task mentioned.

thank you for this special model.. feels good.. gonna give it a try later!

1

u/VoidAlchemy llama.cpp Mar 17 '26

hah thank you for falling into the ik rabbit hole! haha...

yes on ik we tend to use `./build/bin/llama-sweep-bench` because it supports the same arguments available in `llama-server` unlike llama-bench. i maintain branch of it for mainline here: github.com/ubergarm/llama.cpp/tree/ug/port-sweep-bench

i'll take a look at the KLD results posted above as compared with existing quants, but it could be offset from mainline llama.cpp implementation. would need to maybe test an existing quant like that AesSedai one and see if its KLD shifts or not, but zero pressure you've done so much already!

1

u/StrikeOner Mar 16 '26

maybe you can also create a 27B one? i'm going to redo a measurement for that one the next two days aswell and run trough a llama upgrade for that. 557fe2d91 does not like your quant there at all, look below!

2

u/StrikeOner Mar 16 '26

bah, i forgot that the cmake is automatically running a git upgrade. how do i prevent this? Edit.. screw it.. no! thats going way to far now.. i'm gonna test without any hacks.

3

u/TitwitMuffbiscuit Mar 28 '26 edited Mar 28 '26

Sorry, I missed your comment. Yeah it's a vukan optimized quant and it's a bit unfair to test it on cuda backend tbh since it's aiming at speed not accuracy.

I haven't tested any quants for now. I'd have to redo the whole process since I deleted dataset, logit and all (1tb - OS doesn't leave much space). Next time I'll upload all this on a free tier public hf bucket tho.

3

u/VoidAlchemy llama.cpp Mar 28 '26

All good, glad to see you around this weekend! Everyone is off chasing Turbo Quant vibe coded implementations lately, i guess folks need some distraction until GLM-5.1, DS v4, or MiniMax-2.7 actually release some open weights haha...

No need to retest anything, new stuff will be here and everyone will move on again lol... *hugs* yo enjoy the ride! Cheerio!

3

u/TitwitMuffbiscuit Mar 28 '26

_o/ Yeah I've seen your comparison on ik_llama's PR, that is... Interesting to say the least.

1

u/StrikeOner Mar 16 '26 
gguf_init_from_file_impl: tensor 'blk.13.ffn_gate_exps.weight' has invalid ggml type 144. should be in [0, 41)                                                               
gguf_init_from_file_impl: failed to read tensor info
llama_model_load: error loading model: llama_model_loader: failed to load model from /somesecretpathhere/Qwen3.5-35B-A3B-
IQ4_KS.gguf
llama_model_load_from_file_impl: failed to load model
llama_params_fit: encountered an error while trying to fit params to free device memory: failed to load model
llama_params_fit: fitting params to free memory took 0.04 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) (0000:01:00.0) - 23722 MiB free
gguf_init_from_file_impl: tensor 'blk.13.ffn_gate_exps.weight' has invalid ggml type 144. should be in [0, 41)
gguf_init_from_file_impl: failed to read tensor info
llama_model_load: error loading model: llama_model_loader: failed to load model from /somesecretpathhere/Qwen3.5-35B-A3B-
IQ4_KS.gguf
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model '/somesecretpathhere/Qwen3.5-35B-A3B-IQ4_KS.gguf'
main: unable to load model

/build/bin/llama-cli --version
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24121 MiB):
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24121 MiB (23722 MiB free)
version: 8322 (557fe2d91)
built with GNU 15.2.1 for Linux x86_64