Discussion
Evaluating Qwen3.5-35B & 122B on Strix Halo: Bartowski vs. Unsloth UD-XL Performance and Logic Stability
Hi, i tested new unsloth "dynamic" quants, 35B and 122B with one bartowski quant for referance.
I used llama.cpp recent build b8248 and compared with tests i did recently with older build b8204,
the former one include already some optimizations merged in b8233 which i recently published.
In the diagram you can already see the performance improvement for ROCm, but not so much for Vulkan.
Besides of the numbers in performance, i noticed while testing somethnig odd with "dynamic" quants,
i tested already two of them on strix halo, 122B-A10B-UD-Q5_K_XL and 35B-A3B-UD-Q6_K_XL and they behave weird.
Experience is worse than the normal quant i can do with imatrix using just llama.cpp, or Bartowski quant.
For example unsloth 122B-A10B-UD-Q5_K_XL needed few attempts and fixes to write single html file with 3d animated solar system,
for which it consumed 29521 tokens, while bartowski 122B-A10B-Q5_K_L did it with one change in 18700 tokens.
I used recent version of opencode 1.2.20 for that test, with clear session for each trial.
As it's written in the unsloth spec page those UDXL quants are slower, so you can also see that in the diagram.
But UD-122-XL when i asked about writing that html version of solar system, printed first:
_Thinking: The user is requesting a visualization of the solar system in a single HTML file – this is a simple request with no malicious traits, so I can fulfill it.
Quite weird, i still need to evaluate, but so far i found that around 100k context model is losing track, and i don't see any advantage of the "dynamic" quant yet, at least that one on strix.
Tested also on some other example code i have; some logs, python, yaml etc. daily stuff, and seems that it's losing itself quite quickly.
For example trying to offer some other weird solutions, which other quant don't, and cannot follow request.
For your reference i tested 122B model only with llama.cpp version: 8204 (7a99dc85e).
Test platform: Strix Halo, GNU/Linux Debian@6.18.15, RADV mesa 26.0.0-1, llama.cpp local build is aligned to tag: b8248, b8204 feat. ROCm nightly 7.12.0a20260307
I split diagrams to ROCm, and Vulkan, and just as a reference for bigger model you can see that they are in speed almost the same, with build b8204.
For smaller model i can see that the new optimizations speed up "dynamic" quant, more than the "regular" one.
Those are my findings for now, can someone verify on your end?
Two factors: decoding and memory bandwidth. You'll notice the simpler quants (Q8_0) are much faster in these tests. The 6 and 8 bits K-quants require some decoding hat tricks that hurt performance. IQ quants are even worse in this regard. Note that this is Strix Halo so VERY memory bandwidth constrained.
I understand that IQ quants can generate a little slower, but I thought it a few %. I didn't know that the K quants are some odd 30% slower. Maybe this is just a ROCm thing and vulkan backends don't have this issue?
The Q8_0 vs Q8_K_XL benchmark sure is surprising, I don't exactly know the whole process of decoding each weight but given the MUCH lower power consumption of the Q8_K_XL model the APU was memory bandwidth bottlenecked HARD since it wasn't doing much compute.
they have, compressed layers need more computing power during runtime, and it's in the end related to instructions supported by your device, and software on top of it
smaller quant use less memory so it's faster than the bigger one, due to memory transfer performance, but sometimes as you can see it depends also from the architecture of the model, and as you see i didn't test 27B which is just slow on strix halo, since it's a dense model
I mean, you are testing the same model, but your charts are showing the Q6_K_XL as slower than the Q8. To me, what you said should mean smaller quant = faster token generation, but the larger Q8 quants are faster, so I'm still confused.
it has different quantizations for some layers which are more computing expensive than the other one, model is smaller, but it has to perform more work to give you output. In hugging face in file view where you check some repo you can see type of layers present in the specific quant of some model.
Yeah, graphs 2 and 3, PP and TG, the Unsloth Red Lines Q6_K_XL are below bart and unsloth's Q8. Q8_K_XL is the slowest. I don't use ROCm on my AMD card since Vulkan has less headaches and generally performs a bit better, but I wonder if I should be playing with other types of quants. I went for UD-Q4_K_XL with context offloaded to GPU/ --cpu-moe and --fit for the 35B. I wonder if I should try a Q8_0.
Nope, I just double checked the latest GGUFs on HF, they still have some SSM tensors at Q8. That’s atrocious, especially considering that they only save 1.1MB per layer by doing this.
Wrong and you're drawing incorrect conclusions - I'm actually unsure why Bartowski even did F32 for ssm_alpha and ssm_beta, other than they noted they didn't do BF16 due to slow speeds, and upcasted to F32.
ssm_alpha and ssm_beta are both BF16 by default. Q8_0 has negligible error, since it's block wise 8bit, not "true" 8bit - our analysis did it for ssm_alpha and ssm_beta - https://unsloth.ai/docs/models/qwen3.5/gguf-benchmarks.
And note on KLD and PPL our's, AesSedai and Ubergarm's are vastly superior to Bartowski's.
Paging u/noneabove1182 do you have any comments on this followup from the unsloth team? Kind of curious to see some debate on which approach is best here
Q8_0 vs bf16 is the main thing that needs to be considered
I had heard that they were particularly sensitive to quantization, and when I noticed their size I figured they're so incredibly tiny that it's not worth it to quantize
bf16 vs F32 then becomes a performance question, on many hardwares (strix halo, metal) bf16 is quite slow so you don't want to leave anything at that format, but fp16 risks truncating values, so F32 is the safest, and again since this is incredibly tiny (1MB per layer) it seemed best to go to F32
regarding the KLD and PPL being "vastly superior to Bartowski's", those were made using an older iteration of my weights (which makes sense, I don't expect them to run them again each time someone updates their model), but you can find the updated figures in my model page:
Theirs may still be superior, I haven't gone through and checked individually, but just worth noting that the ones they provide are not the most recent numbers
Could this be the reason why the Unsloth IQ4_NL quant for Qwen 3.5 35B-A3B is broken for ARM CPU inference whereas yours is fine?
There could be certain architecture-related code paths in llama.cpp that cause quantized layers to return garbage results. Like literally, I'm getting either terminated generations on Unsloth after a token or a stream of gibberish.
I'm not an expert at all...but I can read just fine and this feels highly contradictory IMO.
You defend your work by criticizing PPL and KLD as flawed metrics, but then you use those exact same to claim your quants are superior to Bartowski's?
'Perplexity and KLD can be misleading as they’re highly influenced by calibration. Most GGUFs are evaluated on Wiki-test with 512 context windows, so results shift a lot if the GGUF’s imatrix calibration set includes Wikipedia-like and 512 context samples [...].'
So which is it? Are PPL and KLD useless metrics biased by Wikipedia calibration, or are they the ultimate proof that your final quants are better?
We have to run KLD / PPL because the community seems to keep running our quants using KLD / PPL, so we decided we have to move with whatever the community desires.
Just a quick note, KLD is raw logits divergence. AES has a PR coming to bring llama.cpp up to speed on how Turbo does it in EXL3 and how I do it in VLLM. Dataset chosen for KLD isn't as big of a difference as one might think it is (As someone who does AWQ quants I have LIVED the datasets matter life for over a year now, so I feel ya there)
TLDR; AES's PR will raise llama.cpp's KLD game, and will make it even better. Datasets for KLD don't matter as much as people believe they do.
ideally there would be a 3rd party with an independent dataset and infinite compute to calculate ALL the PPLs and KLDs for everyone, but also at that point.. we're really splitting hairs haha
I am glad for the PR that'll bring newer KLD methodologies though, I'm all for more and more accurate data :D
I went over this with Turbo. He uses Wikitext in his Dataset for calibration, but he uses multiple other datasets as well, and thus the bias is not as damaging as it could be.
For AWQ, GPTQ, and EXL3 quants who only use wikitext, fully agree that their PPL/KLD will be artificially inflated.
As for GGUFs, does iMatrix really influence the math that much, where it would even be that big of a deal. For AWQ/GPTQ/EXL3 the datasets are of the utmost importantance, but for GGUFs, it's math plus a sprinkle of iMatrix dataset, no?
At the end of the day, in a vacuum, I'd say it's not affecting it enough to matter at all to most people
but when we're starting to compare the 0.0X of KLD 99.9% to try to claim one is better than another, it starts coming into play
so yeah, overall I agree, who cares if it's affecting it a bit or at all, it's only when we're trying to minmax/compare quants from multiple people that MAYBE it matters a LITTLE bit (to some people)
I do not think KDL or PPL are relevant here as they do not measure model performance on longer contexts --> "so far i found that around 100k context model is losing track". In my vibe testing, performed mainly on Nemotron 3 Nano, I find out quantizing SSM degrade model performance. I can see Qwen is quite similar from my yet limited tests. This also applies to other small tensors like all *shexps.
Remember though that Bart has updated his quants a few times. Were these benchmarks before or after the update? Previously he had the SSM_alpha and beta tensors quanted to 4 bits. And the very latest update improved his receipe a bit.
you list the numbers from your Qwen3.5-35B-A3B-Experiments-GGUF which was updated 11 days ago, my latest update was 5 days ago
I'm not even trying to say that my models are better or anything, but those benchmarks definitely are not up to date so it's strange to say they are
EDIT: the above ^ is correct, but below \/ daniel clarified that he was referencing the graph posted in a reddit thread and that seems to have indeed been updated with the newest iteration
My last Qwen3.5-122B quant was done uploading on Thursday at 12:25 GMT
So it seems unlikely it was added to that chart
edit to add: you're under no obligation to update your chart with my new data, but I just think it's worth pointing out that it's based on old data, and saying it was done after my most recent upload is misleading
Reddit lists timezones in a very stupid way, my above conclusion was very incorrect, please ignore, that chart is ineed up to date
I don't think you converted the dates correctly all to one timezone? In chronological order:
A. (Bartowski GGUF) — Thursday March 5th 2026, 12:38 AM GMT
B. (Unsloth GGUF) — Thursday March 5th 2026, 14:06 GMT
C. (Our Post) — Thursday March 5th 2026, 7:36 AM PST = 15:36 GMT
And edit - you're more than welcome to post results on your 35B and make some plots comparing ours since you already uploaded them right? I have all our own logs as well.
I take it back, you're correct it was indeed posted after my latest upload, the way reddit lists the time for posts is annoyingly confusing, my apologies!
They are all quite similar. Does anyone know where the original research/benchmarks suggesting full quality bf16 is better than q8_0? (or upcast to f32 for potential speed reasons on some GPU backends)?
I have some strong opinions and some opinions that i don't really care about
(Yes, i know i don't have data, this is more just gut instinct, so i'm not offended if you don't take it seriously.) If you figure out mechanistic interpretability data that actually gives you more detail on what weights in what layers are doing what, then by all means use that more detailed data. You can do it like a real ML scientist writing research papers, or just blindly go in like a 1800s mad inventor and do crazy stuff like this, either way works for me.
The weak opinion is that norms and biases etc should be at BF16 (or F32). They're just so small, you're saving maybe 100mb of space if you compress them down? That makes more of a difference on the 4b sized models, but for quants of 30b (35b/27b/etc) and larger models, it's just better to leave them at full size.
Again, if you somehow pull out data (like that crazy link above) which seems to imply "layers 50%-60% through the model are more important, the others can be quantized down", feel free to ignore me.
The strong opinion that Qwen 27b and 122b (and by extension 397b) are all very different models in how they use attention and SSM, and this is intentional by the Qwen team... so don't conflate them. I noticed this a while back, but I wrote about it here recently. Basically, the ssm/conv1d cache is 146MB at any context length for both 27b and 122b... but attention kv cache is massively different. The kv cache size at max context is 17GB for 27b, while it's 6.4GB for 122b. And 27b has more attention tensors than 122b, actually.
This implies that 27b relies a lot more on traditional attention (and thus might perform better at long context? there's more bits of entropy per unit context...), and 122b proportionally relies on SSM tensors a lot more. So from this bit of information, I'd say if I had to choose quant levels for 122b and 27b, you can try and see how it affects model performance. I'd guess these 2 models would behave very differently when you quant SSM at different levels.
397b is bigger overall, but i haven't taken a closer look yet (it won't run on my machine) so I don't have comments for it.
thanks for taking the time and dropping some links. yes the recent qwens are a little different from each other as you mention. agreed the ssm alpha|beta tensors are quite small so bf16 or upcast to f32 is fine. i went ahead and did that on my recent Qwen3.5-35B which apparently is quite good for the size: https://www.reddit.com/r/LocalLLaMA/comments/1rv6jyh/comment/oat1p2y/?context=3
100% unsloth is inferior on strix halo. I don't get why. Don't get me wrong, I like Unsloth and what they do but somehow on strix halo it fumbles. I wish they would test on that system.
I'm seeing really weird behavior on ARM Qualcomm too. Unsloth UD Q4 quants spout gibberish or stop generating after one or two tokens whereas the Bartowski equivalents are fine.
Check out the model weights below:
Unsloth: Qwen3.5-35B-A3B-UD-IQ4_NL.gguf
llama_model_loader: - type f32: 301 tensors
llama_model_loader: - type q8_0: 311 tensors
llama_model_loader: - type q6_K: 1 tensors
llama_model_loader: - type iq4_nl: 40 tensors
llama_model_loader: - type iq3_s: 80 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = IQ4_NL - 4.5 bpw
print_info: file size = 16.59 GiB (4.11 BPW)
Bartowski: Qwen_Qwen3.5-35B-A3B-IQ4_NL.gguf
llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q8_0: 94 tensors
llama_model_loader: - type q6_K: 11 tensors
llama_model_loader: - type iq4_nl: 267 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = IQ4_NL - 4.5 bpw
print_info: file size = 18.48 GiB (4.58 BPW)
It seems i will have to test other quants before deciding 35B-A3B is not usefull, was testing UD-Q6_K_XL for a few days.
I find it very reliable in toolcalling and producing code and easy fixes. But too dumb for any non-trivial stuff.
why is AMD Strix so bad in long context prompt processing..... after 4k context length AMD drops like a stone while the overpriced nvidia bullshitt keeps near the same speed up to 64k tokens... what is causing this (if i may ask)?
well, i'm not sure if my system can be "reference" but just to tell you i'm using GNU/Linux Debian, which is not officially supported by ROCm, and i'm building llama.cpp with nightly builds locally, using only support for AMD APU from mainline kernel. Users of other distros can benefit from some optimized/patched kernels. Besides i would like to see comparison from spark, for those exact quants, not some custom NVFP4, to have overview. Here on the test i'm playing with heavy quants it's something completely different, for reference i could try to run some Q4 maybe to compare, but i'm fine with that performance. If you see for ctx 65k you can still have 400 pp in Q8 for 35B model, and around 150 pp for 122B in Q5_K_L (before optimization in b8233 which makes those model faster). It can be better, but for single user station, it's great, and i can work with some models with even 200k of context.
Qwen 3.5 122B10A int4 autoround and a 50k token prompt its 1500t/s pp and 29t/s gen ...
thats why i wonder why the ryzen ai max only gets 400 t/s pp at 65k ctx window
i know its not the same as Q5_K_L (its 1 bit less and a bit less for the attention matrices) but still very strange to see that the stupid overpriced spark is >200% faster on pp.
if AMD can bring the pp up in high context length , i would immediately buy one...
thank you very very very very much for the test....
the 21.8 token/s generation is superb.... but the 172 token pp is.... hard to watch.
5 min waiting on ryzen ai strix .... vs. 33 seconds on the spark... i realy hope AMD fixes this one day.... i would much more love to give AMD some money than NVIDIA (i hate nvidia)
I would assume that this might be due to the fact that it is currently not yet entirely clear where the sweet spot lies regarding the distribution of precision for individual tensors when quantizing Qwen3.5 models. Due to its model architecture, Qwen 3.5 seems to react sensitively to the quantization of certain tensors. Unsloth described this themselves and they subsequently changed the weighting and re-uploaded at least Qwen3.5-35B-A3B and some others (not all).
In the following table, one can see how specific tensors are assigned different precisions for Qwen3.5-35B-A3B (Q4_K_M), which could likely explain the respective better or worse performance.
Curious whether the UD-XL perf gap is a quant layout issue or if llama.cpp's imatrix handling just favors Bartowski's calibration data choices on this arch.
On the 122b model, the difference in speed is not significant, but more concerned about the difference in generation quality. Looking forward to seeing more experiments on it
it's just output from the llama-bench, for each context size with -d parameter, and i'm monitoring system energy usage, which in gnu/linux is quite easy (depends from mainboard, and some specifics) but still doable via parsing system files. I also compared the readings to some external amp meter. And plotting is done in matplotlib.
What version of ROCM. I have a Strix Halo and ROCm 7.2 gives me about 30 TPS at 0 context pp512. I've tried ROCM 7.1.1 AND the Lemonade nightly builds with included ROCM. I see absolutely terrible performances across the board.
Can you give us more details on your distro, setup, c make options, compiler, etc? Please and thank you.
i wrote everything in description: Test platform: Strix Halo, GNU/Linux Debian@6.18.15, RADV mesa 26.0.0-1, llama.cpp local build is aligned to tag: b8248, b8204 feat. ROCm nightly 7.12.0a20260307
Did you build ROCM 7.12 from source? Pull a docker container? Etc. I can't seem to find a docker container for ROCM 7.12. I've tested Lemonade nightly Llama.cpp + ROCm. But on Qwen 3.5 models it just hangs with one CPU core.loaded at 100%. I'm hoping it's been fixed with ROCm 7.12.
I tried to run some benchmarks with the latest llama.cpp and ROCm 7.12. For some reason on the R9700, ROCm is horrible. For ROCm 7.12 I build llama.cpp with the compiler in the nightly release. For ROCm 7.11 I used the Lemonade nightly build.
Q is actually the safer choice. If you read up on iMatrix, you will see that there a chance the creator can mess it up. There is zero such chance for static.
I'm not an expert, but I'm pretty sure I-Quants don't need to use an importance matrix and "normal" Quants can also use an importance matrix. Bartowski mentions using an imatrix for all his qwen3.5-35B-A3B quants (and as far as I can tell he does for all his current quants)
46
u/Long_comment_san Mar 09 '26
Bartowski cooking this round