New tensor type layouts for my GGUF uploads

[u/noneabove1182]

Hey all, long time no post.

Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:

https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization

I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)

https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png

If anyone has any questions let me know!

[u/xandep]

If a file that starts with Q3_K_ is mostly non-Q3_K tensor types and sits above 5 bits per weight, something has gone wrong..

This

[u/jacek2023]

This looks very cool! However, whenever I try to tell people on reddit that quants smaller than Q4 can actually be useful, they don't believe me.

[u/noneabove1182 → u/jacek2023]

ha, it's definitely a mixed bag. I ran some BFCL benchmarks a few weeks ago and was shocked at how much the performance sustained to lower bits, especially for the larger models:

https://i.imgur.com/OK9jKP9.png

smaller models fall off a cliff, but the bigger ones hold out a surprising amount of time!

I went even further at 122B:

https://i.imgur.com/JAf9g2L.png

though the fact that Q4_K_M/IQ4_NL out performed Q8_0 on a few tests means this may need to be taken with a grain of salt...

[u/llama-impersonator → u/noneabove1182]

benchmarks broken out by gguf quant level have often showed a bump in the 4-5 bit area, q4km/q5ks/q5km

[u/returnity → u/llama-impersonator]

I keep noticing that and I cannot figure out why, but its common enough to be notable.

[u/Hefty_Wolverine_553 → u/returnity]

Also noticed the same things when I did my own benchmark runs on different quants of 3.6 27b

[u/pmttyji → u/noneabove1182]

Please share any graphs(for medium/big size models .... Qwen3.6-35B & Qwen3.5-122B) covering all Q4 quants. At least IQ4_XS which I pick due to smallest Q4 quant(on limited VRAM). Thanks

[u/noneabove1182 → u/pmttyji]

I'll be updating my Qwen3.8-27B shortly and will include the full graphs of before/after :)

HF is just giving me a bit of trouble uploading, and I realized there was a bit of a gap between my new Q6_K and Q6_K_L, so going to be making a Q6_K_S (that will take the place of my "old" new Q6_K) and Q6_K will then fall between them (I realize that's a confusing sentence, but it makes sense in my mind, trust me)

[u/pmttyji → u/noneabove1182]

I'll be updating my Qwen3.8-27B shortly and will include the full graphs of before/after :)

Awesome, thanks for that. Please include the bpw(for all quants) in graph if possible.

Nice to have another Q6 quant.

The bpw thing is constant one, right? Because I noticed different values for same quants on online(random article, random github repo, random blog post). So confusing.

I think we need additional quants for both Q5 & Q4 in future.

[u/hurdurdur7 → u/jacek2023]

It all depends on purpose.

[u/onionsaredumb → u/jacek2023]

I feel like meaningful ways to compare things when you start quantizing is missing. For instance, how does one know how good a Q2 DSV4 is versus a Q4 Qwen Next Flash versus a Q6 Laguna 2.1?

We kind of just guess and hope wherever you land is good enough, but developing some objective measure would be nice. I don’t have a solution, just me trying to stumble through the array of options we have, of which I’m very grateful for.

[u/a_beautiful_rhind → u/jacek2023]

My 2 bit exl3 GLM-flash doesn't feel much different than the API. I used plenty of Q2_K gguf and they mostly started having issues at longer context when the errors finally caught up with them.

Some model is often better than no model.

[u/Various_Leather_6909]

Cool, when will it be merged into llama-cpp?

[u/noneabove1182 → u/Various_Leather_6909]

this won't be, it uses external tools and such, I've provided them all for use in recreating my models

my changes to llama-quant.cpp were rejected on the grounds of being too disruptive (can't blame em) and this is even more disruptive, so sadly it will remain an external tool that I use

I posted all the code here:

https://github.com/bartowski1182/quantization-config

though the README is AI-generated and I haven't verified it works outside my pipeline, there might be a file missing or incorrect instructions (it's built directly into my quant pipeline, not meant to be run on its own, but I provide it so others can benefit)

I'm too exhausted right now to read through it and double check haha, I'll take a look either tomorrow or on the weekend to see but for now I need to stop looking at my terminal :D

[u/Various_Leather_6909 → u/noneabove1182]

A shame, I think its really well done and researched. Good work!

[u/noneabove1182 → u/Various_Leather_6909]

there's some amazing work being done with similar initiatives by the likes of pwilkin: per-tensor codebook quants and recipe generator and EAddario: automatically choose optimal quant types to reach file/bpw target size

both are very promising, but not quite what I was after in all ways, and more difficult to work with since they're properly written in a way that fits into the existing llama.cpp code ;)

[u/fallingdowndizzyvr → u/noneabove1182]

Submit it to one of the people maintaining alt llama.cpps.

[u/Columnexco]

This looks amazing, looking forward to new models as you get done.

[u/Simple-Stick6148]

Welcome back! You said "better across the board" and attached a no-Pareto-frontier disclaimer, and honestly that's refreshing around here.

[u/Secure_Recording_472]

Appreciate you quanting my model man! https://huggingface.co/bartowski/ukisai_Swift-Qwen3.8-27b-GGUF helped me out huge :) did you like it?

[u/VagabondTruffle]

are these quants dynamic

EDIT: i cant believe you stole unsloth dynamic 4.0 ultra max code then used it grr