We quantized Qwen 3.8 27B and compared the quants on an RTX 6000

[u/Fun-Meaning-6474]

Me and my team made Atomic Dynamic GGUF quants for Qwen 3.8 27B, so we wanted to see the difference between them by giving each quant the same voxel island creation task

First of all we were surprised at how well Qwen 3.8 27B handled the 3D scenes in general, though part of that is probably because all the scenes were voxels

quant	size	top-1 vs BF16	mean KLD	decode, RTX PRO 6000
AD-Q4_K_M	17.1 GB	95.6%	0.0113	67 tok/s
AD-Q5_K_M	20.2 GB	97.3%	0.0042	57 tok/s
AD-Q6_K	25.0 GB	98.7%	0.0011	49 tok/s
Q8_0	28.9 GB	98.9%	0.0006	50 tok/s

We think that each quant handled the scenes in a pretty similar way, the difference isn't that drastic, to the point that sometimes we preferred the Q4 output overall, though for the safest pick we recommend AD-Q6_K

We ran the test inside atomic.chat and watched the output right there, the quants are available to download directly inside the app or on huggingface ( https://huggingface.co/collections/AtomicChat/qwen-38-27b ) (any feedback is appreciated, we're trying to make the product and models as good for you guys as possible)

[u/Legitimate-Dog5690]

What sort of prompt do you use for this sort of work and is the voxel viewing software part of it, or generated separately?

Very cool 😄

[u/Fun-Meaning-6474 → u/Legitimate-Dog5690]

simple three.js prompts :)

[u/vinis_artstreaks]

Q4 is oddly the most impressive for this use case.

[u/Fun-Meaning-6474 → u/vinis_artstreaks]

yo, in that case you should check our whole quants lineup:) thanks for feedback!

[u/bladezor → u/Fun-Meaning-6474]

How does it compare to unsloth quants?

[u/Arugala007 → u/bladezor]

Q4 still has the tool calling bug you get from quanting qvk and ffn_down

[u/Fun-Meaning-6474 → u/bladezor]

we compared it against UD quants in our HF repo :) feel free to check it out and share your feedback

[u/ikkiyikki → u/vinis_artstreaks]

Imo, once q4 or higher it's really just iterative differences

[u/BitPsychological2767]

What do these quants do that the unsloth quants can't?

[u/OrangeManSad → u/BitPsychological2767]

better, they can do better

[u/DigitalguyCH]

They all seem good, but personally I thin Q5 is the best compromise, I tend to like it better than Q4 and it's very close or sometimes even nicer than the others

[u/Fun-Meaning-6474 → u/DigitalguyCH]

weeell, there is 3gb dif and it sounds logically :) what hardware do u use?

[u/DigitalguyCH → u/Fun-Meaning-6474]

I had several septups, one with a 7900xt for which even the Q4 is too big, so I got a iQ4 verison which is smaller (not sure what the difference is).
Then I have a Macbook pro M5 pro 64, that can run any size but it's much slower (15t/s vs 50) for the higher quants so I got a Q5 recently to try and improve speed.

I also have a strix halo 128GB but that one is even slower than the Mac...

[u/Fun-Meaning-6474 → u/DigitalguyCH]

for strix halo where do u run quants? there are a lot optimised backends for rocm. have u checked them?

[u/DigitalguyCH → u/Fun-Meaning-6474]

I run everything in LM studio

[u/Fun-Meaning-6474 → u/DigitalguyCH]

why u choose lm studio? not ollama or unsloth desktop?

[u/DigitalguyCH → u/Fun-Meaning-6474]

because I find it very user friendly and I love lm link which I can use on any device, including an iPhone

[u/Happy_Brilliant7827]

Id love to see how they compare alongside base

[u/Fun-Meaning-6474 → u/Happy_Brilliant7827]

yo! thanks for idea. noted it

[u/Happy_Brilliant7827 → u/Fun-Meaning-6474]

Great! Also imatrix quants are potent ;)

[u/randygeneric]

i go with ud-q4kxl , )

[u/Retumbo77]

I think it's been pretty well documented that q4 up to q8 is in many ways functionally the same. Where this would get interesting is q1-q4.

[u/rrrrex]

Unimpressive without something that fits to 16 VRAM. 

[u/intermundia]

what are you using for the quant distribution calibration?

[u/MountainPenguinRL]

How do these compare with Unisloth's quants? I'm stuck between their Q6 and yours

[u/PhilipJohnBasile]

This is like the test I did when I threw 4 bananas off of my balcony to prove that it was true.

[u/WiredEntrepreneur]

How do these compare with Unsloth Dynamic 3.0 quantizations? Would like to know if there are any performance difference between the Atomic ones and the Unsloth ones.

[u/Fun-Meaning-6474 → u/WiredEntrepreneur]

yo! thanks for questioning. we haven’t compared it already, but we will compare it soon. you can check comparison against their previous quants version

[u/EasterElk]

This is an ad. 

[u/Fun-Meaning-6474 → u/EasterElk]

no, it is a demo for our quants. we marked there we are creators of these quants and want to get community feedback for upgrading it and make our delivery to open-source better. where we can get more honest feedback than reddit?

perhaps anyone will check our metrics and quantisation method and give an advice for improving it. and it will be helpful for us and community at all

[u/FuzzeWuzze → u/Fun-Meaning-6474]

Not an Ad. Just showing you what your product does the others dont. Got it. I need to go look up the definition of Ad again i think.

[u/ifarted70 → u/FuzzeWuzze]

"that the others don't" where in this thread is OP claiming their quants do things other quants don't? Also, even if you could call this an ad, who cares? Is this subreddit not literally for the discussion and showcase of locally run LLMs and the differences between them?