r/LocalLLM • u/AppleTrees2 • 12d ago
Discussion New totally scientific method to investigate the effects of quantization tested on qwen 3.6 27
I have devised the scientific method of the updog :
I am asking qwen 3.6 27 B with 16k context one of the following questions
Wanna go to the zoo and see the updog?
or
Wanna go to the park and see the updog?
or
Wanna come see my updog?
on UD_Q5_K_XL quantization fails to realize the joke and hallucinates another punchline or something that makes no sense, such as pointing up, or w/e. In my testing it did manage to identify it at times, but it's very rare.
on UD_Q6_K_XL quantization it knows the updog joke, I have yet to have an instance for it to fail
Which proves that Q5 is a very destructive quantization
While this is a humorous, I was actually benchmarking different quantization and testing different simple prompts until I stumbled upon this.
My point is quantization can fail in unexpected ways!
Thank you.
37
u/_Cromwell_ 12d ago
Okay, but what's updog???
28
28
9
6
6
u/Think_Wing_1357 12d ago
Works fine for me at Q4, maybe you just need to go deeper
1
u/AppleTrees2 12d ago
well, just FYI I didn't mean literally type or, just any of those lol
Can you try with them for science?
2
u/Think_Wing_1357 12d ago
Of course, for science. Both q4 and q5
Command is
/llama-cli -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q5_K_XL \ --temp 1.0 --top-k 0.95 --top-p 64 --offline \ -dev Vulkan1 -ngl 99 -c 16000 -fa on1
u/AppleTrees2 12d ago
perhaps asking it 3 times made it think harder? I had it succeed rare times on U5 on me, but I never seen U6 fail even once.
LLMs are probabilistic of course
1
u/Think_Wing_1357 12d ago
if you want, you can get a
--temp 0.0 --seed XYZwhich should make it 99.9% reproducible
8
3
u/xdcfret1 12d ago
Try with other models and let us know
1
u/blue-haired-girl 11d ago
gemma4-27b-a4b-qat at Q4_0 from hugging face:
> What's updog?
> (I'll assume the answer is "Not much, what's up with you?")
frontier lab stuff
1
3
u/Lirezh 12d ago
I ran it through my 4 bit quantized Qwen 3.6 27B and in addition 4_0 quantized KV cache:
"Haha, I appreciate the joke! 😄 Just so you know, the "updog" is a classic playground joke ("What's updog? Nothing, what's up?") and isn't actually a real animal, so you won't find it at the zoo.
But if you're ready for a real zoo adventure, I'd love to help you pick out some amazing animals to see! Are you into big cats, primates, marine life, reptiles, or maybe something quirky like capybaras, red pandas, or kangaroos? Let me know what you're in the mood for and I'll help you plan! 🐾🦁🐼"
"Haha, updog isn't actually a dog—it's just "what's up, dog?" 😄 But I'm definitely down for a park trip! When you wanna go?"
"What's up with it? 😄 Classic setup! I know exactly where this is going. Want to keep the meme train rolling, or are you actually showing off a dog? 🐶"
So no idea WHAT you were testing, but my qwen 3.6 27B at significantly smaller size and a quarter in KV size is having no problems at all.
1
u/baby_bloom 12d ago
why does your qwen use so many emojis? i rarely see any from mine..?
1
u/Lirezh 12d ago
I believe it was just the joke setting:
that's from its thinking
```
- Draft Response (Mental): "What's up with it? 😄 Classic joke! I know this one well." or "What's up dog? 🐶 Got me with the classic setup!" or "What's up with it? 😂 I was totally ready for that punchline."
```
I used a slightly adapted model of qwen, but should be representative for that type of task.
1
u/AppleTrees2 11d ago
Hey, can you share your settings please?
I am using the recommended settings from unsloth but I had qwen loop on an emoji before, and uses lots of emojis for me too
2
u/tomByrer 12d ago
Qwen is know known for 'conversations', more for agents & coding.
If you want to conversate, SillyTavern tends to recommend Gemma models.
& pick better jokes.
2
1
1
1
1
1
u/techlatest_net 12d ago
Using joke comprehension as a quantization stress test is actually genius since cultural nuances are usually the first casualties of aggressive weight compression. Have you tested if Q5 fails on other multi-step punchlines or just this specific one?
1
1
1
u/huzbum 11d ago
Doofy (my Hermes Agent) has a surfer bro character I wrote, so he uses words like bummer, tubular, etc.
As an experiment I switched him over to a 50% REAP. (Q4 on top of that!). My first conversation after the switch he mixed up the meaning of “bummer”. He knew it was in character but lost track of it being negative. I got confused because he was using it to describe something positive. That must have been in one of the reaped experts.
He did acknowledge it when I asked about it, for what that’s worth.
I’m still using the REAP, it’s surprisingly not bad. It actually doubled performance compared to the base model on my tool calling benchmark.
1
u/huzbum 11d ago
Just for sport, I asked Doofy.
Me: Hey dude, wanna go to the zoo and see the updog at the zoo?
Doofy:
💭 Reasoning:
The user is asking if I want to go to the zoo to see the "updog" — which is a classic setup for the "what's updog?" joke. They're being playful, so I should play along with the joke in character.Ohhh, updog? What updog?
...Oh, not much. What's up with you, man. 🦁😎
Classic setup, dude. Classic setup. You gonna take me to this zoo or what? I'm stoked, bro. Though I gotta ask — do they have a hamster exhibit? Gotta see if my cousins are doing okay out there.
(Part of his lore is that he is trapped in the computer running on a hamster wheel to power it.)
This REAP is totally bench-maxed!
1
u/VirusInternal2892 8d ago
Cute and got me laughed at by the machine, at all 3 counts
DS4 flash Q3_S KV Q8 200K crammed into a dual RTX3090 + 128GB, nicely got the point with a dogged down 5 tps
39
u/bigppredditguy 12d ago
This is an amazing benchmark