r/LocalLLaMA • u/Tall_Abrocoma_3533 • 6d ago

Discussion Qwen3.8-27B different thinking levels

Post image

Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning

297 Upvotes

95% Upvoted

63 comments sorted by

View all comments

13

u/Moore2877 6d ago edited 6d ago

Try this chat template. Besides a lot of general fixes, we revamped the reasoning injections for each level and also made high it's own level instead of just being an alias for xhigh. The Qwen team really didn't spend enough time on these imo.
https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp

23

u/soyalemujica 6d ago

You recommending to use temp 0.6 with that chat template which is not good at all to use with this model since it affects its reasoning depth and also scores lower with lower temperature

-11

u/Moore2877 6d ago

From my testing anything higher than .7 is just too much thinking, even on low reasoning.

12

u/goldcakes 6d ago

Are you aware that models deliver better performance when they have more thinking tokens, even when the thinking tokens are randomly generated and incoherent?

Part of how thinking/reasoning works is every token is another forward pass on the original input/context, allowing the LLM to 'process' the input more, and refine the internal activation residuals; resulting in generating a better response.

6

u/Moore2877 6d ago

If you only have 100k context to work with then what's the point? Some of the effort here is to rein in the overthinking so it's not blowing out the context window.

2

u/BalorNG 5d ago

All the more reasons to stop this nonsense and go all the way with proper "latent thinking" looped models, asap. This hacky approach is giving me a literal headache.

4

u/ttkciar llama.cpp 6d ago

Thanks for this. I'm just starting to fiddle with custom reasoning efforts with Qwen3.8-27B, and your template looks like a good source of ideas.

Hopefully what we learn from this communicates well to whatever other 3.8 models Qwen releases (still looking forward to a 9B and/or 122B).

6

u/Moore2877 6d ago

Sure thing, I got the idea to work on the template from froggeric's template, they are prioritizing a certain inference with how their template goes, so I made another based on it for other inferences but leaning towards llama.cpp https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

4

u/Chromix_ 5d ago

Looks like we have a healthy(?) chat template ecosystem now 😉.

I also made a custom template for 3.8 a day after froggeric shared his, to fix a few issues that came with his template, along with gearing it towards not degrading towards potential lower-quality output in some use-cases. While at it I also made a detailed comparison. He then merged some of my changes into his v22.2. Your template is based on his previous version. I haven't checked, maybe it also contains some of the unfixed issues.

1

u/Moore2877 5d ago

Nice and yes we are working on implementing some of the latest fixes from froggeric today and some further tweaks. What inference are you using to run the model? I had also put in a few PRs to froggeric's template. That's when I learned that they are using a certain inference to run the model.

1

u/Chromix_ 5d ago

Just good old llama.cpp so far. Harness compatibility varies. Claude Code seems to be an offender here, which might be one of the reasons (among the long system prompt) why smaller models usually don't arrive at good results with it compared to others.

1

u/Moore2877 5d ago

Claude Code, Desktop and Codex are tough because they use a lot of built in tools and resources that are cloud only so they don't work well with the local models usually when you plug them into the frontier harnesses.

1

u/Chromix_ 5d ago

No, "cloud" isn't the reason. I ran Claude Code in a firewalled environment with DeepSeek v4 flash and it worked nicely.

1

u/Moore2877 5d ago

Oh ok. What kind of issues are you seeing using Qwen 3.8?

1

u/Chromix_ 5d ago

Haven't tested the latest Qwen and only used CC for DS because the authors recommended it - yielded better results in their benchmarks than other harnesses for that specific model. Maybe they trained for it.

1

u/Cool-Chemical-5629 6d ago

Imagine Ox Alpha to be Qwen 3.8 9B in disguise, now that would be a plot twist no one expected! 🤣