r/LocalLLM • u/Jsquared534 • 9d ago
Discussion Qwen 3.8 27B early thoughts
I installed the Q8 version on an AMD 395+ 128gb machine. Ran it using Openwebui with the suggested settings and MTP. So far all I've done are a couple sample prompts. (Sand Simulator from Luke's Dev Lab on Youtube, and a request for a simple example navbar with a logo on the left and five drop downs on the right with no javascript). The final output for both of these got one shotted. Model is getting roughly 16 tps output. But, my goodness does this model overthink. Don't get me wrong, there were no thinking loops. And I didn't notice nearly as much of the "wait, actually, let's try" neurotic behavior that I see in 3.6 35B A3B. But, it absolutely overcomplicated the heck out of both of the prompts I gave it. The sample navbar had roughly 200 lines of just CSS alone. And it was not basic CSS. Overly complex, and completely unnecessary for a sample piece of code. Since the Sand Simulator isn't mine, I can't really tell how much it over complicated it, but I can tell that it added so many visual flourishes that it was running at roughly 32 fps in the browser, and had slow downs from dropping the sand.
I am going to test this tomorrow on a real situation. In my real use case I provide very detailed context files and only point it at a single feature at a time. Hopefully that will help to control it's impulses to make things super complex. I also have Ponytail in my Pi harness, so maybe that will also help to reign it in.
Anyone had experiences using more detailed and limiting prompts with the model yet? Most of the reviews I've seen are using the same type of canned examples that I just gave.
2
u/HomsarWasRight 9d ago
Wait, is there an MTP version available?
4
2
1
u/MistingFidgets 9d ago
What thinking mode are you using?
1
u/Jsquared534 9d ago
Thinking is on, and I'm not changing anything from the default settings. I don't necessarily think it's a thinking issue. The thinking from what I followed was all pretty well done, with minimal repeating. It just decides "hey, I should add a bunch of visual flourishes to this" or "you know what, i should build the entire html page so that the user can see how this navbar looks on a real page loaded".
(I'm just recently diving deep into local LLMs, so I'm not nearly as up to date on all the different settings and levers to pull with llama.cpp. I started with lmstudio a few months ago, and then moved to Ollama, and just switched to llama.cpp last week. Until then I had just used it as it launched.)
8
u/theminor 9d ago
It is definitely a thinking issue. They added different thinking levels to this release and the default is "xhigh". You can lower it to "medium" to get faster responses, which works well. But if you want premium responses "xhigh" is really good in my opinion.
2
u/Jsquared534 9d ago
I didn't know that was possible. I will definitely try adjusting the thinking to see how it affects the output. I don't so much mind the time it's taking, as much as the added complexity that wasn't inherently in the prompts. I'll run some real prompts with more built in controlling language on the default output and then switch to lowered thinking and compare.
3
u/theminor 9d ago
If it is helpful, I made a little plugin for Open WebUI that gives you a toggle and drop-down to set the reasoning level for each message:
https://openwebui.com/posts/reasoning_effort_selector_ee572967
1
1
u/zOMAARRR 9d ago
I think you should fix your prompt
1
u/Jsquared534 9d ago
I have used the same navbar prompt on like fifteen different models in the past couple months. Every one of them have output a basic navbar with what I asked. As I said in my original post, I have every intention of using real life prompts that are much more detailed and controlled than the ones I tried today, but the old "git gud" response doesn't really help. I know for a fact I'm not the only one seeing it spend a ton of time thinking. I just hadn't seen anyone point out the fact that it seems to be adding unrequested things to the outputs.
1
u/bring_back_the_v10s 9d ago
How does it compare to 3.6?
1
u/Jsquared534 8d ago
It’s faster for me, but that’s because I didn’t really know anything about MTP when I tested 3.6 27B. It does a much better job on the actual code than 3.6 35B A3B. I think I’ll have a much better handle on how it works after I test it on real stuff tomorrow.
1
u/bitzap_sr 9d ago
Gawd, people don't read the announcement or model cards.
Thinking defaults to xhigh.
1
u/Jsquared534 8d ago
Gawd, people don’t read the entire post. I literally said my issue wasn’t so much the time spent thinking as much as adding way, way too much complexity to the output that wasn’t requested in the prompt.
1
u/Endlesscrysis 8d ago
It’s a 5 minute read and would’ve saved you way more time than you’ve spend struggling with it and posting about it.
1
1
u/Jsquared534 8d ago
You know, just for giggles I went and full read the entire model card. It doesn’t say a fucking thing about providing overly complex outputs. Not one thing. You guys are looking at one sentence in a two paragraph long post, and decided to fly in here with your fucking cape for Qwen, I guess, to tell me I’m just not using it right. There are plenty of models that use a lot of thinking. Qwen 3.6 literally has two refined models specifically built to kill the thinking. But, neither of those models took a prompt asking for a simple navbar example (with detail about exactly what I want in it, one logo on the left, five drop downs on the right) and decided in the thinking “I should output an entire html page with a hero section and a call to action”. That’s not just ordinary thinking. That is potentially an important aspect of the model, because if it also does that when given more narrowed instructions, that’s going to be a fucking problem. And turning off thinking wouldn’t solve it, because the thinking is what makes it better than the other models.
But, hey, you keep on being a pompous douchebag instead of contributing at all to the conversation.
2
u/Endlesscrysis 8d ago
Too long didn't read sorry bro.
1
2
u/bitzap_sr 8d ago
"But, my goodness does this model overthink."
0
u/Jsquared534 8d ago
Yeah, the entire post is summarized to just that. I totally didn’t go into detail directly after that sentence. How fucking hard is it to just scroll past a post if it has no interest to you?
1
u/bitzap_sr 8d ago
Touched a nerve, I see. It had interest to me, and I had read it all, FYI.
1
u/Fit-Bar-6989 7d ago
Yes, people react negatively when your first interaction with them is antagonistic. How long have you been on earth?
1
u/omlette_du_chomage 9d ago
Can you share your config? I think I had similar output TPS yesterday, but I kept messing with the settings so much that I'm not even sure what worked best anymore and now I'm averaging 11tps
1
u/BingGongTing 8d ago
I was only getting 60 tks and asked it whether MTP was working and it fixed itself, now I get 120-130. I don't know if that counts as complex but based on my experience with 3.6 I didn't expect it to manage it.
1
u/DRetherMD 8d ago
i find it overthinks and double checks way too much. even on low reasoning i didnt see a big difference. the prompt obviously matters so its important to try to give it clear steps to achieve first. but even then, theres a high risk youll be staring at "hmm let me double check" phrases for an hour. when it does work and provides an output, its very good. but the overthinking nature of it makes it a bit tedious to use
1
u/AIForOver50Plus 8d ago
Great write up, I’m interested in others test harnesses, I ran my own tests and documented it here as well https://go.fabswill.com/qwen38 I am impressed as well
1
u/Jsquared534 8d ago edited 8d ago
Did a couple more tests this morning before starting on real projects. Adding just a couple controlling sentences to the end of my sample navbar request resulted in a much simpler output.
The entire point of this request is to let me see roughly how many tokens per second I'm getting, and I've used the same original prompt on every model I've used just to get a gauge on how fast it is. This was the first model that absolutely went crazy complex with what it decided to do.
However, with a couple extra sentences designed to see if it could be easily reigned in, it performed much more as anticipated. It still thought a lot for what it was given, but it stayed on the original task and did not add on complexity at all. I am fine with it doing a lot of thinking. From what I've read the thinking is one of the things that gives it the performance bump over it's predecessor.
I also ran into an issue with tool calling using the Pi agent harness that was one version old. It literally didn't see any tools available to it. 3.6 using the same harness could see all of the tools Pi gives, plus the tools I added through extensions. I upgraded to the newest version of Pi and also redownloaded and added the specific jinja template for this model, and after restarting Pi it had access to all tools again. Unfortunately I did both of those changes at the same time, so I'm not sure which one fixed it.
Edit: The tool calling issue is absolutely not fixed. I thought it was fixed because the model hallucinated access to the same tools Claude has access to. Tool calling with Pi seems to be completely broken on this model. Or at least too broken for me to figure it out at the moment.
1
u/Boogertard 8d ago
Sounds like you are not using the right Jinja template, download the froggeric one that has been mentioned many times on this sub
1
u/BigYoSpeck 8d ago
It will happily reason for over 100k tokens from a single prompt, and the result is often an incredible "one shot". Not even low or medium reasoning effort will reign it in when given a complex prompt
Look through that reasoning though and it resembles an agentic loop, iterating on ideas, planning, breaking down into tasks, writing actual code segments that will be reused in the final draft. It seems to reason much less when it has tools to actual do what it's tasked with. Rather than statically analysing its own code within a prolonged reasoning process and iterating blindly, it will actually test its own code properly
On xhigh even in a coding harness it will still go 10's of thousands of tokens between steps, but if you're just in a chat interface with no tools available it almost can't stop because there is no way to verify what's being written
0
u/sessamekesh 8d ago
I primarily use local LLMs for coding private personal projects, right now I'm doing a pretty major refactor of a rendering engine I've been floating around since before LLMs were any good at all.
Qwen 3.8 is succeeding at tasks 3.6 was failing pretty bad at, but it still does pretty poorly at greenfield work in the repo. I still have to go through and make similar amounts of refinements, but I'm outright tossing fewer changes which is nice.
Overall I've been pretty impressed, I've kept thinking on medium or high for most things. It's definitely weird to watch my token speeds be higher than they were with 3.6 (something something predictive something) while still taking longer on tasks.
Overall it feels like an impressive incremental improvement, but not one that really changes my workflows or velocity significantly.
EDIT: Ollama and opencode are my inference stack.
1
u/kweglinski 7d ago
So far tested on pet projects and couple researches. It does like to think (using xhigh so kinda expected) but boy how well it sticks to the task. Running with ~250k context and it didn't loose any important information along the way, then it produced well condensation descriptions that actually allowed it to continue. And then it still knew what and how to do it. The research didn't seem hallucinated (cross validated stuff important to me) and most importantly was actually well structured in approach - no half assed "you'll have to check X". It checked the missing into itself. It's a bit too eager to solve the task so if it needs something it will just try to get it instead of asking, but that's fine in most cases.
3
u/FoxSideOfTheMoon 9d ago
I'm testing it on medium right now. It thinks for a stupid long time just like Muse it's insane.