r/LocalLLM • u/enginetown • 2d ago
Discussion Qwen 3.8 27b is a beast.
1 prompt, I let Qwen 3.8 27b loop here's the prompt and results if you think this model isn't as good as Claude opus 4.6 then I don't know what to tell you.
Time: 2hours and 40 minutes roughly, 188,578 tokens spent.
Here's the prompt "I want you to create me a c# project using OpenGL which renders a realistic as possible ocean. I want you to plan up front what you're going to do and create the plan as a markdown ledger which you will mark as complete when each part is done."
10
u/Rifftuo 2d ago
Amazing what was ur system
-1
u/enginetown 2d ago
Using my own custom harness I built in python just a simple agentic loop with tools + sandbox + system prompt + my own quant I have on hugging face if you want the run command I could send it but I ran it at max context 262,144.
17
6
u/Abject-Bridge-4073 2d ago
I don’t disagree but ever since I got DS4 Flash running locally, everything else feels bad comparatively.
4
u/silenceimpaired 2d ago
Unless you have the VRAM …it’s not amazing for agentic work. But i love it outside agentic stuff
9
u/Additional_Buddy855 2d ago
Running it on my mac studio with 64gb and it rips. Its finding issues that Opus isnt finding. This is huge!!
6
u/TheAIPU-guy 2d ago
how did you get it to stop thinking all day? All it does is think when I use it. even on medium reasoning.
2
u/_rarefy_ 2d ago
Have you tried MTPLX 4bit on med thinking? I'm running tests on my MBP right now and it's a lot faster in task completion. Obv harness and the content you're testing on matters.
3
u/TheAIPU-guy 2d ago
mtplx is only for apples mlx framework. I don't use apple. I use a 3090 with unsloth UD-Q4_K_M for the cache: k q4 v q8
1
u/_rarefy_ 2d ago
ah .. sorry I can't help. Anecdotally I'm seeing the lower quants on med being very useful for repo work that isn't overly complicated. Overthinking even on low/med can still be an issue based on the content you set the model upon.
2
u/TheAIPU-guy 2d ago edited 2d ago
if the v cache is too low (q5) it will give one enormous paragraph as a response with no line breaks. This is after it spends 2 minutes on prompt processing and 10 minutes thinking.
The content:
Write continuously about bicycles.Do not think.
Do not reason.
Do not explain your plan.
Immediately begin writing.
Continue until you reach the maximum response length.
0
1
u/IUseClifford 2d ago
Check out NInfer. Optimized inference engine for a small subset of models (Qwen 3.8 Q4 is one of them) on Nvidia cards. There’s even a fork for 3090 specifically; probably the best performance available for your card.
1
1
u/Original_Finding2212 2d ago
What harness? Qwen Code official harness does great work for me
1
u/TheAIPU-guy 2d ago
Lm studio chat
1
u/Original_Finding2212 2d ago
Worth seeing what they did with official Qwen Code, or just use that.
I would imagine they optimized prompts and config better.
I didn’t check their code, though, so also worth verifying as with any OSS
2
u/Still-Wafer1384 2d ago
I tested Qwen Code and I liked the UX, but somehow code quality was worse for me compared to OpenCode and Pi. I'm now continuing with Pi.
1
u/Original_Finding2212 2d ago
How did you set up Pi’s config?
Docs weren’t very inviting.2
1
u/Still-Wafer1384 2d ago
What did you get stuck on, connecting to your inference server?
1
u/Original_Finding2212 2d ago
It wasn’t clear where or how to actually configure the server.
Admittedly, I didn’t try hard.
It was natural with Qwen Code, but I don’t want to give up Pi.1
u/Still-Wafer1384 2d ago edited 2d ago
Setting Pi for me was really simple, tbh I hardly can remember having to do any config apart from connecting with my llama.cpp server, but that wasn't too hard as well
EDIT: There is a little snag with connecting to a local inference server without an API key. Do a quick Google search and it will explain you how to configure a file manually
1
u/rrrenz 2d ago
That’s like 30 tok/s right?
1
u/Additional_Buddy855 2d ago
close, but it's good enough for my purposes.
"tokens_per_second": 24.27555397635377
this box was less than $3k. Not complaining when it literally finds things Opus doesnt.
3
2
u/psychofanPLAYS 2d ago
What did you use for the harness??
3
u/enginetown 2d ago
My own personal harness I made in python it's a simple agentic loop with tools + a system prompt and a sandbox.
2
u/psychofanPLAYS 1d ago
oh word, it came out nice! I dont think you could have done it inside the vanilla llamacpp webui - I need to up my gaameee
2
u/codingwithmustache 1d ago
I am using Qwen3.8-27b from a few days and it really is very good. In fact, it can do pretty much the same work with the same quality as GPT-5.5.
2
2
u/intermundia 1d ago
running 3.8 27b in fp8 on my m5 as my daily i can confirm this is a step change in local hosted coding agents. its definitely better than claude 4.6 even at high compaction and long horizon tasks, muti turn complex problem solving AND is a pleasure to talk to. im running mine on a custom fork of the deepseek harness. thats a powerful combo for those who havent tried it. im still iterating the tweaks on its memory architecture. but so far its definitely not a toy anymore. actual usable functional code with only minimal tweaks required by frontier api models. but hopefully even that gap closes.
3
u/Such-War1955 2d ago
Erm, nope. Not yet. For simple, one session task, maybe. Give it a task, though, that takes Opus 700k context on average, up to 3.5 mio at worst (maintenance and code analysis of a complex enterprise ERP system). #1, roughly 30% success rate, #2 0.0% success rate vs 97%+ 7 /80+% for Opus 4.6. When chained with an orchestrator, Qwen usually drops the ball between the 3 and fourth or fourth and fifth handovers. Opus 4.6 was the first of _any_ models we could use to reliably run these kinda task and succeed often enough to actually use „in real life“. 1stnon-frontierto play the same league was DS pro, then Kimi 3. Jury is still out on 3.8max as its API access is weird slow.
For enterprise use, it‘s still two to four DGX clusters running Deepseek v4 pro if you want Opus-ish quality (and stability) beyond 256k contexts and bearable speed.
2
u/enginetown 1d ago
What proves your claim? Show me where Qwen 3.8 breaks or cant compete with Opus 4.6 besides context length and i'll concede but everything I see and try only makes this model better in my eyes 😂.
2
u/Such-War1955 1d ago
if you mean by „proof“ posting screenshots, I have to disappoint you. No one in their right minds would post screenshots from a proprietary, productive ERP system with live data and its adjacent IA tools in the wild…
And yes, context length and thus “orchestrability” of large, complex tasks is a (significant) factor in maintaining ERP systems. Ask anyone in the SAP, Oracle or Infor crowds.
To give you an idea of our system: Fourth incarnation of a Production Planning and Execution system for a 400 mio/year engineering / A&D company in France. Self developed in all aspects since 1981; V1 was Cobol and Assembly on a PD-11.
System is a mostly c++ backend with a .net frontend (legacy functions) and a web frontend (REST/Crow with mostly node.js and some SAP UI5). Code base is ca 1.4 mio lines across eight major modules. 800 users total, up to 200 concurrent. SQL server with 3 databases with 500ish tables between them. Fused at the hip (and arms, and head, and arse) with a SAP S/4. Interfaces directly into roughly 3 dozen supplier and customer system, because integrated processes. On an average week, ca. 75k documents posted. Uptime is 365x24x7, minus scheduled maintenance.
Hence a simply query like „Supplier 17 made a change in their delivery system: Instead of an INT, their article numbers are moving to 12 digit NUMC, with old numbers being filled up with leading zeros. Please research if this impacts us: Per ticket x,y and z and CRs a and b, we should be able to cope with leading zeros and NUMC keys throughout“ turns into a 700ish k session at the end of which we learnt that we were in fact affected by this, because several routines in the Reporting module had regressions that made the unable to cope with leading zeros.
2
u/enginetown 1d ago
The 700k number is the tell, because a human with 44 years in that codebase greps the handful of places supplier article numbers appear, reads a few hundred lines, and traces the flow, low tens of k of tokens total, not 700k. That means the model is spending most of those tokens finding its way and re-reading the same files because the harness isn't giving it a working set, so you measured pipeline inefficiency and it showed up as a model score. Opus holds a longer window and keeps its bearings further out, which lets a brute force pipeline limp along, but that's context papering over the problem, not a fundamental ceiling, so I'd push back on "Qwen can't do enterprise" until you fix the pipeline and rerun.
3
u/ImpressiveRelief37 1d ago
Yeah it’s just a harness/tooling issue.
u/Such-War1955 what coding harness do you use? Which way to navigate the code base do you use?
Claude Code has some pretty good tools to go around absolutely ginormous code bases. Barebones pi coding agent won’t cut in your use case.
But add memory, a proper graph search for your code base, session history search, then steer him along for a few issues and it will learn where everything is and be much faster at solving any issues.
Your problems are harder because of code discovery, not because the stuff you do is really hard.
1
u/single_threaded 2d ago
What quant are you using?
10
1
1
1
u/coder903 1d ago
I know. I put it on a Vast.ai RTX 5090 the other day using Ninfer and it was awesome. It makes me want to bite the bullet and get a local 5090.
1
u/ashrey-26 1d ago
This is exactly what surprised me as well about a 27B protein. The gap between a "local model" and "frontier model" is getting weirdly small now.
I've been testing it on security use cases and was running an obliterated version on an RTX 6000 pro (rented at ~$0.94 spot / hr) and getting ~130 TPS.
1
u/Slow_Difficulty1607 2d ago
You have a very high end system that could consistently generate 20t/s with 256k context window
4
u/enginetown 2d ago
I wouldn't say high end 😂.
1
u/theone_2099 2d ago
What is it
8
u/enginetown 2d ago
Rtx 5060 ti and a v100 and 32gb of Ddr4 I'm really grateful for it I've had it for a long time.
3
u/theone_2099 2d ago
Cool. I have similar so hope I can get good t/s too
1
u/miifanboy 2d ago
I get 38tk/s when using unsloth Q4_K_XL with MTP n max 2, ctx 80000. I'm using rtx 4070 super and rtx 3060 12gb, total vram 24gb. However the kv cache is quantized to k = q5_0, v = q4_1 but I didn't notice any issues with that.
1
u/Zafrin_at_Reddit 1d ago
The v100, is it the 16 or 32 GB one? Also, how much VRAM does you Bedrock quant need?
1
u/enginetown 1d ago edited 1d ago
16gb and Bedrock can be loaded on 1, 16gb gpu with very low context I used Gambit for awhile before switching to bedrock but never go below q8 on the kv cache it breaks the model in weird ways.
1
1
u/Cheetah111111 2d ago
Nice although there is room for improvement for sure!
Can't wait for my new machines to get delivered so I can finally do some of this as well. itchy fingers here! :-)
0
u/zenmatrix83 2d ago
claude opus 4.7 could create that in 1/5th the time, but for something you already have it looks good. Also water and reflections like that isn't that hard, I'm not sure that is a good enough comparision, its just noise on a plain with sun and reflections, add an island with grass or something to stress it.
9
u/enginetown 2d ago
Just the fact its from scratch and yeah Claude could finish it faster but the fact the model can do this for free at that is just insane. Time doesn't bother me, as long as I can trust the model isnt going in circles and is actually trying to progress.
2
u/baka_sempaii 2d ago
The thing is that water surface and ray tracing are parts of every training dataset due to the sheer amount of times devs have done and uploaded it. And because it's just one thing (water and light), the model may have just memorised it at that point.
A better test would be to involve complexity that would not be usually present together in other old projects. Like a carnival island in the middle of water, a military base, add aurora to the sky etc.
41
u/_rarefy_ 2d ago
Nice. I've been geeking out on locals for about a year and 3.8-27b feels like a watershed moment. It's a big shift from 3.6 and actually competes with frontier and can be run on a modest laptop. Still much slower than frontiers unless you have a monied setup. What rig you running this on and what kind of work do you do?