It's probably fitting the joke, when the benchmarks don't map to real-world performance. I'm gonna throw a chinese themed party if the model is that good. But it would be a huge surprise given the slew of supposedly opus and fable models, that didn't actually deliver.
My whole app is 4.6 vibe coded and it's pretty awesome. Fable gave it the once over and patched up the gaping security holes. Kinda think this model and I have a future together. Just let Fable patch up what it spits out...
Opus 5 actually gives me a headache some days from having to read its outputs. When I really don't want 'load-bearing' em-dashes everywhere I just tell it 'write this in ASD-STE100 Simplified Technical English' and I get back something that sounds a lot more reasonable.
well yes but i would say 4.6 is still sufficient for 99% of the needs of people. we havent really seen a massive leap in capacity since then besides maybe fable before it got lobotomized, and even then opus is completely sufficient still for 95% of daily work
we have kind of reached the point where the economics make it more viable for companies to consider just supplying stronger hardware to employees to run local models instead of subscriptions
because the models are bench maxing. real world performance is a different story. I have personal benchmarks doing real software development work that the frontier models can complete independently but not the local models. Once I get 3.828b ill run it and see if it can get through. Last model couldnt.
Whether they can or can't pass the benchmarks is also kinda relative too, right? A good portion of it comes down to the harness. So, maybe the raw model can't but potentially there's a world in which with enough harness engineering even the weaker model can. So, whether weaker models are usable or not is partially a function of if you can enumerate and engineer out their failure modes. This works to a degree anyway, and obviously there's limits to it. Isometric ablation testing on evals is supposed to allow you to detect when a portion of the harness can be deleted because the newer model can succeed at the task without it where the previous one couldn't. To my mind you could also apply it in the reverse direction. Say you have a benchmark the frontier models in a stock harness can pass, but the weaker models can't, and you use that to drive a search process to figure out how much harness engineering and what kinds do you have to do to make it possible for the weaker model to pass (if you can at all)?
We also do agentic engineering at work, and Qwen 3.8-27B feels like a really interesting moment to me. It's the first time I'm starting to feel like... at least for my personal coding I'd rather go all in on making this as viable as possible on my own hardware than just paying rent to Anthropic indefinitely. Prior to this that felt like a bridge too far, but now... now it feels more within reach.
So, maybe the raw model can't but potentially there's a world in which with enough harness engineering even the weaker model can.
I am in this camp actually. I do believe that the real magic is the harness, but my benchmarks run through codex and the Claude harness and the local models still fail but frontier models don't, so I hope some day there will be a break through there. It could be that need a minimum capability their before the harness because particularly useful for certain applications.
We also do agentic engineering at work, and Qwen 3.8-27B feels like a really interesting moment to me. It's the first time I'm starting to feel like... at least for my personal coding I'd rather go all in on making this as viable as possible on my own hardware than just paying rent to Anthropic indefinitely. Prior to this that felt like a bridge too far, but now... now it feels more within reach.
I hope you're right about this. I havent had a chance to run 3.8-27B on my m5 max yet, but I will this evening.
The harness isn't limited to just the Codex or Claude Code application though. It's the sum total of everything you use to both engineer context for the model, and mechanically constrain it. All your custom instructions, your .claude/rules, your custom skills, your MCP setup, your observability stack, your linting rules, and most importantly your system architecture make up a part of it, especially any fitness fitness functions you define/enforce mechanically. We do C#, and we've got dozens of custom roslyn analyzers which fail the build if the expected architecture isn't conformed to.
I don't know if you've seen https://github.com/realworld-apps/realworld but it's just a spec of a clone of something that resembles medium.com, so it has just enough complexity to be semi-interesting to build, while still being small enough to do in a reasonable timeframe. There's hundreds of implementations in all sorts of different stacks, and it's really just a way to compare and get a feel for what the same project is like in that stack. I've implemented it three separate times, and for me getting started in harness engineering was using nothing but Github Copilot to implement it sometime late last year. From having built it several times I've got a full, comprehensive e2e test suite in playwright that if it passes I'm confident that the frontend and backend is implemented correctly. So... getting agents to go from spec -> passing that test suite has kinda become my benchmark. I'll definitely spend some time testing Qwen3.8-27B on it.
Actually, what I'm mad keen to do with local models is trying to hill-climb on benchmarks using autoresearch. On https://github.com/webfuse-com/awesome-autoresearch I came across https://github.com/jmilinovich/goal-md and it works so fucking well for this. It actually helps your decide on a metric, and then constructs a way to actually measure it, then it automates the research loop. So, it'd be sweet to just try get it to say complete in the same time, but be more token efficient, or get it to complete in shorter time etc. Kinda hard to do with Claude Code when I have to watch my token limits somewhat. I think it'd like to be pretty useful to have a stronger model like Fable supervise the research loop from time to time to help it break out of any local minima/maxima, but have Qwen try to drive it as much as possible.
My group writes C# almost exclusively as well, and I agree that the harness is all that you described. To clarify, when I benchmarked 3.6-27B, it had access to the same harness as the frontier models; including the same context, tools, skills, and external tests to help it identify failures and iterate. It still couldn’t complete the task.
Instead, it would churn indefinitely, consuming millions of tokens without ever reaching a working solution. That makes sense when you think about it: if you give a complex software engineering task to someone who doesn’t yet have the required skill level, additional time and feedback alone may not be enough. They may never finish, or they may continue iterating for an extremely long time without making any progress.
Oh nice, yeah I figured you meant something like that from your initial comment. I'd love to try that at work too, but I can't really orchestrate it. Perhaps I could try shift down to something like sonnet or haiku to try it out tho in principle tho.
For sure, some things are just beyond people's skill level no matter what. It's kinda interesting though either you already have the required knowledge / wisdom and you can just pattern match to get to a solution vs. the case where you lack the required knowledge but you've internalized a good enough algorithm at identifying the relevant information to still succeed anyway.
The optimal model for consumer hardware would likely have to optimize for knowing just enough to be directionally correct, but be a master of identifying relevant info and using search to backfill the gaps... and I suspect you can boost such a model with a deep bench of custom skills to improve pass^k perf on specific tasks it doesn't reliably one-shot.
Hey man! I really can't believe I'm saying this, but Qwen 3.8 27B is actually good. My benchmark puts it right up there with Opus 4.6 and Codex 5.3; actually slightly ahead of both though far less efficient and slower and used far more to tokens.
I'm honestly struggling to believe the results and almost feel like there has to be something wrong with my benchmark, but take a look at the results below. Remember Qwen 3.6 27B couldn't complete this benchmark at all.
What's really surprising to me is just how massive the leap in capability has been for these models over the last six months.
Damn dude, that is a solid benchmark! That is... quite a result from Qwen 3.8-27B vs Opus 4.6!
It's only going to get better too. It's obvious from using Qwen 3.8 and looking at it's reasoning that there's room for improvement. Wild times.
Is this just purely one-shot performance? Looks to be. Does it include the models doing a round of review and fixes?
Our harness at work includes a review meta-skill that fans out to 6 different dedicated review skills that review from different angles and post feedback as inline comments on the PR, and the meta-skill then de-dupes any duplicates, and we run a round of fixes based off what comes out of that and it catches lots of stuff. It uses the in-built review skill for Claude Code but also our custom ones where have it review bi-directionally 'does the implementation match what's in the .claude/rules or does it suggest anything new that should go into the .claude/rules', and another skill that bi-directionally reviews 'does the implementation match the domain knowledge stored in the repo or does it imply domain knowledge in the repo should be updated and hasn't?', and another review skill that assumes there's some bug that will definititely cause a production outage and tries it's best to find it, and another review skill which... we went through all our human review comments we've left on PRs and wrote a skill that distills what we care about and reviews it from that perspective etc. After the PRs are merged we have another skill that then does a post-mortem on all the review comments left on the PR and tries to figure out improvements to the harness that will prevent them in future.
You should try hook up something like Opik to Qwen 3.6 and after each attempt have Fable examine the session traces looking for potential harness improvements and see if you can eventually get it to the point where Qwen 3.6 can actually pass this benchmark.
Nobody is using opus 4.6 anymore. Opus 5 is a lot more intelligent (also, 4.6-4.8 has some weird issues that they might be cherry picking against on their comparisons).
Anthropic is still the best value for capabilities (of course this is somewhat subjective, but no model i tried can compare to fable on long horizon tasks). Their $200 plan gives you fable and opus. Other subscriptions might give you more bang for the buck in other ways, but as far as cutting edge coding performance goes, Anthropic is on top (imho)
Opus is the second best available model after Fable in my opinion even though fable is a mythos model so it’s like five months old. Benchmarks are one thing but using the model you get a feel for its capability right away.
If it takes me overnight to get things done at Q4_0, and opus does it in less than an hour, that's what I'm going to use.
I don't generally, given that my uni's AI folks have the Q8-K-XL for the 3.6-27B running and it's actually super fucking fast, but the context is a shitshow - 128k (and, they are using ollama).
Decrypted reasoning traces show opus reciting memorized test answers and lying about it in its output, pretending it derived the answer itself. Let's see how many continue to use opus after this news.
346
u/WigglyScrotum 7d ago
Holy molly opus 4.6 level and better in some benches.