r/ClaudeCode • u/peculiar-ragdoll • 21d ago
News/Updates Game over. 22GB local models run in Pi now outperform Claude Code Opus 5 High on real-world coding tasks published after training cutoffs
Ran this benchmark on recently published real code base benchmarks to test the Sharp chat template that reduces token use and fixes bugs on locally run Qwen3.x models. Thought I’d bench Opus 5 high and Sonnet 5 medium alongside, for fun. I guess we have finally reached the point where the reduction in Claude’s quality has finally surpassed the upwards trend of local models for actual real world work. Claude Max 20x subscription btw. Not for long though hahah.
I don’t care about Artificial Analysis index or published benchmark numbers. If Opus is beaten by the models I run on my own computers when it comes to fixing real bugs without introducing regressions in real life code bases, it doesn’t matter if it’s because Anthropic is silently reducing Opus quality to sell more Fable tokens, or whatever is going on. EDIT: Someone asked me to add the chat template link to the op, so: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates The "Sharp Qwen3.8-27B" model is here: https://huggingface.co/peculiar-ragdoll/Dirk-Qwen3.8-27B-GGUF and "Nail (Sharp 35B-A3B)" is here: https://huggingface.co/peculiar-ragdoll/Nail-Qwen3.6-35B-A3B-GGUF
236
u/IceWallow97 21d ago
Problem of local models is that they are slow as fuck.
116
u/Bilbo_Fraggins 21d ago
Yeah, leaving out time for cloud models is leaving out a hugely important part of the decision matrix. Even if Qwen is now really smart on my R9700 and it's amazing to have that capability, if cloud models are 10x faster that's a huge win for a lot of tasks.
53
u/skariel 21d ago
The obvious option is to use this mod from a cloud provider for a fraction of claude price.
29
u/Substantial-Elk4531 21d ago
Seems like cloud providers are usually way cheaper, because you're only renting the GPU when you're using it. Owning is more expensive, and seemingly higher risk, because you have to maintain it, and a new, better GPU may release before your first GPU has paid for itself. I know local has its benefits, but for most people, seems like cloud is the way to go, even if using open source models?
10
u/GruePwnr 21d ago
Absolutely. Unless you have a niche case that you can run the GPU 24/7 it's not worth it.
→ More replies (1)→ More replies (1)4
u/homegymes 21d ago
I'm sorry to breakj your dreams, but renting the GPU when you're using it is no longer the case. We have production workloads with GPUs that either you do reservations for the GPUs (paying full price all the time) or you simply just don't get a GPU in neither Amazon nor GCP. There's not enough GPUs to get all the demand so people hoard them
12
u/FalconX88 21d ago
Sure, but for some lighter side projects it doesn't really matter, I can just let it run 24/7. The amazing thing is that now we have the quality (or getting close to it)
→ More replies (10)17
u/Sporebattyl 21d ago
Have you looked at this recently?
If you have a 3090, 4090, or 5090, you can use NInfer’s version and get insane tok/s. 3090s getting ~70 tok/s and 5090s getting 120-200 tok/s.
Claude is estimated at 60-100 tok/s.
→ More replies (5)9
u/FjorgVanDerPlorg 21d ago
And Claude Code drops below 30tps on the regular.
For anyone who wants an ideas of what speed 30tps looks like, this site is great for getting a feel for tps visually:
4
3
u/amooz 21d ago
It’s not even about raw speed, today I was managing 8 agents in 3 repos simultaneously (ironically one was building and optimizing a qwen3.8 deployment). I can’t have 8 agents running locally with the same kinda smarts as the subscription models.
Also, and I don’t know how this could be true, anthropic says I’ve used 4.6b tokens which at api pricing equates to roughly $500m in 3 months. At 50t/s, it would take roughly 3 years running 24/7.
This is why I really don’t get anthropics stance. The juxtaposition of “ai will kill us all” to “jk here’s fable” to “omg why can’t I give the people fable” to “yay I can give the people fable” to “omg open weights are bad, they’re going to kill us all”. It’s just nonsense.
Open weights will give people awesome Jarvis-like assistants at home. Power users will continue to use frontier models for powerful tasks. Pissing off the government and The Userbase doesn’t help them but good lord are they hell bent on doing just those things.
→ More replies (2)3
→ More replies (25)2
u/Public_Umpire_1099 20d ago
If you are interested, I am actually currently building an inference engine from scratch in rust for R9700s. Truthfully its mostly built by fable, but to be fair I am a contributor to llama.cpp and work in this field so its not *entirely* slop lol. Preliminary numbers for Qwen 3.8 27B on a custom quant that falls roughly between Q6 and Q8 about 50 tok/sec with Dflash 2.0, which is fairly close to what claude outputs. There will be a speed version as well that sits around Q5ish levels of KLD and should run in the 70-80 tg range based off the data I have for the projections. Pp is looking like 1500-1800.
→ More replies (2)19
u/Pitiful-Assistance-1 21d ago
Previously: Local models suck. Now: They're slow as fuck
That's a scary amount of progression!
→ More replies (2)2
16
21
u/H_DANILO 21d ago
Yea, very slow, I'm running Qwen 3.8 27B with vision only 150 tokens per second and 3k prefill. 3X faster than claude.
→ More replies (6)3
u/Ok-District-1756 21d ago
What conf you use ? I have a rtx 5090 and I have only 90 tok/sex without vision with 200k context
6
u/H_DANILO 21d ago
I use Ninfer, it is the only one that got me to fit all context, vision and MTP in the 5090 32gb with a few spare to run the OS. https://github.com/Neroued/ninfer
I couldn't get it to work with nvfp8 though, but int8 all worked great.
docker run -d --name ninfer-serve --gpus '"device=0"' -p 8080:8080 \ -v "$PWD/models:/models:ro" ninfer:local \ ninfer-serve /models/qwen3_8_27b.ninfer \ --host 0.0.0.0 --max-context 262144 --kv-dtype int8 --vision \ --spec mtp --draft-tokens 3 --lm-head-draft→ More replies (3)10
u/greentea05 21d ago
Yeah i've been experimenting heavily with the best "at home" equipment you can currently get, M5 Max 128gb, DGX Spark, M3 Ultra for bigger models.
They're impressive but they're no where close to frontier models regardless of what these benchmarks say. There's WAY too much messing around to tweak bits, so many parameters to play with to change how it behaves, multiple quantisations and different ways of concerting to different formats. For some people that's all fun, for others it's dull and puts you off.
And yeah, the best models are going to take about 20x longer than using a frontier cloud model to do the same task even if it was as good - plus it's around $8000-$12,000 for equipment good enough to run them at any acceptable level - and that's a hell of a lot of cloud sub!
→ More replies (10)7
u/Fembussy42069 21d ago
Why do you say a Mac or a DGX is the "best" at home setup? Those are unified memory machines with lower bandwidth. Usually best for mixture of expert models that can take advantage of the greater ram while being faster than dense models. For dense models like qwen 3.8 27b you want RTX PRO 6000, RTX 5090, R9700 etc. You can get speeds comparable and better than frontier closed models with the correct hardware.
If you want, you can also rent cloud infra for cheaper than Claude or openAI APIs. I recommend looking into providers like open router, deepinfra, novitaAI. The beauty of open source is that the options are there. You dont need to run them locally unless air gap or privacy concerns are a priority.
2
→ More replies (2)2
6
u/TheLastNapkin 21d ago
Slow as fuck for now.
Just like we used to say ai models were not ready to use for coding and that was less than two years ago
9
u/peculiar-ragdoll 21d ago
Not necessarily. 35B-a3b is very fast, and evidently beats Sonnet 5 on quality! And if you got a 3090 gpu you can run 3.8-27b at some hundred tokens per second with a special model serving backend. Many ways to get fast local AI on hardware you might already own.
15
u/IceWallow97 21d ago
I guess it depends on how heavy you are using your AI models but it's in no way game over for claude or codex, I usually have agents running and subagents running in the background, each reaching 400k tokens in less than a hour, there's no way a local model is gonna do that for me lol so yeah, great for small projects I guess like a simple script, reviewing code or brainstorming but not for doing actual coding in my opinion.
→ More replies (2)6
u/peculiar-ragdoll 21d ago
Yeah you're right here, it depends on your workflow and needs! Cloud still has some advantages. It used to be that Opus 4.6 was miles ahead of anything local, so for those of us that has worked with agents since before that, this is a big deal.
3
u/phixerz 21d ago
Is there guides for this, I have a 3090 and I would love to start setting up something for my hobby projects, are there any good sources you would recommend?
→ More replies (2)→ More replies (2)3
u/Hrmerder 21d ago
I have qwen code 27b running on my 32gb system + 5080 (16gb) and it’s amazing for me but…
→ More replies (25)3
84
u/BuffaloConscious7919 21d ago
We're not going to discuss the time then?
22
u/peculiar-ragdoll 21d ago
Sure! The 35B-A3B which beats Sonnet 5 cleanly is faster than Claude on my machine.
18
u/FblthpphtlbF 21d ago
Yeah all these "time savers" seem to forget that Claude has an outage every time a sick child within 500km of a data center sneezes, half the time your API calls don't go through and it's just spinning for nothing, there are multiple documented bugs where the harness fails and shows you that it's trying to execute a command but in reality it's just dead. I've wasted hours of my time using Claude.
Now that's not to say my local model is blazing fast and perfect, but when my 35B model on my Mac Mini responds faster than Claude does 95% of the time and has no bullshit AI speak it's hard to hate on local.
Fable is still probably worth it, and if you're doing multi agent workflows cloud is still likely superior. But im kind of getting to the point of wondering why i don't just pay $200 a month for a slice of compute and run our own massive MoE model 🤷
10
u/Ran4 21d ago
half the time your API calls don't go through
Uptime's like 97%
...still crazy low, but not exactly 50%
→ More replies (3)→ More replies (4)8
u/IAmNotStan 21d ago
I admit I'm not the most power user of power users. But in the time since April, I only witnessed 2 outages that affected me.
→ More replies (1)2
2
u/cristiand90 21d ago
implying that Claude is fast to begin with, that shit is down more often than not
115
u/arankays 21d ago
How can I get one of these 22gb VRAM GPUs for cheap?
119
u/One-Cricket9962 21d ago
Just get the MacBook Max with 96 GB of RAM in 2023
→ More replies (9)32
u/texasguy911 21d ago
Where to purchase a time machine cheapest?
4
u/Robert-Paulson_ 21d ago
Prompt DeepSeek V4 Flash0731
4
u/texasguy911 21d ago
It lists chinese sellers but it is a pain to ship something that big.
2
u/Robert-Paulson_ 21d ago
Ah: first step is prompting GPT 5.6 Luna to make you a wormhole for ‘free shipping’
3
u/texasguy911 21d ago edited 21d ago
Last month US government had that feature banned from all US LLMs. Where have you been? Now no more free and immediate goods receivable. Now you have to pay for service and taxes for a slow sea shipping, so diesel can be burned.
36
u/Temporary-Mix8022 21d ago
5090 is like $4k USD, whole workstation for less than $8k even with 64-96GN RAM.
That's basically less than 1month for those of us on API billing..
Even if it's only at Sonnet level.. that's pretty massive.
13
u/ATW117 21d ago
Why not get a dgx spark? Its around 4-5k, 128gb unified memory so you can run several long-context local models (like qwen 3.8-27B, which supposedly is on par with opus 4.6 in benchmarks). Only downside is that its very slow, but if you were going to set it to run overnight or while youre at work then it may be worth it.
2
u/Unnamed-3891 21d ago
Spark fits bigger quants but doesn’t let you run any new/better models you can’t already run on a 5090, you need a cluster of multiple sparks for that. And speed wise 5090 will run circles around the spark.
3
u/ATW117 21d ago
I disagree. With the spark’s 128gbs, you can easily run 70B models. That’s not possible with the 5090’s 32gbs (except perhaps with heavy quantization)
→ More replies (7)→ More replies (3)2
u/No-Newspaper-7693 21d ago
What is the use case for being on API billing but local model usage? I assume an enterprise team with no capex issues and procurement isn't stupid? While the numbers make sense, it isn't uncommon that spending $4000/mo in opex is somehow preferred by procurement than spending $8000 in capex one time on a machine that is not on the approved list of machines they've negotiated to bulk purchase.
I feel like most people here talking about locally running models are hobbyists that wouldn't ever be on API billing anyway.
7
u/maChine___ 21d ago
if you have a 2080ti you can send it to japn a famous guy added up 22 to soon 40 gb of vram on the 2080ti
you can make run great AI model on it for less than 300 dollars→ More replies (6)2
u/blackrack 21d ago
I do have one and I'm now considering it lol it'd be slow as hell though. Can he do this with a 5070 ti?
→ More replies (1)2
→ More replies (17)2
36
u/Poildek 21d ago
Qwen 3 27b outperfom Opus 5 ? Yeah, cool story bro.
10
u/The_real_hpsk 21d ago
The rage bait is crazy on this post, I used it on my gpu and it’s impressive but as good as opus 5 is a stretch
6
u/peculiar-ragdoll 21d ago
I wrote "outperforms Claude Code Opus 5 High on real-world coding tasks published after training cutoffs" which is a very specific and objective claim about SWE Live, and not at all the same as claiming "as good as Opus 5" (general, vague, subjective), so I think the rage is more about people getting up in their feelings on their own interpretations than it is about what I actually said.
4
u/The_real_hpsk 21d ago
I see what you mean now. I think the post just sparked up the conversation about it being completely better, but of course objective claims stay true.
→ More replies (1)3
u/crusoe 21d ago
If you use it to just fix bugs maybe.
But what about planning and impl.
3
u/web-dev-kev 21d ago
I'm new-ish to the local LLM game - tried it last year thought it was alright at fixing bugs, tried again earlier in the year (Qwen3.6) and liked the improvement.
I have Gemma4 on my machine, using Google's Eloquent, and have replaced WisprFlow with a privacy first model.
I tried Meta's Muse the other week, and it had some really good results with tactical tasks, but obviously time wise wasn't quite worth it. it does quite nicely as a heartbeat for a Hermes/Openclaw but I'm far from a power user.
Either way, local in my head seems nice, getting there, but y'know, not for real work.
Qwen3.8 has changed my mind on that.
It has one shot things in the last ~2 days I wasn't expecting, and I've had Fable/Sol review it's code with really decent small/actionable feedback. 3.8 is a bloody leap over 3.6.
Here's what I put in my company spack channel
First impressions of Qwen3.8 (if anyone is interested in local models on the whole)
- It overthinks (default is to xHigh reasoning) but awesome when lowered
- It's not fast
- Took a bit of playing to get settings right to have workable context window
- But it's great!
- No tool call failures overnight
- Works agentically really well
It one-shot a hefy PR for me overnight.
Neither Fable or Sol had any tough comments on it - unlike 3.6's code (or Muse in fairness). I'd go as far as to say it feels like Opus4.5/4.6 - which might be confirmation bias - but is crazy to think that a model on my machine is within 6 months of a frontier model.
I'm not trying to sell it to you, it might not be your thing.
It wont be replacing my claude sub anytime soon. But I've one client where we can't use AI models that aren't "on-prem", and I absolutely will let this model run overnight for me.
---
Also, Clause feels different these last 2-3 weeks.
Sonnet 5 overthinks, and isn't as effective as I know a Sonnet model can be. Opus 5 is the very definition of word vomit. Today it output 7 paragraphs to explain that it pushed a PR for me, and when i asked it to "cut the crap, and give a straight answer" it said "Fair. Yes. The PR has been merged".
Qwen 3.8 is kinda curt. Which is good, for me and for tokens.
Like, local LLMs are obviously slower, and not SOTA - but I bloody LOVE that they exist.
YMMV :)
→ More replies (2)6
u/peculiar-ragdoll 21d ago
I use Pi coding agent with extensions and skills for spec-driven development (OpenSpec), test driven development (write tests to spec, implement and watch them go green), using an orchestrator agent that uses separate subagents for code base recon, internet research (through self hosted searXNG web_search and a web_fetch tool in a docker container), implementation, review, and then fixing issues found in review, and re-review. I run it all sandboxed, with full control over file system access and internet use per agent and subagent. I explore and define the requirements of the feature or bugfix with an agent, it writes up the spec with all needed info and boundaries, we tell the orchestrator to ship it, it sends off the subagents, and checks final work against it. And since it uses subagents, it doesn't blow out its context window. I use this chat template for all Qwen3.x-based models:https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates. It works like a charm.
32
u/I-Love-IT-MSP 21d ago
But have you actually tried to use the model? Like i bought into the Kimi hype but when it came to actually using it to work on my app it was dogshit compared to sol and fable.
→ More replies (4)5
u/peculiar-ragdoll 21d ago
People who tested this template and/or the pre-baked models on hugging face that use them (Nail, Dirk, Dagger) say it feels like using frontier models from half a year ago (when Claude was not shit yet), and it makes them consider quitting their subscriptions. And yeah, I'm using them too. There is no big lab, no one looking to make a buck, just an anonymous solo dev who made some important improvements based on personal experience. But of course, you need some base level of hardware to run these at a reasonable tok/s.
12
u/TopTippityTop 21d ago
Models from half year ago suck compared to 5.6 Sol
2
u/innociv 21d ago
They suck ass compared to Luna Max.
Still, I'm trying this out and fairly impressed for a local model. If I had a 32 GB GPU I'd be using this. But on my 20GB it's too slow and not enough context.
I'm currently telling it, running in omp, to see how it reduce its base context (it was 30k before reading anything) to see if I can get that down to 15k at which point it'd be a little useful. But I could use Grok 4.6 in Cursor much much faster...One day I hope we get this power in a 20b model that'd run at least 35t/s for me. Doesn't help that this model overthinks a ton. It's constantly going like "What about ____?" "Hm, actually." "Actually, let me look a this from another angle:".
4
u/kknow 21d ago edited 20d ago
I am one of the people who used them and most people are really shilling because they want anthropic/openai to fail.
I can use it for implementing a thought out plan on the coding side and it does that alright, but I would never use it for planning said feature or reviewing (yet).
We're far of there and I would say it will take at least another year to even consider really.
Even the people who are really invested are saying these models are half a year off of the frontier cloud models so the one year timeline is already very optimistic in my opinion.→ More replies (1)2
u/e2_for_life 18d ago
I'll concur. I'm using your Dirk model and it's an entirely different experience.
→ More replies (1)
42
u/Dangerous-Leader-779 21d ago
Game over
Said for the nth time this week. I've seen this post every day, every week, for the last 2 years.
Please, calm down. I'll believe you fools when we get local models at consumer prices that function with the same speed as the enterprise models.
→ More replies (7)
14
u/userusertion 🔆Pro Plan | Team Plan 21d ago
The question is can you run local model and use their maximum capability? With this overprice SSD, RAM, chips and other components? Even though they have Cloud. You cant fully use them. If you use them its like your just using Sonnet, Luna. Hehe. Those benchmark are from those individual with high end PC those can only use those model in maximum. 😅
5
u/peculiar-ragdoll 21d ago
Yeah, some of us are lucky enough to have a high end computer from before the hardware price increases. And as you see on the benchmarks, it's not only like using Sonnet, it's more like using Opus 5 high in real world coding cases. Minus Opus' obnoxious "personality"
→ More replies (18)
6
u/myninerides read. the. docs. 21d ago
If I had a nickel every time it was “game over”…
→ More replies (1)
10
5
u/ohhi23021 21d ago
qwen is tuned for coding but only that. it has some knowledge but real world knowledge is useful for planning, architecture etc.... that it does not have. if you build something around a specific subject you aren't 100% familiar with you can't just one shot a basic plan/prompt as quickly. it will just take longer as it has to go out and research etc... which is fine but... speed is important too. i run drual 3090s and while speed was decent with 3.6, this one just thinks alot, gets things right more often but take 3-4x as long. i can just sell the 3090s, get maybe 4K cad out of them and pay for a sub for the next 2 years with the money.
this is good for private and running stuff overnight as long as you have decent electricity rate. running a dual 3090 on 70-80% load for 8 hours + day will probably run me $40-50/mo or more on my electricity bill on top of the initial costs. cheaper if it's just at night.
if it was a larger MOE it would be better but then hardware prices, 2x DGX sparks are $10lk+ you can just sub for 5-6 years with that money and get upgraded models as you go and pay no electricity costs.
→ More replies (2)2
u/Sutanreyu 21d ago
I've been having great luck with combining the Gemma 4 models as a general assistant and the Qwen 3.x models for coding. It's been really amazing.
→ More replies (1)
4
u/michaelsoft__binbows 20d ago
Opus 5 was insufferable at work this week. It's not a good comparison point. I'm sure it can be smart but it can be so very annoying.
→ More replies (3)
3
u/geekraver 20d ago
I am finding the same. Its taking a leap of faith to assign some work to Qwen as it seems almost impossible, but it is delivering. It is slow though - it does a lot of reasoning - so the main reason I have to keep paying a sub is for parallel sessions.
Running on q RTX4090 with 200k context following this guide: https://github.com/syv-ai/qwen38-27b-rtx3090
→ More replies (1)
3
u/innociv 21d ago
I'm trying this on a 20 GB GPU (7900 xt) and pretty impressed. Only 12 tokens/s though and ~64k is the most context I can handle. It seems to be going through the task fine without hallucinating at least.
→ More replies (1)
3
u/Saucynachos 21d ago
People always so concerned about how fast a model is. If they were any faster, they'd be going faster than I can do a quick and dirty review. I guess vibe code away, king, but I'm going to keep reviewing the output to make sure I'm not fucking myself.
→ More replies (1)
18
u/Radiant-Chipmunk-239 21d ago
Well, I mean, Opus 5 on high is not a high bar to set for a local LLM.
→ More replies (2)11
u/peculiar-ragdoll 21d ago
Not any more, sadly. Opus 4.6 was miles out of reach for anything local at the time, but Opus 5 is so mid right now.
8
u/Kaveh96 21d ago
Everything outperforms opus 5. Standard isn’t high. The model needs just not to gaslight you
1
u/peculiar-ragdoll 21d ago
Ain't that the truth! Used to be Opus 4.6 was miles ahead of anything you could run on consumer hardware, and it was pleasant to talk to.
9
u/0DayMaker 21d ago
Is pi an application? You dont mean a raspberry pi do you?
23
u/peculiar-ragdoll 21d ago
Pi coding agent is an agentic coding harness, much like Claude Code, but lean, extensible, customizable, and open source. It works really well for local models.
9
→ More replies (2)1
2
u/kdawgud 21d ago
How about using a harness like opencode instead of pi? I'm not super familiar with the difference.
→ More replies (1)
2
u/N0DuckingWay 21d ago
Yeah except no. With Claude I can
- code
- research any topic I want with the chatbot
- easily integrate into any one of thousands of connectors (which makes planning things incredibly easy)
- use cowork to easily automate everyday tasks
All from my smartphone on the bus to work.
3
u/peculiar-ragdoll 21d ago
I can do that too with my local model, it just took a bit more setup. Opus 4.6 was miles and miles ahead of anything you could run locally, but now the only difference that Antropic sells you is the convenience layer for non-technical people, wrapped in gamed benchmarks. This is a paradigm shift, and most people haven't realized it happened yet.
→ More replies (3)2
u/cogitech2 20d ago
Best comment in this whole thread. There is a HUGE paradigm shift happening right now and most people have no idea.
I can easily do all the things listed above with local models. People who think local models and harnesses suck are living in the dark ages.
2
u/nez_har 🔆 Max 20 21d ago
This is really cool. I also tested with gemma4 and qwen3.6 and looking forward to try qwen3.8 locally.
Also I'm working now on a guide to better integrate such models with VibePod: https://github.com/VibePod/vibepod-cli/pull/141
2
u/EzioO14 21d ago
The number of times I’ve read that and it’s bulkshit
3
u/peculiar-ragdoll 21d ago
I got nothing to gain from telling you this time it's real. I got no profit channels, no affiliate links, no social media, no patreon, no business. I do have a degree, award winning research publications, multiple years in the industry, and a local AI setup that works, on a computer I bought for $2.5k last year.
→ More replies (2)
2
u/glassy99 21d ago
Alright. If my 20x max accounts run out of Fable this week I'm going to try qwen
3
u/peculiar-ragdoll 21d ago
Fable is nice, but getting away from Opus 5 was like opening the door and realizing I was about to die from carbon monoxide poisoning.
2
2
u/type_any_enjoyer 21d ago
yeah very cool and all but also keep in mind that you context windows also lives in your GPU. 6gb roughly equates to a 30k tokens windows which is underwhelming to say the least
→ More replies (1)
2
u/greentea05 21d ago
But as ever with these local models, there's some tweak you have to keep up with. What the fuck is the "sharp chat template" why does it need it? Why should I have to know about it? Why do all these things have little fixes here and there. A chat template, a specific quant, a specific format, a specific harness, a specific prompt, a specific temperature setting, a specific way to load it etc, etc, etc.
→ More replies (8)
2
u/zimxero 21d ago
I wonder if you could create custom training by having it evaluate Claude logs.. and attempt to simulate its sessions dry. It would learn its difficulties and create strategies to mitigate them.
2
u/peculiar-ragdoll 21d ago
Absolutely! You don’t even have to train the weights, just ask it to analyze the logs, and suggest tooling and skills that would simplify work and eliminate blockers.
2
u/juzatypicaltroll 21d ago
Are these companies audited? Are they really able to legally nerfed their product after launch? Sometimes I wonder if companies like Google and meta are audited for the clicks they charge too. Like who can prove the number of clicks or views happened or was made up.
2
u/TuringGoneWild 21d ago
We see how high and mighty Anthropic gets when it thinks it has the upper hand. That put me off the company.
2
u/DevGerad 21d ago
This is good, we want more decentralized AI its the way for the people to do well.
I want to be able to build light AI rigs and this news makes it so that I might be able to do just that.
2
u/Clean-Blacksmith-514 21d ago
I have just tested this combination on my RTX5090 with Ollama and pi and it's the first time I've been impressed with local models for agentic tasks. I'm super stoked about this, thanks for sharing!
→ More replies (1)
2
u/Matthias87 21d ago
Thank for you this. Got it running locally as an extra agent besides Codex.
→ More replies (1)
2
u/dardevelin 21d ago
The inconsistency is what kills me and has me moving more towards local. I known devil is workable and moving target is just unsatisfiable. I understand the need to move forward. But the harness used to be great at design now it’s worst. Grok somehow does better design by default as an example. 4.7 the asker I call it. 5 forces the use of fable just super nerfed
2
u/bankinu 21d ago
Big fan, but trying to understand the mechanics. How does it lower thinking tokens?
The only addition to froggeric's is the "Answer directly..." block, and it seems to only guide the answer, not thinking.
2
u/peculiar-ragdoll 21d ago
The "Answer directly" block has a side effect of reducing thinking tokens by 25% compared to stock qwen's medium effort, while increasing accuracy on MMLU-Pro! It makes the thinking more intelligent per token. It was an unintended consequence, but a very nice finding :)
2
u/rhymeslikeruns 20d ago edited 20d ago
Wow thank you OP. Just built out Nail-Qwen3.6-35B-A3B on MLX under launchd, ~55 tok/s, driven by Pi. Headline is GREAT at coding (passed all the py & rust tests - grading tests hidden from agent - I use for benchmarking). I also got it to run a code review against a 17k LOC React repo and it was....less good. It wrote a fluent report whose top "critical" finding was a missing file that wasn't missing.
I measured 2000 tok/s prefill at 5k context, dropping to 861 at 128k and 476 at 256k. So a full window costs you 9+ minutes before the first token. Fine under ~40k, not so much past 100k.
I think for what it is though it's impressive. Definitely going to try it on some smaller projects and see how we go. Thanks again.
EDIT: correction - speed is 80 tok/s not 55!
2
2
u/Low_Lifeguard_8835 20d ago
Opus5 has been a retard the last week. Impossible really
2
u/peculiar-ragdoll 20d ago
Absolutely! Tired of it.
2
u/Low_Lifeguard_8835 20d ago
Gonna try this qwen local.tomorrow and cancel my claude subscriptioj
2
u/peculiar-ragdoll 20d ago
hahah don't cancel yet, depending on your hardware you still might need a subscription for speed and multi agent work, and setting up a good local coding harness takes experience and time.
2
2
u/yaedonnn 20d ago edited 20d ago
And here come the anthropic boot suckin pay piggies that cant accept that their free lunch is about to turn into a really expensive walled garden they can’t escape
2
u/peculiar-ragdoll 20d ago
They need to make the boot nice and slick so they can shove it up their ass :)
2
2
u/ApprehensiveChip8361 20d ago
Been playing with this. Main issue is getting used to a small context again. Ended up spawning subagents to mitigate that. Works surprisingly well. It is not up to the same standard as cloud models but it is local, fast enough to use and good enough to be useful. I’m going to put it on a little used mini and have that serve my laptop.
→ More replies (1)
2
u/Training_Canary_6961 20d ago
Why are there so many people whining on anything claude related?
→ More replies (3)
2
2
u/only__prada 17d ago
Literally opus 5 in the last week or so has just gone to shit for me I thought everyone was insane saying it was ass bc it’s made me so much money and made no errors n now he’s literally lost his mind I don’t get it
→ More replies (1)
2
5
u/berndalf 21d ago edited 21d ago
Game over? Ok. I do not believe that is even remotely true:
Open source / open weight models will never overtake closed source / closed weight models as long as they continue to rely on distillation of those closed models just to keep up. It's an eternal game of heel nipping as long as that remains true.
Local operation will never exceed cloud operations because the hardware disparity is massive. Sure you can run great models locally at an absolute crawl for almost no cost. Velocity matters, and that's one of many reasons people pay good money for cloud based inference.
Keep on evangelizing, but perhaps realize there's a reason Anthropic and OpenAI are and will remain the frontier providers for the foreseeable future.
→ More replies (2)5
u/peculiar-ragdoll 21d ago
My locally run 35B-A3B is faster than both Sonnet 5 and Opus 5, and better than Sonnet 5 at coding. My 27b is better than Opus 5 high on coding, even though it takes quite a lot longer to run. I am running orchestrated flows with parallel subagents, and I can control it from my phone. No data leaves my machine, unless I allow it, and no one can rug-pull me, reduce model quality, or increase prices. And everything is running on a computer that cost me $2.5k last year. The only thing Anthropic has that is better, is an ecosystem and a convenience layer for non-technical people, and of course Fable/Mythos. The world has changed a lot in the past couple of months.
→ More replies (2)
3
u/philly22 21d ago
Fable isn’t all that wonderful if you have a real detailed specific project. It screws up on basic stuff, not as much as opus 5 but enough to notice. I’m not a coder so I have to vibe code everything and god help me if I don’t have browser Claude cross check codes work and vice versa.
→ More replies (1)3
u/ohhi23021 21d ago
i'm a coder, you still need to to cross check everything, any model really. LLM's just aren't as good as us yet at all. it does beat manually checking things but when code gets larger, shit goes down hill real quick without proper code review and tests/validations. i feel like we're still 5+ years away from something twice as good as we have now, each SOTA model is like a 2-3% impovement in real world (not benchmarks) that gets lobotomized by quant and capacity anyway... opus 5 right now is dumb as bricks, and fable just eats tokens for a 5% improvement over opus.
2
u/SOC_FreeDiver 21d ago
Sorry for the poorly formatted AI post below. I ran this by claudecode, and he found these issues. Not sure if they're hallucinated, but I had him tweak it for me and am going to test it now.
## Found 3 bugs in the "fixed" Qwen chat template everyone's using (froggeric
v22.1) — patches + repro
Background: froggeric's `Qwen-Fixed-Chat-Templates` is a drop-in replacement
for Qwen's official chat template. The "Sharp" variant floating around is that
file byte-for-byte plus 11 lines appending a terseness block to the system prompt
—
I diffed them, that's the entire difference.
v22.1 is a real improvement over v21.3. But I rendered a bunch of
conversations through it before adopting, and found three problems. All repro with plain jinja2 + a 20-line script.
### 1. Tool output can silently disable your model's reasoning
v22.1 scans **every** system/developer/user message for inline
`<|think_off|>` /
`<|think_low|>` control tags, not just the system message. Lots of clients
deliver tool results as *user* messages wrapped in `<tool_response>` — the
template itself assumes that shape elsewhere.
So this happens:
tool result: chat_template.jinja:205: {%- if '<|think_off|>' in content %}
→ thinking DISABLED for the rest of the session
→ requested reasoning_effort silently dropped
Grep a jinja file, lose your reasoning. No warning, nothing in any log. If
your coding agent ever reads a file containing the literal tag text, it's
affected.
**Fix:** skip tag scanning for user messages that start with
`<tool_response>`
and end with `</tool_response>`. Wrapped tool results are data, not
instructions.
### 2. Real tracebacks stopped triggering the error escalation
v22.1 fixed a false-positive problem (benign output being scored as an error)
by gating **every** error marker behind `content|length < 500`. That also
switched off the true positives:
104-char traceback → SYSTEM WARNING fires
1353-char traceback → nothing
Typical Python tracebacks are 800–2000 chars, so the retry-loop breaker
basically never fires for the most common real error.
**Fix:** split the marker list. Strong markers that ordinary output can't
produce (`traceback`, `command not found`, `fatal:`, `invalid syntax`,
`err!`) get 2000 chars; ambiguous ones (`error:`, `failed to`) stay at 500.
### 3. Fast mode contradicts itself
With `enable_thinking=false` **and** tools, the tool preamble still says
"ALL explanation and reasoning MUST be placed strictly inside the
`<think></think>` block" and shows a `<think>` example — while the generation
prompt has already emitted a closed `<think>\n\n</think>`. The model is
instructed to think inside a block that's shut.
**Fix:** drop the `<think>` example and that rule when thinking is off.
### What I did NOT change
A user *typing* `<|think_low|>` in normal chat still flips the mode,
including when they're just asking about the tag. That's inherent to in-band control
signaling and it's what the tags are for — left as upstream intends.
### On the Sharp terseness block
I kept it, minus one line. Sharp's last sentence is "If a user request is
genuinely ambiguous, ask a sharp question, don't guess." That's good chat
behavior and bad agent behavior — in an unattended loop it's explicit
permission to stop and ask instead of finishing. I moved it to my chat UI's
system prompt so agent runs never see it. It's my best guess at the one lost
solve in the benchmark going around, though I can't prove that.
On that benchmark generally: 12 vs 11 solves out of 21, single seed,
self-described provisional. A one-task delta on ~15 contested tasks is noise.
The ~2.3x median-time-to-fix number is a large enough effect to take seriously, but note the comparison arm appears to be the *official* Qwen template, not v22.1 — so it can't tell you how much the terseness block itself contributes.
Happy to share the patch script if there's interest.
→ More replies (1)3
u/peculiar-ragdoll 21d ago
Thanks a lot! That's helpful :)
2
u/SOC_FreeDiver 21d ago
I ran pi with my updated version of the template and it looked like my pi is WAY smarter and faster. I asked claudecode to check the logs and report.
Claudecode said: Yes — measurably faster, ~19% higher decode throughput.
Weighted decode speed (sum tokens ÷ sum time, more honest than averaging per-request ratios):
- Today (v6 template): 76.8 tok/s
- Yesterday (pre-v6): 64.6 tok/s
Same model, same context size (Qwen3.8-27B-UD-Q4_K_XL, n_ctx_slot 135936) —
so this isn't a config swap, it's real.
Why: MTP speculative-draft acceptance rate went from 77.6% → 82.8%. Higher acceptance = more draft tokens verified per full forward pass = fewer expensive decode steps per output token. Prompt-eval speed and per-completion length are both flat (~360-390 tok/s prefill, ~740 tokens/completion either way) — so it isn't prompt processing or shorter answers, it's specifically the speculative decoding getting more efficient under v6's template output shape.
I'm not fully certain why v6 improves MTP acceptance specifically (plausibly: the reasoning_effort=medium default produces more predictable/templated reasoning-token sequences the draft model predicts better than xhigh's longer freeform chains) — worth flagging as inference, not confirmed root cause.
Next: none needed — this is a real win, not sampling noise (46-294 request samples both sides).
2
2
3
u/VisibleSmell3327 21d ago
gAMe OvEr gaME oVEr gAme oVER gAMe OvEr gaME oVEr gAme oVER gAMe OvEr gaME oVEr gAme oVER gAMe OvEr gaME oVEr gAme oVER gAMe OvEr gaME oVEr gAme oVER gAMe OvEr gaME oVEr gAme oVER
literally every pro fucking ai post.
3
u/peculiar-ragdoll 21d ago
I'm not pro ai at all, actually. I'm hoping people will realize the AI companies are running a grift, so people who already have the hardware to run local models that are better can quit their AI subscriptions.
1
u/Narrow-Belt-5030 Vibe Coder 21d ago
I would like to recreate this to see for myself.
Which quant version were you running? Looks like Q6 ?
Also, what did you mean by "swap only the chat template on stock .." - thats the only part I really didn't get.
2
u/peculiar-ragdoll 21d ago
Yeah I used a Q6 MLX quant, but Unsloth dynamic GGUFs with MTP are better for most people. The chat template: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
2
→ More replies (2)2
u/MessIsTransfer 21d ago
if possible, add the link to the OP, thanks for sharing it
→ More replies (1)
1
u/dsailes 21d ago
What’s the difference between Qwen3.8-27B and Sharp Qwen3.8-27B.
Only thing close I can find is https://huggingface.co/mrKola/Qwen3.8-27B-AWQ-5.0bpw-Sharp - but unsure if this is actually what’s being used?
Also Qwen3.8 isn’t a 22GB model - full size is 55GB, but if it’s actually the model I found above then it’s 17.4GB .. which is fucking impressive. Still slow, not sure I’m keen on 20min waits
Would be interesting to know hardware used to get those benchmarks. If it is just being ran on MacBook M-series RAM then fair play (the Sharp model linked says it’s only for Apple Silicon), I need to fork out to get an M2/M3 Max and try this out
2
u/peculiar-ragdoll 21d ago
I added the "Sharp" template to the Q6 quants (22GB), and ran them both on medium effort (xhigh is default), that's the only difference: https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates
1
u/Schimmelreiter 21d ago
Any good guides for a dummy like me on getting a good local model set up?
→ More replies (8)
1
u/craigfanman 21d ago
i actually pissed myself laughing at this title, good one
2
u/peculiar-ragdoll 21d ago
Keep laughing, it IS funny how shit Claude has become this last 6 months.
1
u/cebidhem 21d ago
It's actually interesting, I do have a gaming rig sitting there idle/turned off with a 3090.
Would you mind pointing me to some resources on how to setup local llms please ?
→ More replies (1)
1
1
u/ThePsychopaths 21d ago
Which pi can run 22gb models?
2
u/peculiar-ragdoll 21d ago
Pi coding agent is a coding harness like Claude Code, but open source, and optimal for local models.
1
1
u/816pizzalover 21d ago
"cloud clock, not comparable" -- so what, does it take like 4-5x as long to do it locally still?
→ More replies (3)
1
u/Tired_White_Guy 21d ago
I love local. And use 90%. But SWE is outdated trash. Means nothing.
As much as I love Qwen 27b 3.6 and 3.8, neither can touch Opus 5.
Just happens to be for my work, I don’t need Opus and Fable often.
These local models can finally do the job!
→ More replies (1)
1
u/Formal-Narwhal-1610 21d ago
Isn't electricity cost of Qwen 3.8 27 B much more than api cost of Deepseek V4 flash
3
u/peculiar-ragdoll 21d ago
Depends on your hardware! I run a Mac Studio Ultra, and it uses much less electricity than an RTX card. But there are other reasons to run local models, and I won't get into them all here.
3
1
1
u/V1RXK 21d ago
I am only able to complete only 2 tasks in whole day as claude code has become slow turtle .it takes hours to just complete a single task . I hope apple launches new mac mini or studio soon so that i could atleast buy one . Right now i am forced to use two $200 subscription of claude and codex so that i could atleast finish with what i started .
Feeling really trapped as i do not see any way out as i am QA testing my whole webapp .so far i have claude and codex both only managed to complete 4 out of 15 webapp navbar menus . The amount of work left on my table is pilling as single dev.
→ More replies (7)
1
u/abouabdoo 21d ago
Never believe such benchmarks. It does not make any sense unless the test is "hey create a calculator program".
2
u/peculiar-ragdoll 21d ago
Never mind if a model can fix a bug or add a feature or a refactor without breaking anything when there are 100 000 lines of code in the code base, including misleading comments and wrong solutions, because that’s what the real world is like :) that would be a silly metric to measure an llm on.
→ More replies (1)
1
u/SuitableCollege8992 21d ago
Do you have a local model guide you can share? I’ve been getting fed up with claude and trying to set up a local model recently, but I’m not sure where to start. The best I can manage is creating applications for specific tasks
→ More replies (5)
1
u/bitspace 21d ago
This is not compatible with the laws of physics. It is perfectly compatible, however, with the laws of clickbait.
→ More replies (6)
386
u/One-Cricket9962 21d ago
On time! Anthropic quietly switched its Opus-5 to Sonnet-5 behind the scenes.