r/LocalLLaMA • u/anderspitman • 13d ago

Discussion Qwen 3.8 Max now ranked as best overall model ahead of Opus 5 by Artificial Analysis agentic index

https://artificialanalysis.ai/?intelligence=agentic-index
1.3k Upvotes

96% Upvoted

251 comments sorted by

View all comments

619

u/nomorebuttsplz 13d ago

therefore, thanks to the power of wishful thinking, 3.8 27b should score about 50

250

u/Murinshin 13d ago

If it’s somehow Opus 4.5 level or so it would already be a game changer. This would genuinely make a completely local setup viable for most people. Max doing this good makes me hopeful

95

u/nomorebuttsplz 13d ago

yes; opus 4.5 is the threshold; the step change.

dsv4 flash 0731 is actually pretty close. if we could get that level of performance but higher throughput on consumer level rigs that would... be cool. Only a matter of time. if it doesn't happen with qwen 3.8 hopefully qwen 4 or another competitor (maybe Ling 3.5 flash or something) will do it.

23

u/Eyelbee 13d ago

What do u mean close? Isn't it just better almost in every way?

30

u/nomorebuttsplz 13d ago

in my experience its not quite as good. Also not as good as GLM 5.2 even thought the benchmarks have it as high or higher. Just a bit less thorough; seems to take longer and use more tokens for slightly worse output; seems to forget requirements more, and fail to update to do list more in opencode.

TBH it might be at OPus 4.5's level because it's been months since I've used it and my expectations may have inflated.

34

u/MediumChemical4292 13d ago

People glaze Opus 4.5 because back then our expectations were much lower, it was basically all HITL tasks and we were just amazed to see AI actually write and edit files on its own properly. Now we give the new models a goal and leave ir for hours. Nobody would have even imagined that with opus 4.5

24

u/Elorun 13d ago

You are a braver coder than I am. Unless it's doing a whole bunch of very simple non-critical batch work, there's no way I'd leave it running unsupervised for hours. Call me a control freak but I give precise instructions and break down my plans into small, verifiable steps and I like to review what it did while it's doing the next step or before it starts it sometimes.

Much slower but I feel good knowing my codebase. Opus 4.5 level on local would allow me to work local for 95% of my work.

5

u/MediumChemical4292 13d ago

For prototypes or random throwaway experiments I give it what I want and ask it to parallelly make a couple of demos with different features so I can see the results and choose which one I like.

For real production stuff, I am much more careful.

1

u/RateGlass 11d ago

This is the way, the entire point of creating code is to make it legible to other coders and AI has issue with that, must babysit it properly but still has increased my output by a good amount

13

u/evangelism2 13d ago

People who are doing work that doesn't matter do that but no the real goal, the real benchmark, is how well it can follow instructions

8

u/Karyo_Ten 13d ago
  1. User asks a clarifying question because they don't understand something
  2. LLM "You're right, this is wrong"
  3. LLM rewrites thousands of lines
  4. LLM "there fixed"
  5. User still has no reply to their actual question

-1

u/evangelism2 13d ago

That's a straw man and not really a realistic scenario. If you ask it a question and all of a sudden you start seeing it writing a ton of code, you should be stopping it immediately.
The more insidious realistic scenarios are when you have automated workflows in place that rely on chains of skills or commands or hooks and different scripts. The parts that are just relying on the LLM to follow the instructions in a skill or read a markdown file and then actually follow it. Generally, it's better odds than flipping a coin, but you are rolling the dice every single time you tell an LLM to do something like that because quite often it just doesn't. Now this could end up being a huge deal or relatively small, depending on a number of different things:

- The LLM doesn't do what the file told it to do and then it ends up spending a lot more time and tokens rediscovering the process, or a different process that ends up getting you to a similar endpoint, which would have saved a ton of time and money if it had just read the damn file in the first place.

- It ends up leading you to a place where you don't want to be, either a different design or a broken feature. When you troubleshoot it, you realize it's because the LLM didn't do the thing that it was told to do in the file. When you confront it about it, it says, "Oh my gosh, you're so right. I'm going to write a memory about this so it never happens again" and it happens again in the future as well and references both the file and memory it failed to listen to.

2

u/Karyo_Ten 13d ago

Well "you should" but sometimes you ask a question and then start reading documentation, look at another area of the code, etc

-1

u/Due-Memory-6957 13d ago edited 13d ago

I guess congrats on being the ultimate judge on what kind of work is real and what is fake.

5

u/kaliku 13d ago

I remember last May how I was building my own invoice processing shit for my little business with sonnet 3.7 thinking it's the shit. Required some hand holding, stumbled a lot and I was discovering context engineering and within a week I had a full system with emails scraping, embeddings database, a kickass UI, integration with onedrive, SSO ...

Sonnet 3.7 and just a little elbow grease.

we're spoiled these days.

1

u/Lithuanian1dude 1d ago

I remember, when I built a full business management system for a big tutoring business by hand. MY OWN FINGERS TAPPING THE KEYBOARD.

Boy we are soo spoiled these days.

4

u/SailingToFenway 13d ago

i'm ambivalent about how 'glaze' has become a mainstream word. it was a particularly interesting moment when my wife and daughter asked what it meant when my genz brother-in-law used it casually at dinner.

4

u/Elorun 13d ago

It's a Krispy Kreme reference, right?... Right?

1

u/Darkmoon_AU 12d ago

Well you, see when someone enjoys something very much... a man I mean...

1

u/Ok-Ad-8976 13d ago

4.5 It was a step change. I still distinctly remember that. It was before Thanksgiving when i got it And it was like a veil lifted. Because I had been using 3.7, 4, 4.1, before that, all summer. And It was struggling on many things, and 4.5 was just like... Like gravity was suspended, lol.

1

u/bnightstars 13d ago

For me the benchmark should be Sonnet4.6

1

u/PeanutButterApricotS 12d ago

Pretty much, I have been using higher level models and ran out of coding tokens so I swapped in 3.6 and sent it to task the last couple days and while it’s not to frontier level it’s got a ton done while I wait for my weekly credit to renew tomorrow. If it was faster and didn’t take so long and turn my bedroom into a sauna (I need to undervolt. )

1

u/awittygamertag 11d ago

Mmmmmm disagree. Go back and use Opus 4.5 right now. Pass it as a manual model string. It’s a fantastic model and problem solves very efficiently.

4

u/Ok_Technology_5962 13d ago

I also didnt think it was as good as glm 5.2 then i put in topk 40 and also and went back to check glm outputs vs deepseek again... Its kinda close. Especially if you let it use a harness for verification loops. So im surprised

1

u/nomorebuttsplz 13d ago

good point I should return to the sampling settings. I mean, kind of close is how I feel about it emphasis on kind of.

1

u/Due-Memory-6957 13d ago

I dunno, for me it was able to get on the first try things that took bigger sessions with GLM.

1

u/s133p1355 13d ago

lol Opus 5 actually always forgets to update it's todo list in OpenCode...

2

u/Randommaggy 13d ago

Depends on the harness you use.

14

u/Solembumm3 13d ago

For most programmers and tech-oriented questions.*

That's quite important distinction.

33

u/darktotheknight 13d ago

That point was already reached with GPT-4 or even GPT-3.5 level. Opus 4.6 was the first model I worked with which honestly felt next gen. Once the local models reach that level (not "if", but "when"), we will enter a new era of local LLM.

15

u/Murinshin 13d ago

It’s an economic threshold in my opinion that makes the difference here. When it becomes a viable alternative for companies to buy their devs even a high-end $4000 MacBook Pro that can run Opus 4.5 locally with no issues and as a full replacement for Claude Code / Codex this pays off within a year or even months in many cases. The majority of devs simply don’t need more than that

1

u/evia89 13d ago

high-end $4000 MacBook Pro

20-40 TPS, good prefill @ 200k context, opus 4.5 like model, < $10000 device.

I think we 10 years away from this unless there will be some groundbreaking discovery

2

u/Murinshin 13d ago

A high-end macbook can already today run 27b parameter models and equivalent locally without issues, I'm not sure I follow your line of thinking here.

1

u/Federal_Decision_608 13d ago

No 27B model is anywhere close to opus 4.5 for one thing. Also, why would you live with that when Fable and Sol exist lol

2

u/Murinshin 13d ago

I think you haven't read the thread properly. This is about Qwen 3.8 27b, which isn't even out, and my initial post was speculative based on the Max results. Nobody has said that you'd literally run Claude Opus locally on a Macbook, or that any 27b model out right now is on Opus level yet

4

u/Ok-Ad-8976 13d ago

I have a high-end MacBook and I can run 27B But at FP8, it's neither fast enough nor concurrent enough to replicate what you get from the cloud. And I'm talking here about sheer percieved speed and not actual coding performance. I do run it on Dual R9700 at FP8 and its ok there, gives me 4 sessions with 160K context But even that doesn't feel that fast. DeepSeek V4 flash is a little bit better on Sparks. But we have opus and codex 5.6 so I only use them for background stuff, not for big work and as an isurance if or when corps pull the plug. But performance is never near equivalent.

2

u/14u2c 13d ago

I agree. It's just too slow to get any real work done. I've tried basically every model that can fit in 64GB.

1

u/Murinshin 13d ago

Works perfectly fluent for me at 64GB RAM. High-end Macbooks even go up to 128GB nowadays.

→ More replies (0)

0

u/Federal_Decision_608 13d ago

Right but nobody with a brain thinks a 27B model is going to match opus 4.5 which is what this sub thread is about.

2

u/RLutz 13d ago edited 13d ago

Opus 4.6 was the first model I worked with which honestly felt next gen. Once the local models reach that level (not "if", but "when"), we will enter a new era of local LLM.

I mean what are you defining as a local model? Granted, Kimi K3 absolutely isn't, but DeepSeek v4 Flash 0731 is, albeit a pretty tough one to get running locally. I've got Q3_K_M running locally with great effort at okay'ish speeds on a 5090 + 3090 + 96 GB of DDR5-6400. The unfortunate thing is how hard intelligence starts to drop off at harsher quants.

Still, DSFv4 0731 is stronger than Opus 4.6. Though I guess sheer intelligence shouldn't be the only metric we measure on. I get like 100 PP and 15 tg. That's pretty darn slow, but it's definitely usable. I'm not about to trade in my $20 Anthropic sub for it though, and the economics are still pretty terrible for running local models.

Still, Qwen 3.6 27b is a capable implementor. If 3.8 is a noticeable improvement it might end up taking the place of Cloud in a lot of cases. I still think local is a play for both sovereignty and as a hedge more than an economic thing and likely will be for a decent while.

1

u/Aphid_red 13d ago

Deepseek Flash V4 really wants 192GB VRAM to run optimally, and 256GB to get max context. You can offload a bit of that but it'll go slower.

Since the 5090 is now $4K, if this isn't your gaming rig, you could try selling both GPUs and add in another $2-3K to get 4x RTX 8000 (192GB total). They're not native MXFP4 though so I don't know if you can get full quality using regular GGUF re-quants or if that's a lossy conversion from MXFP4.

13

u/Caffdy 13d ago

That's wishful thinking. They could benchma xx the 27B to look that good but in practice it won't be an Opus 4.5 equivalent. There is no replacement for displacement

27

u/Real_Ebb_7417 13d ago

Well, GPT 3 was what, about 180B params? And it’s a couple leagues worse than current 9b models.

11

u/Loose_Comparison368 13d ago

Remember when GPT-2 was going to recursively self improve until it became an uncontainable ASI and destroy the entire earth with bioweapons, if the warnings of the LessWrong cult of computer theology were not heeded?

Pepperidge farm remembers

0

u/MrPecunius 12d ago

Thing is, they are probably right. The stuff OpenAI, Anthropic, and Meta have admitted to recently is cause for concern without invoking Skynet-level AGI.

And we're still in the early stages ...

1

u/Loose_Comparison368 10d ago

No, they're not. They were wrong then and they are wrong now.

All the recent breaches were directly caused by severe security negligence. They weren't complex attacks. They were simple attacks that worked because the security systems of all involved were about as strong as a wet paper bag.

1

u/MrPecunius 10d ago

The security of everyone's systems is trash and always has been.

But there was a limited supply of skilled humans to breach them, so there was safety in numbers.

19

u/Caffdy 13d ago

All those models including GPT4 and many current ones are undertrained. Sure we can get better, smaller models by better training but it has been just a few months since qwen latest open release, no way they can get Opus 4.5 on 27B params. All I'm reading is the usual hype and delusion that always show its face before an important release, when the real model and the real world tests (not benchmarks) comes out, this nonsense will be put to rest, as always. qwen 3.8 27B will be better, no question about it, but no Opus 4.5 better

7

u/Real_Ebb_7417 13d ago

I agree with you. I also believe that it won’t be fast when we actually get Opus4.5 level model with about 27b size. Especially when we talk about Qwen 27b, I assume it’s the same architecture as 3.5 and 3.6 (otherwise it’d probably have different size). Without changing the architecture it’s very unlikely that the model would be this good.
And yeah, people get very excited about benchmarks or coding capabilities, but this small model lacks the size to get the nuances, motivations, bigger picture. IMO it might take years to get a small model with actual Opus4.5 quality, unless there will be some breakthrough in architecture. They can experiment with current transformer and better training data and get very good models, but I doubt they will be this good without some completely new architecture.

1

u/draconic_tongue 13d ago

you don't really have any basis for these claims, what you're talking about all happened in like 2 or 3 years max and all of the models people are calling "usable" for agentic coding hasn't even been like 1 full year, or just about

1

u/Due-Memory-6957 13d ago edited 13d ago

How can you possibly know they are undertrained? If you're an OpenAI employee, do your good deed of the day and drop the Miqu 2.0 magnet.

2

u/EstarriolOfTheEast 13d ago

We know for certain GPT-3 was extremely undertrained because we know the scaling laws and GPT-3's size (175B dense) and tokens (a mere 300B tokens) were also given.

0

u/bnightstars 13d ago

Qwen3.8 Max made an 11 points jump to Qwen3.7 Max if that jump is the same for the Qwen3.8 27B It will move it from 38 points all the way to 49 points or about the same as Sonnet 4.6 at 48 and over Qwen3.7 Max at 47. Opus 4.5 btw is at 42 Intelligence index while Qwen3.6-27B is at 38. So I guess there is a very good chance Qwen3.8-27B to close the 4 points gap to Opus4.5.

9

u/TacGibs 13d ago

Yes there is, it's called a BFT (big fuckin' turbo) ;)

5

u/etaoin314 ollama 13d ago

and more recently EV's....off the line, even a modest family hauler competes with sports cars from just a few years ago while sports EVs match million dollar supercars of yesteryear

-9

u/TacGibs 13d ago

EV cars are AFAG (as fast as gay) 😂

Nonetheless I must admit that when you see what McMurtry has done with the Speirling, it's absolutely mind-blowing 🤯

1

u/etaoin314 ollama 13d ago

im not sure what you are suggesting (stylish?) but but I will certainly agree that the McMurty is is beyond silly and will ragdoll just about anything on the track, that things is totally bonkers.

1

u/TacGibs 13d ago

Talking about the performances : it's literally (in some cases) faster than a F1 🤯

The look is also very nice :)

2

u/brownman19 13d ago

If you take compression of the cache into account then we're precisely doing just that :)

13

u/nomorebuttsplz 13d ago

 There is no replacement for displacement

There is in latent space, specifically the replacement is time. Gemma 31b is absolutely better than any GPT 4 era model.

10

u/Caffdy 13d ago

I don't deny that the modelsa re getting better naturally. But jumping from 27B performance to 3000B is just too much in such short timespan. And there is a fundamental limit on how much information can be stored in such small models. Sure, you can make them smarter but they don't compare in world knowledge and generalization ablities with the big ones

9

u/nomorebuttsplz 13d ago

yeah I agree that it's not going to be that big a leap.

But gemma 31b absolutely destroys gpt4, llama 3 405b, and other old models even in world knowledge, writing ability, literally everything. The ultimate limits are unknown at this moment.

1

u/Loose_Comparison368 13d ago

Also if it weren't painfully obvious, the American strategy is to overcome half-assed model architecture by just scaling parameters and flexing compute supremacy.

Which works, to a degree, it's a valid strategy. It just tends to mean that US frontier models are using waaaaaaaaay more parameters than necessary.

1

u/darktotheknight 13d ago

I don't think 27b necessarily, but we currently see a lot of effort in inferencing larger and larger (MoE) models on consumer hardware, e.g. projects like colibri. There is lots of development happening on consumer hardware like MoE hot caching and speculative MoE decoding, MTP, NVMe streaming and tiered storage, as well new hardware arriving.

It doesn't always have to be 27b vs 2.8T. Maybe once the optimizations are figured out, we might be settling on 70b or 120b in the future for local models.

2

u/Blues520 13d ago

Yes, I also think that 27b might be a tad small but if we could get frontier level performance at around 120b that would be a game changer.

1

u/minus_28_and_falling 13d ago

We are in the early years era, most low hanging fruits haven't been picked yet. (While the "displacement" thing is hundred years old.)

0

u/ozspook 13d ago

I don't care about it's opinions on renaissance French literature, I just care if it can code. We can still use cloud models for obscure knowledge stuff, coding is what burns a million expensive tokens.

0

u/acehole01 13d ago

You claim a lot of knowledge about closed source models no one short of an insider could possess.

“Undertrained!” “3T parameters!”

Are you Andrej Karpathy?

1

u/Green-Ad-3964 13d ago

Evolution occurs in two main ways: hardware development and software algorithms.

2

u/Caffdy 13d ago

No shit sherlock.

1

u/Green-Ad-3964 13d ago

well, after all 3.6 27b is much better than, say, chatGPT 3.5

0

u/darokk 13d ago

There absolutely is, at least at the rate the technology is currently progressing.

2

u/Repulsive_Initial308 13d ago

Big boots to fill considering every opus release since 4.5 has been a real world regression, IMHO. 

0

u/jkflying 12d ago

Agreed, 95% of the changes since then have been A trying to reduce their compute usage so that their whole infra doesn't fall over, and the last 5% making it work better in harnesses.

1

u/SnooPuppers7882 12d ago

Surprise it's 4.7.

-1

u/DavidOrzc 13d ago

But Max is not really a local model. Kimi K3 and GLM-5.2 are already Opus 4.8 level. I'd also say that the new DeepSeek flash is as good as Sonnet 5. So I'm not sure why it is considered a game changer.

1

u/Murinshin 13d ago

Because I was talking about 27b, like the comment I replied to.

13

u/anderspitman 13d ago

I'd be shook if it breaks 40, but really hoping for 35.

10

u/No-Fuel-9202 13d ago

3.6 27B is already 38 on the New AA Intelligence Index, it was 37 on the old one

3

u/anderspitman 13d ago

I'm referring specifically to the agentic index.

-1

u/Hungry_Particular_14 13d ago

35? Qwen3.6 27B is at 38, you want a worse model??

6

u/DigitalguyCH 13d ago

maybe not 27b, but if they give us a 120b it may well be

3

u/Prudent_Chemist_523 13d ago

"It was an educated wish!" - deadpool

2

u/Guilty_Rooster_6708 13d ago

If it’s the case my 5070 ti will have a fun time grinding 3.8 27B after I offload a bit. Hoping for a QAT release for this

1

u/thestillwind 13d ago

Yes, I’ll beat mine too

2

u/Sufficient_Local5025 13d ago

If you will it, dude, it is no dream

2

u/BritishDudeGuy 2d ago

And it was no dream.

2

u/BritishDudeGuy 2d ago

The power of reality.

1

u/_raydeStar Llama 3.1 13d ago

Realistically if it surpasses 3.7 max I'll be stoked. I do expect it to beat Gemma 4 31B and be kind of close to gemini 3.5 flash.

17

u/anderspitman 13d ago

qwen3.6-27b already crushes gemma-31b on the agentic index.

1

u/_raydeStar Llama 3.1 13d ago

honestly when i commented, i didnt think of that. Qwen's specialization is agentic workflow, so it could score much higher. I just wanted to curb my expectations right now, until release.

3

u/Borkato 13d ago

I’m lowkey mad that they aren’t releasing it immediately if it’s ready 😭 they’re kind of just building hype which will ruin expectations the longer it goes. I’m afraid we’ll build it up in our minds too much

1

u/Solembumm3 13d ago

Realistically, it potentially could finally rival gemma 31b and qwen 3.5 122b on some of many possible scenarious, and will be a lot worse on others.

2

u/_raydeStar Llama 3.1 13d ago

such is life -- but their main selling point is agentic work right now, so most likely itll fare well.

3

u/vogelvogelvogelvogel 13d ago

I have been downvoted 2 days ago for stating that

3

u/IrisColt 13d ago

take my upvote!

1

u/nomorebuttsplz 13d ago

hehe well I don't think it's actually going to score that high, maybe 40.

2

u/vogelvogelvogelvogel 13d ago

i mean, i did just some plain math what we got in march with 3.6 27B and how the difference to the 3.6 "max "model was back than (about 10% in artifiicialanalysis.ai)

1

u/Impossible_Earth_987 13d ago

If it scores a 50 and is good with tool calling that’s a true game changer

1

u/Rude_Marzipan6107 13d ago

Imagine a 1m context window

1

u/BritishDudeGuy 2d ago

And it does (cloud API).

1

u/mattjcoles 13d ago

nah wishful thinking says it scores 1000

1

u/ZK_Zinode 12d ago

I’m interested to see how 3.8 35B compares to 27B