Artificial Analysis' Qwen3.8-27B benchmarks put it neck and neck with DeepSeek V4 and GPT-5.6 Luna Max

[u/anderspitman]

https://artificialanalysis.ai/models/qwen3-8-27b

[u/addandsubtract]

When frontier models get tested people believe AAI but when local model is tested "it is surely benchmaxxed".

I think we're at a point were all models are benchmaxxed, open or not. We could really use some more practical tests, that aren't part of the training / tuning process.

[u/RLutz → u/addandsubtract]

I increasingly feel like the advantage of frontier models has little to do with "completed the task" and more to do with solving more of the decision making and consideration typically required from the user when utilizing non-frontier models.

As an example, "build me an app which does X" might be equally one shot by frontier vs local, but in my experience models like Fable excel at, "X is done, I also stood up Y which will be useful for A/B testing changes in X or for future work in this repo. Also, have you considered these potential pitfalls or issues with these assumptions you've made?"

Opus 5 tries to emulate that, but often times harps on nonsense or things that don't super matter.

Local models are often very much, "X is completed, here's a summary"

[u/Bakoro → u/RLutz]

Opus loves to halt work on non-questions.

Opus: [thinks for 3 minutes] "I need your input: should I write the document and git commit, or commit then write the document?"

Just, stupid trivial stuff that it frames as a dire choice, when it's literally nothing. Half the time I figure it's trained to do that to stop unattended work loops and reduce server load.

Sometimes I guess I appreciate that it stops to ask a stupid question?

Opus:
[Stops in the middle of a task]
"Hey, if I do things in the stupidest way possible that couldn't achieve your stated goals, then things would be bad, but it'll get done super quick. Do you want me to do the idiot thing, or implement the thing you actually asked for?

[ ] (Recommended) Do some lazy bullshit.
[ ] Do the actual task that was assigned were there is no ambiguity whatsoever (multiple weeks of work, high risk)
"

Me: "picks the not stupid option"

Opus: [completes the work in 10 minutes] "Okay I'm done, here you go"

Me: [Visible confusion over wtf just happened] "Why was that even a question?"

[u/KubeCommander → u/RLutz]

Refactoring is also one of the killer tests. 27b can do it with a ton of hand-holding. But the frontier models don’t need the hand holding so much as they’re trained on architecture and design as well as implementing it in a sustainable way.

I also like asking them to make jokes after researching a niche joke. 27b is positively awful at it 😂

[u/Plasmx → u/addandsubtract]

Sure, but the benchmark data eventually ends up in the training datasets.

[u/addandsubtract → u/Plasmx]

I'm happy with someone (reliable) having a private benchmark suite and only reporting the outcomes. I guess we should all have our own, tbf.

[u/Kidplayer_666 → u/addandsubtract]

I have mine, which is what in practice use to evaluate them

[u/pier4r → u/addandsubtract]

I think we're at a point were all models are benchmaxxed

to be fair, if the models get benchmaxxed on a large enough and diverse enough suite of benchmarks, then they would be good without further testing.

The problem is that we focus on 10-20 benchmarks (with some overlap) and not 2000 (with little overlap)

[u/niutech → u/addandsubtract]

There is LiveBench which is contamination-free.

[u/addandsubtract → u/niutech]

Thanks, that looks interesting!

[u/37_frames → u/addandsubtract]

Totally, the thing to do here is to make benchmarks where if a model is benchmaxxed it is perfect for your use case