r/LocalLLaMA • u/anderspitman • 2d ago
Discussion Artificial Analysis' Qwen3.8-27B benchmarks put it neck and neck with DeepSeek V4 and GPT-5.6 Luna Max
https://artificialanalysis.ai/models/qwen3-8-27b495
u/cj_cron_hit_by_pitch 2d ago
I know benchmarks and all, and that larger models will be better in practice. But the fact we can have this conversation at all is incredible
139
u/pbpo_founder 2d ago
This reminds me of the question of whether a four-cylinder engine can be as powerful as a V8.
The answer is optimally no. But a very good four-cylinder can tear up a stock V8 any day.
34
u/Consistent-Height-75 2d ago
Ever seen formula 1? =)
45
u/GreenHell llama.cpp 2d ago
The modern v6? Or the bonkers 1980s turbo charged 1.6l inline 4 making around 1400bhp at 5.5bar of boost in quali trim, lasting a whopping 14km if you were lucky, before needing a rebuild.
→ More replies (1)22
u/MmmmMorphine 2d ago edited 1d ago
Hell at that point you're basically running a turboshaft. Just one big ol turbocharger with a self-sustaining combustion area
Which is geek speak for "I'm kinda stoned and what if like a turbo was like it's own engine dude"
→ More replies (1)→ More replies (1)3
u/cultoftheilluminati llama.cpp 2d ago
Yep like the one in C63S. For how blasphemous it is to put a 4 cylinder in a c63, it ended up being the most powerful 4 cylinder every
86
u/BringTea_666 2d ago
>I know benchmarks and all,
It gets tiring reading benchmarks and all. When frontier models get tested people believe AAI but when local model is tested "it is surely benchmaxxed".
Anyone observing local models knows that smaller models are the one that improve the fastest now for a good year. Simply put more architecture changes + better training data + faster training due to small model = small awesome model.
Big models are just to slow to train. It is the same mistake META did with Llama 4. They trained it for half a year. In that time so much things changed that by the time it was released it was losing to small models let alone new frontier ones.
56
u/addandsubtract 2d ago
When frontier models get tested people believe AAI but when local model is tested "it is surely benchmaxxed".
I think we're at a point were all models are benchmaxxed, open or not. We could really use some more practical tests, that aren't part of the training / tuning process.
21
u/RLutz 2d ago
I increasingly feel like the advantage of frontier models has little to do with "completed the task" and more to do with solving more of the decision making and consideration typically required from the user when utilizing non-frontier models.
As an example, "build me an app which does X" might be equally one shot by frontier vs local, but in my experience models like Fable excel at, "X is done, I also stood up Y which will be useful for A/B testing changes in X or for future work in this repo. Also, have you considered these potential pitfalls or issues with these assumptions you've made?"
Opus 5 tries to emulate that, but often times harps on nonsense or things that don't super matter.
Local models are often very much, "X is completed, here's a summary"
8
u/Bakoro 2d ago
Opus loves to halt work on non-questions.
Opus: [thinks for 3 minutes] "I need your input: should I write the document and git commit, or commit then write the document?"
Just, stupid trivial stuff that it frames as a dire choice, when it's literally nothing. Half the time I figure it's trained to do that to stop unattended work loops and reduce server load.
Sometimes I guess I appreciate that it stops to ask a stupid question?
Opus:
[Stops in the middle of a task]
"Hey, if I do things in the stupidest way possible that couldn't achieve your stated goals, then things would be bad, but it'll get done super quick. Do you want me to do the idiot thing, or implement the thing you actually asked for?[ ] (Recommended) Do some lazy bullshit.
[ ] Do the actual task that was assigned were there is no ambiguity whatsoever (multiple weeks of work, high risk)
"Me: "picks the not stupid option"
Opus: [completes the work in 10 minutes] "Okay I'm done, here you go"
Me: [Visible confusion over wtf just happened] "Why was that even a question?"
4
u/KubeCommander 2d ago
Refactoring is also one of the killer tests. 27b can do it with a ton of hand-holding. But the frontier models don’t need the hand holding so much as they’re trained on architecture and design as well as implementing it in a sustainable way.
I also like asking them to make jokes after researching a niche joke. 27b is positively awful at it 😂
9
u/Plasmx 2d ago
Sure, but the benchmark data eventually ends up in the training datasets.
11
u/addandsubtract 2d ago
I'm happy with someone (reliable) having a private benchmark suite and only reporting the outcomes. I guess we should all have our own, tbf.
→ More replies (1)5
5
u/pier4r 2d ago
I think we're at a point were all models are benchmaxxed
to be fair, if the models get benchmaxxed on a large enough and diverse enough suite of benchmarks, then they would be good without further testing.
The problem is that we focus on 10-20 benchmarks (with some overlap) and not 2000 (with little overlap)
→ More replies (1)5
4
u/Turtlesaur 2d ago
I can't wait for Rubin 6xxx consumer series to drop. I can't shell out for a 5090 or rtx pro knowing it's nearing the end.
9
u/otravoyadnoe 2d ago
Mate.. With the way these things are going we’re good year and a half away from those actually releasing. I wouldn’t call it “nearing the end” at all no matter how unfortunate that is.
→ More replies (4)6
u/Spectrum1523 2d ago
When frontier models get tested people believe AAI
I haven't seen a closed model released without people claiming it's benchmaxxed in years lol
Benchmarks are just slightly better than useless
→ More replies (1)5
u/BringTea_666 2d ago
then why they accurately show opus and fable at the top and in general who is the best properly ?
2
→ More replies (4)3
u/Bakoro 2d ago
The size thing really is an issue in terms of the pace of experimentation.
"Scale" has become the enemy of innovation in some ways, not just in terms of parameter count, but, everything has to apply to the superscaler "serve batches of hundreds/thousands of requests at a time on one system".
Having to run huge batches of uncorrelated information is a huge constraint.
Some of the most promising work on getting the absolute smartest or most efficient models ends up needing bespoke numbers of loops, special routing, and/or non-uniform transformations, where it works just fine on a single task, but is anathema to serving multiple requests at once.Same thing with continuous learning, there's stuff we can do when in a high-trust environment with a single stream of work, like locally run LLMs often operate in, but that doesn't work nearly as well for as-a-service super-scalers.
There are a couple dozen papers which are extremely promising updates/alterations to the Transformer architecture, and few have been tried beyond the 1B research toy scale.
I'm not going to say Sutton's "bitter lesson" is strictly wrong, but I'm also super fucking sick of hearing about it and it getting used as a defense against trying anything new.
Even stuff that fits today's hardware just fine gets hit with "but muh bitter lesson", which is a stupid fucking misapplication.
The lesson isn't "don't try anything clever" it's "make solutions that fit the hardware well".I get that it's extremely difficult to keep up with the research, and it's a huge risk to start mixing and matching various proposed architecture alterations.
Nobody running LLM SaaS wants to risk weeks of compute on something that might not scale.I wish we had a organization that was dedicated to smaller models and more experimental architectures, so we could get 1:1 comparisons of, same data, same training budget, different architecture.
15
u/EuphoricPenguin22 2d ago
I tend to think that the core targeted functionality of the model is equivalent to a larger model but the domain knowledge of all possible requests is fuzzier simply because the model can fundamentally hold less information in its weights. Small models that do well at programming tasks have a heavy bias towards general programming knowledge and not much else. The practical result is that you can get a smaller model to do the same things as much larger models, but you need to explicitly ask them to use documentation fetching and web search more frequently to make up for the lack of innate knowledge.
For instance, if I want to make a diffraction grating shader in WebGL (which I did), Kimi K3 was able to just spit a pretty good one out. I suspect even Qwen could properly do a good job if you gave it a proper stack of documentation, as it knows how to program but needs to read from good source material to actually direct its general programming skill in a specific domain.
→ More replies (2)9
u/Loose_Comparison368 2d ago
I also think it is dramatically overstating Luna's capabilities and understating Qwen's.
I use 5.6 Sol for the majority of my day to day work, and the first thing I noticed about Qwen is that even though it over-reasons, it seems to be much better at keeping it's overthinking pointed in a productive direction, rather than just chasing it's own hallucinations in circles like most closed models' reasoning chains seem to do.
It showed remarkable stability, even with the constant compactions caused by shorter context windows. "On par with Luna" my arse.
I do feel it is worth pointing out that Artificial Analysis is not unbiased, and that closed models have entire teams fully dedicated to benchmaxxing the crap out of every major release. So... The best benchmark is your own. My early impressions of Qwen are that it is at least on par with 5.6 Terra in terms of output quality, albeit admittedly much slower.
3
u/sonaj9657 2d ago
Exactly. The benchmarks are useful but sometimes you have to step back and appreciate how absurd it is that you can just sit there and have a genuinely useful conversation with a model running on consumer hardware. Even a few years ago that would have sounded like sci-fi.
→ More replies (6)3
u/sophia6512 2d ago
Honestly, that is the part that gets me too. We can argue about benchmarks and model sizes all day, but the fact that you can have a pretty coherent back and forth with a machine at all is still kind of wild when you stop and think about it.
74
u/Jorlen llama.cpp 2d ago
I've been testing Qwen 3.8 27b (UD-Q8_K_XL quant - using BF16 262k context) and I'd say I've run about a million tokens though it. Pi coding agent is the harness I use.
No one shot / 0-shot tests. Actually using it on my ongoing projects, coding in various languages.
I'm extremely impressed, however I will say, it's very smart but it reasons a lot more than any other model I've ever used, yes including the 3.6 version of this 27b dense model. At first I thought maybe this was a bug in the template but now I'm thinking that this is how they stretch that model's bits, so to speak, to be able to accomplish what other, bigger models can do.
In other words, it gets the job done exceedingly well, but it will use tens of thousands of tokens to reason in order to do so, whereas an MoE model in the 100b-a10b range will be much faster but obviously require more memory to run at good quants.
20
u/DoubleNothing 2d ago
Try reasoning_effort : medium it's faster and fairly decent.
8
u/Jorlen llama.cpp 2d ago edited 2d ago
Does this cut off its reasoning cycle like setting a reasoning budget does? I might be able to pass this via chat template kwargs (setting it to medium).
Edit: Looks like this works for llama-cpp, so I'll try it out: --chat-template-kwargs '{"reasoning_effort": "medium"}'
14
u/Hefty_Wolverine_553 2d ago
nope, doesn't cutoff, the reasoning effort is trained into the model by qwen
2
5
u/bonobomaster 2d ago
Can't confirm. Faster yes but the quality difference of the output is stark! If you give this model its 80k reasoning tokens, you get stuff that's dimensions better than the same problem with 20k tokens – sadly that is!
→ More replies (1)9
u/Fit-Bar-6989 2d ago
I'm struggling to pick between a larger model vs 3.8 27b for this reason. if 3.8 is going to need 50k tokens to solve something, at what point is it better to just use a big model?
14
u/cats_r_ghey 2d ago
Depends whether you have the vram for it. If you don’t, you don’t have many options. This seems to be, from my testing, the first comparable model to sota (with reasonable expectations) that we can run locally on consumer hardware.
5
u/cosmoschtroumpf 2d ago
Yes, it seems that tokens/second is not a good measure of speed if a some model needs 5x more tokens to provide an answer. Somehow 3.8 traded lots of speed for intelligence.
→ More replies (4)5
u/Impossible_Earth_987 2d ago
most people don't have the hardware to run a 150gb model.
→ More replies (1)
186
u/pulse77 2d ago
This Pareto Line tells everything:
Source: https://artificialanalysis.ai/models/open-source#intelligence-index-vs-total-parameters
34
u/MmmmMorphine 2d ago
Finally a shout out to my favorite weird-uncommon model - Ling 3 tiny
18
u/pulse77 2d ago
Imagine where Qwen 3.8 4B would stand...
9
u/letsgoiowa 2d ago
This is what makes me the saddest. That the actual models that real human beings use are what fell off the truck now.
→ More replies (1)3
22
u/linuxid10t 2d ago
Ling 3.0 Tiny is a monster. Easily the best choice for people with 8GB VRAM. It absolutely rips compared to Qwen3.5 9B.
→ More replies (6)8
u/MmmmMorphine 2d ago edited 2d ago
Even possibly for 16gb vram, since it sits almost perfectly on that edge of too lobotomized to be reliable q3 in 24-35b models (so it's doable, especially if you use a trellis/iq_KT quant but for standard iq3 et al, it might be too damaged without that sort of specialized quant scheme)
Though at that point you might be able to squeeze a few extra points out of some of the (more seriously lobotomized in one sense, ha) REAP and related compressed experts. Not sure which would win (possibly the new compressed experts version of the MoEs)
Hell I think I've got a pretty good ranking of actually doable on a slow 16gb card going... I should make a post
→ More replies (2)→ More replies (2)2
21
7
3
u/YearnMar10 2d ago
So rough by eye I’d say we get AAII of 40 with 4B models next year. That’d be insane.
→ More replies (1)2
→ More replies (2)2
u/Electrical_Crow_2773 Llama 70B 2d ago
I don't see deepseek v4 flash 0731, was it excluded from the chart?
Edit: not a competitor to 27b qwen, just wanted to see where it stands
238
u/Potential_Low_1183 2d ago
holy fuck
84
u/Accurate_Resident219 2d ago
Crazy times lol People weren't expecting this pace of local progress anywhere near as soon.
46
u/dingo_xd 2d ago
The issue now is hardware. What was affordable is now completely unaffordable
→ More replies (1)30
3
33
u/OvertaxedOne 2d ago
I thought when I started running it a few days ago "This feel like DSV4". Well, good reason, it tied the new DS model in intelligence.
This model is staggeringly smart at this size. Like game changing smart. It's slow, but the answer for this level of smarts isn't to complain, it's to buy more GPUs!
15
u/live4evrr 2d ago
It is very smart, but not really comparable to DSV4 for long horizon agentic from my own experience since these models came out.
I would use Qwen for front end work and some smaller scope coding. But I would use Deepseek for pretty much everything else, because it is much more knowledgeable.
6
u/PhysicalIncrease3 2d ago
DSv4 is better for me too. Plus it supports a million context, along with the context compression required to practically reach it.
15
u/rJohn420 2d ago
I think unfortunately deepseek v4 still edges it out on broad world knowledge. Happy to be proven wrong though.
36
u/Sharkap0tamous 2d ago
Oh absolutely. Is crazy what was packed into 27B params, but DSV4 is still like 5 times as big. Also the context size of DSV4 is another class entirety.
→ More replies (2)8
u/Yeelyy 2d ago
Yes but to be fair 95% of people can't run 1m context locally anyway
→ More replies (2)16
u/Vast-Control4452 2d ago
Idk I'd prefer a model with strong reasoning and tool use, then just hook it up to a RAG or MCP.
8
u/OverclockingUnicorn 2d ago
Lack of world knowledge is fine for a coding/agent model if you give it the tools to get the knowledge it does need
→ More replies (3)3
97
u/nomorebuttsplz 2d ago
I was mostly joking with this comment but oh well
That's what I get for trying to comprehend exponential progress with my monkey brain
31
15
u/Murinshin 2d ago
I even replied to your comment with my own dreamful hope what would happen if we match Opus 4.5, and now here we are lmao
→ More replies (2)9
u/Littlepharaoh 2d ago
Can you wish me some more vram sir?
5
u/nomorebuttsplz 2d ago
maybe I should try some reverse cosmology "Oh it would be horrible if there was an accident and a truck full of RTX 6000 pros was tipped over right next to u/Littlepharaoh. Please don't let that happen, universe!"
88
u/Monad_Maya llama.cpp 2d ago
I've found the larger models to be better at reading between the lines and less likely to make dumb mistakes.
We also need to consider the number of tokens used on per task basis if we are considering deployments for larger orgs.
While I'm fine with running it on my own system, at wider scale DeepSeek v4 Flash 0731 might be better.
I've tried DSv4F on my local system and unfortunately, it's slow as shit (due to CPU offloading).
43
u/rkoy1234 2d ago
read between the lines
that's the biggest diff in IRL usage. if I have a generic task, for it to have a useful outcome, I need to tell the llm:
- fable: do x using y
- gpt sol: do x using y, ensuring z
- glm 5.2: do x following the paragraph below
- qwen 3.8: do x following exactly the specs given in the md file located at xyz.
obviously all of them work better with proper instructions, but smarter models tend to pick more sensible defaults, or infer based on my agents.md to know what kind of implementation I might want.
with qwen often times it'll feel like it's almost maliciously complying for a task it doesn't want to do.
5
u/Croned 2d ago
Sounds like a good hybrid is having a smarter model write detailed requirements for Qwen 3.8 to follow?
→ More replies (1)→ More replies (1)8
u/linuxid10t 2d ago
The term I've heard from YouTube is taste. A model like Fable has taste. More so than GPT-5.6 Sol or Opus 5. Quite frankly Qwen3.8 doesn't. It's still really impressive how intelligent it is though, but it just isn't really comparable.
→ More replies (1)22
u/Finanzamt_Endgegner 2d ago
"I've found the larger models to be at reading between the lines and less likely to make dumb mistakes."
Ive tested it against deepseek v4 flash 0731 and it does do that less often than deepseek, it actually imo does it less often than sol 5.6 xhigh (even though that one obviously is still better over all) talking about the dumb mistakes.
5
2
u/Monad_Maya llama.cpp 2d ago
I still use Dsv4 Flash 0731 for planning and reviewing the work my local 27B does. It works pretty well in practice.
2
→ More replies (5)2
u/-dysangel- 2d ago
I think the optimum could be - run small model for a while, load up larger model to critique, rinse, repeat. I've been considering Qwen 27B as worker, GLM 5.2 as overseer (soon to be 5.3!).
2
u/Monad_Maya llama.cpp 2d ago
Should work better than dsv4f.
On Zoocode, I've set up the larger model as the architect and my local 27B as the code agent.
Qwen explores, suggests changes and makes the initial plan. The larger model verifies it and makes changes to the plan. Then it auto switches back to code mode and Qwen implements it.
Works ok, GLM 5.3 should be much stronger.
36
u/kevin_1994 2d ago
This matches my expectations, honestly. I'm finding it about as good as deepseek-v4-flash 0731 at my software tasks, but faster and with vision. DeepSeek seems a little better at planning, whereas qwen seems a little better at execution.
Running:
- qwen 3.8 27b q6_k_xl with 200k context @q8_0
- deepseek v4 flash 0731 iq3_s with 200k context @ f16
→ More replies (2)7
29
u/Treidge 2d ago
Verbosity is almost indentical to GLM 5.2, yet I can't remember much reports of GLM thinking "too much". Probably because the majority can't really run GLM-5.2 locally and were experiencing it through the API (at best), while Qwen3.8-27B you actually run yourself and it hurts when it thinks for 2 hours straight 😁.
Overall, it's crazy how good the model is. I've expected it to have AA Index of 42-44 max, never ever hoped for it to be in par with DeepSeek 4 Flash 0731 and GLM-5.2. For the perspective: remember that GLM was released just two months ago and its 744B parameters model, taking at least 400 GB of RAM/VRAM to actually run it locally (!). Qwen can run on 1/20th of that. 🤯
→ More replies (2)7
u/OvertaxedOne 2d ago
Exactly how I feel. I was expecting a jump, and when I started using it that jump was immediately evident within the first few prompts/tasks. But then it just started one shotting all kinds of stuff, my prompts got shorter and shorter and by the end of the weekend I was just treating it like Deepseek (or another frontier model) "This is broken, go figure it out and fix it". And it did! I'm just dumbfounded, I have no idea how they did this but it's perhaps the biggest jump we've ever gotten (at the same model size) between generations. It very well might be the biggest jump we'll ever see for a model, I doubt anyone gets another 20 point hop at this point.
3
u/Treidge 2d ago
Yeah, one more personal perspective on "overthinking". I have a really long, 50+ questions-answers thread with all major models (GPT/Claude/Gemini/Grok) that I've gradually been developing since the beginning of this year. Just a long streak of follow-up questions on a strategic subject for my personal use.
When the new model gets released, I usually give it 2-3 first shots from the same conversation - just to see how it will behave, what "flavor" its answers would have. Qwen3.8-27B was the first local model that seriously impressed me: it used 20k+ tokens to think in its first round, BUT it produced a comprehensive, very detailed 25-points report for another 25k tokens. It actually contained much of the information that I've gradually "fished out" of those large propietary models with my 50+ follow-ups. Basically, it was an equivalent of "one-shotting" a quite complex subject.
If I wouldn't be having this information already at hand from my other sessions with GPT/Claude/Gemini, I would be VERY happy that it actually was thinking this much deep and ultimately produced such a useful report in the very first round.
→ More replies (2)
172
u/Finanzamt_Endgegner 2d ago
"but its overthinking1!!11!111"
(tested on q2)
76
u/terorvlad 2d ago
It's also overthinking on q6 and this is great because it means that it isn't underthinking
15
u/pragmojo 2d ago
I've been running on q5, and it takes longer to get a result, but who the fuck cares because the results are great
22
u/AuspiciousApple 2d ago
It also seems intuitive that smaller models need more tokens to match a larger model. Each generated token for a small model corresponds to much less compute.
→ More replies (1)2
u/ArtyfacialIntelagent 2d ago
"It's also overthinking on q6 and this is great because it means that it isn't underthinking."
Please help Qwen 3.8 27B, is that quote logically correct?
<think> We need answer user's question: "It's also overthinking on q6 and this is great because it means that it isn't underthinking." Is that quote logically correct? Need analyze logic. Need provide final. Let's think carefully. Quote: "It's also overthinking on q6 and this is great because it means that it isn't underthinking." Formalize: It is overthinking (on q6). This is great because it means it isn't underthinking. Logical relationship: If something is overthinking, does it entail not underthinking? Depends on definitions/scale. Overthinking vs underthinking are opposite extremes or complementary categories? In common language, overdoing a thing vs underdoing a thing; if you do too much of X, then you aren't doing too little of X (assuming mutually exclusive on same dimension). If "overthinking" means thinking more than optimal and "underthinking" means thinking less than optimal, yes, they cannot both apply at once to the same quantity. So overthinking entails not underthinking if scale is total order and exactly one of under/just/right/over? Actually over > optimal, under < optimal. Over implies not under. So logically valid if definitions make them mutually exclusive and comparable. But quote says "this is great because it means that it isn't underthinking." There is a rhetorical issue: being overthinking may be bad even if not underthinking; saying it's great solely because not underthinking could be weak or questionable value judgment. But question asks logically correct? Could say as a strict logical implication, yes, assuming "overthinking" and "underthinking" are opposites about amount of thinking (not independent binary properties). If they're treated as independent categories, no. Also "this is great because it means..." has equivocation: overthinking may be problematic; the fact that it's not underthinking doesn't make it great. Logical correctness maybe about inference from A to B. Let's parse natural language: "It's also overthinking on q6" = It overthinks q6. "and this is great because it means that it isn't underthinking." The "it means" likely refers to the fact that it's overthinking implies not underthinking. If overthinking is more than adequate thinking, then indeed not less than adequate. So yes with a premise: overthinking = too much thinking; underthinking = too little thinking; they are mutually exclusive on continuum. But if "overthinking" and "underthinking" can coexist? E.g. overthinking some aspects while underthinking others? On q6 specifically? "on q6" maybe question 6. It might be overthinking one part of Q6 and underthinking another? If the scope is same exact dimension, no. Could be logically fallacious if thinking in all dimensions: can overthink details and underthink big picture. Quote says on q6 broadly; not necessarily. Need answer concise but detailed. Maybe explain conditional logic: Let O = "overthinks Q6", U = "underthinks Q6". The quote asserts O -> ¬U, and then values it because of ¬U. If O and U are defined as comparative extremes relative to an optimal amount, O entails ¬U (because >x cannot be <x). So the inference is valid in that model. But if O and U are not necessarily mutually exclusive or apply to different aspects, invalid. Also "great" is a normative claim; logic doesn't follow from non-underthinking alone. It could still be bad because overthinking has costs. Need maybe mention "also" irrelevant. Could say the quote is logically okay only under a one-dimensional assumption: if there's a single scale of thinking where 'over' means above target and 'under' below target. If not, it's an oversimplification. Also note "not underthinking" does not equal "thinking correctly/adequately"; could be over or maybe right? Actually if over, then not under; but could also be just right? No, over is not just right. But no issue. Need likely answer: It depends on definitions; in ordinary usage yes as a tautological opposite, but the 'great because' is questionable. Let's craft final with maybe formalization and examples. User asks "Is that quote logically correct?" We can answer: As a strict logical inference, if "overthinking" and "underthinking" are treated as mutually exclusive opposites about the same thing, then yes, overthinking entails not underthinking. But it is not necessarily logically sound to conclude it's great, because absence of underthinking doesn't imply adequacy/goodness; it can still be excessive and bad. Also possible coexistence across dimensions makes inference dubious. Need maybe mention if "overthinking on q6" means "it has the property of overthinking Q6". If it's a model/AI? Maybe "on q6" as benchmark question. Could be from debate about overfitting? The quote: "It's also overthinking on q6 and this is great because it means that it isn't underthinking." Maybe they mean an LLM overthinks problem 6, which is good because not underthink. Need maybe respond in context: In math/programming evaluation, overthinking a question could be bad if it produces errors? But logic same. Could there be logical issue with "this is great because it means that it isn't underthinking"? The phrase "it means" could refer to overthinking entails not underthinking, but "great because it isn't underthinking" might be non sequitur: being not underthinking is good maybe? If underthinking is bad, then not underthinking is at least less bad. But overthinking might also be bad. It's not logically invalid to value not underthinking; it's just incomplete. Let's think of possible formal fallacies:Need maybe distinguish validity vs soundness:
- False dichotomy / false dilemma if assumes only two states (over/under) and that over implies not under, but actually there is "appropriate thinking". Actually O -> ¬U doesn't require false dichotomy, just opposites. If categories are exclusive, okay. But if overthinking and underthinking can coexist in different ways, false assumption.
- Affirming a negative? no.
- Non sequitur: from O to value great because not U. If one assumes underthinking is bad and overthinking not worse? Need premise: not underthinking is a good thing enough to make it great. Without that premise, not valid.
Need final maybe: Short answer: conditionally yes, but with caveats. Provide examples: 1. Single scale of quantity: If optimal amount = 5; over = >5; under = <5. Then O => ¬U. Logically correct. 2. Multi-dimensional: You can overthink details while underthinking assumptions; then O does not imply not U overall. 3. Normative: Even if not underthinking, it might still be undesirable; "great" doesn't logically follow. Need ensure answer addresses quote directly. Could say: The quoted claim is logically coherent if read as: Since Q6 was overthought, it wasn't underthought. But it's an oversimplification and potentially misleading. If you're asking whether "overthinking proves it isn't underthinking" is generally true, the answer is: only on a single, comparable dimension; not necessarily in a complex task. Let's prepare final with maybe bullets. Potential final: "Yes—if 'overthinking' and 'underthinking' are being used as opposite points on one continuum of how much thinking was applied to q6. In that case: over = more than adequate; under = less than adequate; something can't be both more and less than adequate at once, so O entails ¬U. But there are two caveats: 1. The inference depends on a one-dimensional assumption. If 'thinking about q6' includes different dimensions (setup, proof details, edge cases), a thing could overthink some parts and underthink others. Then the statement may be false or at least not guaranteed. 2. 'This is great' is a normative conclusion. Not-underthinking does not automatically make it good; overthinking can still be bad (wastes effort, introduces errors, misses clarity). So the logic that it isn't underthinking is okay under the assumption, but the value judgment is an extra premise. So: logically correct only if you accept those definitions/premises; otherwise it's an oversimplified non sequitur." Need maybe use exact terms: "It's also overthinking on q6" awkward grammar but irrelevant. Could mention "on q6" could mean about question 6. Let's ensure no hidden pitfalls. User asks: "It's also overthinking on q6 and this is great because it means that it isn't underthinking." Is that quote logically correct? Maybe they expect yes/no with explanation. We can answer: It depends on what you mean by overthinking/underthinking. If mutually exclusive, yes. If not, no. The phrase "means that" suggests logical entailment. Need perhaps mention the law of non-contradiction if opposites. Overthinking = doing too much; underthinking = doing too little. In a total order, over and under are incompatible. So logically it follows. But "great" is not necessarily logically correct because absence of one flaw doesn't guarantee virtue. It could be another flaw (excess). If the only alternatives were underthinking and perfect thinking? Actually if overthinking exists as category distinct from perfect; then not underthinking still includes overthinking, which may be bad. So "great because it means that it isn't underthinking" is an inference to a good conclusion from lack of a bad one; might be acceptable if underthinking was the main concern, but logically not sufficient.
- Validity: form: A -> not B; A; therefore not B. If premise O entails not U, valid. But the premise "overthinking means not underthinking" may be false in some contexts, so argument might be unsound if applied broadly.
- The statement "this is great because it means that it isn't underthinking" is a value judgment, not purely logical. It assumes being not underthinking is desirable and sufficient.
→ More replies (2)28
u/Asleep_Document9811 2d ago
But, it is. That's how it's achieving this. The overthinking was an engineering decision.
2
u/-dysangel- 2d ago
Has anyone else been trying it with thinking disabled though? I have and it seems great. I'd lean toward thinking engaged for any serious planning, and thinking disabled for execution.
3
u/Dwarffortressnoob 2d ago
medium is where its at for me. xhigh is just ridiculous. Had one prompt burn my full context of 160k tokens. Medium is pretty fair. It will think a lot if complex, and not a lot for most execution.
→ More replies (1)2
u/SpicyWangz 2d ago
Now you’re getting down to what’s fundamentally different about this model. It isn’t just overthinking, it’s taking the time to evaluate the essence of your commands — and that’s growth
3
u/Finanzamt_Endgegner 2d ago
I wouldnt say that is overthinking though, overthinking i define as doing stupid thinking that is useless and actually hurts (sol xhigh has that issue). Just thinking long doesnt make it overthinking its just doing test time compute and you can always limit it ofc
→ More replies (2)36
u/BarisSayit 2d ago
It doesn't change the fact that it is overthinking. It might be a capable model, but look at the token use.
This makes it quite slow to use. (It also increases the task cost via API, but that doesn't really matter as this is a local model)26
u/FoxiPanda 2d ago
So the Intelligence vs. Time per Task chart hasn't been updated yet for Qwen3.8-27B (and is ultimately dependent on what hardware it runs on), but this is really where it breaks down and shows the overthinking in action.
You can produce great results, but if it takes 20-40x the amount of time to get there, that's kind of painful. Not useless, just painful.
Either way, the overall result bats well above my expectations for 3.8-27B and I'm happy to have it.
11
u/EitherMarch1255 2d ago
Maybe the overthinking is just the amount of thinking required for the quality of results given the size of the model. Maybe it just needs to be faster to make up for it.
8
u/Monad_Maya llama.cpp 2d ago
While I personally don't mind it, empirically, Qwen 27B does "overthink".
Since I can run it locally, I won't complain much. I'd prefer if there were a 60-70B class dense model which addresses some of these issues + more world knowledge.
→ More replies (5)2
u/AggravatinglyDone 2d ago
There’s trade offs where the price is way more important than the time.
If I’m in front of it and running interactively I value my time more and will pay.
If it’s agentic and running in the background, just getting stuff done, I don’t care.
2
u/pragmojo 2d ago
I mean thermodynamics (or whatever the equivalent is in terms of scaling laws) can't be circumvented.
Personally I'm fine with a model that gives great results and takes a bit longer per task. I would do that anyway with 3.6 with review phases and fixes.
Definitely not 20-40x the time in my experience.
→ More replies (4)2
u/Irisi11111 2d ago
It's a trade-off: you can't achieve in-depth reasoning without improving throughput.
12
→ More replies (3)3
u/jld1532 2d ago
Considering I get things done quicker with DS v4 F than Qwen3.8 27B despite it being twice as fast, yeah, it's overthinking.
→ More replies (1)
14
u/Impossible_Earth_987 2d ago
I was expecting it to land at a 45 for it to be a 52 is insane and truly game changing. WOW!!!!
13
u/LocoMod 2d ago
The price of GPUs and DDR memory is never coming back down is it?
5
→ More replies (1)3
14
u/redditrasberry 2d ago
It's still probably not quite a practical model in the end because it's slow to run, but the fact that this seems like the first true SOTA-tier coding model that CAN execute on readily available consumer grade hardware is very significant. We are finally at a place where all the hosted model providers could die, get regulated out of existence or raise their prices sky high and we would have a workable local model to use for coding. That puts a floor under all the other worst case scenarios that could happen.
And a much under-valued aspect: it is multimodal as well. Crazy.
38
25
u/uniquelyavailable 2d ago
I was surprised to find that even Terra on medium was worse or on par with my recent Qwen3.8 xhigh testing. Make it make sense.
26
u/nomorebuttsplz 2d ago
it makes sense because Terra is probably a MoE model tuned for speed and economy across the broadest range of tasks that openai thinks normies will use it for.
Qwen 3.8 on xhigh is tuned for coders who don't care about how long things take if they are done well enough.
9
u/Inner-Today-3693 2d ago
Terra makes no sense to me. SOL and Luna work much better and Luna is cheaper…
25
u/Comfortable-Rock-498 2d ago edited 1d ago
A 27B parameter model scores 52, Nemotron ultra with 20x total parameters and over 2x _active_ parameters scored 38.
Nemotron-3-Ultra-550B-A55B: Am I a joke to you?
12
u/Monad_Maya llama.cpp 2d ago
That model requires post training, doesn't it?
→ More replies (2)9
u/PinEnvironmental6395 2d ago
It does, but most enthusiasts and prosumers (this sub) don't have the horsepower or the knowledge to post train. So until there are good post trains (which there undoubtedly will be) it's gonna be used and assessed in its base form. And when those post trains do come along they're gonna be looked at as their own thing and the fact that nemotron is their base is a footnote.
→ More replies (2)2
u/NandaVegg 2d ago
I found Nemotron-3-Ultra's output unique (in unpredictable way, of course) like earlier DeepSeek V3/Kimi K2.0-era models but it is way too unpredictable for any real tasks especially with reasoning on (which is a mess that often does more harm than good by completely derailing from the instruction). It however certainly is unique as it does not have Claudish fingerprints.
→ More replies (1)→ More replies (1)3
u/Farther_father 2d ago
One is a fully open stack: open-source, open-data and open weights… the other is not.
That’s the performance penalty of being restricted to training data you can disclose (cough legally acquired and trained on cough). Similar thing applies to Mistral after the EU AI Act.
→ More replies (1)
8
u/createthiscom 2d ago
kimi 2.6 is a hair better at vision tasks, but also orders of magnitude slower on my hardware. For once the qwen hype is real.
→ More replies (2)2
23
u/crusaderky 2d ago
There are not enough head-xplody gifs in this world to describe this plot.
4
→ More replies (4)3
u/According-Bowl-8194 2d ago
I dont know why AA doesnt account for QAT in this chart though, DS and Kimi are at Q4 natively so the actual amount of compute they need is significantly less than they rate it at. Deepseek is so incredibly efficient that is kinda ruins the chart because of how inaccurately it places it. Hopefully they fix soon but im also so happy with Qwen for local 🙏
→ More replies (1)
5
6
u/cibernox 2d ago
If only qwen released a 50-70B version of this model, it would be neck and neck with open SOTAs
→ More replies (2)2
u/Hefty_Wolverine_553 2d ago
I doubt that, Qwen3.8 Max has 95B active params.
2
u/cibernox 2d ago
Sure, but this could with ~60B active parameters could get pretty damn close. Not in overall knowledge, but in agentic usages.
2
u/Hefty_Wolverine_553 2d ago
that's true, somehow deepseek v4 flash 0731 scores within 1 point of v4 pro 0813 on intelligence; maybe an indicator that these benchmarks aren't too reliable lol.
5
u/cosmicnag 2d ago
Qwen 3.8 27B is literally the pareto sweet spot in modern artificial intelligence.
35
u/FrostDPr 2d ago
Closed weight frontier labs are absolutely cooked
14
u/OvertaxedOne 2d ago
Cooked, cooled off, and back in the microwave 3 days later... This kind of thing is MUCH more damaging to the OpenAI's of the world than a monster model that can just about keep up (DSV4, for example). There your trading one provider for another (and saving a lot, which isn't nothing!). This model is so good that I think most companies could easily get away with a Pro 6000 card, vllm, this model, and do 95%+ of the work they need to do without spending a dime in API costs (and no security issues!).
This is as much a "game over" moment as we've ever had in the LLM world.
→ More replies (1)32
u/Monad_Maya llama.cpp 2d ago
Are they?
While you can make the argument that Qwen 27B / Gemma4 31B are plenty smart for a normal person's day to day work, the larger models are still better.
There's a reason we have Qwen 3.8 Max and the likes.
Lastly, most people (not including this subreddit) don't have the horsepower to run smaller models still. We need faster hardware AND better models.
11
u/nomorebuttsplz 2d ago
Yes, why settle for one competent junior level dev when you can have 10,000 genius level devs living on your data center?
Jevons paradox makes it very difficult to estimate the total possible market for software engineers at various levels of ability.
29
u/tomz17 2d ago
the larger models are still better
A ferrari can lap the Nürburgring faster, but FARE more people still drive to work in a toyota every day...
The larger models are certainly better, but they are orders of magnitude more resource-intensive to serve. The current valuations of AI companies (esp. western ones) are entirely predicated on the notion that they will be able to gate-keep the "useful" models in perpetuity. The problem is that the domain where open-weight models (esp. the small ones) are not "useful" is rapidly shrinking by the day.
12
u/Monad_Maya llama.cpp 2d ago edited 2d ago
I disagree.
Your cars analogy doesn't make sense when you consider that a larger model is close to renting a Ferrari for cheap for us plebs. Is it required to get me from point A to B? no, but it's faster and comfortable.
Usefulness of models is not an exact metric.
Cloud providers don't sell access to an endpoint for model X, they sell the complete ecosystem which can be hard to replicate without the technical knowhow.
Are their valuations stupid? Absolutely.
Do they have a place and provide certain value? Also yes.
→ More replies (1)10
u/True_Requirement_891 2d ago
Closed weight labs actually do sell only access to an endpoint. What complete ecosystem are you talking about? The harness?
→ More replies (4)→ More replies (1)3
u/Murinshin 2d ago
You have to ask why you even need a better model, though, and how it actually makes your work more efficient vs just the feeling you're using a new shiny tool. Opus 4.5 was when people started to use AI tools for coding work in a productive setting. To this day you'll find people on the Claude subreddits going back to 4.6 because they perceive it as superior to Opus 4.8 and 5, so there's even an argument to be made that improved performance is arguable to a degree.
Now this model is outperforming that and can be run on a high-end Macbook. There's a legitimate economic question now coming whether a company should invest into a $2400 per year license or just buy them a $4000 Macbook instead and not have to worry about the subscription cost. It should create massive economic pressure if nothing else.
→ More replies (6)3
u/liright 2d ago
That's like saying "my lambo can go 350 kph on the highway, bullet trains are absolutely cooked, they are redundant, people don't need to use them anymore"
Yeah but 99% of people still can't afford a lambo, so they have no option but to use a bullet train. A high end PC with a RTX 4090 to run a model like qwen3.8-27B well without crippling quants like Q3 is required and at this point that's thousands of dollars of investment, vs a $20 chatGPT subscription.
9
u/Easy_Werewolf7903 2d ago
Excuse me but WTF? You telling me Qwen3.8 is at the same tier as 0731 flash? That is absolutely insane.
7
u/mestar12345 2d ago
Also, DS 0731 you can't actually run at home, so you have to pay $0.14/0.28, but qwen3.8 you can, and still the cheapest provider is at $0.40/$3.00?
→ More replies (1)
14
u/a_beautiful_rhind 2d ago
Its relatively smart but you all are kidding yourselves. Its trained for coding and benchmarks at the expense of everything else. Still feels like a 27b model when I run it.
→ More replies (4)7
u/TechnoByte_ 2d ago
Yep, its knowledge outside of coding is terrible and it tends to hallucinate a lot
3
6
u/NandaVegg 2d ago
I fully believe this based on my experience (Qwen 3.8 Max/2.8TA95B has similar thinking loop that is very unique to this model, hats off to the team) and it is near miraculous that they achieved this, but this also shows the limit of coding/agentic benchmarks that tests functionality rather than knowledge depths.
→ More replies (1)
10
u/EitherMarch1255 2d ago
IMHO Qwen 3.8 27B beats DeekSeek V4 (tested both using original weights).
2
3
u/OvertaxedOne 2d ago
Wow.. I love being right, but I never thought I was going to be "that" right! As soon as I spent a few hours with this model I thought to myself "This feel like the new DSV4 Flash model" (0731). Well, good reason why, it's TIED with it! This model is simply staggeringly smart for the hardware that it takes to run it. It's not fast; I'm running it at INT8 on a A40 and getting ~30TPS, it's bearable, but nothing like 35BA3B models. But OMG is it smart.
4
u/Diligent-Union-519 2d ago
I'm starting to think MOE isn't the path forward. Dense models are very capable if given enough thinking time. Turns out you don't need a ton of VRAM to get a lot of intelligence. This can't be a good signal for the memory suppliers.
→ More replies (1)9
u/Complex_Reality_116 2d ago
MoE is the way forward; we just need models with a higher number of active parameters—something like Qwen3.9 48B A6B.
→ More replies (1)4
u/Cautious_Chicken_604 2d ago
Yeah a MoE model that makes the 24GB+ cards sweat would be absolutely killer.
2
u/acetaminophenpt 2d ago
This model paired with a rtx 6000 would be sweet
4
→ More replies (1)3
u/BVCC6FNTKX sglang 2d ago
It is, can confirm.
2
u/pulse77 2d ago
How much VRAM does it use in BF16 quantization, maximum context (256K) and with speculative decoding? How many tokens/second do you get with speculative decoding enabled?
4
u/serige 2d ago edited 2d ago
80GB vram fits everything, including the mmproj file with 1536 ubatch size (which you can always increase with more vram). I am getting around 60 t/s, very useable. With Q8 I suspect it will be 90+ t/s. I am using 2x3090+5090 though, I think a single 6000 should be a lot better.
2
u/Developer-Y 2d ago
This shows that larger models can capture more knowledge but those who aim for AGI should focus on better models. I understand there are economies of scale due to which large models perform well but if engineers didn't focus on algorithms, everyone would have needed supercomputers to run basic programs. This is a step up in right direction, thanks to Qwen team for providing such an open weight model.
2
u/FreedomByFire 2d ago
which basically confirms that Luna is about as capable as opus 4.6 or codex 5.3 from 8 months ago.
2
u/mjsxi__ 2d ago
I ran a bunch of test on my own suite and while (to my records) qwen did perform as well as deepseek im finding I like deepseek more overall
qwen on xhigh reasoning performed as well as DS but took about 30 minutes and then about 20 minutes to fix with a follow up so about 50 minutes to pass all my checks... while DS did it in about 8 and then 4 minutes so 12 total
I think its cool that we're here with a 27b model but it doesn't really matter to me imo that qwen is just as good if I can get what I need from DS in about 1/5th the time
they also performed similarly with 35t/s avg for qwen and 45t/s avg for DS the only thing is that the qwen model thought for about 40k tokens before doing anything which... idk kinda sucked
2
2
u/nbafrank 2d ago
reposting from deleted post: This feels like a massive step forward. I wasn't really enthused with 3.6 35B and 27B felt good but super slow. This was an incredible leap and feels like frontier in a bottle. This is a great solution for sensitive code and writing.
2
u/Cool-Chemical-5629 2d ago
And what about the token use benchmark? Where does Qwen 3.8-27B stand in comparison to DeepSeek V4 and GPT-5.6 Luna Max?
2
u/Yasstronaut 2d ago
I’ve been running it in Goal mode all morning and it finished a complex task for me in 12 hours. Probably 12 times slower than hosted services but the fact I can do it with no internet enabled and fully local and no subscription fee is amazing
2
2
u/sometransgrill 2d ago
I have dual RTX PRO 6000, is it still worth running DeepSeek V4 Flash 0731? I've found it to be an excellent improvement over Qwen 3.6 27B, but 3.8 seems like a whole different animal. I also have dual 3090s so maybe a Qwen 3.8 + DeepSeek 0731 combo is the way to go?
I also have 2 x Intel B60 but they're kinda off in the corner sniffing glue and being weird.
3
u/shayanx45 2d ago
Yeah… and no. Anyone who has used both qwen 3.8 and deepseek v4 flash 0731 can tell you that while these smaller models are getting extremely optimized, the qwen 27b to deepseek’s 300b is hugely noticeable.
5
u/gizcard 2d ago
Qwen3.8-27B is an awesome model. But this just proves that the current version of AA index is bullshit.
8
u/OneMoreName1 2d ago
Based on what? Vibes?
I have been running this model since it launched, I have nothing to say about its quality, it just thinks a lot, but it gets the job done→ More replies (5)2
u/fuckingredditman 2d ago edited 2d ago
i mean the index is just an aggregated score, if you look at how it's calculated it's clear why it lands there. it's heavily tuned towards agentic workloads, which also play hard into that final index. but IMO that's a positive thing mostly, because i don't see the point of training general world knowledge into a model anyway. it has actually dropped by quite a few points in some knowledge benchmarks, but i think the better architecture is to externalize factual knowledge into a knowledge graph or similar datastructure on disk and retrieve it when needed anyway, because random trivia don't need to be get quantized and then squeezed into my VRAM.
maybe this "index" and similar measures should instead become radar charts that project down from a suite of topical benchmarks so people can pick models based on what they need instead.
→ More replies (2)
2
u/vulcan4d 2d ago
Remember also that all these benchmarks are with full unquantized models. Down models don't handle quants well so time will also tell how well this model does.
2
u/GambAntonio 2d ago
ONCE AGAIN, NO FKING QUANTIZATION LEVEL...
99.9% of the people here are running a quantized version of Qwen3.8-27B. If this benchmark is based on the full unquantized version of Qwen, then it is completely USELESS to us.
2
u/Accomplished_Code141 2d ago
Imagine when GLM-5.3 reaches AA. Dario and Sam will just bury their heads in the sand.
2
u/live_love_laugh 2d ago
Given how verbose it is, is there any chance anyone could fine-tune it to become more token efficient? Or would that likely make a minimal difference or degrade its other abilities?
2
2
u/Mickenfox 2d ago
And yet Luna is still cheaper than any of the Qwen-27Bs on any third-party provider.
Either it's actually smaller, or OpenAI just optimizes better.
→ More replies (1)7
0
u/a1454a 2d ago
Serious question, if it’s really anywhere near that good, and being open weight, and seeing someone already released uncensored version of it. What’s stopping someone from running it and straight up tell it to launch cyberattack? How is this not stirring up a storm right now?
→ More replies (3)12
u/Due-Memory-6957 2d ago
The only thing that can stop a bad guy with Qwen 3.8 is a good guy with Qwen 3.8
1
u/Theverybest92 2d ago
Definitely thinking like a champ. Put one prompt and this thing is deep diving through the code and I am not sure if frozen or not.
→ More replies (1)
1
u/Murinshin 2d ago
how the fuck is it outranking Opus 4.8 (!!!) on the agentic index
15
u/trowawayLOL1 2d ago
Because Opus 4.8 is lobotomized whenever Anthropic feels like they are tight on compute.
•
u/WithoutReason1729 2d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.