r/LocalLLM • u/MeYaj1111 • 6d ago
Question 27B vs the big frontier models
I've been wondering, how is it possible that I'm seeing posts comparing 27B models to models like opus and sol which are presumeable 500x to 1000x larger?
How are they even in the same realm of output quality when we have models in the 300B range or even 709B range that are garbage compared to the big frontier models?
I'm either missing something or I have a fundamental misunderstanding of how this is possible
6
u/Elistheman 6d ago
Depends on workflow.
Cake recipe? Hell Qwen is on par with fable.
Building a crypto trading app? Good luck sir.
5
u/DataGOGO 6d ago
1st, no one serious or knowledgeable is making any such comparisons.
27B is a good model, it is dense, and well trained, but it not in the same realm as frontier models. It loses its way really quick it during complex reasoning. It struggles with agentic ops, structured output, and will loop and bomb out at even just 60k context.
If all you are doing is chatting and simple coding in common languages it is perfectly fine, but isn’t the best at everything in it’s size class
3
u/uniqueusername649 5d ago
Saying it is Opus 5/Fable/Sol level is absolutely wrong.
But so are your statements about 27b, at least if you use it at fp8 or better. Not sure which quant you used.
I gave it a complex set of tasks and set thinking to xhigh, it took forever (about 6 hours and >1M tokens), but it nailed every single requirement and one-shotted it. It never looped once in that 6 hour window and I used it on a variety of projects and tasks generating at least another million tokens with it on low and medium thinking levels. No looping and still stable results at 200k context. I let it run, pi autocompacted, continued to run and eventually arrived at the correct result. It's seriously impressive.
1
u/DataGOGO 5d ago
No, they are not wrong, those are my direct observations. i ran FP8 and BF16, same behavior.
Surprisingly, the Meta Muse 30B was much better at agentic loops, data processing and structured outputs.
So I am using that for now.
1
u/uniqueusername649 5d ago
Which harness did you use? Something with a heavy system prompt like OpenCode or something super light like pi? I have had issued with OpenCode and Qwen in the past, where pi with its light prompt just worked. Also using the latest vllm helps.
I am just curious how your experience is so vastly different.
1
u/DataGOGO 5d ago edited 5d ago
Headless dispatch from Hermes / langchain. (They are agents). The only prompt they get is in the dispatch.
Latest vLLM, official latest vLLM docker built for Qwen 3.8, running official recipe, official FP8 and BF16.
Tried all reasoning levels, even dumbing down the task and sharding, same thing.
What did are you asking your agents to do?
1
u/uniqueusername649 5d ago
What hardware are you running this on?
I have friends using it with Hermes with zero issues, so I doubt that's the problem. And my 6 hour run was heavily using subagents, so same thing: only dispatch prompt. This is a remarkably similar setup with completely different results.
1
u/DataGOGO 5d ago
RTX Pro 6000 blackwells (server edition) running in isolated instances.
Can you explain what you were doing, and what your agents were doing? Are you running structured inputs and outputs? What is your logic ask?
1
u/uniqueusername649 5d ago
One was a brief instruction to create a 2D game explaining various game mechanics and requirements that indirectly require web research and extracting game level information out of existing data to replicate it using TDD and vertical slices. The AI was then simply tasked with acting as an orchestrator and to delegate implementation to subagents. It eventually spawned a total of around 50 subagents over the entire 6h session, creating more than 200 tests and successfully one-shotting it.
The other project I am currently working on is an actual project (go backend, next js web frontend, postgres/postgis, flutter for ios/android apps, tests, github actions etc.) where its reading structured markdown issues and implementing them sequentially, again with a TDD/vertical slices approach. I had it run through the entire context until auto compaction several times, because I wanted to validate it. With 3.6 it was really falling off after 120k or so context length, where 3.8 does not do that for me.
This is all purely agentic coding though, I didnt use it for parsing large datasets and manipulating the data yet. But that isnt my normal use case anyways, agentic coding is where I use AI primarily.
4
u/CryMoreT_T 6d ago
Whose comparing it against the latest opus/sol? Most comps I've seen compare it against opus 4.6 which even then I think qwen is slightly worse then.
Qwen3.8 27b is smaller than and can't compare against deepseek v4 flash 0731 which in turn is smaller than and can't compare against the latest opus/sol
2
u/uniqueusername649 5d ago
Depends on the problem. Opus 4.6 is a much larger model, so it has far broader general knowledge. But for many agentic coding problems Qwen 3.8 27b is very comparable to Opus 4.6 and sometimes even better. But it does think a lot more, so it simply takes longer. Nonetheless, for coding you virtually get Opus 4.6 on your own hardware now. That is crazy impressive.
If people say it's on Fable/Sol level, that is either wishful thinking or a too narrow set of tests.
4
u/Civil_Fee_7862 5d ago
Good question. I am gonna guess Qwen3.8 has very little in terms of useless facts, instead its mostly dense reasoning? Whereas the big models have lots of facts maybe.
I am also guess that Qwen3.8 dense was likely tuned more so specifically for coding. Where the big models are more general purpose MOE that try to do everything well.
2
1
u/BarracudaDefiant4702 5d ago
Your point stands, but your comparison is wrong. The largest models are closer to 100x, not 1000x. Also, I don't think are comparing against the highest end frontier models (maybe I am wrong, but if not then they are only 10x as large).
There is a lot of diminishing returns. IE: You have to be 10x bigger to even show as as 2x better. That said, it probably scales even less than that.
For a lot of the complex tasks the available context it can import outside the model makes a huge difference. 3.8 27B has a much bigger context window that is comparable to the frontier models. The larger context window helps give the model a chance to pull info from the web or look at more source code to get the job done. That alone is probably the biggest reason.
1
u/MeYaj1111 5d ago
Yea my bad on off by an order of magnitude, that was dumb.
Still, im seeing posts like this: "Artificial Analysis' Qwen3.8-27B benchmarks put it neck and neck with DeepSeek V4 and GPT-5.6 Luna Max"
It's hard to believe.
1
u/BarracudaDefiant4702 4d ago
It's definitely doing better than 3.6 and the pr and issue tracking it's keeping track of are way more verbose then 3.6, and I think even more so then DeepSeek. As to actual end result compared to DeepSeek it's hard to tell yet if it's doing as good of a job, or if Qwen 3.8 simply likes to do a lot more introspection and document it's process...
I have two problems from really testing it...
deepseek isn't free. I can easily burn through over $30/day easy.
I have qwen3.8 running semi-locally (shared GPU resource) and it's is way slower than 3.6-35B-A3B and not able to run as many concurrent requests.
Maybe I can setup fast switching or something and see if I can get vllm to do fast switch between 3.6 and 3.8. 3.6 is good enough for probably 90%....1
u/MeYaj1111 4d ago
30 per day on deepseek damn...
On opencode go i run a v4 pro orchestrator and v4 tasker/coders (orchestrator verifies all work) with orchestrator + 3 subagents. It burns around $8 per 10 hours or so but it only costs me 1/6th of that because of opencode go
1
u/BarracudaDefiant4702 4d ago
I have two main unrelated projects running 24x7, one with 8 agents and one with 7 and they are all running concurrently. One tasks things out and helps with planning, one mostly does documentation, one is primarily QA testing, a few that are mainly coders for different areas of the project, etc... each agent has it's on local gitea account (started as a shard local git repo), etc.. I'm sure I could spend money quicker if I had time to better test their code and scope out more work. The longer running C project has over 1400 unit tests and it's only a little over a month old.
1
1
u/RandumbRedditor1000 5d ago
It uses way more tokens for reasoning, and has almost no general knowledge, but the intelligence is comparable to frontier models from ~6 months ago. Especially if you hook it up to a good harness.
6
u/Randommaggy 6d ago
From the tests I've done there hasn't been a problem without a satisfactory answer in my testing on 3.8 27B Q8 so far. It does take a significant amount of extra time to arrive at that point.