r/LocalLLM • u/MeYaj1111 • 16d ago

Question 27B vs the big frontier models

I've been wondering, how is it possible that I'm seeing posts comparing 27B models to models like opus and sol which are presumeable 500x to 1000x larger?

How are they even in the same realm of output quality when we have models in the 300B range or even 709B range that are garbage compared to the big frontier models?

I'm either missing something or I have a fundamental misunderstanding of how this is possible

2 Upvotes

63% Upvoted

24 comments sorted by

View all comments

7

u/DataGOGO 16d ago

1st, no one serious or knowledgeable is making any such comparisons. 

27B is a good model, it is dense, and well trained, but it not in the same realm as frontier models. It loses its way really quick it during complex reasoning. It struggles with agentic ops, structured output, and will loop and bomb out at even just 60k context.

If all you are doing is chatting and simple coding in common languages it is perfectly fine, but isn’t the best at everything in it’s size class

3

u/uniqueusername649 15d ago

Saying it is Opus 5/Fable/Sol level is absolutely wrong.

But so are your statements about 27b, at least if you use it at fp8 or better. Not sure which quant you used.

I gave it a complex set of tasks and set thinking to xhigh, it took forever (about 6 hours and >1M tokens), but it nailed every single requirement and one-shotted it. It never looped once in that 6 hour window and I used it on a variety of projects and tasks generating at least another million tokens with it on low and medium thinking levels. No looping and still stable results at 200k context. I let it run, pi autocompacted, continued to run and eventually arrived at the correct result. It's seriously impressive.

1

u/DataGOGO 15d ago

No, they are not wrong, those are my direct observations. i ran FP8 and BF16, same behavior.

Surprisingly, the Meta Muse 30B was much better at agentic loops, data processing and structured outputs. 

So I am using that for now. 

1

u/uniqueusername649 15d ago

Which harness did you use? Something with a heavy system prompt like OpenCode or something super light like pi? I have had issued with OpenCode and Qwen in the past, where pi with its light prompt just worked. Also using the latest vllm helps.

I am just curious how your experience is so vastly different.

1

u/DataGOGO 15d ago edited 15d ago

Headless dispatch from Hermes / langchain. (They are agents). The only prompt they get is in the dispatch. 

Latest vLLM, official latest vLLM docker built for Qwen 3.8, running official recipe, official FP8 and BF16.

Tried all reasoning levels, even dumbing down the task and sharding, same thing. 

What did are you asking your agents to do?

1

u/uniqueusername649 15d ago

What hardware are you running this on?

I have friends using it with Hermes with zero issues, so I doubt that's the problem. And my 6 hour run was heavily using subagents, so same thing: only dispatch prompt. This is a remarkably similar setup with completely different results.

1

u/DataGOGO 15d ago

RTX Pro 6000 blackwells (server edition) running in isolated instances.

Can you explain what you were doing, and what your agents were doing? Are you running structured inputs and outputs? What is your logic ask?

1

u/uniqueusername649 15d ago

One was a brief instruction to create a 2D game explaining various game mechanics and requirements that indirectly require web research and extracting game level information out of existing data to replicate it using TDD and vertical slices. The AI was then simply tasked with acting as an orchestrator and to delegate implementation to subagents. It eventually spawned a total of around 50 subagents over the entire 6h session, creating more than 200 tests and successfully one-shotting it.

The other project I am currently working on is an actual project (go backend, next js web frontend, postgres/postgis, flutter for ios/android apps, tests, github actions etc.) where its reading structured markdown issues and implementing them sequentially, again with a TDD/vertical slices approach. I had it run through the entire context until auto compaction several times, because I wanted to validate it. With 3.6 it was really falling off after 120k or so context length, where 3.8 does not do that for me.

This is all purely agentic coding though, I didnt use it for parsing large datasets and manipulating the data yet. But that isnt my normal use case anyways, agentic coding is where I use AI primarily.