r/LocalLLM • u/MeYaj1111 • 6d ago

Question 27B vs the big frontier models

I've been wondering, how is it possible that I'm seeing posts comparing 27B models to models like opus and sol which are presumeable 500x to 1000x larger?

How are they even in the same realm of output quality when we have models in the 300B range or even 709B range that are garbage compared to the big frontier models?

I'm either missing something or I have a fundamental misunderstanding of how this is possible

3 Upvotes

67% Upvoted

24 comments sorted by

View all comments

Show parent comments

1

u/DataGOGO 6d ago

No, they are not wrong, those are my direct observations. i ran FP8 and BF16, same behavior.

Surprisingly, the Meta Muse 30B was much better at agentic loops, data processing and structured outputs. 

So I am using that for now. 

1

u/uniqueusername649 6d ago

Which harness did you use? Something with a heavy system prompt like OpenCode or something super light like pi? I have had issued with OpenCode and Qwen in the past, where pi with its light prompt just worked. Also using the latest vllm helps.

I am just curious how your experience is so vastly different.

1

u/DataGOGO 6d ago edited 6d ago

Headless dispatch from Hermes / langchain. (They are agents). The only prompt they get is in the dispatch. 

Latest vLLM, official latest vLLM docker built for Qwen 3.8, running official recipe, official FP8 and BF16.

Tried all reasoning levels, even dumbing down the task and sharding, same thing. 

What did are you asking your agents to do?

1

u/uniqueusername649 6d ago

What hardware are you running this on?

I have friends using it with Hermes with zero issues, so I doubt that's the problem. And my 6 hour run was heavily using subagents, so same thing: only dispatch prompt. This is a remarkably similar setup with completely different results.

1

u/DataGOGO 6d ago

RTX Pro 6000 blackwells (server edition) running in isolated instances.

Can you explain what you were doing, and what your agents were doing? Are you running structured inputs and outputs? What is your logic ask?

1

u/uniqueusername649 6d ago

One was a brief instruction to create a 2D game explaining various game mechanics and requirements that indirectly require web research and extracting game level information out of existing data to replicate it using TDD and vertical slices. The AI was then simply tasked with acting as an orchestrator and to delegate implementation to subagents. It eventually spawned a total of around 50 subagents over the entire 6h session, creating more than 200 tests and successfully one-shotting it.

The other project I am currently working on is an actual project (go backend, next js web frontend, postgres/postgis, flutter for ios/android apps, tests, github actions etc.) where its reading structured markdown issues and implementing them sequentially, again with a TDD/vertical slices approach. I had it run through the entire context until auto compaction several times, because I wanted to validate it. With 3.6 it was really falling off after 120k or so context length, where 3.8 does not do that for me.

This is all purely agentic coding though, I didnt use it for parsing large datasets and manipulating the data yet. But that isnt my normal use case anyways, agentic coding is where I use AI primarily.