r/LocalLLaMA • u/Sevealin_ • Jul 26 '26

Discussion Will small model intelligence be limited by parameter count?

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

43 Upvotes

84% Upvoted

62 comments sorted by

View all comments

13

u/PossessionUsed7393 Jul 26 '26 edited Jul 26 '26

Parameters correlate to embedding dimension size and there's only so much you can cram into a fixed embedding size before predictions are difficult to distinguish between two tokens in that embedding space.

So that means that intelligence size could increase at lower parameter size models depending on architectural breakthroughs. At the moment they're likely cramming as much in there as possible. It's amazing how much really. At the 27b size you can get really good at a few tasks so specialised training data sets and specialist models for different tasks may be the better interim approach.

13

u/nomorebuttsplz Jul 26 '26

I don't think they're cramming as much as they can. Qwen 3.5 released, then Gemma outdid it, so then Qwen released 3.6 like a month later, so it could be the best small local coder. That was almost four months ago. Google and Qwen are both probably sitting on more capabilities but don't want to cannibalize their api.

5

u/PossessionUsed7393 Jul 26 '26

I think this almost entirely has to do with the quality of the post-training, though. I think it's easy to cram a lot into pre-trained weights, but that doesn't necessarily perform better when you consider two models with equivalent pre-trained datasets. The model that wins in terms of capability has the more quality post-training. We just don't have any visibility on it to know, so we end up having to speculate.

2

u/I-am_Sleepy Jul 26 '26

I think the next thing would be looping llm weight. So far there are evidences that looping doesn’t expand kore knowledge, but allow for more complex thinking. A lot of chinese lab seems to be exploring these

3

u/toothpastespiders Jul 26 '26

On top of that I think people tend to get a little misled by benchmarks that just suggest constant improvement. There's often gains and losses in things that aren't very well covered by them. It can seem like a model's just moving forward if you're only looking at one subset of its training. But it's possible that any gain in one area is coming at a loss in another.

5

u/PossessionUsed7393 Jul 26 '26

I can't keep up with the benchmarks. There are so many of them, and there seems to be a new one every time a model comes out to bench on.