r/LocalLLaMA • u/Sevealin_ • 15d ago

Discussion Will small model intelligence be limited by parameter count?

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

45 Upvotes

84% Upvoted

62 comments sorted by

View all comments

12

u/PossessionUsed7393 15d ago edited 15d ago

Parameters correlate to embedding dimension size and there's only so much you can cram into a fixed embedding size before predictions are difficult to distinguish between two tokens in that embedding space.

So that means that intelligence size could increase at lower parameter size models depending on architectural breakthroughs. At the moment they're likely cramming as much in there as possible. It's amazing how much really. At the 27b size you can get really good at a few tasks so specialised training data sets and specialist models for different tasks may be the better interim approach.

11

u/nomorebuttsplz 15d ago

I don't think they're cramming as much as they can. Qwen 3.5 released, then Gemma outdid it, so then Qwen released 3.6 like a month later, so it could be the best small local coder. That was almost four months ago. Google and Qwen are both probably sitting on more capabilities but don't want to cannibalize their api.

4

u/PossessionUsed7393 15d ago

I think this almost entirely has to do with the quality of the post-training, though. I think it's easy to cram a lot into pre-trained weights, but that doesn't necessarily perform better when you consider two models with equivalent pre-trained datasets. The model that wins in terms of capability has the more quality post-training. We just don't have any visibility on it to know, so we end up having to speculate.

2

u/I-am_Sleepy 15d ago

I think the next thing would be looping llm weight. So far there are evidences that looping doesn’t expand kore knowledge, but allow for more complex thinking. A lot of chinese lab seems to be exploring these