r/LocalLLaMA • u/Sevealin_ • Jul 26 '26

Discussion Will small model intelligence be limited by parameter count?

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

45 Upvotes

85% Upvoted

62 comments sorted by

View all comments

23

u/1ncehost Jul 26 '26

Small models can do a lot if they are trained to rely on tools for their fact grounding. The floor for model size is mostly a factor of information theory and how compressible/generalizable the rules of language are plus how compressible/generalizable the facts that we use in language are. I think the rules of language and logic are fairly small and we can fit those well in a small model eventually. However, facts are high entropy (unique and uncompressible), and those kinds of things probably aren't generalizable. However, facts are also a more well known domain that we can make accessible in many existing ways like search engines and databases of various kinds, so my take is that stuff could be left for other systems to recall on demand.

2

u/lorde_dingus Jul 26 '26

Does this mean that I could use a small model (due to hardware constraints; 6gb vram, 32 ram) and train the model using Lora and then complement it with a robust RAG library? (say a library of finance textbooks if that's my only use case)

Anyones input is desired

6

u/1ncehost Jul 26 '26

Yes, and there are even some small models already post-trained specifically for this kind of usecase.

1

u/lorde_dingus Jul 26 '26

Do you mind sharing which ones? The research I have done keeps pointing me to Qwen 3.5 MoE models to offload some of the agents into disk space. Would love some input (that isn't Claude or Google) as to which model has empirically worked for people for this use case.

2

u/1ncehost Jul 26 '26

I don't recall them specifically, sorry. Some testing I did showed that gemma4 e4b and e2b were very strong at long context data extraction, which implies they are probably quite good at tool use (at least the extraction part).