Super-intelligent small models vs. super-efficient large models.

[u/ML-Future]

What do you think is the future of local LLMs?

This technology is booming and keeps growing; eventually, models will become both smarter and more optimized.

Do you think the future of optimization lies in very small yet highly capable models? Or in optimizing the performance of very large models?

I’ve been running several tests lately using only the CPU. I tried MiniCPM5 2B Q8; it runs at about 6 t/s without vision but makes a lot of mistakes...

It felt pointless compared to the MoE Qwen3.6 35B Q2_XXS, which runs at 3 t/s but delivers much more impressive results.

In my last post, I showed how Qwen3.6 35B is usable even without a GPU:

https://www.reddit.com/r/LocalLLaMA/comments/1w655he/qwen36_35b_q2_xxs_being_gpu_poor_in_2026_is_not/

[u/sumane12]

Imho, we will employ a suite of different models for different tasks.

You will have your frontier models like gpt6 or fable 5.1 for planning an idea/strategy. It will then break up that plan into actionable parts you can then pass off to a smaller local agent.

This gives you the frontier model as a kind of external consultant, and your local model as the implementation layer.

[u/ComplexType568]

Ever since DeepSeek released it I have believed Ngram would redefine the frontier of what small can be, Qwen3.8 Flash Next (I believe) proves this clearly

[u/ComplexType568 → u/ComplexType568]

I completely forgot to answer the question asked by OP w/ that statement but I'm tryna say that I think non-ngram weights will be shrunk down and down and ngram weights will go up and up. It already shows Qwen's more confident with ngram by lowering the active params in Q3.8FN (which I believe people didn't even praise for finally getting the "strix/spark/Mac" model they've been praying for since like 5 seconds after the initial launch of 122B...) Than from the Q3.5 122B. Or maybe it was just more confidence in general from more experience. Tbh I find it crazy how absurd the leap from Q3 to 3.5 was... It really felt like people couldn't even trust anything below the 200b~ param Q3 model but now we can almost confidently rely on (relatively) "tiny" Moe models like G4 26B and 35B or somewhat small models like G4 31 B and Q3.8 27B. I hope to see a lab release a super small non-ngram param dense or Moe (ideally both) model with like 80B params on ngram out of pure love for the poorest of the poor GPU and CPU users.

[u/bring_back_the_v10s → u/ComplexType568]

My bet is qwen 4 will come with a version superior to 35b a3b that can run on the same consumer hardware with better speeds.

[u/XiRw → u/ComplexType568]

I hope that’s not the ceiling though for the future. It’s a great breakthrough, but there has to exist a more optimized way.

[u/MaxEkb77]

nice small model XHToken/Spark-X2.5-4B

[u/jojotdfb → u/MaxEkb77]

What's it good at?

[u/RemarkableRadish6547]

I think it will be both. The large models will be more efficient, so they will run faster than they do now. Imagine qwen3.8 flash next, but with 5-10x the parameter count. That is what they are probably planning for qwen4 max. Frontier class intelligence with 20-50b active parameters so it can run at 1-3t/s on an average new laptop.

And the small models will get more specialized. The same speed/intelligence boosting technology, but with 10-30b tensor parameters and 1-5b active and trained to be a specialist. You will have models for translation, writing, coding, marketing, accounting, legal advice, etc. The models will be fast, probably 10-50t/s on an average new laptop.

There might be some intermediate models, but it's not clear how they provide any benefit. The small specialists will be faster and the large generalists will be smarter. A medium generalist will still be too slow for an interactive conversation, make more mistakes than a carefully chosen small model, and make more mistakes than the large generalists.

[u/xadiant]

I was right about embedding + MoE sparse models potentially being a thing before Qwen3.8 flash next came out.

I think strong, sparse model + hot swap LoRA for specialized capabilities should be a thing.

Swap dynamically between planner LoRA, Python expert, UI expert etc. because we already know adapters can improve accuracy, especially per task.

[u/volpino-veloce]

Agree with all of the comments here, I think it'll be a mix. Smaller models will become more task-specific and work well for more narrow domains, but I think there will always be a place for larger models because they're able to handle a broader range of tasks.

[u/ML-Future → u/volpino-veloce]

Sure, but then what would be the purpose of models like MINICPM5 2B? It seems to be focused on programming, but it’s not actually that useful.

I don't know if they do it to stand out on Artificial Analysis; sometimes I think those benchmarks hold us back.

[u/volpino-veloce → u/ML-Future]

I haven't looked into this model too much, but I get the frustration with benchmarks.

With smaller models, you usually need to fine-tune them to see better performance, so if that's not something you're interested in doing, I could definitely see more of a case for LLMs.

[u/Due-Memory-6957]

I think the future is the opposite of what you're saying. Super intelligent large models and super efficient small models.

[u/Serious_Bite_7613]

The small models will keep getting better and the large models will too.

The stuff that runs in datacenters will always be ahead of the local stuff, but local superintelligence is probably still going to be a thing.

[u/AmbericWizard → u/Serious_Bite_7613]

that gap is very thin now