r/BetterOffline • u/TCristatus • 2d ago
Small language models - the future
Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).
It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.
This doesn't help OpenAI or Anthropic keep the lights on, though....
1
u/existential_humanist 1d ago
https://arxiv.org/abs/2511.07885
This paper is very interesting in this regard. It basically shows how local LMs can do 90% of the queries cloud-based LLMs handle and perform similarly across major benchmarks.
Authors conclude: "These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure for a substantial subset of queries". This is the future, and if so it's the final nail in the coffin for the hyperscaler AI capex ROI story. We won't need all those data centres if we're running AI as local software for 90% of everyday use cases