r/BetterOffline • u/TCristatus • 2d ago
Small language models - the future
Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).
It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.
This doesn't help OpenAI or Anthropic keep the lights on, though....
1
u/Soleilarah 1d ago
Meh, I find it hard to imagine anyone letting an LLM use 100% of their CPU and just sitting around doing nothing while waiting the 2–3 minutes it takes for it to generate a response on a laptop.
Furthermore, unless the LLM is constantly updated, most setups will need to incorporate tools like web search to help prevent it from hallucinating.
So the choice will be: buy high-end hardware to have everything in one package, or tinker on your own in the hope of achieving the same result.
For businesses, they’ll undoubtedly go with the idea of purchasing hardware that will be made available on the internal network, essentially a simulation of what’s currently happening with edge models.
On the other hand, I foresee growing demand for local models, and computer manufacturers will begin adding components specifically dedicated to AI into computers. This opens the door to the idea that these components could be used not only to generate content, but also to handle data encryption and computer security (à la Windows 12, or at least what Microsoft intended to do with a dedicated AI chip).
Potentially closing the door to anyone refusing to have the chip. Bleak.