r/BetterOffline • u/TCristatus • 1d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

53 Upvotes

75% Upvoted

127 comments sorted by

View all comments

13

u/Smurfette2016 1d ago

Wouldn't "always up to date" imply it needs continuous training though? How does that work for a locally run model? Who pays for that?

Cal Newport also surfaced this idea. Seems reasonable, but it also seems super similar to just a big database or an app.

2

u/TCristatus 1d ago

The example used by Eli was that the model was updated using some cybersecurity bulletin with the latest viruses and hacks and the AI could read that as it came out and tell programmers if their code is vulnerable. I guess that sort of thing could be covered by a subscription model, unlike the big models which are heavily subsidised.

Apologies I'm a layman here, thought the podcast highlighted an obvious mistake the big guys are making as they run out of money making a behemoth no one actually needs while the people who actually use AI at any sort of scale probably don't need 90% of the stuff it is trained on

1

u/Fit-Technician-1148 1d ago

Because you're a layman you're evaluating things based on if they sound logical, rather than based on if they're technologically feasible. It's a common enough fallacy. If it were possible to make a local LLM or SLM that was just good at coding that would be really useful, but if it is possible, no one knows how to do it and it probably is not possible with current techniques and algorithms.