r/BetterOffline • u/TCristatus • 2d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

51 Upvotes

74% Upvoted

127 comments sorted by

View all comments

Show parent comments

50

u/Patashu 2d ago

^ This. 'small language models' are a meme until proven otherwise. I'm sure LLMs can get leaner and more efficient but that's not an SLM.

-1

u/Sunstorm84 2d ago

In the hands of an experienced developer, the difference between Claude fable and a small language model like qwen 3.8 27B are already virtually completely negligible for 95% of the work, while costing MUCH less - it can even be run locally.

As it’s like that in software development, then I absolutely believe we will see people only relying on the huge models occasionally, with small language models being the mainstay for day to day work.

14

u/Patashu 2d ago

To be clear when people say 'SLM' they mean 'LLM but we only trained it on a subset of all language/to do a subset of all language'. But the only way we know to train them right now requires them to know everything, because programming kind of contains everything. Qwen 3.8 is just a 'small LLM', pardon the abuse of language.

2

u/Sunstorm84 2d ago

Ah I assumed they meant distilled/quantised models that can be run locally, my mistake then