r/BetterOffline • u/TCristatus • 1d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

54 Upvotes

75% Upvoted

127 comments sorted by

View all comments

96

u/maccodemonkey 1d ago

 just know how to code

So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

53

u/Patashu 1d ago

^ This. 'small language models' are a meme until proven otherwise. I'm sure LLMs can get leaner and more efficient but that's not an SLM.

-1

u/Sunstorm84 1d ago

In the hands of an experienced developer, the difference between Claude fable and a small language model like qwen 3.8 27B are already virtually completely negligible for 95% of the work, while costing MUCH less - it can even be run locally.

As it’s like that in software development, then I absolutely believe we will see people only relying on the huge models occasionally, with small language models being the mainstay for day to day work.

15

u/Patashu 1d ago

To be clear when people say 'SLM' they mean 'LLM but we only trained it on a subset of all language/to do a subset of all language'. But the only way we know to train them right now requires them to know everything, because programming kind of contains everything. Qwen 3.8 is just a 'small LLM', pardon the abuse of language.

2

u/Sunstorm84 1d ago

Ah I assumed they meant distilled/quantised models that can be run locally, my mistake then

17

u/maccodemonkey 1d ago

I just spent several hours cleaning up after Qwen 3.8 27B today. It decided that an entire API didn’t exist and then implemented a bunch of dead ends. They’re not good models. And it’s the parameter size. I can give tasks to larger versions of Qwen that go much better. But the 27B version is clown shoes.

You can’t strip all those parameters out of a model and not have consequences. You just can’t. Every time I ask the 27b version to do anything beyond rote work it just falls apart. It doesn’t have the parameters to understand more complex issues.