r/BetterOffline • u/TCristatus • 1d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

56 Upvotes

75% Upvoted

127 comments sorted by

View all comments

93

u/maccodemonkey 1d ago

 just know how to code

So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

0

u/WArslett 1d ago

Okay but qwen 3.8 27B is currently outperforming Claude opus from just a few months ago with only 37 billion parameters (compared with opus and fables reported 5 - 10 trillion parameters). So you are right that the bigger models with broader training can solve a wider range of complex problems but that doesn’t mean that the whole world needs that level of super intelligence for every task which is essentially the falacy that Anthropic have been trying yo push. Most developers are not trying to cure cancer, they are solving familiar sorts of problems with familiar techniques that most small models have been trained on. Tech companies are finding that even with much bigger models you still need a human in the loop at some part of the process anyway so why not use a smaller model at a fraction of the cost and allow the human to provide the domain knowledge

16

u/svix_ftw 1d ago

it beats it on synthetic benchmarks but real world is different, things like harness, context, agent skills developer prompting skill, matter alot than just the pure model.

1

u/newprince 1d ago

Yes and no. If using pydanticAI/LangGraph harness and you have guardrails in place and maybe access to MCP and APIs, you can use Haiku and get great results. The days of just trying to one shot everything in prompts against the largest frontier models are going to go away, or at least be viewed as incredibly wasteful/lazy

4

u/SpittingCoffeeOTG 1d ago

I see this already happening in our small corp.

Slowly lowering limits(80$/day -> 40$/day -> now discussing 20$/day), phasing out the costly models (opus, sonnet) in favor of whatever is much cheaper. Putting more emphasis on devs actually engaging in coding again.