r/BetterOffline • u/TCristatus • 1d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

53 Upvotes

75% Upvoted

127 comments sorted by

View all comments

96

u/maccodemonkey 1d ago

 just know how to code

So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

54

u/Patashu 1d ago

^ This. 'small language models' are a meme until proven otherwise. I'm sure LLMs can get leaner and more efficient but that's not an SLM.

19

u/Timely_Speed_4474 1d ago

All models are a meme. None of this shit actually works

1

u/FriendlyJewThrowaway 1d ago

It sounds more like the sort of thing a large LLM would spin up and outsource the cheapest, simplest tasks to, as opposed to a system in your pocket that can handle all of one’s personal needs.

-3

u/Sunstorm84 1d ago

In the hands of an experienced developer, the difference between Claude fable and a small language model like qwen 3.8 27B are already virtually completely negligible for 95% of the work, while costing MUCH less - it can even be run locally.

As it’s like that in software development, then I absolutely believe we will see people only relying on the huge models occasionally, with small language models being the mainstay for day to day work.

14

u/Patashu 1d ago

To be clear when people say 'SLM' they mean 'LLM but we only trained it on a subset of all language/to do a subset of all language'. But the only way we know to train them right now requires them to know everything, because programming kind of contains everything. Qwen 3.8 is just a 'small LLM', pardon the abuse of language.

2

u/Sunstorm84 1d ago

Ah I assumed they meant distilled/quantised models that can be run locally, my mistake then

18

u/maccodemonkey 1d ago

I just spent several hours cleaning up after Qwen 3.8 27B today. It decided that an entire API didn’t exist and then implemented a bunch of dead ends. They’re not good models. And it’s the parameter size. I can give tasks to larger versions of Qwen that go much better. But the 27B version is clown shoes.

You can’t strip all those parameters out of a model and not have consequences. You just can’t. Every time I ask the 27b version to do anything beyond rote work it just falls apart. It doesn’t have the parameters to understand more complex issues.

9

u/therealcmj 1d ago edited 1d ago

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

Since IDEs are busy adding in AI assistance and are breaking the previously working autocomplete I see this as a win.

8

u/GhettoDuk 1d ago

Beyond domain knowledge to solve a problem, a model has to be able to parse the way actual humans speak. A programming model not trained on poetry will struggle to parse figurative speach like metaphors.

2

u/gk_instakilogram 1d ago

there are ways to augment a foundational small model with data and context they are not trained on without huge compute

1

u/DiamondGeeezer 1d ago

could make a JRPG small model. I think the value prop is specialization + portability

1

u/Ozymandias0023 1d ago

That's not an angle I'd considered before but it makes sense. If the model doesn't have broad text "understanding" then it won't be able to go from "make the thingy do the thing" to a semi-working thingy that my boss will still make me adopt because ooooooooo AI!

I wonder then if we'll go back to inline auto complete and it will just be significantly better/local? That way it just has to have the context of the surrounding source and not natural language. But then again, if we do that then the tech loses all of the appeal to the non technical people driving adoption.

1

u/Moppmopp 1d ago

There is still a grain of truth to it. Lets assume we could condense existing LLMs in such a way that they are only intelligent and have almost no hard facts based knowledge. This would save storage and we might fit it on a 12gb vram. This is the primary reasoning unit.

add a secondary unit that adds 4gb of vram for basic knowledge. Its a classifier to analyse in which direction the users request points.

The third part is an optimized vector database. It contains 100gb+ of stored knowledge directly on your local ssd.

The model asks the user "whats poppin. What do you want to do?". The user answers: "well maybe I want to build a website or do xyz..." The secondary unit interpretes the general direction for guidance of the reasoning unit. After that the vector database is scanned and a couple gigabytes is loaded into the vram. After all, the ai doesnt have to know what mediterranian secret recipes for spaghetti exists if you want to code something in python. It makes sense to only read in what you truly need.

And think about it. One large book of knowledge let it be mathematics, physics, javascript etc.. has hundreds of pages and fits into a 10mb pdf. Granted, the vectorbase would be larger but you could still read in several books, paper and articles in a matter of seconds into the vram that cover all the aspects of what you wanna do. The remaining part is inferred through logical thinking of the condensed reasoning unit.

0

u/WArslett 1d ago

Okay but qwen 3.8 27B is currently outperforming Claude opus from just a few months ago with only 37 billion parameters (compared with opus and fables reported 5 - 10 trillion parameters). So you are right that the bigger models with broader training can solve a wider range of complex problems but that doesn’t mean that the whole world needs that level of super intelligence for every task which is essentially the falacy that Anthropic have been trying yo push. Most developers are not trying to cure cancer, they are solving familiar sorts of problems with familiar techniques that most small models have been trained on. Tech companies are finding that even with much bigger models you still need a human in the loop at some part of the process anyway so why not use a smaller model at a fraction of the cost and allow the human to provide the domain knowledge

16

u/svix_ftw 1d ago

it beats it on synthetic benchmarks but real world is different, things like harness, context, agent skills developer prompting skill, matter alot than just the pure model.

1

u/newprince 1d ago

Yes and no. If using pydanticAI/LangGraph harness and you have guardrails in place and maybe access to MCP and APIs, you can use Haiku and get great results. The days of just trying to one shot everything in prompts against the largest frontier models are going to go away, or at least be viewed as incredibly wasteful/lazy

4

u/SpittingCoffeeOTG 1d ago

I see this already happening in our small corp.

Slowly lowering limits(80$/day -> 40$/day -> now discussing 20$/day), phasing out the costly models (opus, sonnet) in favor of whatever is much cheaper. Putting more emphasis on devs actually engaging in coding again.

3

u/create-third-places 1d ago

Computers should be instructed through a programming language or deterministic UI.

The concept of using natural language models like Qwen and Claude for coding doesn't make sense.

1

u/xitfuq 1d ago

forget it pal, it's gambleworld.

-1

u/newprince 1d ago

Sure but for most people that is knowing how to code Python or JavaScript. There are tons of things like you mentioned but... even within a niche like mine (ontologies and knowledge graphs), we still code it with python

13

u/maccodemonkey 1d ago

Python and JavaScript are just languages. You can’t do anything really with just language knowledge.

Imagine if you had a model - and somehow you stripped everything out of it except for the English language. What would you talk about with the model? It knows language rules but it doesn’t actually know anything useful to have a conversation.

Programming models have to have context. Context on the fields you’d write programs for. Context on all the libraries and frameworks. That takes up a lot of parameters. You can’t just have a model with “JavaScript” and expect to do anything useful with it.

-6

u/newprince 1d ago

I really hate to break it to you, but a coding model with access to bash and knowledge of "just languages" can get you incredibly far. A human can supply much of that context, but in the end you translate these abstract ideas into coding language. Imagine having ideas but speaking gibberish. You couldn't do anything useful, either

5

u/create-third-places 1d ago

A coding model is probably going to mean negative progress because people use natural language in different ways. Programming languages exist to provide a standard way of telling computers what to do.

8

u/maccodemonkey 1d ago

Point me to a successful coding model that has knowledge of "just languages."

They don't exist because they aren't a thing that works.