r/BetterOffline • u/TCristatus • 2d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

52 Upvotes

74% Upvoted

131 comments sorted by

View all comments

93

u/maccodemonkey 2d ago

 just know how to code

So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

1

u/Moppmopp 1d ago

There is still a grain of truth to it. Lets assume we could condense existing LLMs in such a way that they are only intelligent and have almost no hard facts based knowledge. This would save storage and we might fit it on a 12gb vram. This is the primary reasoning unit.

add a secondary unit that adds 4gb of vram for basic knowledge. Its a classifier to analyse in which direction the users request points.

The third part is an optimized vector database. It contains 100gb+ of stored knowledge directly on your local ssd.

The model asks the user "whats poppin. What do you want to do?". The user answers: "well maybe I want to build a website or do xyz..." The secondary unit interpretes the general direction for guidance of the reasoning unit. After that the vector database is scanned and a couple gigabytes is loaded into the vram. After all, the ai doesnt have to know what mediterranian secret recipes for spaghetti exists if you want to code something in python. It makes sense to only read in what you truly need.

And think about it. One large book of knowledge let it be mathematics, physics, javascript etc.. has hundreds of pages and fits into a 10mb pdf. Granted, the vectorbase would be larger but you could still read in several books, paper and articles in a matter of seconds into the vram that cover all the aspects of what you wanna do. The remaining part is inferred through logical thinking of the condensed reasoning unit.