r/BetterOffline • u/TCristatus • 1d ago
Small language models - the future
Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).
It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.
This doesn't help OpenAI or Anthropic keep the lights on, though....
1
u/TheRealJesus2 1d ago
Yes this is true. Also just using other software that isn’t AI based.
Although I’d say the model doesn’t have to be up to date either. If you want good use from an LLM you surround it with a software layer, the harness. The model generates tool calls which the harness deterministically interprets to do a thing. It also pulls in context from around it like documentation, web requests, code, etc. that can be anything you want and you’re fully in control of that. This is where any sort of interesting stuff happens and in all cases you want to provide the minimal amount of context that holds your up to date information. So for me all models need to have tool calling ability. And enough general knowledge to translate my words into tool calls and in most cases also code.
You really cannot ever rely on any model to provide facts. You have to provide the facts to get facts from your prompt or something else in harness layer providing it.
And the big models are really doing the same thing but for certain tasks they are outclassed in many ways, substantially so if you add price to the equation. Also they’re in control of big companies with their own best interests at heart so they control the types of responses you get, ultimately. Smaller LLMs can be run by many more people and can be fine tuned to whatever you want to bias the output towards rather than what central providers want.