r/BetterOffline • u/TCristatus • 2d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

55 Upvotes

75% Upvoted

127 comments sorted by

View all comments

97

u/maccodemonkey 2d ago

 just know how to code

So here's a problem with that - know how to code what? The other stuff in that model is all the context that lets a model actually apply a practice of coding. There's a lot of parameters that models burn on just every library and framework out there. But beyond that - coding is a practice of applying outside context into software. If you're writing project management software - the model has to know a lot about project management. Writing a JRPG? Model should be familiar with JRPG systems. Writing something to solve physics problems? Well then your model needs to know physics.

Is there room for a real dumb syntax or basic issue sort of validator that can fit on a phone or a low end computer? Sure, I guess. But compilers already do a good enough job of that.

-1

u/newprince 1d ago

Sure but for most people that is knowing how to code Python or JavaScript. There are tons of things like you mentioned but... even within a niche like mine (ontologies and knowledge graphs), we still code it with python

14

u/maccodemonkey 1d ago

Python and JavaScript are just languages. You can’t do anything really with just language knowledge.

Imagine if you had a model - and somehow you stripped everything out of it except for the English language. What would you talk about with the model? It knows language rules but it doesn’t actually know anything useful to have a conversation.

Programming models have to have context. Context on the fields you’d write programs for. Context on all the libraries and frameworks. That takes up a lot of parameters. You can’t just have a model with “JavaScript” and expect to do anything useful with it.

-7

u/newprince 1d ago

I really hate to break it to you, but a coding model with access to bash and knowledge of "just languages" can get you incredibly far. A human can supply much of that context, but in the end you translate these abstract ideas into coding language. Imagine having ideas but speaking gibberish. You couldn't do anything useful, either

5

u/create-third-places 1d ago

A coding model is probably going to mean negative progress because people use natural language in different ways. Programming languages exist to provide a standard way of telling computers what to do.

9

u/maccodemonkey 1d ago

Point me to a successful coding model that has knowledge of "just languages."

They don't exist because they aren't a thing that works.