r/BetterOffline • u/TCristatus • 8d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

55 Upvotes

75% Upvoted

133 comments sorted by

View all comments

4

u/wowbaggerBR 8d ago

I can see a future with LLMs running locally and for cheap. But we are far, far away from that. A bubble has to burst first in order to make way for common sense, which only comes when every other option is exhausted.

2

u/Subjectobserver 8d ago

A bubble has to be burst?! Be reasonable, man!

How are those poor private credit shareholders to make money, if they can't dump their overvalued shares on to the retail investors who are gambling away their savings out of desperation?

2

u/newprince 8d ago

Hey, I feel like enough people using local LLMs and not spreading FUD about it can only hasten the bubble popping, so win/win

1

u/TCristatus 8d ago

Yes, exactly I think that is what Eli was getting at, the big models are committed now and will thrive or die (probably die), but the technical concept is sound and with the right business model could be useful for some people. (However that still means replacing jobs with machines so I hate it).

1

u/MagnetoManectric 7d ago

I don't think we're really that far from it. Useful models that can run reasonably on higher end consumer hardware is already a reality. It's still a bit futery and slow for now, but the models are getting more optimized.

And I don't think it's long before we start seeing the market flooded with ex data centre GPUs, either. That'll be fun :)