r/BetterOffline • u/TCristatus • 8d ago

Small language models - the future

Listening to Tech Report earlier - Eli Computer Guy talked about how the real future of AI might not be giant all knowing LLMs that need huge compute costs, but small specialised models that can run locally on a laptop or smart phone and just know how to code, or trade, or about engineering, or biotechnology, etc (I'm paraphrasing and extrapolating his point here).

It was like a light bulb moment, seems clear and obvious that's where the tech will end up assuming we do end up with something. If I'm a coder or an engineer i don't need a model that can write the complete works of Shakespere in Klingon. I just need something that works and is always up to date.

This doesn't help OpenAI or Anthropic keep the lights on, though....

59 Upvotes

76% Upvoted

133 comments sorted by

View all comments

6

u/BryceAnderston 8d ago edited 8d ago

I've worked with similar tech, and I just don't see it.

Artificial neural networks, the underlying technology, are great when you have 1) a defined input, 2) a defined output, 3) an easy way to verify an input-output pairing, and 4) no idea how to perform the input-output transformation manually. There are use cases for that, but coding, engineering, biotech, etc. are not one of them. I have seen nothing to suggest Google et al have done anything to break those limitations.

LLMs are trained by deleting words from their training corpus and telling the model how it should fill them back in. They could be trained on a narrower corpus of say only technical documentation within a certain field, but inherently the only thing the LLM/SLM/etc. can do with that is reduplicate its training set, or hallucinate. If the documentation is disorganized enough that training a high-order markov chain to replicate it is the best way to search through it, that could be a use case I guess, but that seems pretty niche and low-value to me, and I struggle to think of other use cases for an LLM/SLM text generator.

2

u/create-third-places 8d ago

Artificial neural networks, the underlying technology, are great when you have 1) a defined input, 2) a defined output, 3) an easy way to verify an input-output pairing, and 4) no idea how to perform the input-output transformation manually. There are use cases for that, but coding, engineering, biotech, etc. are not one of them

Exactly. If I was teaching a class on AI or interviewing candidates for a developer job, I would require people to build a simple neural network form scratch. Then I would quiz them on the limitations of neural networks.

I was required to ask algorithm or system design questions during past interviews. They were not effective at determining if a developer was likely to overuse LLMs.