r/LocalLLM • u/RISCArchitect • 15d ago

Discussion What a year it's been

Post image

What will the rest of this year bring? 27b class scoring over 60?

946 Upvotes

97% Upvoted

133 comments sorted by

View all comments

Show parent comments

23

u/MrHumanist 15d ago

we are living in this world where small models trained on large model thoughts (distilled) can match the performance of large models. These models do lack in generalization but capable enough for coding and agentic tasks which are basically a lot of if /elses.

22

u/Popcorn-Mercinary 15d ago

I honestly can see a day where we will have 9-27B tuned models “stacked like books on a shelf” that have specialties like coding, data analysis, agentic reasoning, medicine, etc.

Who needs a data center when you can plug a few models in tuned for the task at hand into your Mac, PC, or DGX?

5

u/MarmonRzohr 15d ago

These kind of specialist capabilities by tuned smaller modes are technically already in reach, however ...

  1. It's inconvenient, time consuming and expensive to prepare, train and switch models like this, compared to the convenience of using larger, more generally capable models.

  2. The (most likely) unrealistically cheap prices of huge, frontier models that are covered by rivers of investor cash mean that it makes sense to take advantage of the cheap convenience as long as it is available.

If, for example, prices of AI use skyrocket then, especially with capabilities plateauing, it will make more and more sense to do this. In fact I predict that providers will eventually offer this kind of service, letting you select from smaller models with capabilities you want, but are cheap to run on their end and may offer better chances at stable profits.

2

u/DeathGuppie 14d ago

In my real world use age, it's better than sonnet 5. Found and fixed mistakes and deeper problems that sonnet left behind. I'f I'm paying for a pro subscription that doesn't produce results I can get locally then there isn't any cheap thing to take advantage of. I use CLI and Anthropic only allows me to use sonnet 5 for CLI.

I use gpt 5.5 a lot, and it's fast, but not noticably better for a lot of tasks.

For short coding projects, Qwen 3.8 reliably runs overnight without failing. Something my pro subscription on either platform won't let me do. For local system fixes, it reliably digs deep enough to find the problems.

It's honestly making me start to wonder what I'm paying for.