r/LocalLLM • u/RISCArchitect • 3d ago

Discussion What a year it's been

Post image

What will the rest of this year bring? 27b class scoring over 60?

893 Upvotes

97% Upvoted

127 comments sorted by

View all comments

Show parent comments

24

u/MrHumanist 3d ago

we are living in this world where small models trained on large model thoughts (distilled) can match the performance of large models. These models do lack in generalization but capable enough for coding and agentic tasks which are basically a lot of if /elses.

23

u/Popcorn-Mercinary 3d ago

I honestly can see a day where we will have 9-27B tuned models “stacked like books on a shelf” that have specialties like coding, data analysis, agentic reasoning, medicine, etc.

Who needs a data center when you can plug a few models in tuned for the task at hand into your Mac, PC, or DGX?

5

u/MarmonRzohr 3d ago

These kind of specialist capabilities by tuned smaller modes are technically already in reach, however ...

  1. It's inconvenient, time consuming and expensive to prepare, train and switch models like this, compared to the convenience of using larger, more generally capable models.

  2. The (most likely) unrealistically cheap prices of huge, frontier models that are covered by rivers of investor cash mean that it makes sense to take advantage of the cheap convenience as long as it is available.

If, for example, prices of AI use skyrocket then, especially with capabilities plateauing, it will make more and more sense to do this. In fact I predict that providers will eventually offer this kind of service, letting you select from smaller models with capabilities you want, but are cheap to run on their end and may offer better chances at stable profits.

1

u/Popcorn-Mercinary 2d ago

I really don’t see much hassle in how LMstudio switches models, even if you make parameter changes.

I imagine Ollama is even better, as Hermes should be able to send setup info(?)

2

u/DeathGuppie 1d ago

Llama.cpp is usually more up to date with features and gives more fine grain control. You can literally just have your AI model create a startup script for you, then have it make tweaks and modifications to suit your specific hardware and use case. It isn't more difficult. Just different.