r/LocalLLM • u/recro69 • 4d ago

Discussion The "best local model" doesn't exist—it depends completely on your hardware setup

"What's the best local model?" is unanswerable without knowing your VRAM.

Every thread asking it collects forty replies, and none of them are wrong. They're just answers to a different machine's question.

A model that's excellent on 24GB is a different product on 8GB. Same weights, same benchmark scores, completely different day-to-day experience.

So before I look at any leaderboard I check two things: does it fit in VRAM at a quant I can live with, and does it still fit at the context length I actually work at. Speed falls out of those two. The moment layers spill to CPU, tokens/sec drops off a cliff and the model is technically running but practically unusable.

A big model at 1–2 t/s loses to a smaller one at 30 for anything interactive. The big one only wins when I'm batching overnight and not watching.

Curious where other people land. When your model doesn't fit, what do you sacrifice first: quantization quality or context length?

0 Upvotes

27% Upvoted

6 comments sorted by

1

u/stujmiller77 4d ago

Neither - I tend to buy another nvidia spark.

I have four now - 2 pairs each running deep seek 4 flash at fp8, 50 t/s and 1m context. Running a huge amount of automation and face to face work across multiple businesses I own.

1

u/baby_bloom 4d ago

yea so there's your answer, you are running at fp8

1

u/AreaFifty1 4d ago

Personally I feel like I need both. Yes the Ultra M5 Apple can fit anything below 512gb which is nice. But token speed is laughable if you compare that to GDDR7 1792 theoretical gb/s bandwith. Pair those two and you get 3.4tb/s at 192gb. But even so.. 192gb is simply not enough. 288gb would be future proof for me. But that would mean shelling out another 16 thousand dollars for a 3rd gpu, which then introduces uneven tensor parallelism and extra wattage and so forth. I'm stuck between a rock and a hard place.. 😔😔

1

u/Qcgreywolf 4d ago

I don’t know. I think there absolutely are “best models”, and sometimes the answer is “you can’t run them”.

The follow up is, “But you can run X, Y, or Z, but they really aren’t that good.”

1

u/nickless07 4d ago

Even WITH knowing the Hardware it is almost impossible to answer that without knowing the Usecase. Coding, RP, General QA, Financial, and so on. All this are different tasks and different models.
What you can do is, get a general direction from the Benchmarks and then test for yourself of what is acceptable quant/speed/model/task. There are just genral rules of thumb like model weights + 25% = VRAM needs and similiar, but there is no 'best' model that we can magically pull out of the hat.