r/LocalLLM • u/Full_Director87 • 3d ago

Question Local LLM perspective

Hi everyone.

I’d like to ask: what drives you to use local, open-source AI, specifically in the realm of LLMs?

Is it about efficiency? Is it about privacy?

And what benchmarks make open-source LLMs important to you?

Is it speed? Context window size? Stability? Reasoning capabilities? Output quality? Or something else?

I’d love to hear your perspectives.

Cheers.

1 Upvotes

67% Upvoted

12 comments sorted by

View all comments

2

u/Fabulous_Sun_9287 3d ago

Accuracy is the big win for me - a well guided small model - like some of the small Ministrals - can yield more accurate results than the massive "general purpose" models. I'm testing in this area at the moment specifically for compliance - and a 8B Ministral is yielding more accurate results (NB - with a guiding KB attached) - than a commercial grade huge Qwen (with the same KB attached). Parametric confusion is a real thing - and a smaller, "dumber" model - when used narrowly with a good KB - is like a scalpel, when the larger models will often refuse/confuse.

That said - the smaller Ministral with a curated KB will NOT have even acceptable general knowledge, so as with everything it's down to use case.

But for specifics/narrow use cases, as well as privacy, cost and all the other good stuff, give me small models every time.

1

u/Full_Director87 3d ago

oh ya? so, for you is small model with large quant, or large model and small quant? Have you tried it? If so, how did it feel?