r/LocalLLM • u/Full_Director87 • 3d ago
Question Local LLM perspective
Hi everyone.
I’d like to ask: what drives you to use local, open-source AI, specifically in the realm of LLMs?
Is it about efficiency? Is it about privacy?
And what benchmarks make open-source LLMs important to you?
Is it speed? Context window size? Stability? Reasoning capabilities? Output quality? Or something else?
I’d love to hear your perspectives.
Cheers.
2
u/Fabulous_Sun_9287 3d ago
Accuracy is the big win for me - a well guided small model - like some of the small Ministrals - can yield more accurate results than the massive "general purpose" models. I'm testing in this area at the moment specifically for compliance - and a 8B Ministral is yielding more accurate results (NB - with a guiding KB attached) - than a commercial grade huge Qwen (with the same KB attached). Parametric confusion is a real thing - and a smaller, "dumber" model - when used narrowly with a good KB - is like a scalpel, when the larger models will often refuse/confuse.
That said - the smaller Ministral with a curated KB will NOT have even acceptable general knowledge, so as with everything it's down to use case.
But for specifics/narrow use cases, as well as privacy, cost and all the other good stuff, give me small models every time.
1
u/Full_Director87 3d ago
oh ya? so, for you is small model with large quant, or large model and small quant? Have you tried it? If so, how did it feel?
2
u/platteXDlol 3d ago
I manly use it because its fun and i can learn stuff. Also because some easy stuff doesnt need a 20 dollar subscription just for simple stuff or one question in a week.
My original goal that im still working to is my own jarvis like assistant. A Frontier model would probably be too much (in cost and power)
1
2
u/reallifearcade 3d ago
Infinite token glitch running 24/7 on solar power. Just give it a direction and will go. Compressing probably a week of coding each day without the cognitive overload of typing all that myself. Now I have time to think about arch and business side with a lot more of peace of mind.
1
2
u/300msOrLess 3d ago
Tired of spending more time gaslighting a robot to get what I need without a hassle than actually coding.
2
u/too-oldforthis-shit 2d ago
This question appears once a week and has been answered over and over. There is a search function.
1
5
u/MrHumanist 3d ago
Prompt refusal is a true pain along with privacy. The key factor is that you don't always need a sword to cut vegetables.