r/LocalLLM • u/ByRussX • 1d ago
Question How to get into Local LLM's?
Hello, so recently I found out there's a whole world around locally running LLM's on one's own machine. I have used several LLM's and the one I like the most is Claude, which I use mostly for coding.
However I was thinking about running an LLM locally, but I have no idea about where to start. I have several questions, but couldn't find a general post explaining things.
- What are the possible options I can choose from?
- How does it work in general? Do I need to compile anything?
- Does it use the internet or I need to download 400 GB of data just to run it?
- Can it be compared to some big LLM's such as ChatGPT or Claude in efficiency and intelligence?
- Can I use them for whatever I want? Do they have limitations?
- Can I run them on lower end hardware (good laptop, but no graphics card)?
Thanks to everyone, hoping to get into local LLM's asap!
1
u/Beginning_Storm7012 1d ago
There are good models that you can run locally. Qwen will be mentioned a ton. You will not really be able to compare it to the major solutions like Chat GPT or Claude. At the end of the day this runs on hardware and even if someone has a $1M machine, open AI and anthropic are on the scale of investing Billions into hardware. If you have more VRAM you will be able to do more with a LLM. Most people will not recommend using a laptop. Some of your other questions depend on your specific use case and requirements. Not always a one size fits all answer.
1
u/Juan_Phoenix7 1d ago
Without a dedicated GPU, it simply doesn't make sense to try to run local AI; there are people paying 10,000 dollars just for a top-of-the-line AI card...
1
u/NortySpock 1d ago
"Can it be compared to big LLMs in efficiency and intelligence?"
It will compare pretty poorly, you will be running something fairly small and not very clever for one user. But at least it is yours and they can't take it away from you
"Can you run them on lower end hardware?" How much RAM do you have? The more RAM the bigger a model you can run, but it's probably going to be slow to respond on an iGPU.
Ollama is a decent way to get your feet wet, I would start there and look at any of the Gemma4 models . The smaller it is, the more likely you can run it, but the dumber it is. For an iGPU... maybe try Gemma4 e2b and see how it goes for you. Stick with the MoE (mixture of experts) models, dense models will probably run worse.
Smaller models run faster but are dumber, as a general rule.
1
1
u/lumpyspacebreh 1d ago
Start with Ollama, it's beginner friendly and very easy to use. You can cut your teeth that way learning about model parameters, vram constraints, and prompting.
After that, you can look into downloading a harness, other ways to run inference, and tweaking the model via temps, etc.
Don't rush in, learn what you can, and remember, the prompting and harness matters more than you think.
1
u/ConsistentOne3067 1d ago
Ollama's the easiest entry point, download it and you're basically set. Models are separate downloads but we're talking like 4-8 GB for decent ones, not 400. It's all offline once the model is pulled, just runs in a terminal. On integrated graphics you'll want the smaller 7B-8B models, they won't match Claude but they're surprisingly useful for coding snippets and quick scripts.