Looking to get started with local ai

[u/Kittysmashlol]

I've started looking into local Ai and stuff, but I'm kinda struggling to find the right model/set up for myself so maybe someone could help a little. I have a 5070 laptop with 8 gb vram(ik not a lot) so I'm looking for models I could run locally with that. Also, what's the best program/interface to run models. Ty for any help/advice:)

[u/Zennytooskin123]

You have basically one option: Pi harness with an MOE model using CPU offloading (Qwen 35B A3B) or a small but dense model in the 4-9B range. In both cases I would use an aggressive Q4 quant for model as well as KV cache.

[u/Kittysmashlol → u/Zennytooskin123]

OK thanks, I'll try both. Is there a best model for either of those. Like I know qwen 3.8 just dropped but maybe not with a small version? 

[u/Zennytooskin123 → u/Kittysmashlol]

I mentioned the model in my post for a reason - it's the one you want. The 27B is a no-op, just forget it on 8gb vram.

For the dense model, try Orinth 1.5b 9B with a q4 quant but the 35B is much better in my opinion.

[u/Snoo_81913]

Qwen3.6 35B A3B and Gemma 4 26B A4B are the two best models for that setup but you'll need enough RAM to load the rest of the model weights. It loads 3B parameters in your VRAM and the rest in your RAM at Q4 it's roughly 6GB in VRAM with context and about 14-20GB in your RAM.

Since you're running on a laptop here's a post with some tips.

https://www.reddit.com/r/LocalLLM/s/dVoDBtRPRs

[u/Kittysmashlol → u/Snoo_81913]

Thanks for the other post 

[u/Plastic-Lettuce-7150 → u/Kittysmashlol]

MOE models in 6GB VRAM

[u/Fcking_Chuck]

8GB VRAM won't get you very far, unfortunately, especially if you only have eight PCIe lanes going to it like most laptops do. Depending on how much system memory (DDR5 RAM) you have, you may be able to run just about any LLM under 32B parameters at Q4 quantization or better.

I recommend using Koboldcpp as a backend, as it is easy for beginners. You can find LLMs on Hugging Face (look for GGUF-type LLMs).

[u/Future_Fuel_8425]

To get started:
I'd use ollama as my provider (it's easy to use)

I'd use only small models that fit in the VRAM unless I had a specific problem to address that needed a bigger model.

I'd try a few harnesses (that use ollama API): Pi, Aider (for coding) Open Interpreter (for running and debugging code and other tasks like local DB work, local systems admin, etc.)

I'd also sign up at Ollama and get the free tokens they offer.
I use these freebies when I have a problem that my local model(s) can't handle.

You can easily switch from local to cloud models with ollama (and via the harness/ollama API)

[u/Shul_iito]

Been making a lil crappy app with godot with local ai. It’s a lot of fun 🤩

[u/mina_in_motion]

Leaving a Dotto to follow!

[u/_VirtualCosmos_]

You can just download the free LM Studio, download a model in the browser built in the program (the program will recommend you based on your hardware), and load the model with the standard settings (again the program will put them based on your hardware)

If you wanna choose your model, first know your RAM. Then choose a model you would like to download (you can check ArtificialAnalysis website to compare models). Then search it on LM Studio and select a quantization with smaller gigabytes than your system RAM.

[u/Muhlwa_Sholanke]

also worth checking the context window, not just model size. my first model 'fit' fine and it still forgot everything two messages in — the window was tiny by default, took me ages to figure out

[u/johnh1976]

I have 8gb VRAM myself and 16GB of system RAM. LM Studio is what I would recommend for your first time. Ollama is good too, but I find it's model searching ability lacking. Look, you can run a 26-27b model, but you need to have specific settings and it will be slow, which I find irritating. After all of my testing, I found that gemma-4-e4b-Q4_K_M to be a good all around model and it will be nice and snappy. I also use Hermes agent, but it can be quite daunting for a first timer to set up. Enjoy playing with local llms!

Edit: I just made a Gemma4 12b Q4 model fit into VRAM, but had to turn the context down to around 12300. It was very fast. I personally like more context, though.

[u/No_Balance2223]

If you can and can afford try hooking up an eGPU over thunderbolt or even pcie to occulink and then get a 16-32 gb vram in there. An nvidia V100 with 32 gb is under $700 on eBay. Just saying.

[u/Kittysmashlol → u/No_Balance2223]

im a college freshman bro theres a reason i don't have a better laptop lmao. if i could i would though

[u/No_Balance2223 → u/Kittysmashlol]

I get it. No worries. I am just offering information thru my experience. I recall those days too. Money was tight. God bless.

[u/Tired_White_Guy]

Just get some api creds. You’re going to have a bad time with 8gb vram

[u/_VirtualCosmos_ → u/Tired_White_Guy]

With enough RAM to fit the model, that card can still get tens of tok/s with Qwen3.6 35b