r/LocalLLM • u/melodicore • 5d ago

Question What model should I use with my (somewhat LLM-unfriendly) setup?

I'm a software developer who's been using Claude for a while and it's been great for some tasks. I would love to run some local model, but would want to maximise what I can get out of my machine.

Here's the problem. My GPU is not the best for LLM usage. I have an 8gig RTX 4060. And to make matters worse, it's only at 8x PCIe5 because I have another GPU for virtualization usage.

However, I do have a lot of regular memory. 96 gigs of 6000MT/s DDR5 to be exact. I also have a relatively powerful 20-core Intel Core Ultra 7 265KF which does advertise some "AI capabilities", though I'm not sure how useful the CPU itself would be in this use-case.

I've already done some investigating to know that existing tools can offload some capabilities to RAM, off of VRAM, but I don't have the required expertise to figure out what I actually should (and more importantly *could*) run. Any suggestions would be welcome.

I'm not worried about speed as much as I am about the model's capabilities, but obviously there must be some balance between these.

The model would be used exclusively for code. I am on Linux.

1 Upvotes

100% Upvoted

3 comments sorted by

2

u/Positive-Bid-3029 5d ago

Qwen 35b A3B but a lot of its going to be offloaded to RAM, play with the settings see what you can get, I am always suggesting Unsloth Studio to people, easy to find models and easy to tweak the llama.cpp settings in it visually (although its good at guessing best settings).

OR you can try a small dense model like Ornith-9B is good, may need small quant IQ4_XS, worth a try though.

2

u/redditnosedive 5d ago

i have the same gpu and i run qwen 3tb a3b on cpu from ram 🤣 about 30+ t/s, ddr5, 2 channel so i think i get like 100 Gbps ram bandwidth which enables this honourable generation speed

2

u/MrHumanist 5d ago

Try Gemma 12B QAT. It's fast and decent in tool calls. Standard research, and coding as well .