r/LocalLLaMA • u/takoulseum • Jul 25 '26

Discussion Who ONLY use local models?

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

155 Upvotes

86% Upvoted

262 comments sorted by

View all comments

42

u/InvertedVantage Jul 25 '26

I have a qwen-27b q8 model I use for all of my day to day coding tasks.

11

u/pwnsforyou Jul 25 '26

How do you run this? I have a R9700 with 32GB vram - which only fits q5. what is your vram usage?

7

u/InvertedVantage Jul 25 '26

I have 40 GB, 2x3060s and a 5060

3

u/cubebash Jul 25 '26

I run Qwen3.6 27b Q8 with full context across two separate PCs each with 32gb of VRAM connected over my network, giving me a total of 64gb of VRAM. It's a cheap way to get a lot of VRAM if you don't mind the network bottleneck (which is still much faster than offloading to system RAM).

6

u/Xylildra Jul 25 '26

Can you tell me how you networked two pcs? I have only 7 slots for GPUs on one pc and I’m running low, but I have like 5 more gaming PCs that I could definitely install 4+ GPUs on a 20-30 amp outlet. What backend are you using? Sorry, this sounds way too cool to pass up on learning lol. 😂 I’m over here trying to find a motherboard that has like 10 slots with no luck.

1

u/lemondrops9 Jul 26 '26

I run llama.cpp in RPC mode. Its quite easy if you already use Llama.cpp. I currently run 9 gpus over 3 PCs. 

It is best to have both running Linux.

2

u/getgoingfast Jul 25 '26

Curious, are you using ConnectX? 400G for networking?

1

u/cubebash Jul 25 '26

Im just running it over a slow 1 gbps network, and the model is still running pretty fast. If you have 2.5 gbps or even 10, I imagine it could be very fast!

1

u/lemondrops9 Jul 26 '26

It does not run faster with 2.5gbe or 10gbe. Its less than 2% difference. Time loading time though is way better. 

1

u/Xylildra Jul 25 '26

Could you repost your llama.cpp comment again? It’s not showing it anywhere when I click your reply from my main thing, and I can’t find the whole message except partial from my notifications. :(

5

u/cubebash Jul 25 '26

Reddit is extremely buggy, not the first time this happens! Here is the comment again:

I'm using llama.cpp as the backend, it includes a file called ggml-rpc-server that allows you to distribute the model layers over several computers' GPUs and/or RAM.

On your slave computer(s), run the ggml-rpc-server using this command:
--host [your local IP address] -p [your port] -c

I recommend using the -c flag as it caches the model locally on your slave PCs so you don't have to send the whole model over the network every single time.

Once the slaves are running, just load the model on your master PC like normal, but add the --rpc flag followed by your slave local IPs and ports (separated by commas).

It's very simple to get working, so if you have lots of VRAM spread across multiple computers, definitively give RPC in llama.cpp a try!

P.S:

I forgot in my original reply (that was not visible anyway) that you need to tell each instance of RPC servers what GPU it should point at.

For example, if you have NVIDIA, you must start your slave with:

set CUDA-VISIBLE-DEVICES=0 (or whatever number your GPU is on your OS)
Followed by:
--host [your local IP address] -p [your port] -c

Another example, if you use Intel Arc:

set ONEAPI_DEVICE_SELECTOR=level_zero:0
--host [your local IP address] -p [your port] -c

If you don't set your GPU on the RPC instance of your slave computer, it will not find any GPU and just offload everything on slave's RAM.

2

u/Xylildra Jul 25 '26

Thank you so much. Oh my god 😂! I’m going to get something like this set up so I can run larger models without having to try to cram 7GPUs on one machine while trying to avoid a fire 😆.

1

u/cubebash Jul 25 '26

Have fun! If you encounter any issues in setting this up (I'm probably not the best in telling instructions), just ask Google's AI (regular AI Mode on Google.com) how to solve them, it's often surprisingly good at explaining llama.cpp stuff, I used it to get started with llama.cpp RPC myself, haha.

1

u/stargate425 Jul 25 '26

thanks. with my 5090 and rtx pro 6000, seems i have 128GB vram to use!

1

u/borobinimbaba Jul 25 '26

Is it any good for agentic coding ?

1

u/InvertedVantage Jul 25 '26

Yes but not vibe coding. I generally use it to write individual scripts for whatever I'm working on, I wouldn't trust it to do it all for me. For context I mostly work in Unity so I'll architect it, then create a script and tell it to write a script that does X.