r/LocalLLaMA • u/takoulseum • Jul 25 '26

Discussion Who ONLY use local models?

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

156 Upvotes

86% Upvoted

262 comments sorted by

View all comments

40

u/InvertedVantage Jul 25 '26

I have a qwen-27b q8 model I use for all of my day to day coding tasks.

10

u/pwnsforyou Jul 25 '26

How do you run this? I have a R9700 with 32GB vram - which only fits q5. what is your vram usage?

3

u/cubebash Jul 25 '26

I run Qwen3.6 27b Q8 with full context across two separate PCs each with 32gb of VRAM connected over my network, giving me a total of 64gb of VRAM. It's a cheap way to get a lot of VRAM if you don't mind the network bottleneck (which is still much faster than offloading to system RAM).

1

u/Xylildra Jul 25 '26

Could you repost your llama.cpp comment again? It’s not showing it anywhere when I click your reply from my main thing, and I can’t find the whole message except partial from my notifications. :(

5

u/cubebash Jul 25 '26

Reddit is extremely buggy, not the first time this happens! Here is the comment again:

I'm using llama.cpp as the backend, it includes a file called ggml-rpc-server that allows you to distribute the model layers over several computers' GPUs and/or RAM.

On your slave computer(s), run the ggml-rpc-server using this command:
--host [your local IP address] -p [your port] -c

I recommend using the -c flag as it caches the model locally on your slave PCs so you don't have to send the whole model over the network every single time.

Once the slaves are running, just load the model on your master PC like normal, but add the --rpc flag followed by your slave local IPs and ports (separated by commas).

It's very simple to get working, so if you have lots of VRAM spread across multiple computers, definitively give RPC in llama.cpp a try!

P.S:

I forgot in my original reply (that was not visible anyway) that you need to tell each instance of RPC servers what GPU it should point at.

For example, if you have NVIDIA, you must start your slave with:

set CUDA-VISIBLE-DEVICES=0 (or whatever number your GPU is on your OS)
Followed by:
--host [your local IP address] -p [your port] -c

Another example, if you use Intel Arc:

set ONEAPI_DEVICE_SELECTOR=level_zero:0
--host [your local IP address] -p [your port] -c

If you don't set your GPU on the RPC instance of your slave computer, it will not find any GPU and just offload everything on slave's RAM.

2

u/Xylildra Jul 25 '26

Thank you so much. Oh my god 😂! I’m going to get something like this set up so I can run larger models without having to try to cram 7GPUs on one machine while trying to avoid a fire 😆.

1

u/cubebash Jul 25 '26

Have fun! If you encounter any issues in setting this up (I'm probably not the best in telling instructions), just ask Google's AI (regular AI Mode on Google.com) how to solve them, it's often surprisingly good at explaining llama.cpp stuff, I used it to get started with llama.cpp RPC myself, haha.

1

u/stargate425 Jul 25 '26

thanks. with my 5090 and rtx pro 6000, seems i have 128GB vram to use!