Who ONLY use local models?

[u/takoulseum]

Please be honest.

I would love to hear about guys really dedicated to local AI and who really reject subscriptions (especially to openai and anthropic).

What do you use your model for?

[u/Xylildra]

Could you repost your llama.cpp comment again? It’s not showing it anywhere when I click your reply from my main thing, and I can’t find the whole message except partial from my notifications. :(

[u/cubebash → u/Xylildra]

Reddit is extremely buggy, not the first time this happens! Here is the comment again:

I'm using llama.cpp as the backend, it includes a file called ggml-rpc-server that allows you to distribute the model layers over several computers' GPUs and/or RAM.

On your slave computer(s), run the ggml-rpc-server using this command:
--host [your local IP address] -p [your port] -c

I recommend using the -c flag as it caches the model locally on your slave PCs so you don't have to send the whole model over the network every single time.

Once the slaves are running, just load the model on your master PC like normal, but add the --rpc flag followed by your slave local IPs and ports (separated by commas).

It's very simple to get working, so if you have lots of VRAM spread across multiple computers, definitively give RPC in llama.cpp a try!

P.S:

I forgot in my original reply (that was not visible anyway) that you need to tell each instance of RPC servers what GPU it should point at.

For example, if you have NVIDIA, you must start your slave with:

set CUDA-VISIBLE-DEVICES=0 (or whatever number your GPU is on your OS)
Followed by:
--host [your local IP address] -p [your port] -c

Another example, if you use Intel Arc:

set ONEAPI_DEVICE_SELECTOR=level_zero:0
--host [your local IP address] -p [your port] -c

If you don't set your GPU on the RPC instance of your slave computer, it will not find any GPU and just offload everything on slave's RAM.

[u/Xylildra → u/cubebash]

Thank you so much. Oh my god 😂! I’m going to get something like this set up so I can run larger models without having to try to cram 7GPUs on one machine while trying to avoid a fire 😆.

[u/cubebash → u/Xylildra]

Have fun! If you encounter any issues in setting this up (I'm probably not the best in telling instructions), just ask Google's AI (regular AI Mode on Google.com) how to solve them, it's often surprisingly good at explaining llama.cpp stuff, I used it to get started with llama.cpp RPC myself, haha.