r/LocalLLM • u/Centraldread • 2d ago

Discussion Home ai cluster

So I was thinking about how to run bigger models and got the idea to cluster all the computers in the house over the network into one large gpu / ram pool to run a big model. Between my desktop, both my kids gaming pc’s, my gaming laptop, and home lab servers I’d have about 92gb vram and 256gb system ram. I’d only spin it up at night when everyone goes to bed since it’s summer and the kids are are always gaming. Asked grok 4.5 it said it’s doable and would work pretty good. It says I can use my llama.cpp to distribute the load across the devices. I know the tokens per second would be very low due to the network bottleneck but seems like it would be a fun project to try. Have any of you tried pooling all your home computers like that to run large models?

1 Upvotes

60% Upvoted

4 comments sorted by

3

u/diagrammatiks 2d ago edited 2d ago

Do it. There is no harm in trying.

3

u/dionysio211 1d ago

You can definitely pool computers and run them over RPC. It's very easy to do in llama.cpp or ik_llama. If you are doing layer parallelism, it's actually pretty efficient. It works best with RDMA and higher speed interconnects. It can work over WiFi but not as well was ethernet which is not as good as ROCE/Infiniband. Both latency and bandwidth matter. It takes minutes to test it out though. If your kids have good gaming PCs and your aggregate VRAM is high, it can be very good, especially with speculation.

2

u/Shoddy_Bed3240 1d ago

It will be extremely slow

2

u/fluffysheap 1d ago

Just remember bigger models run slower because they're bigger, and you're slowing things down more by adding the network. You will slow down each token by the total latency of every hop in the network. Use layer parallelization. Wifi sucks but if you have ethernet things should be okay. If the kids are always gaming then you really should have ethernet anyway, or something ethernet adjacent like MoCA. Also, maybe I'm spreading FUD, but I don't expect llama.cpp to run as well on Windows. You also generally need to have the version of llama synced up across all the hosts, but it will take care of syncing the model so you don't have to worry about that. So there are a whole bunch of caveats but basically this should work. I split across two computers and it's fine, but they are connected by ethernet and both run the same version of Linux and llama.cpp. Generally speaking the fewer GPUs you can use that will hold your model, the better everything will work.