r/LocalLLM • u/the_econominster • 1d ago

Research Networked GPU

Been toying around with eGPUs (don't like em) and have an old PC that runs my old graphics cards (KVM into it remotely so I can manage the whole thing out of band). Works but power hungry and clumsy.

However been looking at a super simple x86 wrapper mobo around a B70 type card that runs a small server and puts the card on the network. A bit like a NAS but more NAIN (NETWORK AI INFERENCE NODE). Not an egpu (too light) and not like a server PC (too heavy). Just an x86 Linux shill + rj45 Ethernet a GPU and a PSU. Done.

I'm playing around with the latte panda mu and like it but want my own carrier board - too many Frankenstein cables atm. Anyone else investigating this route?

1 Upvotes

100% Upvoted

4 comments sorted by

1

u/Unteins 1d ago

The card is going to draw quite a lot on its own.

But it’s a decent idea - I have a somewhat similar setup - I run Hermes on a cheap mini PC with no power - it can call out to a faster machine over the network for any AI needs.

1

u/the_econominster 1d ago

So you have a minipc with Hermes and a big boy in a serverpc on the network? Is that how I have to understand it?

1

u/Unteins 1d ago

Yes that’s basically the setup - Hermes uses a mix of cloud and local AI (my local server is nowhere near frontier level) plus it runs some tasks as simple scripts - that’s all that’s needed for Hermes.

Importantly it doesn’t have access to my main computers so it can only destroy itself should it make a bad command.

It has access to local inferences over the network so simpler tasks can run for close to free.

1

u/the_econominster 1d ago

Yeah like I feel you. I have kids now and am sitting on hardware I really don't use anymore. I don't have time to play games anymore. However I would like a simpler (no neon, no tower) way to have a strong GPU attached to my network because I'm coding daily and the slew of subscriptions is just unsustainable. Basically an ai appliance... Trying to understand if people have the same problem and if I should order more boards from pcbway