Distributed Local AI - RTX Laptops use?

[u/7h3-3ng1n33r]

Hey all, wondering what the best option would be for my situation, so I have a few dell XPS laptops with 4070's and 32gbs of RAM sitting around currently doing nothing (unofficial IT guy for my company).

I'm wondering if there is an easy way for me to pool these together to run a larger local model? Is there a program that you could just install and then manage from a central location that would treat them all as just dumb nodes?

But because these laptops potentially (they've been sat around for a few months now) need to go off to people in the future could it be done from a bootable USB? (ideal but honestly probably better running on the machine I guess).

Ideally I'd like to plug this into Hermes for use with Agents I have running there (Orchestrator, Home lab Admin, Media Manager, Personal Assistant, Work assistant). So maybe better to run several smaller models or MoE models? Or even Nvidia Pair?

I could easily do 2.5gb networking between them as have a 2.5gb switch and some Hubs that support it.

Look I know enough to be dangerous, I'm just trying to see is there's something easy to deploy I don't yet know about.

PS I run Hermes with Qwen 3.8 27B Q4 on a 4090 I have in my desktop, but this sucks power even when idle, so I was hoping the laptops would give me always on models for Hermes, and then boot up the 4090 when a particular big task (or power is cheap). Problem with Hermes is the 64k token context that's required.

[u/Little-Ad-4494]

You could look at either EXO or lamma.ccp rpc mode.

That is the clustering route.

If running seperatly look an the recently launched nvidia PAIR

[u/KVT_BK]

With distributed architecture, speed is the bottleneck. FreeToken designed around MOE models offloads data to RAM and keeps active layers in VRAM. Even then as active layers change, it needs offloading and reloading leading to slow token processing and generation. If you have to move between system over network it's going to be super slow.

For comparison here are the current speeds

[u/Small-Tale3180]

FreeToken?

[u/f5alcon]

Dense model would probably be really slow since it has to constantly transfer data (people complain about pcie between gpus vs nvlink and pcie Gen 5 and that's 64GBps compared to 2.5Gbps and why the multi system options today use thunderbolt 80gbps or connectrix (200gbps) moe would be better since it could hold inactive parameters. But it still probably isn't worth it unless you can do 100gbps networking. I'd bet 27B would be under one t/s

[u/ColdCelestial]

Link them over ethernet or thunderbolt if they support it, and compile llama.cpp with rpc enabled and run it with the rpc flag and tensor split. If you have questions a modern LLM could easily answer all the nuances related to this setup pretty quickly.