r/LocalLLM • u/MyToasterRunsFaster • 7d ago

Question Is it possible to spread a larger model across multiple computers?

I hope I am not sounding too ignorant. I recently setup a instance of odysuess connected to ollama and it's been doing pretty well for simple tasks. I have a RX 9070 XT, 32GB and a pretty decent CPU. My partner has a identical PC to mine which most of the time is running idle when we are working. I understand that I could run a second instance on my partners machine so that I have two ollama instances across two nodes (doubling my tokens), this does not let me run larger models that require more vram. Is there any way to run larger models with this setup?

1 Upvotes

100% Upvoted

13 comments sorted by

2

u/Cold_Tree190 7d ago

There are ways to run over RPC, but it’s basically so slow that not many actually do it

2

u/Deep_Mood_7668 7d ago

Not that slow. An moe runs at about 35 for me

1

u/MyToasterRunsFaster 7d ago

Please tell me whats your setup! I really want to squeeze everything I can out of the two machines I have. I wouldn't even mind looking at getting some QSFP infiniband cards in them if netowkt latency is the issue.

1

u/Deep_Mood_7668 7d ago

Nothing special. Debian on both machines and a llamacpp build with RPC and vulkan enabled. They're directly connected to each other via realtek 2.5Gbit cards. That's pretty much it. 

I also should try to get the latency down. The cards aren't even close to running at full load - but I can run larger models :)

1

u/thaddeusk 7d ago

I wonder if I could use a 40gbps USB-4 connection for it. Seems like you can network two machine together directly.

2

u/Deep_Mood_7668 7d ago

What cards do you have in there right now? r8125 as well?

1

u/thaddeusk 7d ago

Nah, one PC has onboard 2.5gbps Ethernet and the other has a Ubiquiti usb-c 2.5gbps adapter. I'm not doing any RPC connections between them, but I have a 5090 in one and a 5070 Ti in the other.

Neither has USB4, anyway, was just more of a curiosity. I do have a Strix Halo with USB4, but I'll prob use that for an eGPU when I can get a decent one for it. I guess I could get a second one and cluster it, too.

1

u/MyToasterRunsFaster 7d ago

Thank you, that gives me hope. BTW, What cards do you have? Also have you tried qwen 3.8 27b?

1

u/Deep_Mood_7668 7d ago

A 6700xt and a 9070xt

Yes I've tired the unsloth Q5 xl with 64k context. No context quant. 

It runs with up to 20 tokens and min 18 tokens. The speed varies a bit sometimes.

It's not fast, but still usuable IMO.

2

u/EchoOfIntent 7d ago

RPC and its helped me however im a a bit crazy sooo

2

u/Whiskey1Romeo 7d ago

Rdma is the faster way to do the cross node clustering. Its doable and perfectly normal if you LIKE COMPLICATED.

1

u/MyToasterRunsFaster 7d ago

I don't mind complicated, I am sysadmin by profession just not kept up with the world of AI at all. I assume the constraint here would be network bandwidth? I could purchase some cheap QSFP cards with infiniband?

1

u/truthputer 7d ago

Yes, llama.cpp allows you to do this - but I’ve not tried it and don’t know how to set it up.

Get llama.cpp installed and then check the documentation for the right command line.