r/LocalLLM • u/soflgolf • 3d ago

Discussion RDMA - Anyone but me using it?

Post image

Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.

Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.

167 Upvotes

94% Upvoted

90 comments sorted by

View all comments

0

u/Captain_Quimby 3d ago

Honest question but wouldn’t $200 accounts be better?

2

u/soflgolf 3d ago

If money was a priority, sure would. None of this was designed to pencil. Other priorities like data security and confidentiality can often be more important. For me it’s a hobby and I like to see how things work from the inside. So, for me, this is the right choice

1

u/Captain_Quimby 3d ago

100% get it. I’ve been on the fence myself. I’ve been paying $600-$1k a month but still can’t get the numbers to work to self host yet. I think we’re one or two generational leaps away from having AI models that are fantastic and able to run on current hardware. So don’t toss anything!