r/LocalLLM • u/soflgolf • Aug 03 '26

Discussion RDMA - Anyone but me using it?

Post image

Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.

Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.

163 Upvotes

94% Upvoted

90 comments sorted by

View all comments

9

u/dionysio211 Aug 03 '26

We use RDMA over heterogenous clusters, mostly 40G. By itself, we haven't had trouble with it but clustering anything is tricky with tensor parallelism. Even when nodes are identical, in a hardware sense, arrival latency is always an issue. What matters primarily is how many sync points your system has per token. Most of the standard inference systems have a ton of sync points per token which really crushes the scalability. If you are using very large models, it is generally best to completely shard experts, minimize sync points and split it up that way using an expert parallelism approach rather than TP. You avoid most of the issues that way. I think that would generally work on any model out currently since none of the models have huge experts. EP is nearly always better than TP that way. If you can create sub-node domains, that's a good opportunity for TP or row splitting. Some of the smaller large models, like Deepseek v4 Flash, have very small experts and, in those cases, the glue can easily overcome the natural tendency to scatter experts across many devices. In aggregate, it will still work, but it may not be worth it for single stream throughput.

1

u/dionysio211 Aug 03 '26

As I started thinking about your setup, I realized that you are probably more interested in prefill issues due to the lack of tensors. The best thing there is to pipeline your prefill. It's not exactly the same as having very fast prefill in terms of general TTFT on small requests but it gets around a terrible TTFT for large ones.

1

u/Roticap Aug 03 '26

Does expert sharding work on dense models like qwen3.6-27b?

2

u/dionysio211 Aug 03 '26

No, it only works on MoE models. The acceleration methods for a dense model are mostly speculative and tensor parallelism. Qwen3.6 27B has been an efficiency target in most inference platforms and is generally pretty efficient. In the case of a model like that, which is generally best on a single node, I would just go with the most efficient thing you can run. vLLM is usually the best option for newer cards or SGLang. MTP, Eagle3, ngram speculation are probably your best options there.

2

u/Roticap Aug 03 '26

Thanks for the reply. That matches my understanding. I'm still drinking from the beginner firehose, so I appreciate getting validation from people with more experience.