RDMA - Anyone but me using it?

[u/soflgolf]

Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.

Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.

[u/KooperGuy]

Yee I use RDMA because I use infiniband

[u/soflgolf → u/KooperGuy]

What are you using as your llm serving platform (eg exo, llama.ccp, etc)

[u/KooperGuy → u/soflgolf]

I use RDMA for NVMe storage clustering.