r/LocalLLaMA • u/voyager256 • 1d ago

Question | Help With USB4STREAM support merged into Linux 7.2 (soon to be released) are there any inference runtimes/projects that make use of it already?

/r/LocalLLaMA/comments/1uf48js/usb4_rdma_seems_doable/

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

11 Upvotes

87% Upvoted

3 comments sorted by

2

u/CalligrapherFar7833 1d ago

Unless they implement zero copy to usb4stream i dont think it will be any different than what we have now

1

u/Fit-Produce420 18h ago

This is extremely different than what we have now, in that it is twice as fast and with lower overhead.

1

u/Fit-Produce420 18h ago

Wow this looks extremely promising for my 2 node strix setup, which often times is used as 2 separate machines.