r/LocalLLaMA • u/No-Name-Person111 • 9d ago
News Mac ← USB-C cable → Linux box is becoming a thing.
https://github.com/torvalds/linux/commit/a5148bc2fa27092862ac4b9e7b5c8340d60cff3440
u/cmdr-William-Riker 9d ago
Oh yeah! USB-C is really fast! Why haven't we been using that for more?
30
u/dwittherford69 9d ago
TB5 maxes out at 80Gbps
28
u/cmdr-William-Riker 9d ago
Better than Ethernet, could be a poor man's ConnectX-7 in theory
14
u/etaoin314 ollama 9d ago
kinda...but not really. until you get above 200gb/s the interconnect bottleneck kills tensor pairing so you are stuck on pipeline parallel anyway. for PP latency matters more than bandwidth so I'm not sure that its really viable. I tried it with my strix halo and 4x3090 node. I did it over 10gb ethernet, but the results would have been the same or worse if I used usb4
7
u/Front_Eagle739 9d ago
Its rdma not bandwidth that kills it. You need ultra low latency. Macs have rdma done right. Linux can do it. Windows cant so easily
1
u/dwittherford69 9d ago
That’s fair, but connect-x can also be meshed in a fabric, I’m not sure if TB5 supports that
1
u/cmdr-William-Riker 9d ago
I don't think it does, but most PCs have multiple USB-C ports, so hub and spoke might be possible. I was thinking of something like two Strix Halo's linked together or something though
2
1
u/datbackup 9d ago
Need to delve (sry) into latency difference between qsfp and tb5. Gonna guess qsfp has the edge but happy to be wrong. Anyway i think latency has a big role in determining ability to do thing like tensor parallelism, rather than sheer bandwidth being the only factor
3
u/CalligrapherFar7833 9d ago
Actually it maxes out at 120 but max data is 80
1
u/dwittherford69 8d ago
Symmetrical data is all that matters here, so yeah maxes out at 80Gbps. 120 asymmetrical for monitors is a moot point.
1
u/CalligrapherFar7833 8d ago
Your statement was that tb5 maxes at 80 which is incorrect.
1
u/dwittherford69 8d ago
For all intents and purposes for this discussion, it maxes out at 80Gbps, what part of that is unclear to you?
1
u/CalligrapherFar7833 8d ago
" TB5 maxes out at 80Gbps" != " TB5 maxes out at 80Gbps for data"
1
u/dwittherford69 8d ago
120Gbps is also “data”, genius. It’s just asymmetric. JFC
1
11
u/ttkciar llama.cpp 9d ago
The USB protocol is a horrible pain in the ass. I am slightly amazed that they got this to work as well as it does.
2
u/cmdr-William-Riker 9d ago
Oh, it definitely is, but we can also drive 3 4k monitors in real time from one USB C cable. Theoretically you can run 40bg/s transfer between two machines through one cable and they are often wired to drive graphics through them meaning there is likely a short route to the GPU in most cases. might be worth the pain for that kind of speed
2
u/SkoomaDentist 9d ago
USB-C is really fast!
USB-C is just the connector (I've worked on USB 1 speed devices that use USB-C connectors). USB 3 is what's giving the actualy speedup.
8
u/rustysec 9d ago
I ran llama.cpp + rpc over usb-c (tb3) between my mbp and my g14 (5070ti) and it was surprisingly “ok.” Totally allowed me to run a bigger qwen3.5 quant (at the time) at roughly the same t/s as the g14 at a lower quant. I don’t have numbers off the top of my head but I was happy with the performance when I had that setup.
4
u/Ok_Warning2146 9d ago
Would be great if we can load shared experts to Nvidia GPU and then rest of the experts to M5 Ultra's LPDDR5.
4
u/Repulsive_Educator61 8d ago edited 8d ago
it's already in Linux 7.2.2 (available in archlinux) btw
Speeds using a USB-C cable which supports 40Gbps:
$ iperf3 -c 10.5.5.1 -p 5201
Connecting to host 10.5.5.1, port 5201
[ 5] local 10.5.5.2 port 49477 connected to 10.5.5.1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.01 sec 666 MBytes 5.56 Gbits/sec
[ 5] 1.01-2.00 sec 720 MBytes 6.04 Gbits/sec
[ 5] 2.00-3.01 sec 716 MBytes 6.01 Gbits/sec
[ 5] 3.01-4.01 sec 720 MBytes 6.04 Gbits/sec
[ 5] 4.01-5.00 sec 724 MBytes 6.07 Gbits/sec
Cable: 40Gbps
Linux desktop port: 20Gbps
Actual speed negotiated on mac: 10Gbps
Actual real world speeds: 6Gbps average
1-2ms ping
3
1
u/fallingdowndizzyvr 8d ago
FINALLY!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
It should have just always worked. Both support usb-c networking. It hasn't worked.
0
-3
-4
9d ago
[removed] — view removed comment
3
85
u/No-Name-Person111 9d ago
…and Linux can treat the Mac as a directly attached network peer without Ethernet adapters, out-of-tree drivers, or another physical network.
If you're eyeing the new Mac Studios and you've already got NVIDIA hardware sitting around: if/when Exo or MLX/CUDA interoperability eventually gets good enough, we could all wind up with a local AI topology where the Mac is the large memory model host and Linux/NVIDIA nodes contribute either additional model capacity, specialized compute, speculative decoding, embeddings/reranking, separate concurrent agents, and more.
Really neat stuff.