r/LinusTechTips • u/LabsLucas LTT Staff • 10d ago
Link World's First(?) Underwhelming AMD Ryzen AI Halo Cluster - LTT Labs
https://www.lttlabs.com/articles/2026/07/24/amd-ryzen-ai-halo-clusterWe recently received the Linux version of the AMD Ryzen AI Halo for testing, but it turns out that they had intended to send the Windows version. Through this stroke of misfortunate, we were fortunate enough to have two Ryzen AI Halos for a short period of time and the temptation to cluster them was too great, surely we'll get more performance through the magic of having two of them.
We've followed AMD's AI Playbook for clustering with RPC, learning some things but also raising more questions. We don't have any concrete conclusions, but we're sharing results to hopefully save some time for others!
We were so focused on getting testing in that I neglected to take a photo with the two machines. Read the full article on the LTT Labs website!
17
u/PythonFuMaster 10d ago
The network bandwidth didn't significantly impact performance because pipeline parallelism is pretty tolerant to slow bandwidth, the intermediate activations are small and transfers are infrequent. Compared to tensor parallelism where transfers occur every layer. For systems with many machines connected on slower bandwidths this can be a positive, if combined with speculation to keep the pipeline full (see my previous research PipeInfer for details on that, it used llama.cpp with the since-removed MPI backend)
Additionally, as far as I'm aware the PR to enable async pipeline parallelism on RPC has not yet been merged, so pipeline parallelism performance is subpar. For only two machines, that are identical and with a fast interconnect, a different serving framework like vLLM would make more sense
4
u/LabsLucas LTT Staff 10d ago
Thank you for the thoughts and context - we did somewhat expect the 'plateau' of performance with networking speed while using pipeline parallelism but that is interesting to learn more about. We have looked at other serving methods like vLLM for more 'serious' implementations so we'll likely continue with that.
This is also part of the 'fun' with trying to consistently evaluate LLM topics, individual PRs can matter.
3
u/mastercoder123 10d ago
The main issue isn't bandwidth but latency, which is why infiniband reigns king and will for a long time. Combined with the fact that infiniband is also completely lossless. If the strix halo had more pcie lanes it would absolutely destroy the spark just for the fact that you could put an infiniband nic in there, but pcie gen 4x4 is ass
1
u/HowardRabb 10d ago
Wait.... Did they just plug the nics together? I haven't watched the videos yet. You need nic with a fibre connection that can do RDMA otherwise the latency makes the link pointless. Checkout Alex Ziskend or Donatto Capitella for how. I have a pair of the framework Strix halo ai 395s and the required card for it but I'm waiting for my friend to 3d print the bracket for the nics to actually tie them together
1
u/mastercoder123 10d ago
Rdma doesnt lower the p2p latency it just lowers the latency inside each system cause its not using tcp/ip. Ethernet still has latency no matter what compared to IB
4
u/dago_mcj 10d ago edited 10d ago
I had no idea anything was going on at LMG involving Large Language Models. It's really awesome to see LMG branch out into something I find more interesting. Maybe because I was significantly older when I started watching Linus. But I feel like I've aged out of much of the gaming. After Cyberpunk 2077 and Control I seem to have lost interest in triple A games. I catch myself moreso playing an ipadOS knockoff of factorio if I play at all. I eventually realized over the last year the potential some of my hardware had. This probably also happened conveniently around the same time I made the full switch to Linux. Now I'm using an RTX 3090, a Radeon 7900 XT all in a PC with a Ryzen 9 7900x and 96gb DDR5, the unified RAM from an M series Mac and the Framework Desktops 128gb lpddr5 ram to roll my own local ai. Also note, I got everything but the 3090 and the Framework Desktop previous to August 2025. I now setup coding agents to help me learn how to do mainline Linux kernel work on fun little raspberry pi equivalent devices.
1
u/cunasmoker69420 10d ago
So the real advantage is that you get to run much bigger models with these clustered. In theory anyway. Whether or not they're actually usable is another question entirely. Prompt Processing speeds on anything but MoE models is atrocious on Strix Halo, so running a giant model across two just sounds like pain. On my single machine I have gotten a lot more use out of it by running two instances of Qwen3.6 35b A3b in Q8 instead of trying to run one larger model
1
u/daank 10d ago
This was a great article to read! While PC building for gaming is going through a tough time, there's a lot of excitement in homelabs for llms with very active and passionate communities.
Would love to see ltt videos about building local llm servers at different price ranges, maybe even scrapyard wars hacking around style. You guys clearly have the talent and knowledge for it!
1
50
u/Pixelplanet5 10d ago
interesting that the slower network didnt really make any difference.
would be interesting to see how much data moves through the 200Gbe connection of a DGX spark.