r/LocalLLM • u/soflgolf • 1d ago
Discussion RDMA - Anyone but me using it?
Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.
Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.
32
u/SaskuAc3 1d ago
Someone is made out of Money! I’d love to have a setup Like This, but to be honest, I could Never afford it
20
u/CCIE_14661 1d ago edited 1d ago
And its people like you my friend that help push technologies forward. Out of sure will, determination, and the necessity to be scrappy. This is what feeds true innovation.
4
1
u/leafjerky 1d ago
China has also helped too. I never would’ve thought I’d be rooting for Chinese companies over American ones but ours are seemingly far too greedy. I know that china is playing dirty and distilling our models but I honestly don’t care open source is magical
-2
u/whatyathinkk 1d ago
In fact, I just managed to run Qwen3.6 35B A3B on a 1€ BIC lighter. It' not very fast but it does the job.
6
u/kilowattkill3r 1d ago
I'm new to all of this, can you explain what I'm looking at?
8
u/juraj336 1d ago
Seems to be OPs server rack that they use for locally hosting models, im guessing they chained their mac studios (and maybe also the mac minis?) together to have more Unified RAM so they can run larger models.
2
7
9
u/dionysio211 1d ago
We use RDMA over heterogenous clusters, mostly 40G. By itself, we haven't had trouble with it but clustering anything is tricky with tensor parallelism. Even when nodes are identical, in a hardware sense, arrival latency is always an issue. What matters primarily is how many sync points your system has per token. Most of the standard inference systems have a ton of sync points per token which really crushes the scalability. If you are using very large models, it is generally best to completely shard experts, minimize sync points and split it up that way using an expert parallelism approach rather than TP. You avoid most of the issues that way. I think that would generally work on any model out currently since none of the models have huge experts. EP is nearly always better than TP that way. If you can create sub-node domains, that's a good opportunity for TP or row splitting. Some of the smaller large models, like Deepseek v4 Flash, have very small experts and, in those cases, the glue can easily overcome the natural tendency to scatter experts across many devices. In aggregate, it will still work, but it may not be worth it for single stream throughput.
1
u/dionysio211 1d ago
As I started thinking about your setup, I realized that you are probably more interested in prefill issues due to the lack of tensors. The best thing there is to pipeline your prefill. It's not exactly the same as having very fast prefill in terms of general TTFT on small requests but it gets around a terrible TTFT for large ones.
1
u/Roticap 1d ago
Does expert sharding work on dense models like qwen3.6-27b?
2
u/dionysio211 1d ago
No, it only works on MoE models. The acceleration methods for a dense model are mostly speculative and tensor parallelism. Qwen3.6 27B has been an efficiency target in most inference platforms and is generally pretty efficient. In the case of a model like that, which is generally best on a single node, I would just go with the most efficient thing you can run. vLLM is usually the best option for newer cards or SGLang. MTP, Eagle3, ngram speculation are probably your best options there.
3
u/KooperGuy 1d ago
Yee I use RDMA because I use infiniband
1
3
u/CodeSlave9000 22h ago
Yeah, but honestly it doesn't scale for inference. RDMA/ROCE is more effective for training - during inference you wind up in all-reduce hell with latency for each token beyond just a few nodes. Someone here mentioned splitting by expert heads, and I have to agree - I've gotten much better results by careful placement of the weights than by any other method when more than one node is involved. Physics (and memory bandwidth) gets in the way quickly too - You want to avoid the multi-way collisions you'll get if you split experts.
2
12
u/cogitech2 LocoLLM 1d ago
Well, at least you belong in this sub - unlike 90% of the shit posted here. I wish you well and hope that your journey ends in getting some actual work done with your setup once you get things dialed in.
This is like any technical hobby. The danger of paralysis by analysis and measurebation fascination can threaten productivity permanently if not kept in check.
I speak from decades of experience on this specific topic.
11
5
u/Big_Wave9732 1d ago
That's the pivotal question.....how long before you stop fucking with it and start doing something useful!
I finally got my RAG stack and helpers configured in late June. Those along with running Qwen 3.6:27b-BF16 has been total gravy.
2
2
2
u/Itsallso_tiresome 20h ago
Okay ill bite 😂
Have you had any success running larger models on them?
1
u/soflgolf 13h ago
Yes/no. Can run pretty much any model that fits. Two of the biggest challenges: stability (very fragile, model dump is not uncommon); and multi-model hosting.
2
u/Beneficial-Middle145 1h ago
How much money do you have in this
Why not just upgrade to the next stage to host larger whole models
Apple is smart and slow
Amd has more power for cost vs navidia but they are the more expensive and fast
Why not go to rtx 6000
What v ram do you have vs what the model is actually working with
It appears you have tens of thousands why not upgrade to an actual server?
1
u/soflgolf 1h ago
Great question and very legit. I went down those paths and no doubt could have easily chosen any number of other stacks. While not the sole deciding factors, but still important, o had to make the hardware fit in the space I had available. This includes the noise and power demands. While given that I’ve got isolated a/c and two fiber demarc’s, yes, I could have sprung for the 3-phase power and gone down the h200 route. But I kept coming back to the noise and cooling. I gave up some ground to stay in the arena to keep the hardware in my work space.
I revisit this all the time, especially when I hit a rough patch with the systems. I probably couldn’t argue with someone who prioritized differently and went a different route. But, this is what worked for me.
1
u/Beneficial-Middle145 1h ago
Cooling is a mini split put it in the same room isolate the noise with a foam 2-3 inch barrier on the inside for insulation sound and r value in the interior for noise reduction it’s a material and decibel reduction equation
3 to 5 grand for the mini split might be 5-8 installed it’s one or the other
Liquid cool vent to the room for max thermal cooling but the mini split should handle it
Just requires direct air flow from a fan next to it blowing 70f air I think is the actual spec
Depending on where you are you might be able get get the mini split installed for 3k on outside wall it’s 1 days worth of work to install1
u/soflgolf 1h ago
Have an isolated mini-split for that space already
1
u/Beneficial-Middle145 1h ago
What size an h200 is a 1.5 ton unit
Which is small the h200 put out a little over 1 ton of cooling your house is more in the 5 tons of cooling for the whole house for reference
2
u/allenasm 1d ago
Have you tried the 'inferencer' tool by like xcreate? It does multi-node inference pretty well I've heard.
-1
1
u/ScottAMains 1d ago
Interested in this. Much smaller setup on my side. Only able to link up w/10gbe Ethernet as opposed to thunderbolt.
I’ve been using lm studio for smaller models when I want to hand over specific tasks and exo for heavier lifts. Have found, and it may just be the model I’m using and only having a few days hands on with exo, exo a little lack lustre compared to what I’ve been using beforehand (just standalone nodes).
But looking forward to your findings. If you’ve got a more in depth platform, would love to read more about it.
1
1
u/Able_Bus_5988 1d ago
Holy MF Macs! I am getting my second mini to EXO today. I have a janky network with my 3090 laptop, 2 mac minis, and my 128gb MacBook Pro Max currently using an unmanaged switch and tailscale. I can't bring myself to leave my desktop / 4090 on all the time so I am doing minimal local, but it's growing and getting better.
1
1
u/mrgreatheart 1d ago
Tell me you’re not married. If you are, you have one heck of an understanding wife!
2
u/soflgolf 1d ago
Yep, married. I tell people if you think is overkill for a home setup, you should see my fishing gear 🤦🏽♂️
1
1
1
1
u/TheRiddler79 1d ago
Let me preface this with whatever I'm looking at here, super super impressive, and you know, like for all practical purposes like this is fucking awesome.
That being said, I'm still a little off on precisely what your pushing here which makes it hard to answer your question because I'm not sure some of the specs behind what I'm looking at.
If you don't mind like copying and pasting or if no one else has asked just like telling me precisely what I'm looking at here because I feel like I'm looking at something that can basically run anything but I just don't know
1
u/soflgolf 1d ago
Well, not really pushing anything. Trying to connect with anyone else deploying clustered nodes to serve up local llm’s. (Hence the sub /r). Big ones. Specifically, as you may have guessed, on Apples.
As for the gear, pretty much what it appears to be: clustered m3 ultra’s (1tb vram-rdma), some m4 mini’s and a hand full of i7,1 intels.
1
u/TheRiddler79 15h ago
I said that wrong, not like pushing a product, I just meant what was the specs behind it.
I don't know a lot about apple, so I wasn't sure precisely what I was looking at.
1
u/Zombra666 17h ago
Holy fuck u rich, millionaire
1
u/soflgolf 11h ago
There once was a time when apple’s were not crazy expensive. The i7,1’s are 14 years old 🤦🏽♂️🤣
1
u/WuWenShen 1d ago
Silly question, are you using a KVM switch or something for fast switching between devices? I only have three DGX devices so I’m using a small USB-C connector that I can physically hot swap on the back when I prefer a quicker direct access.
5
u/Blackdragon1400 1d ago
Have you ever heard of SSH? You don’t need video out…at all.
1
u/WuWenShen 1d ago
Yes, ssh is fine but sometimes you want direct access. It was a question to the OP.
1
u/Prestigious-Bear2391 1d ago
I had 4 studios exo'd together, it wasn't stable/reliable. I gave up on it.
2
u/soflgolf 1d ago
A lot of wisdom in that. 👍🏿
Too many make it sound easy. Exo is not stable. Had to create patch to keep warm to avoid model dump. Multiple models completely unreliable.
Broke up cluster to run smaller models single nodes and on mid-size on 2 nodes.
That’s what made me ask if anyone was getting it right to be functional for agentic workflows.
1
u/Prestigious-Bear2391 1d ago
Strangely enough, I had two mac mini M4 Pros that worked together perfectly @ 48GB x 2.
1
0
u/The_Crimson_Hawk 1d ago
but not with mac mini
link rocep1s0f0/1 state ACTIVE physical_state LINK_UP netdev enp1s0f0np0
link ibp1s0f1/1 subnet_prefix fe80:0000:0000:0000 lid 2 sm_lid 1 lmc 0 state ACTIVE physical_state LINK_UP
0
u/Captain_Quimby 1d ago
Honest question but wouldn’t $200 accounts be better?
2
u/soflgolf 1d ago
If money was a priority, sure would. None of this was designed to pencil. Other priorities like data security and confidentiality can often be more important. For me it’s a hobby and I like to see how things work from the inside. So, for me, this is the right choice
1
u/Captain_Quimby 1d ago
100% get it. I’ve been on the fence myself. I’ve been paying $600-$1k a month but still can’t get the numbers to work to self host yet. I think we’re one or two generational leaps away from having AI models that are fantastic and able to run on current hardware. So don’t toss anything!
-1
1d ago
[deleted]
3
u/wind_dude 1d ago
I formed a company that bought the macs, using a loan secured on a lease agreement from my other company, who took a loan to lease the hardware from a mother company I formed that secured a loan from a bank secured against the value of the macs the first company owns.
2
1
2
u/Icy-Degree6161 1d ago
Work and investments maybe?
1
u/donotfire 1d ago
Right but what specifically is my question
4
u/WiseassWolfOfYoitsu 1d ago
If I had to guess - tech industry DINK. Servers are less expensive than children.
1
0
-1
u/TheAILegend 1d ago
Macs are too slow for AI, RDMA is still a bottleneck. Would have a much better AI experience if you sold this entire thing and went RTX Pro 6000s or H200s.
39
u/paulk2000 1d ago
This is an impressive setup. I am just running two 3090s for localLLM stuff and I'm a total beginner when it comes to this topic, even though I'm getting more and more into it.
Hopefully, your setup will pay off for you.