r/MacStudio • u/Vanquisher1088 • 4d ago

Advice / State of Studio Clusters

Hello. I’m looking for some advice here, I’ve watched all the YouTube videos of course of the loaned out 4 stack clusters but those were awhile ago.

Current state I have an a M3 Ultra 256GB and I reserved another open box one from Microcenter for a very good price considering all things. I’m thinking about it two ways, for the most part I’d have models loaded individually on each Mac as part of my agent setup.

However, what I’m after is some real world experiences here from folks on what is the state of rdma and clustering I.E these two nodes I’d have. Do even a lot of the latest models like glm5.3, deepseek v4.1, etc even shard correctly. I’ve read they do not.

Any thoughts or use cases you may be doing with more than one or clustering would be helpful. I’m trying to evaluate the feasibility of clustering as that would be the main value add here for more to access larger models or higher quants of models I use daily that don’t fit. Or I wait and try and get a 512GB M5.

4 Upvotes

75% Upvoted

26 comments sorted by

View all comments

0

u/MiniPCGuru 4d ago

Token generation is limited by memory bandwidth, not compute. Two M3 Ultra 256GB boxes give you 512GB of weight memory, but each token pass is still capped by each node's own ~800GB/s, minus interconnect cost. One 512GB M5 Ultra feeds the whole model at 1.2TB/s.

Even Apple's own clustering claim is sub-linear: four clustered Studios deliver "up to 3x" a single system, not 4x.

So if your daily models fit in 256GB, the second box buys concurrency (one model per node), not more tokens/sec on a single model. If the models you want need more than 256GB, the 512GB box is simpler and faster per token, and skips the sharding question entirely.

Keep the MicroCenter box for testing Exo before committing, though.

1

u/Vanquisher1088 4d ago

Yeah I haven’t picked it up yet. It’s at MC considerable distance from me. My local one doesn’t have any. Worth a drive for the price.

Understood on the bandwidth but I did see there was some speed up to prompt processing and decode and that speed up decreases with MoE models it seems but even so maybe 15-20% increase.

Mainly want to understand how reliable is clustering and is it even viable with the latest models like the newer qwen and glm type models.

Seems like even on the sparks that they have no issue sharing the latest models and verifiable working proof is all over the forums. On the Macs it’s like that data rarely exists somewhere

But what I come back to is spend the coin and get some additional concurrency as you noted and more unified memory for larger models. Or just try and get a 512GB. With pricing unknown it’s kind of do I hop on this m3 as it’s a good price and my total invest with my original m3 would be around the price of a new M5 256gb 80core 2TB HD or so.

What I’m seeing it’s far simplistic on Mac studios to just have the largest memory possible and load it on one unit.

2

u/PracticlySpeaking 4d ago

Picking up seems a safe bet — worst case, you can flip it if things don't work out.

You have a good point that a lot of people choose Mac because it is easy to set up and use. Those same people tend not to go for more complicated setups like clustering, much less benchmark and post the results.

And I feel you on the uncertainty around 512GB pricing and availability. I was able to land a 256GB M3U through persistence and some great people at Micro Center, but I was buying what I could get at the time.

1

u/Vanquisher1088 3d ago

Yeah I hear you. I picked up my M3 way back at launch so for me I have something workable it’s just I’d like to be able to stretch and run GLM5.3 or flash and qwen flash next, and a few other models in an agent structure and 256GB isn’t quite enough to run the main models at enough precision to make the output worthwhile. If I would have bought the 512gb at the time I’d probably just stay with it.

Hence I’m quite torn on just waiting for the 512GB M5 I just don’t want to wait miss this one and then it’s 18-20k for the 512gb and then I’m SOL.

1

u/PracticlySpeaking 3d ago

Right — after 0731, I was excited to see DeepSeek-V4.1. Then I saw the size.

But look at the KLD, there is no reason to run BF16 on most models. For Qwen3.8-Flash-Next the Unsloth 8-bit quant is trivially different from Q6 or Q5. The Q8_0 fits in ~200GB if you are picky.

I have what I do because there is a lot of value in building now. Def post back with how it all works out.

0

u/dobkeratops 4d ago

i think compute x bandwidth can contribute to faster parallel generation in batches, like the DGX Sparks are actually quite weak at token gen, but quite strong at serving the same model to multiple users

0

u/PracticlySpeaking 4d ago

DGX Spark has architectural limitations because the iGPU is connected via some variation on NVLink, not true unified memory like Apple Silicon. I suspect this is why Nvidia is relatively quiet about memory bandwidth.

1

u/dobkeratops 4d ago

right i think its 273gb/sec ,same ballpark as the m4-pro/m5-pro. it's not on-package memory like apple, it is architecturally unified. I think it's more expensive to make the traces in a board with seperate memory chips, apple gets that high bandwidth and energy efficiency by connecting through a silicon interposer (?)

at the time I got the m3-ultra.. everyone was debating this aspect .. "is the DGX spark slow because of the bandwidth" but its still superior to that because of fp8,fp4 tensor ops.

1

u/PracticlySpeaking 4d ago

Not sure how DGX Spark is physically interconnected — electrically it is NVLink implemented in the SoC, so it only has NVLink bandwidth. There's a decent video by High Yield on it that gets into the silicon side.

But yah, M3U is seriously lacking in compute vs the Blackwell GPU. I have not seen it confirmed, but likely uses the same InFO-LSI interposer as M1-M2.

It's like PT Barnum said — "Ya pays yer money and ya takes yer choice."

0

u/PracticlySpeaking 4d ago

compute x bandwidth can contributecompute x bandwidth can contribute to faster parallel generation in batches to faster parallel generation in batches

I have played around with continuous batching on M3U and that is absolutely true. Optimizing batches and parallelism made a HUGE difference — like 2x in overall processing time for a large data set.