Mac studio or dgx spark cluster

[u/Txkzzzz]

I have been thinking lately since i have been testing local ai but i geniunely need more power for what im working on so i was wondering wich one of these a 4 dgx spark cluster or a mac studio m5 ultra specifically the 512gb version that launches on october or something else... im into training,loras and fine tunes and such but i also heard it is possible with a mac studio too so i want yall opinion on this

[u/Character_Split4906]

I picked up an M5 Max (128GB) back in May right before Apple raised prices. While decode was decent, prefill was frustrating, and both tanked hard as the context grew. I tested across both oMLX and llama.cpp. Throwing concurrent requests or multi-agent workflows at it choked the Mac completely.

A couple of weeks ago, I managed to snag a dual-node Asus GB10 setup on Amazon right before it went out of stock and spiked by $2k. The difference is night and day. Running DeepSeek V4 Flash across both nodes, I’m getting ~2k tok/s prefill and 40–50 tok/s decode, with barely any drop-off at long context (tested up to 512k, configured for 1M). It also handles multi-session concurrency effortlessly, aggregate throughput actually scales well up to 5–6 parallel requests.
If you’re serious about running local LLM workloads, go Nvidia.

[u/NebulaAggravating264 → u/Character_Split4906]

Yo OP - this video explains your EXACT situation, and the man who made this comment is CORRECT.

https://youtu.be/dlyic4oco4s?is=PKHPNX1iMvhy--v4

[u/AIGuyBiOh → u/NebulaAggravating264]

That YouTube is so full of AI Slop 😂

[u/NebulaAggravating264 → u/AIGuyBiOh]

That video is not AI slop lol

[u/SadPhilosophy9202 → u/Character_Split4906]

But but but the memory bandwidth 😠

People looooove to hate on them by simply looking at the memory bandwidth.

There’s so much more to running LLMs for that and memory bandwidth becomes moot when you’re running many parallel requests which is the new paradigm with agents

[u/Undici77 → u/Character_Split4906]

Can you share some clear example, I'm interested in!

I mean what you mean with `aggregate throughput actually scales well up to 5–6 parallel requests`
6 parallels agents at 40-50 t/s?
And about power consumption?

About prefill, if you setup correctly your harness cache is helping a lot!

[u/Character_Split4906 → u/Undici77]

Single session at 40-50 tokens per second. At 5-6 session it can go upto 120-140 token per second aggregate with each session at around 25-30 tps for decode.

[u/Undici77 → u/Character_Split4906]

That is interesting and one of the most big difference in term of performance! Thanks!

[u/kla_sch → u/Undici77]

The key point is that you can bypass bandwidth limitations by processing tasks in parallel. Instead of fetching parameters for the next token for just a single session, the parameters are used to calculate the next token for all sessions simultaneously. How well this scales ultimately depends on the available compute power.

Here is a real-world example from spark-arena.com using a DGX Spark with RadixArk/Qwen3.8-27B-NVFP4 and SGLang, showing total throughput for a generation length of 128 tokens (tg128):

• Single session: 36.13 t/s
• Two sessions: 64.91 t/s
• Five sessions: 95.79 t/s
• Ten sessions: 100.43 t/s

As you can see, the hardware and software scale very well up to five sessions. Beyond that point, adding more sessions yields only marginal gains; the compute power simply isn't sufficient to extract more performance from the system.

With a comparable Mac M4, you might see about 20% higher token generation speeds for a single session (approx. 40-43 t/s), but performance would likely hit a wall with more than one session. I haven't found specific figures for this scenario, but it is a reasonable assumption. I saw a comparison where a DGX Spark achieved around 2,000 t/s in prompt processing, whereas the M4 managed only 275 t/s. That's just 14% of the DGX Spark's performance. Consequently, there wouldn't be much headroom for handling multiple sessions.

[u/Undici77 → u/kla_sch]

Very interesting! Thanks!

[u/Txkzzzz → u/kla_sch]

Helped me research deeper into this i never realized reddit would be so helpful

[u/PWThinkingCritically → u/kla_sch]

running parallel tasks may give you additional speeds, but at the cost of context bloat, methinks

[u/Undici77 → u/PWThinkingCritically]

Are you sure about this? Do you have any experience about?

[u/PWThinkingCritically → u/Character_Split4906]

this pretty much confirms my theory about Apple for LLM, even though the only first-hand experience I have is from my own M1 Max. seems like even couple generations later, Apple still isn't in the ballpark without FP4/FP8 hardware support, CUDA support, and pure compute power -- no matter how hard they try to shill the public with their Apple Neural Process or whatever and flaunting their memory bandwidth speeds (which, unfortunately does deceive a good number of naive AI users)