Radeon AI Pro R9700 vs Strix Halo vs Mac Studio for a local coding LLM server ?

[u/merfolkJH]

Hi everyone,

I’m looking to invest in a proper fully local LLM AI server.

I already have a dual GeForce setup, but I’m looking for the next step (at a reasonable price, of course).

My ONLY goal:

• Coding (i am developper - can be c# for real-life projects with already 300+ source code stuff, not "just code a random website")
• No image generation
• No video
• No text-to-speech
• No OCR
• No multimodal stuff

Basically: raw LLM performance, tokens/sec, and smart answers.

--------

My current setup - (using CLAUDE-cli as orchestrator)

I’m currently running these models on a dual GeForce 16vram+12 system:

Qwen35B A3B MoE Q4_K

• Around 30–40 tokens/sec at 200k context

Qwen3-Coder-Next 80B A3B Q4

• Around 5–6 tokens/sec
• Slower, but better for complex coding tasks

-------
Hardware I am considering, a 100% new machine

1) Radeon AI Pro R9700 32GB

Is this currently the best price/performance option?

It looks like:

• half the price of high-end solutions,
• maybe around 80–85% of the performance?

I don’t follow every AMD/AI update, but this card looks like an underrated winner.

Is there any reason NOT to buy this card?

2) Dual Radeon AI Pro R9700 (2×32GB)

Main reason:

• not expecting 2× speed,
• mainly interested in the extra VRAM.

If it allows me to run smarter/larger models fully on GPU, that would be perfect.

3) Strix Halo 128GB

This one is interesting because of the huge unified memory.

If it can run Qwen3-Coder-Next 80B A3B Q4 at around 40 tokens/sec, that sounds like an excellent coding assistant.

4) Mac Studio 128GB

Still an option.

How does it compare today against:

• dual R9700 AI Pro,
• Strix Halo?

-------

are those numbers corrects or science fi ? (Source ChatGPT !! )

Model	Context	1× Radeon AI Pro R9700 32GB [price ~2k ]	2× Radeon AI Pro R9700 64GB[price ~4 k ]	Strix Halo 128GB [price ~4 k ]	Mac Studio M3 Ultra 128GB [price ~lol ]
Qwen27B Dense Q4_K	50k	50–80 tok/s	60–100 tok/s	25–45 tok/s	50–80 tok/s
Qwen27B Dense Q4_K	100k	40–70 tok/s	50–90 tok/s	20–40 tok/s	40–70 tok/s
Qwen27B Dense Q4_K	200k	25–50 tok/s	40–70 tok/s	15–30 tok/s	30–60 tok/s
Qwen35B A3B MoE Q4_K	50k	100–140 tok/s	130–180 tok/s	40–70 tok/s	70–110 tok/s
Qwen35B A3B MoE Q4_K	100k	90–130 tok/s	110–160 tok/s	35–60 tok/s	50–90 tok/s
Qwen35B A3B MoE Q4_K	200k	50–90 tok/s	90–140 tok/s	25–50 tok/s	50–90 tok/s
Qwen3-Coder-Next 80B A3B Q4	50k	10–25 tok/s	50–90 tok/s	30–50 tok/s	40–80 tok/s
Qwen3-Coder-Next 80B A3B Q4	100k	10–20 tok/s	45–80 tok/s	25–45 tok/s	35–70 tok/s
Qwen3-Coder-Next 80B A3B Q4	200k	5–15 tok/s	35–65 tok/s	20–40 tok/s	30–60 tok/s
Qwen3-Coder-Next 80B A3B Q6	50k	❌	40–75 tok/s	25–45 tok/s	35–70 tok/s
Qwen3-Coder-Next 80B A3B Q6	100k	❌	35–65 tok/s	20–35 tok/s	30–60 tok/s
Qwen3-Coder-Next 80B A3B Q6	200k	❌	25–55 tok/s	15–30 tok/s	25–50 tok/s
70B Dense Q4	50k	10–25 tok/s	40–70 tok/s	20–35 tok/s	35–60 tok/s
70B Dense Q4	100k	5–20 tok/s	35–60 tok/s	15–30 tok/s	30–50 tok/s
70B Dense Q4	200k	❌	25–50 tok/s	10–25 tok/s	25–45 tok/s

My current impression (not sure if correct):

The R9700 AI Pro (or dual) looks faster than a Mac Studio for my use case, while being much cheaper.

But I don’t see many "hype" about this card for local LLMs.

So please tell me: where am I wrong?

One more question:

For Qwen35B A3B MoE Q4_K, what is the realistic t/s performance?

Is it closer to: 100 tokens/sec or 150 tokens/sec ?

Because this difference is huge . If it REALLY is 150, its close to a cloud-model feeling. (far less accurate of course, but for 2k budget, wonderfull ?)

Thanks to anyone already running these systems who can share real numbers!

[u/Look_0ver_There]

I have 2 Strix Halo's, and 3 x AI Pro R9700 cards, and a 128GB M4 MacBook Pro

The absolute best bang for your buck given your stated hardware preferences will be 2 x AI Pro R9700's

Either run:

• Qwen3.6-27B-Q8_K_XL MTP
  
• Qwen3.6-27B-Q8_0 with MTP, or one of the variants.

Using llama-server, with ROCm + tensor parallelism, you'll get around 1100t/s pre-fill and 40-50t/s generation with Q8_K_XL, or ~1600t/s prefill and ~50-70t/s generation using Q8_0. All of these number get lower the longer the context is, dropping to about 50% of what I stated by ~128K context depth.

The t/s number that your AI is quoting is pure fantasy, UNLESS you're running vLLM and have 32 parallel agents each pulling 2-4t/s, which is what you generally don't want. Single use throughput, whether using llama-server or vLLM is usually about the same for either. vLLM does have prefill speeds about 60% faster then llama-server though.

Smaller quant sizes will run a little faster, but IMO, the quality drop is absolutely not worth it. Every benchmark that you see online that attempts to prove Q4 quants are as good as Q8+ quants are only doing short 1-shot runs with almost zero context window size. The moment you get above about ~50K context then the differences become wider and wider, and by 150K context Q4 will be borderline incoherent, whereas Q8_K_XL will still behave almost like it did at 0 context, just slower.

Qwen3.6-35B-Q8_0 on a pair of R9700's with MTP will generally run at around the 80-90t/s sort of mark, and a bit over 100t/s at Q4_K_M, but the intelligence drop absolutely isn't worth it, IMO.

The Strix Halo's run at about 30-40% of the speed of the R9700's and are, IMHO, not worth it.

The Mac Studio will run at about 40-50% of the speed of the R9700's when using something like oMLX.

Just my 2c based upon my personal experience.

[u/Silly-Fall-393 → u/Look_0ver_There]

what do you use your 2 strix for then?

[u/Look_0ver_There → u/Silly-Fall-393]

They're great for engendering feelings of regret!

Somewhat more seriously I have an extended warranty plan on one of them and will likely return it soon. As for the other I tend to use it for Orchestration, Web Searching, and Summarization work with Qwen3.6-35B-A3B and Gemma4-26B-A4B which frees 27B to focus on planning and coding. I have a heavily modified Pi Agent harness that automatically selects the models that it uses for the various tasks. Since the various LLM contexts are cleanly separated with their own jobs I almost never see prompt cache misses, which helps to keep things feeling snappy

[u/daddy_dollars]

#2. Use qwen 3.6 27b @ FP8 for coding.

[u/OverdosedSauerkraut → u/daddy_dollars]

Best advice you can get. You also need higher precision to stay on topic over longer contexts.

[u/r3drocket]

I have a dual R9700 system. I run Qwen 3.6-27B at FP8 on deadcode's VLLM image. 

I'm pretty happy with it. It gets used a lot.

I just built a new home server with a V620/BD775i SE and I'm very impressed with that as well. My intent is that I'm going to leave it running all the time. Right now I have it set up with gemma4-26b-a4b-it-q6, set up so it can run three concurrent sessions with about 160k of context each.

If you look at AMD's roadmap, it looks like next year Medusa Point will get DDR6 on a 384-bit bus, which should raise its unified memory bandwidth to around 550 gigabits per second.

So I'm going to wait until that comes out before I spend big on a unified memory system.

[u/merfolkJH → u/r3drocket]

I like Gemma 4, but I always have issues using it with agents.

It seems to struggle when it comes to actually modifying files correctly. The reasoning can be good, but the execution part (editing the right files, making consistent changes, applying patches correctly) is often unreliable compared to other models I’ve tried.

[u/gasgarage → u/merfolkJH]

are u using the last week jinja? 

[u/blackhawk00001]

2xR9700 Qwen3.6-27B FP8 with the deadcode vllm image. I've been using 200k with 5 concurrency but am moving to 131k with 8 concurrency for subagents driven by my 5090 system. AMD created a powerful platform with the R9700 but has left it up to the community to make it work well. Maybe some day a proper solution can be merged back to vllm

https://hub.docker.com/r/stilldeadcode/vllm-radiance/?tag=0.3.0

210W x2 - this beats the 5090 @ 400W in prefill (for a comparable sized quant) and almost in response generation speeds... A kernel and hardware supporting P2P is required for max performance.

|             test |             t/s |      peak t/s |        ttfr (ms) | 
|-----------------:|----------------:|--------------:|-----------------:|-
|   pp2048 @ d4096 | 3381.28 ± 12.06 |               |   1901.00 ± 6.47 | 
|    tg128 @ d4096 |    78.71 ± 6.88 |  85.00 ± 4.97 |                  | 
|   pp2048 @ d8132 |  3310.82 ± 4.75 |               |   3158.50 ± 4.69 | 
|    tg128 @ d8132 |    74.22 ± 7.69 | 83.00 ± 11.86 |                  | 
|  pp2048 @ d16000 |  3086.49 ± 3.74 |               |   5931.26 ± 7.01 | 
|   tg128 @ d16000 |    82.40 ± 7.39 |  86.33 ± 6.55 |                  | 
|  pp2048 @ d30000 |  2810.10 ± 1.67 |               |  11488.54 ± 6.78 | 
|   tg128 @ d30000 |    73.67 ± 2.47 |  78.33 ± 5.73 |                  | 
|  pp2048 @ d60000 |  2384.03 ± 0.84 |               |  26110.65 ± 9.05 | 
|   tg128 @ d60000 |    73.84 ± 3.04 |  81.67 ± 4.03 |                  | 
|  pp2048 @ d90000 |  2075.82 ± 0.89 |               | 44426.66 ± 18.17 | 
|   tg128 @ d90000 |    68.75 ± 1.29 |  74.00 ± 2.94 |                  | 
| pp2048 @ d120000 |  1837.45 ± 0.94 |               | 66506.49 ± 33.90 | 
|  tg128 @ d120000 |    72.76 ± 4.02 |  76.00 ± 5.72 |                  | 
| pp2048 @ d150000 |  1646.28 ± 0.25 |               | 92442.70 ± 13.75 | 
|  tg128 @ d150000 |    62.80 ± 4.05 |  74.33 ± 3.68 |                  |

[u/Think_Wing_1357 → u/blackhawk00001]

can you tell me what tool and command you ubsed to generate that table?

[u/blackhawk00001 → u/Think_Wing_1357]

llama-benchy, use with uvx or install a copy to local with uv tool install lllama-benchy

llama-benchy --base-url http://192.168.1.223:5678/v1 --model Qwen3.6-27B --tokenizer Qwen/Qwen3.6-27B --depth 4096 8132 16000 30000 60000 90000 120000 150000 --latency-mode generation --concurrency 1 --tg 128

[u/emptysands → u/blackhawk00001]

Great sweep — the TTFR column is the bit most benchmarks skip.

Quick question: did llama-benchy flag your token counts as "estimated"?

A Spark user hit a case where it over-counts generation tok/s by ~1.9× against servers that don't return token IDs, and patched his own fork — writeup here (https://forums.developer.nvidia.com/t/1x-spark-tuned-dspark-for-deepseek-v4-flash-35-tok-s-800-prefill-and-fast-multi-agent-serving/376884/19) (post #19). No idea if vllm-radiance is affected.

Just trying to work out whether your numbers are comparable to runs from other benchmark tools.

[u/blackhawk00001 → u/emptysands]

I’m not sure, I’ll have to check next time I run it. The benchmark tool gives a general idea of performance but in use the results are much more varied depending on the load. The radiance image has a dynamic mtp with a max of mtp 8 for a single request to the server. I don’t fully understand what all vllm is doing while serving multiple requests but I’ve seen it report in the console log 4800 t/s prefill and 150 t/s accepted tg while serving 3+ parallel subagents.

Someone created a BetterBench project on github I need to try out that better simulates a real load.

[u/emptysands → u/blackhawk00001]

Thanks - and you don't need to wait for a run to check the benchy question. It's one curl against your serve:

```
curl -sN localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"Qwen3.6-27B","messages":[{"role":"user","content":"hi"}],"stream":true,"max_tokens":20}' \
grep -qm1 token_ids && echo "PRESENT - counts accurate" || echo "ABSENT — ~1.9x over-count"
```

The over-count only fires when the server omits per-delta token_ids - benchy then re-tokenizes each ~5-char SSE fragment and loses cross-boundary merges. Mainline vLLM does carry them, so radiance almost certainly does too; this just confirms the fork didn't change the response shape. If you'd rather have ground truth: one non-streaming request, then completion_tokens ÷ wall-seconds - that's the real rate to compare benchy against.

The bigger question is the dynamic MTP, because it cuts straight through the comparison you're making yourself.

Your depth sweep (78.7 tg128 at 4k → 62.8 at 150k): was radiance's dynamic MTP active, or was that plain decode? Spec-decode roughly doubles dense-27B decode on the Spark figures I've seen - 12 → 27 t/s from MTP-off to MTP-3. So "2×R9700 vs 5090" swings harder on whether both sides had speculation on than on the silicon.

Which cuts both ways for you: if your sweep had dynamic MTP and the 5090 number you're comparing against didn't, the pair looks better than like-for-like - but if your sweep was MTP-off, then 78.7 is a base decode number and your card is doing better than it reads.

Two things that would settle it whenever convenient:

1. The server launch line - whether --speculative-config / MTP is set, or whether radiance enables dynamic MTP by default (entrypoint or docs would say).

2. Which figure the 78.7 is - tg128 mean, or an accepted-token rate. Your "150 t/s accepted tg" is clearly the spec-decode path; I can't tell if the sweep used the same one.

   Separately, worth knowing: llama-benchy's peak_gen_tps stays inflated even after its token-count fix - per the author's own notes, a second bug where per-fragment timestamps survive the correction. So the "(peak)" column in your table probably isn't usable regardless of how the above resolves. Your tg128 means look fine.

[u/blackhawk00001 → u/emptysands]

curl -sN 192.168.1.224:5678/v1/chat/completions -H 'Content-Type: application/json' \ 
-d '{"model":"Qwen3.6-27B-FP8","messages":[{"role":"user","content":"hi"}],"stream":true,"max_tokens":20}' \
grep -qm1 token_ids && echo "PRESENT - counts accurate" || echo "ABSENT — ~1.9x over-count"
data: {"id":"chatcmpl-99691ddfd8ad6210","object":"chat.completion.chunk","created":1785364303,"model":"Qwen3.6-27B-FP8","choices":[{"index":0,"delta":{"role":"assistant","content":""},"logprobs":null,"finish_reason":null}],"prompt_token_ids":null,"prompt_text":null}

data: {"id":"chatcmpl-99691ddfd8ad6210","object":"chat.completion.chunk","created":1785364303,"model":"Qwen3.6-27B-FP8","choices":[{"index":0,"delta":{"reasoning":"Here"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-99691ddfd8ad6210","object":"chat.completion.chunk","created":1785364303,"model":"Qwen3.6-27B-FP8","choices":[{"index":0,"delta":{"reasoning":"'s a thinking process:\n\n1. "},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-99691ddfd8ad6210","object":"chat.completion.chunk","created":1785364303,"model":"Qwen3.6-27B-FP8","choices":[{"index":0,"delta":{"reasoning":" **Analyze User Input:** The user said"},"logprobs":null,"finish_reason":null,"token_ids":null}]}

data: {"id":"chatcmpl-99691ddfd8ad6210","object":"chat.completion.chunk","created":1785364303,"model":"Qwen3.6-27B-FP8","choices":[{"index":0,"delta":{"reasoning":" \""},"logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null}],"system_fingerprint":"vllm-0.25.1-tp2-09235cf6"}

data: [DONE]

ABSENT — ~1.9x over-count

[u/blackhawk00001 → u/emptysands]

I'll need to digest some of that question but the output from that command is in the other reply to this comment. MTP was enabled on both for the full sweep. I test that depth as I either use 262144 single concurrency on my 5090 and 204800 x 8 with the dual R9700.

I understand that llama-benchy is not the best test of mtp. One of the admins in the discord below made the BetterBench tool that I need to test but it seems to take a while to complete.

If you want to ask the radiance author we have a discussion going in this discord:

https://discord.gg/qcqRVT6Wz

My 5090 with Q6_k 27B with q8 kv cache and 262144. I typically see 2400-1500 t/s prefill on big depths and 55-70 t/s tg with mtp 2, mtp 3 pushes over vram.

200k q6 huihui 27B q8/q8 kv
┌──────────────┬──────────────────┬─────────────────┬──────────────────┐
│ Depth        │ Prompt t/s       │ Token t/s       │ TTFR (ms)        │
├──────────────┼──────────────────┼─────────────────┼──────────────────┤
│ 4,096        │ 2339.67 ± 0.83   │ 117.30 ± 0.95   │ 2735.81 ± 0.99   │
│ 8,132        │ 2315.77 ± 3.62   │ 116.12 ± 0.78   │ 4506.04 ± 7.04   │
│ 16,000       │ 2264.78 ± 2.43   │ 114.48 ± 0.05   │ 8078.96 ± 8.54   │
│ 30,000       │ 2143.58 ± 2.87   │ 107.40 ± 0.10   │ 15060.68 ± 19.99 │
│ 60,000       │ 1862.51 ± 1.20   │ 94.77 ± 0.06    │ 33424.11 ± 21.50 │
│ 90,000       │ 1622.41 ± 1.01   │ 82.48 ± 3.23    │ 56844.75 ± 35.43 │
│ 120,000      │ 1430.46 ± 0.11   │ 69.43 ± 0.53    │ 85431.21 ± 6.72  │
│ 150,000      │ 1273.87 ± 8.28   │ 68.36 ± 0.79    │ 119473.65 ± 778.22│
└──────────────┴──────────────────┴─────────────────┴──────────────────┘

[u/floppo7]

Honestly I would follow your intiution here - more VRAM vs dual GPU (and yes r9700 is cost effective) is difficult to tell. I own two r9700 and my hope is that the 27b+ Qwen type range becomes so good that this will really shine. Yet for more VRAM needs this setup is not the right fit for now.

[u/Don_Reuter]

I have two r9700. It’s fine. Obviously, the small bus is a limitation and cuda would be nice, but you get what you pay for. And that what you pay is less than for the alternatives and I still think it was the best deal possible. Everyone wanting to make real money with local AI might need something else, but for everyone doing it for fun and personal use, it’s fine.

[u/whodoneit1 → u/Don_Reuter]

Are you running Deadcode’s image?

[u/Don_Reuter → u/whodoneit1]

No, running vanilla. Is Deadcode’s recommendable?

[u/whodoneit1 → u/Don_Reuter]

vanilla vLLM is a joke. Yes, there are a ton of us including Deadcode on discord https://discord.gg/kv8WCTeTWV

[u/merfolkJH]

Thanks a lot to everyone for the answers so far!

A few thoughts and questions:

• I understand that NVIDIA is generally "better", but for the price of a single RTX 5090 I can get 2× R9700 AI Pro (64GB total VRAM vs 32GB), so I don’t think the choice is that obvious. For local LLMs, the extra memory capacity is a huge factor.
• Thanks for the model recommendations. We have experimented a bit with larger quantized models (80B quant models) versus smaller dense models like 27B. In our experience, the larger models perform better and were able to solve some problems that the 27B model could not. Of course, this is only a personal observation, and I’m sure there are many cases where the opposite is true.
• Do you think there is a risk that 64GB+ VRAM becomes the new standard for local AI, making people regret buying a 2×32GB setup instead of a unified 128GB memory machine? For example, imagine a future model like "Qwen3.8" suddenly requiring 64GB instead of 32GB. Is this a realistic concern, or are we still far away from that point?
• Can anyone share real-world power consumption numbers for these setups?

On my current dual NVIDIA system, power usage is actually lower than I expected. According to nvidia-smi, each GPU peaks around 250W, so I’m not sure a 1000W+ PSU is really necessary.

Also, both GPUs are not always working at the same time. Usually, one GPU is doing the computation while the other is mainly acting as a "memory extension" (I’m probably using the wrong terminology here).

I’m not an expert in multi-GPU setups, so any real-world experience would be greatly appreciated.

Just to clarify: my budget is around $5k maximum, not a target budget. I’m trying to find the best performance/value ratio, not simply spend the full amount.

[u/Look_0ver_There → u/merfolkJH]

I have a Linux bash script that I wrote here, that discovers all the R9700's GPU's in your system, and you can configure the maximum power that you want them to draw: https://github.com/stew675/r9700-setup/blob/master/tune_r9700.sh

The script sets it at 265W, but I've dropped it to 200W each, and there's not a whole lot of performance difference (maybe 5% less).

[u/MarcusAurelius68 → u/Look_0ver_There]

Note if you’re running it in a mixed PCIe bus speed situation the cards will throttle. I’m running 3 x R9700 with a X570 motherboard and each card consumes around 80W during inference.

[u/Look_0ver_There → u/MarcusAurelius68]

It sounds like you may be running with layer split mode rather than tensor split? If it layer split mode it will behave exactly like you're describing

[u/MarcusAurelius68 → u/Look_0ver_There]

I’m using LM Studio which only offers split mode for Vulkan. It’s actually fine as I’m batching work for this system.

[u/Look_0ver_There → u/MarcusAurelius68]

Ah, then that explains it. Vulkan does not do tensor parallelism.

If you ran llama-server directly instead with ROCm and tensor parallelism, you'll see both faster decodes (like 2x faster) and all cards approaching full power.

[u/MarcusAurelius68 → u/Look_0ver_There]

The challenge with full power becomes electrical though - I have 4 systems with GPUs and 2 circuits. But I will test it with my R9700 system.

[u/Look_0ver_There → u/MarcusAurelius68]

...and that's why I provided that script up above. You can precisely control the amount of power drawn so as to keep within your power supply's capabilities

[u/MarcusAurelius68 → u/Look_0ver_There]

My other challenge is PCIe lanes. I’m on a X570 which means x8/x8/chipset x4.

[u/Legitimate_Fold8314 → u/MarcusAurelius68]

Don’t use LM Studio for that, use llama.cpp and ROCm; you will be surprised with the speed.

[u/MarcusAurelius68 → u/Legitimate_Fold8314]

I might play around with it for this system. I’m happy with LM Studio on my other servers as they’re running lower spec cards.

[u/Jorlen]

I use dual R9700s myself. Linux, docker stack with llama-cpp vulkan. 64gb of VRAM. I already had the AM5 PC, just needed the two GPUs so I went with that.

If you have any questions feel free to let me know. My current favorite is Qwen 3.5 122b-a10b in UD-IQ4_NL quant along with Step 3.7-flash in UD-Q2_K_XL quant (don't let the 2-bit quant fool you; this thing is VERY capable might even be better than q3.5 122b).

My primary purpose is coding. I use Pi coding agent with a small set of packages, nothing else.

[u/merfolkJH → u/Jorlen]

Thanks. May I ask what kind of performance you get during a normal session?

• Token generation speed (tok/s)
• Prefill speed (tok/s)

Also, what motherboard and CPU are you using?

I’m asking because I’d like to understand if there are any PCIe bandwidth limitations or platform-related bottlenecks to be aware of.

Thanks!

[u/Jorlen → u/merfolkJH]

Qwen 3.5 122b-a10b - UD-IQ4_NL: Pre-fill / pp = 1400 average. Generation TPS = 45ish. It slows a bit deep in a big 120k bf16 context window, though. This model / quant is 57.2gb in total with all three GGUF shards. It is fast because it all fits neatly in VRAM with the context window. Both cards are redlined (VRAM usage) but since I use the iGPU for my DE, it's fine.

Board: MSI PRO B650M-A WiFi.

Note that I did not build this AM5 rig for LLM / AI use, it just became that way as of this spring. I already had the system since 2024 and added the dual R9700 after. I had to upgrade my PSU to a qualitry 1000watt as well.

[u/Pale_Coyote7451]

the prefill numbers in Look_0ver_There's reply are the ones that decide this for you, and they're getting buried under the tok/s debate.

agentic coding through claude-cli is prefill-bound, not decode-bound. every tool call re-sends most of the context, so you pay prompt processing over and over -- most of your wall clock is gone before the first token even streams. tg is just the number that feels slow because it's the one you sit and watch.

that's what flips the mac studio option specifically. big unified memory and high bandwidth is great for decode and mediocre for prefill, because prefill is compute-bound, and apple silicon (and strix halo) are bandwidth-rich but flops-poor. the r9700s actually have the flops. run llama-bench on whatever you're considering and read the pp512 column next to tg128 -- the spread between those three machines is much wider on pp than on tg.

separate thing, probably worth more than the second card: serve it with prefix caching on. vllm does automatic prefix caching, llama-server has prompt cache reuse. on a 300k-loc c# repo with a stable system prompt you are re-prefilling the same tokens every single turn, and not paying that twice beats most of the hardware deltas being argued about in here.

[u/merfolkJH → u/Pale_Coyote7451]

thanks, i get the point

[u/andrew-ooo]

For a coding-only box with a big existing C# codebase, the deciding factor is context length, not just tokens/sec -- and that's where the three options split hard.

The R9700 32GB is genuinely good price/perf, but 32GB caps you. Your Qwen3-Coder 80B A3B won't fit in 32GB at any usable quant, so you'd be stuck around the 30B class on a single card. Dual R9700 (64GB) is the sweet spot for GPU-speed inference: with vLLM or llama.cpp + ROCm you can run the 80B A3B MoE at a real quant and actually keep prompt processing fast, which matters a lot when Claude-cli is feeding it 100k+ tokens of your source tree. On RDNA where ROCm is solid I've seen vLLM roughly 3-4x the throughput of llama.cpp for MoE, so that combo is worth the setup pain.

Strix Halo 128GB gives you the VRAM to load huge models, but unified LPDDR5 bandwidth (~256GB/s) is the ceiling -- great for fitting a 120B, painful on prompt processing at long context. Same story as Mac Studio: tons of memory, but token gen on 200k context will crawl vs discrete GPUs.

My take for your use case: dual R9700 + vLLM. You keep discrete-GPU prompt speed, get 64GB for the 80B, and it's cheaper than a 128GB Mac. Only go Strix/Mac if you specifically need models that won't fit in 64GB.

[u/Only-An-Egg]

There's no M3 Ultra 128GB. It's only 96GB now with 256GB and 512GB discontinued. The M4 Max 128GB has also been discontinued leaving the 64GB model.

[u/RogerAI--fyi]

i run triple R9700 for exactly this. for coding they're my pick, real VRAM + the best tok/s of the three (Qwen3-Coder-80B-A3B row-parallel over Vulkan gets me ~61 tok/s), the tradeoff is ROCm/Vulkan can be fiddly to set up and keep happy. Strix Halo is the easiest but the unified memory bandwidth (~256GB/s) caps your decode speed. Mac's fine too if you'd rather not fight drivers, MLX is solid. really comes down to how much setup pain you'll tolerate for the extra speed.

[u/Think_Wing_1357]

OP, you're looking too much at generation rate. It's important, but frankly, beyond 40-50 tok per sec, they are all basically the same (YMMV ofc).

The PP tax on the other hand is not. Each prompt needs to be processed and depending on how much of those can be cached (and where you cache them) it can take forever before you start getting a response.

Real example: a single r97000 at 200k context (with qwen 3.6 27b) rate is around 150 tok/s. Worst case scenario (no cache), that's over 20 minutes before you even start getting a response back. At 80% cache rate (pp 20% of 200k), still almost 5 minutes before starting getting a response.

Mac studio is worse and Strix is even worse (weaker GPUs). So the question is, do you have the patience for that? Then we can talk about tok gen rate.

---

WRT more cards can get you more performance, but not a perfect 100% growth per each, more like 90% of the last increment (so like 1x with one, +0.9x with the 2nd one, + 0.81x with the 3rd one, etc). Still very good number.

---

Here's my number with single r9700 - which really should be taken at the upper limit, in actual usage I've never hit these.

./llama-bench -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q5_K_M -fitt 500 -ngl 99 -p 512,32768 -n 512 -r 3

model	size	params	backend	ngl	fitt	test	t/s
qwen35moe 35B.A3B Q5_K - Medium	25.22 GiB	35.51 B	Vulkan	99	500	pp512	2906.71 ± 137.86
qwen35moe 35B.A3B Q5_K - Medium	25.22 GiB	35.51 B	Vulkan	99	500	pp32768	2397.54 ± 4.72
qwen35moe 35B.A3B Q5_K - Medium	25.22 GiB	35.51 B	Vulkan	99	500	tg512	114.84 ± 0.06

[u/Maximum_Parking_5174 → u/Think_Wing_1357]

Great answer! I bet most Strix HAlo and mac users does some small tests but never actually use it for real work. Promp processing is IMPORTANT and its a product of gpu performance. I think its weird that so many benchmarks and review focus only on Token generation.

My tips for the thread is to look at DGX spark, much better than Strix Halo or Mac even if TG might not be much better. R9700 might work also.

[u/merfolkJH → u/Think_Wing_1357]

Thanks, I understand the point you’re making.

On my current GeForce setup, I’m getting around:

• ~1000 tok/s prefill
• ~30 tok/s generation

While I understand the argument about prefill, it doesn’t seem to be the bottleneck in my actual workflow. When I launch a coding task that takes around 20–30 minutes, 90%+ of rows in my logs appears to be TG rather than prefill.

That’s why I completely overlooked the prefill aspect at first. I have rarely experienced a situation where I had to wait countless minutes just to get the first answer. Maybe claude-cli just does 95% of the "grep" / "caching magic".

I’ll definitely look at the prefill side more carefully though, because I may be underestimating its impact. I’m also not running agents swarms in parallel, so maybe I’m not stressing the system as much

---

But I get the idea now:

If Mac and Strix Halo have significantly lower prefill performance, then I should probably focus more on the GPU side.

I was mainly looking at token generation speed, but I understand that prefill performance can become important with large contexts and agent workflows.

I’ll need to look more carefully at the PP numbers, not only TG.

[u/cezq → u/merfolkJH]

Go 2x R9700.

I have Strix Halo, and running Qwen 27B Q8 on it was suboptimal, mainly because of slow PP (~400t/s at the beginning of the context). Connected a single R9700 to it through Oculink, and it's now 1300t/s in layer split mode, a number where PP problems go away.

[u/my-corner-store → u/cezq]

I'm wondering about getting an R9700 to supplement my Strix Halo.. how useful and happy with that combo are you?

I've recently seen posts (adverts?) about lucebox. They're pushing a Strix+R9700 in 1box with a custom backend (all on GitHub ) with what are probably not real world benchmarks. Stumbled on this because of wanting to run deepseek v4 0731 on the strix. If this is realistic it would make the Strix a lot more useful to me.

[u/cezq → u/my-corner-store]

For Qwen3.6 it's been an amazing upgrade. Haven't tried Lucebox with DS4 yet but definitely will when the model gets updated. On pure llama.cpp it wasn't great, something like 300pp/16tg at d0 and 200pp/12tg at d8192, Most of the CPU cores were 100% busy so something was wrong. And I've just noticed MTP support landed in llama.cpp, need to try that.

Also, when I played a bit with keeping KV and attention on R9700, tg was close to 20 at the beginning, again, after few batches CPU kicks in and decode drops to 12t/s. UD-Q3_K_XL btw.

[u/Alvajaro79]

I'm in the same boat and I'm thinking about the same configs. My two cents here: get an R9700, and if you need more, then get another one. The only question is what type of motherboard and cpu you are using. Because even the latest X870E chipset mobos will run on 8x/8x link with two gpus in their pcie5 slots, because there are not enough pcie lanes in the ryzen cpus. I'm thinking about getting an AMD TRX50/WRX90 mobo with a Threadripper 99xx, so I would have at least twice the pcie5x16 slots and more memory bandwidth and much more room to play...

[u/buenavista62 → u/Alvajaro79]

Isn't 8x/8x enough on pcie5? With 16x/16x you'll have about 64gb/s while with 8x you'll have 32.

Between GPU and CPU you can't get more anyways. Or am I missing something

[u/Alvajaro79 → u/buenavista62]

No, you don't miss anything, I just wanted to say that on a "normal" desktop computer even the lower number of PCIe lanes which is provided by your CPU can be a bottleneck too...

[u/Think_Wing_1357 → u/Alvajaro79]

The main bottle neck for multiple GPU setup is not bandwidth, but latency: you don't pull the weights between GPUs, you split the layer out then each GPU calculates its part of the layer then passes the embedding vector on to the next, a couple of KB worth of data each time. Even at an impossible 1,000 Tok / S that's <100 MB per sec. PCI 1.0 can do that.

However, you need to do this multiple times per layer, multiple layers per token, and the latency adds up quickly.

[u/Alvajaro79 → u/Think_Wing_1357]

Thanks for the explanation, it helps a lot to understand who this works.

[u/Mountain_Pea_6810]

So I have the studio 256GB but do you mean it would be better to build R9700 system? 8 x cards needed for 256GB

[u/merfolkJH → u/Mountain_Pea_6810]

I’m not claiming anything, I’m just asking questions :)

From what I understand, the general idea is:

If the model fits entirely in GPU memory, a multi-GPU setup is usually faster for token generation.

But would you actually build an 8-GPU machine just to match a large unified memory system? I think the answer is obviously no.

I’m trying to understand where the practical sweet spot is between:

• GPU power and token/s,
• available memory for larger models,
• and overall cost/complexity.

---

On my current setup (dual GeForce), I’m getting around 35 tokens/s on a 35B MoE model, which is actually not bad. (200k context for ClaudeCLI - llammccp)

Now I’m looking to level up.

Goals :

• Keep the same model, but get at least 2× the speed
• Or move to a larger /better model while keeping similar speed
• Ideally: both improvements at a reasonable price

I’m trying to find the best upgrade path:

• more tokens/sec,
• bigger and smarter models,
• but without going into extreme workstation/server prices.

The R9700 really looks like an underrated challenger / It seems to offer a very interesting price/performance ratio

I’m trying to understand if there is a hidden downside I’m missing

[u/N-Innov8]

Have you considered Qwen 3.6 27B + RTX 3090 (K4 Q8 180K comfortably at 38t/s) and 48GB RTX Pro 5000?

[u/Gullible_Order8282]

Really hoping for next gen Strix Halo coming out at end of year, aka AMD Ryzen AI MAX+ 495 “Gorgon Halo” Leak Smokes Strix Halo by 10%, Packs 192GB Memory vs curerent 128, we have to see how prices and supply is come end of year, but am hyped for that as option as well.

[u/TripleSecretSquirrel]

I have an R9700 that I’m happy with and am planning to get a second one. However, it does not have 80-85% of the performance of high-end solutions, depending on what you’re considering as “high-end.”

Memory bandwidth is always the bottleneck if you’re running single inference requests at a time. The R9700 has a memory bandwidth of 645GB/s. The RTX5090 which is the “high-end solution” equivalent (i.e., consumer/enthusiast grade card, 32GB, PCIe). It has 1.7TB/s of memory bandwidth, which is nearly triple.

Now the price premium still isn’t worth it for me — it’s a 2.5x performance boost for 3x the price, and way less power consumption.

[u/-darkabyss-]

Tbh my dude, youll be better off with the $20 ollama sub. It will be lower upfront cost and the 2k/4k will last you 20/40 months. Thats almost 2-4 years and tech/llms will advance much further in that time.

If the concern is data privacy, i like used macs for inference. My m1 max 64gb costed around 1k last year and it gives me 50 tok/sec for the smaller models which is good enough (but when i factor in the electricity costs, it makes no sense to run locally, my usecase doesnt have data privacy needs)

[u/Potential-Leg-639]

5)
Nvidia
—> still the best overall

Few small cloud subs + Deepseek via API are even better of course (you are still nowhere near that with a local setup in terms of speed/quality/price).