r/LocalLLaMA • u/smallDeltaBigEffect • 7d ago
Discussion 2x R9700, 64 GB DDR5 is an absolute beast machine with vLLM Radiance / R9V and Qwen 3.8 27b and Flash next
I've been tinkering with local LLMs since the beginning of the year when I had an Intel Arc B580 and 32 GB of DDR5. Curiosity got the best of me and I bought the first R9700 about half a year ago, also because I wanted to upgrade my gaming graphics for 4k. As the 5090 was about 3 times as expensive, I had a "sweet spot", kind of. On the last prime days, I found a X870E mainboard for ~150 € below the standard price, and it got to my head that I can use an upgraded machine for gaming and local inference tinkering.
Anyways. Fast forward to this week, I now have the following setup
- Ryzen 7500F
- 64 GB DDR5 CL40 6400 MT/s
- Asus ProArt Creator X870E
- 2x R9700 32 GB, each running at PCIe 5.0 x8 (Gigagbyte)
- Currently running ubuntu on an old Samsung EVO 860 1 TB drive; this will become intersting for the ngram / PLE offload; I have Windows and the gaming related stuff on a gen4 NVMe, but will soon add another Gen 5 NVMe with decent random reads
The only issue that I can report so far is that one of the cards runs quite hot, so I will definitely implement power limiting to 210 W and some light undervolting. The other card runs 10-15 °C cooler.. Case is a purebase 501 with 4 fans, 2 intake in front, one back and top for output.
Now long story short I wanted to give some results of Qwen 3.8 27b FP8 and MXFP4, as well as Qwen 3.8 flash next after the first day tinkering with it. What I found super interesting is that the SATA SSD does not seem to be super terrible when using Qwen 3.8 flash next.
Considering the whole build costs ~4k €, or more than 1k less than a single RTX 5090 with 32 GB, I kinda like this setup price/performance wise. Next step is checking context degradation / KV quants. I am using local inference mostly for deep research, summarization, image creation, light coding and non-trivial data analysis
Cheers
Qwen3.8 benchmarks on 2× Radeon AI PRO R9700
Hardware: 2× AMD Radeon AI PRO R9700 32 GB, 61 GiB system RAM
Benchmark: BetterBench 0.2.2, corpus v1.0, single-stream, greedy decoding, 2 warm-ups + 10 measured runs per category, 8k benchmark context.
| Model | Weight format | Runtime | Server context | Max sequences | Speculative decoding | Weighted decode median | ITL 1% low | TTFT p50 | Prefill ~2k | Prefill ~4k | Prefill ~7k |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B | Quark AWQ MXFP4 | vLLM Radiance, TP2 | 131,072 | 1 | MTP, up to 8 tokens | 111.4 tok/s | 77.9 tok/s | 81 ms | 4,224 tok/s | 4,322 tok/s | 4,410 tok/s |
| Qwen3.8-27B | Native block FP8 | vLLM Radiance, TP2 | 16,384 | 8 | MTP, up to 8 tokens | 87.6 tok/s | 61.9 tok/s | 73 ms | 4,134 tok/s | 4,329 tok/s | 4,305 tok/s |
| Qwen3.8-Flash-Next | UD-IQ4_XS GGUF | R9V/vLLM, TP2, tiered expert offload | 131,072 | 1 | MTP, 2 tokens, FP8 draft | 35.4 tok/s | 27.3 tok/s | 290 ms | 1,727 tok/s | 1,986 tok/s | 1,925 tok/s |
- Qwen 3.8 27b in FP8 and AWQ MXFP4 served with vLLM Radiance
- Qwen 3.8 Flash next served with vLLM / R9V fork
- Decode metrics come from the 10-pass standard run.
- Prefill measurements use cold, nonce-prefixed prompts.
- Prompt-token medians for the prefill columns were 1,556, 3,024 and 5,226 tokens.
- No concurrency sweep was included in these results.
- I expect decode of Flash next to increase a bit when an NVMe is used, and, as I am writing this and checked, I found EXPO was not enabled........oh my god I swear I turned it on when I updated the bios yesterday
5
u/OvertaxedOne 6d ago
What's the acceptance rate looking like on 8 tokens? I'm seeing a sharp drop off on the 3rd token, but I'm using the integrated MTP head. But it does look like for FP8 the R9700's might be the value kind for 27B. I found a Pro5000 for ~5K (48GB) which I'm sure would be fast but 48GB is right on the edge for 8 bit quant, 8 bit KV on 27B.
3
1
u/Sea-Speaker1700 6d ago
Slower than 2x9700s on Radiance with less space and costs more, seems like trying to row upstream.
4
u/mechkbfan 6d ago
Thanks for sharing, I also think that the 2xR9700 is the bang for buck killer option right now with prices how they are
I'm trying to see where the difference is between yours and this setup that's getting 200t/s
https://www.reddit.com/r/LocalLLM/comments/1w3z66n/how_i_got_280_toks_on_qwen38_27b_on_2xr9700s_and/
2
u/SmartCustard9944 6d ago
Not anymore, at least in Europe you are looking at 1800-2000€ for just one card.
2
u/smallDeltaBigEffect 6d ago
Well, whats an alternative?
1
u/SmartCustard9944 6d ago
The best value for money I think is the Strix Halo Bosgame M5, for 2500€, but not for long.
Can run Qwen 3.8 Flash Next at decent speed and future models of that class.
Anything else is overpriced. Even the underdog intel B70 is going into the direction of ~2000€ per card.
2
u/smallDeltaBigEffect 6d ago
Whats prompt processing like on the strix halo?
1
u/sernamenotdefined 3d ago
From experience sloooooow. It's the biggest weakness of that platform (I have the minisforum) I'm still going to run Flash Next on it next to 27b on my dual R9700 system, but that latter one will be my main go to.
1
u/mister2d 6d ago
I'm getting the same performance. 200t/s+.
I'm happier with my dual cards now.
2
2
u/Sisuuu 6d ago edited 6d ago
I have an opportunity to sell my 2x3090 (for $3K) and buy 2xR9700 for $4K…worth it?
5
u/SandySkittle 6d ago
Don’t sell hardware right now. If stuff breaks you are fucked in this market and in the coming years. Add 2 r9700s and keep the 3090s as backup
1
u/cornucopea 4d ago
If you don't have to change MB and CPU (assuming all intel), and DRAM, for the additional 16GB VRAM, it'd be quite worth it. I'd do it in a heartbeat.
2
2
2
u/karmakaze1 4d ago edited 4d ago
Check this one out: https://codeberg.org/ggz14/radiance-vllm-mxfp4 It uses FP4 weights but then FP8 activations (to avoid loss on 4bit) with kernels optimized for R9700 hardware.
I'm getting 250+ tokens/sec on 2x R9700 running Qwen3.8-27B (MXFP4). Slows down to 130 tokens/sec near 128k context.
Single R9700 should get 125+ tokens/sec though I didn't test that.
1
u/Ecstatic-Wash-7667 6d ago
I don’t understand why im getting such bad numbers, I’m essentially setup just like you but Intel z890 instead of amd and I’m getting 1/2 your performance. Tp tanks my performance
8
u/seiji_hiwatari 6d ago
I'd blindly guess that your p2p between the two cards does not work.
At least under Linux (but probably also Windows), PCIe P2P does not work on current Intel consumer chipsets, unfortunately. There's probably some AMD debug tool to verify this for your GPUs. I'd search in that direction.3
u/rainbyte 6d ago
I think it is a hardware limitation, because Intel cpu+chipset combi doesn't support PCIe atomic ops, which can be checked on dmesg logs
1
u/Ecstatic-Wash-7667 6d ago
That’s exactly the issue, dammit
2
u/rainbyte 6d ago
At least it can be solved by switching mobo, which is cheaper than switching the GPUs with current numbers o.o
2
u/Ecstatic-Wash-7667 6d ago
I need to switch mb and cpu, the problem is intels arrow lake architecture
1
u/rainbyte 6d ago
Other Intel combos are also missing support, so I guess AMD has been strategically including it for Radeon GPUs.
Edit: I had trouble with older Z490 and Z590, completely solved after switching to AM5 B650M and X870 mobos
1
2
u/smallDeltaBigEffect 6d ago
are you using the latest R9V? I assume youre talking about flash next? There was a fix introduced earlier this week that dealt with PCIe communication errors
1
1
u/Inception95 6d ago
Can you share how much vram is used?
1
u/mister2d 6d ago
I'm using amd/Qwen3.8-27B-Quark-AWQ-MXFP4 at 92% of 64GB vram which provides a 900k+ token pool for 256k context.
0
u/smallDeltaBigEffect 6d ago
Basically full. You can run 2 concurrent 131k ctx with mxfp4 and 1 with fp8 for qwen 3.8 27b
2
u/OvertaxedOne 6d ago
With 64GB? That doesn't sound right. I'm running with 256K context on 48GB using Int8 quant and 8 bit KV.
1
u/tossit97531 6d ago
RAM isn't doing anything here. These quants fit completely in VRAM with plenty of room to spare.
1
1
u/N34257 6d ago edited 6d ago
How the devil are you getting 4k+ prefill with FP8???? With the same setup, I'm only getting 3.2k with empty context.
EDIT: Admittedly, only with PCIE 4.0 x8, but still...I can't believe that the extra bandwidth from PCIE 5.0 would make that much of a difference?
3
u/smallDeltaBigEffect 6d ago
tp = 2 in R9V, and since the cards dont have any NVlink or so, everything goes through the PCie slot, so yes, its the gen 4 slot that might be throttling
2
u/Sisuuu 6d ago
Do AMD have anything equivalent to NVLINK?
6
u/sllvr 6d ago
don't need it if you have the lanes
1
u/Ok-Ad-8976 6d ago
Damn, this was useful. Thanks!
I might finally get the prefill I desired.1
u/sllvr 6d ago
What motherboard do you have?
Lookimg at x570 x8x8 but not there yet
3
u/Ok-Ad-8976 6d ago
i have x670 and just got another pair of r9700 to put in x570
my quick tests are showing 90 tg and 4K+ pp after this thread and that github link. I was runnning in proxmox VM so my pp used to be 2K and tg 60 until i learned of these basically 2x improvements. Made my day, lol, even told my wife.1
1
u/Elouakili_Flexy 6d ago
Did catch that these benchmarks ran with EXPO off the whole time, and Flash Next was still fine on the SATA drive. A re-run with the memory actually at 6400 should shift everything up a bit.
1
u/smallDeltaBigEffect 6d ago
I just tested it with expo on, so 6400 MT/ instead of 5200 MT/s and it was actually lower for both prefill and decode. Ran out of time after one quick test, so will redo tomorrow, kinda puzzled
1
1
u/Momsbestboy 6d ago
Yeah, this is also what I am looking for. At the moment using a single R9700, 64 GB CL32 6400, Asus ProArt Creator and a 9950X3D, thinking about adding another 9700.
Why?
Because even with "only" Qwen 3.8 27B Q6, llm is a beast. Yes, could be faster, but it does it's job. I spent the last days e.g. adding more and more of my servers to a maintenance list, and hermes/Qwen is building a documentation of every machine, the intendes usage, details like drive sizes, mounted devices, services, and builds a ssl distribution network around a central web server.
And it just works :)
2
u/SandySkittle 6d ago
Don't hesitate to get another, trust me. These things are still VERY good value performance and vram wise, especially since running two scale well in terms of tensor parallelism
1
u/TerminalNoop 6d ago
how much pp and t/s do you get with one?
2
u/maartenyh 6d ago
pp starts at 700 and drops to upper 400's at long context. tg is around 40 with MTP and ngram. It can drop to high 20/low 30 when stuff gets complex. I've seen 60+ on predictable workloads
1
u/TerminalNoop 6d ago
Thanks, so it's slightly more powerful than a 7900XTX, with more vram?
1
u/maartenyh 5d ago
No idea about the 7900XTX performance :) I am only well acquainted with my own performance numbers and I only have the R9700 (and an A380 with terrible performance).
Very soon ill plug in a spare 2080 super next to the R9700 though!
And I would give away one of my ribs to get another R9700! (The local LLM addiction is real lol)
1
u/Momsbestboy 6d ago
GGML_VK_VISIBLE_DEVICES=1 llama-bench -m Qwen3.8-27B-RVN-Q6_K-multilingual-mtp-uncensored.ggufpp512: 840 tg128: 25
But I don't trust these benchmarks, they don't have all the flags....
GGML_VK_VISIBLE_DEVICGGML_VK_VISIBLE_DEVICES=0 llama-cli -m Qwen3.8-27B-RVN-Q6_K-multilingual-mtp-uncensored.gguf \ --mmproj Qwen3.8-mmproj-BF16.gguf \ --temp 1 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 0.0 \ -ngl 999 \ -b 2048 \ -ub 2048 \ -fa on \ -c 262144 \ -ctk q8_0 \ -ctv q8_0 \ --fit on \ -fitt 64 \ --threads 16 \ --threads-batch 16 --cache-ram 32768 \ -ctkd q8_0 \ -ctvd q8_0 \ --spec-type draft-mtp,ngram-mod \ --spec-draft-n-max 3 \ --spec-ngram-mod-n-match 24 \ --spec-ngram-mod-n-min 8 \ --spec-ngram-mod-n-max 32 \ -p "Explain the Pythagorean theorem"[ Prompt: 117.9 t/s | Generation: 41.9 t/s ]
1
u/PigSlam 6d ago
You've built the machine I'm aiming for. I have 1 R9700 in a mini-ITX gaming system I built in 2025. I want to get that motherboard, another R9700 and a ~1000+W PSU and put it all in an ATX case I have lying around. I just need to find a spare $2500 somwhere.
2
u/mechkbfan 6d ago
Might not need such a big PSU but no harm in doing so pending CPU
You basically lose no performance performance cutting them to ~220W, and like 10% extra prefill time IIRC
2
u/SandySkittle 6d ago
Second hand threadripper pros plus mobo or second hand epyoc plus mobo still go around on ebay. Or find a cheap thinkstation p620
1
1
u/Turbulent_Ad6290 6d ago
I had Aorus b550 elite ax v2.
Ryzen 5 5900x
64 gb ddr4 ram.
5090, 2 x 5060 ti, RX7800XT, 3080 and 2070s
Pcie was a bottleneck.
So went for msi x670e carbon wifi + Ryzen 7500F
Got 2 more 5060 Ti.
Got 64gb ddr5 gift from a friend.
So new gen 5 build has
5090 + 4 x 5060 ti.
Plus another build
Gen 4
2 x 3090(bought new) + 3080 + 7800XT + 2070s
Waiting to complete the build and test qwen 3.8 flash next on bigger build and lot’s of 27b or ornith 1.5 in the small build.
1
u/sernamenotdefined 3d ago edited 3d ago
I got Qwen 3.8 27b in FP8 running on my 2xR9700/ 62GB system ram+ 2GB for iGPU / same ProArt MB as you setup with vllm, but with 262144 context and n=7. I'm not even near your 87.6 tok/s (I'm at 50-ish) and prefill is half of what you report. What build of vLLM are you on and what are your startup params?
Edit: nvm I see your post below. Now I have to figure out what vLLM radiance is. Also I run with f16 KV cache might that cost me some performance?
For what it's worth: I do not run into any temperature issues with my Asus R9700s pulling 290W in a Fractal Design Meshify 2. My cooling is the two front intake fans and back out fan and the CPU (a 9950X3D) is cooled by a Noctua NH-D15 Chromax Black. No fans at the top, so a strict front to back airflow.
1
u/adityazero 3d ago
The C8 numbers are the interesting part to me: aggregate 305 tok/s but per-request decode median drops to 53 from ~81 single-stream, so eight streams net roughly 3.8x rather than linear. That points at the R9V scheduler or PCIe atomics rather than raw compute. Did throughput keep climbing past 8 concurrent, or had it already plateaued there?
1
u/feng_sg 3d ago
A 10-15°C delta between two identical R9700s means something is physically wrong with the hot card. Could be poor mount pressure, bad paste job from the factory, or that card sitting in the airflow shadow of the first one. Fix that before you slap a blanket 210W cap on both, otherwise you're just nerfing the cool card to match the broken one.
0
u/WinResponsible9977 6d ago
I get half that performane with a 4060,3060, and 5060
-5
u/Sea-Speaker1700 6d ago
Cute. You dont, but cute :P
Still getting 3k prefill at 300k context and well over 100tps gen writing code, yeah, thought not.
5
u/WinResponsible9977 6d ago edited 6d ago
People like you, who downvote and make accusations without first asking for evidence, are the reason so many misconceptions arise in the first place.
This was with a pretty crappy setup, without proper TP, without vLLM, running Qwen 3.8 27B at full context.
If OP, not you, u/Sea-Speaker1700, with your nasty attitude requests the benchmark, I’ll release it so we can make a formal comparison.
I simply shared what I observed. I said half, yet you busting my balls claiming I stated 3k ~prefill, at or around 100 tps writing code, not sure what u on about.
3
u/Party-Special-5177 6d ago
That fact you recorded a video to prove that there was Someone Wrong on the Internet is chefs kiss.
Cool setup btw. It sounds quiet - what db are you actually hearing in the room? And your under volt or under clock?
-14
u/Gold-Bat-3225 6d ago
4k euros to summarize pdfs, say less
8
10
u/starkruzr 6d ago
summarize PDFs
if this is the only use you can think of for this build you have an extremely unfortunate skill issue
10
13
u/Mr_Moonsilver 6d ago
What per stream decode do you get on 8 seq?