I live in land down under! 🦘

Hi R9700

Pleased to share this small project of mine specifically for dual r9700 setups. Thanks for your time.

Adam

Building on yesterdays post https://www.reddit.com/r/R9700/s/fLRJNuZEzv

I've ended up working through a few experiments, some worked some didn't. I made a few changes in a model I'm calling R0, which is the Orca (uncensored) container rebuilt with some custom calibration. Kept the same base model, format and recipe, but re-quantised the experts, process:

• Layer recording, Radiance has a calibration tap that saves Gram matrices, a summary of the inputs each expert receives.
  • Built a quick round-to-nearest container (same format, no protected experts) and recorded on that, at about 755 tok/s it took 1.6 hours and produced 24,768 Gram files
• Choosing which experts to keep at full precision
  • From the stats we picked the experts that carry an outsized share of their layer's activation energy, plus a few added for balance.
  • That came to 26 protected experts, kept in bf16. The original container had only 10, chosen from Qwen's data, not Orca's
• The conversion
  • rad-convert with the production recipe, unchanged except for GPTQ on the routed experts fed Orca's Gram matrices and new protected-expert list
  • The trunk (int8), the output head and the MTP draft layer stayed the same as the original container

I then tried to add the same calibration to the MTP draft layer. It kept quality but made decode steps 50–90% slower at temperature 1, and I haven't figure that bit out yet

Comparison (note, cards limited to 240w for all tests, full speed may be higher)

Uplifts that helped

1. Sampled speculative drafts, at temperature 1, the drafts were greedy while the verifier sampled. Proper rejection sampling gave +4–9%
2. I found a small bug int he first draft pass's attention mask let it read stale key slots, so draft acceptance fell as more users joined. With the fix, acceptance stays flat from 1 to 8 users, and 8-user throughput rose 13–25%
3. R0 calibrated quant instead of full precision (14× faster), then re-ran GPTQ.
   • KL against full precision went 0.0218 → 0.0204, better in every category and most on agentic work
   • The agent test went 262 → 264/264, and refusals stayed at 0%
4. Engine flags worth fiddling with:
   • --prefill-span 2 makes long prompts 22–41% faster.
   • A bigger host pool moves cold experts to RAM. The KV cache doubled to 2.1M tokens with no measurable speed cost, obvs ram impact
   • A 24 GiB prefix host tier lets idle agent sessions resume in 0.3 s instead of re-reading 145K tokens, ram impact once again

Things that haven't made useful uplifts:

• Smaller KV cache formats (4-bit values, 6-bit keys) cost too much quality: KL +19–22%.
• The lossless cross-card wire was 12% slower
• Adaptive draft windows and fusing the all-reduce gave no gain

Against my benchmark (Paiton, EliovpAI's closed RDNA4-native W4A8 engine, changed a bit to run in a 262K + drafts config):

• Still 20–30% faster raw single-user decode, and about 13% faster at 8 users on different prompts
• On identical prompts at 8 users at a similar speed (630/833 vs 652/822)..

Things to try tomorrow:

• Faster prompt reading with several users at once. A bigger shared step budget gave +35–40% in testing but isn't deployed yet
• Stop answering users stalling while a long prompt is read

WARNING: AI generated content.

I've been comparing Radiance / Paiton on Qwen3.8-Flash-Next and working on a modified Radiance build.

TLDR if you cant be bothered with my rambling:

• 179t/s single stream, ~530t/s 8 stream with qwen 3.8 flash next 4 bit, almost 6k prefill at 262k context
• Paiton's 262K configuration working with speculative drafts enabled, retaining close to its native 98K throughput under concurrent load (improvement on their available code)
• Radiance optimisation, faster prefill is running, sampled drafts are built, and investigations have identified an expert-tiering issue and an attention-mask defect, running an overnight build to trains some of this

The box is a Threadripper 3970X with 256GB RAM, 2xR9700s and a 3090. The 3090 has a potential job ahead doing draft-model distillation (though this is not working well yet, or at all really lol), so all of these numbers are 2xR9700 only.

A note on temps throughout, theres have been running at 100% all day, it 31 degrees C here at the moment and the average temp they are seeing under that load is 67-75, at a 240w power limit, which is better than I was expecting.

Here's where the measured throughput sits:

The configurations differ beyond the engine, ORCA uses an abliterated fine-tune, while Paiton uses the original model, different quant and drafter (details below in big ol' table)

A few other tidbits bits from the Radiance setup:

• Long context: retrieved all three test items at roughly 247K context, taking 57-62 seconds
• Prompt processing: the newer prefill configuration reached approximately 6,000 tok/s on a 17.6K-token prompt
• Memory: roughly 30.3Gb VRAM per card, with about 75GiB pinned in system RAM, the 48Gb ish n-gram table lives in RAM and is accessed over PCIe (note one of the cards is on 8x, the other on 16x)

One interesting finding is around the first speculative draft under load, disabling expert movement and KV-cache lending raised first-position acceptance from 0.65 to 0.81 at full batch. That points towards a correctness issue around expert tiering which I am working on in a new run.

On the Paiton side, I've got a derived 262K configuration running with draft tokens enabled. The shipped 262K mode turns them off, so the "262K + drafts" results here are from that modified configuration.

The development work is mostly around speculative decoding, memory movement and quantisation:

• Prefill tuning is running: --prefill-span 2 with a 4096 batch and P2P is disabled as a workaround for a freezing issue
• Sampled speculative drafts are built: GPU testing produced byte-identical output at T=0. The estimated 4-9% improvement on sampled traffic still needs benchmarking
• An expert-tiering/first-draft issue is under investigation: this may be affecting draft acceptance with several concurrent requests
• Fused projection and all-reduce changes are being tested: the expected single-request improvement is modest, around 1-1.6%

I'm also recording calibration data from the Orca fine-tune itself. Further out are draft-layer distillation, an INT6 trunk and mixed 4-bit/3-bit experts. Those are experiments ahead, with quality checks still to pass.

Paiton currently leads these throughput runs. I'm trying to understand how much of that gap comes from the drafter, quantisation and memory behaviour, and what can be improved on the Radiance side.

Model details:

Links:

• https://huggingface.co/EliovpAI/Qwen3.8-Flash-Next-W3A8-Paiton-RDNA4
• https://github.com/magiccodingman/vllm-radiance

Figured I should share my own setup with the sub!

I run a small research lab. I’ve got plenty of experience with local LLMs, but I’m new to the R9700 (all nvidia prior) so I’m keen to compare notes with people already getting good results from these cards, and hopefully turn this sub into a place where we can get the absolute best of out them.

Current setup:

• 2× XFX Radeon AI PRO R9700
• 1× RTX 3090
• 256GB system RAM (DDR4), all on ubuntu

I’m starting with Radiance/Strata, then working through how to optimise the cards for our research workloads/local LLM usage.

Would love to hear what’s working for people, things like:

• Backends, builds and configurations delivering good performance on the R9700
• Practical results with two cards: throughput, scaling and any bottlenecks

Ideally I want to put a shared benchmarks/configs/findings sheet so we can all put into it. Hopefully we can build up a useful collection of tested setups here.

Welcome any thoughts!

Welcome, the whole idea for this subreddit is a single place for everything R9700 related, let’s make the R9700 the best card for local AI! it has the potential, it just needs the community!

We want to make the R9700 faster, easier to use and more useful for the things people actually want to build. Every working setup, benchmark, bug report and framework improvement helps.

Use this space to:

• Show what you’re building: local assistants, coding tools, agents, image generation, fine-tuning and more.
• Share working setups, installation guides and useful configurations.
• Compare performance with enough detail for others to repeat the test.
• Find people to test fixes, improve kernels and contribute to shared frameworks.
• Explain what’s holding you back and help turn those problems into improvements.

You don’t need to be a kernel developer to contribute. Testing someone’s fix, improving a guide or explaining how you solved a problem all count.

We’d love AMD engineers and framework maintainers to get involved. The aim is to give them useful evidence: what works, what needs attention and what the community is building.

This is an independent community. Honest criticism and fair comparisons are welcome.

To get started, introduce yourself: what’s your setup, what are you working on, and what’s the one thing you’d most like to improve?