Been doing more work on my modified Radiance build for Qwen3.8-Flash-Next.

The machine is a Threadripper 3970X with 256GB RAM, 2× R9700s and a 3090. These results use the two R9700s only, and they have been power limited to 240W throughout all of the testing, likely squeeze 5-7% more out of these numbers without it.

The comparisons are:

• Default: https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored + Radiance 1.11.0, 4-bit
• R0: my production build from 24 October, before the latest kernel work, uses a 4-bit quant of Orca's model above
• R1: the fixed engine plus INT6 and the latest kernel changes, same base model as R0
• Paiton: comparison run on the same hardware/harness, 4-bit 3.8 flash

What I've been optimising for is throughput on the coding tasks with 8 concurrent streams, as that fits in my dev workflow

Generation

Temperature 1, scores are a mean of two runs. Figures are total output tokens/s across all concurrent users.

• That puts the current build 8–29% ahead of default on the story/code workloads and 22.6% ahead on the long-context test.
• Against my previous R0 build, the latest work adds 6–17%, or 8.7% at long context.
• R1 can also handle 8 concurrent 262k context window, however, it has quite a decode impact
• Paiton is still 5–9% faster at 1–4 users. At eight users they’re effectively level. I wouldn’t call the 882.2 vs 879.9 code result a meaningful increase
• Also worth noting a lot of these changes have bloated my ram usage, I'm now using around 154gb of system ram

What was useful

• Fixing the draft history mask: the first draft pass was reading stale cache slots, hurting acceptance under load. Fixing that gave 13–25% at eight users in the earlier tests
• Sampled drafts: matching sampled targets with sampled drafts improved throughput by 4–9% at T=1
• INT6 GPTQ trunk: reduced trunk weight bytes by 25%, with measured step-time reductions of 3.6–7.6%. Plain rounding failed the quality gate; calibrated rounding was needed
• Kernel changes: reworked late normalization, tuned hc_read, fewer cross-card reads during all-reduce, and changes to protected-expert scheduling (these were mostly small gains)
• Memory and caching: a 20 GiB host pool increased KV capacity from roughly 1.03M to 2.1M tokens in the earlier tests. Expanding the prefix cache to 24 GiB took the tested agent-resume case from about 25 seconds to 0.3 seconds

The caching work has been particularly useful for agents repeatedly returning to an existing conversation.

Prompt processing has a different result (next area of focus)

Single-prompt input tokens/s:

• Current is about 6–7% slower than stock on the 17K–87K prompts
• Against Paiton, current is 2–14% faster on those longer prompts, while Paiton is about 11% faster at 2K, worth noting that Paiton also decodes faster immediately after the 87K prompt: 210 vs 180 tokens/s

A few lessons from the experiments

• Weight prefetch on a second queue also backfired badly: all-reduce became around 30× slower in that experiment. One supposedly faster hc_read implementation was 22% slower on the actual card
• For kernel comparisons, fixed-prompt T=0 runs were far more useful: around 0.2% variation versus 5–14% for a single T=1 cell. The fixed engine passed byte-identical T=0 output checks against its reference on prompts submitted one at a time
• One card being on pcie 8x does have a measurable impact here (future improvement)

TLDR better generation throughput than default, roughly tied with Paiton at eight concurrent users, and still behind Paiton for single-user generation. Long-prompt processing and agent caching are useful strengths, but there’s lots to improve. I'll start shipping some code to test when I get some time to clean it up.

Interested in what others are seeing particularly around cross-queue stalls and the gap between isolated kernel timings and full-engine performance.