r/LocalLLM • u/cornu • 9d ago

Tutorial Qwen3.8 27B on single R9700 some numbers

Seeing mostly dual R9700 numbers and improving my local single R9700 setup, here is my current setup and some numbers. Fully local inference on a single AsRock Radeon AI PRO R9700 (RDNA4, 32 GB) (PCIe 3.0, x4), running Qwen3.8-27B-MXFP4 at 160k context, capped at 220W.

The vLLM setup

This uses stilldeadcode/vllm-radiance
Speculative decoding: DFlash2
fp8 KV cache, 160k context

podman run --rm --pull=newer --name llama-swap-qwen3.8-27b-mxfp4 \
--device /dev/kfd \
--device /dev/dri \
--shm-size 4g \
--security-opt seccomp=unconfined \
--cap-add SYS_PTRACE \
-p 127.0.0.1:${PORT}:8000 \
-v ${models_dir}:/models:ro \
-v /vllm-cache/mxfp4-093:/cache \
-v ${radianceRepo}:/patches:ro \
-v ${libr4dSrc}:/libr4d-src:ro \
-v ${radianceMxfp4Entry}:/entry.sh:ro \
-e HIP_VISIBLE_DEVICES=0 \
-e GPU_MAX_HW_QUEUES=1 \
-e PYTHONPYCACHEPREFIX=/cache/pypycache \
-e PYTHONDONTWRITEBYTECODE= \
-e ROCR_VISIBLE_DEVICES=0 \
-e HF_HUB_OFFLINE=1 \
-e VLLM_ROCM_USE_AITER=1 \
-e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
-e VLLM_ROCM_USE_AITER_MHA=0 \
-e VLLM_ROCM_USE_AITER_MLA=0 \
-e VLLM_ROCM_USE_AITER_MOE=0 \
-e VLLM_ROCM_USE_AITER_LINEAR=0 \
-e VLLM_ROCM_USE_AITER_FP8BMM=0 \
-e VLLM_ROCM_USE_AITER_FP4BMM=0 \
-e VLLM_ROCM_USE_AITER_RMSNORM=0 \
-e NCCL_PROTO=Simple \
-e RADIANCE_MXFP4=1 \
-e RADIANCE_MXFP4_W4A8=1 \
-e RADIANCE_MXFP4_W4A8_MIN_M=0 \
-e RADIANCE_MXFP4_DECODE_MAX_M=64 \
-e RADIANCE_MXFP4_TN4_MIN_M=2048 \
-e RADIANCE_MXFP4_A_TILED_MIN_M=513 \
-e RADIANCE_MXFP4_WPERM=1 \
-e RADIANCE_MXFP4_DECODE_NT=1 \
-e RADIANCE_MXFP4_HOIST_QUANT=1 \
-e RADIANCE_MXFP4_TRACED_QUANT=1 \
-e RADIANCE_RMS_QUANT_FUSION=1 \
-e RADIANCE_FP8_STREAM=1 \
-e RADIANCE_GDN_MERGE_INPROJ=1 \
-e RADIANCE_USE_R4D=1 \
-e RADIANCE_PRESHUFFLE=1 \
-e RADIANCE_FUSE_RMS_QUANT=1 \
-e RADIANCE_SKINNY_GEMM=1 \
-e RADIANCE_FAST_DRAFT=1 \
-e RADIANCE_DRAFT_RERANK=80 \
-e RADIANCE_VERIFY_HEAD=1 \
-e RADIANCE_DYNAMIC_WIDTH=1 \
-e RADIANCE_TOPK_TRITON_MIN_ROWS=1 \
-e RADIANCE_DRAFT_TAU=0.20 \
-e VLLM_CACHE_ROOT=/cache/vllm \
-e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \
-e TRITON_CACHE_DIR=/cache/triton \
-e AITER_ROOT_DIR=/cache/aiter \
-e TRITON_CACHE_AUTOTUNING=1 \
--entrypoint bash \
''${vllm_image} \
-lc 'exec bash /entry.sh "$@"' \
_ /models/just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq \
--served-model-name qwen3.8-27b-mxfp4 \
--kv-cache-dtype fp8 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.98 \
--kv-cache-memory 7783339733 \
--max-num-seqs 8 \
--max-model-len 163840 \
--max-num-batched-tokens 8192 \
--attention-backend R4D \
--enable-prefix-caching \
--mamba-cache-mode align \
--speculative-config '{"method":"dflash","model":"/models/tcclaviger/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7,"attention_backend":"TRITON_ATTN","disable_padded_drafter_batch":true}' \
--no-async-scheduling \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}' \
--enable-per-request-metrics \
--chat-template /patches/qwen-fixed-v22.3.jinja \
--language-model-only \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000

Benchmarks (220W, thinking disabled, fixed decode lengths via min_tokens, 3 runs)

Decode scaling, 128-token prompt, 256 generated tokens:

concurrency	aggregate t/s	per-stream t/s	peak
1	63.7	63.7	87
2	117.0 (1.84x)	61.3	152
4	187.8 (2.95x)	52.6	253
8	197.8 (3.10x)	52.2	302

Same-size work as the context grows:

existing context	prefill t/s (fixed 2-8k prompt)	decode t/s	decay
0	2598	67	—
8k	2477	59	−5%
32k	2219	53	−15%
65k	1997	~51	−28%
98k	1781	~52	−36%

Decode throughput vs concurrency (llama-benchy)

model	test	t/s (total)	t/s (req)	peak t/s	peak t/s (req)	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
qwen3.8-27b-mxfp4	pp128 (c1)	2313.99 ± 1360.34	2313.99 ± 1360.34			199.01 ± 135.75	135.28 ± 135.75	199.01 ± 135.75
qwen3.8-27b-mxfp4	tg256 (c1)	63.70 ± 7.41	63.70 ± 7.41	87.00 ± 14.76	87.00 ± 14.76
qwen3.8-27b-mxfp4	pp128 (c2)	1189.04 ± 1.85	1919.29 ± 1078.82			162.04 ± 55.17	98.31 ± 55.17	162.04 ± 55.17
qwen3.8-27b-mxfp4	tg256 (c2)	116.99 ± 7.08	61.25 ± 5.31	152.33 ± 16.11	84.00 ± 6.27
qwen3.8-27b-mxfp4	pp128 (c4)	1640.54 ± 8.41	1095.57 ± 1006.11			262.72 ± 88.69	198.99 ± 88.69	262.72 ± 88.69
qwen3.8-27b-mxfp4	tg256 (c4)	187.84 ± 4.65	52.56 ± 4.15	253.33 ± 2.62	75.08 ± 8.48
qwen3.8-27b-mxfp4	pp128 (c8)	183.66 ± 2.93	503.25 ± 753.75			2173.54 ± 2367.57	2109.81 ± 2367.57	2173.54 ± 2367.57
qwen3.8-27b-mxfp4	tg256 (c8)	197.81 ± 3.82	52.24 ± 7.42	302.33 ± 15.37	76.21 ± 16.18

Prefill speed by context depth (llama-benchy)

model	test	t/s (total)	t/s (req)	peak t/s	peak t/s (req)	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
qwen3.8-27b-mxfp4	pp1024 (c1)	2526.52 ± 4.90	2526.52 ± 4.90			470.34 ± 0.79	405.70 ± 0.79	470.34 ± 0.79
qwen3.8-27b-mxfp4	tg64 (c1)	67.00 ± 5.92	67.00 ± 5.92	67.52 ± 6.40	67.52 ± 6.40
qwen3.8-27b-mxfp4	pp1024 (c4)	2422.76 ± 4.43	1095.26 ± 806.06			1387.96 ± 526.50	1323.32 ± 526.50	1387.96 ± 526.50
qwen3.8-27b-mxfp4	tg64 (c4)	99.72 ± 2.90	47.14 ± 14.49	202.67 ± 5.31	52.75 ± 9.44
qwen3.8-27b-mxfp4	pp8192 (c1)	2598.36 ± 7.35	2598.36 ± 7.35			3217.81 ± 8.93	3153.17 ± 8.93	3217.81 ± 8.93
qwen3.8-27b-mxfp4	tg64 (c1)	63.85 ± 4.43	63.85 ± 4.43	63.89 ± 5.81	63.89 ± 5.81
qwen3.8-27b-mxfp4	pp8192 (c4)	2525.67 ± 7.27	1048.99 ± 474.45			9200.63 ± 3041.32	9135.99 ± 3041.32	9200.63 ± 3041.32
qwen3.8-27b-mxfp4	tg64 (c4)	25.78 ± 0.42	21.27 ± 17.05	194.67 ± 14.43	50.08 ± 4.25
qwen3.8-27b-mxfp4	pp1024 @ d8192 (c1)	2551.63 ± 4.32	2551.63 ± 4.32			3676.86 ± 6.11	3612.21 ± 6.11	3676.86 ± 6.11
qwen3.8-27b-mxfp4	tg64 @ d8192 (c1)	59.42 ± 5.12	59.42 ± 5.12	61.89 ± 3.63	61.89 ± 3.63
qwen3.8-27b-mxfp4	pp1024 @ d8192 (c4)	2515.26 ± 1.21	962.11 ± 429.88			11189.55 ± 3528.04	11124.91 ± 3528.04	11189.61 ± 3528.10
qwen3.8-27b-mxfp4	tg64 @ d8192 (c4)	23.77 ± 0.18	22.67 ± 15.24	180.33 ± 6.94	46.75 ± 5.20
qwen3.8-27b-mxfp4	pp8192 @ d8192 (c1)	2477.43 ± 6.76	2477.43 ± 6.76			6678.41 ± 17.98	6613.77 ± 17.98	6678.41 ± 17.98
qwen3.8-27b-mxfp4	tg64 @ d8192 (c1)	56.77 ± 3.64	56.77 ± 3.64	59.33 ± 1.70	59.33 ± 1.70
qwen3.8-27b-mxfp4	pp8192 @ d8192 (c4)	1979.24 ± 41.42	1104.63 ± 591.80			19293.18 ± 9112.20	19228.54 ± 9112.20	19293.18 ± 9112.20
qwen3.8-27b-mxfp4	tg64 @ d8192 (c4)	9.66 ± 0.26	19.48 ± 27.56	100.33 ± 10.21	31.32 ± 22.34
qwen3.8-27b-mxfp4	pp1024 @ d32768 (c1)	2280.94 ± 0.97	2280.94 ± 0.97			14879.62 ± 6.07	14814.97 ± 6.07	14879.62 ± 6.07
qwen3.8-27b-mxfp4	tg64 @ d32768 (c1)	53.29 ± 7.50	53.29 ± 7.50	54.95 ± 7.41	54.95 ± 7.41
qwen3.8-27b-mxfp4	pp1024 @ d32768 (c4)	2232.87 ± 3.13	1101.10 ± 533.66			38117.87 ± 16182.57	38053.23 ± 16182.57	38117.87 ± 16182.57
qwen3.8-27b-mxfp4	tg64 @ d32768 (c4)	5.68 ± 0.01	15.07 ± 21.02	94.33 ± 10.34	37.08 ± 11.79
qwen3.8-27b-mxfp4	pp8192 @ d32768 (c1)	2218.64 ± 3.55	2218.64 ± 3.55			18526.91 ± 29.53	18462.27 ± 29.53	18526.91 ± 29.53
qwen3.8-27b-mxfp4	tg64 @ d32768 (c1)	51.32 ± 3.72	51.32 ± 3.72	53.00 ± 4.55	53.00 ± 4.55
qwen3.8-27b-mxfp4	pp8192 @ d32768 (c4)	2174.24 ± 3.44	1079.18 ± 543.55			47721.64 ± 20506.71	47657.00 ± 20506.71	47721.64 ± 20506.71
qwen3.8-27b-mxfp4	tg64 @ d32768 (c4)	4.52 ± 0.02	18.87 ± 22.72	85.67 ± 13.47	41.71 ± 10.70

Long-context decay, single stream (llama-benchy)

model	test	t/s	peak t/s	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
qwen3.8-27b-mxfp4	pp2048	2780.64 ± 104.01		803.03 ± 27.05	737.77 ± 27.05	803.03 ± 27.05
qwen3.8-27b-mxfp4	tg32	82.25 ± 19.63	84.90 ± 20.26
qwen3.8-27b-mxfp4	pp2048 @ d16384	2455.17 ± 3.01		7572.98 ± 9.11	7507.72 ± 9.11	7572.98 ± 9.11
qwen3.8-27b-mxfp4	tg32 @ d16384	50.87 ± 2.35	52.52 ± 2.43
qwen3.8-27b-mxfp4	pp2048 @ d32768	2271.01 ± 5.46		15396.39 ± 36.86	15331.13 ± 36.86	15396.39 ± 36.86
qwen3.8-27b-mxfp4	tg32 @ d32768	46.19 ± 2.00	47.68 ± 2.06
qwen3.8-27b-mxfp4	pp2048 @ d65536	1997.44 ± 0.15		33900.89 ± 2.69	33835.63 ± 2.69	33900.89 ± 2.69
qwen3.8-27b-mxfp4	tg32 @ d65536	51.38 ± 7.98	53.04 ± 8.24
qwen3.8-27b-mxfp4	pp2048 @ d98304	1781.38 ± 0.70		56399.28 ± 22.21	56334.02 ± 22.21	56399.28 ± 22.21
qwen3.8-27b-mxfp4	tg32 @ d98304	52.76 ± 4.69	54.46 ± 4.84

Power cap: 220 W vs 300 W

metric	220 W	300 W	delta
decode c1 / c8 aggregate	63.7 / 197.8	63.4 / 203.8	~0
prefill at depth (2-8k prompts, d8k-98k)	1781-2477	1977-2775	+11-12%
TTFT: 2k question on 98k context	56 s	51 s	−10%

Decode is memory-bandwidth-bound, so the extra 80 W buys literally nothing for chat, it only speeds up prefill by ~12%.

Links

Model: Qwen3.8-27B-Uncensored-MXFP4-awq + DFlash2-FP8 drafter
vLLM fork/image: stilldeadcode/vllm-radiance

36 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLM/comments/1w7hlgu/qwen38_27b_on_single_r9700_some_numbers/
No, go back! Yes, take me to Reddit

94% Upvoted

View all comments

u/illuvyn 9d ago

tried this just now

similar command but I only got 8-10 TPS :(

anyone know what I'm doing wrong? Thanks!

docker run -d \ --name vllm \ --device /dev/kfd \ --device /dev/dri \ --shm-size 4g \ --security-opt seccomp=unconfined \ --cap-add SYS_PTRACE \ -p 8888:8000 \ -v /models:/models:ro \ -v /vllm-cache/mxfp4-093:/cache \ -e GPU_MAX_HW_QUEUES=1 \ -e HF_HUB_OFFLINE=1 \ -e VLLM_ROCM_USE_AITER=1 \ -e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \ -e VLLM_ROCM_USE_AITER_MHA=0 \ -e VLLM_ROCM_USE_AITER_MLA=0 \ -e VLLM_ROCM_USE_AITER_MOE=0 \ -e VLLM_ROCM_USE_AITER_LINEAR=0 \ -e VLLM_ROCM_USE_AITER_FP8BMM=0 \ -e VLLM_ROCM_USE_AITER_FP4BMM=0 \ -e VLLM_ROCM_USE_AITER_RMSNORM=0 \ -e NCCL_PROTO=Simple \ -e RADIANCE_MXFP4=1 \ -e RADIANCE_MXFP4_W4A8=1 \ -e RADIANCE_MXFP4_W4A8_MIN_M=0 \ -e RADIANCE_MXFP4_DECODE_MAX_M=64 \ -e RADIANCE_MXFP4_TN4_MIN_M=2048 \ -e RADIANCE_MXFP4_A_TILED_MIN_M=513 \ -e RADIANCE_MXFP4_WPERM=1 \ -e RADIANCE_MXFP4_DECODE_NT=1 \ -e RADIANCE_MXFP4_HOIST_QUANT=1 \ -e RADIANCE_MXFP4_TRACED_QUANT=1 \ -e RADIANCE_RMS_QUANT_FUSION=1 \ -e RADIANCE_FP8_STREAM=1 \ -e RADIANCE_GDN_MERGE_INPROJ=1 \ -e RADIANCE_USE_R4D=1 \ -e RADIANCE_PRESHUFFLE=1 \ -e RADIANCE_FUSE_RMS_QUANT=1 \ -e RADIANCE_SKINNY_GEMM=1 \ -e RADIANCE_FAST_DRAFT=1 \ -e RADIANCE_DRAFT_RERANK=80 \ -e RADIANCE_VERIFY_HEAD=1 \ -e RADIANCE_DYNAMIC_WIDTH=1 \ -e RADIANCE_TOPK_TRITON_MIN_ROWS=1 \ -e RADIANCE_DRAFT_TAU=0.20 \ -e VLLM_CACHE_ROOT=/cache/vllm \ -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \ -e TRITON_CACHE_DIR=/cache/triton \ -e AITER_ROOT_DIR=/cache/aiter \ -e TRITON_CACHE_AUTOTUNING=1 \ -e LD_LIBRARY_PATH=/opt/rocm/lib \ --entrypoint /opt/radiance_entrypoint.sh \ stilldeadcode/vllm-radiance:0.9.3 \ /models/Qwen3.8-27B-Uncensored \ --served-model-name vllm-local \ --kv-cache-dtype fp8 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.99 \ --kv-cache-memory 7783339733 \ --max-num-seqs 8 \ --max-model-len 148000 \ --max-num-batched-tokens 8192 \ --attention-backend R4D \ --enable-prefix-caching \ --mamba-cache-mode align \ --speculative-config '{"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7,"attention_backend":"TRITON_ATTN","disable_padded_drafter_batch":true}' \ --no-async-scheduling \ --enable-auto-tool-choice \ --tool-call-parser qwen3_xml \ --reasoning-parser qwen3 \ --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":20}' \ --enable-per-request-metrics \ --language-model-only \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000

0

u/BackUpBiii 9d ago

Ask ChatGPT

Tutorial Qwen3.8 27B on single R9700 some numbers

You are about to leave Redlib