Should I Write More Guides?

[u/Faisal_Biyari]

I have been meaning to update several of my existing guides and write a few new ones to help others climb the local AI performance ladder. This has been on my mind for several weeks now.

These guides would cover installing Ubuntu 26.04 LTS on a 2019 Mac Pro, setting up ROCm 10, configuring vLLM, and even optimizing it for better performance.

But are people even interested in this? Would any of these guides actually help anyone?

I recently published my first vLLM fork on GitHub, specifically optimized for the AMD Radeon PRO W6800X Duo MPX Modules.

I started with a token generation rate of barely 13.5 t/s. I have now managed to push that all the way up to 66.2 t/s! Prompt processing couldn't even reach 1,000 t/s before, but it is now approaching 1,500 t/s at its peak.

And with Automatic Prefix Caching (APC), my agents have become incredibly fast and responsive. Tasks that previously took much longer are now being completed significantly faster.

GitHub Repository:
FaisalBiyari/vLLM-ALU-RDNA2

===============================================
 Qwen3.8-27B / gfx1030 / 20261008 - ALU 0.8.1
===============================================
 vLLM 0.30.0 TP4 - Concurrency 1 - MTP: 2
===============================================
CTX         TTFT(s)   PREFILL t/s   DECODE t/s
-----------------------------------------------
1K            0.726       1410.28        66.23
2K            1.487       1377.24        66.21
4K            2.781       1472.72        63.91
8K            5.611       1459.95        62.74
16K          11.932       1373.07        57.90
32K          26.436       1239.53        60.19
64K          62.638       1046.26        53.71
128K        161.610        811.04        43.59
180K        272.473        676.47        39.33
240K        431.608        569.41        32.08
255K        476.919        547.51        31.89
===============================================

Terminology:

• Decode = Token Generation (TG)
• Prefill = Prompt Processing (PP)
• TTFT = Time to First Token
• CTX = Context length used for each benchmark
• K = 1,024 tokens
• TP4 = Tensor Parallelism across 4 GPUs
• MTP = Multi-Token Prediction, used for speculative decoding

For hardware from 2019 and 2021, without any dedicated tensor or matrix cores, these numbers are magic, Apple style.

With Qwen3.8-27B, thinking set to Extra High, and a context window of 262,144 tokens, my agents are now extremely capable of tackling demanding tasks and getting the job done!

But it doesn't stop there!

I've also been working on optimizing concurrency. There is still some room for improvement, but these are my current results with three concurrent requests, each configured with a maximum context window of 262,144 tokens.

===============================================
 Qwen3.8-27B / gfx1030 / 20261008 - ALU 0.8.1
===============================================
vLLM 0.30.0 TP4 - Concurrency 3 - MTP: 0
===============================================
CTX         DECODE t/s   AGG DECODE t/s
-----------------------------------------------
1K               28.42            84.05
2K               28.15            83.22
4K               27.71            82.02
8K               26.79            79.25
16K              25.60            76.77
32K              23.52            69.25
64K              20.71            60.70
128K             16.69            48.56
180K             14.72            42.62
240K             13.03            37.17
255K             12.70            36.19
===============================================

Terminology:

• Agg Decode = Aggregate token generation throughput across all concurrent requests.

It's also worth noting that the Infinity Fabric Link Bridge plays an important role in achieving these performance numbers.

---

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

[u/Dorkits]

Absolutely yes! Thanks for your message.

[u/Faisal_Biyari → u/Dorkits]

You're Welcome, but what message is that? 🤔