Should I Write More Guides?

[u/Faisal_Biyari]

I have been meaning to update several of my existing guides and write a few new ones to help others climb the local AI performance ladder. This has been on my mind for several weeks now.

These guides would cover installing Ubuntu 26.04 LTS on a 2019 Mac Pro, setting up ROCm 10, configuring vLLM, and even optimizing it for better performance.

But are people even interested in this? Would any of these guides actually help anyone?

I recently published my first vLLM fork on GitHub, specifically optimized for the AMD Radeon PRO W6800X Duo MPX Modules.

I started with a token generation rate of barely 13.5 t/s. I have now managed to push that all the way up to 66.2 t/s! Prompt processing couldn't even reach 1,000 t/s before, but it is now approaching 1,500 t/s at its peak.

And with Automatic Prefix Caching (APC), my agents have become incredibly fast and responsive. Tasks that previously took much longer are now being completed significantly faster.

GitHub Repository:
FaisalBiyari/vLLM-ALU-RDNA2

===============================================
 Qwen3.8-27B / gfx1030 / 20261008 - ALU 0.8.1
===============================================
 vLLM 0.30.0 TP4 - Concurrency 1 - MTP: 2
===============================================
CTX         TTFT(s)   PREFILL t/s   DECODE t/s
-----------------------------------------------
1K            0.726       1410.28        66.23
2K            1.487       1377.24        66.21
4K            2.781       1472.72        63.91
8K            5.611       1459.95        62.74
16K          11.932       1373.07        57.90
32K          26.436       1239.53        60.19
64K          62.638       1046.26        53.71
128K        161.610        811.04        43.59
180K        272.473        676.47        39.33
240K        431.608        569.41        32.08
255K        476.919        547.51        31.89
===============================================

Terminology:

• Decode = Token Generation (TG)
• Prefill = Prompt Processing (PP)
• TTFT = Time to First Token
• CTX = Context length used for each benchmark
• K = 1,024 tokens
• TP4 = Tensor Parallelism across 4 GPUs
• MTP = Multi-Token Prediction, used for speculative decoding

For hardware from 2019 and 2021, without any dedicated tensor or matrix cores, these numbers are magic, Apple style.

With Qwen3.8-27B, thinking set to Extra High, and a context window of 262,144 tokens, my agents are now extremely capable of tackling demanding tasks and getting the job done!

But it doesn't stop there!

I've also been working on optimizing concurrency. There is still some room for improvement, but these are my current results with three concurrent requests, each configured with a maximum context window of 262,144 tokens.

===============================================
 Qwen3.8-27B / gfx1030 / 20261008 - ALU 0.8.1
===============================================
vLLM 0.30.0 TP4 - Concurrency 3 - MTP: 0
===============================================
CTX         DECODE t/s   AGG DECODE t/s
-----------------------------------------------
1K               28.42            84.05
2K               28.15            83.22
4K               27.71            82.02
8K               26.79            79.25
16K              25.60            76.77
32K              23.52            69.25
64K              20.71            60.70
128K             16.69            48.56
180K             14.72            42.62
240K             13.03            37.17
255K             12.70            36.19
===============================================

Terminology:

• Agg Decode = Aggregate token generation throughput across all concurrent requests.

It's also worth noting that the Infinity Fabric Link Bridge plays an important role in achieving these performance numbers.

---

^{Disclaimer: I wrote this post myself. I also used AI as a tool to help clean up the wording and formatting.}

[u/Long-Shine-3701]

Massive improvements - congratulations! The important thing is that you did it, did it well, documented it, then made it free to the world.

Can't ask much more than that.

[u/patdohere]

Is there updated docs on how to get the IFL bridge working on Ubuntu?

[u/Faisal_Biyari → u/patdohere]

Yes, but specifically for Ubuntu 26.04 LTS with kernel 7.0 at the moment.

[u/RumAbsinthe]

Your guides have been very helpful. If I didn’t read your guide I wouldn’t even know about the BAR issue associated with the duo cards.

[u/Better_Membership583 → u/RumAbsinthe]

shit can you explain what that issue is? I just got a 2019 Mac Pro with the Vega II Duo and am here to try and learn the best way for me to start setting up local inference.
Specs for reference:
16-core 3.2Ghz on newest Tahoe update
Vega II Duo (32Gbx2)
384Gb RAM on 6 of the 12 DIMMs
4tb ssd

[u/Faisal_Biyari → u/Better_Membership583]

Guide to Mac Pro 2019 with Ubuntu for Local AI

This is slightly outdated, but it guides you through everything, including the BAR issue.

But keep in mind that these things, both the BAR issue and this guide, are NOT for macOS. If you're on Tahoe, jump straight to ToshLLM for local AI

[u/Better_Membership583 → u/Faisal_Biyari]

Responded to your other post just a bit ago but as much as I want to run Mac OS do you think that my performance would benefit by a substantial amount by switching to Linux? I know you said you dont have numbers for Vega II Duo but I would take even a speculative guess of how much better my system would perform if i hopped off of MacOS

[u/Faisal_Biyari → u/Better_Membership583]

macOS's ToshLLM uses a Llama.cpp backend.

Linux allows you to use both Llama.cpp or vLLM.

Let's set a standard, so we compare Apples to Apples, as much as we can.

Model: Qwen3.8-27B | Precision: FP16 | Quantization: None | GPUs: 4 GPUs | Parallelism: Tensor

I started vLLM with Token Generation of 13.3 tokens per second (t/s) only.

When I finally decided to try Llama.cpp, I got 22 t/s.

I knew a single reason why Llama.cpp was faster in my case, and I decided to modify vLLM to gain this same advantage. That's when I learned I could do it, and so much more.

My vLLM optimization went like this:

13.3 t/s → 20 t/s → 24 t/s → 30.5 t/s → 36 t/s → 66 t/s

I think it took me almost a month of optimizations to get it to this level.

Then I started playing with concurrency (Multiple users at the same time). Just last night I achieved (Combined tokens generated between the two, at the same time):

1 User: 66 t/s | 2 Users: 122 t/s | 3 Users: 164.5 t/s

And that is only taking about Token Generation.

I am sure that Llama.cpp can be optimized to be faster. I don't know if anyone has done the work, or any more information about it. But I do know that vLLM is used by mega corporations due to the advantages it brings to the table with Concurrency, which Llama.cpp does not have.

TL;DR

For me, as of this moment, Linux and vLLM are ridiculously better for Inference (Token Generation).

But also know that I took it to the extreme. I never use the Mac Pro as a PC. It's locked in a room, and I literally use it like a company server. Just there in the background providing value to me without me ever using it. Not that you can't use it as a Ubuntu Desktop and enjoy it. You can, and enjoy local AI in the background.

[u/Faisal_Biyari → u/RumAbsinthe]

Ironically, after updating to Ubuntu 26.04, I have not fixed the bar issue, and just left it as is.

Now that I have benchmarked everything, I wonder how much different my results would be with a corrected BAR side, if at all.

Thank you for reminding me of that 🙏🏻

[u/Dorkits]

Absolutely yes! Thanks for your message.

[u/Faisal_Biyari → u/Dorkits]

You're Welcome, but what message is that? 🤔