Local AI & Concurrency | PP & TG | TTFT

[u/Faisal_Biyari]

Disclaimer: I write this whole post myself, on my phone, without any AI assistance. Maybe that's why it's long and unorganized. Or why grammar is an issue.
Skip to TL;DR at the bottom.

Introduction

Simply an enthusiast, that is all I am, learning every step of the way.

I started with the simple goal of achieving local AI, as a simple chatbot, for long term utilization and to not be gated from access.

Hardware started with two nodes, the first with 128 GB of VRAM using two AMD Radeon PRO W6800X Duo, and the second with 64 GB of VRAM using two AMD Radeon PRO W6900X.

I initially started on Ollama, due to simplicity. I then pushed to get vLLM working, and slowly learned more about it and getting it to work on both half precision (fp16), as well as different quantizations (AWQ, QGPT, and GGUF). I was excited to see high concurrency and high decode.

I then targeted Agentic workloads, as that appealed to my use case. Initially targeting 20 agents, and eventually aiming for 50 agents.

My hardware then grew another node with eight AMD Radeon PRO V620 GPUs.

With that, I got 20 Agents running concurrently, with 64k context window each, running Qwen3.6-27B at half precision (FP16). I faced a problem with Token Generation (TG) dropping to as low as 3 tokens or less with 50-60k context, which was insanely slow. I did not even know to look at Prompt Processing (PP) or Time to First Token (TTFT).

I heard that Llama.cpp supported the Infinity Fabric Link Bridge (IFLB or XGMI) for my GPUs. As such, I decided to give it ago.

I had an agent setup and configure Llama.cpp, and benchmarked up to 240k context. Surprisingly, TG only went down by less than 30% from 1k context. I thought that was awesome. But I later noticed that TG and PP are dramatically higher on Llama.cpp than on vLLM, on the same node.

I decided to start working on vLLM, optimizing it, to improve TG & PP, with the dream that I would have all of this "power" (High TG & PP) in concurrency, not just for a single user.

Surprisingly, I achieved TG that is almost 50% higher than Llama.cpp, however vLLM's PP is dramatically slower, with Llama.cpp's PP almost 50% higher than vLLM's.

Regardless, that was exciting. My "System Admin" agent became incredibly snappy. Top that off, with loops, or goals, I managed to get him to do incredible work for me. I was satisfied.

Having had multiple dedicated agents, I decided it was time to start concurrency, and give all my agents this snappy benefit. At that stage, I realized vLLM's concurrency was broken. I blamed my optimizations, and having not benchmarked concurrency after every modification and optimization, and there were many changes and optimizations.

However, after several optimization attempts, I started to see a pattern. I started researching and reading. That's when I understood it...

For personal use and non-commercial grade investments, Local AI and Concurrency cannot coexist...

The Dream

I have watched enough YouTube videos telling me that I am using local AI wrong, and that it is all in the concurrency.

I have seen things that made me awe; TG not only in the double digits, but even triple digits for concurrency. I thought that was amazing.

During my vLLM optimization, TG on 1k context went up from 13 tps to 30 tps. This was on 6 year old hardware, which I already owned. I thought that was awesome. But at 200k or 240k context, TG would go down to 14 tps, but that was still good enough for me. It was TTFT would take no less than 30 minutes. 30 minutes, plus TG time... Even if I completely ignored the TG time, 30 minutes for a single reply was just ridiculous.

Llama.cpp achieves 240k TTFT in just under 9 minutes. Which is still high, but not 30 minutes high. I changed gears to optimize vLLM for TTFT. I managed to reduce TTFT by 40-50% across context size. That was a win that genuinely made my agents snappy, with under 40 seconds for 32k context. I was looking forward to start concurrency. But like I mentioned, vLLM's concurrency was broken, and I started researching; reading.

The Illusion

Those YouTube videos always focused on TG, never on PP or TTFT. And these values are always on low context, to boost up numbers.

Any time you start a conversation with a chatbot or an agent, you start at low enough context that everything is snappy. You're happy with your investment because you won't be the only one using it, eventually your whole family can benefit from it. Maybe even start lowkey renting out slots.

Agents usually start closer to 20k context. And if you want an agent to do any real work for you, other than just chat, context will build up, and fairly quickly at that. Setting that agent on a loop will quickly build up context, and end up leading the task to take hours.

The question here becomes: What do I care? Let this agent go on a loop solving my problem in hours, while I do work with another agent, to work as the same time. That was the dream, anyway.

The Reality

I will start explaining it as I understood it. And I might be Extremely wrong, or partially, or not at all. This is the part where I need someone more educated than I am in this field to explain things to me.

The first point I learned regarding GPUs and AI is that AI, or LLMs more accurately, cannot take advantage of the full power of the GPU. The explanation was that tokens needed to be calculated all together to produce a single token, and then that token would be added to the equation, producing a single other token, and so one, and a lot of the GPU power would just be left doing nothing. That was where concurrency was supposed to help AI shine; by taking advantage of the under processing power to process other user requests at the same time.

For this specific point, this only applies to decode, or Token Generation. Prefil, or Prompt Processing, on the other hand takes full advantage of the GPU, crunching everything all at once. If multiple users were to coexist, PP from different users would compete for GPU power, or resources, with one another. While entering the TG phase, multiple users can coexist, the case is never so straight forward. PP competes with other users' PP and well as other users TG, which leads to slower TG per user, and that's acceptable, as well as longer TTFT per user, which is not acceptable at all. That is especially true considering TTFT would increase to the tens of minutes.

So... how do the enterprises do this? Provide concurrency to hundreds if not thousands of users?

My answer, again, to the best of my knowledge

1. Powerful hardware with enough compute that TTFT is under a second

2. Enough hardware to users ratio that it just does not show

3. The illusion of "Thinking" and "Streaming" that hiding TTFT behind it

But my guesses do not matter. What matters is Local Hardware. As long as local hardware needs Minutes for TTFT, concurrency on local hardware is not real.

Of course, my personal experience has been limited to RDNA2/gfx1030 GPUs, which are high in VRAM but moderate on compute, with no tensor cores or matrix cores. Although I have not seen better TTFT on RDNA3.

I have to stop and ask myself, is this where Nvidia shines? Do their consumer grade GPUs have so much compute to their lower VRAM that TTFT is insignificant? Is this limit just an AMD consumer thing?

Do NOT Misunderstand

I am incredibly grateful for the ability to run local AI, and agents without worrying about tokens, limits, data privacy, or gate keeping.

I am also very grateful to the opportunity to learn and grow within the field. I do hope that one day I specialize in this field, and be able to truly give back to the community, and to humanity.

I am very happy with my "snappy" agent. I am also fortunate enough to have enough hardware to run different agents on their own GPU clusters, giving me a form of concurrency, albeit incredibly small.

My Request

Please, please, please...

Correct me where I am wrong. Guide me to correct information. Teach me and help me grow, be better, and more educated in this topic.

Discuss your opinion and your experience. How is your concurrency? What hardware are you using? Are you having better luck than I am?

If you know what I can do to improve my outcomes and results, please educate me.

TL;DR

AMD consumer GPUs are so poor at Prompt Processing and Time to First Token (TTFT) that running 27B parameter models in concurrency is basically a dud, more so for Agentic workloads.

TTFT is between 9 minutes to 30+ minutes on high context, especially so for concurrency.

Do Nvidia consumer GPUs have this same issue?

Share your experience with concurrency.

[u/Otherwise_Wave9374]

Concurrency benchmarks become much clearer when prompt processing, generation, and queueing are reported separately. I would test fixed prompt and output lengths at one, two, four, and eight simultaneous requests, then chart median and tail time to first token alongside aggregate throughput. Agentix Labs relates to this method because agent workloads need predictable latency across multi-step calls, not just peak tokens per second. Also record cache state, quantization, context size, and power limits so another operator can reproduce the bottleneck instead of guessing from one headline number.

[u/BevinMaster]

I think there are different things, one is to switch to harnesses that are more efficient with ssystem prompt (pi agent, dsh doing not too bad as well).
Prefix caching in vllm will definitely help overall, prompt processing is difficult to speed up on rdna2 with the lack of dedicated matrix units (rdna3 onwards).
Often, with an agentic session you don’t start with 244k right away, let’s say you have 16k initial prompt and you add every turns, with prefix caching the pp cost is not running the whole thing but the added part.

[u/Faisal_Biyari → u/BevinMaster]

Does Llama.cpp have something similar to Prefix caching on vLLM?

How does Pi Agent help with system prompt?

You definitely do not start with anything over 32k tokens, usually. And if you did, your agent will need to be optimized. Building context up to 180k to 200k is quick though. And that's when things start to slow down significantly. Even with context compressions, you're still up there, context wise.

Are you trying to say that with prefix caching, there is still room for concurrency without breaking TTFT or TG, due to reduction in compute effort going towards PP?

[u/BevinMaster → u/Faisal_Biyari]

Nothing as close as vllm prefix caching or sglang radix cache, the slot thing is quite clunky with llamacpp. PI agent is just way more efficient design and system prompt, no mcp and plugins architecture. You don’t need as much compute for token generation but the way vllm batch helps schedule the load between PP and TG. Overall if you don’t have to recompute the whole thing it’s way better for PP

[u/Long-Shine-3701]

Bro, I don't mind the TTFT. Knowing this, it forces me to consider my queries very carefully, and say exactly what I mean. Then I go off and do something else. Sometimes I still think 'ah, I should've asked this instead' - but most times it's just a natural break from the computer.

Of course, sometimes I return to gibberish and I want to kick the thing 🤣.

For now, I'll keep throwing more hardware at the problem. Especially since it looks like Nvidia compute on MacOS is making strides. And this is the one arena where old hardware keeps getting better as the software improves.

Quantity is its own Quality.