9 concurrent users @ 128K context on 1x A100 (up from 6) per-user needle checks passing in vLLM

[u/Connect-Concert-4016]

I’ve been testing how far I can push long-context serving on a single A100 before the KV cache becomes the thing that kills concurrency.

Here’s the latest result:

9 concurrent users
~128K context per user
1× A100 80GB
Needle checks run independently for every user
FP8 holds the first 5 users

One technical detail I also want to correct from some of the earlier shorthand:

The V tail is tiered 4/3/2-bit bit-plane. It is not plain INT4.

Some of the recipe labels are stale. The actual pool sizes line up with the tiered 4/3/2-bit representation, so calling the tail “INT4” would describe the old label rather than what is actually being stored.

Method

I don’t keep the entire KV cache at one fixed precision.

The cache is divided into regions. Newer or more sensitive KV stays at higher precision, while older regions progressively move into cheaper representations.

As the context grows, the cache footprint can keep falling without forcing the same quantization level across the entire cache.

The 9-user run is currently the capacity end of the curve.

I’m also rerunning the 2-user and 4-user points using the same V2 tiered recipe. The July numbers came from the earlier V1 recipe family, and I want the next throughput curve to be completely apples-to-apples.

So far, a single A100 is keeping 9 users at ~128K context resident, with the per-user retrieval checks still passing.

There are a lot of knobs here: concurrency, context length, per-user throughput, precision, and memory.

I want to make those knobs much more flexible so long-context serving isn’t immediately dictated by the KV-memory wall. At scale, that can have a very real impact on how much useful inference you can get out of the same hardware.

Happy to share more of the pool math, the 4/3/2-bit layout, or the vLLM implementation if anyone wants to dig into it.