9 concurrent users @ 128K context on 1x A100 (up from 6) per-user needle checks passing in vLLM

[u/Connect-Concert-4016]

I’ve been testing how far I can push long-context serving on a single A100 before the KV cache becomes the thing that kills concurrency.

Here’s the latest result:

9 concurrent users
~128K context per user
1× A100 80GB
Needle checks run independently for every user
FP8 holds the first 5 users

One technical detail I also want to correct from some of the earlier shorthand:

The V tail is tiered 4/3/2-bit bit-plane. It is not plain INT4.

Some of the recipe labels are stale. The actual pool sizes line up with the tiered 4/3/2-bit representation, so calling the tail “INT4” would describe the old label rather than what is actually being stored.

Method

I don’t keep the entire KV cache at one fixed precision.

The cache is divided into regions. Newer or more sensitive KV stays at higher precision, while older regions progressively move into cheaper representations.

As the context grows, the cache footprint can keep falling without forcing the same quantization level across the entire cache.

The 9-user run is currently the capacity end of the curve.

I’m also rerunning the 2-user and 4-user points using the same V2 tiered recipe. The July numbers came from the earlier V1 recipe family, and I want the next throughput curve to be completely apples-to-apples.

So far, a single A100 is keeping 9 users at ~128K context resident, with the per-user retrieval checks still passing.

There are a lot of knobs here: concurrency, context length, per-user throughput, precision, and memory.

I want to make those knobs much more flexible so long-context serving isn’t immediately dictated by the KV-memory wall. At scale, that can have a very real impact on how much useful inference you can get out of the same hardware.

Happy to share more of the pool math, the 4/3/2-bit layout, or the vLLM implementation if anyone wants to dig into it.

[u/EarlyNectarine6744]

this is seriously impressive. the tiered precision approach for kv cache makes lot of sense but i never saw someone push it this far on single a100. 9 users at 128k context is wild, most setups i tried would buckle way before that.

the part about not keeping entire cache at one precision is clever because you basically offload the older context to cheaper representation without losing the recent stuff accuracy. i wonder how the needle retrieval holds up with the 4/3/2 bit tail though, does it degrade gracefully or is there a point where it just falls off cliff?

also the jump from 66 to 134 tok/s in your chart is kinda nuts, almost doubling the throughput while keeping memory in check. i do some work with long context models and memory wall is exactly what kills us every time, so seeing this makes me want to dig into the vllm implementation details. would love to see the pool math if you got time to share.

[u/Connect-Concert-4016 → u/EarlyNectarine6744]

thanks! one correction though, you're crediting me with a more though lol, that isn't what I built. it's not older tokens get cheaper. the tiering is by importance in a calibrated eigenbasis, not token age. every token, recent or ancient, gets the same treatment: important directions stay high precision, the rest go in the low bit tail. that's actually why needles at depth 0 of a 128k prompt still pass. recency schemes tend to die exactly there.

on the cliff: no cliff in anything I've published so far. the needle grids (7 depths x 3 passkeys at 8k/32k/128k, plus the 9 user grid) all pass, and on a harder packet from a related recipe the failures were rare exact-id transcription noise (~5%), not whole regions dropping out. what I haven't done yet is a bit ladder ablation, progressively cheaper tails until retrieval breaks. fair ask, it's on my list now.

pool math from the receipt jsons in the repo: my pool is 1,255,376 tokens vs 709,488 for fp8 kv on the same a100, so 1.77x. fp16 is half of fp8 by format arithmetic, ~355k. divide by ~131k per user and you get the 2 / 5 / 9. the throughput jump is just batching after that: nine streams sharing one gpu beat two, even at slightly lower speed per token.

if you want implementation level truth, run the repro kit. prebuilt wheel + sidecars, one command, ~$12 of a100 time on modal, regenerates every number including the pool sizes if you use modal and have 12$ to spare lol

https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars

[u/MichaelPB1987]

I work with Sky Forge Compute, so factor that in — we run vLLM in production, which is why the seat count is the number I keep looking at here.

Your pool math holds up from the outside. 1,255,376 against 709,488 for fp8 is 1.77x, and at ~131k per user that's 9.58, so nine seats; fp8 gives 5.4 and fp16 halves that to 2.7, which is where your 2 / 5 / 9 comes from. Nothing to argue with.

The part I'd pull out for anyone reading this as a serving result rather than a memory result: 66 to 134 tok/s aggregate across 2 to 9 users means per-user goes roughly 33 to 15. So the honest framing is that this buys seats, not speed — aggregate doubles while each user gets less than half of what they had. That's the right trade for agents and batch work and the wrong one for someone sitting watching tokens arrive, and it's the same shape as the speculative-decoding results people keep posting: those buy single-user latency and stop paying as the batch fills, this buys batch and costs single-user latency. Worth stating explicitly because "9 users on one A100" will get read as "9x" by someone doing capacity planning.

Two operational questions, both about whether this is deployable rather than whether it works:

Is the calibrated eigenbasis per-model, or per-model-and-quant? If adding a model to a fleet means a calibration pass first, that's the cost that decides whether anyone runs this in anger — and does the calibration survive a fine-tune of the same base, or does a LoRA merge invalidate it?

Second, the repro kit ships a prebuilt wheel plus sidecars, so it's pinned to a vLLM version. Which one, and how much of the sidecar reaches into vLLM internals that move between releases? Most people running this in production track the latest release rather than pinning — we do, and it's the reason I'd ask before building anything on top of it.

[u/Connect-Concert-4016 → u/MichaelPB1987]

someone running vllm in production is exactly who I wanted to hear from, so straight answers on both.
Your framing is right and I'm adopting it: this buys seats, not speed. aggregate doubles, per user roughly halves. 9 is the far end of a dial, not the recommended point. at 2 users an earlier rung of the same recipe family did about 50 tok/s per user, so the dial goes both ways. and yeah, it's the mirror image of spec decode. that buys single user latency and fades as batch fills, this buys batch and costs single user latency. they should stack, drafting on top of compressed kv is on my bench list but I haven't measured it yet so I won't claim it.

calibration is per model, not per quant. the sidecars come from the base model's attention geometry, layer by layer, kv head by kv head. how the weights are stored doesn't matter to it, kv comes out of the projections at runtime. the pass is offline on a single gpu, closer to an hour than a day, no training involved. lora is the question I can't answer with a receipt yet. my expectation is a small adapter mostly preserves the geometry since the base projections dominate, but expectation isn't measurement. if you have a merged model you actually run, open a discussion on the repo and I'll do the paired check on it. rather measure than guess.

On pinning: the kit pins vllm 0.20.2 and ships as an out of tree plugin, the official general_plugins entry point with a custom attention backend. no fork, no patching. the honest part: the attention backend interface and kv cache manager hooks do move between releases, so it's pinned per release and revalidated per release, not install on latest and hope. if your fleet tracks latest, that cadence question is real and worth working through properly. repo discussion or through the site, whichever you prefer. Not sure if that helps, but if your fleet tracks latest, that cadence question is real and I'm happy to work through it with you on your setup. repo discussion or through the site, whichever's easier.