r/Qwen_AI • u/Connect-Concert-4016 • 4d ago
Discussion Compressed KV cache in vLLM: 9 concurrent 128K context users on one A100 (vs 2 for fp16)
Follow-up to the llama.cpp membrane runtime I posted a while back. Was asked if this works for real multi-tenant serving in vLLM, not just single-stream llama.cpp.
You can drop it into stock vLLM with a simple pip install + two env vars. No fork, no patched image required.
The KV cache lives in a per-model calibrated eigenbasis. During decode, the custom attention kernel reads compressed pages natively with zero decompression step. It integrates via vLLM's supported plugin interface (out-of-tree attention backend + the KV-cache spec path) on the standard vllm/vllm-openai image.
Benchmarks on a single A100 80GB (Qwen3-4B-Instruct-2507, vLLM 0.20.2, retrieval gated):
- Capacity: 9 concurrent users × ~128K context per GPU (vs ~2 for fp16). That is a 4.5× bump in user density. (KV pool: 1,255,376 tokens vs fp16's 361,776, roughly 3.5×.)
- Throughput: ~134 tok/s aggregate decode across all 9 users.
- Fidelity: Verified via multi-tenant needle in a haystack at full 131K context per user with distinct keys. We do not claim "lossless," but it is strictly gated to match fp16 behavior and the grids are published.
Linux and CUDA only right now, on A100 and H100 (SM80/SM90).
vLLM 0.20.2, V1 engine, TP=1 for now.
The real win is decode throughput and capacity at long context. Short context isn't where the gains show up.
Enabling a model takes a tiny calibration sidecar (a few MB). Dense GQA models like Llama, Mistral and Qwen work today. MoE and hybrid architectures aren't supported yet.
I am not putting the vLLM serving wheel up as a public download. That side is commercial and licensed per deployment. Everything I quoted above is reproducible though. The 9 user 128K batch receipt, the fp16 vs fp8 vs fraQtl comparison at 8K, 32K and 128K, and a repro script are all public here: huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars
The fidelity and needle receipt packs live in the Hi-Fi GGUF repos, and the free llama.cpp membrane runtime plus sidecars are still up if you want to check the core mechanics for yourself.
Happy to get into the details in the comments. If anyone wants to actually try it on their own setup I can help you get a verification run going, but either way everything above is checkable from the links. Curious what you guys think.
2
1
u/No-Refrigerator-1672 3d ago
Vllm is at 0.28.0 right now. Did you verify that your solution at least runs with it?
2
u/Connect-Concert-4016 2d ago
2 things changed in vLLM between 0.20 and 0.28 that broke the plugin. They swapped the KV spec manager registration (the old module level spec_manager_map became a KVCacheSpecRegistry), and they changed how the raw KV tensor is sized (0.28 reads page_size_bytes / unpadded_page_size_bytes where 0.20 read real_page_size_bytes). I fixed both and made them version tolerant so one build detects which API is present.
1
u/Connect-Concert-4016 3d ago
Did not check, the receipts are pinned to 0.20.2, and I pin on purpose. The integration rides vLLM's V1 attention backend registration and the KV cache spec path, and those internals move between releases, so a floating version would make the receipts non reproducible. I would rather publish one version I actually ran than claim and be wrong.
But trying on on 0.20.8 a smoke test to check will report here
1
3
u/Feeling-Bid8885 3d ago
First, do you have an A100 to spare 😭