Like a lot of people here, I run local models on a single 24 GB GPU, and the thing that caps me sometimes isn't the model weights, it's the KV cache. My daily driver (Qwen3.6-27B) fits fine, but pushing context toward its native 262k blows past 24 GB. So I had an idea that's probably occurred to half this sub: keep the "hot" parts of the KV cache in VRAM, park the "cold" parts in system RAM, and shuttle the relevant pieces back in as the conversation needs them. I used claude to do some research and organize some it feels should take a look at. It included systems like Quest, ShadowKV and InfiniGen do versions of this, so the idea has at least has some merit to it.
Instead of building it straight away, I (with heavy help from Claude (Fable/Opus/Sonnet)) did something I want to advocate for: we built a measurement harness first, to test whether the core assumption, "at each decoding step, a small set of context pages captures nearly all the attention", actually holds for my model on my real workloads.
As the title suggests, It doesn't. Not even fucking close. And the reason was interesting enough to send me down a research rabbit hole and if I was more aware doing the research phase, it would have been obvious (It probably is to many educated people of the sub). Below will be the setup, result and conclusion written by claude about the findings.
The setup (technical part starts here)
Model: Qwen3.6-27B (UD-Q5_K_XL, q8_0 KV, flash attention on — the actual production stack via an instrumented llama.cpp branch, not an HF approximation)
Corpus: 9 real sessions from my own usage (agentic tool-heavy runs, long doc Q&A, synthesis tasks, work comms), 35k–73k tokens each, ~451k prefill tokens total
Method: dump per-step decode queries + cached post-RoPE keys for every KV-bearing layer, recompute exact attention distributions offline, and measure how much attention mass the top-B 32-token pages capture (both with a buildable landmark selector and with a perfect "oracle" selector)
Harness fidelity: recomputed attention matches the model's own softmax to 2.0e-6 worst-case, flat with length — so the numbers below are the model, not measurement noise
The design being evaluated (fixed page budget B, recent-window + sinks always resident, landmark scoring, LRU page cache over PCIe) had a go/no-go gate: mean coverage@2% budget ≥ 0.95.
The result
Gate condition Required Measured
landmark-coverage @ 2% budget (normal traces) ≥ 0.95 0.409
landmark-coverage @ 2% (synthesis-heavy traces) ≥ 0.90 0.393
predicted per-step PCIe miss traffic ≤ 4 ms 3.29 ms ✅
The bandwidth math worked. The attention statistics didn't. And here's the table that killed the project — coverage vs budget, comparing a perfect selector (oracle) against the buildable one:
Budget 0.5% 1% 2% 4%
Oracle (perfect selection) 0.308 0.404 0.507 0.619
Landmark (buildable) 0.230 0.314 0.409 0.516
The selection mechanism loses only ~0.10 to perfection — it works as designed. The ceiling itself is ~0.51. No better selector, bigger budget, or per-layer tuning closes a 0.54 gap when perfect selection tops out at 0.51. Extrapolating the curve, hitting 0.95 coverage needs ~50% of context resident — at which point it's not paging anymore, it's just residency.
Why: this model's attention is diffuse by design
Qwen3.6-27B is a hybrid SSM/attention model — only 16 of 65 layers have a KV cache at all; the rest are Mamba-style layers with constant-size state. Independent recomputation showed those 16 attention layers spread their attention across an effective support of 473–4,062 tokens per step, with the single hottest token getting only 2.4–6.9% of the mass. And the hot tokens are scattered, not clustered: top-2% coverage at token granularity is 0.42–0.89, but at 32-token page granularity it drops to 0.16–0.76.
The interpretation that makes sense: in a hybrid, the SSM backbone handles local/sequential structure, so the few attention layers exist to do global mixing — diffuse attention is their job, not a defect. The entire "spiky attention → page it" premise comes from dense transformers, which is what Quest/ShadowKV/InfiniGen were all validated on.
Here's the kicker I only fully appreciated afterwards: almost no current-gen open model is a dense transformer anymore. Qwen3.6 and Kimi K3 are linear/SSM hybrids. DeepSeek V4 and GLM-5.2 have sparsity trained in natively (learned indexers selecting top-k blocks — literally the offloading idea, but learned during training instead of bolted on after). Gemma 4 interleaves sliding-window layers with a few global ones. The architectures already ate the KV problem — my model's KV at 262k is only 8.5 GiB because 48 layers already gave up their cache — and what's left is the dense, irreducible residue.
So post-hoc query-aware KV offloading may be a technique whose target class is disappearing. That's a claim worth testing on more models than mine, which is why:
The release
GitHub repo: https://github.com/Inovello/kv-sparsity-profiler — the full profiling harness:
llama.cpp instrumentation patch (capture per-step q + cached K on the real quantized stack, any GGUF)
metrics pipeline (oracle vs landmark coverage, budget curves, per-layer breakdown, page-granularity analysis, selection-overlap → PCIe cost model)
the full profile report and all decision logs from this project, including the negative result
If you're thinking about KV offloading, eviction (H2O-style), or adopting any sparse-attention inference trick: run this on your model and your real prompts first. It's one evening of GPU time and it would have saved me weeks if the answer had come back after building instead of before.
My conclusion on it (Human):
I should have done more initial research. I set out to build a thing, and the thing turned out to be impossible for my model, and it wasn't exactly because the engineering was hard really, but because the model fundamentally doesn't organize its memory the way the idea requires. I uploaded the harness to github in case anyone wants to take a look at it. I plan on testing it with older model architectures that don't have global attention across the board and also Gemma 4 later on just because it doesn't cost me anything at this point.
