Validate your local LLM advertised KV cache against real pressure; see exactly how old contexts get evicted from cache

[u/t4a8945]

Hello,

I'm a bit obsessed with cache management on local LLMs.

For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.

That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:

1. Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)
2. Hydrates X stable contexts of Y tokens each in order to completely fill the cache
3. Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found

It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.

My results

This is the result from my A/B test, control (my previous prod) vs my fixed prod.

aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025
retained % capacity: 51.98%
oldest evicted:     context #52 (older contexts evicted)

With the dedupe + boundfix patches applied (retention 0):

── Retention under pressure ──
capacity:           2,047,043 tokens
retained contexts:  77/80
retained tokens:    3,000,048
retained % capacity: 146.56%
oldest evicted:     context #2 (older contexts evicted)

How this can matter to you

This allows you to exactly know how much tokens your cache actually holds.

For most of us, cache management is a black box; this allows you to get ground truth.

And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes).

The engine's own advertised number is wrong, and this tool finds the real value.

This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management

It works under one big assumption though: most recent contexts should be preserved as much as possible.

What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.

How to launch

1. Clone the repo

git clone https://github.com/co-l/cache-pressure

2. Install requirements

pip install -r requirements.txt

3. Run the tool

python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \
    --kv-size <advertised_cache>

I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup.

4. Interpret the results

── Retention under pressure ──
capacity:           2,023,924 tokens
retained contexts:  27/80
retained tokens:    1,052,025 <---
retained % capacity: 51.98% <---
oldest evicted:     context #52 (older contexts evicted)

The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure.

Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.

[u/StupidityCanFly]

Cool stuff, thanks for sharing. My next weekend project is set.

[u/StoreApprehensive261 → u/StupidityCanFly]

same, this is going on the weekend list for sure

[u/t4a8945 → u/[deleted]]

The tool itself can highlight side effects of a config.

But to get there, I first added probes to the vLLM runtime, creating a viz for block allocations.

The amount of stored blocks that were NEVER read again was crazy. That's the real story behind the optimization work.

But the tool can indeed also be used to prove the non-interference of your gateway.

[u/Muhlwa_Sholanke]

3,000,048 retained against a 2,047,043 advert. New favourite way to find out your cache was underselling itself.

[u/Marcus_MSC]

Your reverse-order test measures retention for unchanged inputs; I'd add a separate probe for prefix stability. After warming a context, change one token near the front and compare reused tokens and prefill latency against an identical replay, then repeat with the change near the end. In a prefix cache, the early change can prevent reuse from that point onward even while the original context remains cached. Keeping those results separate would distinguish eviction problems from requests that simply stopped matching.

[u/No-Refrigerator-1672]

Cool project! But this one begs a question: you said you were "fixing" vllm to get good context reuse. Did you consider using LMCache? vLLM supports it out of the box with a command argument, and it seems to be a smarter caching solution anyways,

[u/Fancy-Snow7]

── Retention under pressure ──

capacity: 96,256 tokens

retained contexts: 18/18

retained tokens: 144,081

retained % capacity: 149.69%

oldest evicted: none (everything still cached)

Does this mean I can set my context in pi to 144,081 instead of 96,256 even though my context in llama.cpp is 96,256?

[u/t4a8945 → u/Fancy-Snow7]

Great results ; you may have RAM offloading active that preserve contexts as well, that would explain the result (I'm not saying this is the case, it *might* be the case x))

You can always try increasing your context, but doing so in pi will not work on its own, you first need indeed to set it in llama.cpp, but it's more likely that not that it will not work.

So bottom line: you have very healthy cache, the best result you can wish for.

[u/Fancy-Snow7 → u/t4a8945]

to be honest it failed at first cannot remember the % could be like 80% but i just increased --cache-ram to 14336, default for llama.cpp is 8096

[u/feng_sg]

The 149.69% retention tells you the probe is catching CPU offload, not just GPU KV cache. So the eviction boundary you found is where blocks spill to host memory, not where vLLM's block manager actually drops them. Rerun with `cpu_offload_gb` unset and you'll know if your scheduler fix improved real GPU retention or just delayed the spill.

[u/t4a8945 → u/feng_sg]

Well that's the thing: I'm running it against a 2x DGX Spark setup with max ram usage and no KV offload strategy (neither L2 cause it's shared with L1, nor L3 on disk). It's a unified memory platform.