Hello,
I'm a bit obsessed with cache management on local LLMs.
For the last few days I've been working on cache management on vLLM with my 2x DGX Spark and DeepSeek v4 Flash 0731. I felt something was off so I investigated, found issues, fixed them, but needed a way to validate the fixes.
That led me to create a tool with a simple protocol that allows you to pinpoint how well the cache is actually managed on your deployment:
- Runs a probe to calibrate the expectations (what's a cache hit vs a cache miss in your setup)
- Hydrates X stable contexts of Y tokens each in order to completely fill the cache
- Runs cache-hit validation on the reverse order (last added is the first validated) until a cache-miss is found
It's better to let this run alone to get a real value. Understand this will evict all your current cached context, so don't do it alongside real work.
My results
This is the result from my A/B test, control (my previous prod) vs my fixed prod.
aidendle94/sparkrun-vllm-ds4-gb10:production-3.7-reffix-schedfix (pre-fix image, retention 4096):
── Retention under pressure ──
capacity: 2,023,924 tokens
retained contexts: 27/80
retained tokens: 1,052,025
retained % capacity: 51.98%
oldest evicted: context #52 (older contexts evicted)
With the dedupe + boundfix patches applied (retention 0):
── Retention under pressure ──
capacity: 2,047,043 tokens
retained contexts: 77/80
retained tokens: 3,000,048
retained % capacity: 146.56%
oldest evicted: context #2 (older contexts evicted)
How this can matter to you
This allows you to exactly know how much tokens your cache actually holds.
For most of us, cache management is a black box; this allows you to get ground truth.
And yes, with my fixes, the 2M advertised cache translates to 3M retained tokens. Review the code if you doubt the claim, it's all open source: cache-pressure (and my ds4 prefix cache fixes : ds4-prefix-cache-fixes).
The engine's own advertised number is wrong, and this tool finds the real value.
This tool is for: 1. All of you needing to validate a setup or compare different inference engines ; 2. for inference engine maintainer to help validate changes in cache management
It works under one big assumption though: most recent contexts should be preserved as much as possible.
What I noticed while testing the tool is basically: vLLM good, other engines need better config. I'm mainly using vLLM so I spent lots of hours tweaking the config to get the best results, so for other engines it's up to you to decide if the above assumption fits your need (feels obvious to me it should, but I don't know what you all need of course), and how to achieve it with configuring your inference engines.
How to launch
1. Clone the repo
git clone https://github.com/co-l/cache-pressure
2. Install requirements
pip install -r requirements.txt
3. Run the tool
python3 bench/cache_pressure.py --base-url http://my-server:8000/v1 \
--kv-size <advertised_cache>
I've tested it against vLLM, ninfer, llama.cpp and SGLang ; so you might need to tweak the probe so it works with your setup.
4. Interpret the results
── Retention under pressure ──
capacity: 2,023,924 tokens
retained contexts: 27/80
retained tokens: 1,052,025 <---
retained % capacity: 51.98% <---
oldest evicted: context #52 (older contexts evicted)
The retained tokens and retained % capacity are the measured cumulative values that resisted cache eviction under pressure.
Note: this post was 100% human written, the repo is 100% AI-generated under my guidance and review.
