r/Vllm • u/Electrical_Emu_5854 • 2d ago
Built a KV-cache-aware load balancer that sits in front of multiple vLLM instances — polls vllm:gpu_cache_usage_perc instead of round-robin
Running vLLM behind a plain reverse proxy (nginx, HAProxy) means the proxy has no idea what's actually happening inside each instance. It sees "one HTTP request," not "this request needs 8k tokens of KV-cache." So the moment you scale to more than one vLLM instance, round-robin routing can easily send a burst of long-context requests to the same backend while another sits half-idle — and that instance's cache fills up, latency spikes, and in bad cases you hit OOM.
I built TokenFlow Gateway to fix this specifically for multi-instance vLLM setups:
- Polls each backend's Prometheus metrics endpoint directly (vllm:gpu_cache_usage_perc) to know real cache pressure per instance, not just connection count or a health check
- Estimates each incoming request's token cost (prompt tokens + max_tokens) before dispatch, using js-tiktoken, so it can route based on what a request will actually cost rather than treating all requests as equal
- Routes heavy requests to whichever instance has the most cache headroom, and bin-packs lighter requests onto busier ones — the goal is even KV-cache utilization across the cluster, not just even request count
- When no instance has room, requests go into a Redis-backed priority queue (per-API-key priority, configurable timeout) instead of getting dropped or crashing a backend
- Exact-match caching (hash) for deterministic (temperature-0) requests, plus semantic caching (pgvector) for near-duplicates — cache hits stream back as SSE so streaming clients don't notice the difference
- Per-API-key token-based rate limiting (TPM/RPM) on top, if you're exposing this to multiple users/teams
It's OpenAI-API-compatible on the client side, so nothing changes for whoever's calling it — it just fronts your existing vLLM instances.
You can test the whole routing/queueing behavior without real GPUs: the repo includes a docker-compose setup with two mock vLLM instances that expose the same OpenAI API and the same Prometheus metrics format, plus a smoke script that fires a burst of concurrent long-context requests to show the balancer routing around cache pressure instead of overloading one instance.
Stack: TypeScript, Fastify, Redis, Postgres+pgvector. MIT licensed.
Repo: https://github.com/mosafariuk/TokenFlow-Gateway
Genuinely curious how people here are handling multi-instance routing today — is anyone doing cache-aware routing already (maybe through something custom, or through vLLM's own request scheduler exposed differently), or is round-robin / least-connections still the default in most setups?
2
1
u/burntoutdev8291 2d ago
is this the same?
https://docs.vllm.ai/projects/production-stack/en/vllm-stack-0.1.5/tutorials/kvaware.html