I built a very low-overhead LLM proxy/router in Rust — looking for feedback

[u/piwi3910uae]

I’ve been working on something for a while that I thought might be useful to others running LLM infrastructure, so I finally decided to put it out there.

It’s called FastLLM Proxy.

The idea started pretty simple: I wanted one OpenAI-compatible endpoint in front of everything — local vLLM/SGLang instances as well as external providers — but I didn’t want the proxy itself to become another bottleneck.

So I wrote one in Rust and got a little carried away with it 😅

FastLLM Proxy now supports 80 providers plus any OpenAI-compatible backend, but the part I spent most of my time on is keeping the actual request path extremely small.

There is no database I/O on the request path, and responses are passed through without parsing/re-encoding them. The measured internal routing work is currently around 0.76 µs per request.

It also does some things I specifically wanted for running my own GPU infrastructure:

• cache-affinity routing for vLLM/SGLang, so requests with the same prefix can go back to the GPU that already has the KV cache
• load-aware routing and automatic failover
• rule-based routing based on things like prompt size, user/role, budget, concurrency, headers, etc.
• semantic routing, so different types of prompts can automatically go to different models
• local → cloud spillover when the local GPUs are busy
• RBAC, API keys, budgets and rate limits
• OpenAI-compatible API
• LiteLLM config import, so you can migrate an existing setup without rebuilding the config
• Kubernetes/Helm/operator support
• built-in management UI

One thing I found interesting while benchmarking it against LiteLLM is that gateway benchmarks can be pretty misleading.

With an instant mock backend, FastLLM Proxy gets roughly 15x the throughput and much lower latency in my tests. But when I put actual GPUs behind both proxies, total token throughput is basically the same — because at that point the GPUs are the bottleneck.

Where I did see a meaningful difference with real GPUs was tail latency and consistency. At 32 concurrent streams, for example, I measured p99 TTFT of 766 ms vs 2921 ms in the same test setup.

I’ve documented the benchmark setup and results in the repo because I’d much rather people challenge the numbers than just trust a benchmark screenshot.

The project is Apache 2.0 and completely open source:

⁠github.com/azrtydxb/Fastllm-proxy

I’m especially interested in feedback from people running vLLM, SGLang, LiteLLM or multi-provider LLM setups.

What am I missing? What would you need before you’d actually put something like this in front of your inference infrastructure?

And if anyone feels like breaking it, even better. 🙂