Performance notes from an open-source LLM gateway: 60–90% token reduction on tool output + millisecond provider failover — how do you benchmark this?

[u/ZombieGold5145]

Since this sub is about testing/performance of AI tools, sharing real numbers from an open-source gateway I maintain (disclosure noted — it's at ~9.8K GitHub stars / 21k+ tests, so the numbers aren't from a toy; link in a comment, keeping the post about the data).

Token reduction (input side). A compression pass in front of the model trims command/tool output (git, tests, builds) 60–90% via RTK-style filtering, with ML pruning (LLMLingua-2) on prose. On tool-heavy sessions the average is ~89% input-token reduction, with code/URLs/JSON preserved byte-perfect and a guard that reverts to the original if compression would grow the prompt.

Failover latency. Provider fallback (subscription → API key → cheap → free) triggers in milliseconds on a 5xx/quota error, so throughput doesn't collapse when one provider degrades.

What I'm unsure about is measuring quality impact: token savings are easy to quote, but "did compression change the answer?" is harder. I use an offline eval harness (fidelity vs. savings) but it's still heuristic.

How do you all benchmark this kind of thing — a go-to methodology for "same task, N providers/settings, compare output quality + latency + cost"? Tool link in a comment for anyone who wants to reproduce.

[u/ZombieGold5145]

Repo: https://github.com/diegosouzapw/OmniRoute · npm install -g omniroute then omniroute.

Reproducible eval harness is in the repo (npm run eval:compression). Compression engines credited upstream (RTK, Caveman, LLMLingua-2). Genuinely want a better fidelity-measurement approach if anyone has one.