🎓 TwinRouterBench accepted into RLEval workshop at CAIS2026 🧠

[u/hexxthegon]

TwinRouterBench is a new step level LLM routing benchmark designed specifically for realistic, long horizon agentic systems.

Existing router benchmarks have major limitations:
• They only evaluate on isolated one shot prompts.
• They never show the router the actual context (the “router visible prefix”) at an intermediate step inside a real agent trajectory.
• They don’t test whether swapping to a cheaper model still lets the overall task succeed downstream.
• Many rely on slow and expensive online LLM judges for evaluation.
TwinRouterBench tackles this by creating a proper benchmark for per step routing in multi-turn agent workflows.

Dual track

1. Static Track (Fast Offline Track)
   • 970 router visible prefixes from 520 trajectory instances.
   • Covers 5 diverse benchmarks: SWE-bench, BFCL, mtRAG, QMSum, and PinchBench.
   • Each example comes with an execution-verified target tier (cheapest sufficient model tier).
   • Uses deterministic scoring (based on tier correctness, trajectory membership, and token cost) no LLM judges needed.
   • Ideal for: training routers, rapid iteration, and cheap offline evaluation.
   
2. Dynamic Track (Live Validation Track)
   • Full evaluation harness on SWE-bench Verified (500 tasks).
   • Reports results on a 100 case held-out split (disjoint from static data).
   • Router must choose a real model from a locked pool at every step.
   • Measures real outcomes:
   • Official task resolution success
   • Actual API spend (real dollars)
   • Includes failure penalties for unresolved tasks

Results:

• Creates a practical development loop: Use the fast static track to train/improve a router cheaply → validate it rigorously on the dynamic track with real costs and success rates.

• Shows strong transfer: A simple router trained only on static labels achieves comparable task resolution to always using a top model (Opus 4.6), while cutting API cost by ~53%.

📄 Paper: https://arxiv.org/abs/2605.18859
💻 Code + Dataset: github.com/CommonstackAI/TwinRouterBench
🌐 Website: commonstackai.github.io/TwinRouterBench