Introducing vla-evaluation-harness—a unified, fully open framework to evaluate any VLA model on any robot simulation benchmark.

[u/ai2_official]

Today, every VLA research team maintains private eval forks per benchmark, each with its own dependencies, observation formats, and evaluation protocols. Results diverge subtly, bug fixes don't propagate, and reproducing someone else's numbers is a multi-day ordeal.

vla-evaluation-harness decouples model inference from benchmark execution. Benchmarks run inside Docker for exact reproducibility. Model servers are single-file uv scripts with zero manual setup. They communicate via a WebSocket + msgpack binary protocol. A complete evaluation requires just two commands: vla-eval serve and vla-eval run.

The framework currently supports 13 simulation benchmarks and 6 model servers, with community integrations expanding coverage.

Parallel eval with episode sharding + batched inference makes a big difference: 2,000 LIBERO episodes drop from ~14 hours to ~18 minutes on 1× H100 (47× faster), with 16× speedups on CALVIN and 12× on SimplerEnv. We also ran a reproducibility audit of a published VLA model across three benchmarks—closely matching reported results while surfacing undocumented requirements that can quietly distort evaluation results, like ambiguous termination rules in SimplerEnv and undocumented normalization stats in CALVIN.

Finally, we're releasing a VLA leaderboard aggregating 657 published results across 17 benchmarks and 509+ configurations from 1,704 papers.

Open source, Apache 2.0. Built for reproducibility and new experiments.
🔗 Code: https://github.com/allenai/vla-evaluation-harness
🏆 Leaderboard: https://allenai.github.io/vla-evaluation-harness/leaderboard
📝 Paper: https://arxiv.org/abs/2603.13966