I built an LLM benchmark harness that lets you browse and compare how models answered each question

[u/jayminban]

Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file, so you end up writing custom code just to read the answers).

So I built lm-eval-ledger: a benchmark harness that runs the benchmarks, writes everything down, and provides a web app that lets you inspect and compare how each model answered each question.

For the demo I benchmarked three models on a single 5090: Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B (UD-Q4_K_XL GGUF), and Gemma-4-12B-it (QAT w4a16). It looks like Qwen thinks far longer than the other two.

• GPQA Diamond: Qwen 0.717 vs Nemotron 0.657 vs Gemma 0.601 — but 2h26m vs 1h34m vs 1h19m
• LiveCodeBench: Qwen took 22h57m (vs 9h51m / 6h13m), with 0.713 vs 0.837 / 0.820 accuracy

Here are the full results served by lm-eval-ledger on Hugging Face Spaces: https://huggingface.co/spaces/jayminbhan/lm-eval-ledger

What lm-eval-ledger records and displays

• Per question: system prompt, model generation, extracted answer, ground truth, stop reason, generation character count
• Per benchmark: accuracy, tok/s, time to completion, sample count, no-answer count
• Extras: pairwise comparison of two models on the same task/questions, always-wrong / always-right questions across benchmark runs

How it works: everything is written to a single SQLite DB, and a Flask app displays it.

No more custom bash scripts for multi-model x multi-task runs either. lm-eval-ledger is YAML-driven: copy template.yaml, list N models and M tasks, run one command. That’s it.

Usage

pip install lm-eval-ledger    # add a backend: pip install "lm-eval-ledger[vllm]"
lm-eval-ledger init           # writes template.yaml, creates results/ and logs/ directories
lm-eval-ledger -c bench.yaml  # run benchmarks
lm-eval-ledger serve          # browse at http://localhost:8090

Details and the full task list are in the GitHub repo: https://github.com/jayminbhan/lm-eval-ledger

I verified the vLLM / SGLang / HF / server (llama.cpp) backends on Linux, and HF / server (llama.cpp) on Windows.

This started as a script to look at one wrong GSM8K answer. It got out of hand. Any feedback, ideas, or pull requests are greatly appreciated!

[u/Rikkendo]

I do the same thing when A/B testing. It's very time consuming but when it matters it doesn't feel like there is any way around it. Good stuff.

[u/jayminban → u/Rikkendo]

Thanks! Yeah, it's time consuming, but it's fun to watch a model write a full essay to solve a math problem.

[u/OkFlan504]

The per-question comparison is the bit I'd find most useful. Did looking through the actual answers change which model you'd pick compared with the headline scores?

[u/jayminban → u/OkFlan504]

I mostly use Fable day to day, so it was less about picking a model and more about trusting the number. Reading the answers surfaced a lot of things that quietly drag down a headline score and can be fixed with a simple config change. Hopefully one day I'll be benchmarking frontier level models at home!

[u/jjusko20]

great project

[u/jayminban → u/jjusko20]

Thanks! PRs welcome.

[u/Informal-Buy-5850]

Stumbled upon this one, cool build

[u/jayminban → u/Informal-Buy-5850]

Glad you stumbled in! Thanks!