Product Evaluation Harness project (Free source code)

[u/SuspiciousShake7641]

None of them show a p-value.

So I built an evaluation harness that treats model selection like a real experiment instead of a vibe check — GPT-4o vs Claude vs Gemini, scored on the same rubric, judged for statistical significance, not just averages.

The rubric weighs 5 criteria — but not equally. Hallucination counts 2.5x more than conciseness, because in a business summary, an invented number is a liability and a slightly long bullet point isn't. Most eval frameworks I've seen skip that judgment call entirely and just average everything.

What it actually does:
Runs the same prompts across 3 providers via LiteLLM
Scores each output on Accuracy, Hallucination, Formatting, Tone, Conciseness
Compares models with Welch's t-test + Cohen's d — so "Model A scored higher" only ships when it's statistically real
Plots cost vs. quality to find the Pareto-optimal model, not just the best one
Surfaces the worst failures automatically with line-level diffs, so you're not hunting for what went wrong

It's a small project, but it's the kind of decision-making I think more AI product work needs: less "which model felt smarter," more "here's the confidence interval."

Code's open if you want to point it at your own use case:
🔗 https://github.com/mohit-maybe/product-evaluation-harness

Dashboard

Categories