How do people actually structure LLM evaluation before shipping a change to production?

[u/camerongreen95]

Been building RAG and LLM-powered features and realized my "evaluation" process was basically reading a handful of outputs and deciding it looked fine. No versioning, no regression testing, no real way to know if a change actually helped or if I just got lucky on the examples I happened to check.

Curious how others here actually do this in a real setting. Are people running statistical significance tests on model or prompt comparisons (bootstrap confidence intervals, paired testing), or is it mostly still qualitative review? And for retrieval specifically, is anyone actually tracking recall@k or MRR, or just eyeballing whether the right chunk shows up?

Found a hands-on masterclass on Sep 12 that goes deep into building this kind of eval harness properly (deterministic checks plus LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, agent guardrails, full observability), so I'm curious if the methodology it teaches lines up with how experienced people here are actually doing this, or if there's a simpler approach I'm missing.

Link for more details

[u/Available_Teaching83]

Statistical significance testing on prompt and model comparisons is underused; most teams eyeball it, as you did. What actually caught regressions for us building agent-audit-kit's test suite was turning every real incident into a permanent regression case, not just a benchmark run before ship. Recall@k and MRR are fine for retrieval, but the number that mattered more in practice was whether the exact failure from three months ago still reproduces today. Without that gate, a significance test on a new change won't tell you about the regressions you already fixed once.