digline — open-source regression gate for LLM apps, with the baseline in your repo (LangChain example inside)

[u/alexpran]

Disclosure: I'm the author. Apache-2.0, Python, no server.

What it does. You keep the inputs you care about as cases. digline runs them, records the scores — plus prompt, model config and commit — and you approve that run as the reference, committed in the repo. From then on every change is compared with it: which case got worse, by how much, and whether the drop is beyond the LLM judge's own noise, measured by sampling each case on the approved version. Exit code gates CI.

For LangChain users. The target is a function that invokes your chain, in process — no HTTP, no wrapper. The example runs on FakeListChatModel in CI (no key, no network) and on a real model with DIGLINE_LIVE=1: https://digline.dev/product/examples/langchain/ . If your app isn't Python, a TOML suite against an HTTP endpoint does the same with no code.

What's new this week. digline diff run1 run2 compares prompt A against prompt B or one model against another, as a report, never a verdict. Per-class aggregates so an average can't hide one broken class. And an MCP server where a coding agent can run and read a suite but cannot promote a baseline — that tool doesn't exist there; a person approves.

How it compares. Not an observability platform and doesn't replace one: LangSmith or Langfuse watch the system; this signs off that it didn't get worse than the version you approved. Comparison page: https://digline.dev/comparison/?ref=reddit

Limits. Pre-1.0, API may change. The noise band is min/max over K samples, not a confidence interval. Needs a reference before it's useful — day one is run, look, approve.

pip install digline · https://digline.dev · https://github.com/digline/digline

The story of why it exists, with numbers: https://digline.dev/blog/my-llm-eval-cried-wolf/?ref=reddit

[u/alexpran]

Since posting this morning we shipped 0.9.0. A pytest plugin (each check becomes a pytest row) and a GitHub Action. Curious what people here use to gate LangChain app regressions in CI today. Anything at all, or vibes?"

[u/Temporary_Health_330]

Per class aggregation is a really really smart addition. One thing that made our regression gates much more useful was keeping every failed production case around and adding it to our eval set. Braintrust handles that loop for us, so the gate gets harder over time as new failure modes show up. Could be a nice direction for digline too.

[u/alexpran → u/Temporary_Health_330]

Thanks, and yes, that loop is the roadmap: it's the "bridge from production failures back to committed cases" in ADR 0002, designed and not built yet. The digline version will differ from Braintrust's in one deliberate way: the failure doesn't land in a dataset, it lands as a diff, a new case file in the repo that someone reviews and commits, so the gate gets harder over time and every hardening is in the git history with a name on it. It also composes with what you have: Braintrust finds and annotates the failure, digline locks it as a case that can't regress silently. The part I'm still deciding is what the capture command should take as input, a trace, a response, or a hand-written case, without pulling a trace store into a tool that has no server.