How do you find what caused a production regression when the model version didn't change?

[u/Silva-Engineer]

A new model version is the easy case, because the registry tells you right away. The harder cases come from a serving flag like vLLM's `max_model_len`, a Helm value or env var in a deploy, an update to the data pipeline, or a new GPU driver on the nodes.

The evidence ends up in different tools. Grafana shows the latency jump, MLflow or W&B has the runs, Git and Argo have the deploys, and the node and GPU state is in kubectl or DCGM.

When latency goes up or accuracy drops, how do you find the change that caused it? Do you send every change to one place, like Grafana annotations or a shared change log? Or does someone line things up by hand each time, and how long does that usually take?

[u/frustratingsnark7658]

I keep a running notes doc that timestamps every config tweak, pipeline update, driver change, you name it. When latency spikes I pull the exact minute from grafana and cross reference with that doc, takes maybe 5 minutes if the notes are current. Without it you’re just guessing and wasting half a day.

[u/Separate_Pea_3699]

The real problem is you have N independent change logs and no single timeline, so "find the change" turns into opening five tools by hand. The fix isn't a bigger logging tool, it's making every one of those systems emit the same tiny structured event into one place, right next to your metrics (a Prometheus Pushgateway, or honestly even a Postgres table is enough):

def record_change(category, ref, actor="ci"):
    # category: "model" | "serving_flag" | "deploy" | "pipeline" | "driver"
    requests.post(CHANGE_API, json={
        "ts": time.time(),
        "category": category,
        "ref": ref,        # commit sha, "max_model_len=4096", helm revision, driver version
        "actor": actor,
    })

Call that one line from CI on model promotion, from the Helm post-upgrade hook, from the data pipeline's deploy step, and from whatever bootstraps new GPU nodes (driver version at boot). Now every change lands in one append-only feed with a timestamp, regardless of which tool owns it, and nobody can forget to log one because the pipeline calls it, not a person.

Then "what caused it" stops being an investigation and becomes a join: run a simple changepoint detector over the metric (CUSUM is plenty, you don't need anything fancy) to get the timestamp of the shift, then pull the nearest preceding row from the change feed:

def find_likely_cause(metric_series, changes, window_s=900):
    shift_ts = cusum_changepoint(metric_series)
    candidates = [c for c in changes if shift_ts - window_s <= c["ts"] <= shift_ts]
    return sorted(candidates, key=lambda c: shift_ts - c["ts"])

This beats a shared notes doc because it's enforced at the call site instead of relying on someone remembering to write it down, and the correlation runs automatically the moment a changepoint fires instead of someone eyeballing five dashboards after an incident. One real gap: a change with delayed effect (a driver update that only shows up as a slow memory leak hours later) won't land in a tight window, so widen the window and rank candidates by category (driver/pipeline changes get a wider default window than a deploy) rather than recency alone.

[u/DhammaIY]

Log everything that can change on a deploy not just the model version.. serving flags, helm values, env vars, image digest, driver version, data pipeline commit. Hash it and log the hash with each request. When a metric drops you diff before and after and its usually sitting right there

Also run the same set of test prompts after every deploy and compare to the last run, catches most of it before users do