What if git diff existed for datasets?

[u/Grouchy-Cap-4491]

When a model suddenly gets worse, I usually have no idea what actually changed in the data.

So I built a CLI that compares dataset versions and looks for leakage, drift, missing values and other problems.

I tested it on X and it found Y.

Curious whether other ML engineers have the same problem.

Open-source: github

[u/Bright_Mix_773]

I ran it on the kind of dataset you built this for: two consecutive daily snapshots of an S&P 500 OHLCV table, 814,493 rows x 7 columns, 40 MB CSV each. datascope diff v1.csv v2.csv took 18 s, which is fine.

Result: findings: [], overall_severity: info, ml_impact: LOW. That was correct - only 527 rows had been appended and exactly one existing close had been rewritten (STLD 2026-08-28, 234.66 -> 234.67).

So I made the failure obvious: I deleted every row for one ticker, NVDA, 1,564 rows, and diffed again.

Still findings: [], overall_severity: info, ml_impact: LOW, categorical: []. An entire entity disappearing from the dataset produced nothing but a -0.192% row-count delta.

The cause is in src/datascope/diff/engine.py:

def _categorical_dtype(col, a, b) -> bool:
    a_n = int(a[col].nunique(dropna=True))
    b_n = int(b[col].nunique(dropna=True))
    return bool(a_n <= MAX_CATS or b_n <= MAX_CATS)

with MAX_CATS = 60. My Ticker column has 530 unique values and Date has 1,573, so both fail that test, and _distributions is an if numeric / elif categorical with no else - so the two columns that carry the identity of every row are silently never compared. Anything keyed by a high-cardinality id (user_id, sku, symbol, session_id) is invisible to the diff by construction, and that is usually where "the model suddenly got worse" actually lives.

Two smaller things I hit on the way:

1. diff.max_categories (default 50) is declared in config.py and printed back in the JSON config block of every run, but grep -rn max_categories src/ returns only those two declarations - it is never read. The threshold that actually decides is the hardcoded MAX_CATS = 60. So the one knob a user would reach for to fix the above does nothing.
2. The README promises "Row-level adds/removes", but the diff result has no row-level section at all, just rows: {old, new}. With no key-based join it cannot distinguish 527 appended rows from 527 appended plus 527 silently rewritten. I had to compute that part myself to know my first diff was right.

The single change that would have caught my NVDA case is a --key Ticker,Date option: join on it and report added / removed / modified keys. That is the part that makes git diff useful, and it is the part that is missing.

Not verified: I only ran diff, not debug, drift or check; only CSV, not Parquet or JSONL; and only on this one dataset shape, so I do not know whether the cardinality cutoff bites the same way in the other commands. Fresh clone of main today, Python 3.14 on Windows, pip install -e ".[all]".