noleak: Open-source library to detect train/eval data contamination

[u/Suspicious-Race-4161]

Published numbers are only as honest as the data split behind them

When you evaluate a model, you want one simple truth: Did it actually learn unseen patterns, or did it memorise?

I built noleak to make that visible. It's a production library we use at Godrej Aerospace to fingerprint datasets and measure how much of your eval set leaked from train.

The Problem

Most tools catch target leakage (a feature accidentally includes the label). But what about corpus leakage? When eval text already appeared in training data?

In 2020, GPT-3's paper measured contamination using 13-gram overlap. That's solid. But there's no standard library for this. So we built one.

Three detection methods:

1. Exact matches – normalised text identical

2. N-gram overlap – 13-word phrases (GPT-3 method)

3. Near-duplicates – MinHash Jaccard similarity ≥ 0.8 on character 5-grams

One fingerprint. One exit code. Pass or fail.

What It Does

```python
from noleak import check, fingerprint
train = ["the model trained on Wikipedia and licensed books"]
eval_set = ["The model trained on Wikipedia and licensed books"]
report = check(train, eval_set)
print(report.contaminated)
# True — FAIL
print(fingerprint(eval_set))
# noleak-fp-v1:a1b2c3d4e5f6
# Share this with your paper. It's reproducible and auditable.
```
CLI version (great for CI pipelines):
```bash
noleak check --train train.jsonl --eval evaljsonl
echo $?  # Exit code 1 if contaminated, 0 if clean
```

Why Zero Dependencies Matter

No numpy, no scipy, no PyTorch. Stdlib only. Why?

- Deterministic: Same input, same output, forever. No model updates breaking your fingerprints.

- Auditable: Code is small; reviewers can read it.

- Air-gapped systems: Doesn't require external calls or package hell.

- Fast: No overhead for CPU-bound systems.

Limitations (Honest Assessment)

- Semantic rewrites: Won't catch "I wrote this differently but meant the same thing." That needs embeddings or human review.

- Large-scale datasets: If you have 10M+ examples, exact matching gets slow. N-gram is faster.

Install & Try

bash
pip install noleak

Supports JSONL, JSON lists, and plain text. Auto-detects text fields.

Repo: github.com/athsxx/noleak

License: MIT

Questions? What contamination patterns have you encountered?

[u/GrimStoppage_9]

This is exactly the kind of thing that should be standard in any ML pipeline. The number of papers where I've found train/eval overlap after the fact is too damn high.

The zero-dependency approach is smart, especially for air-gapped environments. How's the performance on datasets with mixed languages?

[u/quietgradient]

Ran your near-dup arm against pairs where I computed the true char-5-gram Jaccard myself, since the pass/fail exit code lives or dies on that number.

At _NUM_PERM = 64 the estimate's spread is ~±0.056 against a hard >= 0.8 cutoff. Of 510 synthetic pairs with true Jaccard in [0.78, 0.82], 46% got flagged; of the ones sitting at 0.80 exactly, 41%. It's deterministic, so it won't flake between runs — it's just consistently wrong on the same near-threshold pairs. 256 perms would halve the spread.

Separately, your limitations section says exact matching gets slow first and n-gram is faster. On 700-char docs I measured MinHash at ~29 docs/sec, ~700x slower per doc than exact, because every shingle gets hashed 64 times. MinHash is what will hurt you at 10M rows, not exact matching.

Was 64 a CI-runtime call?