leakhunt

Find data leakage and suspicious evaluation behaviour in machine-learning pipelines.

Everyone tells you to avoid leakage. Almost nobody measures it. Measuring turns out to be the more useful thing, because it tells you whether a published number is plausible, and it upgrades "your split is wrong" into "your split is wrong and it's worth 18 points".

leakhunt does both jobs. There are cheap model-free checks you can drop into CI, and a diagnostic that fits models to work out what a leaky split is actually buying you.

The idea

Rows often aren't independent. Several EEG epochs come from one subject. Several images come from one patient. Several flows come from one capture session. Each entity leaves its own signature on top of whatever signal you care about.

Split those rows at random and the model can identify the entity. The entity determines the label. So it scores well without having learned anything that transfers.

leakhunt holds everything fixed except the splitter. Any difference in score is down to leakage and nothing else.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

from leakhunt import make_grouped_classification, measure_inflation

data = make_grouped_classification(n_groups=40, rows_per_group=15,
                                   group_signature=2.0, seed=0)
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))

result = measure_inflation(model, data.X, data.y, data.groups)

print(result.to_finding().summary)
# Leaky splitting is worth +18.5 points of balanced_accuracy
# (0.556 correct -> 0.741 leaky).

Does it actually track leakage?

A diagnostic that fires on everything is worthless. What matters is that it responds to entity structure and stays quiet when there isn't any.

examples/dose_response.py sweeps the per-entity signature in synthetic data from zero up to strong. These are measured numbers, not estimates. Reproduce them with python examples/dose_response.py:

Entity signature	Correct protocol	Leaky protocol	Inflation	SD ratio
0.0	0.629 ± 0.040	0.634 ± 0.044	+0.4 pp	0.90×
0.5	0.642 ± 0.067	0.686 ± 0.045	+4.4 pp	1.49×
1.0	0.613 ± 0.097	0.721 ± 0.038	+10.9 pp	2.54×
2.0	0.556 ± 0.100	0.741 ± 0.039	+18.5 pp	2.60×
3.0	0.545 ± 0.099	0.745 ± 0.040	+20.0 pp	2.47×

Balanced accuracy, 5 folds × 5 seeds, logistic regression, 40 groups × 15 rows. Python 3.12.10, scikit-learn 1.9.0. Takes about a second to run.

With no entity signature it reports +0.4 points, so it doesn't fire on clean data. As the signature gets stronger, inflation climbs steadily to +20 points.

The stability signature

Now look at the last column. Under the leaky protocol the fold-to-fold standard deviation shrinks, from 0.099 down to 0.040 at the strongest setting.

So a leaky protocol doesn't only look more accurate. It looks more reproducible, because every fold is recycling the same entities. That inverts the usual intuition. A suspiciously tight confidence interval is a reason to worry, not a reason to relax. InflationResult.stability_ratio reports it, and anything past roughly 2× deserves a second look.

Does it find anything in real data?

Synthetic data shows the diagnostic behaves correctly when you know the ground truth. It says nothing about real cohorts. So here's UCI Human Activity Recognition (Anguita et al., 2013): 30 volunteers, 561 accelerometer and gyroscope features, subject ids included with the data. Public, CC BY 4.0.

Three binary activity pairs, all of them physically similar and therefore genuinely hard. Each subject contributes a lot of windows, so splitting windows at random puts the same person on both sides. Reproduce with python examples/har_validation.py, which downloads about 60 MB once and then caches it:

Task	n	Correct (grouped)	Leaky (random)	Inflation	SD ratio
SITTING vs STANDING	3,683	0.9338 ± 0.0211	0.9661 ± 0.0066	+3.2 pp	3.2×
WALKING_UP vs WALKING_DOWN	2,950	0.9670 ± 0.0271	0.9993 ± 0.0013	+3.2 pp	21.2×
WALKING vs WALKING_UP	3,266	0.9613 ± 0.0375	0.9992 ± 0.0010	+3.8 pp	36.2×

Balanced accuracy, 5 folds × 3 seeds, logistic regression on the published feature set.

Read the last column rather than the inflation column. These tasks sit close to ceiling, so leakage only buys 3 to 4 points in the mean. What collapses is the spread. Fold-to-fold standard deviation drops from 0.0375 to 0.0010, a factor of 36.

Picture a paper reporting 0.9992 ± 0.0010 on this task. It reads as an outstanding result that replicates beautifully. It's an artifact of splitting one person's windows across train and test. Going by the headline number the leak is easy to miss. Going by the interval it's obvious.

Checks

Model-free, fast enough for a pre-commit hook, and they report every problem in one pass instead of stopping at the first one.

Check	Finds
check_group_overlap	Entities appearing in both train and test
check_duplicate_rows	Test rows byte-identical to training rows
check_target_leakage	Single features that predict the label almost alone
check_preprocessing_leakage	Scaling fitted on the full dataset before splitting
check_temporal_order	Training rows that postdate the test period

from leakhunt import audit_split, grouped_kfold, to_text

train_idx, test_idx = next(grouped_kfold(len(data), data.groups, n_splits=5))
report = audit_split(data.X, data.y, train_idx, test_idx, groups=data.groups)

assert report.passed          # gate CI on this
print(to_text(report, verbose=True))
print(report.to_json())       # machine-readable, diffable between runs

Two design decisions worth knowing about:

A skipped check never counts as a pass. Leave out groups and the group-overlap check simply doesn't run. The report records that in metadata["skipped_checks"] so you can't mistake silence for a clean bill of health.

check_preprocessing_leakage looks for one specific signature. Fit a StandardScaler correctly and your training rows end up at mean 0, sd 1. Fit it before the split and the whole frame lands at mean 0, sd 1 while the training rows sit slightly off. That gap is detectable, which matters, because otherwise this mistake is invisible. The pipeline produces perfectly plausible numbers and raises nothing.

Audit every fold automatically

Wrap any scikit-learn cross-validator and pass it anywhere a cv argument is accepted. Each fold gets audited before it's handed over, so a contaminated split fails where it was created rather than surviving all the way into a published number.

from sklearn.model_selection import GroupKFold, cross_validate
from leakhunt import AuditingSplitter

cv = AuditingSplitter(GroupKFold(n_splits=5))
scores = cross_validate(model, X, y, cv=cv, groups=groups)

Swap in KFold on grouped data and it raises LeakageError on the first bad fold. on_fail="warn" keeps going and warns instead. on_fail="collect" says nothing and records every fold in cv.reports for you to inspect afterwards. Raising is the default, because a warning buried in a training log is a warning nobody reads.

Install

pip install -e ".[dev]"    # from a clone

Needs Python 3.10 or newer. Runtime dependencies are numpy, pandas, scikit-learn and scipy.

Command line

leakhunt demo                                  # see what it catches, on synthetic data
leakhunt audit data.csv --target label --group subject_id
leakhunt audit data.csv --target label --group subject_id --split-on-group
leakhunt audit data.csv --target label --group subject_id --format markdown

Exit code is 1 when the audit fails and 2 on a usage error, so it slots into CI directly. The audit is strict by default, meaning a WARNING fails it. Pass --fail-on critical if you want warnings to come through while you're still working on them.

Data

The generators in leakhunt.datasets produce synthetic data only. Nothing in this repository comes from a real cohort, recording or network capture. The design is written up in the module: each group draws a random signature vector, then every row from that group carries the signature plus the class effect plus noise. Signature strength is the knob the dose-response table turns.

Synthetic data is the right tool for that table specifically, because you need known ground truth. To show the diagnostic reports zero inflation on a clean dataset, the dataset has to be provably clean. The UCI HAR results above are the real-world half.

The HAR dataset itself is downloaded at run time and cached under examples/.cache/, which is gitignored. It's never committed.

Limitations

Worth reading before you trust a result.

• Balanced accuracy only. measure_inflation doesn't take a custom scorer yet. On heavily imbalanced problems, precision at a fixed low false-positive rate would usually be the better call.
• check_target_leakage handles binary targets. Multiclass gets skipped with an INFO finding rather than quietly passing.
• The correct arm is deterministic. GroupKFold doesn't shuffle, so its fold scores repeat across seeds. Seeds only vary the leaky arm. That means the correct arm's spread reflects fold composition rather than seed variance.
• check_duplicate_rows finds exact matches. Near-duplicates from augmented images, resampled signals or rounded exports get through.
• check_preprocessing_leakage detects standardisation. Min-max scaling, quantile transforms and target encoding leak the same way and aren't covered yet.
• One real cohort so far. UCI HAR has clean subject ids and a well-behaved feature set. Nested grouping (site and subject), heavy class imbalance and noisy group labels are all untested.
• Not a proof of correctness. A passing audit means these particular checks found nothing. It doesn't mean the pipeline is sound.

Roadmap

• Custom scorers, and precision at fixed FPR
• Nested grouping, site and subject together
• Near-duplicate detection via embedding distance
• Min-max, quantile and target-encoding leakage detection

Development

pytest                  # 68 tests, 98% coverage
ruff check .
mypy

_{If pytest won't start and complains with a PluginValidationError mentioning nengo, something unrelated in your environment ships an incompatible pytest plugin. Run pytest -p no:nengo, or work in a clean virtual environment.}

Citing

If leakhunt is useful in published work, please cite it via CITATION.cff.

Related reading

• Anguita et al., A Public Domain Dataset for Human Activity Recognition Using Smartphones, ESANN 2013.
• Kaufman et al., Leakage in Data Mining: Formulation, Detection, and Avoidance, TKDD 2012.
• Arp et al., Dos and Don'ts of Machine Learning in Computer Security, USENIX Security 2022.
• Kapoor & Narayanan, Leakage and the Reproducibility Crisis in ML-based Science, Patterns 2023.

License

MIT. See LICENSE.