Your model is not smarter. It has hindsight.
Open the live demo · nothing to install
A bank builds a model to predict who will repay a loan. It scores almost perfectly in testing, so they ship it, and it fails.
The model was never good. One of the facts it was shown could only be known after the loan decision had already been made, so in testing it was reading the answer, and in production that answer does not exist yet.
This is called target leakage. It is one of the most expensive silent failures in production ML, and what makes it hard is that the mistake usually happens in a data pipeline that the person training the model never looks at.
Hindsight proves whether that happened, before the model is released.
A student aces a practice exam. Impressive, until you notice the answer sheet was on the desk. The score was real; the ability was not. Take the answers away, retake the exam, and you learn what they actually know.
That retake is what Hindsight does, and it does it with dates.
git clone https://github.com/usv240/hindsight && cd hindsight
uv sync --extra dev
uv run hindsight demoNo Docker, no DataHub, no API key, no network. It prints the column lineage path, the before-and-after scores, and the verdict, in about forty lines.
demo is a self-check and exits 0 when the demo itself passed. The commands that act as
release gates are the ones that carry the verdict in their exit code: 3 blocks, 2 means
the evidence was insufficient, 0 clears.
Or open the live demo and click a scenario.
Four steps. The unusual one is the first, and it is the one that needs DataHub.
flowchart TB
subgraph DH["DataHub"]
L["<b>Column-level lineage</b><br/>which column was built from which"]
end
subgraph HS["Hindsight: deterministic, no LLM in the path"]
A["<b>1 · Trace it back</b><br/><i>where did each fact come from?</i>"]
B["<b>2 · Check the clock</b><br/><i>when did it become knowable?</i>"]
C["<b>3 · Read the code</b><br/><i>is there a cutoff guard?</i>"]
D["<b>4 · Rebuild and re-test</b><br/><i>remove the future, retrain</i>"]
A --> B --> C --> D
end
L -->|"read"| A
D --> V{"Verdict"}
V -->|"exit 0 / 2 / 3"| CI["CI release gate"]
V --> HA["Human approval"]
HA -->|"approved writes"| W["Tag · property<br/>Document · incident"]
W -->|"written back, then re-read"| L
| Step | What happens |
|---|---|
| 1. Trace it back | Follow DataHub's column-level lineage backwards from the model to the real source of every feature it used |
| 2. Check the clock | For each source, ask when its rows became knowable, and compare that to the moment the decision had to be made |
| 3. Read the code | Parse the SQL that built the feature. If it joins a post-decision source with no available_at <= prediction_time guard, that is proof, and no retraining is needed |
| 4. Rebuild and re-test | Reconstruct the feature using only records that existed at the cutoff, retrain, compare. A leaked feature loses its advantage; a legitimate one keeps it |
Either step 3 or step 4 is sufficient alone. Hindsight reports which one fired rather than blending them, so you can see exactly what settled it.
Only after a human approves is the finding written back into DataHub, so the next engineer inherits it instead of rediscovering it.
The leak is created upstream, in the warehouse, often three joins away from the model. So:
| Where you might look | Why it cannot see the problem |
|---|---|
| The training notebook | The defect was created far upstream, before read_parquet |
| The feature store | It sees finished features, not where they came from |
| Monitoring | It notices months later, once the losses arrive |
| A data quality tool | Every value is valid. Nothing is null, nothing is out of range |
Only a map spanning raw tables, transformations, features and models can answer "could this have been known at the time?" — and that map is DataHub's column-level lineage.
No LLM decides anything. A language model may explain the evidence in prose. Reaching a verdict runs through deterministic code with no model in the path.
Four levels, each self-contained. Stop whenever you are satisfied.
uv sync --extra dev
uv run hindsight demoA complete audit on metadata recorded from a real DataHub instance.
uv run hindsight serve # http://127.0.0.1:8100Click a scenario. Five are offered across three industries, and one is expected to pass — a gate that can only say no is indistinguishable from one that is not looking. Toggle Plain English / Technical. Press View evidence to read the raw record the page argues from, or Download to take it away.
Hindsight's own scenarios are generated by code in this repository, so on their own they
cannot answer the fair question. examples/adapter/ is committed CSV in
a different domain — subscription churn — with different column names and a different leak
mechanism.
uv run hindsight validate-point-in-time --scenario examples/adapter/scenario.jsonSUSPECT FEATURE
baseline, honest features only 0.825 AUC
with the suspect feature 0.993
rebuilt as of the decision 0.826
advantage retained 0.4%
post-cutoff records excluded 664
LEGITIMATE CONTROL (audited the same way, must stay clean)
observed 0.960 -> point-in-time 0.960
advantage retained 100.0%
collapsed False
VERDICT: confirmed
The feature looked worth 0.168 AUC. Once records that did not exist at the decision were removed, it was worth 0.0007.
When you do not know which feature is guilty, sweep all of them:
uv run hindsight sweep-features --scenario examples/adapter/scenario_wide.jsonfeature obs AUC pit AUC adv lost collapsed
plan_changes_to_date 0.9932 0.8257 0.1674 True
support_tickets_snapshot 0.9601 0.9601 0.0000 False
logins_30d 0.8251 0.8251 0.0000 False
tenure_snapshot 0.8252 0.8252 0.0000 False
spend_snapshot 0.8252 0.8252 0.0000 False
FLAGGED (1): plan_changes_to_date
Read the second row before the first. support_tickets_snapshot scores 0.96 against a 0.825
baseline — one of the most predictive features present, and entirely legitimate. Rank this
list by importance and it lands near the top. Rank by advantage lost to the cutoff and it
sits at zero, where it belongs. That distinction is the whole method.
The same data is also available as a wide snapshot table, which is the shape most feature stores actually use. A test asserts both shapes agree on the verdict to within 1e-9.
Full walkthrough, including how to point it at your own files: examples/adapter/README.md
uv run datahub docker quickstart --quickstart-compose-file docker/datahub.quickstart.yml
uv run hindsight serveThe status pill turns green on its own. The write-back panel becomes live: approve it and Hindsight publishes a tag, a structured property, an audit Document and an incident, then re-reads every one to prove it persisted. Full steps in QUICKSTART.md.
Needs no DataHub and no seeded data:
uv run hindsight scan-sql path/to/dbt/models --post-outcome-table payments_after_decisionScanned 214 SQL file(s) under models
VIOLATIONS (2) - post-outcome data with no cutoff:
models/marts/fct_applications.sql
reads payments_after_decision without an availability guard
clean: 11 violations: 2 unchecked: 0 no post-outcome source: 201
Exit 3 blocks a pull request. Files that could not be parsed are reported as
unchecked, never as clean — a file that was never examined has not passed.
uv run hindsight benchmark sweeps the defect from total reach down to almost none, against
a matched clean control for every case. 42 cases, 0 false positives, 0 false negatives.
The breakdown matters more than the headline:
| Reach of the defect | Mean AUC delta | Statistical route fires | Caught overall |
|---|---|---|---|
| 100% | 0.1768 | yes | yes |
| 70% | 0.1578 | yes | yes |
| 40% | 0.1086 | no | yes |
| 15% | 0.0451 | no | yes |
| 2% | 0.0066 | no | yes |
The statistical route stops firing below 40% reach. At 2% the performance difference is 0.0066 of AUC, invisible to any threshold anyone would realistically set, and the deterministic SQL/time proof still catches it. That is the argument for two independent routes, measured rather than claimed.
Read the perfect score with care. Ground truth is structural — "leaked" means the query joins a post-outcome source with no guard, and the deterministic route reads that same query — so its score is close to true by construction. What is measured honestly is the statistical route's collapse, the absence of false positives across 21 guarded queries, and the vanishing AUC delta. Full caveats in evidence/benchmark/2026-07-28.md.
Leakage detection is an active field. Everything below is real, works, and is worth using — the point of this table is to be precise about the gap, not to dismiss any of it.
| Prior work | What it analyses | Covers upstream temporal leakage? |
|---|---|---|
| Yang et al., ASE 2022 | Static analysis of notebook code | No — notebook scope |
| LeakageDetector, ICSME 2025 | PyCharm plugin over Python ML code | No — detects Overlap, Multi-test, Preprocessing |
| LeakageDetector 2.0, ICSME 2025 | The same, extended to Jupyter | No |
| A Grammar of ML Workflows, 2026 | Type system rejecting leakage at call time | Partly — has split_temporal with embargo, but scoped to one execution context |
| Feature stores (Feast, Tecton, Azure ML) | As-of joins at retrieval time | Prevents it, if every feature goes through the store |
Two things follow, and both are worth stating plainly.
The three leakage types current tooling detects are all notebook-internal. Overlap, Multi-test and Preprocessing are mistakes made while splitting and fitting. The class Hindsight targets — a feature whose values were computed from records that did not exist at the decision — is not in that taxonomy. The 2026 grammar paper comes closest and names the boundary itself: it explicitly lists "manual feature derivation before split" among its enforcement gaps, and scopes itself to "one notebook session, one pipeline run".
A feature store with correct as-of joins prevents this by construction, and that is a better answer than detecting it. If every feature your model uses is retrieved through point-in-time joins, you do not need Hindsight for this class of defect. It is for the very common case where features are assembled in SQL upstream with no such guarantee — and for the separate problem of proving, after the fact and to someone else, that a specific model was clean.
That second part is why the output is an evidence record with a re-read write-back rather than a warning in an IDE.
Run against dbt-labs/jaffle-shop, a public project
written by someone else with no knowledge of this tool. It parsed all 13 models, resolved
ref() templating, and correctly reported which model reads a nominated source without a
guard.
It also found two bugs in Hindsight. Every dbt model was being reported as clean, because Jinja templating made them unparseable and unparseable was falling through to "no post-outcome source" — a false negative wearing a pass, which is precisely the failure this project exists to prevent. Both are fixed and covered by tests.
| Objection | What stops it |
|---|---|
| "It probably flags everything." | A legitimate feature with a larger importance score than the leaked one is audited on every run and must come back clean. It does. |
| "It can only ever say no." | One scenario is expected to pass, and does: the same model after the repair. |
| "The numbers were tuned." | The generator is frozen and its seed committed. Whatever it produces is published, including a collapse threshold missed by 5.1 points, disclosed rather than hidden. |
| "An LLM decided this." | It cannot. Reaching a verdict runs through deterministic code with no model in the path. |
| Verdict | Minimum evidence | CI exit |
|---|---|---|
insufficient_metadata |
Required lineage or time evidence is absent | 2 |
needs_review |
Ancestry is suspicious, but direction or time is unproven | 2 |
high_confidence |
Directional outcome lineage and an availability violation | 3 |
confirmed |
Deterministic cutoff proof or qualified point-in-time reconstruction | 3 |
clear_for_release |
Configured checks and planted safe controls pass | 0 |
Plain ablation is explanatory context only. It is structurally unreachable as a confirmation branch, because a legitimate feature can easily have a larger ablation delta than a leaked one.
Hindsight does not guess which feature is suspicious, and it does not read your warehouse. It asks DataHub one thing, through the Agent Context Kit: is there a path from this column to that one, and which way does it run?
hindsight trace-lineage \
--source-column payment_recorded_at \
--target-column days_since_last_payment
{ "available": true, "found": true, "hops": 2,
"path": ["payment_recorded_at",
"urn:li:query:d47c395f75a3ca4422cf5f44739b620e001eeb455358b323ff42161e4acc9377",
"days_since_last_payment"],
"resolved_by": "agent_context_kit" }Look at the middle hop. It is a DataHub query entity. The catalog does not only say these two columns are connected — it identifies the transformation that connects them. So the evidence chain cites the catalog's own record of the SQL, rather than a file someone pointed the tool at. Recorded live against DataHub Core v1.5.0.6; committed at evidence/integrations/lineage-trace.json.
available and found are separate fields. "The catalog says there is no path" is
evidence against leakage. "We could not reach the catalog" is not evidence at all.
Collapsing them would let an outage read as a clean bill of health — the first version
did exactly that, and it was fixed.
| Surface | Used | How |
|---|---|---|
| Context graph | yes | Column-level lineage, ML entities, schema, profiles |
| MCP Server | yes | Discovery, lineage reads, governed mutations |
| Agent Context Kit | yes | get_lineage_paths_between, the column-level directional path query this project is built on |
| DataHub Skills | yes | datahub-ml-release-audit |
| DataHub Actions | yes | Audits a model the moment it appears, with nobody triggering it |
| Analytics Agent | — | Not applicable; this is not a text-to-SQL product |
It can watch instead of waiting. The isolated
docker/hindsight-action.compose.yml deployment runs
on DataHub's official Actions image, rejects unrelated entity types and URNs, audits one
exact-bound model, and emits a local JSONL proof. It never publishes evidence itself, because
a process that silently rewrites governed metadata is the thing this project argues against.
Live result: confirmed / block / exit 3. See docker/ACTION.md.
After human approval, then re-read to prove it persisted:
hindsight:leakage-confirmedfield tag on the offending columnhindsight.auditVerdictstructured property on the model- a linked audit Document with the evidence path, control results and remediation
- an active
ML_LEAKAGEincident, reused on retry rather than duplicated
Publication is dry-run by default and requires explicit --approve-writeback. A write
that reports success but cannot be read back is treated as a failure.
| Overview and scenario picker | Technical evidence view |
|---|---|
 |
 |
Screenshots are generated from the running app by scripts/capture_screenshots.py, so they
cannot drift from what it renders. Every route is checked against WCAG 2.1 AA contrast on
every build by scripts/check_accessibility.py.
Read this before assuming what works. The components are at different maturity levels.
| Component | Works on your own data today? |
|---|---|
| SQL / temporal verification | Yes. scan-sql walks a whole dbt project. No DataHub needed |
| Verdict lattice and evidence grading | Yes. Generic over any AuditCase |
| Point-in-time reconstruction | Yes, with a contract. Mapped CSV/Parquet, long-form or wide snapshots |
| DataHub write-back and re-read | Yes, for an exact bound dataset URN whose field exists in schema metadata |
Point-in-time reconstruction needs an availability timestamp. A nightly overwrite that keeps no history cannot be audited this way by anyone, including this tool: the information required to answer the question was never stored. The remaining boundary is transport — Hindsight reads exported snapshots rather than connecting directly to every warehouse engine.
The synthetic scenarios are total by construction. The planted leak reaches every record,
so its observed AUC of 1.000000 is expected, and real leakage can be subtler. The generator
is frozen; measured results are published rather than retuned.
| Measure | Observed | Point-in-time |
|---|---|---|
| Planted case AUC | 1.000000 | 0.833630 |
| Advantage over baseline | 0.301712 | 0.135342 |
| Advantage retained | — | 44.858% |
| Legitimate control AUC | 0.924842 | 0.924842 |
Measured margin on the collapse rule: 5.142 percentage points. The 50% boundary is a visible, configurable policy, not a universal constant.
All three came out of building this, not out of looking for something to contribute.
| Status | |
|---|---|
datahub#18705 — document the required customType on CUSTOM incidents |
Merged |
datahub#18822 — stop docker quickstart crashing on legacy Windows code pages |
Open, awaiting review |
datahub-skills#68 — add the datahub-ml-release-audit skill |
Open, awaiting review |
The first is small and worth explaining. Hindsight raises a DataHub incident during
write-back, and the incidents tutorial lists CUSTOM as a supported type without mentioning
that customType is required alongside it. Following the guide as written fails with
customType is required: Failed to create incident. Verified against DataHub Core v1.5.0.6.
The second we hit bringing DataHub up for this project. datahub docker quickstart printed a
check mark through a cp1252 console and raised UnicodeEncodeError after the stack was
already healthy, so a successful install reported failure. The same crash reaches the strings
that explain how to recover from a broken install.
| Command | What it does |
|---|---|
hindsight demo |
Full offline audit. The judge path |
hindsight serve |
The evidence console on :8100 |
hindsight validate-point-in-time --scenario ... |
Reconstruct one scenario, generated or external |
hindsight sweep-features --scenario ... |
Audit every candidate feature and rank them |
hindsight scan-sql <dir> --post-outcome-table ... |
Walk a dbt project. Exit 3 blocks a PR |
hindsight verify-sql <file> --post-outcome-table ... |
Check one transformation for a cutoff guard |
hindsight benchmark |
The 42-case sweep |
hindsight trace-lineage ... |
Column-level path query via the Agent Context Kit |
hindsight publish-audit --approve-writeback |
Write the finding into DataHub |
Define your own audit target with audits/, then:
uv run hindsight demo-audit --audit audits/my_pipeline.json
uv run hindsight serve --audit audits/my_pipeline.json --target-urn "<exact-dataset-urn>"Frozen seeds, committed lockfile, CI on Ubuntu and Windows across Python 3.11 and 3.12. Raw
machine reports stay ignored as *.local.json; the sanitized summaries in
evidence/ are versioned with the code that produced them.
- Kaufman, Rosset & Perlich, Leakage in Data Mining: Formulation, Detection, and Avoidance, KDD 2011 — defines leakage through legitimacy and learn-predict separation. Their second class, leakage in training examples, needs record-level provenance and is explicitly out of scope here rather than implied.
- Yang, Brower-Sinning, Lewis & Kästner, Data Leakage in Notebooks, ASE 2022 — found leakage pervasive across 100,000+ public notebooks. Their static analysis works inside the notebook; the class Hindsight targets originates upstream, in the pipeline. arXiv:2209.03345
- Kapoor & Narayanan, Leakage and the Reproducibility Crisis in ML-based Science, Patterns 2023 — 294 papers across 17 disciplines affected.
Apache 2.0. See LICENSE.