EvalRepro

Detect semantic drift in AI evaluation inputs and contracts before it silently changes benchmark results.

An evaluation can keep importing and its tests can stay green while a dependency, dataset revision, task configuration, or adapter changes sample membership, ordering, targets, choices, metadata, or types. EvalRepro creates hash-only manifests in isolated environments and compares the evaluation contract rather than relying on import success alone.

dependency / dataset / config change
                 ↓
       create two manifests
                 ↓
     compare scope + coverage +
     samples + semantic fields
                 ↓
reproducible | order drift | semantic drift

Status: public alpha. The manifest schema and adapter API may change before v1. Raw evaluation records are never written to a manifest, but hashes are not anonymisation; review the privacy notes before publishing manifests from sensitive datasets.

30-second terminal demo

The recording snapshots two synthetic one-record JSONL evaluations and detects a target change as semantic_drift. The generated manifests contain hashes and compact diagnostics, not the raw sample text. The exact synthetic inputs are committed in examples/terminal-demo/.

Replay it locally with asciinema:

asciinema play docs/terminal-demo.cast

Download the terminal recording.

What EvalRepro checks

• evaluation identity, task version, adapter parameters, and selected semantic fields;
• complete versus partial coverage and declared/processed sample counts;
• ordered and unordered sample digests;
• input, target, choices, and metadata digests by default;
• top-level field type distributions;
• added/removed sample hashes and the first ordered mismatch;
• runtime and source provenance without treating dependency versions as semantic scope.

Verdicts

Verdict	Meaning
reproducible	Scope, coverage, sample content/order, field digests, and types match.
order_drift	The same records remain, but their order changed.
semantic_drift	Record content, semantic fields, membership, or types changed.
coverage_mismatch	The manifests cover different ranges or completeness levels.
scope_mismatch	The manifests describe different tasks, parameters, or fields.

Install

Install the public alpha from PyPI:

python -m pip install "evalrepro==0.1.0a2"

For the optional Inspect adapter:

python -m pip install "evalrepro[inspect]==0.1.0a2"

To install the same immutable release directly from source:

python -m pip install \
  "evalrepro @ git+https://github.com/seva9523/EvalRepro.git@v0.1.0a2"

Quick start: JSONL

Create one JSON object per line:

{"id":"1","input":"What is 2+2?","target":"4","metadata":{"split":"test"}}

Snapshot the baseline and candidate:

evalrepro snapshot jsonl baseline.jsonl --name arithmetic-v1 -o baseline.manifest.json
evalrepro snapshot jsonl candidate.jsonl --name arithmetic-v1 -o candidate.manifest.json

If sample IDs should not appear in a published manifest, add --no-id-preview to either snapshot command. The IDs still contribute to the sample digests, so this changes the diagnostic presentation only.

Compare them:

evalrepro compare baseline.manifest.json candidate.manifest.json \
  --json report.json \
  --markdown report.md

Exit codes are 0 for a reproducible comparison, 2 for detected drift/mismatch, and 3 for an invalid source or manifest.

Quick start: Inspect AI

Run the same task in two isolated environments:

evalrepro snapshot inspect inspect_evals.bbq.bbq:bbq \
  --kwargs '{"subsets":"Age"}' \
  -o artifacts/datasets-4.8.5.json

Repeat under the candidate dependency version, then compare the manifests. Inspect message IDs are removed by the adapter because they are runtime-generated rather than evaluation semantics. Local image content is represented by a content digest instead of an environment-specific path.

Quick start: Harvey LAB

The first-party Harvey LAB adapter reads a local benchmark checkout without importing the Harvey harness or running a model or judge.

evalrepro snapshot harvey-lab ./harvey-labs \
  --task all \
  -o artifacts/harvey-baseline.json

evalrepro snapshot harvey-lab ./harvey-labs-candidate \
  --task all \
  -o artifacts/harvey-candidate.json

evalrepro compare artifacts/harvey-baseline.json artifacts/harvey-candidate.json

--task accepts all, a practice-area/task prefix, or one exact task ID. The adapter hashes effective instructions, rubrics, deliverables, unknown task fields, repository-relative source paths, and source file bytes. Raw task text and document contents are not written to the manifest. Use --no-id-preview when task IDs should also be omitted.

See the Harvey LAB adapter contract for scope, provenance, safety, and current semantic decisions.

GitHub Action

Use the immutable public-alpha tag:

- uses: seva9523/EvalRepro@v0.1.0a2
  with:
    baseline: artifacts/baseline.json
    candidate: artifacts/candidate.json
    report-json: artifacts/evalrepro-report.json
    report-markdown: artifacts/evalrepro-report.md
    fail-on-drift: "true"

The Markdown report is appended to the GitHub Actions job summary.

Founding case study

EvalRepro grew out of a dependency investigation for UKGovernmentBEIS/inspect_evals#2049. A fork-side matrix compared Hugging Face datasets==4.8.5 and datasets==5.0.1 across seven complete Inspect datasets and 22,773 records. The tested sample membership, ordering, inputs, targets, choices, metadata, types, task scope, and provenance matched exactly. The case study is documented in case-studies/inspect-datasets-5.md. Its status is explicitly recorded as fork-validated, not upstream-accepted.

Harvey LAB pinned-revision case study

An opt-in workflow compared Harvey LAB firm-knowledge revisions 55510f0e609ffa5cf6f5df17d9a813ce4bb33d0c and 60071cc424d6479569626b8c76d90b958fe2d6c0. Both manifests covered 250 task contracts and the same shared DMS inventory of 9,288 files (520,597,269 bytes). EvalRepro classified 250 of 250 task contracts as changed, isolated the drift to input and target, and returned semantic_drift; scope, coverage, choices, metadata, and top-level field types matched.

The case study and hash-only evidence are EvalRepro workflow-validated, not upstream-reviewed or upstream-merged. No Harvey adoption or endorsement is claimed.

Built-in adapters

• Generic JSON Lines
• Inspect AI / Inspect Evals (optional extra)
• Harvey LAB task contracts from a local checkout
• Framework-neutral SnapshotSource API for custom adapters

Planned adapters are tracked in the roadmap. The most useful contributions are adapters backed by a real reproducibility case, not thin wrappers added only to increase framework count.

Contributing

External contributors are central to this project. There are three useful entry points:

1. submit a reproducibility case or bug report;
2. add fixtures, normalisers, reports, or platform tests;
3. propose and implement a framework adapter using the public adapter contract.

Start with CONTRIBUTING.md, the Adapter specification, and issues labelled good first issue or help wanted.

Non-goals

EvalRepro does not run or grade language models, replace evaluation frameworks, compare model quality, or claim that matching inputs guarantee identical model outputs. It protects the evaluation contract and data path so score changes can be interpreted with greater confidence.

Security and privacy

Manifests contain hashes, compact ID previews by default, runtime details, and provenance. They do not contain raw sample text. Hashes of small or predictable values can still be brute-forced and ID previews can be sensitive. Use --no-id-preview when the preview should be omitted, and do not publish manifests from confidential evaluations without reviewing the remaining fields. See docs/privacy.md, SECURITY.md, and METHODOLOGY.md.

Project stewardship

EvalRepro was created and is maintained by Sevinj Ahmadova, whose work focuses on AI evaluation, evaluation reproducibility, human-in-the-loop systems, and enterprise AI adoption. Research and professional background are available on LinkedIn.

Accepted external contributors are credited in the changelog and release notes. Substantive, sustained contributors can grow into component-maintainer roles through the governance process.

Licence

Apache-2.0. See LICENSE.