Build the prospective evaluation foundation

[CmsChase]

Stack order

This is a stacked draft on top of #17. It must not merge before #17. After #17 merges, retarget this PR to main and let the normal Linux CI run on the resulting standalone diff.

Summary

This PR builds the pre-enrollment foundation for a genuinely prospective evaluation without opening the cohort or claiming a frozen protocol.

• publish a v1 pre-enrollment retirement record without rewriting the old artifacts
• add a v2 machine-readable protocol with status: draft and a tentative 2026-08-15 start
• add a dedicated T0 capture command driven only by explicit contest id, problem indices, and contest start
• fetch direct English problem statement pages without a metadata API or local metadata table
• emit an exact allowlisted feature CSV and a separate provenance sidecar
• reject partial contests, stale cache reuse, non-200 pages, parse failures, deadline crossings, and overwrite attempts
• inspect CSV headers or Parquet schemas before reading prospective row values
• reject forbidden, extra, reordered, multi-contest, duplicate, malformed, and infinite inputs
• reparse verified raw HTML and compare every allowlisted feature before prediction
• lock Ridge parameters, source hashes, runtime/BLAS versions, training-input hashes, and dependency-spec hash
• bind predictions to the protocol, model, manifest, input, sidecar, raw pages, and decoded statement hashes
• require clean committed source and real system time for model freezing
• pre-register cohort census, external GitHub timestamp evidence, one fixed outcome snapshot, bootstrap details, and minimum-threshold denominators

Safety properties covered by tests

• forbidden CSV/Parquet schemas cause zero pandas row reads
• all requested problem keys are present exactly once
• failed or challenge pages write an audited failure sidecar and no model CSV
• prediction and capture cannot be backdated through CLI options
• crossing the 30-minute deadline before or after publication removes the output
• changing an allowlisted feature and updating the sidecar hash still fails because features are reconstructed from raw HTML
• raw, decoded, model, manifest, source, runtime, and sidecar tampering are rejected
• interrupted exclusive writes remove partial artifacts

Validation

• PYTHONPATH=src python -m pytest -q ? 187 passed
• python -m compileall -q src
• python -m json.tool configs/prospective_protocol_v2.json
• git diff --check research/evaluation-governance...HEAD
• independently reviewed for blocker-level research-governance and implementation issues
• scanned the proposed diff and public names for prohibited tool-identity naming

Explicit non-goals and freeze gates

This PR intentionally does not freeze v2, commit a v2 model bundle, open the cohort, or add outcome data.

Before status may change to frozen, a later sequential PR must add and test:

1. append-only prediction, miss, and outcome ledgers;
2. reveal separation and the fixed cohort-close outcome snapshot;
3. GitHub Actions push-event timestamps proving each T0 commitment reached the default branch before the 30-minute deadline;
4. cohort census reconciliation that detects silent whole-contest omission;
5. the deterministic confirmatory-analysis implementation and fixtures, with a pre-close execution guard;
6. an end-to-end dry run on the merged default branch.

If those gates are not complete with a review buffer, the draft start date must move forward before freezing; it must never be backdated.

[CmsChase]

Paused while the project moves to a focused forward-time historical statement-only backtest. This draft will not be merged as a unit in its current form. The reusable capture and validation ideas—sealed statement pages, SHA-256 provenance, an exact feature allowlist, schema-before-values checks, forbidden-input rejection, and feature reconstruction from sealed pages—will be reimplemented in a smaller reviewable branch. The prospective protocol and freeze machinery remain deferred for a later, smaller automated pilot. The branch is being retained as a design and implementation record.

[CmsChase]

Closed as superseded by #20, which merged the focused historical statement-only backtest and reimplemented the relevant cache sealing, feature allowlist, schema/leakage checks, and artifact verification in a smaller auditable scope. This branch is intentionally retained as a design and implementation record; the prospective machinery remains deferred to a separate future pilot.