Add a forward-time historical statement-only backtest

[CmsChase]

Summary

This PR completes a focused forward-time historical statement-only backtest after the corrected evaluation controls merged in #17.

It adds:

• a frozen protocol and machine-readable configuration;
• deterministic cache archiving, verification, and a sealed source manifest;
• separate validation-selection and locked-test stages with leakage guards;
• committed selection, prediction, uncertainty, coverage, and error-analysis artifacts;
• a public result report with explicit evidence limits.

Frozen comparison

Complete contest start-time buckets are split chronologically at 70/10/20.

• Comparator: index_rank + index_number.
• Primary: the comparator plus the frozen 41 statement-structure features.
• Forbidden inputs include ratings, points, tags, solved/acceptance counts, submissions, participation, and hacks.
• Ridge alpha is selected using validation MAE only.
• The selected models are refit on train + validation.
• The primary contrast uses 10,000 paired bootstrap resamples clustered by contest.
• Error examples are the automatically ranked ten largest primary-model absolute test errors.

Locked result

Model	Test MAE	RMSE	R²
Index-only comparator	477.2151	575.4347	0.5005
Index + statement structure	401.2704	503.2983	0.6179

The frozen contrast is primary MAE - comparator MAE = -75.9447, about a 15.9% reduction relative only to the predeclared index-only comparator. The paired contest-cluster bootstrap 95% interval is [-86.7339, -65.0783].

The test split contains 2,468 problems from 355 contests. Statement parsing coverage is 10,906 / 10,979 overall and 2,422 / 2,468 on test.

Evidence chronology

• Correct experiment configuration and model selection #17 merged the public erratum and corrected evaluation controls.
• acea3be froze the backtest design and exact allowlists.
• f2b979f sealed the 10,979-entry cache manifest.
• 76c69f1 added the two-stage runner and leakage guards.
• 1260f4bd54db42ac985797bd99a1d462cc0289e5 committed and pushed the validation selection before the locked test output was produced.
• 12b85070180d9170981e8ff4c54b2c5c30980947 committed the locked test outputs.
• 18e51dcb106ee70d0ceb7acbddb7f8e54469a504 added the result report, coverage artifact, and final-result hash manifest.

Git history therefore records one committed locked evaluation after the public selection commit. The output guard prevents overwriting the same output path, but this is not a technical proof that no unrecorded execution occurred elsewhere.

Evidence boundary

This is a forward-time retrospective evaluation, not a prospective test or independent preregistration.

• Pages were collected after their contests and may include later corrections.
• Earlier project work had prior exposure to the historical data.
• The legacy cache is decoded response text re-encoded as UTF-8, not byte-for-byte HTTP responses.
• The 718 MB page cache is not committed. The prepared dataset can reproduce the model selection and reported metrics, but the repository alone cannot reproduce HTML-to-feature extraction.
• The 15.9% figure is not a comparison with the legacy full-API model or the state of the art.
• Extreme ICPC-style overpredictions remain visible in the automatic Top-10 and are discussed without retroactive model changes.

See docs/HISTORICAL_STATEMENT_BACKTEST_RESULTS.md for the full interpretation and artifact links.

Validation

• 148 repository tests passed locally.
• Selection-lock and final-result SHA-256 manifests verify.
• Coverage and headline metrics were independently recomputed from committed artifacts.
• git diff --check passes.
• GitHub Actions is green on current HEAD 18e51dcb106ee70d0ceb7acbddb7f8e54469a504.

TF-IDF is outside this primary result. #18 and #19 are not dependencies and can be closed as superseded after this PR merges.