Skip to content

Correct experiment configuration and model selection - #17

Merged
CmsChase merged 5 commits into
mainfrom
research/evaluation-governance
Jul 12, 2026
Merged

CmsChase merged 5 commits into
mainfrom
research/evaluation-governance

Conversation

@CmsChase

Copy link
Copy Markdown
Owner

Summary

This PR corrects the historical evaluation pipeline and publishes the corresponding interpretation note as one reviewable change.

  • enforce a strict, versioned experiment config
  • record effective configuration fingerprints across generated artifacts
  • preserve the historical 70/15/15 split behavior in a dedicated audit config
  • select models and feature settings only on validation MAE
  • attach test metrics only after the candidate is fixed
  • remove test-selected exposure winner claims
  • add disagreement fixtures where validation and test favor different candidates
  • publish a calm public erratum while retaining historical papers and tags
  • consolidate the two-minute project introduction in QUICK_OVERVIEW.md
  • move error analysis under docs/

Root cause

The previous loader did not consume the nested split keys in the checked-in config and fell back to 70/15/15 for both strategies. Several downstream summaries also selected a reported winner after observing test MAE. Training code used training rows, but repeated test-informed selection means the historical test outcomes are retrospective rather than unseen confirmatory evidence.

Reproducibility impact

  • configs/experiment.yaml is the corrected current pipeline.
  • configs/experiment_legacy_v6.yaml records the effective historical ratios for audit.
  • The legacy config alone does not recreate a paper; exact historical reproduction also requires the matching Git tag and original local data snapshot.
  • Historical generated tables and PDFs are intentionally not overwritten.

Validation

  • python -m pytest -q ? 131 passed
  • python -m compileall -q src
  • git diff --check origin/main...HEAD
  • scanned the proposed diff for prohibited public naming and prospective files

Scope boundary

Prospective input capture, frozen prediction, protocol, and ledger work are intentionally excluded and will be reviewed separately after this PR.

@CmsChase
CmsChase marked this pull request as ready for review July 12, 2026 07:48
@CmsChase
CmsChase merged commit 6ad7bdc into main Jul 12, 2026
1 check passed
@CmsChase
CmsChase deleted the research/evaluation-governance branch July 13, 2026 06:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant