fix(heldout): declare assignment-design trial count - #1098
Conversation
Logging-design evidence must fail closed without a positive trial_count, and the declared count must be the actual assignment loop.
Remove the hidden 24,000-trial logging default. The harness run still writes 24,000 as this run's choice and records trials.
ADR 0049 is Proposed. Production route/conduct defaults stay locked.
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Signed-off-by: Seongho Bae <me@seonghobae.me>
|
Normal predecessor integration completed at ae5aabf, preserving original assignment-trial delta e39a99f and incorporating predecessor 0771905. Includes the no-detection censoring repair, separate false-alarm/detection denominators, null candidate when no calibration threshold is eligible, and retained calibration results. Merged-tree validation: 27 tests passed in 29.96 seconds (boundary suite and ADR identifier suite), Ruff passed, diff whitespace check passed. CodeGraph indexing completed. No force push or discarded delta. Prior-head hosted checks are not evidence for this new head; protected checks/reviews and release remain pending. |
Stack
Draft successor of #1097 (
525991db). Parents #1096, #1095, #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1097.Exact head:
e39a99f0ecd1e331f904f03df6a546559d9924eb.Problem
Held-out inverse-propensity assignment evidence hid
ASSIGNMENT_TRIALS = 24_000, so logging-design RMSE used a repository-selected Monte Carlo sample size.Change
trial_counton_validate_assignment_design. Missing, boolean, or non-positive values fail closed.trials.Verification
Local smallest tests:
tests/test_psychometric_routing.py,tests/test_psychometric_benchmark_boundaries.py,tests/test_planning_adr_identifiers.py)This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.
Remaining no-heuristics work
Other harness sample sizes (DIF, reliability, exploration rate) remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.