fix(heldout): declare sequential-drift horizon and record censored delays

[seonghobae]

Stack

Draft successor of #1093 (d40a4981). Parents #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1093.

Exact head: 318370e84aa4a2a2d280493057a6b19b23484a7e.

Static-analysis follow-up: eight inherited findings were reproduced on predecessor 74c27e7e, then corrected without moving the runtime guard after numerical imports. The two changed files pass Ruff and 20 boundary tests pass in 7.84s on this source. Earlier seeded-screen results below apply to df31500a, not a fresh run of this head.

Problem

The held-out CUSUM screen hid a 500-replication, 250-observation, change-at-100 horizon, aborted when a replication never alarmed, and stored a 95% Wilson bound as false_alarm_rate_upper_95. Missed detections could not enter the delay KPI.

Change

• Require declared replications, horizon_observations, change_after_observations, and confidence_level. Missing, non-positive, change_after >= horizon, or non-exclusive-unit-interval coverage fail closed.
• Count no-alarm replications as censored non-detections; record horizon - change_after separately as the observation limit, never as an observed detection. Quantiles describe post-change detections only and are null when empty. Preserve failure denominators and all calibration results; no eligible candidate returns null and false acceptance.
• Store the Wilson upper bound as false_alarm_rate_upper_bound using the declared coverage.
• The harness run still writes 500, 250, 100, and 0.95 as this run's choices. ADR 0046 is Proposed. Production route/conduct defaults stay locked.

Verification

Current repair: 20 boundary tests passed in 17.22s, after reproducing the false horizon-valued median. A separate full-size seeded screen preserves threshold 6.6, false-alarm rate 0.024, and detected-only p95 20. These are local unit/screen results, not buyer outcomes.

Historical tests on predecessor 74c27e7e (not transferred to the current head):

• 54 passed in 19.24s (tests/test_psychometric_routing.py, tests/test_psychometric_benchmark_boundaries.py, tests/test_planning_adr_identifiers.py)
• interrogate on the held-out script: 100%

This is declaration-and-censoring-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.

Remaining no-heuristics work

Other harness sample sizes remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Integration finding at exact head 74c27e7: _evaluate_sequential_drift_threshold appends horizon-change_after into detection_delays for a no-alarm replication, then reports ordinary detection_delay_p50/p95 from that combined list. A horizon-censored non-detection is not an observed detection at that boundary; retaining censored_replications alone does not make those percentile labels accurate. Also _validate_sequential_drift still calls min(eligible_candidates) without an empty-candidate path, and all pre-change alarms cause a ValueError that loses the otherwise valid false-alarm summary. Parent #1067 now preserves these outcomes at fb12256 (current parent84a60523): explicit non-detection denominator, detected-only delay population with null empty quantiles, no eligible candidate -> null candidate/false acceptance and retained calibration results. Please preserve that effective delta while retaining this PR’s required horizon/coverage declarations during normal stack integration. No close, force push, or replacement of this PR’s declaration delta is requested. This is source review evidence, not a new executed test or current-head approval.

[seonghobae]

Repair df31500 preserves required horizon/coverage declarations and the parent non-detection semantics. RED: one no-alarm regression failed (median10.0 despite zero detections). GREEN: 20 boundary tests passed in17.22s. Separate full-size seeded screen (500/250/100, coverage0.95) retains threshold6.6, false-alarm rate0.024, detected-only p95=20 and zero censored candidate replications. No-alarm and all-premature-alarm outcomes now preserve counts/null quantiles, no eligible candidate returns null/false plus full calibration results. ADR0046, doctoring sequence and gap baseline corrected; Proposed status stays. This replaces the older body claim that censoring boundaries enter the observed delay distribution. No protected merge, release or buyer KPI improvement is claimed.