fix(heldout): declare held-out context population

[seonghobae]

Stack

Draft successor of #1096 (8ccf011d). Parents #1095, #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1096.

Exact head: 525991db1ebb0adc58a1a5a895329769ad445672.

Problem

Held-out accuracy and decision-latency averages hid TRAIN_CONTEXTS = 24, so buyer-facing Brier, log loss, regret, and latency p50/p95 used a repository-selected population.

Change

• Require declared context_count on _build_evidence, _evaluate_quality, and _measure_paired_latency. Missing, boolean, or non-positive values fail closed.
• The harness run still writes 24 as this run's choice and records contexts_held_out / contexts_train.
• ADR 0048 is Proposed. Production route/conduct defaults stay locked.

Verification

Local smallest tests:

• 58 passed in 17.78s (tests/test_psychometric_routing.py, tests/test_psychometric_benchmark_boundaries.py, tests/test_planning_adr_identifiers.py)
• interrogate on the held-out script: 100%

This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.

Remaining no-heuristics work

Other harness sample sizes (assignment, DIF, reliability) remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Integrated predecessor b8a7b3b by a normal merge, preserving the original 525991d context-population delta. New head: 0771905. This carries the no-detection censoring repair, explicit false-alarm/detection denominators, null candidate when calibration has no eligible threshold, and all calibration results into this successor. Validation on the merged tree: 25 boundary/ADR identifier tests passed in 21.47 seconds; Ruff passed; git diff --cached --check passed. An initial test command referenced a nonexistent ADR test filename and ran no tests; the corrected command used tests/test_planning_adr_identifiers.py and completed with exit 0. No delta discarded, no force push, no production-default change or release claim. Earlier hosted checks belong to the previous head and must be rerun for this head.