Skip to content

fix(heldout): declare assignment-design trial count - #1098

Draft
seonghobae wants to merge 4 commits into
codex/declared-heldout-context-count-20260908from
codex/declared-assignment-trials-20260908
Draft

fix(heldout): declare assignment-design trial count#1098
seonghobae wants to merge 4 commits into
codex/declared-heldout-context-count-20260908from
codex/declared-assignment-trials-20260908

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Stack

Draft successor of #1097 (525991db). Parents #1096, #1095, #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1097.

Exact head: e39a99f0ecd1e331f904f03df6a546559d9924eb.

Problem

Held-out inverse-propensity assignment evidence hid ASSIGNMENT_TRIALS = 24_000, so logging-design RMSE used a repository-selected Monte Carlo sample size.

Change

  • Require declared trial_count on _validate_assignment_design. Missing, boolean, or non-positive values fail closed.
  • The harness run still writes 24,000 as this run's choice and records trials.
  • ADR 0049 is Proposed. Production route/conduct defaults stay locked.

Verification

Local smallest tests:

  • 60 passed in 15.74s (tests/test_psychometric_routing.py, tests/test_psychometric_benchmark_boundaries.py, tests/test_planning_adr_identifiers.py)
  • interrogate on the held-out script: 100%

This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.

Remaining no-heuristics work

Other harness sample sizes (DIF, reliability, exploration rate) remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.

Logging-design evidence must fail closed without a positive trial_count,
and the declared count must be the actual assignment loop.
Remove the hidden 24,000-trial logging default. The harness run still
writes 24,000 as this run's choice and records trials.
ADR 0049 is Proposed. Production route/conduct defaults stay locked.
@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Normal predecessor integration completed at ae5aabf, preserving original assignment-trial delta e39a99f and incorporating predecessor 0771905. Includes the no-detection censoring repair, separate false-alarm/detection denominators, null candidate when no calibration threshold is eligible, and retained calibration results. Merged-tree validation: 27 tests passed in 29.96 seconds (boundary suite and ADR identifier suite), Ruff passed, diff whitespace check passed. CodeGraph indexing completed. No force push or discarded delta. Prior-head hosted checks are not evidence for this new head; protected checks/reviews and release remain pending.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: medium Normal-priority or P2 work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant