Skip to content

fix(heldout): declare decision-latency repetitions per context - #1096

Draft
seonghobae wants to merge 4 commits into
codex/declared-sequential-drift-horizon-20260908from
codex/declared-latency-repetitions-20260908
Draft

fix(heldout): declare decision-latency repetitions per context#1096
seonghobae wants to merge 4 commits into
codex/declared-sequential-drift-horizon-20260908from
codex/declared-latency-repetitions-20260908

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Stack

Draft successor of #1095 (74c27e7e). Parents #1093, #1092, #1091, #1090, #1074, and psychometric KPI owner #1067 stay open. This PR does not close #1095.

Exact head: 8ccf011dbf74f7140dd29fc2661234117c7cb36b.

Problem

Paired held-out ranking timings hid LATENCY_REPETITIONS = 200, so buyer-facing decision-latency p50/p95 used a repository-selected Monte Carlo sample size.

Change

  • Require declared repetitions_per_context on _measure_paired_latency. Missing, boolean, or non-positive values fail closed.
  • The harness run still writes 200 as this run's choice and records latency_repetitions_per_context.
  • ADR 0047 is Proposed. Production route/conduct defaults stay locked.

Verification

Local smallest tests:

  • 56 passed in 18.66s (tests/test_psychometric_routing.py, tests/test_psychometric_benchmark_boundaries.py, tests/test_planning_adr_identifiers.py)
  • interrogate on the held-out script: 100%

This is declaration-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge.

Remaining no-heuristics work

Other harness sample sizes remain later slices. No production policy or release is authorized. Parent #1067 still needs independent review.

Paired ranking timings must fail closed without a positive per-context
repetition count, and the declared count must be the actual timing loop.
Remove the hidden 200-repetition timing default. The harness run still
writes 200 as this run's choice and records latency_repetitions_per_context.
ADR 0047 is Proposed. Production route/conduct defaults stay locked.
@coderabbitai

coderabbitai Bot commented Sep 7, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Normal integration checkpoint b8a7b3b: retains both predecessor8ccf011dbf74f7140dd29fc2661234117c7cb36b and repaired parent318370e84aa4a2a2d280493057a6b19b23484a7e. Only the gap-baseline insertion conflicted; both entries were preserved. Required latency repetitions remain in place, while censored no-detection semantics and parent static-analysis repairs are retained. Integrated boundary and ADR identifier tests:23 passed in33.65s, exit0. This is local focused evidence, not protected-main merge, current-head hosted success, release or measured buyer latency. Draft/parent-first review boundaries remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: medium Normal-priority or P2 work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant