fix(benchmark): require declared paired-bootstrap coverage

[seonghobae]

Stack

Draft successor of #1074 (9f5d0ce2). Parent #1067 remains the psychometric KPI owner and still needs independent review. This PR does not close #1074.

Exact head: 5e952933d4dd97057ba76b70ea17983314a4babe.

Problem

Paired NIM comparisons used a hidden 2,000-resample 95% percentile interval and a baked-in policy subset (conduct_bounded vs route_once, optional cheapest vs route_once, plus hindsight pairs). Those choices were not reconstructible as operator declarations.

Change

• resample_count, confidence_level, seed, and comparison_pairs are required declarations. None is a fail-closed sentinel, not a statistical default.
• Coverage must be a finite exclusive unit interval. A coverage that cannot be represented with the resample count fails closed.
• Method name is paired_bootstrap_percentile; coverage is a separate numeric field.
• Unobserved or disjoint declared pairs are omitted, not imputed. Hindsight identity stays a measurement field; comparing against it requires an explicit pair.
• Report schema 4.0.0 records the declarations in provenance. CLI and the NIM workflow must pass them; 2,000 and 0.95 in those files are run declarations, not code defaults.
• ADR 0042 is Proposed. Efron (1979) and Efron & Tibshirani (1993) are cited; PDFs are not bundled.

Verification

Local three-file coverage command matching hosted quality:

• 175 passed in 10.01s
• nim_benchmark.py 1328 statements / 504 branches, 0 missed
• interrogate 100%

This is calculation-contract evidence, not buyer-held-out accuracy, p95 latency, or protected merge. Production route/conduct defaults stay locked.

Remaining no-heuristics work

Repository-authored MAX_WORKFLOW_DEPTH and DEFAULT_MAX_OUTPUT_TOKENS, and the psychometric held-out harness's 2,000-sample 95% interval on parent #1067, remain later slices. No production policy or release is authorized.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}