fix(heldout): require declared paired-bootstrap coverage

.github/workflows/nim-benchmark.yml

@@ -72,6 +72,9 @@ jobs:
             --task-manifest examples/nim_task_manifest.json \
             --output-dir benchmark_artifacts \
             --max-total-requests "$MAX_REQUESTS" \
+            --bootstrap-resample-count 2000 \
+            --confidence-level 0.95 \
+            --comparison-pair conduct_bounded,route_once \
             --max-workflow-depth 5 \
             --max-output-tokens 264 \
             --git-sha "$PROVENANCE_GIT_SHA" \
@@ -140,6 +143,9 @@ jobs:
             --task-manifest examples/nim_task_manifest.json \
             --output-dir benchmark_artifacts \
             --max-total-requests "$MAX_REQUESTS" \
+            --bootstrap-resample-count 2000 \
+            --confidence-level 0.95 \
+            --comparison-pair conduct_bounded,route_once \
             --max-workflow-depth 5 \
             --max-output-tokens 264 \
             --git-sha "$PROVENANCE_GIT_SHA" \

CHANGELOG.md

@@ -13,6 +13,15 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
 
 ### Changed
 
+- The psychometric held-out harness now requires a declared resample count,
+  percentile coverage, and seed for paired intervals. Hidden 2,000-sample
+  95% defaults are removed. The script entry still writes 2,000, 0.95, and
+  seed 568 as this run's choices.
+- NIM paired comparisons now require a declared resample count, percentile
+  coverage, and policy-pair list. Hidden 2,000-resample 95% defaults and the
+  baked-in conduct/route/cheapest/hindsight subset are removed. Report schema
+  4.0.0 records those declarations; older reports must be regenerated.
+
 - NIM benchmark workflow depth and per-call output-token budgets are required
   declarations (`max_workflow_depth` / `max_output_tokens`); omitted values fail
   closed instead of inventing five steps or 264 tokens (ADR 0043).

docs/doctoring/nim-benchmark-evidence-grade.md

@@ -211,6 +211,90 @@ Current-head full/hosted verification, independent review and protected release
 remain separate gates. This corrects report provenance and cohort isolation;
 it does not establish probability sampling, model accuracy or faster decisions.
 
+### Declared paired-bootstrap coverage (2026-09-07, proposed)
+
+The previous paired comparison used a hidden 2,000-resample 95% percentile
+interval and a hard-coded policy subset (`conduct_bounded` versus `route_once`,
+optional cheapest versus `route_once`, and hindsight pairs when a unique
+winner existed). Those choices were not reconstructible as operator
+declarations. Report version 4 requires `bootstrap_resample_count`,
+`confidence_level`, and `comparison_pairs` in provenance. The method name is
+`paired_bootstrap_percentile`. A coverage that cannot be represented with the
+resample count fails closed. CLI and workflow flags carry the same
+declarations; 2,000 and 0.95 in those files are run choices, not code
+defaults. Hindsight identity remains a measurement field; comparing against it
+requires an explicit pair.
+
+Efron (1979) grounds resampling observed task units. Efron and Tibshirani
+(1993) ground the percentile interval and treat *B* as Monte Carlo precision.
+This slice does not add a statistical dependency or change production
+route/conduct defaults. Token and workflow-depth budgets are the successor
+slice recorded below.
+
+```mermaid
+sequenceDiagram
+    participant Operator as Run declaration
+    participant Compare as Paired comparison
+    participant Interval as Percentile interval
+    participant Report as Schema 4 report
+    Operator->>Compare: Policy pairs, resample count, coverage, seed
+    Compare->>Compare: Fail closed on missing or invalid declarations
+    Compare->>Interval: Shared locked-task differences
+    Interval->>Report: Mean difference and declared-coverage interval
+    Note over Operator,Report: Unobserved pairs are omitted; production gates still apply
+```
+
+### Declared workflow depth and token budgets (2026-09-07, proposed)
+
+The previous equal-budget cell used a hidden five-step workflow and a
+264-token per-call output cap. Those numbers allocated evaluation compute
+and shaped request planning without an operator declaration. Planning,
+evaluation, CLI, and provenance now require positive integer declarations.
+`None` is a fail-closed sentinel. The equal cell token budget is the product
+of the declared output cap and the declared workflow depth. Five and 264 in
+the workflow YAML are run choices, not code defaults.
+
+This slice does not change production route/conduct defaults. Held-out
+bootstrap coverage is the successor slice recorded below.
+
+```mermaid
+sequenceDiagram
+    participant Operator as Run declaration
+    participant Plan as Request plan
+    participant Cell as Equal-budget cell
+    participant Report as Schema 4 report
+    Operator->>Plan: Workflow depth and output-token cap
+    Plan->>Plan: Fail closed on missing or non-positive declarations
+    Plan->>Cell: Equal call envelope and token product
+    Cell->>Report: Configured budget and observed usage
+    Note over Operator,Report: Production route and conduct defaults stay locked
+```
+
+### Declared held-out bootstrap coverage (2026-09-07, proposed)
+
+The psychometric held-out harness used a hidden 2,000-sample 95% paired
+interval. Those numbers chose Monte Carlo precision without an operator
+declaration. `_paired_bootstrap_mean_ci`, `run_benchmark`, and the adaptive
+calibration helper now require resample count, exclusive-unit-interval
+coverage, and seed. The existing floor/ceil index mapping is kept so
+synthetic fixtures stay comparable. The script entry writes 2,000, 0.95, and
+seed 568 as this run's choices. The report records those fields.
+
+Efron (1979) grounds resampling observed units. This slice does not change
+production route/conduct defaults. Other harness sample sizes remain later
+work.
+
+```mermaid
+sequenceDiagram
+    participant Operator as Run declaration
+    participant Interval as Held-out interval
+    participant Report as Held-out report
+    Operator->>Interval: Resample count, coverage, seed
+    Interval->>Interval: Fail closed on missing or non-representable declarations
+    Interval->>Report: Mean difference and declared-coverage interval
+    Note over Operator,Report: Production route and conduct defaults stay locked
+```
+
 ### Failure-inclusive comparison repair (2026-09-05, proposed)
 
 The previous paired comparison selected only jointly successful cells even
@@ -486,6 +570,9 @@ Chen, L., Zaharia, M., & Zou, J. (2023). FrugalGPT: How to use large language
 models while reducing cost and improving performance. *arXiv*.
 https://doi.org/10.48550/arXiv.2305.05176
 
+Efron, B., & Tibshirani, R. J. (1993). *An introduction to the bootstrap*.
+Chapman & Hall. https://doi.org/10.1201/9780429246593
+
 Efron, B. (1979). Bootstrap methods: Another look at the jackknife.
 *The Annals of Statistics, 7*(1), 1–26.
 https://doi.org/10.1214/aos/1176344552

docs/nim_benchmark.md

@@ -19,15 +19,24 @@ The detailed engineering and evidence record is
 python -m contextual_orchestrator nim-benchmark --dry-run \
   --pricing-scenario examples/nim_pricing_scenario.json \
   --output-dir benchmark_artifacts \
+  --bootstrap-resample-count 2000 \
+  --confidence-level 0.95 \
+  --comparison-pair conduct_bounded,route_once \
   --max-workflow-depth 5 \
   --max-output-tokens 264
 
 # Live CI run: the workflow injects NVIDIA_NIM_API_KEY only into the live step.
 # The process bootstraps it into the credential registry and runtime access
-# resolves the credential by name.
+# resolves the credential by name. Resample count, coverage, comparison pairs,
+# workflow depth, and output-token cap are required declarations; the values
+# below are this run's choices, not hidden code defaults.
 python -m contextual_orchestrator nim-benchmark \
   --max-total-requests 2000 \
+  --max-workflow-depth 5 \
   --max-output-tokens 264 \
+  --bootstrap-resample-count 2000 \
+  --confidence-level 0.95 \
+  --comparison-pair conduct_bounded,route_once \
   --git-sha "$GITHUB_SHA" \
   --workflow-run-id "$GITHUB_RUN_ID"
 ```
@@ -210,18 +219,27 @@ failure record retains `task_score: null`; zero delivery reward is not an
 estimate of an unobserved answer's correctness or a psychometric response.
 
 Each comparison reports A-minus-B mean delivered-score and elapsed-time
-differences with paired 95% bootstrap intervals, successful outcome counts on
-the shared tasks, and unmatched task counts. Elapsed time ends at the recorded
-terminal outcome, including a failure or timeout. A fast failure is therefore
-visible alongside its zero delivery reward; lower elapsed time alone is not an
-improvement in service. The intervals condition on the common task set and the
-selected policies, including the explicitly labelled hindsight worker.
-When direct workers tie for the highest quality, no unique hindsight worker
-is selected and its comparisons are omitted. The observations and other policy
-comparisons remain available; model names never break a quality tie.
+differences with paired percentile bootstrap intervals, successful outcome
+counts on the shared tasks, and unmatched task counts. Coverage, resample
+count, and the compared policy pairs are declared in provenance; they are not
+hidden 2,000-resample or 95% defaults and not a baked-in conduct/route subset.
+Elapsed time ends at the recorded terminal outcome, including a failure or
+timeout. A fast failure is therefore visible alongside its zero delivery
+reward; lower elapsed time alone is not an improvement in service. The
+intervals condition on the common task set and the declared policy pairs.
+Hindsight worker identity remains a separate measurement field; comparing
+against it requires an explicit pair declaration.
+
+### Comparison report version 4
+
+Version 4 requires declared `bootstrap_resample_count`, `confidence_level`,
+and `comparison_pairs` in report provenance. The percentile method name is
+`paired_bootstrap_percentile`; coverage is a separate numeric field. A
+declared coverage that cannot be represented with the resample count fails
+closed. Older reports must be regenerated.
 
 Reports using version 1 compared only jointly successful tasks. Their values
-must not be pooled with version 2 or 3, and the validator rejects old schemas.
+must not be pooled with version 2, 3, or 4, and the validator rejects old schemas.
 Version 2 observations require their independently preserved evaluation plan
 before migration; do not infer a complete plan from surviving observations.
 No task-count or completion-fraction threshold authorizes production promotion.

docs/papers/README.md

@@ -222,6 +222,15 @@ verified observed-task evidence and the protected release process.
   unobserved psychometric response. Mean intervals do not establish p95
   performance. Citation and summary only; redistribution was not established.
 
+- Efron, B., & Tibshirani, R. J. (1993). *An introduction to the bootstrap*.
+  Chapman & Hall. https://doi.org/10.1201/9780429246593
+  Grounds the percentile interval: ordered bootstrap replications, with
+  coverage α taken from the operator declaration rather than a hidden 95%
+  constant, and resample count B as a Monte Carlo precision declaration.
+  NIM report version 4 records both values in provenance and fails closed
+  when coverage cannot be represented with B. Citation and summary only;
+  redistribution was not established.
+
 - **Holistic Evaluation of Language Models (HELM)** — Percy Liang, Rishi
   Bommasani, Tony Lee, et al. arXiv:2211.09110, 2022 (TMLR 2023).
   `helm-holistic-evaluation-2211.09110.pdf`

docs/planning/adrs/0042-declared-paired-bootstrap-coverage.md

@@ -0,0 +1,83 @@
+---
+id: "0042"
+title: "Declare paired-bootstrap coverage and comparison pairs"
+status: proposed
+proposed_date: "2026-09-07"
+deciders:
+  - "repository maintainer"
+affected_components:
+  - "contextual_orchestrator/nim_benchmark.py"
+related:
+  - path: "docs/planning/adrs/0034-anti-heuristic-routing-evidence.md"
+    relation: extends
+success_criteria:
+  - metric: "no hidden interval default"
+    target: "omitted resample_count or confidence_level fails closed"
+    source: "tests/test_nim_benchmark.py::test_paired_bootstrap_rejects_undeclared_or_invalid_coverage"
+  - metric: "no baked-in policy subset"
+    target: "omitted comparison_pairs fails closed; unobserved pairs are skipped, not invented"
+    source: "tests/test_nim_benchmark.py::test_paired_policy_comparisons_reject_undeclared_or_invalid_pairs"
+---
+
+# ADR 0042: Declare paired-bootstrap coverage and comparison pairs
+
+- Status: Proposed
+- Date: 2026-09-07
+- Doctoring record: [`docs/doctoring/nim-benchmark-evidence-grade.md`](../../doctoring/nim-benchmark-evidence-grade.md)
+
+## Product requirement
+
+A buyer comparing `route` and `conduct` needs to know which policies were
+compared and how the uncertainty interval was formed. A hidden 2,000-resample
+95% interval, or a hard-coded conduct/route/cheapest/hindsight subset, cannot
+be defended as measurement design. The operator must declare resample count,
+percentile coverage, and policy pairs for each run.
+
+## Decision
+
+In the context of NIM paired policy evidence, facing hidden 2,000-resample
+95% intervals and a baked-in comparison subset, we chose required declarations
+and against restoring those constants or auto-comparing every observed pair,
+to keep the estimand explicit, accepting that a run without flags fails closed
+and that undeclared hindsight comparisons are omitted.
+
+`paired_bootstrap_mean_difference` takes keyword-only `resample_count`,
+`confidence_level`, and `seed`. `None` is a fail-closed sentinel, not a
+statistical default. Coverage must be a finite exclusive unit interval. The
+percentile indices follow Efron and Tibshirani (1993); if the lower and upper
+indices collapse, the declaration cannot be represented and the run fails.
+The method name is `paired_bootstrap_percentile`; coverage is a numeric field.
+
+`paired_policy_comparisons` takes declared `comparison_pairs`. Empty, malformed,
+duplicate, or identical-name pairs fail closed. Unobserved or disjoint pairs
+are omitted rather than imputed. Hindsight identity remains a separate
+measurement; comparing against it requires an explicit pair.
+
+Report schema 4.0.0 records the declarations in provenance. Production
+route/conduct defaults stay locked. Token and workflow-depth budgets are the
+successor slice in ADR 0043.
+
+## Alternatives considered
+
+- Keep 2,000 and 0.95 as code defaults. Rejected: they hide Monte Carlo
+  precision and coverage from the report consumer.
+- Auto-compare every observed locked policy pair. Rejected for this slice:
+  exhaustive pairing is a different estimand and would change dry-run reports
+  without an operator declaration. It can be declared later as an explicit
+  pair list.
+- Adopt RankWeave's released comparison API. Rejected: v0.18.0 does not accept
+  generic response times or a paired percentile of mean differences.
+
+## Consequences
+
+Positive: interval coverage and compared policies are reconstructible from the
+report and the workflow flags.
+
+Negative: existing CLI, workflow, and library callers must pass the
+declarations; schema 3 reports cannot be reused.
+
+## Remaining work
+
+Token and workflow-depth budgets move to ADR 0043. Held-out bootstrap
+coverage moves to ADR 0044. This ADR is Proposed until independent review
+and protected delivery.

docs/planning/adrs/0043-declared-workflow-token-budgets.md

@@ -10,6 +10,8 @@ affected_components:
 related:
   - path: "docs/planning/adrs/0041-generalize-models-dev-cost-classification.md"
     relation: related
+  - path: "docs/planning/adrs/0044-declared-heldout-bootstrap-coverage.md"
+    relation: followed_by
 success_criteria:
   - metric: "no hidden workflow envelope"
     target: "omitted maximum_calls or max_workflow_depth fails closed"
@@ -51,8 +53,8 @@ Workflow YAML and tests may still write 5 and 264 as this run's choices.
 
 Report schema stays 4.0.0; those fields already exist in provenance and are
 now validated as declarations. Production route/conduct defaults stay locked.
-The psychometric held-out harness's 2,000-sample 95% interval remains a later
-slice.
+The psychometric held-out harness's 2,000-sample 95% interval is the
+successor slice in ADR 0044.
 
 ## Alternatives considered
 
@@ -74,5 +76,5 @@ declarations; omitting them fails closed.
 
 ## Remaining work
 
-The psychometric held-out harness still uses a 2,000-sample 95% interval.
-This ADR is Proposed until independent review and protected delivery.
+Held-out bootstrap coverage moves to ADR 0044. This ADR is Proposed until
+independent review and protected delivery.

docs/planning/adrs/0044-declared-heldout-bootstrap-coverage.md

@@ -0,0 +1,70 @@
+---
+id: "0044"
+title: "Declare held-out paired-bootstrap coverage"
+status: proposed
+proposed_date: "2026-09-07"
+deciders:
+  - "repository maintainer"
+affected_components:
+  - "scripts/benchmark_psychometric_heldout.py"
+related:
+  - path: "docs/planning/adrs/0042-declared-paired-bootstrap-coverage.md"
+    relation: extends
+success_criteria:
+  - metric: "no hidden held-out interval default"
+    target: "omitted resample_count, confidence_level, or seed fails closed"
+    source: "tests/test_psychometric_routing.py::test_paired_bootstrap_interval_requires_declared_coverage"
+---
+
+# ADR 0044: Declare held-out paired-bootstrap coverage
+
+- Status: Proposed
+- Date: 2026-09-07
+- Doctoring record: [`docs/doctoring/nim-benchmark-evidence-grade.md`](../../doctoring/nim-benchmark-evidence-grade.md)
+
+## Product requirement
+
+Buyer-facing accuracy and decision-latency intervals on the psychometric
+held-out harness must be reconstructible. A hidden 2,000-sample 95% interval
+is Monte Carlo precision chosen by the repository, not a declared analysis.
+
+## Decision
+
+In the context of the held-out warm-start harness, facing
+`BOOTSTRAP_SAMPLES = 2_000` and a baked-in 95% percentile, we chose required
+declarations and against restoring those constants, to keep coverage explicit,
+accepting that `run_benchmark()` without kwargs fails closed.
+
+`_paired_bootstrap_mean_ci` takes keyword-only `resample_count`,
+`confidence_level`, and `seed`. Percentile indices keep the existing
+floor/ceil mapping so historical synthetic fixtures stay comparable; a
+coverage that cannot be represented with the resample count fails closed.
+The script entry passes 2,000, 0.95, and seed 568 as this run's choices.
+The report records `bootstrap_samples`, `bootstrap_confidence_level`, and
+`bootstrap_seed`.
+
+Production route/conduct defaults stay locked. Nested `*_ci95` JSON key names
+remain a later naming slice. Other harness sample sizes (assignment trials,
+DIF, reliability) are unchanged.
+
+## Alternatives considered
+
+- Keep 2,000 and 0.95 as module constants. Rejected: they hide Monte Carlo
+  precision from the report consumer.
+- Switch to the NIM integer-index formula from ADR 0042. Rejected for this
+  slice: it would move the 2,000-sample lower index and invalidate existing
+  synthetic fixtures without a new estimand.
+
+## Consequences
+
+Positive: held-out interval coverage is reconstructible from the report and
+the script entry.
+
+Negative: library callers of `run_benchmark` and the calibration helper must
+pass the declarations.
+
+## Remaining work
+
+Other repository-authored harness sample sizes stay open. Nested `*_ci95`
+key names still embed 95. This ADR is Proposed until independent review and
+protected delivery.