Skip to content

feat: #162D certification machinery for P3 kind-decoupled candidate - #177

Merged
ezutfen merged 1 commit into
mainfrom
162d-certification
Sep 5, 2026
Merged

feat: #162D certification machinery for P3 kind-decoupled candidate#177
ezutfen merged 1 commit into
mainfrom
162d-certification

Conversation

@ezutfen

@ezutfen ezutfen commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Closes #176 (machinery phase).

What this adds

evals/admission/certification/ — extends the merged #162C machinery rather than duplicating it:

  • Doctrine (doctrine.py + committed doctrine-162d-v1.json): the versioned, digest-pinned pre-run gate freeze required by the ticket. Pins P3 candidate-kind-decoupled-v1 (declaration digest + #162C freeze digest), current/P0 as controls, all numerical gates G0–G7 as literals cross-checked at load, the predeclared paired-bootstrap uncertainty method (10,000 resamples, fixed deterministic seed, percentile 95%), corpus doctrine (N=100, both spent corpora excluded BEFORE selection), review/adjudication workflow, unknown-signal rules, terminal statuses, authorization scope, and fail-closed events. load_doctrine() fails closed on digest/gate/candidate/freeze drift.
  • Corpus selection (select.py): fresh N=100 selection gated on the doctrine being loadable; #162B dev + #162C holdout removed from the pool before selection (the #162C fail-closed lesson); derived zero-overlap proofs; population shortfall records a pre-declared INCONCLUSIVE run; digest-pinned manifest; verify_certification_freeze_gate binds every later stage.
  • Dual-review freeze (review.py): Reviewer A ledger expansion (generalized to N=100, #162C blindness attestations), derived Reviewer B queue = high-consequence ∪ substantive disagreement, blind B packet builder, and finalize_certification_corpus which refuses unresolved substantive disagreement / missing B coverage / any policy field, preserves raw reviewer records separately from adjudicated labels, reports inter-rater agreement by field, and must run before any reveal (assert_certification_reveal_gate).
  • Runner (runner.py): pure deterministic replay (no DB/network; run twice + byte-compare in the CLI). G0 exact P0/current parity on storage + automatic admission; G1 paired +5pp storage accuracy with bootstrap CI (passing point estimate + too-wide interval → INCONCLUSIVE, never silent pass); G2 15% relative held-back reduction (current=0 → not computable → INCONCLUSIVE path, never vacuous pass); G3 zero high-consequence reject→retain and required-review bypass; G4 ≤35% review rate with exactly-35-passes boundary + 100% high-consequence required-review coverage; G5 zero high-consequence false governed/startup eligibility; G6 automatic admission locked disabled — zero positives → ppv=null + INSUFFICIENT_EVIDENCE, unexpected positive fails closed (run invalid for scope); G7 unknown/unavailable preserved, labels structurally absent from candidate signatures. Terminal decision is exactly one of CERTIFIED_STORAGE_POLICY / NOT_CERTIFIED / INCONCLUSIVE.
  • CLI (__main__.py): write-doctrine / select-corpus / expand-reviewer-a / finalize-corpus / run, all private outputs outside Git (enforced), 0600, opaque errors.

Test matrix (36 tests, tests/test_certification_162d.py)

Covers the ticket's required tests 1–26: freeze-before-membership, digest drift fail-closed (both artifact-tamper and gate-value-tamper paths), N=100 exact, zero overlap detection, policy-field rejection in ledgers, label/candidate isolation, +5pp and 15% boundary behavior (exact-boundary pass, one-below fail), current-held-back=0 vacuity, reject→retain / review-bypass / false-governed / false-startup hard failures, 35%/36% review boundary, high-consequence required-review miss, ppv-null + INSUFFICIENT_EVIDENCE, unexpected-positive fail-closed, unknown preservation, reviewer-record separation, B-queue derivation, bootstrap determinism, private-writer repo rejection, per-case-free public projection, pure-runner proof, production import isolation, runner-digest glob stability (baselines byte-valid), and reveal-gate membership.

Verification

  • pytest tests/test_certification_162d.py — 36 passed
  • Full admission suites (test_admission_eval_contract, test_admission_shadow_162c, test_holdout_adjudication_162c, test_holdout_reviewer_provenance_162c) — 108 passed total
  • ruff check clean; mypy --strict clean (engram/ 76 files, evals/admission/ 34 files)
  • Doctrine artifact generated and load-verified: digest 145d82831dc3df01ef2472923aacba94ceff8ed33a4bd4ee90a6eacd9142e7d8, code_sha de55483 (merged #162C head)

Scope notes

  • No production behavior change; engram/ untouched.
  • Operational run (fresh snapshot on engram01 → select-corpus → blind review → adjudication freeze → run) proceeds after merge, exactly as #162C did.
  • Local pre-commit hook bypassed per repo convention (known-flaky local test_usage_report.py failures, documented in #162C PR); authoritative checks are the exact-head Compose/PostgreSQL CI paths.

Versioned, digest-pinned certification doctrine (gates G0-G7 frozen before
corpus selection), fresh N=100 corpus selection with derived zero-overlap
proofs against the spent #162B/#162C corpora, generalized dual-review corpus
freeze (Reviewer A over all cases; Reviewer B on the high-consequence union
substantive-disagreement queue; operator adjudication; reveal gate), and a
deterministic certification runner with paired bootstrap uncertainty and
fail-closed terminal decision logic. Automatic admission remains disabled
and is always reported INSUFFICIENT_EVIDENCE with ppv=null at zero
positives. No production behavior change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENG-CALIBRATION-001D — P3 certification gates and fresh blind certification run

1 participant