feat: #162D certification machinery for P3 kind-decoupled candidate - #177
Merged
Conversation
Versioned, digest-pinned certification doctrine (gates G0-G7 frozen before corpus selection), fresh N=100 corpus selection with derived zero-overlap proofs against the spent #162B/#162C corpora, generalized dual-review corpus freeze (Reviewer A over all cases; Reviewer B on the high-consequence union substantive-disagreement queue; operator adjudication; reveal gate), and a deterministic certification runner with paired bootstrap uncertainty and fail-closed terminal decision logic. Automatic admission remains disabled and is always reported INSUFFICIENT_EVIDENCE with ppv=null at zero positives. No production behavior change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #176 (machinery phase).
What this adds
evals/admission/certification/— extends the merged #162C machinery rather than duplicating it:doctrine.py+ committeddoctrine-162d-v1.json): the versioned, digest-pinned pre-run gate freeze required by the ticket. Pins P3candidate-kind-decoupled-v1(declaration digest + #162C freeze digest), current/P0 as controls, all numerical gates G0–G7 as literals cross-checked at load, the predeclared paired-bootstrap uncertainty method (10,000 resamples, fixed deterministic seed, percentile 95%), corpus doctrine (N=100, both spent corpora excluded BEFORE selection), review/adjudication workflow, unknown-signal rules, terminal statuses, authorization scope, and fail-closed events.load_doctrine()fails closed on digest/gate/candidate/freeze drift.select.py): fresh N=100 selection gated on the doctrine being loadable; #162B dev + #162C holdout removed from the pool before selection (the #162C fail-closed lesson); derived zero-overlap proofs; population shortfall records a pre-declared INCONCLUSIVE run; digest-pinned manifest;verify_certification_freeze_gatebinds every later stage.review.py): Reviewer A ledger expansion (generalized to N=100, #162C blindness attestations), derived Reviewer B queue = high-consequence ∪ substantive disagreement, blind B packet builder, andfinalize_certification_corpuswhich refuses unresolved substantive disagreement / missing B coverage / any policy field, preserves raw reviewer records separately from adjudicated labels, reports inter-rater agreement by field, and must run before any reveal (assert_certification_reveal_gate).runner.py): pure deterministic replay (no DB/network; run twice + byte-compare in the CLI). G0 exact P0/current parity on storage + automatic admission; G1 paired +5pp storage accuracy with bootstrap CI (passing point estimate + too-wide interval → INCONCLUSIVE, never silent pass); G2 15% relative held-back reduction (current=0 → not computable → INCONCLUSIVE path, never vacuous pass); G3 zero high-consequence reject→retain and required-review bypass; G4 ≤35% review rate with exactly-35-passes boundary + 100% high-consequence required-review coverage; G5 zero high-consequence false governed/startup eligibility; G6 automatic admission locked disabled — zero positives → ppv=null + INSUFFICIENT_EVIDENCE, unexpected positive fails closed (run invalid for scope); G7 unknown/unavailable preserved, labels structurally absent from candidate signatures. Terminal decision is exactly one of CERTIFIED_STORAGE_POLICY / NOT_CERTIFIED / INCONCLUSIVE.__main__.py): write-doctrine / select-corpus / expand-reviewer-a / finalize-corpus / run, all private outputs outside Git (enforced), 0600, opaque errors.Test matrix (36 tests,
tests/test_certification_162d.py)Covers the ticket's required tests 1–26: freeze-before-membership, digest drift fail-closed (both artifact-tamper and gate-value-tamper paths), N=100 exact, zero overlap detection, policy-field rejection in ledgers, label/candidate isolation, +5pp and 15% boundary behavior (exact-boundary pass, one-below fail), current-held-back=0 vacuity, reject→retain / review-bypass / false-governed / false-startup hard failures, 35%/36% review boundary, high-consequence required-review miss, ppv-null + INSUFFICIENT_EVIDENCE, unexpected-positive fail-closed, unknown preservation, reviewer-record separation, B-queue derivation, bootstrap determinism, private-writer repo rejection, per-case-free public projection, pure-runner proof, production import isolation, runner-digest glob stability (baselines byte-valid), and reveal-gate membership.
Verification
pytest tests/test_certification_162d.py— 36 passedtest_admission_eval_contract,test_admission_shadow_162c,test_holdout_adjudication_162c,test_holdout_reviewer_provenance_162c) — 108 passed totalruff checkclean;mypy --strictclean (engram/76 files,evals/admission/34 files)145d82831dc3df01ef2472923aacba94ceff8ed33a4bd4ee90a6eacd9142e7d8, code_sha de55483 (merged #162C head)Scope notes
engram/untouched.