[Deep-Sixed]

Adds the #15 item "Full-corpus production verification and sign-off. The baseline is today's bubblewrap-only setup."

stele.verification checks a new sandbox setup against the current baseline on the whole corpus, then records a named person's sign-off on the result. The runbook is docs/verification.md.

Steps

1. Pin the corpus (corpus.py, stele.corpus v1).
   • Every regular file is listed by SHA-256 and size.
   • Everything left out (symlinks, FIFOs, unreadable files) is listed with the reason.
   • The manifest's digest is the corpus id.
   • check() catches changed, missing or added documents.
2. Run the campaign (campaign.py). Every document goes through the baseline lane and each candidate lane; the backend is always forced, never auto-selected.
   • Lanes: bubblewrap, oci-runc, oci-runsc, oci-runc-gpu and wasmtime.
   • Parsers: a script parser, a packaged ML parser, or a Wasm extractor.
   • The input Snapshot the parser read must hash to the manifest entry.
   • Successful runs are sealed in the ledger.
   • Every observation goes to a fsynced JSON-lines journal, including failures with their structured reason and telemetry.
   • The journal is resumable: a rerun skips what is done, and a torn last line is dropped.
3. Build the report (report.py, stele.verification-report v1).
   • Re-checks every sealed record:
     • still SEALED, and the same run, parser and input the journal names;
     • its bundle verifies in the archive;
     • its canonical extraction resolves, where a normalizer exists.
   • Compares each candidate with the baseline, per document: identical, equivalent (under the parser's replay policy), diverged, regression, recovered, consistent_failure or failure_mismatch.
   • Gates: corpus unchanged, coverage, baseline produced output, no errors, records verified, ledger verified, candidates agree, and waivers used. Optional gates cover the baseline failure rate and the median slowdown.
   • Waivers: comparison findings can be waived per document and lane, each with a written reason. Integrity problems cannot be waived.
4. Sign off (signoff.py).
   • The sign-off rebuilds the report from the ledger, journal and corpus and refuses anything that differs, so an edited report with its gates flipped to passed is refused. It also refuses when a gate failed or when the report no longer holds (the chain was rewritten, or a record was invalidated).
   • On success it stores the report in the archive and appends verification.signed_off to the hash-chained event log.
   • check_sign_off later reports the sign-off as stale once any record it covers is invalidated.

CLI: python -m stele.verification {corpus,run,report,sign-off,check,list}.

Tests

tests/test_verification.py has 29 tests:

• the corpus manifest;
• the campaign: resume, torn journal lines, a mismatched journal, not-applicable documents, a document changed mid-campaign;
• every comparison outcome and gate;
• waivers;
• sign-off, including refusals and stale sign-offs;
• real sandboxes: a bubblewrap baseline end to end, an oci-runc candidate against it (3 identical, 1 consistent failure), and the whole CLI.

Local run: the full suite passes, 750 passed. The 29 verification tests also pass with Docker running, including the OCI lane.

Review points

• The journal is trusted only for what the ledger cannot know: failure reasons and timings. Everything about a sealed run is re-checked against the ledger and the archive.
• An orphan record after a crash: if a crash lands between sealing a record and journaling it, the document runs again on resume, and the earlier record stays in the ledger unreferenced by the report.
• Equivalence is only as strong as the parser's comparison policy. Script parsers have none, so any difference is diverged.
• Publishing the anchor is what makes it hold: as with the event log, the printed sign-off anchor (seq:hash) must be published outside the ledger to hold against someone who can rewrite the SQLite file.

🤖 Generated with Claude Code

https://claude.ai/code/session_013P1MkGppjFZfBUKsToHzez

---

Generated by Claude Code

[Deep-Sixed]

Review: I found and fixed one integrity gap, in 0988c1f.

A lane could claim another lane's record. The report trusted the journal for which sealed record a lane produced. It checked only that the record was the named run, of the right parser, over the right document. I rewrote a journal so the candidate's lines cite the baseline's records, and reproduced this:

• a candidate that diverges on every document came out IDENTICAL everywhere;
• the report passed;
• sign_off() accepted it, because its rebuild reads the same journal.

_RecordChecker now also requires, from the ledger:

• record.backend is the backend of the lane that claims the record;
• record.parser_config matches the campaign's parser_config_digest, so a run of the same parser under another config can't stand in;
• no other document or lane claims the same record.

A violation is a records_verified integrity problem, which can't be waived. Two new tests forge the journal in each way, and both fail without the fix. docs/verification.md lists the new checks.

The rest looks sound:

• the sign-off rebuild-and-compare;
• the refusal on a rewritten chain;
• stale detection after invalidation;
• the torn-line handling in the resumable journal;
• non-waivable integrity gates;
• the verification.signed_off event (verify_ledger ignores event kinds it doesn't replay, so it doesn't break ledger verification).

Full suite: 852 passed, 89 skipped. ruff is clean.

---

Generated by Claude Code

---

Generated by Claude Code