Skip to content

Production build: parallel certificates, a process store by content identity, certificates named by their root's program - #558

Open
afflom wants to merge 8 commits into
claude/evaluation-prooffrom
claude/export-parallel-store-2
Open

afflom wants to merge 8 commits into
claude/evaluation-prooffrom
claude/export-parallel-store-2

Conversation

@afflom

@afflom afflom commented Nov 16, 2026

Copy link
Copy Markdown
Member

Stacked on claude/evaluation-proof (2bf5c1e, the revision uorc pins today). This PR adds two commits.

Why

The uorc client export took 196 min on a 4 vCPU / 16 GB host. Of that, 183 min went to nine preservation certificates, which compiled one after another. Every certificate A imported all 78 project modules, and every certificate was named after the project's semantic id, so any change re-proved all nine.

What changes

  • Parallel certificate stage (SPEC 22.11 item 6). The environment and the certificates compile in the topological frontiers of their import headers, at LEXLEAN_VERIFY_WIDTH and within LEXLEAN_VERIFY_MEMORY_KIB, using the existing pool::run_ordered_with and hints. Each leanchecker replay runs in its compile's slot. A certificate inside another's namespace waits for that one's replay, because leanchecker replays a module prefix. Records are pushed in canonical order, so the attestation is byte-identical at any width.
  • Process store (LEXLEAN_VERIFY_STORE, new SPEC 22.13). The key is the tool, the Lean executable digest, the normalized argv, and the Merkle κ of the module text and every import it reaches. The header parser handles module, public import, meta import and import all. A replay's identity covers its prefix. Outputs are stored by stem, together with Lean's .olean.server, .olean.private and .ir companions. Every restored file is digest-checked, and an entry is restored only for the same executable and exit 0. A miss means the process runs.
  • Certificates named by their root's lowered program (SPEC 17.17). The name is LexLeanPreserve.C<sha256("lexlean/certificate-name/1" ‖ program)[..32]>.R<i>. Certificate A imports only the user modules that own a constant its terms name.
  • A reused telemetry event is added. SPEC 23.1 lists the five operational variables.

Measured

examples/production-coverage publishes the same attestation in all three runs:

run time
width 1 683 s
width 2, cold store 387 s
width 2, warm store 12 s
after a one-module edit 24 s (5 of 298 certificate processes re-ran)

uorc client export:

run time certificates stage
before 196 min 183 min
cold store 153 min
warm store 13.5 min 33 s

The warm export published the same result id as the cold one. The rest of the warm time goes to certificate generation (344 s) and extraction (253 s), which the store does not cover yet.

Unit tests cover the import-header parser and Merkle propagation. uorc pins this head in UOR-Foundation/compression#186.

Porting to release/v0.5 waits on the staged-certificates and evaluation-proof chain landing there. A cherry-pick conflicts only in verify/mod.rs's certificate stage, which that chain rewrites.


Generated by Claude Code

…rocess store by content identity, certificates named by their root's program

The certificate stage compiled the preservation environment and every
certificate one after another on one core (the uorc client: 9 certificates,
about 1 GB of Lean, 2 h of a 2.3 h export). They now run in the topological
frontiers of their import headers at the verification width and within the
memory budget, each replay in its compilation's slot, records in the
stage's canonical order (SPEC 22.11 item 6). Certificate labels drop their
content segment, so measured hints carry to the next run.

A process store (LEXLEAN_VERIFY_STORE, SPEC 22.13) reuses an elaboration,
replay, audit member or certificate process whose key (tool, executable
digest, normalized argv, Merkle content identity of the module and every
module its imports reach) an earlier run established; restored files and
records are digest-checked and judged as the stage judges its own, so the
published directory is byte-identical with and without it.

Certificates are named by their root's lowered program instead of the
project's semantic id, and certificate A imports only the user modules its
terms name, so a root a change does not reach keeps its certificates' text
and content identity and is not proved again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…ay; a replay's identity covers its prefix

leanchecker replays a module prefix (SPEC 22.4), so certificate A's replay
checked its root's B and E as well once they compiled beside it (30 min in
place of 2). Certificates B and E now start after their root's A, as at
width 1, and a replay's store key covers every staged module in its
namespace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q

afflom commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

The red checks fall into two groups, and neither comes from a defect in this diff.

  • honesty / model: gate failed: §17.17: crates/lexlean/src/production/certificate.rs:103: an if let with an implicit default. The flagged line is Certificate::released from the staged-certificates commits underneath this PR. I ran cargo xtask validate-model on the base 2bf5c1e on its own and got the same failure.
  • just vv legs / repro: LLC0102: lexlean.lock is stale … the compiler semantics changed since locking. Naming certificates by their root's program changes the compiler's output, so the semantics identity moves as intended. Every lock then has to be relocked and the records regenerated.

This branch is stacked on claude/evaluation-proof, which is not on release/v0.5. The way to land it is to port it onto the release line once the staged-certificates and evaluation-proof chain is there. On release/v0.5, just relock-identity and regenerate.yml handle the identity move. Until then, uorc pins this head (UOR-Foundation/compression#186), the same way it pins 2bf5c1e today.


Generated by Claude Code

… process store

Both are lean processes whose output is a function of their staged source
and the modules it imports; with the store a warm client export spends 8.6 s
in extraction instead of 253 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: the new process store does not yet bind all inputs whose behavior it reuses. The inline finding identifies a reachable stale-success path; unchanged output digests and timing comparisons do not establish this proof boundary. This is a source finding, not a reproduced exploit.

The inspected CI failures are separate: the honesty log flags Certificate::released's implicit default, which is unchanged in base 2bf5c1e; repro reports stale compiler-semantics locks. Those explain the observed failures but do not clear the new store implementation. The seven-file delta has parser/Merkle unit checks; final evidence still needs the applicable cold/warm, invalidation/refusal and output-integrity guarantees before accepting skipped compile/replay/audit work.

Existing owner retains the stacked change, release-line port and coordinated identity products. Resolve the input-binding gap by authenticating the dependencies and effective tool/environment inputs used, or treating unbound inputs as cache misses. Reuse valid evidence and preserve original failures; no new producer, experiment or retry is requested here. This review does not admit deferred #338/#438 scope or waive #391/#468, final-head full gates, actual-tree equality, or STOP constraints.

Comment thread crates/lexlean/src/verify/store.rs Outdated
.iter()
.find_map(|root| std::fs::read(root.join(&relative).as_std_path()).ok())
else {
return Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes());

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Bind external imports and the effective tool environment before reusing success. An import absent from Session.roots gets an identity derived only from its name here. Session::run then keys on that identity, normalized argv and one executable digest, omitting the effective extra_env/Lake wrapper and imported toolchain artifacts. With the same staged sources and Lean/leanchecker executable bytes but changed imported Init/Std .olean files or effective Lake environment, the key can remain unchanged. stored returns the cached ChildRecord without running the process; replay and audit callers pass no outputs, so their success can also be reused without observing the changed dependency. Authenticate the relevant dependency/tool/environment closure, or force a miss where that binding is unavailable.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You were right that the key left these inputs unbound. 8eec295 binds them as follows:

  • External imports. A module the stage does not hold is bound only when the pinned toolchain's lib/lean holds its .olean. Any other module is unbound (None), and so is every module whose imports reach it. A process about an unbound module runs and is not stored. Test: a_module_neither_the_stage_nor_the_toolchain_holds_is_unbound.

  • Toolchain artifacts. Each key now includes a context, and that context contains a tree digest of the toolchain directory. The tree digest covers every file a lean, lake or leanchecker process loads: bin/*, the shared libraries, and every .olean, .olean.server, .olean.private and .ir. It is read whole once per session. If any file can't be read, the result is a miss. Test: the_toolchain_digest_follows_every_file_a_process_loads changes each kind of file in turn and checks the digest moves; a .ilean change leaves it unchanged.

  • The Lake wrapper and environment. The context also includes:

    • the digest of the program started (the lake wrapper or leanchecker);
    • the normalized toolchain bin and working directory;
    • the digests of the working directory's lakefile.lean, lakefile.toml, lake-manifest.json and lean-toolchain, each recorded as absent when missing;
    • the tree digest of its .lake;
    • every extra_env row with its normalized value.

    A process that does not run under the toolchain home is never answered by the store.

SPEC §22.13 now states the context and the unbound rule. The stage still judges a restored record the way it judges its own, and files are still digest-checked on restore. I'm running a cold uorc export with this head now to reseed the store under the new keys. A warm run will follow, and I'll report both times and result ids here.


Generated by Claude Code

@afflom afflom mentioned this pull request Nov 16, 2026
A key's identity now adds the process's context: the digest of the program
it starts, the tree digest of every toolchain file a lean, lake or
leanchecker process loads (executables, shared libraries, .olean with its
companions, .ir), the working directory's Lake configuration and .lake tree,
and the normalized environment rows. A module neither the stage nor the
toolchain's library holds is unbound, and so is everything importing it: such
a process is not answered by the store and is not stored. A context that
cannot be read whole, or a process not under the toolchain's home, is a miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
A and the B and E certificates are made from the lowered program and the
linked modules alone and neither reads the other; A's result is judged first,
as when they were generated one after the other. Generation of the uorc
client's three roots took 344 s with A, B and E one after another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…constructs

Certificate::released names the certificate without data; the named-constant
filter of certificate A names every core node; the Option pair in Update's
evaluation is a zip; the prefix test uses contains.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new executable/toolchain/Lake-context bindings and propagation of unbound imports to cache misses address substantial parts of the original finding. One concrete P1 remains: the tree inventory silently omits symlinked directories, while the import eligibility check follows them. The inline finding describes how that can preserve a cache key across changed imported artifacts. No runtime experiment or exploitation claim is made.

Honesty now passes on eba5. Current CI and repro still fail; those results remain separate from this source finding. Existing owner retains the repair, release-line port, coordinated products and final qualification. Preserve original receipts and the cold/warm/refusal/invalidation evidence requirement; no repeated producer or cache execution is requested.

Comment thread crates/lexlean/src/verify/store.rs Outdated
for entry in std::fs::read_dir(directory.as_std_path()).ok()? {
let entry = entry.ok()?;
let path = Utf8PathBuf::from_path_buf(entry.path()).ok()?;
let kind = entry.file_type().ok()?;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not silently omit symlinked library directories from the cache context. DirEntry::file_type() does not follow symlinks, so a toolchain with lib/lean symlinked to an external library takes the non-directory branch and is omitted because that directory name is not a loaded-file suffix. Later visit() uses library.is_file(), which follows the link and treats Init.olean as bound by this toolchain digest. Changing an artifact under the linked library therefore leaves the context/key unchanged while compile/replay/audit can return cached success. Either authenticate the resolved directory closure with appropriate cycle/error handling or return an unbound/cache-miss result for unsupported symlinks; the traversal and import-eligibility rules must agree.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in c06c769. The traversal and the import eligibility check now follow symbolic links the same way.

  • tree now gets each entry's kind from std::fs::metadata, which follows links. A directory linked in from outside the toolchain is therefore walked and bound, the same as a directory inside it.
  • Each directory is canonicalized before it is read. If the canonical directory was already reached (a link cycle or an alias), the result is None. A link that doesn't resolve also gives None. Both cases are a miss.
  • Test a_linked_library_is_bound_and_a_link_cycle_is_unreadable (unix) covers these cases:
    • lib/lean symlinked to an outside library: Init is bound.
    • Changing Init.olean under the linked directory moves the digest.
    • A back-link cycle returns None.
    • A dangling link returns None.
  • SPEC §22.13 now states the rule.

The pinned toolchain here has only file symlinks (the lib/*.so aliases). Those were already read through their targets, so its digest and the existing keys are unchanged.


Generated by Claude Code

…e do

A library directory linked into the toolchain was skipped by the traversal
while import eligibility followed the link; both now follow it. A link that
does not resolve, or a directory reached twice (a cycle or an alias), makes
the tree unreadable, which is a miss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q

@afflom afflom left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The concrete symlink P1 from review5918813164 is repaired in c06. The traversal now follows metadata through links, tracks canonical directories, and returns None for unresolved links or repeated directories; the existing None path makes these cases cache misses. The specification matches that behavior.

The new Unix unit source covers a linked Init library, a target-byte change moving its digest, and cycle/dangling-link refusal. This review inspected the source; it did not execute the unit or claim a current gate pass. No new actionable defect was found in this one-commit/two-file delta.

Credit this targeted repair alongside the earlier context bindings. Full process-input and normative cold/warm/refusal/invalidation/output-equivalence qualification remains broader than this delta. Existing owner retains final source/evidence disposition, release-line port, coordinated products and exact-head/actual-tree acceptance; no producer or completed check was repeated.

Receipt: /workspace/work/pr558-c06-delta-review-20261116.json; SHA256 77f273ee11f05aa872b6f7cfe5787e789fe8fb2697054c42770521194b44eb72.

afflom commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

atlas (join) on c06c769 fails with R10: uor-atlas: missing oracle expected/verify/audit/LexLeanAudit.A841bf522….LexLeanExample.Atlas.lean. The committed oracles are named LexLeanAudit.Aa04981d2…. The audit module name is the project semantic id (verify/mod.rs:879), and that id folds in compiler_semantics_id_for (link.rs:1229). So any compiler-source change renames these oracles, for the same reason the locks go stale (LLC0102). This is the class already noted above, not a defect of the store or the parallel certificates. The repair is the release line's relock-identity plus regenerated records, which the claude/evaluation-proof base does not carry.


Generated by Claude Code

afflom commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

Store evidence on c06c769, from the uorc client export (78 modules, 3 roots, 4 vCPU / 16 GB, store on a shared network mount):

run wall processes run / restored result id
cold under the new keys (resumed after a host restart cut the first run off at frontier 7) 9249 s 22 / 262 e5f0d61a…
warm, no change 303 s (first read of the mount: 466 s) 3 / 281 e5f0d61a… (same)
  • Cold. The run is bound by the two B certificates, which run one after the other under the MemAvailable gate. R1 B used 8.9 GB peak, with 2071 s to compile and 1970 s for leanchecker.
  • Invalidation. The keys from 3808fac (before the context binding) were all misses under c06c769: the cold run restored only what the cut-off run had stored under the new keys, and nothing from the earlier store. For the same compiler, c06c769 restored everything 1992437 stored, because the symlink change does not alter any key on a toolchain without linked directories.
  • Refusal is covered by the unit tests named in the PR: an unbound module, a toolchain file change, and a link cycle.
  • Warm cost. Certificate generation is 171 s of the 303 s, in-process Rust, and is the next distance.

Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants