Repository navigation
Production build: parallel certificates, a process store by content identity, certificates named by their root's program - #558
Conversation
…rocess store by content identity, certificates named by their root's program The certificate stage compiled the preservation environment and every certificate one after another on one core (the uorc client: 9 certificates, about 1 GB of Lean, 2 h of a 2.3 h export). They now run in the topological frontiers of their import headers at the verification width and within the memory budget, each replay in its compilation's slot, records in the stage's canonical order (SPEC 22.11 item 6). Certificate labels drop their content segment, so measured hints carry to the next run. A process store (LEXLEAN_VERIFY_STORE, SPEC 22.13) reuses an elaboration, replay, audit member or certificate process whose key (tool, executable digest, normalized argv, Merkle content identity of the module and every module its imports reach) an earlier run established; restored files and records are digest-checked and judged as the stage judges its own, so the published directory is byte-identical with and without it. Certificates are named by their root's lowered program instead of the project's semantic id, and certificate A imports only the user modules its terms name, so a root a change does not reach keeps its certificates' text and content identity and is not proved again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…ay; a replay's identity covers its prefix leanchecker replays a module prefix (SPEC 22.4), so certificate A's replay checked its root's B and E as well once they compiled beside it (30 min in place of 2). Certificates B and E now start after their root's A, as at width 1, and a replay's store key covers every staged module in its namespace. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
|
The red checks fall into two groups, and neither comes from a defect in this diff.
This branch is stacked on Generated by Claude Code |
… process store Both are lean processes whose output is a function of their staged source and the modules it imports; with the store a warm client export spends 8.6 s in extraction instead of 253 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
afflom
left a comment
There was a problem hiding this comment.
P1: the new process store does not yet bind all inputs whose behavior it reuses. The inline finding identifies a reachable stale-success path; unchanged output digests and timing comparisons do not establish this proof boundary. This is a source finding, not a reproduced exploit.
The inspected CI failures are separate: the honesty log flags Certificate::released's implicit default, which is unchanged in base 2bf5c1e; repro reports stale compiler-semantics locks. Those explain the observed failures but do not clear the new store implementation. The seven-file delta has parser/Merkle unit checks; final evidence still needs the applicable cold/warm, invalidation/refusal and output-integrity guarantees before accepting skipped compile/replay/audit work.
Existing owner retains the stacked change, release-line port and coordinated identity products. Resolve the input-binding gap by authenticating the dependencies and effective tool/environment inputs used, or treating unbound inputs as cache misses. Reuse valid evidence and preserve original failures; no new producer, experiment or retry is requested here. This review does not admit deferred #338/#438 scope or waive #391/#468, final-head full gates, actual-tree equality, or STOP constraints.
| .iter() | ||
| .find_map(|root| std::fs::read(root.join(&relative).as_std_path()).ok()) | ||
| else { | ||
| return Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes()); |
There was a problem hiding this comment.
[P1] Bind external imports and the effective tool environment before reusing success. An import absent from Session.roots gets an identity derived only from its name here. Session::run then keys on that identity, normalized argv and one executable digest, omitting the effective extra_env/Lake wrapper and imported toolchain artifacts. With the same staged sources and Lean/leanchecker executable bytes but changed imported Init/Std .olean files or effective Lake environment, the key can remain unchanged. stored returns the cached ChildRecord without running the process; replay and audit callers pass no outputs, so their success can also be reused without observing the changed dependency. Authenticate the relevant dependency/tool/environment closure, or force a miss where that binding is unavailable.
There was a problem hiding this comment.
You were right that the key left these inputs unbound. 8eec295 binds them as follows:
-
External imports. A module the stage does not hold is bound only when the pinned toolchain's
lib/leanholds its.olean. Any other module is unbound (None), and so is every module whose imports reach it. A process about an unbound module runs and is not stored. Test:a_module_neither_the_stage_nor_the_toolchain_holds_is_unbound. -
Toolchain artifacts. Each key now includes a context, and that context contains a tree digest of the toolchain directory. The tree digest covers every file a
lean,lakeorleancheckerprocess loads:bin/*, the shared libraries, and every.olean,.olean.server,.olean.privateand.ir. It is read whole once per session. If any file can't be read, the result is a miss. Test:the_toolchain_digest_follows_every_file_a_process_loadschanges each kind of file in turn and checks the digest moves; a.ileanchange leaves it unchanged. -
The Lake wrapper and environment. The context also includes:
- the digest of the program started (the
lakewrapper orleanchecker); - the normalized toolchain
binand working directory; - the digests of the working directory's
lakefile.lean,lakefile.toml,lake-manifest.jsonandlean-toolchain, each recorded as absent when missing; - the tree digest of its
.lake; - every
extra_envrow with its normalized value.
A process that does not run under the toolchain home is never answered by the store.
- the digest of the program started (the
SPEC §22.13 now states the context and the unbound rule. The stage still judges a restored record the way it judges its own, and files are still digest-checked on restore. I'm running a cold uorc export with this head now to reseed the store under the new keys. A warm run will follow, and I'll report both times and result ids here.
Generated by Claude Code
A key's identity now adds the process's context: the digest of the program it starts, the tree digest of every toolchain file a lean, lake or leanchecker process loads (executables, shared libraries, .olean with its companions, .ir), the working directory's Lake configuration and .lake tree, and the normalized environment rows. A module neither the stage nor the toolchain's library holds is unbound, and so is everything importing it: such a process is not answered by the store and is not stored. A context that cannot be read whole, or a process not under the toolchain's home, is a miss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
A and the B and E certificates are made from the lowered program and the linked modules alone and neither reads the other; A's result is judged first, as when they were generated one after the other. Generation of the uorc client's three roots took 344 s with A, B and E one after another. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
…constructs Certificate::released names the certificate without data; the named-constant filter of certificate A names every core node; the Option pair in Update's evaluation is a zip; the prefix test uses contains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
afflom
left a comment
There was a problem hiding this comment.
The new executable/toolchain/Lake-context bindings and propagation of unbound imports to cache misses address substantial parts of the original finding. One concrete P1 remains: the tree inventory silently omits symlinked directories, while the import eligibility check follows them. The inline finding describes how that can preserve a cache key across changed imported artifacts. No runtime experiment or exploitation claim is made.
Honesty now passes on eba5. Current CI and repro still fail; those results remain separate from this source finding. Existing owner retains the repair, release-line port, coordinated products and final qualification. Preserve original receipts and the cold/warm/refusal/invalidation evidence requirement; no repeated producer or cache execution is requested.
| for entry in std::fs::read_dir(directory.as_std_path()).ok()? { | ||
| let entry = entry.ok()?; | ||
| let path = Utf8PathBuf::from_path_buf(entry.path()).ok()?; | ||
| let kind = entry.file_type().ok()?; |
There was a problem hiding this comment.
[P1] Do not silently omit symlinked library directories from the cache context. DirEntry::file_type() does not follow symlinks, so a toolchain with lib/lean symlinked to an external library takes the non-directory branch and is omitted because that directory name is not a loaded-file suffix. Later visit() uses library.is_file(), which follows the link and treats Init.olean as bound by this toolchain digest. Changing an artifact under the linked library therefore leaves the context/key unchanged while compile/replay/audit can return cached success. Either authenticate the resolved directory closure with appropriate cycle/error handling or return an unbound/cache-miss result for unsupported symlinks; the traversal and import-eligibility rules must agree.
There was a problem hiding this comment.
Fixed in c06c769. The traversal and the import eligibility check now follow symbolic links the same way.
treenow gets each entry's kind fromstd::fs::metadata, which follows links. A directory linked in from outside the toolchain is therefore walked and bound, the same as a directory inside it.- Each directory is canonicalized before it is read. If the canonical directory was already reached (a link cycle or an alias), the result is
None. A link that doesn't resolve also givesNone. Both cases are a miss. - Test
a_linked_library_is_bound_and_a_link_cycle_is_unreadable(unix) covers these cases:lib/leansymlinked to an outside library:Initis bound.- Changing
Init.oleanunder the linked directory moves the digest. - A back-link cycle returns
None. - A dangling link returns
None.
- SPEC §22.13 now states the rule.
The pinned toolchain here has only file symlinks (the lib/*.so aliases). Those were already read through their targets, so its digest and the existing keys are unchanged.
Generated by Claude Code
…e do A library directory linked into the toolchain was skipped by the traversal while import eligibility followed the link; both now follow it. A link that does not resolve, or a directory reached twice (a cycle or an alias), makes the tree unreadable, which is a miss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q
afflom
left a comment
There was a problem hiding this comment.
The concrete symlink P1 from review5918813164 is repaired in c06. The traversal now follows metadata through links, tracks canonical directories, and returns None for unresolved links or repeated directories; the existing None path makes these cases cache misses. The specification matches that behavior.
The new Unix unit source covers a linked Init library, a target-byte change moving its digest, and cycle/dangling-link refusal. This review inspected the source; it did not execute the unit or claim a current gate pass. No new actionable defect was found in this one-commit/two-file delta.
Credit this targeted repair alongside the earlier context bindings. Full process-input and normative cold/warm/refusal/invalidation/output-equivalence qualification remains broader than this delta. Existing owner retains final source/evidence disposition, release-line port, coordinated products and exact-head/actual-tree acceptance; no producer or completed check was repeated.
Receipt: /workspace/work/pr558-c06-delta-review-20261116.json; SHA256 77f273ee11f05aa872b6f7cfe5787e789fe8fb2697054c42770521194b44eb72.
|
Generated by Claude Code |
|
Store evidence on c06c769, from the uorc client export (78 modules, 3 roots, 4 vCPU / 16 GB, store on a shared network mount):
Generated by Claude Code |
Stacked on
claude/evaluation-proof(2bf5c1e, the revision uorc pins today). This PR adds two commits.Why
The uorc client export took 196 min on a 4 vCPU / 16 GB host. Of that, 183 min went to nine preservation certificates, which compiled one after another. Every certificate A imported all 78 project modules, and every certificate was named after the project's semantic id, so any change re-proved all nine.
What changes
LEXLEAN_VERIFY_WIDTHand withinLEXLEAN_VERIFY_MEMORY_KIB, using the existingpool::run_ordered_withand hints. Eachleancheckerreplay runs in its compile's slot. A certificate inside another's namespace waits for that one's replay, becauseleancheckerreplays a module prefix. Records are pushed in canonical order, so the attestation is byte-identical at any width.LEXLEAN_VERIFY_STORE, new SPEC 22.13). The key is the tool, the Lean executable digest, the normalized argv, and the Merkle κ of the module text and every import it reaches. The header parser handlesmodule,public import,meta importandimport all. A replay's identity covers its prefix. Outputs are stored by stem, together with Lean's.olean.server,.olean.privateand.ircompanions. Every restored file is digest-checked, and an entry is restored only for the same executable and exit 0. A miss means the process runs.LexLeanPreserve.C<sha256("lexlean/certificate-name/1" ‖ program)[..32]>.R<i>. Certificate A imports only the user modules that own a constant its terms name.reusedtelemetry event is added. SPEC 23.1 lists the five operational variables.Measured
examples/production-coveragepublishes the same attestation in all three runs:uorc client export:
The warm export published the same result id as the cold one. The rest of the warm time goes to certificate generation (344 s) and extraction (253 s), which the store does not cover yet.
Unit tests cover the import-header parser and Merkle propagation. uorc pins this head in UOR-Foundation/compression#186.
Porting to
release/v0.5waits on the staged-certificates and evaluation-proof chain landing there. A cherry-pick conflicts only inverify/mod.rs's certificate stage, which that chain rewrites.Generated by Claude Code