From e8753cefa7ad78e015f09e2fe054fa688554ed1f Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 02:00:09 +0000 Subject: [PATCH 1/8] Export cost follows what a change touches: parallel certificates, a process store by content identity, certificates named by their root's program The certificate stage compiled the preservation environment and every certificate one after another on one core (the uorc client: 9 certificates, about 1 GB of Lean, 2 h of a 2.3 h export). They now run in the topological frontiers of their import headers at the verification width and within the memory budget, each replay in its compilation's slot, records in the stage's canonical order (SPEC 22.11 item 6). Certificate labels drop their content segment, so measured hints carry to the next run. A process store (LEXLEAN_VERIFY_STORE, SPEC 22.13) reuses an elaboration, replay, audit member or certificate process whose key (tool, executable digest, normalized argv, Merkle content identity of the module and every module its imports reach) an earlier run established; restored files and records are digest-checked and judged as the stage judges its own, so the published directory is byte-identical with and without it. Certificates are named by their root's lowered program instead of the project's semantic id, and certificate A imports only the user modules its terms name, so a root a change does not reach keeps its certificates' text and content identity and is not proved again. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- SPEC.md | 22 +- crates/lexlean/src/production/certificate.rs | 9 +- .../src/production/certificate/data_a.rs | 34 +- crates/lexlean/src/verify/leanchecker.rs | 13 +- crates/lexlean/src/verify/mod.rs | 327 +++++++++---- crates/lexlean/src/verify/store.rs | 440 ++++++++++++++++++ crates/lexlean/src/verify/telemetry.rs | 18 + 7 files changed, 766 insertions(+), 97 deletions(-) create mode 100644 crates/lexlean/src/verify/store.rs diff --git a/SPEC.md b/SPEC.md index 84cfaa287d..9445af5468 100644 --- a/SPEC.md +++ b/SPEC.md @@ -5141,8 +5141,15 @@ certificate generator, and their shared source reader obey §17.13's exhaustiven audit: they name every construct and match none by default. **Statement.** The certificate of the root `r` with parameters `x₁ … xₙ` is -the module `LexLeanPreserve.C.R`, `i` the -root's position in module then report order, and proves +the module `LexLeanPreserve.C.R`, `i` the root's position in module +then report order and `` the first 32 hex digits of the SHA-256 of +`lexlean/certificate-name/1`, NUL, the root's module, NUL, its name, NUL, and +the bytes of its lowered program (`preserve/program/R.json`): the name is a +function of what the certificates are about, so a root whose program a change +of the project leaves as it was keeps its certificates' names. Certificate A +imports the preservation library and those generated user modules whose +constants its terms name, and no other (Lean's imports are transitive, so the +modules those constants are defined from come with them). It proves ```lean theorem root (x₁ … xₙ) : @@ -9207,6 +9214,7 @@ The profile changes how processes are scheduled and nothing else: 3. Results are consumed in the canonical order and never in completion order: the members of a frontier in module-declaration order, replays in sorted full module name, audit members in the order of the audit family. The process records, the staged files, the merge of the observed axiom sets, the policy rows, and the attestation body are functions of that order. When several processes fail, the diagnostic reported is the one of the first failing process in that order, which is the diagnostic width 1 reports; no process later in that order than a known failure is started, and every process started is finished before `verify` returns. Width 1 is exactly the sequence the verifier ran before there was a width: a process starts only after the result of the one before it has been consumed, so a failure that only the consumer can see (a missing `.olean`, a staging error) stops the stage before another process starts. 4. The verifier does not measure the host. A width above 1 is for a host whose memory envelope is known; a width the host cannot hold is a failure of that host's run, never a verification result. The normative job's `examples` leg names its profile (§9.4); every other run uses the default unless its environment selects one, and the conformance cases exercise widths above 1 on small projects. 5. Hints and the memory budget are schedule inputs, never verdicts. A hint only reorders the start of items inside a stage, and the memory budget only gates a worker from starting an item whose expected peak (plus the expected peaks of the running items) would pass it. An item whose expected peak alone passes the budget runs alone rather than waiting forever: a wrong hint is a wrong schedule, never a fence and never a kill. A process that outgrows its hinted peak runs to completion; the stage measures it for the next hints. Hints change which ready item a worker starts first; results are consumed in the canonical order exactly as at width 1, so a failure reported is still the first failure in that order. +6. Certificates (§17.17, stage 12) compile as one stage with the preservation environment they import, in the topological frontiers of their import headers, like module elaboration: the certificates A and B of every root import nothing from each other and run together, at most `W` at a time, and a certificate E starts when the A and B it composes have finished. Each certificate's `leanchecker` replay runs in the slot of its compilation, right after it. The records are listed in the stage's canonical order (the environment's compilations, then each certificate's compilation and replay, in certificate order) whatever order the processes ran in. The label of a certificate's processes, for the hints, is its module name without its `C` segment, so a root's measured cost schedules its next certificates. For every project, the directory published at any profile equals, byte for byte, the directory published at width 1 without hints and without a budget, on the same host and toolchain; in particular the attestation ID, the observed axiom sets, the policy rows, the canonical compiler input, and every normalized verification record (§29.5) are identical. @@ -9218,6 +9226,14 @@ Wall time, resident-set and swap measurements, scheduling and completion order, Telemetry is write-only: no stage reads it, no ID or verified artifact contains any of it, and a failure to write it changes no result. Every key it uses is in a closed vocabulary (`wall_ms`, `started_unix_ms`, `peak_rss_kib`, `peak_swap_kib`, `mem_total_kib`, `mem_available_kib`, `swap_total_kib`, `swap_used_kib`, `host_cpus`, `process_width`, `completion_seq`), and a JSON key of that vocabulary anywhere in a verified directory fails the gate: `cargo xtask verify-examples` checks every example's verified directory for one. +### 22.13 Process store + +A verification process is a function of what it reads, so the verifier may take a process another run on the same host already established instead of running it again. The store is selected only by the environment variable `LEXLEAN_VERIFY_STORE`, the path of a directory; an unset or empty variable selects none. Like the resource profile (§22.11) it is operational: it is no field of `lexlean.toml`, the lock, a request, or the command line, no artifact records it, it enters no ID, and the directory a verification publishes with a store equals, byte for byte, the one it publishes without. + +The content identity of a staged Lean module is the SHA-256 of the domain `lexlean/verify-store/1`, its name, the digest of its staged text, and, for each module its import header names (the `import` commands before its first other command, with their `public`, `private`, `meta` and `all` modifiers), that module's name and content identity; a module the stage does not hold (the toolchain's `Init`, `Lean`, `Std`) is identified by its name, which the toolchain digest fixes. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the content identity of the module (or, for an audit member, of its source) it is about, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. + +Only a process that exited 0 is stored, with its normalized record and every file it wrote beside its `.olean` (the companions of that file's stem). A stored process is used only when its record names this run's digest of the executable and every stored file has the digest it was stored with; its files are restored to where the process writes them and its record is judged as the stage judges its own (§22.3 to §22.6), so a stored process that a stage would refuse is refused. Anything else, an unreadable or partial entry included, is a miss and the process runs. A failure to store is dropped, since the store never decides a result. Telemetry (§22.12) reports a stored process as a `reused` event of its stage. + --- ## 23. Command-line interface @@ -9238,7 +9254,7 @@ Global options: --help ``` -No environment variable changes semantic configuration. Four operational variables exist and change no output: `LEXLEAN_VERIFY_WIDTH` selects the verification process width, `LEXLEAN_VERIFY_HINTS` names the scheduling-hints file and `LEXLEAN_VERIFY_MEMORY_KIB` the process memory budget (§22.11), and `LEXLEAN_CI_TELEMETRY` names the directory that receives performance telemetry (§22.12). +No environment variable changes semantic configuration. Five operational variables exist and change no output: `LEXLEAN_VERIFY_WIDTH` selects the verification process width, `LEXLEAN_VERIFY_HINTS` names the scheduling-hints file and `LEXLEAN_VERIFY_MEMORY_KIB` the process memory budget (§22.11), `LEXLEAN_CI_TELEMETRY` names the directory that receives performance telemetry (§22.12), and `LEXLEAN_VERIFY_STORE` names the process store (§22.13). ### 23.2 Project discovery diff --git a/crates/lexlean/src/production/certificate.rs b/crates/lexlean/src/production/certificate.rs index 8f368c5fd3..c50c138425 100644 --- a/crates/lexlean/src/production/certificate.rs +++ b/crates/lexlean/src/production/certificate.rs @@ -566,9 +566,12 @@ pub fn certificate( generator .within(source.len(), "certificate A") .map_err(failure)?; - let mut imports: Vec = modules - .values() - .map(|linked| linked.lean_module.to_owned()) + let users: BTreeSet<&str> = modules.values().map(|linked| linked.lean_module).collect(); + let mut imports: Vec = data + .imports + .iter() + .filter(|import| users.contains(import.as_str())) + .cloned() .collect(); imports.sort(); Ok(Certificate { diff --git a/crates/lexlean/src/production/certificate/data_a.rs b/crates/lexlean/src/production/certificate/data_a.rs index 022aed2147..90df121c98 100644 --- a/crates/lexlean/src/production/certificate/data_a.rs +++ b/crates/lexlean/src/production/certificate/data_a.rs @@ -630,10 +630,42 @@ impl Gen<'_> { .map(|linked| linked.lean_module) .collect(); users.sort_unstable(); + let users_all: BTreeSet = users.iter().map(|module| (*module).to_owned()).collect(); imports.extend(users.into_iter().map(str::to_owned)); let mut writer = Writer::new(self, name); writer.write(root_module, root_name, report, &graph)?; - writer.finish(imports) + let mut data = writer.finish(imports)?; + // A certificate imports the user modules whose constants its terms + // name, and no other: Lean's imports are transitive, so the modules + // those constants are defined from come with them, and a module the + // root's closure never reaches is not among what the certificate + // reads (§17.17, §22.13). + let named: BTreeSet<&str> = data + .nodes + .iter() + .filter_map(|node| match node { + crate::ir::core::CoreNode::Const { n, .. } => Some(n.as_str()), + _ => None, + }) + .collect(); + let owner = |constant: &str| -> Option { + users_all + .iter() + .filter(|module| { + constant.len() > module.len() + && constant.starts_with(module.as_str()) + && constant.as_bytes()[module.len()] == b'.' + }) + .max_by_key(|module| module.len()) + .cloned() + }; + let reached: BTreeSet = named + .iter() + .filter_map(|constant| owner(constant)) + .collect(); + data.imports + .retain(|import| !users_all.contains(import) || reached.contains(import)); + Ok(data) } } diff --git a/crates/lexlean/src/verify/leanchecker.rs b/crates/lexlean/src/verify/leanchecker.rs index e165e573d5..705db1dc0d 100644 --- a/crates/lexlean/src/verify/leanchecker.rs +++ b/crates/lexlean/src/verify/leanchecker.rs @@ -9,7 +9,7 @@ use camino::Utf8Path; use crate::code; use crate::config::Limits; use crate::diagnostic::Diagnostic; -use crate::verify::child::{run, ChildHome, ChildRecord, ChildSpec, Normalizer}; +use crate::verify::child::{ChildHome, ChildRecord, ChildSpec, Normalizer}; use crate::verify::toolchain::Toolchain; /// Run `lake env ` (the preflighted executable by @@ -21,9 +21,11 @@ pub fn run_leanchecker( workspace: &Utf8Path, limits: &Limits, normalizer: &Normalizer, + session: Option<&crate::verify::store::Session>, ) -> Result { let bin_dir = toolchain.root.join("bin"); - run( + crate::verify::store::run( + session, &ChildSpec { tool: "leanchecker", module: Some(module.to_owned()), @@ -45,6 +47,8 @@ pub fn run_leanchecker( }, limits, normalizer, + |session| Some(session.module(module)), + &[], ) } @@ -57,8 +61,11 @@ pub fn replay_module( workspace: &Utf8Path, limits: &Limits, normalizer: &Normalizer, + session: Option<&crate::verify::store::Session>, ) -> Result { - let record = run_leanchecker(toolchain, module, lean_path, workspace, limits, normalizer)?; + let record = run_leanchecker( + toolchain, module, lean_path, workspace, limits, normalizer, session, + )?; if record.exit_code != 0 { return Err(Diagnostic::new( code!("LLV7003"), diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index e5380da5e0..201166af9b 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -28,6 +28,7 @@ pub mod operational; pub mod partition; pub mod pool; pub mod source_audit; +pub mod store; pub mod telemetry; pub mod toolchain; pub mod workspace; @@ -650,6 +651,7 @@ fn leanchecker_identity( workspace_root, limits, normalizer, + None, )?; let combined = format!("{}{}", record.stdout.trim_end(), record.stderr.trim_end()); let observed = format!("{}\n", combined.trim_end()); @@ -883,7 +885,21 @@ pub fn run_part( use crate::production::rust_cert; for (index, root) in roots(checked).map_err(fail)?.iter().enumerate() { let (lowered, program) = lowered_roots[index].clone(); - let module = format!("LexLeanPreserve.C{semantic_hex32}.R{index}"); + // A root's certificates are named by what they are about, the + // root and its lowered program, and not by the project's semantic + // identity: a root whose program a change leaves as it was keeps + // its certificates' names and text, so their processes keep their + // content identities (§22.13). + let root_hex32: String = { + let mut bytes = format!( + "lexlean/certificate-name/1\0{}\0{}\0", + root.module, root.name + ) + .into_bytes(); + bytes.extend_from_slice(&program); + Sha256Digest::of(&bytes).to_hex()[..32].to_owned() + }; + let module = format!("LexLeanPreserve.C{root_hex32}.R{index}"); let generated = certificate( &linked, &root.module, @@ -1273,6 +1289,16 @@ pub fn run_part( )?; } release_non_diagnostic_buffers(build); + // The process store (§22.13), when the environment selects one: the + // staged sources the identities of this run's processes are read from, + // in the order of the search paths their processes read imports from. + let store_session = store::Store::from_env().map(|selected| { + store::Session::new( + selected, + vec![staging_utf8.join("preserve-src"), src_root.clone()], + ) + }); + let store_session = store_session.as_ref(); let lean_path_env = format!("{olean_root}"); // The records of this run's processes, by what they are and the module // or audit member they are of, and the records of the remaining stages @@ -1495,7 +1521,8 @@ pub fn run_part( )) })?; } - let record = run_child( + let record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(module.name.clone()), @@ -1518,6 +1545,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| Some(session.module(&module.name)), + std::slice::from_ref(&olean), ) .map_err(fail)?; if record.exit_code != 0 @@ -1643,7 +1672,8 @@ pub fn run_part( let module_path = module_name.replace('.', "/"); let source = src_root.join(format!("{module_path}.lean")); let olean = olean_root.join(format!("{module_path}.olean")); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(module_name.clone()), @@ -1666,6 +1696,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| Some(session.module(module_name)), + std::slice::from_ref(&olean), ) }, &|result: &Result| { @@ -1744,6 +1776,7 @@ pub fn run_part( &workspace_root, &limits, &normalizer, + store_session, ) .map_err(fail)?; write_staged( @@ -1776,6 +1809,7 @@ pub fn run_part( &workspace_root, &limits, &normalizer, + store_session, ) // A replay failure is about this module, so it points at the // module's source rather than at the project manifest (§20.1). @@ -1828,7 +1862,8 @@ pub fn run_part( let audit = &audit_modules[audit_index]; let source_relative = format!("audit/{}.lean", audit.name); let audit_source = staging_utf8.join(&source_relative); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(audit.name.clone()), @@ -1847,6 +1882,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&audit_source), + &[], ) }, &|result: &Result| { @@ -2401,8 +2438,7 @@ pub fn run_part( .iter() .map(|module| module.lean_module.clone()) .collect(); - let mut stage = - preserve::stage(&generated_modules, &staged_certificates).map_err(fail)?; + let mut stage = preserve::stage(&generated_modules, &staged_certificates).map_err(fail)?; staged_certificates = staged_certificates .into_iter() .map(crate::production::certificate::Certificate::released) @@ -2502,6 +2538,7 @@ pub fn run_part( }; let compile = |file: &preserve::StagedFile, output: bool| { let source = preserve_src.join(&file.path); + let mut outputs: Vec = Vec::new(); let mut argv = vec![ "env".to_owned(), "lean".to_owned(), @@ -2517,9 +2554,11 @@ pub fn run_part( } argv.push("-o".to_owned()); argv.push(olean.to_string()); + outputs.push(olean); } argv.push(source.to_string()); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(file.module.clone()), @@ -2534,93 +2573,207 @@ pub fn run_part( }, &limits, &normalizer, + |session| Some(session.module(&file.module)), + &outputs, ) }; - for file in &stage.environment { - // A runtime module the calculus modules import was compiled, - // replayed, and audited with the project's (stage 7), and its - // olean is on this search path. - if runtime_modules - .iter() - .any(|module| module.name == file.module) - { - continue; - } - let record = compile(file, true).map_err(fail)?; - if record.exit_code != 0 - || !record.stdout.trim().is_empty() - || !record.stderr.trim().is_empty() - { - let combined = format!("{}{}", record.stdout, record.stderr); - if let Some(exhausted) = resource_death( - &file.module, - record.exit_code, - &combined, - file.text.len() as u64, - &limits, - ) { - return Err(fail(exhausted)); - } - return Err(fail(Diagnostic::new( - code!("LLV7014"), - format!( - "preservation environment: `{}` does not compile silently under the pinned Lean: {}", - file.module, - first_error(&format!("{}{}", record.stdout, record.stderr)) - ), + // The environment and the certificates compile as one stage, in the + // topological frontiers of their imports (§22.11): a certificate A or + // B imports the library and the project's modules, a certificate E + // the A and B it composes, so the certificates of different roots, + // and A and B of one root, import nothing from each other and run + // together, at most `W` at a time and within the memory budget. Each + // certificate's `leanchecker` replay runs in the same process slot + // right after its compilation. The records are listed in the stage's + // canonical order (the environment, then each certificate's + // compilation and replay) whatever order the processes ran in, so the + // published directory is the one width 1 publishes. + let environment_items: Vec<&preserve::StagedFile> = stage + .environment + .iter() + .filter(|file| { + !runtime_modules + .iter() + .any(|module| module.name == file.module) + }) + .collect(); + let items: Vec<&preserve::StagedFile> = environment_items + .iter() + .copied() + .chain(&stage.certificates) + .collect(); + let environment_count = environment_items.len(); + let staged_imports = |file: &preserve::StagedFile| -> Vec { + store::imports( + &std::fs::read_to_string(preserve_src.join(&file.path).as_std_path()) + .unwrap_or_default(), + ) + }; + let depends: Vec> = items + .iter() + .map(|file| { + let imports = staged_imports(file); + items + .iter() + .enumerate() + .filter(|(_, other)| imports.iter().any(|import| *import == other.module)) + .map(|(index, _)| index) + .collect() + }) + .collect(); + // A stable label per item, for the scheduling hints: a certificate's + // module name without its content segment, so that the observed cost + // of a root's certificate schedules the next run's. + let labels: Vec = items + .iter() + .map(|file| { + file.module + .split('.') + .filter(|segment| { + !(segment.len() == 33 + && segment.starts_with('C') + && segment[1..].bytes().all(|byte| byte.is_ascii_hexdigit())) + }) + .collect::>() + .join(".") + }) + .collect(); + type Run = ( + Result, + Option>, + ); + let silent = |record: &ChildRecord| { + record.exit_code == 0 + && record.stdout.trim().is_empty() + && record.stderr.trim().is_empty() + }; + let mut results: Vec)>> = vec![None; items.len()]; + let mut done: BTreeSet = BTreeSet::new(); + let mut frontier_index = 0usize; + while done.len() < items.len() { + let ready: Vec = (0..items.len()) + .filter(|index| !done.contains(index)) + .filter(|index| { + depends[*index] + .iter() + .all(|dependency| done.contains(dependency)) + }) + .collect(); + if ready.is_empty() { + return Err(fail(internal( + "the preservation modules' imports do not converge", ))); } - write_staged( - staging.path(), - &format!("process/preserve/{}.json", file.module), - &record.to_json().to_file_bytes(), + let frontier_started = std::time::Instant::now(); + pool::run_ordered_with( + width, + &ready, + hints, + memory_kib, + &|index: usize| labels[index].clone(), + &|index: usize| -> Run { + let file = items[index]; + let compiled = compile(file, true); + if index < environment_count { + return (compiled, None); + } + match &compiled { + Ok(record) if silent(record) => { + let replay = leanchecker::run_leanchecker( + &toolchain, + &file.module, + &preserve_path, + &workspace_root, + &limits, + &normalizer, + store_session, + ); + (compiled, Some(replay)) + } + _ => (compiled, None), + } + }, + &|run: &Run| match run { + (Ok(record), replay) => { + !silent(record) + || matches!(replay, Some(Err(_))) + || matches!(replay, Some(Ok(replay)) if replay.exit_code != 0) + } + (Err(_), _) => true, + }, + |index, (compiled, replay)| { + let file = items[index]; + let record = compiled.map_err(fail)?; + if !silent(&record) { + let combined = format!("{}{}", record.stdout, record.stderr); + if index < environment_count { + if let Some(exhausted) = resource_death( + &file.module, + record.exit_code, + &combined, + file.text.len() as u64, + &limits, + ) { + return Err(fail(exhausted)); + } + return Err(fail(Diagnostic::new( + code!("LLV7014"), + format!( + "preservation environment: `{}` does not compile silently under the pinned Lean: {}", + file.module, + first_error(&combined) + ), + ))); + } + return Err(fail(reject( + &file.module, + &combined, + record.exit_code, + &staged_text(file), + ))); + } + write_staged( + staging.path(), + &format!("process/preserve/{}.json", file.module), + &record.to_json().to_file_bytes(), + )?; + let replay = match replay { + None => None, + Some(replay) => { + let replay = replay.map_err(fail)?; + if replay.exit_code != 0 { + let combined = format!("{}{}", replay.stdout, replay.stderr); + return Err(fail(reject( + &file.module, + &combined, + replay.exit_code, + &staged_text(file), + ))); + } + write_staged( + staging.path(), + &format!("process/leanchecker/{}.json", file.module), + &replay.to_json().to_file_bytes(), + )?; + Some(replay) + } + }; + results[index] = Some((record, replay)); + Ok(()) + }, )?; - tail_records.push(record); + telemetry_run.frontier( + "certificates", + frontier_index, + ready.len(), + frontier_started, + ); + frontier_index += 1; + done.extend(ready); } - for file in &stage.certificates { - let record = compile(file, true).map_err(fail)?; - if record.exit_code != 0 - || !record.stdout.trim().is_empty() - || !record.stderr.trim().is_empty() - { - let combined = format!("{}{}", record.stdout, record.stderr); - return Err(fail(reject( - &file.module, - &combined, - record.exit_code, - &staged_text(file), - ))); - } - write_staged( - staging.path(), - &format!("process/preserve/{}.json", file.module), - &record.to_json().to_file_bytes(), - )?; + for (record, replay) in results.into_iter().flatten() { tail_records.push(record); - let replay = leanchecker::run_leanchecker( - &toolchain, - &file.module, - &preserve_path, - &workspace_root, - &limits, - &normalizer, - ) - .map_err(fail)?; - if replay.exit_code != 0 { - let combined = format!("{}{}", replay.stdout, replay.stderr); - return Err(fail(reject( - &file.module, - &combined, - replay.exit_code, - &staged_text(file), - ))); - } - write_staged( - staging.path(), - &format!("process/leanchecker/{}.json", file.module), - &replay.to_json().to_file_bytes(), - )?; - tail_records.push(replay); + tail_records.extend(replay); } let audit_record = compile(&stage.audit, false).map_err(fail)?; let audit_output = format!("{}{}", audit_record.stdout, audit_record.stderr); diff --git a/crates/lexlean/src/verify/store.rs b/crates/lexlean/src/verify/store.rs new file mode 100644 index 0000000000..bdd5630f05 --- /dev/null +++ b/crates/lexlean/src/verify/store.rs @@ -0,0 +1,440 @@ +//! The process store (SPEC.md §22.13): a verification process is a function +//! of its content identity, so a process whose identity an earlier run on +//! this host already established is not run again. +//! +//! The content identity κ of a Lean module is the digest of its staged text +//! and of the identities of the modules its import header names, by name: a +//! Merkle identity of everything a `lean` or `leanchecker` process of that +//! module can read beyond the pinned toolchain. A module the stage does not +//! hold (the toolchain's own `Init`, `Lean`, `Std`) is identified by its name, +//! which the toolchain digest fixes. The key of one process adds the tool, +//! the digest of its executable, and its normalized argument vector, so the +//! same module compiled, replayed, or compiled without output are three +//! processes. +//! +//! The store is operational, like the width (§22.11): it is selected only by +//! `LEXLEAN_VERIFY_STORE`, is no input of any stage, enters no ID, and +//! changes no published byte. Only a process that succeeded is stored, with +//! its normalized record and the bytes of the files it wrote; a stored entry +//! is used only when its record names this run's digest of the executable +//! and every stored file still has the digest it was stored with. Anything +//! else, including an unreadable entry, is a miss and the process runs. + +use std::collections::BTreeMap; +use std::path::PathBuf; +use std::sync::{Mutex, PoisonError}; + +use camino::{Utf8Path, Utf8PathBuf}; + +use crate::artifact::canonical_json::Json; +use crate::artifact::content_id::Sha256Digest; +use crate::config::Limits; +use crate::diagnostic::Diagnostic; +use crate::verify::child::{ChildRecord, ChildSpec, Normalizer}; + +/// The environment variable that names the store directory. +pub const STORE_VARIABLE: &str = "LEXLEAN_VERIFY_STORE"; + +const DOMAIN: &str = "lexlean/verify-store/1"; + +/// A process store directory. +#[derive(Debug, Clone)] +pub struct Store { + root: PathBuf, +} + +impl Store { + /// The store the environment selects, if any: an unset or empty variable + /// selects none. + #[must_use] + pub fn from_env() -> Option { + let value = std::env::var_os(STORE_VARIABLE)?; + if value.is_empty() { + return None; + } + Some(Self { + root: PathBuf::from(value), + }) + } + + fn entry(&self, key: &Sha256Digest) -> PathBuf { + let hex = key.to_hex(); + self.root.join(&hex[..2]).join(hex) + } + + /// The stored record of the process `key`, after restoring the files it + /// wrote to `outputs`, or `None` when the store does not hold it whole. + fn restore( + &self, + key: &Sha256Digest, + outputs: &[Utf8PathBuf], + executable: Sha256Digest, + ) -> Option { + let entry = self.entry(key); + let manifest = std::fs::read_to_string(entry.join("outputs.txt")).ok()?; + let record = ChildRecord::from_json( + &Json::parse(&std::fs::read(entry.join("record.json")).ok()?).ok()?, + ) + .ok()?; + if record.executable_sha256 != executable || record.exit_code != 0 { + return None; + } + // One line per file written: the output it belongs to, the suffix + // after that output's name (Lean writes `.olean` with companions), and + // the file's digest. + let mut files = Vec::new(); + for (position, line) in manifest.lines().enumerate() { + let mut fields = line.splitn(3, ' '); + let index: usize = fields.next()?.parse().ok()?; + let digest = fields.next()?; + let suffix = fields.next().unwrap_or(""); + let output = outputs.get(index)?; + let stored = std::fs::read(entry.join(format!("{position}.out"))).ok()?; + if Sha256Digest::of(&stored).to_hex() != digest { + return None; + } + files.push(( + Utf8PathBuf::from(format!("{}{suffix}", output.with_extension(""))), + stored, + )); + } + if outputs.iter().enumerate().any(|(index, _)| { + !manifest + .lines() + .any(|line| line.split(' ').next() == Some(&index.to_string())) + }) { + return None; + } + for (path, stored) in files { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent.as_std_path()).ok()?; + } + std::fs::write(path.as_std_path(), stored).ok()?; + } + // The entry was used: its time orders what a pruning keeps. + let _ = std::fs::File::options() + .append(true) + .open(entry.join("outputs.txt")) + .and_then(|file| file.set_modified(std::time::SystemTime::now())); + Some(record) + } + + /// Store a process that succeeded. A failure to store is dropped: the + /// store never decides a result. + fn save(&self, key: &Sha256Digest, outputs: &[Utf8PathBuf], record: &ChildRecord) { + let entry = self.entry(key); + let Some(parent) = entry.parent() else { return }; + if entry.is_dir() || std::fs::create_dir_all(parent).is_err() { + return; + } + let Ok(scratch) = tempfile::Builder::new() + .prefix(".entry-") + .tempdir_in(parent) + else { + return; + }; + let mut digests = String::new(); + let mut position = 0usize; + for (index, output) in outputs.iter().enumerate() { + // Lean writes the `.olean` with companions of the same stem + // (`.olean.server`, `.olean.private`, `.ir`): every file of that + // stem in the output's directory is what the process wrote. + let (Some(directory), Some(stem)) = (output.parent(), output.file_stem()) else { + return; + }; + let name = format!("{stem}."); + let name = name.as_str(); + let Ok(entries) = std::fs::read_dir(directory.as_std_path()) else { + return; + }; + let mut written: Vec = entries + .flatten() + .filter_map(|entry| entry.file_name().into_string().ok()) + .filter(|file| file.starts_with(name)) + .collect(); + written.sort(); + for file in written { + let Ok(bytes) = std::fs::read(directory.join(&file).as_std_path()) else { + return; + }; + let suffix = &file[stem.len()..]; + digests.push_str(&format!( + "{index} {} {suffix}\n", + Sha256Digest::of(&bytes).to_hex() + )); + if std::fs::write(scratch.path().join(format!("{position}.out")), bytes).is_err() { + return; + } + position += 1; + } + } + if std::fs::write( + scratch.path().join("record.json"), + record.to_json().to_file_bytes(), + ) + .is_err() + || std::fs::write(scratch.path().join("outputs.txt"), digests).is_err() + { + return; + } + let scratch = scratch.keep(); + if std::fs::rename(&scratch, &entry).is_err() { + // Another run stored the same process first. + let _ = std::fs::remove_dir_all(&scratch); + } + } +} + +/// The key of one process: its tool, executable, normalized arguments, and +/// the content identity of the module it is about. +#[must_use] +pub fn key( + tool: &str, + executable: Sha256Digest, + argv: &[String], + identity: Sha256Digest, +) -> Sha256Digest { + let mut bytes = Vec::new(); + for part in [DOMAIN, tool, &executable.to_hex(), &identity.to_hex()] { + bytes.extend_from_slice(part.as_bytes()); + bytes.push(0); + } + for argument in argv { + bytes.extend_from_slice(argument.as_bytes()); + bytes.push(0); + } + Sha256Digest::of(&bytes) +} + +/// Run `process` unless the store holds the process `key`; a process that +/// succeeded silently is stored with the files `outputs` it wrote. +/// +/// # Errors +/// +/// What `process` returns. +pub fn stored( + store: Option<&Store>, + key: Option, + outputs: &[Utf8PathBuf], + executable: Sha256Digest, + process: impl FnOnce() -> Result, +) -> Result { + let (Some(store), Some(key)) = (store, key) else { + return process(); + }; + if let Some(record) = store.restore(&key, outputs, executable) { + crate::verify::telemetry::reused(record.module.as_deref(), &record.tool); + return Ok(record); + } + let record = process()?; + if record.exit_code == 0 && outputs.iter().all(|output| output.as_std_path().is_file()) { + store.save(&key, outputs, &record); + } + Ok(record) +} + +/// The import header of a Lean source: the module each `import` command of +/// its header names, in order. The header is what precedes the first command +/// that is not an import: blank lines, line comments, the `module` and +/// `prelude` keywords, and imports with their `public`, `private`, `meta`, and +/// `all` modifiers (Lean's module system). +#[must_use] +pub fn imports(text: &str) -> Vec { + let mut found = Vec::new(); + for line in text.lines().map(str::trim) { + if line.is_empty() || line.starts_with("--") || line == "module" || line == "prelude" { + continue; + } + let mut words = line.split_whitespace().peekable(); + while matches!(words.peek(), Some(&("public" | "private" | "meta"))) { + words.next(); + } + if words.next() != Some("import") { + break; + } + if words.peek() == Some(&"all") { + words.next(); + } + found.extend(words.map(str::to_owned)); + } + found +} + +/// The store as one verification uses it: the store, and the staged source +/// roots its modules' identities are read from, in the order of the search +/// path the processes read their imports from. +pub struct Session { + store: Store, + roots: Vec, + known: Mutex>, +} + +impl Session { + /// The session of a verification whose sources are staged under `roots`. + #[must_use] + pub fn new(store: Store, roots: Vec) -> Self { + Self { + store, + roots, + known: Mutex::new(BTreeMap::new()), + } + } + + /// The content identity of the staged module `name`: its text and the + /// identities of its imports. A module no root holds is the toolchain's + /// and is identified by its name. + pub fn module(&self, name: &str) -> Sha256Digest { + let mut open = Vec::new(); + self.visit(name, &mut open) + } + + /// The content identity of the source at `path` and its imports. + pub fn source(&self, path: &Utf8Path) -> Option { + let text = std::fs::read(path.as_std_path()).ok()?; + let mut open = Vec::new(); + // Labelled by its file name: the staging directory it lies in is a + // fresh one in every run. + Some(self.of_text(path.file_name()?, &text, &mut open)) + } + + fn visit(&self, name: &str, open: &mut Vec) -> Sha256Digest { + if let Some(identity) = self + .known + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(name) + { + return *identity; + } + let relative = format!("{}.lean", name.replace('.', "/")); + let Some(text) = self + .roots + .iter() + .find_map(|root| std::fs::read(root.join(&relative).as_std_path()).ok()) + else { + return Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes()); + }; + if open.iter().any(|other| other == name) { + return Sha256Digest::of(format!("{DOMAIN}\0cycle\0{name}").as_bytes()); + } + open.push(name.to_owned()); + let identity = self.of_text(name, &text, open); + open.pop(); + self.known + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(name.to_owned(), identity); + identity + } + + fn of_text(&self, label: &str, text: &[u8], open: &mut Vec) -> Sha256Digest { + let mut bytes = format!( + "{DOMAIN}\0module\0{label}\0{}\0", + Sha256Digest::of(text).to_hex() + ) + .into_bytes(); + for import in imports(&String::from_utf8_lossy(text)) { + let identity = self.visit(&import, open); + bytes.extend_from_slice(format!("{import}\0{}\0", identity.to_hex()).as_bytes()); + } + Sha256Digest::of(&bytes) + } + + /// Run the process `spec` about the content `identity` unless the store + /// holds it; `outputs` are the files it writes. + /// + /// # Errors + /// + /// What the process returns. + pub fn run( + &self, + spec: &ChildSpec<'_>, + limits: &Limits, + normalizer: &Normalizer, + identity: Sha256Digest, + outputs: &[Utf8PathBuf], + ) -> Result { + let argv: Vec = spec + .argv + .iter() + .map(|argument| normalizer.normalize_arg(argument)) + .collect(); + let key = key(spec.tool, spec.executable_sha256, &argv, identity); + stored( + Some(&self.store), + Some(key), + outputs, + spec.executable_sha256, + || crate::verify::child::run(spec, limits, normalizer), + ) + } +} + +/// Run `spec` through the session's store when there is one. +/// +/// # Errors +/// +/// What the process returns. +pub fn run( + session: Option<&Session>, + spec: &ChildSpec<'_>, + limits: &Limits, + normalizer: &Normalizer, + identity: impl FnOnce(&Session) -> Option, + outputs: &[Utf8PathBuf], +) -> Result { + match session.and_then(|session| identity(session).map(|identity| (session, identity))) { + Some((session, identity)) => session.run(spec, limits, normalizer, identity, outputs), + None => crate::verify::child::run(spec, limits, normalizer), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_import_header_reads_the_module_system_forms() { + let text = "module\n-- a comment\npublic import Init\nimport all A.B\nmeta import C\nimport D E\n\nset_option x 1\nimport F\n"; + assert_eq!(imports(text), ["Init", "A.B", "C", "D", "E"]); + assert_eq!( + imports("prelude\nimport Init.Core\ndef x := 1\n"), + ["Init.Core"] + ); + } + + #[test] + fn an_identity_follows_every_module_an_import_reaches() { + let root = tempfile::tempdir().expect("scratch"); + let base = Utf8PathBuf::from_path_buf(root.path().to_path_buf()).expect("utf-8"); + let write = |name: &str, text: &str| { + let path = base.join(format!("{}.lean", name.replace('.', "/"))); + std::fs::create_dir_all(path.parent().expect("parent").as_std_path()).expect("dir"); + std::fs::write(path.as_std_path(), text).expect("write"); + }; + write("P.A", "module\npublic import Init\ndef a := 1\n"); + write("P.B", "module\npublic import P.A\ndef b := 2\n"); + write("P.C", "module\npublic import P.B\ndef c := 3\n"); + let before = Session::new( + Store { + root: PathBuf::new(), + }, + vec![base.clone()], + ); + let (a, c) = (before.module("P.A"), before.module("P.C")); + write("P.A", "module\npublic import Init\ndef a := 4\n"); + let after = Session::new( + Store { + root: PathBuf::new(), + }, + vec![base.clone()], + ); + assert_ne!(after.module("P.A"), a); + assert_ne!( + after.module("P.C"), + c, + "a change two imports away moves the identity" + ); + assert_eq!(after.module("Init"), before.module("Init")); + } +} diff --git a/crates/lexlean/src/verify/telemetry.rs b/crates/lexlean/src/verify/telemetry.rs index 6d5b7c7366..69789b05d5 100644 --- a/crates/lexlean/src/verify/telemetry.rs +++ b/crates/lexlean/src/verify/telemetry.rs @@ -210,6 +210,17 @@ impl Context { .timed("process", report.started, Value::Object(fields)); } + /// Report a process the store held (§22.13): it did not run. + pub fn reused(&self, tool: &str, module: Option<&str>) { + let mut fields = serde_json::Map::new(); + fields.insert("stage".to_owned(), json!(self.stage)); + fields.insert("tool".to_owned(), json!(tool)); + if let Some(module) = module { + fields.insert("module".to_owned(), json!(module)); + } + self.sink.emit("reused", Value::Object(fields)); + } + /// Report that the item at `item` of the stage completed as the /// `sequence`th to do so. This is the one place completion order is /// visible, and it is visible only here. @@ -241,6 +252,13 @@ thread_local! { static CURRENT: RefCell> = const { RefCell::new(None) }; } +/// Report a process the store held, in the context in force on this thread. +pub fn reused(module: Option<&str>, tool: &str) { + if let Some(context) = current() { + context.reused(tool, module); + } +} + /// The context in force on this thread. #[must_use] pub fn current() -> Option { From 3808fac75376d23ecea896598ac70e0b7209425c Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 04:53:31 +0000 Subject: [PATCH 2/8] A certificate inside another's namespace starts after that one's replay; a replay's identity covers its prefix leanchecker replays a module prefix (SPEC 22.4), so certificate A's replay checked its root's B and E as well once they compiled beside it (30 min in place of 2). Certificates B and E now start after their root's A, as at width 1, and a replay's store key covers every staged module in its namespace. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- crates/lexlean/src/verify/leanchecker.rs | 2 +- crates/lexlean/src/verify/mod.rs | 13 +++++++- crates/lexlean/src/verify/store.rs | 38 ++++++++++++++++++++++++ 3 files changed, 51 insertions(+), 2 deletions(-) diff --git a/crates/lexlean/src/verify/leanchecker.rs b/crates/lexlean/src/verify/leanchecker.rs index 705db1dc0d..157cba08c8 100644 --- a/crates/lexlean/src/verify/leanchecker.rs +++ b/crates/lexlean/src/verify/leanchecker.rs @@ -47,7 +47,7 @@ pub fn run_leanchecker( }, limits, normalizer, - |session| Some(session.module(module)), + |session| Some(session.prefix(module)), &[], ) } diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index 201166af9b..a96b776624 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -2616,7 +2616,18 @@ pub fn run_part( items .iter() .enumerate() - .filter(|(_, other)| imports.iter().any(|import| *import == other.module)) + .filter(|(_, other)| { + imports.iter().any(|import| *import == other.module) + // `leanchecker` replays a module prefix (§22.4): + // a certificate inside another's namespace (B and + // E inside their root's A) starts only after that + // one's replay, so each replay checks what it + // checks at width 1. + || file + .module + .strip_prefix(other.module.as_str()) + .is_some_and(|rest| rest.starts_with('.')) + }) .map(|(index, _)| index) .collect() }) diff --git a/crates/lexlean/src/verify/store.rs b/crates/lexlean/src/verify/store.rs index bdd5630f05..08b7db1e33 100644 --- a/crates/lexlean/src/verify/store.rs +++ b/crates/lexlean/src/verify/store.rs @@ -288,6 +288,44 @@ impl Session { self.visit(name, &mut open) } + /// The content identity of what a `leanchecker` replay of the prefix + /// `name` reads: the module and every staged module inside its namespace, + /// which the replay checks with it (§22.4). + pub fn prefix(&self, name: &str) -> Sha256Digest { + let mut inside: Vec = Vec::new(); + let relative = name.replace('.', "/"); + for root in &self.roots { + let mut stack = vec![root.join(&relative)]; + while let Some(directory) = stack.pop() { + let Ok(entries) = std::fs::read_dir(directory.as_std_path()) else { + continue; + }; + for entry in entries.flatten() { + let Ok(path) = Utf8PathBuf::from_path_buf(entry.path()) else { + continue; + }; + if path.is_dir() { + stack.push(path); + } else if path.extension() == Some("lean") { + if let Ok(rest) = path.strip_prefix(root) { + inside.push(rest.with_extension("").as_str().replace('/', ".")); + } + } + } + } + } + inside.sort(); + inside.dedup(); + let mut bytes = + format!("{DOMAIN}\0prefix\0{name}\0{}\0", self.module(name).to_hex()).into_bytes(); + for module in inside { + bytes.extend_from_slice( + format!("{module}\0{}\0", self.module(&module).to_hex()).as_bytes(), + ); + } + Sha256Digest::of(&bytes) + } + /// The content identity of the source at `path` and its imports. pub fn source(&self, path: &Utf8Path) -> Option { let text = std::fs::read(path.as_std_path()).ok()?; From 99f6ba2d46c648dfcedc0931d15913c984397494 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 08:31:58 +0000 Subject: [PATCH 3/8] The named-root extraction and the interface probe are answered by the process store Both are lean processes whose output is a function of their staged source and the modules it imports; with the store a warm client export spends 8.6 s in extraction instead of 253 s. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- crates/lexlean/src/verify/mod.rs | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index a96b776624..5a7f523d18 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -1359,7 +1359,8 @@ pub fn run_part( let probe_source = staging_utf8 .join("probe") .join(format!("{probe_name}.lean")); - let probe_record = run_child( + let probe_record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(probe_name.clone()), @@ -1378,6 +1379,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&probe_source), + &[], ) .map_err(fail)?; if probe_record.exit_code != 0 { @@ -2325,7 +2328,10 @@ pub fn run_part( let source = staging_utf8 .join("extract") .join(format!("{}.lean", driver.name)); - let record = run_child( + // The driver's report is a function of its text and the modules it + // imports, so the store answers it by that identity (§22.13). + let record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(driver.name.clone()), @@ -2340,6 +2346,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&source), + &[], ) .map_err(fail)?; if record.exit_code != 0 { From 8eec2955a4c1fa3cbce6ea528c35e38a38bd3979 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 08:57:00 +0000 Subject: [PATCH 4/8] The process store binds every input a reused process reads A key's identity now adds the process's context: the digest of the program it starts, the tree digest of every toolchain file a lean, lake or leanchecker process loads (executables, shared libraries, .olean with its companions, .ir), the working directory's Lake configuration and .lake tree, and the normalized environment rows. A module neither the stage nor the toolchain's library holds is unbound, and so is everything importing it: such a process is not answered by the store and is not stored. A context that cannot be read whole, or a process not under the toolchain's home, is a miss. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- SPEC.md | 4 +- crates/lexlean/src/verify/leanchecker.rs | 2 +- crates/lexlean/src/verify/mod.rs | 7 +- crates/lexlean/src/verify/store.rs | 326 ++++++++++++++++++++--- 4 files changed, 291 insertions(+), 48 deletions(-) diff --git a/SPEC.md b/SPEC.md index 9445af5468..0fe8ade915 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9230,7 +9230,9 @@ Telemetry is write-only: no stage reads it, no ID or verified artifact contains A verification process is a function of what it reads, so the verifier may take a process another run on the same host already established instead of running it again. The store is selected only by the environment variable `LEXLEAN_VERIFY_STORE`, the path of a directory; an unset or empty variable selects none. Like the resource profile (§22.11) it is operational: it is no field of `lexlean.toml`, the lock, a request, or the command line, no artifact records it, it enters no ID, and the directory a verification publishes with a store equals, byte for byte, the one it publishes without. -The content identity of a staged Lean module is the SHA-256 of the domain `lexlean/verify-store/1`, its name, the digest of its staged text, and, for each module its import header names (the `import` commands before its first other command, with their `public`, `private`, `meta` and `all` modifiers), that module's name and content identity; a module the stage does not hold (the toolchain's `Init`, `Lean`, `Std`) is identified by its name, which the toolchain digest fixes. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the content identity of the module (or, for an audit member, of its source) it is about, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. +The content identity of a staged Lean module is the SHA-256 of the domain `lexlean/verify-store/1`, its name, the digest of its staged text, and, for each module its import header names (the `import` commands before its first other command, with their `public`, `private`, `meta` and `all` modifiers), that module's name and content identity. A module the stage does not hold is bound only when the pinned toolchain's library holds its `.olean`; it is then identified by its name, and its bytes by the toolchain digest below. A module neither holds is unbound, and so is every module whose imports reach it: a process about an unbound module is not answered by the store and is not stored. + +The context of one process is the SHA-256 of the domain, the digest of the program it starts, the toolchain digest, its normalized toolchain `bin` directory and working directory (§22.7), the digest of each of `lakefile.lean`, `lakefile.toml`, `lake-manifest.json` and `lean-toolchain` in its working directory (or that it is absent), the tree digest of the working directory's `.lake` (or that it is absent), and its environment rows with normalized values. The toolchain digest is the tree digest of the toolchain directory, and a tree digest is the SHA-256 of the domain and, in path order, the relative path and digest of every file under it a `lean`, `lake` or `leanchecker` process loads: the executables under `bin`, the shared libraries, and every `.olean`, `.olean.server`, `.olean.private` and `.ir`. A process whose context cannot be read whole, or that does not run under the toolchain's home, is not answered by the store. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the digest of the content identity of the module (or, for an audit member, the extraction driver and the interface probe, of its source) it is about together with its context, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), the interface probe (§18.8), the named-root extraction (§22.10), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. Only a process that exited 0 is stored, with its normalized record and every file it wrote beside its `.olean` (the companions of that file's stem). A stored process is used only when its record names this run's digest of the executable and every stored file has the digest it was stored with; its files are restored to where the process writes them and its record is judged as the stage judges its own (§22.3 to §22.6), so a stored process that a stage would refuse is refused. Anything else, an unreadable or partial entry included, is a miss and the process runs. A failure to store is dropped, since the store never decides a result. Telemetry (§22.12) reports a stored process as a `reused` event of its stage. diff --git a/crates/lexlean/src/verify/leanchecker.rs b/crates/lexlean/src/verify/leanchecker.rs index 157cba08c8..936f83bbec 100644 --- a/crates/lexlean/src/verify/leanchecker.rs +++ b/crates/lexlean/src/verify/leanchecker.rs @@ -47,7 +47,7 @@ pub fn run_leanchecker( }, limits, normalizer, - |session| Some(session.prefix(module)), + |session| session.prefix(module), &[], ) } diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index 5a7f523d18..be2d7902ba 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -1296,6 +1296,7 @@ pub fn run_part( store::Session::new( selected, vec![staging_utf8.join("preserve-src"), src_root.clone()], + toolchain.root.clone(), ) }); let store_session = store_session.as_ref(); @@ -1548,7 +1549,7 @@ pub fn run_part( }, &limits, &normalizer, - |session| Some(session.module(&module.name)), + |session| session.module(&module.name), std::slice::from_ref(&olean), ) .map_err(fail)?; @@ -1699,7 +1700,7 @@ pub fn run_part( }, &limits, &normalizer, - |session| Some(session.module(module_name)), + |session| session.module(module_name), std::slice::from_ref(&olean), ) }, @@ -2581,7 +2582,7 @@ pub fn run_part( }, &limits, &normalizer, - |session| Some(session.module(&file.module)), + |session| session.module(&file.module), &outputs, ) }; diff --git a/crates/lexlean/src/verify/store.rs b/crates/lexlean/src/verify/store.rs index 08b7db1e33..9865b34515 100644 --- a/crates/lexlean/src/verify/store.rs +++ b/crates/lexlean/src/verify/store.rs @@ -6,10 +6,15 @@ //! and of the identities of the modules its import header names, by name: a //! Merkle identity of everything a `lean` or `leanchecker` process of that //! module can read beyond the pinned toolchain. A module the stage does not -//! hold (the toolchain's own `Init`, `Lean`, `Std`) is identified by its name, -//! which the toolchain digest fixes. The key of one process adds the tool, -//! the digest of its executable, and its normalized argument vector, so the -//! same module compiled, replayed, or compiled without output are three +//! hold is bound only when the toolchain's library holds it (the toolchain's +//! own `Init`, `Lean`, `Std`); it is then identified by its name, and the +//! toolchain by the digest of every file of it a process loads. A module +//! neither holds is unbound, and a process whose closure reaches it is not +//! answered by the store. The key of one process adds the tool, the digest of +//! its executable, its normalized argument vector, and its context: the +//! digest of the program it starts, the toolchain, the Lake configuration of +//! its working directory, and its normalized environment rows. The same +//! module compiled, replayed, or compiled without output are three //! processes. //! //! The store is operational, like the width (§22.11): it is selected only by @@ -266,24 +271,175 @@ pub fn imports(text: &str) -> Vec { pub struct Session { store: Store, roots: Vec, - known: Mutex>, + toolchain: Utf8PathBuf, + known: Mutex>>, + files: Mutex>>, +} + +/// The file names under a toolchain or a Lake build directory that a `lean`, +/// `lake`, or `leanchecker` process loads: the executables, the shared +/// libraries, and each module's `.olean` with its companions and `.ir`. +fn loaded(relative: &str) -> bool { + let name = relative.rsplit('/').next().unwrap_or(relative); + relative.starts_with("bin/") + || name.ends_with(".olean") + || name.ends_with(".olean.server") + || name.ends_with(".olean.private") + || name.ends_with(".ir") + || name.ends_with(".so") + || name.contains(".so.") + || name.ends_with(".dylib") + || name.ends_with(".dll") +} + +/// The digest of every loaded file under `root`, by its path relative to +/// `root`, or `None` when the tree cannot be read whole. +fn tree(root: &Utf8Path) -> Option { + let mut files: Vec = Vec::new(); + let mut stack = vec![root.to_path_buf()]; + while let Some(directory) = stack.pop() { + for entry in std::fs::read_dir(directory.as_std_path()).ok()? { + let entry = entry.ok()?; + let path = Utf8PathBuf::from_path_buf(entry.path()).ok()?; + let kind = entry.file_type().ok()?; + if kind.is_dir() { + stack.push(path); + } else { + let relative = path.strip_prefix(root).ok()?.as_str().to_owned(); + if loaded(&relative) { + files.push(relative); + } + } + } + } + files.sort(); + // The files are read on every core: the toolchain is the largest input a + // key binds, and the store answers only once it is read. + let width = std::thread::available_parallelism().map_or(1, std::num::NonZeroUsize::get); + let chunk = files.len().div_ceil(width).max(1); + let digests: Vec>> = std::thread::scope(|scope| { + files + .chunks(chunk) + .map(|part| { + scope.spawn(move || { + part.iter() + .map(|relative| { + std::fs::read(root.join(relative).as_std_path()) + .ok() + .map(|bytes| Sha256Digest::of(&bytes)) + }) + .collect::>>() + }) + }) + .collect::>() + .into_iter() + .map(|handle| handle.join().ok().flatten()) + .collect() + }); + let mut bytes = format!("{DOMAIN}\0tree\0").into_bytes(); + let mut names = files.iter(); + for part in digests { + for digest in part? { + bytes.extend_from_slice(format!("{}\0{}\0", names.next()?, digest.to_hex()).as_bytes()); + } + } + Some(Sha256Digest::of(&bytes)) } impl Session { /// The session of a verification whose sources are staged under `roots`. #[must_use] - pub fn new(store: Store, roots: Vec) -> Self { + pub fn new(store: Store, roots: Vec, toolchain: Utf8PathBuf) -> Self { Self { store, roots, + toolchain, known: Mutex::new(BTreeMap::new()), + files: Mutex::new(BTreeMap::new()), } } + /// The digest of a file or a tree, read once in a session. + fn digest_of( + &self, + path: &Utf8Path, + read: impl FnOnce(&Utf8Path) -> Option, + ) -> Option { + if let Some(known) = self + .files + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(path) + { + return *known; + } + let digest = read(path); + self.files + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(path.to_path_buf(), digest); + digest + } + + /// The context of the process `spec`, beyond its module's identity: the + /// program it starts, the toolchain it loads, the Lake configuration and + /// build directory of its working directory, and its normalized + /// environment rows. `None` when one of them cannot be read whole, or + /// when the process does not run under the toolchain's home. + fn context(&self, spec: &ChildSpec<'_>, normalizer: &Normalizer) -> Option { + let crate::verify::child::ChildHome::Toolchain { toolchain_bin } = spec.home else { + return None; + }; + let program = self.digest_of(spec.program, |path| { + std::fs::read(path.as_std_path()) + .ok() + .map(|bytes| Sha256Digest::of(&bytes)) + })?; + let toolchain = self.digest_of(&self.toolchain, tree)?; + let mut bytes = format!( + "{DOMAIN}\0context\0{}\0{}\0{}\0{}\0", + program.to_hex(), + toolchain.to_hex(), + normalizer.normalize_arg(toolchain_bin.as_str()), + normalizer.normalize_arg(spec.cwd.as_str()), + ) + .into_bytes(); + // `lake env` reads the workspace's configuration and puts its build + // directories on the search path. + for name in [ + "lakefile.lean", + "lakefile.toml", + "lake-manifest.json", + "lean-toolchain", + ] { + let path = spec.cwd.join(name); + let row = match std::fs::read(path.as_std_path()) { + Ok(text) => Sha256Digest::of(&text).to_hex(), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => "absent".to_owned(), + Err(_) => return None, + }; + bytes.extend_from_slice(format!("{name}\0{row}\0").as_bytes()); + } + let lake = spec.cwd.join(".lake"); + let lake_row = if lake.as_std_path().exists() { + self.digest_of(&lake, tree)?.to_hex() + } else { + "absent".to_owned() + }; + bytes.extend_from_slice(format!(".lake\0{lake_row}\0").as_bytes()); + for (key, value) in &spec.extra_env { + bytes.extend_from_slice( + format!("{key}\0{}\0", normalizer.normalize_arg(value)).as_bytes(), + ); + } + Some(Sha256Digest::of(&bytes)) + } + /// The content identity of the staged module `name`: its text and the - /// identities of its imports. A module no root holds is the toolchain's - /// and is identified by its name. - pub fn module(&self, name: &str) -> Sha256Digest { + /// identities of its imports. A module no root holds is bound when the + /// toolchain's library holds it, and is then identified by its name; + /// otherwise it, and everything that imports it, is unbound (`None`). + pub fn module(&self, name: &str) -> Option { let mut open = Vec::new(); self.visit(name, &mut open) } @@ -291,7 +447,7 @@ impl Session { /// The content identity of what a `leanchecker` replay of the prefix /// `name` reads: the module and every staged module inside its namespace, /// which the replay checks with it (§22.4). - pub fn prefix(&self, name: &str) -> Sha256Digest { + pub fn prefix(&self, name: &str) -> Option { let mut inside: Vec = Vec::new(); let relative = name.replace('.', "/"); for root in &self.roots { @@ -316,14 +472,17 @@ impl Session { } inside.sort(); inside.dedup(); - let mut bytes = - format!("{DOMAIN}\0prefix\0{name}\0{}\0", self.module(name).to_hex()).into_bytes(); + let mut bytes = format!( + "{DOMAIN}\0prefix\0{name}\0{}\0", + self.module(name)?.to_hex() + ) + .into_bytes(); for module in inside { bytes.extend_from_slice( - format!("{module}\0{}\0", self.module(&module).to_hex()).as_bytes(), + format!("{module}\0{}\0", self.module(&module)?.to_hex()).as_bytes(), ); } - Sha256Digest::of(&bytes) + Some(Sha256Digest::of(&bytes)) } /// The content identity of the source at `path` and its imports. @@ -332,10 +491,10 @@ impl Session { let mut open = Vec::new(); // Labelled by its file name: the staging directory it lies in is a // fresh one in every run. - Some(self.of_text(path.file_name()?, &text, &mut open)) + self.of_text(path.file_name()?, &text, &mut open) } - fn visit(&self, name: &str, open: &mut Vec) -> Sha256Digest { + fn visit(&self, name: &str, open: &mut Vec) -> Option { if let Some(identity) = self .known .lock() @@ -350,10 +509,23 @@ impl Session { .iter() .find_map(|root| std::fs::read(root.join(&relative).as_std_path()).ok()) else { - return Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes()); + // The toolchain's own module: its bytes are bound by the + // toolchain's digest in every key's context. Any other module + // the stage does not hold is unbound. + let library = self + .toolchain + .join("lib") + .join("lean") + .join(format!("{}.olean", name.replace('.', "/"))); + return library + .as_std_path() + .is_file() + .then(|| Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes())); }; if open.iter().any(|other| other == name) { - return Sha256Digest::of(format!("{DOMAIN}\0cycle\0{name}").as_bytes()); + return Some(Sha256Digest::of( + format!("{DOMAIN}\0cycle\0{name}").as_bytes(), + )); } open.push(name.to_owned()); let identity = self.of_text(name, &text, open); @@ -365,17 +537,17 @@ impl Session { identity } - fn of_text(&self, label: &str, text: &[u8], open: &mut Vec) -> Sha256Digest { + fn of_text(&self, label: &str, text: &[u8], open: &mut Vec) -> Option { let mut bytes = format!( "{DOMAIN}\0module\0{label}\0{}\0", Sha256Digest::of(text).to_hex() ) .into_bytes(); for import in imports(&String::from_utf8_lossy(text)) { - let identity = self.visit(&import, open); + let identity = self.visit(&import, open)?; bytes.extend_from_slice(format!("{import}\0{}\0", identity.to_hex()).as_bytes()); } - Sha256Digest::of(&bytes) + Some(Sha256Digest::of(&bytes)) } /// Run the process `spec` about the content `identity` unless the store @@ -397,6 +569,19 @@ impl Session { .iter() .map(|argument| normalizer.normalize_arg(argument)) .collect(); + // A context that cannot be read whole binds nothing: the process + // runs and is not stored. + let Some(context) = self.context(spec, normalizer) else { + return crate::verify::child::run(spec, limits, normalizer); + }; + let identity = Sha256Digest::of( + format!( + "{DOMAIN}\0bound\0{}\0{}\0", + identity.to_hex(), + context.to_hex() + ) + .as_bytes(), + ); let key = key(spec.tool, spec.executable_sha256, &argv, identity); stored( Some(&self.store), @@ -441,32 +626,41 @@ mod tests { ); } - #[test] - fn an_identity_follows_every_module_an_import_reaches() { - let root = tempfile::tempdir().expect("scratch"); - let base = Utf8PathBuf::from_path_buf(root.path().to_path_buf()).expect("utf-8"); - let write = |name: &str, text: &str| { - let path = base.join(format!("{}.lean", name.replace('.', "/"))); - std::fs::create_dir_all(path.parent().expect("parent").as_std_path()).expect("dir"); - std::fs::write(path.as_std_path(), text).expect("write"); - }; - write("P.A", "module\npublic import Init\ndef a := 1\n"); - write("P.B", "module\npublic import P.A\ndef b := 2\n"); - write("P.C", "module\npublic import P.B\ndef c := 3\n"); - let before = Session::new( + fn session(roots: Vec, toolchain: &Utf8Path) -> Session { + Session::new( Store { root: PathBuf::new(), }, - vec![base.clone()], - ); + roots, + toolchain.to_path_buf(), + ) + } + + fn write(base: &Utf8Path, relative: &str, text: &str) { + let path = base.join(relative); + std::fs::create_dir_all(path.parent().expect("parent").as_std_path()).expect("dir"); + std::fs::write(path.as_std_path(), text).expect("write"); + } + + fn scratch() -> (tempfile::TempDir, Utf8PathBuf) { + let root = tempfile::tempdir().expect("scratch"); + let base = Utf8PathBuf::from_path_buf(root.path().to_path_buf()).expect("utf-8"); + (root, base) + } + + #[test] + fn an_identity_follows_every_module_an_import_reaches() { + let (_root, base) = scratch(); + let (src, toolchain) = (base.join("src"), base.join("toolchain")); + write(&toolchain, "lib/lean/Init.olean", "init"); + write(&src, "P/A.lean", "module\npublic import Init\ndef a := 1\n"); + write(&src, "P/B.lean", "module\npublic import P.A\ndef b := 2\n"); + write(&src, "P/C.lean", "module\npublic import P.B\ndef c := 3\n"); + let before = session(vec![src.clone()], &toolchain); let (a, c) = (before.module("P.A"), before.module("P.C")); - write("P.A", "module\npublic import Init\ndef a := 4\n"); - let after = Session::new( - Store { - root: PathBuf::new(), - }, - vec![base.clone()], - ); + assert!(a.is_some() && c.is_some()); + write(&src, "P/A.lean", "module\npublic import Init\ndef a := 4\n"); + let after = session(vec![src.clone()], &toolchain); assert_ne!(after.module("P.A"), a); assert_ne!( after.module("P.C"), @@ -475,4 +669,50 @@ mod tests { ); assert_eq!(after.module("Init"), before.module("Init")); } + + #[test] + fn a_module_neither_the_stage_nor_the_toolchain_holds_is_unbound() { + let (_root, base) = scratch(); + let (src, toolchain) = (base.join("src"), base.join("toolchain")); + write(&toolchain, "lib/lean/Init.olean", "init"); + write( + &src, + "P/A.lean", + "module\npublic import Elsewhere.Built\ndef a := 1\n", + ); + write(&src, "P/B.lean", "module\npublic import P.A\ndef b := 2\n"); + let session = session(vec![src.clone()], &toolchain); + assert_eq!(session.module("Elsewhere.Built"), None); + assert_eq!(session.module("P.A"), None); + assert_eq!(session.module("P.B"), None, "unbound through every import"); + assert!(session.module("Init").is_some()); + } + + #[test] + fn the_toolchain_digest_follows_every_file_a_process_loads() { + let (_root, base) = scratch(); + write(&base, "bin/lean", "lean"); + write(&base, "lib/libleanshared.so", "shared"); + write(&base, "lib/lean/Init/Core.olean", "core"); + write(&base, "lib/lean/Init/Core.ir", "ir"); + write(&base, "lib/lean/Init/Core.ilean", "server index"); + let first = tree(&base).expect("readable"); + write(&base, "lib/lean/Init/Core.ilean", "another index"); + assert_eq!( + tree(&base), + Some(first), + "a file no process loads is not bound" + ); + for (relative, text) in [ + ("lib/lean/Init/Core.olean", "changed"), + ("lib/lean/Init/Core.ir", "changed"), + ("lib/libleanshared.so", "changed"), + ("bin/lean", "changed"), + ("lib/lean/Init/New.olean.private", "new"), + ] { + let before = tree(&base); + write(&base, relative, text); + assert_ne!(tree(&base), before, "{relative} is bound"); + } + } } From 199243705c9eb62f3d8af8567362ff1cf0d0d772 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 09:03:03 +0000 Subject: [PATCH 5/8] Certificate A is generated beside its root's renderings A and the B and E certificates are made from the lowered program and the linked modules alone and neither reads the other; A's result is judged first, as when they were generated one after the other. Generation of the uorc client's three roots took 344 s with A, B and E one after another. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- crates/lexlean/src/verify/mod.rs | 173 +++++++++++++++++-------------- 1 file changed, 95 insertions(+), 78 deletions(-) diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index be2d7902ba..856e2fbe82 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -900,87 +900,104 @@ pub fn run_part( Sha256Digest::of(&bytes).to_hex()[..32].to_owned() }; let module = format!("LexLeanPreserve.C{root_hex32}.R{index}"); - let generated = certificate( - &linked, - &root.module, - &root.name, - root.report, - &lowered, - &module, - limits.max_file_bytes, - ) - .map_err(fail)?; - // Certificate B: the root's crate in each of its targets - // simulates the lowered program. - let mut renderings = Vec::new(); - for row in &root.report.targets { - let profile = - crate::calculus::rust::Profile::named(&row.target).ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust profile", row.target))) - })?; - let rendered = - crate::calculus::rust::lower(&lowered.program, profile).map_err(|reason| { - fail(internal(format!( - "{} has no rendering in {}: {reason}", - root.report.root, row.target - ))) - })?; - let module_b = rust_cert::module_for(&module, &row.target).ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust target", row.target))) - })?; - let certificate_b = rust_cert::certificate_b( - &lowered.program, - &rendered, - &module_b, - limits.max_file_bytes, - ) - .map_err(|reason| match reason.strip_prefix(crate::production::lower::LIMIT) { - Some(limit) => fail(Diagnostic::new( - code!("LLS8002"), - format!( - "root `{}`: max_file_bytes exceeded in phase certificates: certificate B of its {} rendering: {limit}", - root.report.root, row.target - ), - )), - None => fail(Diagnostic::new( - code!("LLV7015"), - format!( - "certificate B: `{module_b}`: no derivation relates {} to its {} rendering: {reason}", - root.report.root, row.target - ), - )), - })?; - // Certificate E: A and B composed, for the root's rendering. - let module_e = rust_cert::module_for(&format!("{module}.Compose"), &row.target) - .ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust target", row.target))) - })?; - let fallible = crate::calculus::rust::fallible_functions(&lowered.program) - .map_err(|reason| fail(internal(reason)))? - .get(lowered.entry() as usize) - .copied() - .ok_or_else(|| fail(internal("a lowered program without its entry")))?; - let composed = crate::production::certificate::certificate_e( - &linked, - &root.module, - &root.name, - &lowered, - &module, + // Certificate A and the renderings (B and E) are made from the + // lowered program and the linked modules alone, and neither reads + // the other: A is generated beside them, and its result is judged + // first, as when they were generated one after the other. + let (generated, renderings) = std::thread::scope(|scope| { + let generated_a = scope.spawn(|| { + certificate( + &linked, + &root.module, + &root.name, + root.report, + &lowered, + &module, + limits.max_file_bytes, + ) + }); + let renderings = (|| -> Result, LexLeanError> { + // Certificate B: the root's crate in each of its targets + // simulates the lowered program. + let mut renderings = Vec::new(); + for row in &root.report.targets { + let profile = crate::calculus::rust::Profile::named(&row.target) + .ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust profile", row.target))) + })?; + let rendered = crate::calculus::rust::lower(&lowered.program, profile) + .map_err(|reason| { + fail(internal(format!( + "{} has no rendering in {}: {reason}", + root.report.root, row.target + ))) + })?; + let module_b = + rust_cert::module_for(&module, &row.target).ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust target", row.target))) + })?; + let certificate_b = rust_cert::certificate_b( + &lowered.program, + &rendered, &module_b, - &module_e, - fallible, limits.max_file_bytes, ) - .map_err(fail)?; - renderings.push(CertifiedRendering { - target: row.target.clone(), - certificate: certificate_b.into_certificate(), - composed, - crate_text: crate::calculus::rust::ast::print(&rendered) - .map_err(|reason| fail(internal(reason)))?, - entry: lowered.entry(), - }); - } + .map_err(|reason| match reason.strip_prefix(crate::production::lower::LIMIT) { + Some(limit) => fail(Diagnostic::new( + code!("LLS8002"), + format!( + "root `{}`: max_file_bytes exceeded in phase certificates: certificate B of its {} rendering: {limit}", + root.report.root, row.target + ), + )), + None => fail(Diagnostic::new( + code!("LLV7015"), + format!( + "certificate B: `{module_b}`: no derivation relates {} to its {} rendering: {reason}", + root.report.root, row.target + ), + )), + })?; + // Certificate E: A and B composed, for the root's rendering. + let module_e = + rust_cert::module_for(&format!("{module}.Compose"), &row.target) + .ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust target", row.target))) + })?; + let fallible = crate::calculus::rust::fallible_functions(&lowered.program) + .map_err(|reason| fail(internal(reason)))? + .get(lowered.entry() as usize) + .copied() + .ok_or_else(|| fail(internal("a lowered program without its entry")))?; + let composed = crate::production::certificate::certificate_e( + &linked, + &root.module, + &root.name, + &lowered, + &module, + &module_b, + &module_e, + fallible, + limits.max_file_bytes, + ) + .map_err(fail)?; + renderings.push(CertifiedRendering { + target: row.target.clone(), + certificate: certificate_b.into_certificate(), + composed, + crate_text: crate::calculus::rust::ast::print(&rendered) + .map_err(|reason| fail(internal(reason)))?, + entry: lowered.entry(), + }); + } + Ok(renderings) + })(); + let generated = generated_a + .join() + .unwrap_or_else(|panic| std::panic::resume_unwind(panic)) + .map_err(fail)?; + Ok::<_, LexLeanError>((generated, renderings?)) + })?; // Every certificate held as data weighs what it is recorded as, // the notation: the Lean text is the core backend's rendering, // and its size is the backend's, not the certificate's. From aef1279f0f485cf457ee2be46f1d9604cd7d4ef5 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 09:12:48 +0000 Subject: [PATCH 6/8] The honesty and lint gates: explicit arms where a default classified constructs Certificate::released names the certificate without data; the named-constant filter of certificate A names every core node; the Option pair in Update's evaluation is a zip; the prefix test uses contains. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- crates/lexlean/src/calculus/interp.rs | 2 +- crates/lexlean/src/production/certificate.rs | 10 +++++++--- crates/lexlean/src/production/certificate/data_a.rs | 11 ++++++++++- crates/lexlean/src/verify/mod.rs | 2 +- 4 files changed, 19 insertions(+), 6 deletions(-) diff --git a/crates/lexlean/src/calculus/interp.rs b/crates/lexlean/src/calculus/interp.rs index ad3727a4e1..3c764129c4 100644 --- a/crates/lexlean/src/calculus/interp.rs +++ b/crates/lexlean/src/calculus/interp.rs @@ -239,7 +239,7 @@ fn primitive(operation: &Prim, values: &[Rv]) -> Out { usize::try_from(*i) .ok() .filter(|at| *at < a.len()) - .and_then(|at| u8::try_from(*byte).ok().map(|value| (at, value))) + .zip(u8::try_from(*byte).ok()) .map(|(at, value)| { let mut out = a.clone(); out[at] = value; diff --git a/crates/lexlean/src/production/certificate.rs b/crates/lexlean/src/production/certificate.rs index c50c138425..0e2c446bfe 100644 --- a/crates/lexlean/src/production/certificate.rs +++ b/crates/lexlean/src/production/certificate.rs @@ -100,9 +100,13 @@ impl Certificate { pub fn released(mut self) -> Self { self.text = String::new(); self.source = None; - if let Some(data) = &mut self.data { - data.nodes = Vec::new(); - data.proof_nodes = Vec::new(); + match &mut self.data { + Some(data) => { + data.nodes = Vec::new(); + data.proof_nodes = Vec::new(); + } + // A certificate without data has no expression DAG to release. + None => {} } self } diff --git a/crates/lexlean/src/production/certificate/data_a.rs b/crates/lexlean/src/production/certificate/data_a.rs index 90df121c98..40b532b000 100644 --- a/crates/lexlean/src/production/certificate/data_a.rs +++ b/crates/lexlean/src/production/certificate/data_a.rs @@ -645,7 +645,16 @@ impl Gen<'_> { .iter() .filter_map(|node| match node { crate::ir::core::CoreNode::Const { n, .. } => Some(n.as_str()), - _ => None, + crate::ir::core::CoreNode::BVar { .. } + | crate::ir::core::CoreNode::Sort { .. } + | crate::ir::core::CoreNode::App { .. } + | crate::ir::core::CoreNode::Lambda { .. } + | crate::ir::core::CoreNode::Forall { .. } + | crate::ir::core::CoreNode::Let { .. } + | crate::ir::core::CoreNode::Nat { .. } + | crate::ir::core::CoreNode::String { .. } + | crate::ir::core::CoreNode::Mdata { .. } + | crate::ir::core::CoreNode::Projection { .. } => None, }) .collect(); let owner = |constant: &str| -> Option { diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index 856e2fbe82..797f847a4f 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -2643,7 +2643,7 @@ pub fn run_part( .iter() .enumerate() .filter(|(_, other)| { - imports.iter().any(|import| *import == other.module) + imports.contains(&other.module) // `leanchecker` replays a module prefix (§22.4): // a certificate inside another's namespace (B and // E inside their root's A) starts only after that From eba5d304753d84f2cf63dc444cd32e80e01e8269 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 09:16:27 +0000 Subject: [PATCH 7/8] cargo fmt over the files the base left unformatted Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- .../src/cases/bootstrap/calculus_check.rs | 8 +------- crates/lexlean/src/calculus/mod.rs | 8 ++++++-- crates/lexlean/src/ir/semantic.rs | 5 ++++- .../src/production/certificate/data_a/prove.rs | 14 ++++++++------ 4 files changed, 19 insertions(+), 16 deletions(-) diff --git a/crates/conformance/src/cases/bootstrap/calculus_check.rs b/crates/conformance/src/cases/bootstrap/calculus_check.rs index 180f43b83a..e90df3a3a3 100644 --- a/crates/conformance/src/cases/bootstrap/calculus_check.rs +++ b/crates/conformance/src/cases/bootstrap/calculus_check.rs @@ -1382,13 +1382,7 @@ fn signature(operation: &Prim, pool: &[Ty], seeded: &mut Seeded) -> Vec { Prim::Join => vec![list_of(string()), string()], Prim::FormatDecimal | Prim::Convert { .. } => vec![seeded.pick(pool).clone()], Prim::ParseDecimal { .. } => vec![string()], - Prim::Update => vec![ - bytes(), - nat(), - Ty::Fixed { - width: IntKind::U8, - }, - ], + Prim::Update => vec![bytes(), nat(), Ty::Fixed { width: IntKind::U8 }], } } diff --git a/crates/lexlean/src/calculus/mod.rs b/crates/lexlean/src/calculus/mod.rs index 66e4587f15..e354126a2d 100644 --- a/crates/lexlean/src/calculus/mod.rs +++ b/crates/lexlean/src/calculus/mod.rs @@ -337,8 +337,12 @@ pub enum Prim { Join, FormatDecimal, Compare, - Convert { target: IntKind }, - ParseDecimal { target: Ty }, + Convert { + target: IntKind, + }, + ParseDecimal { + target: Ty, + }, /// One byte of a byte string replaced: `none` when the position is out /// of range (language 1.2). Update, diff --git a/crates/lexlean/src/ir/semantic.rs b/crates/lexlean/src/ir/semantic.rs index 1aa5e5da4e..2081b6c47f 100644 --- a/crates/lexlean/src/ir/semantic.rs +++ b/crates/lexlean/src/ir/semantic.rs @@ -9864,7 +9864,10 @@ mod tests { }; let rejected = parse(&module("lexlean/semantic-module/1"), "1.1") .expect_err("a language-1.1 module has no evaluation proof"); - assert!(format!("{rejected:?}").contains("evaluation proof"), "{rejected:?}"); + assert!( + format!("{rejected:?}").contains("evaluation proof"), + "{rejected:?}" + ); parse(&module("lexlean/semantic-module/2"), "1.2") .expect("a language-1.2 theorem may be closed by evaluation"); } diff --git a/crates/lexlean/src/production/certificate/data_a/prove.rs b/crates/lexlean/src/production/certificate/data_a/prove.rs index 566aac14bd..eaa7232cec 100644 --- a/crates/lexlean/src/production/certificate/data_a/prove.rs +++ b/crates/lexlean/src/production/certificate/data_a/prove.rs @@ -1386,12 +1386,14 @@ impl<'a> Writer<'_, 'a> { self.b.konst("Bool.false_ne_true") }; let lemma = self.b.konst1(if polarity { "if_pos" } else { "if_neg" }); - let fits_eq = self - .b - .apps(lemma, &[condition, decide, witness, bool, then_fits, else_fits]); - let src_eq = self - .b - .apps(lemma, &[condition, decide, witness, lean, then_src, else_src]); + let fits_eq = self.b.apps( + lemma, + &[condition, decide, witness, bool, then_fits, else_fits], + ); + let src_eq = self.b.apps( + lemma, + &[condition, decide, witness, lean, then_src, else_src], + ); let selected_fits = self.ite(bool, side, then_fits, else_fits); let selected_src = self.ite(lean, side, then_src, else_src); let selected_value = self.b.app(encoder, selected_src); From c06c769c044335a3e87d5d1ac96ee255687f493b Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 16 Nov 2026 10:56:49 +0000 Subject: [PATCH 8/8] A tree digest follows symbolic links as the processes reading the tree do A library directory linked into the toolchain was skipped by the traversal while import eligibility followed the link; both now follow it. A link that does not resolve, or a directory reached twice (a cycle or an alias), makes the tree unreadable, which is a miss. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_013Asa3QjGTCZCDMLxRDZ33Q --- SPEC.md | 2 +- crates/lexlean/src/verify/store.rs | 52 +++++++++++++++++++++++++++++- 2 files changed, 52 insertions(+), 2 deletions(-) diff --git a/SPEC.md b/SPEC.md index 0fe8ade915..04caec55a9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9232,7 +9232,7 @@ A verification process is a function of what it reads, so the verifier may take The content identity of a staged Lean module is the SHA-256 of the domain `lexlean/verify-store/1`, its name, the digest of its staged text, and, for each module its import header names (the `import` commands before its first other command, with their `public`, `private`, `meta` and `all` modifiers), that module's name and content identity. A module the stage does not hold is bound only when the pinned toolchain's library holds its `.olean`; it is then identified by its name, and its bytes by the toolchain digest below. A module neither holds is unbound, and so is every module whose imports reach it: a process about an unbound module is not answered by the store and is not stored. -The context of one process is the SHA-256 of the domain, the digest of the program it starts, the toolchain digest, its normalized toolchain `bin` directory and working directory (§22.7), the digest of each of `lakefile.lean`, `lakefile.toml`, `lake-manifest.json` and `lean-toolchain` in its working directory (or that it is absent), the tree digest of the working directory's `.lake` (or that it is absent), and its environment rows with normalized values. The toolchain digest is the tree digest of the toolchain directory, and a tree digest is the SHA-256 of the domain and, in path order, the relative path and digest of every file under it a `lean`, `lake` or `leanchecker` process loads: the executables under `bin`, the shared libraries, and every `.olean`, `.olean.server`, `.olean.private` and `.ir`. A process whose context cannot be read whole, or that does not run under the toolchain's home, is not answered by the store. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the digest of the content identity of the module (or, for an audit member, the extraction driver and the interface probe, of its source) it is about together with its context, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), the interface probe (§18.8), the named-root extraction (§22.10), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. +The context of one process is the SHA-256 of the domain, the digest of the program it starts, the toolchain digest, its normalized toolchain `bin` directory and working directory (§22.7), the digest of each of `lakefile.lean`, `lakefile.toml`, `lake-manifest.json` and `lean-toolchain` in its working directory (or that it is absent), the tree digest of the working directory's `.lake` (or that it is absent), and its environment rows with normalized values. The toolchain digest is the tree digest of the toolchain directory, and a tree digest is the SHA-256 of the domain and, in path order, the relative path and digest of every file under it a `lean`, `lake` or `leanchecker` process loads: the executables under `bin`, the shared libraries, and every `.olean`, `.olean.server`, `.olean.private` and `.ir`. The traversal follows symbolic links, as the processes reading the tree do; a link that does not resolve, or a directory reached a second time, makes the tree unreadable. A process whose context cannot be read whole, or that does not run under the toolchain's home, is not answered by the store. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the digest of the content identity of the module (or, for an audit member, the extraction driver and the interface probe, of its source) it is about together with its context, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), the interface probe (§18.8), the named-root extraction (§22.10), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. Only a process that exited 0 is stored, with its normalized record and every file it wrote beside its `.olean` (the companions of that file's stem). A stored process is used only when its record names this run's digest of the executable and every stored file has the digest it was stored with; its files are restored to where the process writes them and its record is judged as the stage judges its own (§22.3 to §22.6), so a stored process that a stage would refuse is refused. Anything else, an unreadable or partial entry included, is a miss and the process runs. A failure to store is dropped, since the store never decides a result. Telemetry (§22.12) reports a stored process as a `reused` event of its stage. diff --git a/crates/lexlean/src/verify/store.rs b/crates/lexlean/src/verify/store.rs index 9865b34515..48ed698821 100644 --- a/crates/lexlean/src/verify/store.rs +++ b/crates/lexlean/src/verify/store.rs @@ -294,14 +294,23 @@ fn loaded(relative: &str) -> bool { /// The digest of every loaded file under `root`, by its path relative to /// `root`, or `None` when the tree cannot be read whole. +/// +/// The traversal follows symbolic links as a process reading the tree does, +/// so a library directory linked from elsewhere is bound like one inside it. +/// A link that does not resolve, or a directory reached a second time (a +/// link cycle or an alias), makes the tree unreadable: `None`, a miss. fn tree(root: &Utf8Path) -> Option { let mut files: Vec = Vec::new(); + let mut seen: std::collections::BTreeSet = std::collections::BTreeSet::new(); let mut stack = vec![root.to_path_buf()]; while let Some(directory) = stack.pop() { + if !seen.insert(std::fs::canonicalize(directory.as_std_path()).ok()?) { + return None; + } for entry in std::fs::read_dir(directory.as_std_path()).ok()? { let entry = entry.ok()?; let path = Utf8PathBuf::from_path_buf(entry.path()).ok()?; - let kind = entry.file_type().ok()?; + let kind = std::fs::metadata(path.as_std_path()).ok()?.file_type(); if kind.is_dir() { stack.push(path); } else { @@ -688,6 +697,47 @@ mod tests { assert!(session.module("Init").is_some()); } + #[cfg(unix)] + #[test] + fn a_linked_library_is_bound_and_a_link_cycle_is_unreadable() { + let (_root, base) = scratch(); + let (toolchain, library) = (base.join("toolchain"), base.join("library")); + write(&toolchain, "bin/lean", "lean"); + write(&library, "Init.olean", "init"); + std::fs::create_dir_all(toolchain.join("lib").as_std_path()).expect("dir"); + std::os::unix::fs::symlink( + library.as_std_path(), + toolchain.join("lib/lean").as_std_path(), + ) + .expect("link"); + let session = session(Vec::new(), &toolchain); + assert!( + session.module("Init").is_some(), + "the linked library holds Init" + ); + let before = tree(&toolchain).expect("readable"); + write(&library, "Init.olean", "changed"); + assert_ne!( + tree(&toolchain), + Some(before), + "a file under the linked library is bound" + ); + std::os::unix::fs::symlink(toolchain.as_std_path(), library.join("back").as_std_path()) + .expect("cycle"); + assert_eq!(tree(&toolchain), None, "a link cycle is a miss"); + std::fs::remove_file(library.join("back").as_std_path()).expect("unlink"); + std::os::unix::fs::symlink( + base.join("nowhere").as_std_path(), + library.join("dangling").as_std_path(), + ) + .expect("dangling"); + assert_eq!( + tree(&toolchain), + None, + "a link that does not resolve is a miss" + ); + } + #[test] fn the_toolchain_digest_follows_every_file_a_process_loads() { let (_root, base) = scratch();