diff --git a/SPEC.md b/SPEC.md index 84cfaa287d..04caec55a9 100644 --- a/SPEC.md +++ b/SPEC.md @@ -5141,8 +5141,15 @@ certificate generator, and their shared source reader obey §17.13's exhaustiven audit: they name every construct and match none by default. **Statement.** The certificate of the root `r` with parameters `x₁ … xₙ` is -the module `LexLeanPreserve.C.R`, `i` the -root's position in module then report order, and proves +the module `LexLeanPreserve.C.R`, `i` the root's position in module +then report order and `` the first 32 hex digits of the SHA-256 of +`lexlean/certificate-name/1`, NUL, the root's module, NUL, its name, NUL, and +the bytes of its lowered program (`preserve/program/R.json`): the name is a +function of what the certificates are about, so a root whose program a change +of the project leaves as it was keeps its certificates' names. Certificate A +imports the preservation library and those generated user modules whose +constants its terms name, and no other (Lean's imports are transitive, so the +modules those constants are defined from come with them). It proves ```lean theorem root (x₁ … xₙ) : @@ -9207,6 +9214,7 @@ The profile changes how processes are scheduled and nothing else: 3. Results are consumed in the canonical order and never in completion order: the members of a frontier in module-declaration order, replays in sorted full module name, audit members in the order of the audit family. The process records, the staged files, the merge of the observed axiom sets, the policy rows, and the attestation body are functions of that order. When several processes fail, the diagnostic reported is the one of the first failing process in that order, which is the diagnostic width 1 reports; no process later in that order than a known failure is started, and every process started is finished before `verify` returns. Width 1 is exactly the sequence the verifier ran before there was a width: a process starts only after the result of the one before it has been consumed, so a failure that only the consumer can see (a missing `.olean`, a staging error) stops the stage before another process starts. 4. The verifier does not measure the host. A width above 1 is for a host whose memory envelope is known; a width the host cannot hold is a failure of that host's run, never a verification result. The normative job's `examples` leg names its profile (§9.4); every other run uses the default unless its environment selects one, and the conformance cases exercise widths above 1 on small projects. 5. Hints and the memory budget are schedule inputs, never verdicts. A hint only reorders the start of items inside a stage, and the memory budget only gates a worker from starting an item whose expected peak (plus the expected peaks of the running items) would pass it. An item whose expected peak alone passes the budget runs alone rather than waiting forever: a wrong hint is a wrong schedule, never a fence and never a kill. A process that outgrows its hinted peak runs to completion; the stage measures it for the next hints. Hints change which ready item a worker starts first; results are consumed in the canonical order exactly as at width 1, so a failure reported is still the first failure in that order. +6. Certificates (§17.17, stage 12) compile as one stage with the preservation environment they import, in the topological frontiers of their import headers, like module elaboration: the certificates A and B of every root import nothing from each other and run together, at most `W` at a time, and a certificate E starts when the A and B it composes have finished. Each certificate's `leanchecker` replay runs in the slot of its compilation, right after it. The records are listed in the stage's canonical order (the environment's compilations, then each certificate's compilation and replay, in certificate order) whatever order the processes ran in. The label of a certificate's processes, for the hints, is its module name without its `C` segment, so a root's measured cost schedules its next certificates. For every project, the directory published at any profile equals, byte for byte, the directory published at width 1 without hints and without a budget, on the same host and toolchain; in particular the attestation ID, the observed axiom sets, the policy rows, the canonical compiler input, and every normalized verification record (§29.5) are identical. @@ -9218,6 +9226,16 @@ Wall time, resident-set and swap measurements, scheduling and completion order, Telemetry is write-only: no stage reads it, no ID or verified artifact contains any of it, and a failure to write it changes no result. Every key it uses is in a closed vocabulary (`wall_ms`, `started_unix_ms`, `peak_rss_kib`, `peak_swap_kib`, `mem_total_kib`, `mem_available_kib`, `swap_total_kib`, `swap_used_kib`, `host_cpus`, `process_width`, `completion_seq`), and a JSON key of that vocabulary anywhere in a verified directory fails the gate: `cargo xtask verify-examples` checks every example's verified directory for one. +### 22.13 Process store + +A verification process is a function of what it reads, so the verifier may take a process another run on the same host already established instead of running it again. The store is selected only by the environment variable `LEXLEAN_VERIFY_STORE`, the path of a directory; an unset or empty variable selects none. Like the resource profile (§22.11) it is operational: it is no field of `lexlean.toml`, the lock, a request, or the command line, no artifact records it, it enters no ID, and the directory a verification publishes with a store equals, byte for byte, the one it publishes without. + +The content identity of a staged Lean module is the SHA-256 of the domain `lexlean/verify-store/1`, its name, the digest of its staged text, and, for each module its import header names (the `import` commands before its first other command, with their `public`, `private`, `meta` and `all` modifiers), that module's name and content identity. A module the stage does not hold is bound only when the pinned toolchain's library holds its `.olean`; it is then identified by its name, and its bytes by the toolchain digest below. A module neither holds is unbound, and so is every module whose imports reach it: a process about an unbound module is not answered by the store and is not stored. + +The context of one process is the SHA-256 of the domain, the digest of the program it starts, the toolchain digest, its normalized toolchain `bin` directory and working directory (§22.7), the digest of each of `lakefile.lean`, `lakefile.toml`, `lake-manifest.json` and `lean-toolchain` in its working directory (or that it is absent), the tree digest of the working directory's `.lake` (or that it is absent), and its environment rows with normalized values. The toolchain digest is the tree digest of the toolchain directory, and a tree digest is the SHA-256 of the domain and, in path order, the relative path and digest of every file under it a `lean`, `lake` or `leanchecker` process loads: the executables under `bin`, the shared libraries, and every `.olean`, `.olean.server`, `.olean.private` and `.ir`. The traversal follows symbolic links, as the processes reading the tree do; a link that does not resolve, or a directory reached a second time, makes the tree unreadable. A process whose context cannot be read whole, or that does not run under the toolchain's home, is not answered by the store. The key of one process is the SHA-256 of the domain, its tool, the digest of its executable, the digest of the content identity of the module (or, for an audit member, the extraction driver and the interface probe, of its source) it is about together with its context, and its normalized argument vector (§22.7). The elaborations (§22.3), the replays (§22.4), the audit members (§22.5), the interface probe (§18.8), the named-root extraction (§22.10), and the compilations and replays of the preservation environment and the certificates (§17.17) go through the store. + +Only a process that exited 0 is stored, with its normalized record and every file it wrote beside its `.olean` (the companions of that file's stem). A stored process is used only when its record names this run's digest of the executable and every stored file has the digest it was stored with; its files are restored to where the process writes them and its record is judged as the stage judges its own (§22.3 to §22.6), so a stored process that a stage would refuse is refused. Anything else, an unreadable or partial entry included, is a miss and the process runs. A failure to store is dropped, since the store never decides a result. Telemetry (§22.12) reports a stored process as a `reused` event of its stage. + --- ## 23. Command-line interface @@ -9238,7 +9256,7 @@ Global options: --help ``` -No environment variable changes semantic configuration. Four operational variables exist and change no output: `LEXLEAN_VERIFY_WIDTH` selects the verification process width, `LEXLEAN_VERIFY_HINTS` names the scheduling-hints file and `LEXLEAN_VERIFY_MEMORY_KIB` the process memory budget (§22.11), and `LEXLEAN_CI_TELEMETRY` names the directory that receives performance telemetry (§22.12). +No environment variable changes semantic configuration. Five operational variables exist and change no output: `LEXLEAN_VERIFY_WIDTH` selects the verification process width, `LEXLEAN_VERIFY_HINTS` names the scheduling-hints file and `LEXLEAN_VERIFY_MEMORY_KIB` the process memory budget (§22.11), `LEXLEAN_CI_TELEMETRY` names the directory that receives performance telemetry (§22.12), and `LEXLEAN_VERIFY_STORE` names the process store (§22.13). ### 23.2 Project discovery diff --git a/crates/conformance/src/cases/bootstrap/calculus_check.rs b/crates/conformance/src/cases/bootstrap/calculus_check.rs index 180f43b83a..e90df3a3a3 100644 --- a/crates/conformance/src/cases/bootstrap/calculus_check.rs +++ b/crates/conformance/src/cases/bootstrap/calculus_check.rs @@ -1382,13 +1382,7 @@ fn signature(operation: &Prim, pool: &[Ty], seeded: &mut Seeded) -> Vec { Prim::Join => vec![list_of(string()), string()], Prim::FormatDecimal | Prim::Convert { .. } => vec![seeded.pick(pool).clone()], Prim::ParseDecimal { .. } => vec![string()], - Prim::Update => vec![ - bytes(), - nat(), - Ty::Fixed { - width: IntKind::U8, - }, - ], + Prim::Update => vec![bytes(), nat(), Ty::Fixed { width: IntKind::U8 }], } } diff --git a/crates/lexlean/src/calculus/interp.rs b/crates/lexlean/src/calculus/interp.rs index ad3727a4e1..3c764129c4 100644 --- a/crates/lexlean/src/calculus/interp.rs +++ b/crates/lexlean/src/calculus/interp.rs @@ -239,7 +239,7 @@ fn primitive(operation: &Prim, values: &[Rv]) -> Out { usize::try_from(*i) .ok() .filter(|at| *at < a.len()) - .and_then(|at| u8::try_from(*byte).ok().map(|value| (at, value))) + .zip(u8::try_from(*byte).ok()) .map(|(at, value)| { let mut out = a.clone(); out[at] = value; diff --git a/crates/lexlean/src/calculus/mod.rs b/crates/lexlean/src/calculus/mod.rs index 66e4587f15..e354126a2d 100644 --- a/crates/lexlean/src/calculus/mod.rs +++ b/crates/lexlean/src/calculus/mod.rs @@ -337,8 +337,12 @@ pub enum Prim { Join, FormatDecimal, Compare, - Convert { target: IntKind }, - ParseDecimal { target: Ty }, + Convert { + target: IntKind, + }, + ParseDecimal { + target: Ty, + }, /// One byte of a byte string replaced: `none` when the position is out /// of range (language 1.2). Update, diff --git a/crates/lexlean/src/ir/semantic.rs b/crates/lexlean/src/ir/semantic.rs index 1aa5e5da4e..2081b6c47f 100644 --- a/crates/lexlean/src/ir/semantic.rs +++ b/crates/lexlean/src/ir/semantic.rs @@ -9864,7 +9864,10 @@ mod tests { }; let rejected = parse(&module("lexlean/semantic-module/1"), "1.1") .expect_err("a language-1.1 module has no evaluation proof"); - assert!(format!("{rejected:?}").contains("evaluation proof"), "{rejected:?}"); + assert!( + format!("{rejected:?}").contains("evaluation proof"), + "{rejected:?}" + ); parse(&module("lexlean/semantic-module/2"), "1.2") .expect("a language-1.2 theorem may be closed by evaluation"); } diff --git a/crates/lexlean/src/production/certificate.rs b/crates/lexlean/src/production/certificate.rs index 8f368c5fd3..0e2c446bfe 100644 --- a/crates/lexlean/src/production/certificate.rs +++ b/crates/lexlean/src/production/certificate.rs @@ -100,9 +100,13 @@ impl Certificate { pub fn released(mut self) -> Self { self.text = String::new(); self.source = None; - if let Some(data) = &mut self.data { - data.nodes = Vec::new(); - data.proof_nodes = Vec::new(); + match &mut self.data { + Some(data) => { + data.nodes = Vec::new(); + data.proof_nodes = Vec::new(); + } + // A certificate without data has no expression DAG to release. + None => {} } self } @@ -566,9 +570,12 @@ pub fn certificate( generator .within(source.len(), "certificate A") .map_err(failure)?; - let mut imports: Vec = modules - .values() - .map(|linked| linked.lean_module.to_owned()) + let users: BTreeSet<&str> = modules.values().map(|linked| linked.lean_module).collect(); + let mut imports: Vec = data + .imports + .iter() + .filter(|import| users.contains(import.as_str())) + .cloned() .collect(); imports.sort(); Ok(Certificate { diff --git a/crates/lexlean/src/production/certificate/data_a.rs b/crates/lexlean/src/production/certificate/data_a.rs index 022aed2147..40b532b000 100644 --- a/crates/lexlean/src/production/certificate/data_a.rs +++ b/crates/lexlean/src/production/certificate/data_a.rs @@ -630,10 +630,51 @@ impl Gen<'_> { .map(|linked| linked.lean_module) .collect(); users.sort_unstable(); + let users_all: BTreeSet = users.iter().map(|module| (*module).to_owned()).collect(); imports.extend(users.into_iter().map(str::to_owned)); let mut writer = Writer::new(self, name); writer.write(root_module, root_name, report, &graph)?; - writer.finish(imports) + let mut data = writer.finish(imports)?; + // A certificate imports the user modules whose constants its terms + // name, and no other: Lean's imports are transitive, so the modules + // those constants are defined from come with them, and a module the + // root's closure never reaches is not among what the certificate + // reads (§17.17, §22.13). + let named: BTreeSet<&str> = data + .nodes + .iter() + .filter_map(|node| match node { + crate::ir::core::CoreNode::Const { n, .. } => Some(n.as_str()), + crate::ir::core::CoreNode::BVar { .. } + | crate::ir::core::CoreNode::Sort { .. } + | crate::ir::core::CoreNode::App { .. } + | crate::ir::core::CoreNode::Lambda { .. } + | crate::ir::core::CoreNode::Forall { .. } + | crate::ir::core::CoreNode::Let { .. } + | crate::ir::core::CoreNode::Nat { .. } + | crate::ir::core::CoreNode::String { .. } + | crate::ir::core::CoreNode::Mdata { .. } + | crate::ir::core::CoreNode::Projection { .. } => None, + }) + .collect(); + let owner = |constant: &str| -> Option { + users_all + .iter() + .filter(|module| { + constant.len() > module.len() + && constant.starts_with(module.as_str()) + && constant.as_bytes()[module.len()] == b'.' + }) + .max_by_key(|module| module.len()) + .cloned() + }; + let reached: BTreeSet = named + .iter() + .filter_map(|constant| owner(constant)) + .collect(); + data.imports + .retain(|import| !users_all.contains(import) || reached.contains(import)); + Ok(data) } } diff --git a/crates/lexlean/src/production/certificate/data_a/prove.rs b/crates/lexlean/src/production/certificate/data_a/prove.rs index 566aac14bd..eaa7232cec 100644 --- a/crates/lexlean/src/production/certificate/data_a/prove.rs +++ b/crates/lexlean/src/production/certificate/data_a/prove.rs @@ -1386,12 +1386,14 @@ impl<'a> Writer<'_, 'a> { self.b.konst("Bool.false_ne_true") }; let lemma = self.b.konst1(if polarity { "if_pos" } else { "if_neg" }); - let fits_eq = self - .b - .apps(lemma, &[condition, decide, witness, bool, then_fits, else_fits]); - let src_eq = self - .b - .apps(lemma, &[condition, decide, witness, lean, then_src, else_src]); + let fits_eq = self.b.apps( + lemma, + &[condition, decide, witness, bool, then_fits, else_fits], + ); + let src_eq = self.b.apps( + lemma, + &[condition, decide, witness, lean, then_src, else_src], + ); let selected_fits = self.ite(bool, side, then_fits, else_fits); let selected_src = self.ite(lean, side, then_src, else_src); let selected_value = self.b.app(encoder, selected_src); diff --git a/crates/lexlean/src/verify/leanchecker.rs b/crates/lexlean/src/verify/leanchecker.rs index e165e573d5..936f83bbec 100644 --- a/crates/lexlean/src/verify/leanchecker.rs +++ b/crates/lexlean/src/verify/leanchecker.rs @@ -9,7 +9,7 @@ use camino::Utf8Path; use crate::code; use crate::config::Limits; use crate::diagnostic::Diagnostic; -use crate::verify::child::{run, ChildHome, ChildRecord, ChildSpec, Normalizer}; +use crate::verify::child::{ChildHome, ChildRecord, ChildSpec, Normalizer}; use crate::verify::toolchain::Toolchain; /// Run `lake env ` (the preflighted executable by @@ -21,9 +21,11 @@ pub fn run_leanchecker( workspace: &Utf8Path, limits: &Limits, normalizer: &Normalizer, + session: Option<&crate::verify::store::Session>, ) -> Result { let bin_dir = toolchain.root.join("bin"); - run( + crate::verify::store::run( + session, &ChildSpec { tool: "leanchecker", module: Some(module.to_owned()), @@ -45,6 +47,8 @@ pub fn run_leanchecker( }, limits, normalizer, + |session| session.prefix(module), + &[], ) } @@ -57,8 +61,11 @@ pub fn replay_module( workspace: &Utf8Path, limits: &Limits, normalizer: &Normalizer, + session: Option<&crate::verify::store::Session>, ) -> Result { - let record = run_leanchecker(toolchain, module, lean_path, workspace, limits, normalizer)?; + let record = run_leanchecker( + toolchain, module, lean_path, workspace, limits, normalizer, session, + )?; if record.exit_code != 0 { return Err(Diagnostic::new( code!("LLV7003"), diff --git a/crates/lexlean/src/verify/mod.rs b/crates/lexlean/src/verify/mod.rs index e5380da5e0..797f847a4f 100644 --- a/crates/lexlean/src/verify/mod.rs +++ b/crates/lexlean/src/verify/mod.rs @@ -28,6 +28,7 @@ pub mod operational; pub mod partition; pub mod pool; pub mod source_audit; +pub mod store; pub mod telemetry; pub mod toolchain; pub mod workspace; @@ -650,6 +651,7 @@ fn leanchecker_identity( workspace_root, limits, normalizer, + None, )?; let combined = format!("{}{}", record.stdout.trim_end(), record.stderr.trim_end()); let observed = format!("{}\n", combined.trim_end()); @@ -883,88 +885,119 @@ pub fn run_part( use crate::production::rust_cert; for (index, root) in roots(checked).map_err(fail)?.iter().enumerate() { let (lowered, program) = lowered_roots[index].clone(); - let module = format!("LexLeanPreserve.C{semantic_hex32}.R{index}"); - let generated = certificate( - &linked, - &root.module, - &root.name, - root.report, - &lowered, - &module, - limits.max_file_bytes, - ) - .map_err(fail)?; - // Certificate B: the root's crate in each of its targets - // simulates the lowered program. - let mut renderings = Vec::new(); - for row in &root.report.targets { - let profile = - crate::calculus::rust::Profile::named(&row.target).ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust profile", row.target))) - })?; - let rendered = - crate::calculus::rust::lower(&lowered.program, profile).map_err(|reason| { - fail(internal(format!( - "{} has no rendering in {}: {reason}", - root.report.root, row.target - ))) - })?; - let module_b = rust_cert::module_for(&module, &row.target).ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust target", row.target))) - })?; - let certificate_b = rust_cert::certificate_b( - &lowered.program, - &rendered, - &module_b, - limits.max_file_bytes, - ) - .map_err(|reason| match reason.strip_prefix(crate::production::lower::LIMIT) { - Some(limit) => fail(Diagnostic::new( - code!("LLS8002"), - format!( - "root `{}`: max_file_bytes exceeded in phase certificates: certificate B of its {} rendering: {limit}", - root.report.root, row.target - ), - )), - None => fail(Diagnostic::new( - code!("LLV7015"), - format!( - "certificate B: `{module_b}`: no derivation relates {} to its {} rendering: {reason}", - root.report.root, row.target - ), - )), - })?; - // Certificate E: A and B composed, for the root's rendering. - let module_e = rust_cert::module_for(&format!("{module}.Compose"), &row.target) - .ok_or_else(|| { - fail(internal(format!("`{}` is not a Rust target", row.target))) - })?; - let fallible = crate::calculus::rust::fallible_functions(&lowered.program) - .map_err(|reason| fail(internal(reason)))? - .get(lowered.entry() as usize) - .copied() - .ok_or_else(|| fail(internal("a lowered program without its entry")))?; - let composed = crate::production::certificate::certificate_e( - &linked, - &root.module, - &root.name, - &lowered, - &module, + // A root's certificates are named by what they are about, the + // root and its lowered program, and not by the project's semantic + // identity: a root whose program a change leaves as it was keeps + // its certificates' names and text, so their processes keep their + // content identities (§22.13). + let root_hex32: String = { + let mut bytes = format!( + "lexlean/certificate-name/1\0{}\0{}\0", + root.module, root.name + ) + .into_bytes(); + bytes.extend_from_slice(&program); + Sha256Digest::of(&bytes).to_hex()[..32].to_owned() + }; + let module = format!("LexLeanPreserve.C{root_hex32}.R{index}"); + // Certificate A and the renderings (B and E) are made from the + // lowered program and the linked modules alone, and neither reads + // the other: A is generated beside them, and its result is judged + // first, as when they were generated one after the other. + let (generated, renderings) = std::thread::scope(|scope| { + let generated_a = scope.spawn(|| { + certificate( + &linked, + &root.module, + &root.name, + root.report, + &lowered, + &module, + limits.max_file_bytes, + ) + }); + let renderings = (|| -> Result, LexLeanError> { + // Certificate B: the root's crate in each of its targets + // simulates the lowered program. + let mut renderings = Vec::new(); + for row in &root.report.targets { + let profile = crate::calculus::rust::Profile::named(&row.target) + .ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust profile", row.target))) + })?; + let rendered = crate::calculus::rust::lower(&lowered.program, profile) + .map_err(|reason| { + fail(internal(format!( + "{} has no rendering in {}: {reason}", + root.report.root, row.target + ))) + })?; + let module_b = + rust_cert::module_for(&module, &row.target).ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust target", row.target))) + })?; + let certificate_b = rust_cert::certificate_b( + &lowered.program, + &rendered, &module_b, - &module_e, - fallible, limits.max_file_bytes, ) - .map_err(fail)?; - renderings.push(CertifiedRendering { - target: row.target.clone(), - certificate: certificate_b.into_certificate(), - composed, - crate_text: crate::calculus::rust::ast::print(&rendered) - .map_err(|reason| fail(internal(reason)))?, - entry: lowered.entry(), - }); - } + .map_err(|reason| match reason.strip_prefix(crate::production::lower::LIMIT) { + Some(limit) => fail(Diagnostic::new( + code!("LLS8002"), + format!( + "root `{}`: max_file_bytes exceeded in phase certificates: certificate B of its {} rendering: {limit}", + root.report.root, row.target + ), + )), + None => fail(Diagnostic::new( + code!("LLV7015"), + format!( + "certificate B: `{module_b}`: no derivation relates {} to its {} rendering: {reason}", + root.report.root, row.target + ), + )), + })?; + // Certificate E: A and B composed, for the root's rendering. + let module_e = + rust_cert::module_for(&format!("{module}.Compose"), &row.target) + .ok_or_else(|| { + fail(internal(format!("`{}` is not a Rust target", row.target))) + })?; + let fallible = crate::calculus::rust::fallible_functions(&lowered.program) + .map_err(|reason| fail(internal(reason)))? + .get(lowered.entry() as usize) + .copied() + .ok_or_else(|| fail(internal("a lowered program without its entry")))?; + let composed = crate::production::certificate::certificate_e( + &linked, + &root.module, + &root.name, + &lowered, + &module, + &module_b, + &module_e, + fallible, + limits.max_file_bytes, + ) + .map_err(fail)?; + renderings.push(CertifiedRendering { + target: row.target.clone(), + certificate: certificate_b.into_certificate(), + composed, + crate_text: crate::calculus::rust::ast::print(&rendered) + .map_err(|reason| fail(internal(reason)))?, + entry: lowered.entry(), + }); + } + Ok(renderings) + })(); + let generated = generated_a + .join() + .unwrap_or_else(|panic| std::panic::resume_unwind(panic)) + .map_err(fail)?; + Ok::<_, LexLeanError>((generated, renderings?)) + })?; // Every certificate held as data weighs what it is recorded as, // the notation: the Lean text is the core backend's rendering, // and its size is the backend's, not the certificate's. @@ -1273,6 +1306,17 @@ pub fn run_part( )?; } release_non_diagnostic_buffers(build); + // The process store (§22.13), when the environment selects one: the + // staged sources the identities of this run's processes are read from, + // in the order of the search paths their processes read imports from. + let store_session = store::Store::from_env().map(|selected| { + store::Session::new( + selected, + vec![staging_utf8.join("preserve-src"), src_root.clone()], + toolchain.root.clone(), + ) + }); + let store_session = store_session.as_ref(); let lean_path_env = format!("{olean_root}"); // The records of this run's processes, by what they are and the module // or audit member they are of, and the records of the remaining stages @@ -1333,7 +1377,8 @@ pub fn run_part( let probe_source = staging_utf8 .join("probe") .join(format!("{probe_name}.lean")); - let probe_record = run_child( + let probe_record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(probe_name.clone()), @@ -1352,6 +1397,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&probe_source), + &[], ) .map_err(fail)?; if probe_record.exit_code != 0 { @@ -1495,7 +1542,8 @@ pub fn run_part( )) })?; } - let record = run_child( + let record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(module.name.clone()), @@ -1518,6 +1566,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.module(&module.name), + std::slice::from_ref(&olean), ) .map_err(fail)?; if record.exit_code != 0 @@ -1643,7 +1693,8 @@ pub fn run_part( let module_path = module_name.replace('.', "/"); let source = src_root.join(format!("{module_path}.lean")); let olean = olean_root.join(format!("{module_path}.olean")); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(module_name.clone()), @@ -1666,6 +1717,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.module(module_name), + std::slice::from_ref(&olean), ) }, &|result: &Result| { @@ -1744,6 +1797,7 @@ pub fn run_part( &workspace_root, &limits, &normalizer, + store_session, ) .map_err(fail)?; write_staged( @@ -1776,6 +1830,7 @@ pub fn run_part( &workspace_root, &limits, &normalizer, + store_session, ) // A replay failure is about this module, so it points at the // module's source rather than at the project manifest (§20.1). @@ -1828,7 +1883,8 @@ pub fn run_part( let audit = &audit_modules[audit_index]; let source_relative = format!("audit/{}.lean", audit.name); let audit_source = staging_utf8.join(&source_relative); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(audit.name.clone()), @@ -1847,6 +1903,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&audit_source), + &[], ) }, &|result: &Result| { @@ -2288,7 +2346,10 @@ pub fn run_part( let source = staging_utf8 .join("extract") .join(format!("{}.lean", driver.name)); - let record = run_child( + // The driver's report is a function of its text and the modules it + // imports, so the store answers it by that identity (§22.13). + let record = store::run( + store_session, &ChildSpec { tool: "lean", module: Some(driver.name.clone()), @@ -2303,6 +2364,8 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.source(&source), + &[], ) .map_err(fail)?; if record.exit_code != 0 { @@ -2401,8 +2464,7 @@ pub fn run_part( .iter() .map(|module| module.lean_module.clone()) .collect(); - let mut stage = - preserve::stage(&generated_modules, &staged_certificates).map_err(fail)?; + let mut stage = preserve::stage(&generated_modules, &staged_certificates).map_err(fail)?; staged_certificates = staged_certificates .into_iter() .map(crate::production::certificate::Certificate::released) @@ -2502,6 +2564,7 @@ pub fn run_part( }; let compile = |file: &preserve::StagedFile, output: bool| { let source = preserve_src.join(&file.path); + let mut outputs: Vec = Vec::new(); let mut argv = vec![ "env".to_owned(), "lean".to_owned(), @@ -2517,9 +2580,11 @@ pub fn run_part( } argv.push("-o".to_owned()); argv.push(olean.to_string()); + outputs.push(olean); } argv.push(source.to_string()); - run_child( + store::run( + store_session, &ChildSpec { tool: "lean", module: Some(file.module.clone()), @@ -2534,93 +2599,218 @@ pub fn run_part( }, &limits, &normalizer, + |session| session.module(&file.module), + &outputs, ) }; - for file in &stage.environment { - // A runtime module the calculus modules import was compiled, - // replayed, and audited with the project's (stage 7), and its - // olean is on this search path. - if runtime_modules - .iter() - .any(|module| module.name == file.module) - { - continue; - } - let record = compile(file, true).map_err(fail)?; - if record.exit_code != 0 - || !record.stdout.trim().is_empty() - || !record.stderr.trim().is_empty() - { - let combined = format!("{}{}", record.stdout, record.stderr); - if let Some(exhausted) = resource_death( - &file.module, - record.exit_code, - &combined, - file.text.len() as u64, - &limits, - ) { - return Err(fail(exhausted)); - } - return Err(fail(Diagnostic::new( - code!("LLV7014"), - format!( - "preservation environment: `{}` does not compile silently under the pinned Lean: {}", - file.module, - first_error(&format!("{}{}", record.stdout, record.stderr)) - ), + // The environment and the certificates compile as one stage, in the + // topological frontiers of their imports (§22.11): a certificate A or + // B imports the library and the project's modules, a certificate E + // the A and B it composes, so the certificates of different roots, + // and A and B of one root, import nothing from each other and run + // together, at most `W` at a time and within the memory budget. Each + // certificate's `leanchecker` replay runs in the same process slot + // right after its compilation. The records are listed in the stage's + // canonical order (the environment, then each certificate's + // compilation and replay) whatever order the processes ran in, so the + // published directory is the one width 1 publishes. + let environment_items: Vec<&preserve::StagedFile> = stage + .environment + .iter() + .filter(|file| { + !runtime_modules + .iter() + .any(|module| module.name == file.module) + }) + .collect(); + let items: Vec<&preserve::StagedFile> = environment_items + .iter() + .copied() + .chain(&stage.certificates) + .collect(); + let environment_count = environment_items.len(); + let staged_imports = |file: &preserve::StagedFile| -> Vec { + store::imports( + &std::fs::read_to_string(preserve_src.join(&file.path).as_std_path()) + .unwrap_or_default(), + ) + }; + let depends: Vec> = items + .iter() + .map(|file| { + let imports = staged_imports(file); + items + .iter() + .enumerate() + .filter(|(_, other)| { + imports.contains(&other.module) + // `leanchecker` replays a module prefix (§22.4): + // a certificate inside another's namespace (B and + // E inside their root's A) starts only after that + // one's replay, so each replay checks what it + // checks at width 1. + || file + .module + .strip_prefix(other.module.as_str()) + .is_some_and(|rest| rest.starts_with('.')) + }) + .map(|(index, _)| index) + .collect() + }) + .collect(); + // A stable label per item, for the scheduling hints: a certificate's + // module name without its content segment, so that the observed cost + // of a root's certificate schedules the next run's. + let labels: Vec = items + .iter() + .map(|file| { + file.module + .split('.') + .filter(|segment| { + !(segment.len() == 33 + && segment.starts_with('C') + && segment[1..].bytes().all(|byte| byte.is_ascii_hexdigit())) + }) + .collect::>() + .join(".") + }) + .collect(); + type Run = ( + Result, + Option>, + ); + let silent = |record: &ChildRecord| { + record.exit_code == 0 + && record.stdout.trim().is_empty() + && record.stderr.trim().is_empty() + }; + let mut results: Vec)>> = vec![None; items.len()]; + let mut done: BTreeSet = BTreeSet::new(); + let mut frontier_index = 0usize; + while done.len() < items.len() { + let ready: Vec = (0..items.len()) + .filter(|index| !done.contains(index)) + .filter(|index| { + depends[*index] + .iter() + .all(|dependency| done.contains(dependency)) + }) + .collect(); + if ready.is_empty() { + return Err(fail(internal( + "the preservation modules' imports do not converge", ))); } - write_staged( - staging.path(), - &format!("process/preserve/{}.json", file.module), - &record.to_json().to_file_bytes(), + let frontier_started = std::time::Instant::now(); + pool::run_ordered_with( + width, + &ready, + hints, + memory_kib, + &|index: usize| labels[index].clone(), + &|index: usize| -> Run { + let file = items[index]; + let compiled = compile(file, true); + if index < environment_count { + return (compiled, None); + } + match &compiled { + Ok(record) if silent(record) => { + let replay = leanchecker::run_leanchecker( + &toolchain, + &file.module, + &preserve_path, + &workspace_root, + &limits, + &normalizer, + store_session, + ); + (compiled, Some(replay)) + } + _ => (compiled, None), + } + }, + &|run: &Run| match run { + (Ok(record), replay) => { + !silent(record) + || matches!(replay, Some(Err(_))) + || matches!(replay, Some(Ok(replay)) if replay.exit_code != 0) + } + (Err(_), _) => true, + }, + |index, (compiled, replay)| { + let file = items[index]; + let record = compiled.map_err(fail)?; + if !silent(&record) { + let combined = format!("{}{}", record.stdout, record.stderr); + if index < environment_count { + if let Some(exhausted) = resource_death( + &file.module, + record.exit_code, + &combined, + file.text.len() as u64, + &limits, + ) { + return Err(fail(exhausted)); + } + return Err(fail(Diagnostic::new( + code!("LLV7014"), + format!( + "preservation environment: `{}` does not compile silently under the pinned Lean: {}", + file.module, + first_error(&combined) + ), + ))); + } + return Err(fail(reject( + &file.module, + &combined, + record.exit_code, + &staged_text(file), + ))); + } + write_staged( + staging.path(), + &format!("process/preserve/{}.json", file.module), + &record.to_json().to_file_bytes(), + )?; + let replay = match replay { + None => None, + Some(replay) => { + let replay = replay.map_err(fail)?; + if replay.exit_code != 0 { + let combined = format!("{}{}", replay.stdout, replay.stderr); + return Err(fail(reject( + &file.module, + &combined, + replay.exit_code, + &staged_text(file), + ))); + } + write_staged( + staging.path(), + &format!("process/leanchecker/{}.json", file.module), + &replay.to_json().to_file_bytes(), + )?; + Some(replay) + } + }; + results[index] = Some((record, replay)); + Ok(()) + }, )?; - tail_records.push(record); + telemetry_run.frontier( + "certificates", + frontier_index, + ready.len(), + frontier_started, + ); + frontier_index += 1; + done.extend(ready); } - for file in &stage.certificates { - let record = compile(file, true).map_err(fail)?; - if record.exit_code != 0 - || !record.stdout.trim().is_empty() - || !record.stderr.trim().is_empty() - { - let combined = format!("{}{}", record.stdout, record.stderr); - return Err(fail(reject( - &file.module, - &combined, - record.exit_code, - &staged_text(file), - ))); - } - write_staged( - staging.path(), - &format!("process/preserve/{}.json", file.module), - &record.to_json().to_file_bytes(), - )?; + for (record, replay) in results.into_iter().flatten() { tail_records.push(record); - let replay = leanchecker::run_leanchecker( - &toolchain, - &file.module, - &preserve_path, - &workspace_root, - &limits, - &normalizer, - ) - .map_err(fail)?; - if replay.exit_code != 0 { - let combined = format!("{}{}", replay.stdout, replay.stderr); - return Err(fail(reject( - &file.module, - &combined, - replay.exit_code, - &staged_text(file), - ))); - } - write_staged( - staging.path(), - &format!("process/leanchecker/{}.json", file.module), - &replay.to_json().to_file_bytes(), - )?; - tail_records.push(replay); + tail_records.extend(replay); } let audit_record = compile(&stage.audit, false).map_err(fail)?; let audit_output = format!("{}{}", audit_record.stdout, audit_record.stderr); diff --git a/crates/lexlean/src/verify/store.rs b/crates/lexlean/src/verify/store.rs new file mode 100644 index 0000000000..48ed698821 --- /dev/null +++ b/crates/lexlean/src/verify/store.rs @@ -0,0 +1,768 @@ +//! The process store (SPEC.md §22.13): a verification process is a function +//! of its content identity, so a process whose identity an earlier run on +//! this host already established is not run again. +//! +//! The content identity κ of a Lean module is the digest of its staged text +//! and of the identities of the modules its import header names, by name: a +//! Merkle identity of everything a `lean` or `leanchecker` process of that +//! module can read beyond the pinned toolchain. A module the stage does not +//! hold is bound only when the toolchain's library holds it (the toolchain's +//! own `Init`, `Lean`, `Std`); it is then identified by its name, and the +//! toolchain by the digest of every file of it a process loads. A module +//! neither holds is unbound, and a process whose closure reaches it is not +//! answered by the store. The key of one process adds the tool, the digest of +//! its executable, its normalized argument vector, and its context: the +//! digest of the program it starts, the toolchain, the Lake configuration of +//! its working directory, and its normalized environment rows. The same +//! module compiled, replayed, or compiled without output are three +//! processes. +//! +//! The store is operational, like the width (§22.11): it is selected only by +//! `LEXLEAN_VERIFY_STORE`, is no input of any stage, enters no ID, and +//! changes no published byte. Only a process that succeeded is stored, with +//! its normalized record and the bytes of the files it wrote; a stored entry +//! is used only when its record names this run's digest of the executable +//! and every stored file still has the digest it was stored with. Anything +//! else, including an unreadable entry, is a miss and the process runs. + +use std::collections::BTreeMap; +use std::path::PathBuf; +use std::sync::{Mutex, PoisonError}; + +use camino::{Utf8Path, Utf8PathBuf}; + +use crate::artifact::canonical_json::Json; +use crate::artifact::content_id::Sha256Digest; +use crate::config::Limits; +use crate::diagnostic::Diagnostic; +use crate::verify::child::{ChildRecord, ChildSpec, Normalizer}; + +/// The environment variable that names the store directory. +pub const STORE_VARIABLE: &str = "LEXLEAN_VERIFY_STORE"; + +const DOMAIN: &str = "lexlean/verify-store/1"; + +/// A process store directory. +#[derive(Debug, Clone)] +pub struct Store { + root: PathBuf, +} + +impl Store { + /// The store the environment selects, if any: an unset or empty variable + /// selects none. + #[must_use] + pub fn from_env() -> Option { + let value = std::env::var_os(STORE_VARIABLE)?; + if value.is_empty() { + return None; + } + Some(Self { + root: PathBuf::from(value), + }) + } + + fn entry(&self, key: &Sha256Digest) -> PathBuf { + let hex = key.to_hex(); + self.root.join(&hex[..2]).join(hex) + } + + /// The stored record of the process `key`, after restoring the files it + /// wrote to `outputs`, or `None` when the store does not hold it whole. + fn restore( + &self, + key: &Sha256Digest, + outputs: &[Utf8PathBuf], + executable: Sha256Digest, + ) -> Option { + let entry = self.entry(key); + let manifest = std::fs::read_to_string(entry.join("outputs.txt")).ok()?; + let record = ChildRecord::from_json( + &Json::parse(&std::fs::read(entry.join("record.json")).ok()?).ok()?, + ) + .ok()?; + if record.executable_sha256 != executable || record.exit_code != 0 { + return None; + } + // One line per file written: the output it belongs to, the suffix + // after that output's name (Lean writes `.olean` with companions), and + // the file's digest. + let mut files = Vec::new(); + for (position, line) in manifest.lines().enumerate() { + let mut fields = line.splitn(3, ' '); + let index: usize = fields.next()?.parse().ok()?; + let digest = fields.next()?; + let suffix = fields.next().unwrap_or(""); + let output = outputs.get(index)?; + let stored = std::fs::read(entry.join(format!("{position}.out"))).ok()?; + if Sha256Digest::of(&stored).to_hex() != digest { + return None; + } + files.push(( + Utf8PathBuf::from(format!("{}{suffix}", output.with_extension(""))), + stored, + )); + } + if outputs.iter().enumerate().any(|(index, _)| { + !manifest + .lines() + .any(|line| line.split(' ').next() == Some(&index.to_string())) + }) { + return None; + } + for (path, stored) in files { + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent.as_std_path()).ok()?; + } + std::fs::write(path.as_std_path(), stored).ok()?; + } + // The entry was used: its time orders what a pruning keeps. + let _ = std::fs::File::options() + .append(true) + .open(entry.join("outputs.txt")) + .and_then(|file| file.set_modified(std::time::SystemTime::now())); + Some(record) + } + + /// Store a process that succeeded. A failure to store is dropped: the + /// store never decides a result. + fn save(&self, key: &Sha256Digest, outputs: &[Utf8PathBuf], record: &ChildRecord) { + let entry = self.entry(key); + let Some(parent) = entry.parent() else { return }; + if entry.is_dir() || std::fs::create_dir_all(parent).is_err() { + return; + } + let Ok(scratch) = tempfile::Builder::new() + .prefix(".entry-") + .tempdir_in(parent) + else { + return; + }; + let mut digests = String::new(); + let mut position = 0usize; + for (index, output) in outputs.iter().enumerate() { + // Lean writes the `.olean` with companions of the same stem + // (`.olean.server`, `.olean.private`, `.ir`): every file of that + // stem in the output's directory is what the process wrote. + let (Some(directory), Some(stem)) = (output.parent(), output.file_stem()) else { + return; + }; + let name = format!("{stem}."); + let name = name.as_str(); + let Ok(entries) = std::fs::read_dir(directory.as_std_path()) else { + return; + }; + let mut written: Vec = entries + .flatten() + .filter_map(|entry| entry.file_name().into_string().ok()) + .filter(|file| file.starts_with(name)) + .collect(); + written.sort(); + for file in written { + let Ok(bytes) = std::fs::read(directory.join(&file).as_std_path()) else { + return; + }; + let suffix = &file[stem.len()..]; + digests.push_str(&format!( + "{index} {} {suffix}\n", + Sha256Digest::of(&bytes).to_hex() + )); + if std::fs::write(scratch.path().join(format!("{position}.out")), bytes).is_err() { + return; + } + position += 1; + } + } + if std::fs::write( + scratch.path().join("record.json"), + record.to_json().to_file_bytes(), + ) + .is_err() + || std::fs::write(scratch.path().join("outputs.txt"), digests).is_err() + { + return; + } + let scratch = scratch.keep(); + if std::fs::rename(&scratch, &entry).is_err() { + // Another run stored the same process first. + let _ = std::fs::remove_dir_all(&scratch); + } + } +} + +/// The key of one process: its tool, executable, normalized arguments, and +/// the content identity of the module it is about. +#[must_use] +pub fn key( + tool: &str, + executable: Sha256Digest, + argv: &[String], + identity: Sha256Digest, +) -> Sha256Digest { + let mut bytes = Vec::new(); + for part in [DOMAIN, tool, &executable.to_hex(), &identity.to_hex()] { + bytes.extend_from_slice(part.as_bytes()); + bytes.push(0); + } + for argument in argv { + bytes.extend_from_slice(argument.as_bytes()); + bytes.push(0); + } + Sha256Digest::of(&bytes) +} + +/// Run `process` unless the store holds the process `key`; a process that +/// succeeded silently is stored with the files `outputs` it wrote. +/// +/// # Errors +/// +/// What `process` returns. +pub fn stored( + store: Option<&Store>, + key: Option, + outputs: &[Utf8PathBuf], + executable: Sha256Digest, + process: impl FnOnce() -> Result, +) -> Result { + let (Some(store), Some(key)) = (store, key) else { + return process(); + }; + if let Some(record) = store.restore(&key, outputs, executable) { + crate::verify::telemetry::reused(record.module.as_deref(), &record.tool); + return Ok(record); + } + let record = process()?; + if record.exit_code == 0 && outputs.iter().all(|output| output.as_std_path().is_file()) { + store.save(&key, outputs, &record); + } + Ok(record) +} + +/// The import header of a Lean source: the module each `import` command of +/// its header names, in order. The header is what precedes the first command +/// that is not an import: blank lines, line comments, the `module` and +/// `prelude` keywords, and imports with their `public`, `private`, `meta`, and +/// `all` modifiers (Lean's module system). +#[must_use] +pub fn imports(text: &str) -> Vec { + let mut found = Vec::new(); + for line in text.lines().map(str::trim) { + if line.is_empty() || line.starts_with("--") || line == "module" || line == "prelude" { + continue; + } + let mut words = line.split_whitespace().peekable(); + while matches!(words.peek(), Some(&("public" | "private" | "meta"))) { + words.next(); + } + if words.next() != Some("import") { + break; + } + if words.peek() == Some(&"all") { + words.next(); + } + found.extend(words.map(str::to_owned)); + } + found +} + +/// The store as one verification uses it: the store, and the staged source +/// roots its modules' identities are read from, in the order of the search +/// path the processes read their imports from. +pub struct Session { + store: Store, + roots: Vec, + toolchain: Utf8PathBuf, + known: Mutex>>, + files: Mutex>>, +} + +/// The file names under a toolchain or a Lake build directory that a `lean`, +/// `lake`, or `leanchecker` process loads: the executables, the shared +/// libraries, and each module's `.olean` with its companions and `.ir`. +fn loaded(relative: &str) -> bool { + let name = relative.rsplit('/').next().unwrap_or(relative); + relative.starts_with("bin/") + || name.ends_with(".olean") + || name.ends_with(".olean.server") + || name.ends_with(".olean.private") + || name.ends_with(".ir") + || name.ends_with(".so") + || name.contains(".so.") + || name.ends_with(".dylib") + || name.ends_with(".dll") +} + +/// The digest of every loaded file under `root`, by its path relative to +/// `root`, or `None` when the tree cannot be read whole. +/// +/// The traversal follows symbolic links as a process reading the tree does, +/// so a library directory linked from elsewhere is bound like one inside it. +/// A link that does not resolve, or a directory reached a second time (a +/// link cycle or an alias), makes the tree unreadable: `None`, a miss. +fn tree(root: &Utf8Path) -> Option { + let mut files: Vec = Vec::new(); + let mut seen: std::collections::BTreeSet = std::collections::BTreeSet::new(); + let mut stack = vec![root.to_path_buf()]; + while let Some(directory) = stack.pop() { + if !seen.insert(std::fs::canonicalize(directory.as_std_path()).ok()?) { + return None; + } + for entry in std::fs::read_dir(directory.as_std_path()).ok()? { + let entry = entry.ok()?; + let path = Utf8PathBuf::from_path_buf(entry.path()).ok()?; + let kind = std::fs::metadata(path.as_std_path()).ok()?.file_type(); + if kind.is_dir() { + stack.push(path); + } else { + let relative = path.strip_prefix(root).ok()?.as_str().to_owned(); + if loaded(&relative) { + files.push(relative); + } + } + } + } + files.sort(); + // The files are read on every core: the toolchain is the largest input a + // key binds, and the store answers only once it is read. + let width = std::thread::available_parallelism().map_or(1, std::num::NonZeroUsize::get); + let chunk = files.len().div_ceil(width).max(1); + let digests: Vec>> = std::thread::scope(|scope| { + files + .chunks(chunk) + .map(|part| { + scope.spawn(move || { + part.iter() + .map(|relative| { + std::fs::read(root.join(relative).as_std_path()) + .ok() + .map(|bytes| Sha256Digest::of(&bytes)) + }) + .collect::>>() + }) + }) + .collect::>() + .into_iter() + .map(|handle| handle.join().ok().flatten()) + .collect() + }); + let mut bytes = format!("{DOMAIN}\0tree\0").into_bytes(); + let mut names = files.iter(); + for part in digests { + for digest in part? { + bytes.extend_from_slice(format!("{}\0{}\0", names.next()?, digest.to_hex()).as_bytes()); + } + } + Some(Sha256Digest::of(&bytes)) +} + +impl Session { + /// The session of a verification whose sources are staged under `roots`. + #[must_use] + pub fn new(store: Store, roots: Vec, toolchain: Utf8PathBuf) -> Self { + Self { + store, + roots, + toolchain, + known: Mutex::new(BTreeMap::new()), + files: Mutex::new(BTreeMap::new()), + } + } + + /// The digest of a file or a tree, read once in a session. + fn digest_of( + &self, + path: &Utf8Path, + read: impl FnOnce(&Utf8Path) -> Option, + ) -> Option { + if let Some(known) = self + .files + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(path) + { + return *known; + } + let digest = read(path); + self.files + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(path.to_path_buf(), digest); + digest + } + + /// The context of the process `spec`, beyond its module's identity: the + /// program it starts, the toolchain it loads, the Lake configuration and + /// build directory of its working directory, and its normalized + /// environment rows. `None` when one of them cannot be read whole, or + /// when the process does not run under the toolchain's home. + fn context(&self, spec: &ChildSpec<'_>, normalizer: &Normalizer) -> Option { + let crate::verify::child::ChildHome::Toolchain { toolchain_bin } = spec.home else { + return None; + }; + let program = self.digest_of(spec.program, |path| { + std::fs::read(path.as_std_path()) + .ok() + .map(|bytes| Sha256Digest::of(&bytes)) + })?; + let toolchain = self.digest_of(&self.toolchain, tree)?; + let mut bytes = format!( + "{DOMAIN}\0context\0{}\0{}\0{}\0{}\0", + program.to_hex(), + toolchain.to_hex(), + normalizer.normalize_arg(toolchain_bin.as_str()), + normalizer.normalize_arg(spec.cwd.as_str()), + ) + .into_bytes(); + // `lake env` reads the workspace's configuration and puts its build + // directories on the search path. + for name in [ + "lakefile.lean", + "lakefile.toml", + "lake-manifest.json", + "lean-toolchain", + ] { + let path = spec.cwd.join(name); + let row = match std::fs::read(path.as_std_path()) { + Ok(text) => Sha256Digest::of(&text).to_hex(), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => "absent".to_owned(), + Err(_) => return None, + }; + bytes.extend_from_slice(format!("{name}\0{row}\0").as_bytes()); + } + let lake = spec.cwd.join(".lake"); + let lake_row = if lake.as_std_path().exists() { + self.digest_of(&lake, tree)?.to_hex() + } else { + "absent".to_owned() + }; + bytes.extend_from_slice(format!(".lake\0{lake_row}\0").as_bytes()); + for (key, value) in &spec.extra_env { + bytes.extend_from_slice( + format!("{key}\0{}\0", normalizer.normalize_arg(value)).as_bytes(), + ); + } + Some(Sha256Digest::of(&bytes)) + } + + /// The content identity of the staged module `name`: its text and the + /// identities of its imports. A module no root holds is bound when the + /// toolchain's library holds it, and is then identified by its name; + /// otherwise it, and everything that imports it, is unbound (`None`). + pub fn module(&self, name: &str) -> Option { + let mut open = Vec::new(); + self.visit(name, &mut open) + } + + /// The content identity of what a `leanchecker` replay of the prefix + /// `name` reads: the module and every staged module inside its namespace, + /// which the replay checks with it (§22.4). + pub fn prefix(&self, name: &str) -> Option { + let mut inside: Vec = Vec::new(); + let relative = name.replace('.', "/"); + for root in &self.roots { + let mut stack = vec![root.join(&relative)]; + while let Some(directory) = stack.pop() { + let Ok(entries) = std::fs::read_dir(directory.as_std_path()) else { + continue; + }; + for entry in entries.flatten() { + let Ok(path) = Utf8PathBuf::from_path_buf(entry.path()) else { + continue; + }; + if path.is_dir() { + stack.push(path); + } else if path.extension() == Some("lean") { + if let Ok(rest) = path.strip_prefix(root) { + inside.push(rest.with_extension("").as_str().replace('/', ".")); + } + } + } + } + } + inside.sort(); + inside.dedup(); + let mut bytes = format!( + "{DOMAIN}\0prefix\0{name}\0{}\0", + self.module(name)?.to_hex() + ) + .into_bytes(); + for module in inside { + bytes.extend_from_slice( + format!("{module}\0{}\0", self.module(&module)?.to_hex()).as_bytes(), + ); + } + Some(Sha256Digest::of(&bytes)) + } + + /// The content identity of the source at `path` and its imports. + pub fn source(&self, path: &Utf8Path) -> Option { + let text = std::fs::read(path.as_std_path()).ok()?; + let mut open = Vec::new(); + // Labelled by its file name: the staging directory it lies in is a + // fresh one in every run. + self.of_text(path.file_name()?, &text, &mut open) + } + + fn visit(&self, name: &str, open: &mut Vec) -> Option { + if let Some(identity) = self + .known + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(name) + { + return *identity; + } + let relative = format!("{}.lean", name.replace('.', "/")); + let Some(text) = self + .roots + .iter() + .find_map(|root| std::fs::read(root.join(&relative).as_std_path()).ok()) + else { + // The toolchain's own module: its bytes are bound by the + // toolchain's digest in every key's context. Any other module + // the stage does not hold is unbound. + let library = self + .toolchain + .join("lib") + .join("lean") + .join(format!("{}.olean", name.replace('.', "/"))); + return library + .as_std_path() + .is_file() + .then(|| Sha256Digest::of(format!("{DOMAIN}\0toolchain\0{name}").as_bytes())); + }; + if open.iter().any(|other| other == name) { + return Some(Sha256Digest::of( + format!("{DOMAIN}\0cycle\0{name}").as_bytes(), + )); + } + open.push(name.to_owned()); + let identity = self.of_text(name, &text, open); + open.pop(); + self.known + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(name.to_owned(), identity); + identity + } + + fn of_text(&self, label: &str, text: &[u8], open: &mut Vec) -> Option { + let mut bytes = format!( + "{DOMAIN}\0module\0{label}\0{}\0", + Sha256Digest::of(text).to_hex() + ) + .into_bytes(); + for import in imports(&String::from_utf8_lossy(text)) { + let identity = self.visit(&import, open)?; + bytes.extend_from_slice(format!("{import}\0{}\0", identity.to_hex()).as_bytes()); + } + Some(Sha256Digest::of(&bytes)) + } + + /// Run the process `spec` about the content `identity` unless the store + /// holds it; `outputs` are the files it writes. + /// + /// # Errors + /// + /// What the process returns. + pub fn run( + &self, + spec: &ChildSpec<'_>, + limits: &Limits, + normalizer: &Normalizer, + identity: Sha256Digest, + outputs: &[Utf8PathBuf], + ) -> Result { + let argv: Vec = spec + .argv + .iter() + .map(|argument| normalizer.normalize_arg(argument)) + .collect(); + // A context that cannot be read whole binds nothing: the process + // runs and is not stored. + let Some(context) = self.context(spec, normalizer) else { + return crate::verify::child::run(spec, limits, normalizer); + }; + let identity = Sha256Digest::of( + format!( + "{DOMAIN}\0bound\0{}\0{}\0", + identity.to_hex(), + context.to_hex() + ) + .as_bytes(), + ); + let key = key(spec.tool, spec.executable_sha256, &argv, identity); + stored( + Some(&self.store), + Some(key), + outputs, + spec.executable_sha256, + || crate::verify::child::run(spec, limits, normalizer), + ) + } +} + +/// Run `spec` through the session's store when there is one. +/// +/// # Errors +/// +/// What the process returns. +pub fn run( + session: Option<&Session>, + spec: &ChildSpec<'_>, + limits: &Limits, + normalizer: &Normalizer, + identity: impl FnOnce(&Session) -> Option, + outputs: &[Utf8PathBuf], +) -> Result { + match session.and_then(|session| identity(session).map(|identity| (session, identity))) { + Some((session, identity)) => session.run(spec, limits, normalizer, identity, outputs), + None => crate::verify::child::run(spec, limits, normalizer), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_import_header_reads_the_module_system_forms() { + let text = "module\n-- a comment\npublic import Init\nimport all A.B\nmeta import C\nimport D E\n\nset_option x 1\nimport F\n"; + assert_eq!(imports(text), ["Init", "A.B", "C", "D", "E"]); + assert_eq!( + imports("prelude\nimport Init.Core\ndef x := 1\n"), + ["Init.Core"] + ); + } + + fn session(roots: Vec, toolchain: &Utf8Path) -> Session { + Session::new( + Store { + root: PathBuf::new(), + }, + roots, + toolchain.to_path_buf(), + ) + } + + fn write(base: &Utf8Path, relative: &str, text: &str) { + let path = base.join(relative); + std::fs::create_dir_all(path.parent().expect("parent").as_std_path()).expect("dir"); + std::fs::write(path.as_std_path(), text).expect("write"); + } + + fn scratch() -> (tempfile::TempDir, Utf8PathBuf) { + let root = tempfile::tempdir().expect("scratch"); + let base = Utf8PathBuf::from_path_buf(root.path().to_path_buf()).expect("utf-8"); + (root, base) + } + + #[test] + fn an_identity_follows_every_module_an_import_reaches() { + let (_root, base) = scratch(); + let (src, toolchain) = (base.join("src"), base.join("toolchain")); + write(&toolchain, "lib/lean/Init.olean", "init"); + write(&src, "P/A.lean", "module\npublic import Init\ndef a := 1\n"); + write(&src, "P/B.lean", "module\npublic import P.A\ndef b := 2\n"); + write(&src, "P/C.lean", "module\npublic import P.B\ndef c := 3\n"); + let before = session(vec![src.clone()], &toolchain); + let (a, c) = (before.module("P.A"), before.module("P.C")); + assert!(a.is_some() && c.is_some()); + write(&src, "P/A.lean", "module\npublic import Init\ndef a := 4\n"); + let after = session(vec![src.clone()], &toolchain); + assert_ne!(after.module("P.A"), a); + assert_ne!( + after.module("P.C"), + c, + "a change two imports away moves the identity" + ); + assert_eq!(after.module("Init"), before.module("Init")); + } + + #[test] + fn a_module_neither_the_stage_nor_the_toolchain_holds_is_unbound() { + let (_root, base) = scratch(); + let (src, toolchain) = (base.join("src"), base.join("toolchain")); + write(&toolchain, "lib/lean/Init.olean", "init"); + write( + &src, + "P/A.lean", + "module\npublic import Elsewhere.Built\ndef a := 1\n", + ); + write(&src, "P/B.lean", "module\npublic import P.A\ndef b := 2\n"); + let session = session(vec![src.clone()], &toolchain); + assert_eq!(session.module("Elsewhere.Built"), None); + assert_eq!(session.module("P.A"), None); + assert_eq!(session.module("P.B"), None, "unbound through every import"); + assert!(session.module("Init").is_some()); + } + + #[cfg(unix)] + #[test] + fn a_linked_library_is_bound_and_a_link_cycle_is_unreadable() { + let (_root, base) = scratch(); + let (toolchain, library) = (base.join("toolchain"), base.join("library")); + write(&toolchain, "bin/lean", "lean"); + write(&library, "Init.olean", "init"); + std::fs::create_dir_all(toolchain.join("lib").as_std_path()).expect("dir"); + std::os::unix::fs::symlink( + library.as_std_path(), + toolchain.join("lib/lean").as_std_path(), + ) + .expect("link"); + let session = session(Vec::new(), &toolchain); + assert!( + session.module("Init").is_some(), + "the linked library holds Init" + ); + let before = tree(&toolchain).expect("readable"); + write(&library, "Init.olean", "changed"); + assert_ne!( + tree(&toolchain), + Some(before), + "a file under the linked library is bound" + ); + std::os::unix::fs::symlink(toolchain.as_std_path(), library.join("back").as_std_path()) + .expect("cycle"); + assert_eq!(tree(&toolchain), None, "a link cycle is a miss"); + std::fs::remove_file(library.join("back").as_std_path()).expect("unlink"); + std::os::unix::fs::symlink( + base.join("nowhere").as_std_path(), + library.join("dangling").as_std_path(), + ) + .expect("dangling"); + assert_eq!( + tree(&toolchain), + None, + "a link that does not resolve is a miss" + ); + } + + #[test] + fn the_toolchain_digest_follows_every_file_a_process_loads() { + let (_root, base) = scratch(); + write(&base, "bin/lean", "lean"); + write(&base, "lib/libleanshared.so", "shared"); + write(&base, "lib/lean/Init/Core.olean", "core"); + write(&base, "lib/lean/Init/Core.ir", "ir"); + write(&base, "lib/lean/Init/Core.ilean", "server index"); + let first = tree(&base).expect("readable"); + write(&base, "lib/lean/Init/Core.ilean", "another index"); + assert_eq!( + tree(&base), + Some(first), + "a file no process loads is not bound" + ); + for (relative, text) in [ + ("lib/lean/Init/Core.olean", "changed"), + ("lib/lean/Init/Core.ir", "changed"), + ("lib/libleanshared.so", "changed"), + ("bin/lean", "changed"), + ("lib/lean/Init/New.olean.private", "new"), + ] { + let before = tree(&base); + write(&base, relative, text); + assert_ne!(tree(&base), before, "{relative} is bound"); + } + } +} diff --git a/crates/lexlean/src/verify/telemetry.rs b/crates/lexlean/src/verify/telemetry.rs index 6d5b7c7366..69789b05d5 100644 --- a/crates/lexlean/src/verify/telemetry.rs +++ b/crates/lexlean/src/verify/telemetry.rs @@ -210,6 +210,17 @@ impl Context { .timed("process", report.started, Value::Object(fields)); } + /// Report a process the store held (§22.13): it did not run. + pub fn reused(&self, tool: &str, module: Option<&str>) { + let mut fields = serde_json::Map::new(); + fields.insert("stage".to_owned(), json!(self.stage)); + fields.insert("tool".to_owned(), json!(tool)); + if let Some(module) = module { + fields.insert("module".to_owned(), json!(module)); + } + self.sink.emit("reused", Value::Object(fields)); + } + /// Report that the item at `item` of the stage completed as the /// `sequence`th to do so. This is the one place completion order is /// visible, and it is visible only here. @@ -241,6 +252,13 @@ thread_local! { static CURRENT: RefCell> = const { RefCell::new(None) }; } +/// Report a process the store held, in the context in force on this thread. +pub fn reused(module: Option<&str>, tool: &str) { + if let Some(context) = current() { + context.reused(tool, module); + } +} + /// The context in force on this thread. #[must_use] pub fn current() -> Option {