diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c0eea6a00b..29566be0d0 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -311,6 +311,11 @@ jobs: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Install the pinned Rust toolchain run: rustup toolchain install + # CL-25 builds the executable for WASI with the pinned toolchain and runs + # it under the runner's Node (§8.3); the target is added here rather than + # in rust-toolchain.toml, which the export custody records byte for byte. + - name: Add the WASI target of the pinned toolchain + run: rustup target add wasm32-wasip1 - uses: taiki-e/install-action@b6b84cf49ebfe0176417bdce007c624f0db37f20 # v2.86.2 with: tool: just,cargo-deny diff --git a/CHANGELOG.md b/CHANGELOG.md index 65df9295aa..739d413e43 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,20 @@ workspace carried at those versions without a tag. ## Unreleased +- `lexlean check` builds and runs on `wasm32-wasip1`, #494 (SPEC.md §8.3, + `CL-25`). WASI has no threads, no file lock, and no processes: the check + pipeline runs on the instance's stack, a command that changes the project + is refused with `LLV7010` before it touches it, and the compiler-semantics + digest reads its files on the one thread. A file's identity there is the + device and inode WASI reports, read through `rustix`, so `LLC0104` still + refuses one file under two names; `fs4` and `same-file` are dependencies of + the other targets only. The executable is linked there with the 80 MiB + stack the depth plan assumes (`config::WASI_STACK_BYTES`, read by + `build.rs` and the plan alike), so deep input is `LLS8002` on WASI too, at a + hosted depth of 1,024. Native code is unchanged but + for the digest's reader, which reads on the calling thread when no thread + can be had. `CL-25` builds the target and compares it with the native + executable under Node's `node:wasi`; the register is 494. - Breaking security update: replace the single Lean 4.32.1 pin with `leanprover/lean4:v4.32.2`, source commit `f3b06c705e6c85f5314019d5d3baab0fec5b580c` (SPEC.md §8.2). The diff --git a/CONFORMANCE.md b/CONFORMANCE.md index 2e967d5954..9a0033212e 100644 --- a/CONFORMANCE.md +++ b/CONFORMANCE.md @@ -132,6 +132,7 @@ the error registry does not sanction (R5, `ERRORS.md`). | `CL-22` | `build` | A host that embeds the library ends every child process group the library started and has not seen exit through the public function `lexlean::verify::end_live_children`, which installs no signal handler, alters no signal disposition, never signals the host's own process or group, reports each group as ended, already gone, or failing with the OS error, and refuses a later child start with LLV7018 until the host re-enables it, and the `lexlean` executable ends its interrupted children through the same function. | | `CL-23` | `build` | lexlean production targets prints the canonical target-descriptor document in either output mode, production build --request publishes the bundle its request names and reports the bundle and the project's source, semantic, build, and attestation IDs in its command result, production verify-artifact replays a bundle with no project, and each maps a malformed request, a refused request, and a rejected replay to the exact exit code of its registered class. | | `CL-24` | `build` | lexlean migrate --to 1.2 migrates a language-1.1 data-form project to language 1.2 in one command: it rewrites every module to the authoring notation through lexlean::sdk::to_notation with the same declarations, names, statements, and axiom policies, moves the builtin glossary references of every source and the imports of every path lexicon package to 1.2.0, rewrites the language row, and regenerates the lock as lexlean/lock/2, so the result checks, has a current lock, and is canonical; --check proves the migration and writes nothing; and a project that is not language 1.1 is refused with LLC0103 and a module that is not a language-1.1 data-form semantic module, a data form that is not canonical JSON included, with LLP2003, each leaving every file of the project unchanged. | +| `CL-25` | `build` | The lexlean executable builds for wasm32-wasip1 with the pinned toolchain and, run by a WASI host, gives the native executable's exit code and JSON output for check on every single-step check fixture and every example; there a module with two names of one file is refused with LLC0104, nesting beyond the depth its linked stack hosts is refused with LLS8002, and a command that changes the project is refused with LLV7010 and leaves the project unchanged. | ## configuration-lock diff --git a/README.md b/README.md index 068400012f..5d91624b50 100644 --- a/README.md +++ b/README.md @@ -162,7 +162,7 @@ just vv # the complete normative acceptance gate (SPEC.md §9.2) just release # vv, then the §30 release criterion; refused until 1.0.0 ``` -All 496 registered conformance IDs have their named tests; `just vv` checks them with the pinned toolchain installed. +All 497 registered conformance IDs have their named tests; `just vv` checks them with the pinned toolchain installed. The differential against the independent generator `lean4-prod` (SPEC.md §17.19, `OR-01`..`OR-08`) acquires it at its pinned commit: from the checkout `LEXLEAN_ORACLE_DIR` names, or by fetching the commit with `git` into the Cargo target directory (`LEXLEAN_ORACLE_CACHE` overrides), and builds it with the pinned Lean and Rust toolchains. It is validation evidence only and no input to any identity. An oracle that cannot be acquired, is not the pinned files, or fails is unavailable: that fails the gate on Linux x86-64 and is printed as `NOT VALIDATED` elsewhere, and is never reported as agreement. @@ -188,7 +188,7 @@ Every row is validated by `just vv`; the IDs link the claim to its register row, | Canonical LaTeX regeneration and the optional hash-checked PDF provider | `TX-01`..`TX-12` | `build` | | Canonical diagnostics, source maps, coverage, manifests, and reproducible builds | `AR-01`..`AR-14` | `build` | | Sixteen-stage verification with leanchecker replay and exact axiom audit | `VR-01`..`VR-23` | `build` | -| The exact CLI contract, the stable seven-method Rust `Engine` API, the host-driven end of the children the library started, the `production` subcommands, and the `migrate` command | `CL-01`..`CL-24` | `build` | +| The exact CLI contract, the stable seven-method Rust `Engine` API, the host-driven end of the children the library started, the `production` subcommands, the `migrate` command, and `check` on `wasm32-wasip1` | `CL-01`..`CL-25` | `build` | | Filesystem confinement, no shell, no hidden network, closed failure model | `SE-01`..`SE-12` | `build` | | Language-1.2 production eligibility: closed targets, effects, and construct dispositions; runtime closures; target-dependent, fail-closed root analysis; deterministic eligibility reports; an exhaustiveness audit; measured stack bounds for every recursive root a harness runs; the effect each host request realizes | `PD-01`..`PD-10` | `build` | | Named-root extraction through Lean's compiler front end: an adapter that is a closed tooling module over a registry derived from the pinned Lean and probed on every extraction, checked against a handwritten oracle; a closed, canonical, root-independent compiler input; fail-closed rejection; and an exact closure cross-check against production eligibility | `NE-01`..`NE-06` | `build` | diff --git a/SPEC.md b/SPEC.md index ed98afddce..2fe180be2b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -571,6 +571,8 @@ On Windows x86-64 that same complete workspace test workload runs as lanes of on Non-UTF-8 project paths are outside language 1.0 and produce a registered environment diagnostic. +The `lexlean` executable MUST also build for `wasm32-wasip1` with the pinned toolchain, so that `check` runs where a WASI host is the only host, such as a browser worker over a file system held in memory. Run by a WASI host, `check` MUST give the native executable's exit code and JSON output on the same project. WASI has no threads, no file lock, and no processes, and three things follow on that target alone. The check pipeline runs on the instance's one stack rather than on a thread, and the executable is linked with the stack `Limits::compile_stack_plan` plans on there, `lexlean::config::WASI_STACK_BYTES` (80 MiB, which hosts a depth of 1,024), so nesting beyond the configured or the hosted depth is `LLS8002` as on every host (§25.5). The host supplies the rest of the instance's call stack; Node 22's default is sufficient at the hosted depth, and a host whose call stack ends sooner stops the instance with a trap, which is never a verdict. An embedder that links the library into a module of its own links it with at least that stack. A command that changes the project is refused with `LLV7010` before it touches the project, since the exclusive lock of §21.8 cannot be taken; `check` takes no lock. `verify` starts no toolchain process. A file's identity (§25.1) is the device and inode WASI reports for it, so a host that gives one file two names is refused with `LLC0104` there as well. `CL-25` builds the executable for the target from the tree under test and runs it under Node's `node:wasi`, over the real file system, beside the native executable on every single-step `check` fixture of §28.2 that sets no toolchain, overlay, or environment, and on every example. + ### 8.4 Cargo dependency rules - Every dependency requirement MUST be explicit; wildcard requirements are forbidden. @@ -606,9 +608,9 @@ Every dependency the shipped crate declares, for any target and of any kind but | --- | --- | | `camino` | UTF-8 path handling | | `clap` | CLI parsing | -| `fs4` | file locking | -| `rustix` | process-group signalling (`kill` of a group, without an external `kill`; safe wrappers, the crate forbids unsafe code) | -| `same-file` | file identity | +| `fs4` | file locking (every target but WASI, which has no file lock; §8.3) | +| `rustix` | process-group signalling (`kill` of a group, without an external `kill`; safe wrappers, the crate forbids unsafe code), and file identity on WASI (the device and inode of `stat`; §8.3) | +| `same-file` | file identity (every target but WASI, where it refuses every path; §8.3) | | `semver` | semantic-version parsing | | `serde` | JSON and TOML deserialization and serialization | | `serde_json` | JSON parsing and serialization | @@ -11927,6 +11929,7 @@ Every row below is normative, has honesty level `build`, and MUST be copied byte | `CL-22` | `cli-api` | A host that embeds the library ends every child process group the library started and has not seen exit through the public function `lexlean::verify::end_live_children`, which installs no signal handler, alters no signal disposition, never signals the host's own process or group, reports each group as ended, already gone, or failing with the OS error, and refuses a later child start with LLV7018 until the host re-enables it, and the `lexlean` executable ends its interrupted children through the same function. | §17.17, §24.6 | | `CL-23` | `cli-api` | lexlean production targets prints the canonical target-descriptor document in either output mode, production build --request publishes the bundle its request names and reports the bundle and the project's source, semantic, build, and attestation IDs in its command result, production verify-artifact replays a bundle with no project, and each maps a malformed request, a refused request, and a rejected replay to the exact exit code of its registered class. | §24.7 | | `CL-24` | `cli-api` | lexlean migrate --to 1.2 migrates a language-1.1 data-form project to language 1.2 in one command: it rewrites every module to the authoring notation through lexlean::sdk::to_notation with the same declarations, names, statements, and axiom policies, moves the builtin glossary references of every source and the imports of every path lexicon package to 1.2.0, rewrites the language row, and regenerates the lock as lexlean/lock/2, so the result checks, has a current lock, and is canonical; --check proves the migration and writes nothing; and a project that is not language 1.1 is refused with LLC0103 and a module that is not a language-1.1 data-form semantic module, a data form that is not canonical JSON included, with LLP2003, each leaving every file of the project unchanged. | §17.12, §23.4 | +| `CL-25` | `cli-api` | The lexlean executable builds for wasm32-wasip1 with the pinned toolchain and, run by a WASI host, gives the native executable's exit code and JSON output for check on every single-step check fixture and every example; there a module with two names of one file is refused with LLC0104, nesting beyond the depth its linked stack hosts is refused with LLS8002, and a command that changes the project is refused with LLV7010 and leaves the project unchanged. | §8.3, §21.8, §25.1 | | `SE-01` | `security` | Source, package, workspace, resource, and output paths are confined and symlinks are rejected. | §25.1 | | `SE-02` | `security` | Special files, duplicate filesystem identities, and case-fold collisions are rejected before processing. | §25.1 | | `SE-03` | `security` | All child processes use direct executable and argv invocation with no shell. | §25.2 | @@ -12175,7 +12178,7 @@ Every row below is normative, has honesty level `build`, and MUST be copied byte | `LM-09` | `language-models` | No prompt is code: no member of the closed language maps `string` or `bytes` to a function, a term, or source — every operation of the primitive registry that takes a `string` or `bytes` operand is classified by the audit with a data result, the registry's schema enumeration and the production registry's primitive rows equal one another and equal that classified set, and the text declarations of `lexlean/semantic-module/2` state no `string`-typed member, so a prompt reaches a model only as the tokens `encodeText` makes of it; a planted operation or text member of that shape is caught by the audit with its name. | §17.12 | | `LM-13` | `language-models` | The text laws a tokenizer elaborates through are LexLean source and not authored text: the module that states them is one language-1.2 native core module of the embedded `language/text-1.2` project, the project holds no file of Lean source, no independently authored Lean file anywhere in the release tree declares the `LexLeanText` namespace or lies under its module root, and the one-time migration input is absent; the registry lists only that module, the declarations and their exact axiom policies are read from its native data, which the compiler embeds and hashes into the language-1.2 semantics, and a module that declares a text component imports the generated module rather than splicing any Lean text. | §17.10, §17.12 | -**Total required capability IDs:** 496. +**Total required capability IDs:** 497. No row may be downgraded to `some-true` or `open`. Upstream Lean facts are ledger/authority rows, not substitutions for these build behaviors. diff --git a/VERIFICATION.md b/VERIFICATION.md index 597977606f..ba49e222ed 100644 --- a/VERIFICATION.md +++ b/VERIFICATION.md @@ -958,6 +958,42 @@ Planted: `mentions_family_group` ignored the arguments again (the comparison test ir::semantic::family_order_tests::a_group_instantiated_at_its_own_type_has_two_motives ... FAILED ``` +### check on WASI can trap, merge two names of a file, or differ (CL-25) + +`CL-25` holds `lexlean check` on `wasm32-wasip1` (§8.3). `conformance_cl_25` +builds the executable for the target in the release profile, runs it under +Node's `node:wasi` over the real file system, and compares it with +`lexlean::cli::run`. It covers every single-step `check` fixture with no +toolchain, overlay or environment of its own, and every example: 277 projects +on the first run, each with the same exit code and byte-identical stdout and +stderr. A hard-linked module is `LLC0104` on both builds. `fmt` on WASI exits 3 +with `LLV7010` and leaves the project as it was. With `max_scope_depth = +100000000`, nesting 1,000 is identical on both, and nesting 1,040 is `LLS8002` +(exit 4) on WASI, where the linked 80 MiB stack hosts 1,024. The case ran in +435 s, the module build excluded, on a shared Linux x86-64 host. + +Planted defect: `build.rs` without the line that links the executable with +`WASI_STACK_BYTES`, so it gets Rust's default 1 MiB stack. `cargo test -p +repo-conformance --all-features --test conformance conformance_cl_25` still +agreed on all 277 projects, then failed in 393 s at the depth vector: + +```text +assertion `left == right` failed: CL-25: nesting 1000 on WASI + left: (1, "", "... RuntimeError: memory access out of bounds + at ...lexlean7grammar4mathNtB2_7MathAst5atoms ... +``` + +With the line restored, the case passes. Before the stack was linked, the +same depth was measured directly: on WASI, nesting 512 checked and nesting 1,024 +trapped when the project allowed it, while natively both checked. This is the +trap the depth vector exists to catch. + +Node 22 collects garbage inside its fast call of WASI `path_filestat_get` +and dies with SIGSEGV on a language-1.2 project. The backtrace runs from +`uvwasi_path_filestat_get` through `MallocImpl` to `Heap::CollectGarbage`. The +case runs Node with `--no-turbo-fast-api-calls`, under which every project +checks. + ### a declaration ID can miss a reference or keep a proof (SM-38) `SM-38` holds the declaration and statement IDs of §21.9. `conformance_sm_38` diff --git a/crates/conformance/src/cases/cli_api.rs b/crates/conformance/src/cases/cli_api.rs index dd1b476648..2d163f2404 100644 --- a/crates/conformance/src/cases/cli_api.rs +++ b/crates/conformance/src/cases/cli_api.rs @@ -1235,6 +1235,7 @@ pub(crate) fn run(id: &str) { } "CL-23" => super::production_export::cli(), "CL-24" => super::migration::cli(), + "CL-25" => super::wasi::cli(), other => panic!("no cli-api case is wired for {other}"), } } diff --git a/crates/conformance/src/cases/mod.rs b/crates/conformance/src/cases/mod.rs index 62a5dd2eb9..baba52ebaf 100644 --- a/crates/conformance/src/cases/mod.rs +++ b/crates/conformance/src/cases/mod.rs @@ -74,6 +74,7 @@ mod stdlib_sha256; mod tooling; mod typed_diagnostics; mod verification; +mod wasi; /// The wall time of one case, written to the CI evidence directory when the /// environment names one (SPEC.md §22.12) and nowhere else. It is written when diff --git a/crates/conformance/src/cases/repository.rs b/crates/conformance/src/cases/repository.rs index 5c208be925..d4fb742ec7 100644 --- a/crates/conformance/src/cases/repository.rs +++ b/crates/conformance/src/cases/repository.rs @@ -1316,7 +1316,7 @@ pub(crate) fn run(id: &str) { let model = repo_model::Model::load(&root.join("model").into_std_path_buf()) .expect("the model loads"); let table = spec_table(); - assert_eq!(table.len(), 496, "§31 has 496 rows"); + assert_eq!(table.len(), 497, "§31 has 497 rows"); assert_eq!(model.ids.id.len(), table.len(), "register row count"); for ((spec_id, spec_suite, spec_statement), row) in table.iter().zip(model.ids.id.iter()) diff --git a/crates/conformance/src/cases/wasi.rs b/crates/conformance/src/cases/wasi.rs new file mode 100644 index 0000000000..b95f585193 --- /dev/null +++ b/crates/conformance/src/cases/wasi.rs @@ -0,0 +1,382 @@ +//! The `cli-api` suite, `check` on WASI: CL-25 (SPEC.md §8.3). +//! +//! The executable is built for `wasm32-wasip1` from this tree and run by +//! Node's `node:wasi`, a WASI host over the real file system, so a project +//! is read through the same paths, confinement checks, and file identities +//! as on the native host. The native side is `lexlean::cli::run`, the +//! function `main` calls, as the fixture runner runs it (§28.2). + +use std::process::Command; + +use camino::{Utf8Path, Utf8PathBuf}; + +use crate::fixtures; +use crate::support::{self, P}; + +const TARGET: &str = "wasm32-wasip1"; + +/// The WASI host: the module, the directory it sees as `/`, then the +/// arguments after `lexlean`. The project is the root of the instance's file +/// system because a WASI program starts in `/` and LexLean looks for +/// `lexlean.toml` from where it starts (§23.2). The environment is empty, as +/// it is for a native run that sets nothing (§25.4). +const HOST: &str = r#"import { readFile } from "node:fs/promises"; +import { WASI } from "node:wasi"; +const [module, root, ...args] = process.argv.slice(2); +const wasi = new WASI({ + version: "preview1", + args: ["lexlean", ...args], + env: {}, + preopens: { "/": root }, + returnOnExit: true, +}); +const instance = await WebAssembly.instantiate( + await WebAssembly.compile(await readFile(module)), + wasi.getImportObject(), +); +process.exitCode = wasi.start(instance); +"#; + +/// What one run produced: the exit code, stdout, and stderr. +type Run = (i32, String, String); + +/// Is there a WASI host and the pinned toolchain's standard library for the +/// target? On the normative host both must be present (`rust-toolchain.toml` +/// names the target, and CI installs Node's `node:wasi` with the runner); on +/// any other host the case says it did not run, never passes vacuously. +fn wasi_backed(id: &str) -> bool { + let node = Command::new("node") + .args(["--input-type=module", "-e", "await import('node:wasi')"]) + .output() + .is_ok_and(|output| output.status.success()); + let target = Command::new("rustc") + .args(["--print", "target-libdir", "--target", TARGET]) + .current_dir(support::repo_root().as_std_path()) + .output() + .ok() + .filter(|output| output.status.success()) + .and_then(|output| String::from_utf8(output.stdout).ok()) + .is_some_and(|dir| Utf8Path::new(dir.trim()).is_dir()); + if node && target { + return true; + } + assert!( + !support::normative_host(), + "{id}: the normative host (Linux x86-64) must have `node` with `node:wasi` (found: {node}) and the pinned toolchain's {TARGET} standard library (found: {target}); this case never passes vacuously here (§8.3)" + ); + eprintln!( + "{id}: no WASI host or no {TARGET} standard library on this host; nothing was run (§8.3)" + ); + false +} + +/// Build the executable for the target from this tree. It is the release +/// profile the executable ships with, in the workspace's target directory, +/// where nothing else builds for this target. +fn build() -> Utf8PathBuf { + let target_dir = std::env::var("CARGO_TARGET_DIR") + .map_or_else(|_| support::repo_root().join("target"), Utf8PathBuf::from); + let cargo = std::env::var("CARGO").unwrap_or_else(|_| "cargo".to_owned()); + let output = Command::new(cargo) + .args([ + "build", + "--locked", + "--release", + "--target", + TARGET, + "-p", + "lexlean", + "--bin", + "lexlean", + ]) + .current_dir(support::repo_root().as_std_path()) + .output() + .expect("cargo runs"); + assert!( + output.status.success(), + "CL-25: lexlean does not build for {TARGET}:\n{}", + String::from_utf8_lossy(&output.stderr) + ); + let module = target_dir.join(TARGET).join("release/lexlean.wasm"); + assert!(module.is_file(), "CL-25: {module} was built"); + module +} + +/// Run the module on `root` under the WASI host. +fn wasi(module: &Utf8Path, host: &Utf8Path, root: &Utf8Path, arguments: &[&str]) -> Run { + // Node 22 collects garbage inside its fast call of WASI's + // `path_filestat_get` and dies with SIGSEGV on a language-1.2 project; + // the ordinary call path does not. + let output = Command::new("node") + .args(["--no-warnings", "--no-turbo-fast-api-calls"]) + .arg(host.as_str()) + .arg(module.as_str()) + .arg(root.as_str()) + .args(arguments) + .output() + .expect("node runs"); + ( + output.status.code().unwrap_or(-1), + String::from_utf8_lossy(&output.stdout).into_owned(), + String::from_utf8_lossy(&output.stderr).into_owned(), + ) +} + +/// Run the executable natively on `root`, holding the environment lock as +/// the fixture runner does. +fn native(root: &Utf8Path, arguments: &[&str]) -> Run { + let _guard = support::env_shared(); + support::cli_in(root, arguments) +} + +/// A run with the project's own location written as `$PROJECT`: the native +/// run sees its temporary directory, the instance sees `/`. +fn normalized((exit, stdout, stderr): Run, root: &str) -> Run { + let root = root.trim_end_matches('/'); + let replace = |text: &str| { + if root.is_empty() { + text.replace("\"/", "\"$PROJECT/") + } else { + text.replace(root, "$PROJECT") + } + }; + (exit, replace(&stdout), replace(&stderr)) +} + +/// A copy of `source` for one side of a comparison, symlinks kept. +fn copy(source: &Utf8Path) -> P { + let project = P::empty(); + fixtures::copy_tree_preserving_links(source.as_std_path(), project.root.as_std_path()) + .expect("the project copies"); + for generated in [".lexlean", "expected"] { + let path = project.root.join(generated); + if path.is_dir() { + std::fs::remove_dir_all(path.as_std_path()).expect("remove a generated directory"); + } + } + project +} + +/// One project to check on both hosts: its source and the `check` arguments. +struct Subject { + name: String, + source: Utf8PathBuf, + args: Vec, +} + +/// Every single-step `check` fixture that needs no toolchain, overlay, or +/// environment of its own, and every example project. +fn subjects() -> Vec { + let root = support::repo_root(); + let mut subjects = Vec::new(); + for dir in fixtures::discover(&root) { + let case = fixtures::load_case(&dir).expect("a fixture loads"); + let [invocation] = case.invocations.as_slice() else { + continue; + }; + if invocation.command != "check" + || !case.environment.is_empty() + || dir.join("toolchain").exists() + || dir.join("overlay").exists() + { + continue; + } + subjects.push(Subject { + name: dir.strip_prefix(&root).unwrap_or(&dir).to_string(), + source: dir.join("project"), + args: invocation.args.clone(), + }); + } + let fixtures = subjects.len(); + assert!( + fixtures >= 100, + "CL-25: {fixtures} check fixtures were found" + ); + let mut examples: Vec = std::fs::read_dir(root.join("examples").as_std_path()) + .expect("examples") + .flatten() + .filter(|entry| entry.path().join("lexlean.toml").is_file()) + .filter_map(|entry| Utf8PathBuf::from_path_buf(entry.path()).ok()) + .collect(); + examples.sort(); + for source in examples { + subjects.push(Subject { + name: source.strip_prefix(&root).unwrap_or(&source).to_string(), + source, + args: Vec::new(), + }); + } + assert!(subjects.len() > fixtures, "CL-25: the examples were found"); + subjects +} + +/// Check `subject` on both hosts and report any difference. +fn compare(module: &Utf8Path, host: &Utf8Path, subject: &Subject) -> Option { + let mut arguments = vec!["--diagnostic-format", "json", "check"]; + arguments.extend(subject.args.iter().map(String::as_str)); + let left = copy(&subject.source); + let right = copy(&subject.source); + let on_native = normalized(native(&left.root, &arguments), left.root.as_str()); + let on_wasi = normalized(wasi(module, host, &right.root, &arguments), "/"); + (on_native != on_wasi).then(|| { + format!( + "{}: native exit {} stdout {} stderr {}\n wasi exit {} stdout {} stderr {}", + subject.name, on_native.0, on_native.1, on_native.2, on_wasi.0, on_wasi.1, on_wasi.2 + ) + }) +} + +/// CL-25. +pub(crate) fn cli() { + if !wasi_backed("CL-25") { + return; + } + let module = build(); + let scratch = tempfile::Builder::new() + .prefix("lexlean-wasi-host-") + .tempdir() + .expect("tempdir"); + let host = Utf8PathBuf::from_path_buf(scratch.path().join("host.mjs")).expect("utf8 path"); + std::fs::write(host.as_std_path(), HOST).expect("write the host"); + + // The same exit code, command result, and diagnostics on every project. + let subjects = subjects(); + let next = std::sync::atomic::AtomicUsize::new(0); + let differences = std::sync::Mutex::new(Vec::new()); + let workers = std::thread::available_parallelism() + .map_or(1, usize::from) + .min(8); + std::thread::scope(|scope| { + for _ in 0..workers { + scope.spawn(|| loop { + let at = next.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let Some(subject) = subjects.get(at) else { + break; + }; + if let Some(difference) = compare(&module, &host, subject) { + differences + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .push(difference); + } + }); + } + }); + let differences = differences + .into_inner() + .unwrap_or_else(std::sync::PoisonError::into_inner); + assert!( + differences.is_empty(), + "CL-25: {} of {} projects differ between the native and the WASI executable:\n{}", + differences.len(), + subjects.len(), + differences.join("\n") + ); + eprintln!( + "CL-25: {} projects agree between the native and the WASI executable", + subjects.len() + ); + + // Two names of one file are refused on WASI too (§25.1): WASI reports the + // device and inode that tell them apart. + let example = support::repo_root().join("examples/nat-add-zero"); + let linked = |project: &P| { + std::fs::hard_link( + project.root.join("src/Main.lex.tex").as_std_path(), + project.root.join("src/Twin.lex.tex").as_std_path(), + ) + .expect("a hard link"); + }; + let left = copy(&example); + let right = copy(&example); + linked(&left); + linked(&right); + let arguments = ["--diagnostic-format", "json", "check"]; + let on_native = normalized(native(&left.root, &arguments), left.root.as_str()); + let on_wasi = normalized(wasi(&module, &host, &right.root, &arguments), "/"); + assert!( + on_native.1.contains("LLC0104") && on_native.1.contains("are the same file"), + "CL-25: the native executable refuses a hard-linked module: {on_native:?}" + ); + assert_eq!(on_wasi, on_native, "CL-25: a hard-linked module on WASI"); + + // A command that changes the project is refused before it touches it: + // WASI has no file lock (§21.8). + let project = copy(&example); + let before = support::file_set(&project.root); + let (exit, stdout, stderr) = wasi( + &module, + &host, + &project.root, + &["--diagnostic-format", "json", "fmt"], + ); + assert_eq!(exit, 3, "CL-25: fmt on WASI exits 3: {stdout}{stderr}"); + assert!( + stdout.contains("LLV7010") && stdout.contains("this host has no file lock"), + "CL-25: fmt on WASI is refused with LLV7010: {stdout}{stderr}" + ); + assert_eq!( + support::file_set(&project.root), + before, + "CL-25: a refused fmt leaves the project as it was" + ); + + // The depth the linked stack hosts is enforced as on every host: input + // within it checks as it does natively, and input beyond it is LLS8002, + // never a trap, however deep the project allows (§8.3, §25.5). + let hosted = lexlean::config::Limits { + max_scope_depth: u64::MAX, + ..support::limits_of(&P::example()) + }; + let hosted = usize::try_from(hosted.compile_stack_plan().effective_max_scope_depth) + .expect("the native plan hosts a depth"); + assert!( + hosted > 1024, + "CL-25: the native host hosts more than WASI's 1,024" + ); + for (nesting, refused) in [(1000, false), (1040, true)] { + let project = copy(&example); + let nested = format!("{}n{}", "(".repeat(nesting), ")".repeat(nesting)); + for (file, from, to) in [ + ( + "lexlean.toml", + "max_scope_depth = 1024".to_owned(), + "max_scope_depth = 100000000".to_owned(), + ), + ( + "src/Main.lex.tex", + r"\(n + 0 = n\)".to_owned(), + format!(r"\({nested} + 0 = n\)"), + ), + ] { + let path = project.root.join(file); + let text = std::fs::read_to_string(path.as_std_path()).expect("a project file"); + assert!(text.contains(&from), "CL-25: {file} holds {from}"); + std::fs::write(path.as_std_path(), text.replace(&from, &to)).expect("edit"); + } + let locked = native(&project.root, &["lock"]); + assert_eq!( + locked.0, 0, + "CL-25: the deep project locks natively: {locked:?}" + ); + let arguments = ["--diagnostic-format", "json", "check"]; + let on_native = normalized(native(&project.root, &arguments), project.root.as_str()); + let on_wasi = normalized(wasi(&module, &host, &project.root, &arguments), "/"); + assert_eq!( + on_native.0, 0, + "CL-25: nesting {nesting} checks natively: {on_native:?}" + ); + if refused { + assert_eq!( + on_wasi.0, 4, + "CL-25: nesting {nesting} on WASI exits 4: {on_wasi:?}" + ); + assert!( + on_wasi.1.contains("LLS8002") && on_wasi.1.contains("max_scope_depth exceeded"), + "CL-25: nesting {nesting} on WASI is LLS8002: {on_wasi:?}" + ); + } else { + assert_eq!(on_wasi, on_native, "CL-25: nesting {nesting} on WASI"); + } + } +} diff --git a/crates/conformance/src/fixtures.rs b/crates/conformance/src/fixtures.rs index 340cc3b752..893b01e4e8 100644 --- a/crates/conformance/src/fixtures.rs +++ b/crates/conformance/src/fixtures.rs @@ -321,7 +321,7 @@ pub fn discover(root: &Utf8Path) -> Vec { /// Copy a tree preserving symlinks as symlinks (a fixture may deliberately /// contain one, SPEC.md §28.5 "a path symlink"). -fn copy_tree_preserving_links(from: &Path, to: &Path) -> Result<(), String> { +pub(crate) fn copy_tree_preserving_links(from: &Path, to: &Path) -> Result<(), String> { for entry in walkdir::WalkDir::new(from) .follow_links(false) .into_iter() diff --git a/crates/conformance/tests/conformance.rs b/crates/conformance/tests/conformance.rs index c83090918d..2b6d5483e6 100644 --- a/crates/conformance/tests/conformance.rs +++ b/crates/conformance/tests/conformance.rs @@ -1255,6 +1255,11 @@ fn conformance_cl_24() { repo_conformance::cases::run("CL-24"); } +#[test] +fn conformance_cl_25() { + repo_conformance::cases::run("CL-25"); +} + #[test] fn conformance_se_01() { repo_conformance::cases::run("SE-01"); diff --git a/crates/lexlean/Cargo.toml b/crates/lexlean/Cargo.toml index 3c89a70347..f1875c69e1 100644 --- a/crates/lexlean/Cargo.toml +++ b/crates/lexlean/Cargo.toml @@ -19,8 +19,6 @@ categories.workspace = true [dependencies] camino = { workspace = true } clap = { workspace = true } -fs4 = { workspace = true } -same-file = { workspace = true } semver = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } @@ -39,6 +37,17 @@ walkdir = { workspace = true } rustix = { workspace = true } signal-hook = { workspace = true } +# WASI has no file lock, so `fs4` does not build there, and `same-file` builds +# but refuses every file at run time. WASI does report a file's device and +# inode through `stat`, which `rustix` reads without unsafe code, so the +# identity check of §25.1 holds there too (§8.3). +[target.'cfg(not(target_os = "wasi"))'.dependencies] +fs4 = { workspace = true } +same-file = { workspace = true } + +[target.'cfg(target_os = "wasi")'.dependencies] +rustix = { workspace = true, features = ["fs"] } + # The build script records the address of the `rustc` that compiles this crate # (SPEC.md §22.11): the SHA-256 of its executable and driver library. [build-dependencies] diff --git a/crates/lexlean/build.rs b/crates/lexlean/build.rs index 5b94ae8d54..2d43ad0510 100644 --- a/crates/lexlean/build.rs +++ b/crates/lexlean/build.rs @@ -28,10 +28,18 @@ const SEMANTIC_DIRS: [(&str, &str); 4] = [ ("tests/golden/canonical-json", "tests/golden/canonical-json"), ]; +include!("src/wasi_stack.rs"); + /// The embedded directory the `production-data` feature carries. const PRODUCTION_DATA_DIR: &str = "language/preservation-1.2/library"; fn main() { + // WASI fixes the one stack when the executable is linked; the depth the + // check pipeline accepts there is planned on exactly this stack (§8.3). + println!("cargo:rerun-if-changed=src/wasi_stack.rs"); + if env::var("CARGO_CFG_TARGET_OS").as_deref() == Ok("wasi") { + println!("cargo:rustc-link-arg-bins=-zstack-size={WASI_STACK_BYTES}"); + } let manifest = PathBuf::from(env::var("CARGO_MANIFEST_DIR").expect("cargo sets this")); let mut files: Vec<(String, PathBuf)> = Vec::new(); diff --git a/crates/lexlean/src/api.rs b/crates/lexlean/src/api.rs index 400fa59f31..49013077f8 100644 --- a/crates/lexlean/src/api.rs +++ b/crates/lexlean/src/api.rs @@ -566,6 +566,7 @@ pub(crate) struct MutationGuard { /// Acquire the project mutation lock at `/.lock` (§21.8). The /// build root's components are checked for symlinks first (§25.1). +#[cfg(not(target_os = "wasi"))] fn acquire_lock(project: &Project) -> Result { use fs4::fs_std::FileExt; let build_root = project @@ -592,6 +593,17 @@ fn acquire_lock(project: &Project) -> Result { Ok(MutationGuard { _file: file }) } +/// WASI has no file lock, and a WASI host may share its directories with +/// other processes, so a command that changes the project is refused there +/// before it touches anything rather than run without the lock (§8.3, §21.8). +#[cfg(target_os = "wasi")] +fn acquire_lock(project: &Project) -> Result { + Err(host_failure(format!( + "{}/.lock: this host has no file lock, so it runs no command that changes the project", + project.config.build_root + ))) +} + /// Publish a rendered build atomically at its content-addressed path /// (§21.5, §21.8): staging, fsync, validate-before-reuse, rename. pub fn publish_build( diff --git a/crates/lexlean/src/config.rs b/crates/lexlean/src/config.rs index 2f9cc39e6e..099cd1d003 100644 --- a/crates/lexlean/src/config.rs +++ b/crates/lexlean/src/config.rs @@ -56,8 +56,17 @@ const COMPILE_STACK_BYTES_PER_SCOPE: usize = 64 * 1024; /// a resource reservation, not a language limit: nesting beyond what it /// hosts is refused as `LLS8002` naming both the configured and the /// effective bound, never by exhausting the stack. +#[cfg(not(target_os = "wasi"))] const MAX_COMPILE_STACK_BYTES: usize = 1024 * 1024 * 1024; +/// On WASI the pipeline runs on the executable's one stack, so the plan hosts +/// no more than that stack holds and deeper input is `LLS8002` there too +/// (§8.3); an embedder links its module with at least this stack. +#[cfg(target_os = "wasi")] +const MAX_COMPILE_STACK_BYTES: usize = WASI_STACK_BYTES; + +include!("wasi_stack.rs"); + impl Limits { /// Size the compile thread's stack from `max_scope_depth` with checked /// arithmetic, and report the nesting depth that stack hosts. A diff --git a/crates/lexlean/src/lib.rs b/crates/lexlean/src/lib.rs index d5fae9cd5d..3b6d37ec64 100644 --- a/crates/lexlean/src/lib.rs +++ b/crates/lexlean/src/lib.rs @@ -263,15 +263,24 @@ fn embedded_digest(keep: impl Fn(&str) -> bool) -> Sha256Digest { let workers = std::thread::available_parallelism() .map_or(1, usize::from) .min(8); + let read = || loop { + let at = next.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let Some((path, bytes)) = selected.get(at) else { + break; + }; + let _ = slots[at].set(identity_bytes(path, bytes)); + }; std::thread::scope(|scope| { for _ in 0..workers { - scope.spawn(|| loop { - let at = next.fetch_add(1, std::sync::atomic::Ordering::Relaxed); - let Some((path, bytes)) = selected.get(at) else { - break; - }; - let _ = slots[at].set(identity_bytes(path, bytes)); - }); + // A host without threads (WASI) reads every file on this one, as + // `identity_bytes` does; the rows, and so the digest, are the same. + if std::thread::Builder::new() + .spawn_scoped(scope, read) + .is_err() + { + read(); + break; + } } }); let rows: Vec<(&str, std::borrow::Cow<'static, [u8]>)> = selected diff --git a/crates/lexlean/src/link.rs b/crates/lexlean/src/link.rs index 50bbbb8108..8c7ba00eec 100644 --- a/crates/lexlean/src/link.rs +++ b/crates/lexlean/src/link.rs @@ -358,6 +358,7 @@ struct ModuleSource { /// bounded by `max_scope_depth`, and a configured depth larger than the /// stack can host is enforced at the depth the stack does host, so deep /// input is `LLS8002` and never a stack overflow. +#[cfg(not(target_os = "wasi"))] pub fn check_project( project: &Project, selection: &Selection, @@ -387,6 +388,20 @@ pub fn check_project( }) } +/// Run the complete check pipeline for a selection on the caller's stack: +/// WASI has no threads. The executable is linked with the stack the plan is +/// sized on there (`config::WASI_STACK_BYTES`), so the depth it hosts is +/// enforced as on any other host and deeper input is `LLS8002` (§8.3, §25.5). +#[cfg(target_os = "wasi")] +pub fn check_project( + project: &Project, + selection: &Selection, + lock: &Lock, + packages: Vec, +) -> Result { + check_project_inline(project, selection, lock, packages) +} + /// Read every configured model artifact (§10.1): a confined regular file of /// at most `max_file_bytes` whose SHA-256 is its configured digest. A missing /// file or a digest mismatch is `LLR3007`. diff --git a/crates/lexlean/src/project.rs b/crates/lexlean/src/project.rs index c1fb0eea43..bb48265c9c 100644 --- a/crates/lexlean/src/project.rs +++ b/crates/lexlean/src/project.rs @@ -81,6 +81,27 @@ fn security(message: impl Into) -> Diagnostic { Diagnostic::new(code!("LLS8001"), message) } +/// What makes two paths one file (§25.1), compared by equality. +#[cfg(not(target_os = "wasi"))] +type FileIdentity = same_file::Handle; + +#[cfg(not(target_os = "wasi"))] +fn file_identity(path: &Utf8Path) -> std::io::Result { + same_file::Handle::from_path(path.as_std_path()) +} + +/// `same-file` has no WASI implementation and refuses every path there, but +/// WASI reports a device and an inode, which is what `same-file` compares on +/// unix; so a host that links two names to one file is still refused. +#[cfg(target_os = "wasi")] +type FileIdentity = (u64, u64); + +#[cfg(target_os = "wasi")] +fn file_identity(path: &Utf8Path) -> std::io::Result { + let stat = rustix::fs::stat(path.as_std_path())?; + Ok((stat.st_dev, stat.st_ino)) +} + /// A non-UTF-8 path is outside language 1.0 and an environment failure /// (§8.3), never a security violation. pub fn non_utf8_path(path: &std::path::Path) -> Diagnostic { @@ -422,7 +443,7 @@ impl Project { pub fn all_modules(&self) -> Result, Vec> { let mut diagnostics = Vec::new(); let mut modules: BTreeMap = BTreeMap::new(); - let mut identities: Vec<(same_file::Handle, String)> = Vec::new(); + let mut identities: Vec<(FileIdentity, String)> = Vec::new(); for root in &self.config.source_roots { let absolute_root = match self.confined(root, Expect::Creatable) { Ok(absolute) => absolute, @@ -484,7 +505,7 @@ impl Project { continue; } let relative = self.display(&path); - match same_file::Handle::from_path(path.as_std_path()) { + match file_identity(&path) { Ok(handle) => { if let Some((_, existing)) = identities .iter() diff --git a/crates/lexlean/src/verify/lifeline.rs b/crates/lexlean/src/verify/lifeline.rs index 841038b749..da68ea6d1a 100644 --- a/crates/lexlean/src/verify/lifeline.rs +++ b/crates/lexlean/src/verify/lifeline.rs @@ -419,6 +419,16 @@ pub(crate) fn group_members(group: i32) -> Option> { Some(members) } +/// WASI starts no process (spawning one is refused there), so there is never +/// a leader to end; being asked to end one is a failure, never success +/// (SPEC.md §24.6). +#[cfg(target_os = "wasi")] +pub(crate) fn end_leader(id: u32) -> GroupEnd { + GroupEnd::Failed(format!( + "process {id} cannot be ended: this host starts no process" + )) +} + /// End the tree rooted at `id` on windows, and report a `taskkill` that /// failed like a group that could not be ended (the one place a tree is /// ended; see the unix function of the same name). diff --git a/crates/lexlean/src/wasi_stack.rs b/crates/lexlean/src/wasi_stack.rs new file mode 100644 index 0000000000..21780d7432 --- /dev/null +++ b/crates/lexlean/src/wasi_stack.rs @@ -0,0 +1,7 @@ +/// The stack the `lexlean` executable is linked with on WASI (SPEC.md §8.3). +/// WASI has no threads, so the check pipeline runs on this one stack, and +/// `Limits::compile_stack_plan` hosts its depth on it as it does on a compile +/// thread elsewhere: 16 MiB for the pipeline and 64 KiB for each of 1,024 +/// levels. `build.rs` links the executable with it, and the plan reads it, from +/// this one file. +pub const WASI_STACK_BYTES: usize = 80 * 1024 * 1024; diff --git a/features/suites/cli-api.feature b/features/suites/cli-api.feature index 5b738f2763..b3f5f4c230 100644 --- a/features/suites/cli-api.feature +++ b/features/suites/cli-api.feature @@ -179,3 +179,12 @@ Feature: cli-api Given the language-1.1 data-form example with a path lexicon package, the language-1.2 example, and a language-1.1 project whose module the notation cannot express When the example is migrated with --check and then without it, and each of the others is migrated Then --check writes nothing, the migration leaves a language-1.2 project that checks, locks, and formats canonically with the same modules, and the refused projects are refused with LLC0103 and LLP2003 and left unchanged + + @CL-25 @build + Scenario: The lexlean executable builds for wasm32-wasip1 with the pinned toolchain and, run by a WASI host, gives the native executable's exit code and JSON output for check on every single-step check fixture and every example; there a module with two names of one file is refused with LLC0104, nesting beyond the depth its linked stack hosts is refused with LLS8002, and a command that changes the project is refused with LLV7010 and leaves the project unchanged. + Given the lexlean executable built for wasm32-wasip1 from this tree and Node's node:wasi over the real file system + When check runs natively and under the WASI host on every single-step check fixture and every example, on a project whose module has a second name by a hard link, and on a project nested just within and just beyond the depth the WASI stack hosts, and fmt runs under the WASI host + Then every exit code and JSON output is the native one + And the hard-linked module is refused with LLC0104 on both hosts + And fmt is refused with LLV7010 and leaves every file of the project as it was + And nesting within the hosted depth checks as it does natively, and nesting beyond it is LLS8002 on WASI diff --git a/model/ids.toml b/model/ids.toml index a2d3400de8..a8ed485f31 100644 --- a/model/ids.toml +++ b/model/ids.toml @@ -1507,6 +1507,12 @@ level = "build" suite = "cli-api" statement = "lexlean migrate --to 1.2 migrates a language-1.1 data-form project to language 1.2 in one command: it rewrites every module to the authoring notation through lexlean::sdk::to_notation with the same declarations, names, statements, and axiom policies, moves the builtin glossary references of every source and the imports of every path lexicon package to 1.2.0, rewrites the language row, and regenerates the lock as lexlean/lock/2, so the result checks, has a current lock, and is canonical; --check proves the migration and writes nothing; and a project that is not language 1.1 is refused with LLC0103 and a module that is not a language-1.1 data-form semantic module, a data form that is not canonical JSON included, with LLP2003, each leaving every file of the project unchanged." +[[id]] +id = "CL-25" +level = "build" +suite = "cli-api" +statement = "The lexlean executable builds for wasm32-wasip1 with the pinned toolchain and, run by a WASI host, gives the native executable's exit code and JSON output for check on every single-step check fixture and every example; there a module with two names of one file is refused with LLC0104, nesting beyond the depth its linked stack hosts is refused with LLS8002, and a command that changes the project is refused with LLV7010 and leaves the project unchanged." + [[id]] id = "SE-01" level = "build" diff --git a/xtask/src/spec_links.rs b/xtask/src/spec_links.rs index f97eb8c07c..83dab19361 100644 --- a/xtask/src/spec_links.rs +++ b/xtask/src/spec_links.rs @@ -61,8 +61,8 @@ fn parse_table(spec: &str) -> Result, Fail> { pub fn validate(root: &Path) -> Result<(), Fail> { let spec = std::fs::read_to_string(root.join("SPEC.md"))?; let rows = parse_table(&spec)?; - if rows.len() != 496 { - return Err(format!("RP-07: the §31 table has {} rows, not 496", rows.len()).into()); + if rows.len() != 497 { + return Err(format!("RP-07: the §31 table has {} rows, not 497", rows.len()).into()); } // The prose total under the table is part of the contract a reader // relies on; a row added without it would leave §31 contradicting itself.