diff --git a/.github/bdd-parts.json b/.github/bdd-parts.json index cdb36ec045..47aa46068f 100644 --- a/.github/bdd-parts.json +++ b/.github/bdd-parts.json @@ -19,6 +19,9 @@ "bootstrap-structural-grammar": [ "BS-21" ], + "bootstrap-surface-lex": [ + "BS-22" + ], "bootstrap-check": [ "BS-04" ], diff --git a/.github/vv-rows.json b/.github/vv-rows.json index cbea2aa417..00037bec15 100644 --- a/.github/vv-rows.json +++ b/.github/vv-rows.json @@ -125,6 +125,15 @@ "memory_kib": "", "hints": "" }, + { + "part": "bdd-bootstrap-surface-lex", + "recipes": "bdd", + "examples_part": "", + "bdd_part": "bootstrap-surface-lex", + "width": "", + "memory_kib": "", + "hints": "" + }, { "part": "bdd-bootstrap-terms", "recipes": "bdd", diff --git a/CHANGELOG.md b/CHANGELOG.md index 369e5e8653..7b4ef27d3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -599,6 +599,12 @@ of this tree show. hand-built vectors, the committed modules, and 2,000 seeded edits; 44 planted defects are each detected by a vector of their own. It admits no `recursion`. The largest certificate part is 2,879,430 bytes. The register is 469 IDs. +- The lexer of the authoring notation as language-1.2 source, second slice of + the front end (SPEC.md §17.23, §17.21, `BS-22`, #269; no new diagnostic code, + no change to the language-1.2 identity). `lexText` and `quoteText` of + `SurfaceLex` return what `surface::lex::lex` and `surface::lex::quote` return, + the tokens with their byte ranges or the first lexical error, and a quoted + text; they admit no `recursion`. The register is 470 IDs. - Explicit shared byte literals (#68, `SM-19`, `MD-02`; languages 1.1 and 1.2): a modeled byte literal is lowered as `_root_.ByteArray.mk #[…]` with every octet written `_root_.UInt8.ofNat (nat_lit n)`, an octet occurring four or diff --git a/CONFORMANCE.md b/CONFORMANCE.md index a509ee22cd..eff53b5154 100644 --- a/CONFORMANCE.md +++ b/CONFORMANCE.md @@ -76,6 +76,7 @@ the error registry does not sanction (R5, `ERRORS.md`). | `BS-14` | `build` | The crate around the bodies of the Rust lowering of SPEC.md §17.16 (the Function item of each function: its parameters, each a name when the body mentions it and its type is not unit; its result type, wrapped when the function can overflow; the body of a function with a parameter no value inhabits, which is the empty match on that parameter; and the order of all the items, the types' first) is language-1.2 source in the `bootstrap/` project, the last pass of the lowering, as two production roots of `RustAssemble` that the host composes with the roots of `BS-07` and `BS-11` to `BS-13`: `functionShells`, which for a program as calculus data, the failing functions, and the frame of `BS-11` returns a Function item of each function, complete for a function with an uninhabited parameter and with a placeholder body for the others, and `assembleItems`, which puts the items of `lowerTypes`, those shells, and the parameters and body that `loopFunction` returned for each live function into one chain of items; each is eligible for `rust-std` with exactly its declared effects and has no part of its certificate beyond 3,145,728 bytes; for the programs that `rust::lower` lowers, on the normative host the lowered programs under the calculus evaluator agree with the oracle (the chain `assembleItems` returns, given to `applyMoves` of `BS-08`, is the whole chain of items of the crate to the byte, the items of the types too, the host passing the counter of the fresh names from one live function to the next and the profile) on 337 vectors and 580 seeded mutants, the rendered packages agree with the evaluator under the pinned rustc, 11 planted defects in the source and three hosts that do more than thread the counter (one that starts it again at each function, one that lowers the functions last to first, one that leaves out the items of the first live function) are each detected, and the certified packages the production export publishes for the project are the committed ones; the checks that `rust::lower` runs on the crate (`validate` and `correspond`), which are the roots of `BS-05` and `BS-06`, are outside the claim. | | `BS-15` | `build` | The package renderer of SPEC.md §17.16 (the checks `package::lower` makes of a manifest's own fields before it reads the program: the spec, the crate name against the Rust keywords and the reserved crates, the version, the profile, the form and the order of the sources, and the effects against the registry, with `allocation` refused on `rust-core`; the Cargo manifest over the scaffold record; and the provenance in canonical JSON with the digests of the runtime, the Cargo manifest, and the library) is language-1.2 source in the `bootstrap/` project, as two production roots of `RustPackage` that the host composes around the roots of `BS-04` to `BS-14`: `checkManifest`, which for the fields of a manifest and the registered effects returns whether the profile is `rust-std`, or the refusal of `package::lower` to the byte, and `renderPackage`, which for the scaffold record, the manifest's name, version, profile, sources, effects and exports, the program's identity, the compiler-semantics identity, the two runtime texts, and the library returns the `Cargo.toml` and the `provenance.json` of `package::package` to the byte; each is eligible for `rust-std` with exactly its declared effects and has no part of its certificate beyond 3,145,728 bytes; the oracle of `checkManifest` is `package::lower` on the manifest with its program and exports replaced by a pair every later check accepts, which agrees with each committed negative manifest's refusal, and the oracle of `renderPackage` is `package::package` on the 282 committed manifests it accepts and, on 137 hand-built inputs (names, exports, sources and effects that reach the escaper, and scaffold records that differ from the committed one) and 282 seeded mutants, `cargo_toml_of` and the canonical JSON of §21.7 by the recipe `package::package` is held to; on the normative host the lowered programs under the calculus evaluator agree with the oracle on the 301 committed manifests, 182 rule manifests and 2,000 seeded mutants, and on the 241 renders whose library is at most 16,384 bytes; the rendered packages agree with the oracle under the pinned rustc on all 3,184 runs; 21 planted defects in the source are each detected; and the certified packages the production export publishes for the project are the committed ones; the checks of the program, the exports and the crate, and the bytes of the library, which are the rows `BS-04` to `BS-14`, are outside the claim. | | `BS-21` | `build` | The fixed structural grammar of SPEC.md §15.1 and §15.2 (`grammar::structural::parse_module`: the module header, the blocks and declarations, the axiom policies, the sentences, the structured proof environments, and the closed data forms, with the coverage rows of every structural atom it consumes) is language-1.2 source in the `bootstrap/` project: `parseModule` of `StructuralGrammar`, for the atoms `scanSource` of `BS-03` returns, the path, the name the path gives the module, and `max_scope_depth`, is a production root eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and has no part of its certificate beyond 3,145,728 bytes; it returns the module with its rows, or the refusal's code, message, and span, and on 267 hand-built vectors, 146 of them refused, which reach every refusal of `parse_module`, and on the committed modules, whole or by the atoms of both their ends, its lowered program under the calculus evaluator on the normative host and its rendered package under the pinned rustc on every host agree with the oracle to the byte; on the normative host both also agree on 2,000 seeded edits of those texts, each of 44 planted defects in the source is detected by the one vector it names, and the certified package the production export publishes for the project is the committed one; the grammars of mathematical islands, propositions, and proof sentences, which read the lexicon's closure, and the interpretation of the module this root returns are outside the claim. | +| `BS-22` | `build` | The lexing of the authoring notation of SPEC.md §17.21 (`surface::lex::lex`: a text into its delimiters, `:=` and `::`, quoted strings with their escapes decoded, and words, which end at white space in the sense of `char::is_whitespace` or at a delimiter, each with its byte range, or the offset and message of the first lexical error; and `surface::lex::quote`: a text as the notation quotes it, with the quotation mark, the backslash, and every control character escaped) is language-1.2 source in the `bootstrap/` project, as two production roots of `SurfaceLex`: `lexText` and `quoteText`, each eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and with no part of its certificate beyond 3,145,728 bytes; on every host their rendered `rust-std` packages under the pinned rustc, and on the normative host their lowered programs under the calculus evaluator, agree with the oracle to the byte on the hand-built vectors, which reach every lexical error and every escape, on windows of the committed sources, and on seeded edits of those texts; each planted defect in the source is detected by the one vector it names, and the certified packages the production export publishes for the project are the committed ones; the readers of the tokens (the generic notation and the sugar of the closed schemas), the pretty-printing algebra, and canonical JSON are outside the claim. | ## calculus diff --git a/README.md b/README.md index 00594e8c2f..e853d3fd96 100644 --- a/README.md +++ b/README.md @@ -147,7 +147,7 @@ just vv # the complete normative acceptance gate (SPEC.md §9.2) just release # vv, then the §30 release criterion; refused until 1.0.0 ``` -All 469 registered conformance IDs are implemented and pass; `just vv` runs clean from a checkout with the pinned toolchain installed. +All 470 registered conformance IDs are implemented and pass; `just vv` runs clean from a checkout with the pinned toolchain installed. The differential against the independent generator `lean4-prod` (SPEC.md §17.19, `OR-01`..`OR-08`) acquires it at its pinned commit: from the checkout `LEXLEAN_ORACLE_DIR` names, or by fetching the commit with `git` into the Cargo target directory (`LEXLEAN_ORACLE_CACHE` overrides), and builds it with the pinned Lean and Rust toolchains. It is validation evidence only and no input to any identity. An oracle that cannot be acquired, is not the pinned files, or fails is unavailable: that fails the gate on Linux x86-64 and is printed as `NOT VALIDATED` elsewhere, and is never reported as agreement. @@ -190,7 +190,7 @@ Every row is validated by `just vv`; the IDs link the claim to its register row, | The language-1.2 authoring notation: a readable `.lex.tex` text for a semantic module, in bijection with its canonical JSON, whose every position and variant is read from the schema, with readable forms for types and terms and a generic form for every other value, formatted canonically by `fmt` failing closed, the only form in which a language-1.2 source of the product is written (the data form is refused under language 1.2, and `lexlean::sdk` is the explicit conversion), the same for the compiler-tooling modules in the generic notation, with no compiler-semantics ID that depends on which form an embedded module is written in, with no product source owned by a generator but five named records, read and written by programs only through the authoring API (`lexlean::sdk`), and, for a native core module, a readable program over the shared table of kernel terms whose numbering is the order of its text | `AU-01`..`AU-13` | `build` | | The runtime of generated Lean as native modules: the portable, collection, model, and reasoning runtimes of language 1.2 are LexLean source that every generated module imports, migrated losslessly and audited against their exact axioms, the released runtime text of languages 1.0 and 1.1 is frozen data equal to their released output, and no other Lean runtime text remains in the compiler but the decoder of core module 2 | `RT-01`..`RT-04` | `build` | | The standard library: language-1.2 source beside the compiler whose members are certified production roots, each validated against its oracle by a seeded differential of its evaluator and rustc realizations; bytes and text over `Bytes`, an output buffer of a given capacity written in order; decimal and hexadecimal; the byte order and stable sorts over it; SHA-256 against FIPS 180-4, the NIST CAVP vectors, and the `sha2` crate; and the `std-models` and `std-reasoning` source packages, generic over types, whose laws are kernel-checked theorems and whose application is the `helpdesk` example | `SL-01`, `SL-02`, `SL-03`, `SL-04`, `SL-05`, `SL-09`, `SL-10` | `build` | -| The bootstrap project: the compiler's own components as language-1.2 source compiled by LexLean's production pipeline to certified Rust packages, the exact axiom-output parser and its closed Lean-name check, the primitive scanner of §12.2, the preservation lexer and token audit, the target calculus's checker, the Rust crate validator, the Rust printer, the runtime syntax validator, the failure typing and the types of the Rust lowering, its last-use moves, its literal values and its terms, the package renderer, and the structural grammar, each held to the Rust it replaces (the stage-0 oracle) by the committed vectors and a seeded differential of its evaluator and rustc realizations, with planted defects detected and the production export committed | `BS-01`..`BS-21` | `build` | +| The bootstrap project: the compiler's own components as language-1.2 source compiled by LexLean's production pipeline to certified Rust packages, the exact axiom-output parser and its closed Lean-name check, the primitive scanner of §12.2, the preservation lexer and token audit, the target calculus's checker, the Rust crate validator, the Rust printer, the runtime syntax validator, the failure typing and the types of the Rust lowering, its last-use moves, its literal values and its terms, the package renderer, the structural grammar, and the lexer of the authoring notation, each held to the Rust it replaces (the stage-0 oracle) by the committed vectors and a seeded differential of its evaluator and rustc realizations, with planted defects detected and the production export committed | `BS-01`..`BS-22` | `build` | | Source packages: a language-1.2 project imports the modules of a locked, content-addressed package of its workspace: `[[source_package]]` and `lexlean-workspace.toml`, the pins in `lexlean.lock` recomputed by every command, imports closed at link, no re-export of a package's production roots, and the package's modules replayed by the importer's `verify` with their origin in the records | `PK-01`..`PK-05` | `build` | | The literal `nat-add-zero` example, the Lean-verified feature examples, and the complete negative fixture suite | `EX-01`..`EX-09` | `build` | | Language-1.2 text components: tokenizers and their round-trip laws, with the laws carried as one native core module the generated modules import rather than authored Lean text | `LM-01`, `LM-02`, `LM-03`, `LM-13` | `build` | diff --git a/SPEC.md b/SPEC.md index fb9083c191..dd72b3e4b1 100644 --- a/SPEC.md +++ b/SPEC.md @@ -8496,6 +8496,30 @@ one into the chain when it pops a frame, so no definition recurses over a chain. What the module means (its notation, its IR, its islands and proof sentences) reads the lexicon's closure or the data forms and belongs to the later slices. +The lexing of the authoring notation (§17.21) is the second component of the +front end (`BS-22`, issue 269), the module `SurfaceLex` with two production +roots for `rust-std`: + +| Root | Takes | Returns | Oracle | +| --- | --- | --- | --- | +| `LexLeanBoot.SurfaceLex.lexText` | a text | its tokens, each with its byte range, or the offset and message of the first lexical error | `surface::lex::lex` | +| `LexLeanBoot.SurfaceLex.quoteText` | a text | the text between quotation marks, with the quotation mark, the backslash, and every control character escaped | `surface::lex::quote` | + +A token is a delimiter (`{ } [ ] ( ) , ; :`, with `:=` and `::` one token each), +a quoted string with its escapes decoded, or a word, the longest run of scalars +that are neither white space nor a delimiter nor a quotation mark. White space +is the Unicode `White_Space` property, as `char::is_whitespace` has it, and the +module takes its width at a position from `AxiomOutput.whiteLength` (`BS-01`), +which holds the same list. The text is valid UTF-8, as a Rust `&str` is: the +module reads bytes, and a continuation byte is never a delimiter, so it steps a +byte at a time and takes a scalar's width only where the oracle's message names +the scalar. A string is read by a loop over its pieces (a run of raw bytes or +one escape), joined once at the end, so reading it costs its length and not its +square. The lexical errors are an unterminated string, a `\u` escape without four +hexadecimal digits (either case), a `\u` escape of a surrogate, an escape that is +none, and a control character in a string; each is reported at the offset of the +byte the oracle names, the first one standing. + The bootstrap project's production export is the first project whose roots together approach `max_total_source_bytes` (§10.2): the limit counts the programs, the crates, and the certificates of every production root of the @@ -11409,6 +11433,7 @@ Every row below is normative, has honesty level `build`, and MUST be copied byte | `BS-14` | `bootstrap` | The crate around the bodies of the Rust lowering of SPEC.md §17.16 (the Function item of each function: its parameters, each a name when the body mentions it and its type is not unit; its result type, wrapped when the function can overflow; the body of a function with a parameter no value inhabits, which is the empty match on that parameter; and the order of all the items, the types' first) is language-1.2 source in the `bootstrap/` project, the last pass of the lowering, as two production roots of `RustAssemble` that the host composes with the roots of `BS-07` and `BS-11` to `BS-13`: `functionShells`, which for a program as calculus data, the failing functions, and the frame of `BS-11` returns a Function item of each function, complete for a function with an uninhabited parameter and with a placeholder body for the others, and `assembleItems`, which puts the items of `lowerTypes`, those shells, and the parameters and body that `loopFunction` returned for each live function into one chain of items; each is eligible for `rust-std` with exactly its declared effects and has no part of its certificate beyond 3,145,728 bytes; for the programs that `rust::lower` lowers, on the normative host the lowered programs under the calculus evaluator agree with the oracle (the chain `assembleItems` returns, given to `applyMoves` of `BS-08`, is the whole chain of items of the crate to the byte, the items of the types too, the host passing the counter of the fresh names from one live function to the next and the profile) on 337 vectors and 580 seeded mutants, the rendered packages agree with the evaluator under the pinned rustc, 11 planted defects in the source and three hosts that do more than thread the counter (one that starts it again at each function, one that lowers the functions last to first, one that leaves out the items of the first live function) are each detected, and the certified packages the production export publishes for the project are the committed ones; the checks that `rust::lower` runs on the crate (`validate` and `correspond`), which are the roots of `BS-05` and `BS-06`, are outside the claim. | §17.16, §17.23 | | `BS-15` | `bootstrap` | The package renderer of SPEC.md §17.16 (the checks `package::lower` makes of a manifest's own fields before it reads the program: the spec, the crate name against the Rust keywords and the reserved crates, the version, the profile, the form and the order of the sources, and the effects against the registry, with `allocation` refused on `rust-core`; the Cargo manifest over the scaffold record; and the provenance in canonical JSON with the digests of the runtime, the Cargo manifest, and the library) is language-1.2 source in the `bootstrap/` project, as two production roots of `RustPackage` that the host composes around the roots of `BS-04` to `BS-14`: `checkManifest`, which for the fields of a manifest and the registered effects returns whether the profile is `rust-std`, or the refusal of `package::lower` to the byte, and `renderPackage`, which for the scaffold record, the manifest's name, version, profile, sources, effects and exports, the program's identity, the compiler-semantics identity, the two runtime texts, and the library returns the `Cargo.toml` and the `provenance.json` of `package::package` to the byte; each is eligible for `rust-std` with exactly its declared effects and has no part of its certificate beyond 3,145,728 bytes; the oracle of `checkManifest` is `package::lower` on the manifest with its program and exports replaced by a pair every later check accepts, which agrees with each committed negative manifest's refusal, and the oracle of `renderPackage` is `package::package` on the 282 committed manifests it accepts and, on 137 hand-built inputs (names, exports, sources and effects that reach the escaper, and scaffold records that differ from the committed one) and 282 seeded mutants, `cargo_toml_of` and the canonical JSON of §21.7 by the recipe `package::package` is held to; on the normative host the lowered programs under the calculus evaluator agree with the oracle on the 301 committed manifests, 182 rule manifests and 2,000 seeded mutants, and on the 241 renders whose library is at most 16,384 bytes; the rendered packages agree with the oracle under the pinned rustc on all 3,184 runs; 21 planted defects in the source are each detected; and the certified packages the production export publishes for the project are the committed ones; the checks of the program, the exports and the crate, and the bytes of the library, which are the rows `BS-04` to `BS-14`, are outside the claim. | §17.16, §17.23 | | `BS-21` | `bootstrap` | The fixed structural grammar of SPEC.md §15.1 and §15.2 (`grammar::structural::parse_module`: the module header, the blocks and declarations, the axiom policies, the sentences, the structured proof environments, and the closed data forms, with the coverage rows of every structural atom it consumes) is language-1.2 source in the `bootstrap/` project: `parseModule` of `StructuralGrammar`, for the atoms `scanSource` of `BS-03` returns, the path, the name the path gives the module, and `max_scope_depth`, is a production root eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and has no part of its certificate beyond 3,145,728 bytes; it returns the module with its rows, or the refusal's code, message, and span, and on 267 hand-built vectors, 146 of them refused, which reach every refusal of `parse_module`, and on the committed modules, whole or by the atoms of both their ends, its lowered program under the calculus evaluator on the normative host and its rendered package under the pinned rustc on every host agree with the oracle to the byte; on the normative host both also agree on 2,000 seeded edits of those texts, each of 44 planted defects in the source is detected by the one vector it names, and the certified package the production export publishes for the project is the committed one; the grammars of mathematical islands, propositions, and proof sentences, which read the lexicon's closure, and the interpretation of the module this root returns are outside the claim. | §17.23, §15.1, §15.2 | +| `BS-22` | `bootstrap` | The lexing of the authoring notation of SPEC.md §17.21 (`surface::lex::lex`: a text into its delimiters, `:=` and `::`, quoted strings with their escapes decoded, and words, which end at white space in the sense of `char::is_whitespace` or at a delimiter, each with its byte range, or the offset and message of the first lexical error; and `surface::lex::quote`: a text as the notation quotes it, with the quotation mark, the backslash, and every control character escaped) is language-1.2 source in the `bootstrap/` project, as two production roots of `SurfaceLex`: `lexText` and `quoteText`, each eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and with no part of its certificate beyond 3,145,728 bytes; on every host their rendered `rust-std` packages under the pinned rustc, and on the normative host their lowered programs under the calculus evaluator, agree with the oracle to the byte on the hand-built vectors, which reach every lexical error and every escape, on windows of the committed sources, and on seeded edits of those texts; each planted defect in the source is detected by the one vector it names, and the certified packages the production export publishes for the project are the committed ones; the readers of the tokens (the generic notation and the sugar of the closed schemas), the pretty-printing algebra, and canonical JSON are outside the claim. | §17.23, §17.21 | | `PK-01` | `packages` | A language-1.2 project that declares a source package by its identifier, version, kind `path`, and the member directory a `lexlean-workspace.toml` lists locks it by the tree digest of its manifest and source files, its manifest digest, its version, its module prefix, and its own packages, and the configuration and the lock read back to the bytes written; a project without a source package keeps its bytes, and a package of another kind, a source package in another language, a changed byte or version (LLC0105), an undeclared package (LLC0106), a module prefix claimed twice (LLC0107), a project or a package outside the workspace (LLC0108), a path that escapes it, and a symlink (LLS8001) are each refused with the named code. | §10.5, §11.1, §25.1 | | `PK-02` | `packages` | A module of a project imports only modules of its own project and modules of the packages the project declares, and a module of a package only modules of itself and of the packages its own manifest declares: a package module is a module of the importing project under its own module prefix, a module of a package the project does not declare fails with LLR3009, and an import cycle through a package fails with LLR3003. | §10.5, §15.1 | | `PK-03` | `packages` | An importing project builds and verifies under the pinned Lean offline, and its package modules are generated, elaborated, replayed with `leanchecker`, and audited for their exact axioms under the package's module prefix with the package as their origin in the records; the source ID of the project changes with the pins, and no file of the package is written by `fmt`, `build`, or `verify`. | §10.5, §21.3, §22 | @@ -11425,7 +11450,7 @@ Every row below is normative, has honesty level `build`, and MUST be copied byte | `LM-09` | `language-models` | No prompt is code: no member of the closed language maps `string` or `bytes` to a function, a term, or source — every operation of the primitive registry that takes a `string` or `bytes` operand is classified by the audit with a data result, the registry's schema enumeration and the production registry's primitive rows equal one another and equal that classified set, and the text declarations of `lexlean/semantic-module/2` state no `string`-typed member, so a prompt reaches a model only as the tokens `encodeText` makes of it; a planted operation or text member of that shape is caught by the audit with its name. | §17.12 | | `LM-13` | `language-models` | The text laws a tokenizer elaborates through are LexLean source and not authored text: the module that states them is one language-1.2 native core module of the embedded `language/text-1.2` project, the project holds no file of Lean source, no independently authored Lean file anywhere in the release tree declares the `LexLeanText` namespace or lies under its module root, and the one-time migration input is absent; the registry lists only that module, the declarations and their exact axiom policies are read from its native data, which the compiler embeds and hashes into the language-1.2 semantics, and a module that declares a text component imports the generated module rather than splicing any Lean text. | §17.10, §17.12 | -**Total required capability IDs:** 469. +**Total required capability IDs:** 470. No row may be downgraded to `some-true` or `open`. Upstream Lean facts are ledger/authority rows, not substitutions for these build behaviors. diff --git a/VERIFICATION.md b/VERIFICATION.md index 29e04a1515..e04efb015b 100644 --- a/VERIFICATION.md +++ b/VERIFICATION.md @@ -10787,6 +10787,36 @@ landing regeneration produces the `bootstrap/parse_module` package: `conformance_bs_21` stops at the stack-bound step with `bootstrap/parse_module is no package of the committed exports`, after every differential above has agreed. +### the lexer of the authoring notation is source (BS-22) + +`BS-22` holds the two production roots of `LexLeanBoot.SurfaceLex` +(`bootstrap/src/SurfaceLex.lex.tex`), `lexText` and `quoteText`, to +`surface::lex::lex` and `surface::lex::quote`, on the tokens with their byte +ranges and on the offset and message of the first lexical error. Neither root +admits `recursion`: a scan is a loop under `iterate_until`, and a string is a +loop over its pieces, joined once by `Buffer.joinAll` so that reading one costs +its length. `AxiomOutput.whiteLength` is the list of white space, which `BS-01` +holds to `char::is_whitespace`. + +The case has the shape of `BS-21`: hand-built vectors that reach each of the +five lexical errors, windows of the committed sources, and seeded edits, under +the evaluator on the normative host and under the pinned rustc on every host; +each planted defect names the one vector that catches it, and the case asserts +that the site of each plant occurs once. The row's statement carries no counts. + +Measured with the compiler binary of the `BS-21` tree, on a copy of the project +whose only entrypoint is `SurfaceLex`: `lexlean check` and `lexlean build` pass, +and the eligibility report gives both roots `rust-std` with exactly +`allocation` and `overflow`. On the project: the lock is current, all 18 modules +are canonical, and 22 modules check. The register agrees with the §31 table on +470 IDs, by the same parse `validate-spec-links` makes. + +The case has not run. It has not been compiled either, and `lexlean verify`, +which holds the axioms each definition declares to the pinned Lean, has not been +run on the module. What it owes is its first run, to pin the counts of its +corpora, and the landing regeneration, which produces the `bootstrap/lex_text` +and `bootstrap/quote_text` packages. + ## The `bdd` recipe as the parts of one run (Issue #131) With UOR Atlas verified as the parts of one run, the `rest` leg became the diff --git a/bootstrap/README.md b/bootstrap/README.md index ad11a75016..3d577f2e8d 100644 --- a/bootstrap/README.md +++ b/bootstrap/README.md @@ -19,3 +19,6 @@ library's `Buffer` (`stdlib/src/Buffer.lex.tex`), which it writes through. `src/StructuralGrammar.lex.tex` imports `PrimitiveScan` for the atoms it reads and the refusal it returns, and `Buffer` and `Numeral` from the standard library. + +`src/SurfaceLex.lex.tex` imports `AxiomOutput`, whose `whiteLength` holds the list of +white space the oracle's `char::is_whitespace` has, and the standard library's `Buffer` and `Text`. diff --git a/bootstrap/lexlean.lock b/bootstrap/lexlean.lock index 9a9b51057a..bf0e67ad0f 100644 --- a/bootstrap/lexlean.lock +++ b/bootstrap/lexlean.lock @@ -1,7 +1,7 @@ spec = "lexlean/lock/2" language = "1.2" compiler_semantics = "aac30cb0bda069a55aca6ce2417be48726fd578218b3e8fbaf31634694695b64" -project_config_sha256 = "46467ebf533a6f9ae1680362f34bb13645384258f23a9138764cb856df1df33d" +project_config_sha256 = "80b0cdf4c2dda3d097913cf2a900054237ebf3860e9eec2924c7b1b43191b2f5" lean_toolchain = "leanprover/lean4:v4.32.1" [[workspace_file]] diff --git a/bootstrap/lexlean.toml b/bootstrap/lexlean.toml index aaeb2e85e8..01f37f55d4 100644 --- a/bootstrap/lexlean.toml +++ b/bootstrap/lexlean.toml @@ -3,7 +3,7 @@ name = "bootstrap" language = "1.2" module_prefix = "LexLeanBoot" source_roots = ["src"] -entrypoints = ["src/AxiomOutput.lex.tex", "src/CalculusCheck.lex.tex", "src/PrimitiveScan.lex.tex", "src/RuntimeValidate.lex.tex", "src/RustAssemble.lex.tex", "src/RustAst.lex.tex", "src/RustBodies.lex.tex", "src/RustLoops.lex.tex", "src/RustLower.lex.tex", "src/RustMoves.lex.tex", "src/RustPackage.lex.tex", "src/RustPrint.lex.tex", "src/RustSucceed.lex.tex", "src/RustTerms.lex.tex", "src/RustValidate.lex.tex", "src/StructuralGrammar.lex.tex", "src/TokenAudit.lex.tex"] +entrypoints = ["src/AxiomOutput.lex.tex", "src/CalculusCheck.lex.tex", "src/PrimitiveScan.lex.tex", "src/RuntimeValidate.lex.tex", "src/RustAssemble.lex.tex", "src/RustAst.lex.tex", "src/RustBodies.lex.tex", "src/RustLoops.lex.tex", "src/RustLower.lex.tex", "src/RustMoves.lex.tex", "src/RustPackage.lex.tex", "src/RustPrint.lex.tex", "src/RustSucceed.lex.tex", "src/RustTerms.lex.tex", "src/RustValidate.lex.tex", "src/StructuralGrammar.lex.tex", "src/SurfaceLex.lex.tex", "src/TokenAudit.lex.tex"] build_root = ".lexlean" lockfile = "lexlean.lock" lean_workspace = "." diff --git a/bootstrap/src/SurfaceLex.lex.tex b/bootstrap/src/SurfaceLex.lex.tex new file mode 100644 index 0000000000..3e55f1e1eb --- /dev/null +++ b/bootstrap/src/SurfaceLex.lex.tex @@ -0,0 +1,425 @@ +\begin{lexlean}{SurfaceLex} +\useglossary{lexlean.std.nat@1.2.0} +\importmodule{AxiomOutput} +\importmodule{LexLeanStd.Buffer} +\importmodule{LexLeanStd.Text} +\title{Natural number addition} +\begin{semanticmodule} +inductive{ + name: Token, + type_parameters: [], + parameters: [], + constructors: [ + {name: LBrace, fields: []}, + {name: RBrace, fields: []}, + {name: LBrack, fields: []}, + {name: RBrack, fields: []}, + {name: LParen, fields: []}, + {name: RParen, fields: []}, + {name: Comma, fields: []}, + {name: Colon, fields: []}, + {name: ColonEq, fields: []}, + {name: Scope, fields: []}, + {name: Semi, fields: []}, + {name: StrToken, fields: [Bytes]}, + {name: WordToken, fields: [Bytes]} + ] +} + +structure{ + name: Spanned, + type_parameters: [], + parameters: [], + fields: [{name: token, type: Token}, {name: start, type: Nat}, {name: stop, type: Nat}] +} + +structure{ + name: Failure, + type_parameters: [], + parameters: [], + fields: [{name: offset, type: Nat}, {name: message, type: Bytes}] +} + +structure{ + name: Lx, + type_parameters: [], + parameters: [], + fields: [ + {name: pos, type: Nat}, + {name: tokens, type: (List Spanned)}, + {name: error, type: (Option Failure)} + ] +} + +structure{ + name: Scanned, + type_parameters: [], + parameters: [], + fields: [ + {name: pos, type: Nat}, + {name: pieces, type: (List Bytes)}, + {name: finished, type: Bool}, + {name: error, type: (Option Failure)} + ] +} + +structure{ + name: Quoting, + type_parameters: [], + parameters: [], + fields: [{name: pos, type: Nat}, {name: pieces, type: (List Bytes)}] +} + +definition{ + name: lit, + parameters: [literal : String], + result: Bytes, + body: (prim utf8_encode literal : Bytes), + executable: true +} + +definition{ + name: cat, + parameters: [left : Bytes, right : Bytes], + result: Bytes, + body: (prim append left right : Bytes), + executable: true +} + +definition{ + name: byteOf, + parameters: [n : Nat], + result: Bytes, + body: (LexLeanStd.Text::sliceOr 0x000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff n 1), + executable: true +} + +definition{ + name: hexChar, + parameters: [n : Nat], + result: Bytes, + body: (LexLeanStd.Text::sliceOr 0x30313233343536373839616263646566 (prim remainder n 16 0 : Nat) 1), + executable: true +} + +definition{ + name: hexAbove, + parameters: [code : Nat, place : Nat], + result: Bytes, + body: (hexChar (prim quotient code place 0 : Nat)), + executable: true +} + +definition{ + name: hex4, + parameters: [code : Nat], + result: Bytes, + body: (cat (cat (hexAbove code 4096) (hexAbove code 256)) (cat (hexAbove code 16) (hexChar code))), + executable: true +} + +definition{ + name: hex2, + parameters: [code : Nat], + result: Bytes, + body: (cat (hexAbove code 16) (hexChar code)), + executable: true +} + +definition{ + name: joined, + parameters: [pieces : (List Bytes)], + result: Bytes, + body: (LexLeanStd.Buffer::joinAll (LexLeanStd.Text::reverse pieces)), + executable: true +} + +definition{ + name: reverseSpans, + parameters: [items : (List Spanned)], + result: (List Spanned), + body: (prim list_fold (fun (turned : (List Spanned)) (item : Spanned) => (cons item turned)) (nil Spanned) items : (List Spanned)), + executable: true +} + +definition{ + name: byteIs, + parameters: [subject : Bytes, position : Nat, value : UInt8], + result: Bool, + body: (match (LexLeanStd.Text::byteAt subject position) + | Option.some byte => (prim equal byte value : Bool) + | Option.none => false), + executable: true +} + +definition{ + name: byteNum, + parameters: [subject : Bytes, position : Nat], + result: Nat, + body: (match (LexLeanStd.Text::byteAt subject position) + | Option.some byte => (prim to_nat byte : Nat) + | Option.none => 0), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: hexOf, + parameters: [byte : UInt8], + result: (Option Nat), + body: (let value : Nat := (prim to_nat byte : Nat); (if (value (hexOf byte) + | Option.none => (ctor Option.none [Nat])), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: fourHex, + parameters: [subject : Bytes, position : Nat], + result: (Option Nat), + body: (match (hexAt subject position) + | Option.some a => (match (hexAt subject (position + 1)) + | Option.some b => (match (hexAt subject (position + 2)) + | Option.some c => (match (hexAt subject (position + 3)) + | Option.some d => (ctor Option.some [Nat] ((prim multiply ((prim multiply ((prim multiply a 16 : Nat) + b) 16 : Nat) + c) 16 : Nat) + d)) + | Option.none => (ctor Option.none [Nat])) + | Option.none => (ctor Option.none [Nat])) + | Option.none => (ctor Option.none [Nat])) + | Option.none => (ctor Option.none [Nat])), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: utf8Of, + parameters: [code : Nat], + result: Bytes, + body: (if (code (wordStep subject n position)) (prim subtract n start : Nat) start : (Nat * Bool))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: rawStep, + parameters: [subject : Bytes, n : Nat, position : Nat], + result: (Option Nat), + body: (if (position (rawStep subject n position)) (prim subtract n start : Nat) start : (Nat * Bool))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: taken, + parameters: [state : Scanned, stop : Nat, piece : Bytes], + result: Scanned, + body: (record Scanned {pos := stop, pieces := (cons piece (project state pieces)), finished := false, error := (ctor Option.none [Failure])}), + executable: true +} + +definition{ + name: halted, + parameters: [state : Scanned, offset : Nat, message : Bytes], + result: Scanned, + body: (record Scanned {pos := (project state pos), pieces := (project state pieces), finished := false, error := (ctor Option.some [Failure] (failed offset message))}), + executable: true +} + +definition{ + name: unicodeEscape, + parameters: [subject : Bytes, state : Scanned], + result: Scanned, + body: (let position : Nat := (project state pos); (match (fourHex subject (position + 2)) + | Option.some code => (if (55296 <=? code) then (if (code <=? 57343) then (halted state position (cat (cat (lit "`\\u") (hex4 code)) (lit "` is not a Unicode scalar value"))) else (taken state (position + 6) (utf8Of code))) else (taken state (position + 6) (utf8Of code))) + | Option.none => (halted state position (lit "a `\\u` escape takes four hexadecimal digits")))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: escapeAt, + parameters: [subject : Bytes, n : Nat, state : Scanned], + result: Scanned, + body: (let position : Nat := (project state pos); (match (LexLeanStd.Text::byteAt subject (position + 1)) + | Option.none => (record Scanned {pos := n, pieces := (project state pieces), finished := false, error := (ctor Option.none [Failure])}) + | Option.some escape => (if (prim equal escape (34 : UInt8) : Bool) then (taken state (position + 2) (lit "\"")) else (if (prim equal escape (92 : UInt8) : Bool) then (taken state (position + 2) (lit "\\")) else (if (prim equal escape (47 : UInt8) : Bool) then (taken state (position + 2) (lit "/")) else (if (prim equal escape (98 : UInt8) : Bool) then (taken state (position + 2) (byteOf 8)) else (if (prim equal escape (102 : UInt8) : Bool) then (taken state (position + 2) (byteOf 12)) else (if (prim equal escape (110 : UInt8) : Bool) then (taken state (position + 2) (byteOf 10)) else (if (prim equal escape (114 : UInt8) : Bool) then (taken state (position + 2) (byteOf 13)) else (if (prim equal escape (116 : UInt8) : Bool) then (taken state (position + 2) (byteOf 9)) else (if (prim equal escape (117 : UInt8) : Bool) then (unicodeEscape subject state) else (halted state position (cat (cat (lit "`\\") (LexLeanStd.Text::sliceOr subject (position + 1) (widthAt subject (position + 1)))) (lit "` is not an escape")))))))))))))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: scanStep, + parameters: [subject : Bytes, n : Nat, state : Scanned], + result: (Option Scanned), + body: (match (project state error) + | Option.some stopped => (ctor Option.none [Scanned]) + | Option.none => (if (project state finished) then (ctor Option.none [Scanned]) else (let position : Nat := (project state pos); (if (position (scanStep subject n state)) ((prim subtract n start : Nat) + 1) (record Scanned {pos := (start + 1), pieces := (nil Bytes), finished := false, error := (ctor Option.none [Failure])}) : (Scanned * Bool))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: pushed, + parameters: [state : Lx, token : Token, stop : Nat], + result: (Option Lx), + body: (ctor Option.some [Lx] (record Lx {pos := stop, tokens := (cons (record Spanned {token := token, start := (project state pos), stop := stop}) (project state tokens)), error := (ctor Option.none [Failure])})), + executable: true +} + +definition{ + name: quoted, + parameters: [subject : Bytes, n : Nat, state : Lx], + result: (Option Lx), + body: (let start : Nat := (project state pos); (let scanned : Scanned := (readString subject n start); (match (project scanned error) + | Option.some failure => (ctor Option.some [Lx] (record Lx {pos := start, tokens := (project state tokens), error := (ctor Option.some [Failure] failure)})) + | Option.none => (if (project scanned finished) then (pushed state (ctor Token.StrToken (joined (project scanned pieces))) (project scanned pos)) else (ctor Option.some [Lx] (record Lx {pos := start, tokens := (project state tokens), error := (ctor Option.some [Failure] (failed start (lit "unterminated string")))})))))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: colonAt, + parameters: [subject : Bytes, state : Lx], + result: (Option Lx), + body: (let position : Nat := (project state pos); (if (byteIs subject (position + 1) (61 : UInt8)) then (pushed state (ctor Token.ColonEq) (position + 2)) else (if (byteIs subject (position + 1) (58 : UInt8)) then (pushed state (ctor Token.Scope) (position + 2)) else (pushed state (ctor Token.Colon) (position + 1))))), + executable: true +} + +definition{ + name: wordAt, + parameters: [subject : Bytes, n : Nat, state : Lx], + result: (Option Lx), + body: (let position : Nat := (project state pos); (let stop : Nat := (wordEnd subject n position); (pushed state (ctor Token.WordToken (LexLeanStd.Text::sliceOr subject position (prim subtract stop position : Nat))) stop))), + axioms: [Quot.sound, propext], + executable: true +} + +definition{ + name: lexStep, + parameters: [subject : Bytes, n : Nat, state : Lx], + result: (Option Lx), + body: (match (project state error) + | Option.some stopped => (ctor Option.none [Lx]) + | Option.none => (let position : Nat := (project state pos); (if (position (lexStep text n state)) (n + 1) (record Lx {pos := 0, tokens := (nil Spanned), error := (ctor Option.none [Failure])}) : (Lx * Bool))); (match (project final error) + | Option.some failure => (ctor Result.error [(List Spanned), Failure] failure) + | Option.none => (ctor Result.ok [(List Spanned), Failure] (reverseSpans (project final tokens)))))), + axioms: [Quot.sound, propext], + executable: true, + production: {effects: [allocation, overflow], targets: [rust-std]} +} + +definition{ + name: escapePiece, + parameters: [value : Nat], + result: Bytes, + body: (if (value =? 34) then (lit "\\\"") else (if (value =? 92) then (lit "\\\\") else (if (value =? 8) then (lit "\\b") else (if (value =? 12) then (lit "\\f") else (if (value =? 10) then (lit "\\n") else (if (value =? 13) then (lit "\\r") else (if (value =? 9) then (lit "\\t") else (cat (lit "\\u00") (hex2 value))))))))), + executable: true +} + +definition{ + name: quoteStep, + parameters: [subject : Bytes, n : Nat, state : Quoting], + result: (Option Quoting), + body: (let position : Nat := (project state pos); (if (position (quoteStep text n state)) (n + 1) (record Quoting {pos := 0, pieces := (nil Bytes)}) : (Quoting * Bool))); (cat (cat (lit "\"") (joined (project final pieces))) (lit "\"")))), + axioms: [Quot.sound, propext], + executable: true, + production: {effects: [allocation, overflow], targets: [rust-std]} +} +\end{semanticmodule} +\end{lexlean} diff --git a/crates/conformance/src/cases/bootstrap.rs b/crates/conformance/src/cases/bootstrap.rs index 93f612a4a4..af7190bffb 100644 --- a/crates/conformance/src/cases/bootstrap.rs +++ b/crates/conformance/src/cases/bootstrap.rs @@ -34,6 +34,7 @@ mod rust_terms; mod rust_validate; mod scan; mod structural_grammar; +mod surface_lex; /// The module the first component lives in. const MODULE: &str = "LexLeanBoot.AxiomOutput"; @@ -2454,6 +2455,7 @@ fn case(id: &str) { "BS-14" => rust_assemble::case(), "BS-15" => rust_package::case(), "BS-21" => structural_grammar::case(), + "BS-22" => surface_lex::case(), other => panic!("no bootstrap case is wired for {other}"), } } diff --git a/crates/conformance/src/cases/bootstrap/surface_lex.rs b/crates/conformance/src/cases/bootstrap/surface_lex.rs new file mode 100644 index 0000000000..dbe25f4d34 --- /dev/null +++ b/crates/conformance/src/cases/bootstrap/surface_lex.rs @@ -0,0 +1,1275 @@ +//! `BS-22` (SPEC.md §17.23, §17.21): the lexing of the authoring notation as +//! language-1.2 source, `LexLeanBoot.SurfaceLex`, held to the byte to +//! `lexlean::surface::lex::{lex, quote}`. +//! +//! `lexText` reads a text and returns its tokens with their byte ranges, or the +//! offset and message of the first lexical error; `quoteText` spells a text as +//! the notation quotes it. Neither recurses: a scan is a loop under +//! `iterate_until`, and a string is a loop over its pieces, joined once at the +//! end. +//! +//! Each planted defect names the one vector that catches it. `BS-05` and +//! `BS-11` each lost a plant when their corpus grew, because a plant caught by +//! "some run of the corpus" is only as good as that corpus. + +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::OnceLock; + +use lexlean::calculus::Value; +use lexlean::surface::lex::{lex, quote, Spanned, Token}; +use lexlean::surface::SurfaceError; +use serde_json::Value as Json; + +use super::rust_validate::{eligibility_of, rustc_differential}; +use super::{list, nat_val, text, Run, Seeded}; +use crate::production_export as harness; +use crate::support::{self, repo_root}; + +const MODULE: &str = "LexLeanBoot.SurfaceLex"; +const SOURCE: &str = "bootstrap/src/SurfaceLex.lex.tex"; +const ROOTS: [(&str, &[&str]); 2] = [ + ("lexText", &["allocation", "overflow"]), + ("quoteText", &["allocation", "overflow"]), +]; + +/// The most a part of a certificate of a root may be, as the row states. +const CERTIFICATE_BUDGET: u64 = 3_145_728; +/// The bytes of each committed source the evaluator reads, and rustc reads: a +/// window from a place the path fixes. The harness writes a text as an array +/// of its bytes, so the total under rustc is held to about a megabyte. +const EVALUATOR_WINDOW: usize = 192; +const RUSTC_WINDOW: usize = 1_024; +/// Seeded edits under the evaluator, and under rustc. +const EVALUATOR_MUTANTS: usize = 600; +const RUSTC_MUTANTS: usize = 2_400; + +// ---------------------------------------------------------------- the sorts + +/// The constructors of each inductive of the module, in declaration order, +/// read from its text, so that the encoders name a constructor and never +/// number it, and one moved in the module fails here. +fn sorts() -> &'static BTreeMap> { + static SORTS: OnceLock>> = OnceLock::new(); + SORTS.get_or_init(|| { + let source = + std::fs::read_to_string(repo_root().join(SOURCE).as_std_path()).expect("the module"); + let mut out = BTreeMap::new(); + for block in source.split("inductive{").skip(1) { + let block = block.split("\n}\n").next().expect("a block"); + let name = block + .split("name: ") + .nth(1) + .and_then(|rest| rest.split(',').next()) + .expect("a sort name") + .trim() + .to_owned(); + let constructors: Vec = block + .split("{name: ") + .skip(1) + .map(|rest| { + rest.split(',') + .next() + .expect("a constructor") + .trim() + .to_owned() + }) + .collect(); + out.insert(name, constructors); + } + out + }) +} + +fn adt(sort: &str, constructor: &str, fields: Vec) -> Value { + let found = sorts() + .get(sort) + .unwrap_or_else(|| panic!("the module has no sort `{sort}`")) + .iter() + .position(|name| name == constructor) + .unwrap_or_else(|| panic!("`{sort}` has no constructor `{constructor}`")); + Value::Adt { + constructor: u64::try_from(found).expect("an index"), + fields, + } +} + +/// A record: a structure is the one constructor of its own sort. +fn record(fields: Vec) -> Value { + Value::Adt { + constructor: 0, + fields, + } +} + +// ------------------------------------------------------------- the encoders + +fn enc_token(token: &Token) -> Value { + let plain = |name: &str| adt("Token", name, Vec::new()); + match token { + Token::LBrace => plain("LBrace"), + Token::RBrace => plain("RBrace"), + Token::LBrack => plain("LBrack"), + Token::RBrack => plain("RBrack"), + Token::LParen => plain("LParen"), + Token::RParen => plain("RParen"), + Token::Comma => plain("Comma"), + Token::Colon => plain("Colon"), + Token::ColonEq => plain("ColonEq"), + Token::Scope => plain("Scope"), + Token::Semi => plain("Semi"), + Token::Str(value) => adt("Token", "StrToken", vec![text(value)]), + Token::Word(value) => adt("Token", "WordToken", vec![text(value)]), + } +} + +fn enc_spanned(spanned: &Spanned) -> Value { + record(vec![ + enc_token(&spanned.token), + nat_val(spanned.start), + nat_val(spanned.end), + ]) +} + +fn enc_failure(failure: &SurfaceError) -> Value { + record(vec![nat_val(failure.offset), text(&failure.message)]) +} + +/// The run of `lexText` on a text, and the oracle's message when it refuses. +fn lex_run(source: &str) -> (Run, Option) { + let (value, refusal) = match lex(source) { + Ok(tokens) => ( + Value::Ok { + value: Box::new(list(tokens.iter().map(enc_spanned).collect())), + }, + None, + ), + Err(failure) => ( + Value::Error { + value: Box::new(enc_failure(&failure)), + }, + Some(failure.message), + ), + }; + ( + ( + vec![text(source)], + serde_json::to_value(value).expect("a value is JSON"), + ), + refusal, + ) +} + +/// The run of `quoteText` on a text. +fn quote_run(source: &str) -> Run { + ( + vec![text(source)], + serde_json::to_value(text("e(source))).expect("a value is JSON"), + ) +} + +// ------------------------------------------------------------------ vectors + +/// One named vector: a text, and the root it is a run of. +struct Vector { + label: String, + quoting: bool, + source: String, +} + +impl Vector { + fn run(&self) -> (Run, Option) { + if self.quoting { + (quote_run(&self.source), None) + } else { + lex_run(&self.source) + } + } +} + +/// The scalars `char::is_whitespace` accepts that are not ASCII, one of each +/// width: the oracle splits a word at each. +const WHITE: &[&str] = &[ + "\u{b}", "\u{c}", "\r", "\u{85}", "\u{a0}", "\u{1680}", "\u{2000}", "\u{2003}", "\u{200a}", + "\u{2028}", "\u{2029}", "\u{202f}", "\u{205f}", "\u{3000}", +]; +/// Scalars next to white space that are none. +const LOOKALIKES: &[&str] = &[ + "\u{200b}", + "\u{180e}", + "\u{feff}", + "\u{2060}", + "\u{1c}", + "\u{1f}", + "\u{a1}", + "\u{1fff}", + "\u{2010}", + "\u{2029}\u{2029}", +]; + +#[allow(clippy::too_many_lines)] +fn named() -> Vec { + let mut out: Vec = Vec::new(); + let mut lexed = |label: &str, source: &str| { + out.push(Vector { + label: label.to_owned(), + quoting: false, + source: source.to_owned(), + }); + }; + + // Tokens and words. + lexed("empty", ""); + lexed("white space only", " \n\t \u{a0} "); + lexed( + "the oracle's own text", + "add{left: x, right: 1} := (a -> b);", + ); + for (label, source) in [ + ("an opening brace", "{"), + ("a closing brace", "}"), + ("an opening bracket", "["), + ("a closing bracket", "]"), + ("an opening parenthesis", "("), + ("a closing parenthesis", ")"), + ("a comma", ","), + ("a semicolon", ";"), + ("a colon", ":"), + ("a colon equals", ":="), + ("a double colon", "::"), + ] { + lexed(label, source); + } + for (label, source) in [ + ("a colon and a colon equals", "::="), + ("a colon equals and a colon", ":=:"), + ("three colons", ":::"), + ("two colon equals", ":=:="), + ("a colon before the end", "a:"), + ("a colon after a word", "a: b"), + ("a colon equals inside a word", "a:=b"), + ("a double colon inside a word", "a::b"), + ("a colon equals at the end", "x :="), + ("every delimiter", "{}[](),:;"), + ("every delimiter and white space", "{ } [ ] ( ) , : ; :: :="), + ("a word", "word"), + ("a word of symbols", "->=>+-*/<>=!?#%&|^~@$"), + ( + "a word with a dot, a quote mark, and a hyphen", + "a.b'c-d_e/f", + ), + ("a number", "1234567890"), + ("a negative number", "-12"), + ("a word ended by a comma", "a,b"), + ("a word ended by a semicolon", "a;b"), + ("a word ended by a colon", "a:b"), + ("a word ended by a brace", "a{b}c"), + ("a word ended by a bracket", "a[b]c"), + ("a word ended by a parenthesis", "a(b)c"), + ("a word ended by a quote mark", "a\"b\"c"), + ("a tab between words", "a\tb"), + ("a line feed between words", "a\nb"), + ("a carriage return between words", "a\rb"), + ("a form feed between words", "a\u{c}b"), + ("a vertical tab between words", "a\u{b}b"), + ("a no-break space between words", "a\u{a0}b"), + ("a next line between words", "a\u{85}b"), + ("an ogham space mark between words", "a\u{1680}b"), + ("an en quad between words", "a\u{2000}b"), + ("a hair space between words", "a\u{200a}b"), + ("a line separator between words", "a\u{2028}b"), + ("a paragraph separator between words", "a\u{2029}b"), + ("a narrow no-break space between words", "a\u{202f}b"), + ("a medium mathematical space between words", "a\u{205f}b"), + ("an ideographic space between words", "a\u{3000}b"), + ( + "a word of scalars beyond ASCII", + "\u{e9}\u{2115}\u{65e5}\u{1f600}", + ), + ( + "a word of a scalar before white space", + "\u{e9}\u{a0}\u{e9}", + ), + ( + "scalars beyond ASCII among delimiters", + "\u{2115}:\u{1f600},\u{3b1}", + ), + ("a control character in a word", "a\u{0}b\u{1}c\u{7f}d"), + ("a nul alone", "\u{0}"), + ] { + lexed(label, source); + } + for (index, white) in WHITE.iter().enumerate() { + lexed( + &format!("white space {index} around words"), + &format!("{white}a{white}{white}b{white}"), + ); + } + for (index, other) in LOOKALIKES.iter().enumerate() { + lexed( + &format!("lookalike {index} among words"), + &format!("a{other}b {other} {other}c"), + ); + } + + // Strings. + for (label, source) in [ + ("an empty string", "\"\""), + ("a string", "\"abc\""), + ("a string with a space", "\"a b\""), + ("a string after tokens", "{ a: \"x\" }"), + ("a string beyond ASCII", "\"\u{e9}\u{2115}\u{1f600}\""), + ("a string with a delete", "\"a\u{7f}b\""), + ("a string with a brace and a colon equals", "\"{:=}\""), + ("two strings", "\"a\"\"b\""), + ("a string between words", "a\"b\"c"), + ( + "a string with a raw high scalar after an escape", + "\"\\n\u{e9}\"", + ), + ("a string with an escape after raw text", "\"ab\\ncd\""), + ("an escaped quote mark", "\"a\\\"b\""), + ("an escaped backslash", "\"a\\\\b\""), + ("an escaped solidus", "\"a\\/b\""), + ("an escaped backspace", "\"\\b\""), + ("an escaped form feed", "\"\\f\""), + ("an escaped line feed", "\"\\n\""), + ("an escaped carriage return", "\"\\r\""), + ("an escaped tab", "\"\\t\""), + ("every escape", "\"\\\"\\\\\\/\\b\\f\\n\\r\\t\""), + ("an escaped backslash before a quote mark", "\"a\\\\\" b"), + ("an escape at the end of a string", "\"\\n\""), + ("a unicode escape", "\"\\u0041\""), + ("a unicode escape of a lower-case digit", "\"\\u00e9\""), + ("a unicode escape of an upper-case digit", "\"\\u00E9\""), + ("a unicode escape of a mixed-case digit", "\"\\uAbCd\""), + ("a unicode escape of nul", "\"\\u0000\""), + ("a unicode escape of the delete", "\"\\u007f\""), + ("a unicode escape of the last ASCII scalar", "\"\\u007e\""), + ( + "a unicode escape of the first two-byte scalar", + "\"\\u0080\"", + ), + ( + "a unicode escape of the last two-byte scalar", + "\"\\u07ff\"", + ), + ( + "a unicode escape of the first three-byte scalar", + "\"\\u0800\"", + ), + ( + "a unicode escape of the last scalar before the surrogates", + "\"\\ud7ff\"", + ), + ( + "a unicode escape of the first scalar after the surrogates", + "\"\\ue000\"", + ), + ("a unicode escape of the last scalar", "\"\\uffff\""), + ("a unicode escape of a letter", "\"\\u2115\""), + ("a unicode escape of a number", "\"\\u1234\""), + ( + "a unicode escape of each place", + "\"\\u1000\\u0100\\u0010\\u0001\"", + ), + ("a unicode escape among raw text", "\"a\\u00e9b\\u2115c\""), + ("a unicode escape of a quote mark", "\"\\u0022\""), + ("two unicode escapes", "\"\\u0041\\u0042\""), + ] { + lexed(label, source); + } + + // Lexical errors. + for (label, source) in [ + ("an unterminated string", "\"abc"), + ("an unterminated string after tokens", "{ a: b } \"abc"), + ("an unterminated string after a word", "word \"abc"), + ("an unterminated string at the end of the text", "\""), + ( + "an unterminated string after white space beyond ASCII", + "\u{a0}\u{3000}\"abc", + ), + ( + "an unterminated string after scalars beyond ASCII", + "\u{e9}\u{2115} \"abc", + ), + ("a string ending with a backslash", "\"abc\\"), + ("a backslash alone in a string", "\"\\"), + ("an escape at the end of the text", "\"a\\n"), + ("a quote mark escaped at the end", "\"a\\\""), + ("an escape that is not one", "\"\\q\""), + ("an escape of a digit", "\"\\1\""), + ("an escape of a space", "\"\\ \""), + ("an escape of a quote mark and a letter", "\"\\\"\\x\""), + ("an escape of a scalar beyond ASCII", "\"\\\u{e9}\""), + ("an escape of a three-byte scalar", "\"\\\u{2115}\""), + ("an escape of a four-byte scalar", "\"\\\u{1f600}\""), + ("an escape of a line feed", "\"\\\n\""), + ("an escape of a nul", "\"\\\u{0}\""), + ("an escape of an upper-case letter", "\"\\N\""), + ("an escape after tokens", "a b \"\\q\""), + ( + "an escape after scalars beyond ASCII", + "\u{e9}\u{2115} \"\\q\"", + ), + ("a unicode escape of no digit", "\"\\u\""), + ("a unicode escape of one digit", "\"\\u1\""), + ("a unicode escape of two digits", "\"\\u12\""), + ("a unicode escape of three digits", "\"\\u123\""), + ("a unicode escape of three digits and the end", "\"\\u123"), + ( + "a unicode escape of a letter that is no digit", + "\"\\u12g4\"", + ), + ("a unicode escape of a letter first", "\"\\ug234\""), + ("a unicode escape of a letter last", "\"\\u123g\""), + ("a unicode escape of a sign", "\"\\u+123\""), + ("a unicode escape of a hyphen", "\"\\u-123\""), + ( + "a unicode escape of a scalar beyond ASCII", + "\"\\u12\u{e9}4\"", + ), + ("a unicode escape cut short by a quote mark", "\"\\u12\"4"), + ( + "a unicode escape of the digits just short of the first lower-case letter", + "\"\\u000`\"", + ), + ( + "a unicode escape of the digits just past the last lower-case letter", + "\"\\u000g\"", + ), + ( + "a unicode escape of the digits just short of the first upper-case letter", + "\"\\u000@\"", + ), + ( + "a unicode escape of the digits just past the last upper-case letter", + "\"\\u000G\"", + ), + ( + "a unicode escape of the character just past nine", + "\"\\u000:\"", + ), + ( + "a unicode escape of the character just short of zero", + "\"\\u000/\"", + ), + ("a first surrogate", "\"\\ud800\""), + ("a last high surrogate", "\"\\udbff\""), + ("a first low surrogate", "\"\\udc00\""), + ("a last surrogate", "\"\\udfff\""), + ("a surrogate pair", "\"\\ud83d\\ude00\""), + ("a surrogate in upper case", "\"\\uD800\""), + ("a surrogate after tokens", "a: \"\\ud800\""), + ("a surrogate after text", "\"ab\\ud800\""), + ("a nul in a string", "\"a\u{0}b\""), + ("a start of heading in a string", "\"\u{1}\""), + ("a line feed in a string", "\"a\nb\""), + ("a tab in a string", "\"a\tb\""), + ("a carriage return in a string", "\"a\rb\""), + ("a unit separator in a string", "\"\u{1f}\""), + ("a control character after tokens", "a: \"b\u{1}\""), + ( + "a control character after scalars beyond ASCII", + "\u{e9}\u{2115} \"\u{e9}\u{7}\"", + ), + ("a control character after an escape", "\"\\n\u{1}\""), + ("an error after a string", "\"ok\" \"\\q\""), + ( + "an error after a first error is not read", + "\"\\q\" \"\\u12\" \"abc", + ), + ("an unterminated string after strings", "\"a\" \"b\" \"c"), + ] { + lexed(label, source); + } + + // The quoting of a text. + let mut quoted = |label: &str, source: &str| { + out.push(Vector { + label: label.to_owned(), + quoting: true, + source: source.to_owned(), + }); + }; + quoted("quote the empty text", ""); + quoted("quote a text", "plain"); + quoted("quote a space", " "); + quoted("quote a quote mark", "a\"b"); + quoted("quote a backslash", "a\\b"); + quoted("quote a backslash and a quote mark", "\\\""); + quoted("quote a backspace", "\u{8}"); + quoted("quote a form feed", "\u{c}"); + quoted("quote a line feed", "a\nb"); + quoted("quote a carriage return", "\r"); + quoted("quote a tab", "\t"); + quoted("quote the delete", "a\u{7f}b"); + quoted("quote the first scalar not escaped", " "); + quoted("quote scalars beyond ASCII", "\u{e9}\u{2115}\u{1f600}"); + quoted( + "quote scalars beyond ASCII among escapes", + "\u{e9}\n\u{2115}\"\u{1f600}", + ); + quoted( + "quote a long text of nothing to escape", + &"abcdefgh".repeat(200), + ); + quoted("quote a long text of escapes", &"\n\"\\".repeat(100)); + quoted( + "quote a text of every control character", + &(0..32u8).map(char::from).collect::(), + ); + for value in 0..32u8 { + quoted( + &format!("quote U+{value:04X}"), + &format!("a{}b", char::from(value)), + ); + } + quoted( + "quote a text the notation reads back", + "add{left: x} := \"y\"\n", + ); + out +} + +/// Every error the lexer can state, by a part of its message, so that a corpus +/// that never reached one fails here and not silently (R2). +const REFUSALS: &[&str] = &[ + "unterminated string", + "a `\\u` escape takes four hexadecimal digits", + "is not a Unicode scalar value", + "is not an escape", + "a control character must be escaped in a string", +]; + +// ------------------------------------------------------------------- seeded + +/// A whole token or a part of one, of each shape the notation reads. +const PIECES: &[&str] = &[ + "{", + "}", + "[", + "]", + "(", + ")", + ",", + ":", + ":=", + "::", + ";", + " ", + " ", + "\n", + "\t", + "\r", + "\u{a0}", + "\u{2003}", + "\u{3000}", + "\u{85}", + "\u{200b}", + "word", + "a-b", + "x1", + "->", + "=>", + "1", + "-1", + "\"", + "\"\"", + "\"abc\"", + "\"a b\"", + "\"\\n\"", + "\"\\\"\"", + "\"\\\\\"", + "\"\\/\"", + "\"\\u00e9\"", + "\"\\u00E9\"", + "\"\\ud800\"", + "\"\\udfff\"", + "\"\\ue000\"", + "\"\\q\"", + "\"\\u12\"", + "\"\\u12g4\"", + "\\", + "\\u", + "\\n", + "\\u00", + "\u{e9}", + "\u{2115}", + "\u{1f600}", + "\u{0}", + "\u{1}", + "\u{1f}", + "\u{7f}", + "\"\u{1}\"", + "\"\n\"", + "\"\u{e9}\"", +]; + +/// One seeded edit of a text, at a scalar boundary: a deletion, an insertion, +/// or a replacement, up to four of them. +fn edited(seeded: &mut Seeded, source: &str) -> String { + let mut scalars: Vec = source.chars().collect(); + for _ in 0..=seeded.below(4) { + let at = seeded.below(scalars.len() + 1); + let piece: Vec = seeded.pick(PIECES).chars().collect(); + match seeded.below(3) { + 0 if at < scalars.len() => { + scalars.remove(at); + } + 1 => { + drop(scalars.splice(at..at, piece)); + } + _ if at < scalars.len() => { + drop(scalars.splice(at..=at, piece)); + } + _ => scalars.extend(piece), + } + } + scalars.into_iter().collect() +} + +/// A text of pieces, one edit away from nothing: an edge text. +fn edge_text(seeded: &mut Seeded) -> String { + let mut out = String::new(); + for _ in 0..=seeded.below(12) { + out.push_str(seeded.pick(PIECES)); + if seeded.chance(40) { + out.push_str(seeded.pick(&[" ", "\n", " "])); + } + } + out +} + +/// The seeded corpus: `count` edits of the texts of the vectors, of windows of +/// the committed sources, and edge texts, as runs of `lexText`, and the same +/// number of runs of `quoteText` on them. +fn mutants(count: usize, bases: &[String]) -> (Vec, Vec) { + let mut seeded = Seeded(0x0B55_0022); + let mut lexing = Vec::with_capacity(count); + let mut quoting = Vec::with_capacity(count); + while lexing.len() < count { + let source = if seeded.chance(25) { + edge_text(&mut seeded) + } else { + let base = seeded.pick(bases).clone(); + edited(&mut seeded, &base) + }; + lexing.push(lex_run(&source).0); + if seeded.chance(40) { + quoting.push(quote_run(&source)); + } + } + (lexing, quoting) +} + +/// A hash of a path (FNV-1a), which fixes where a window lies. +fn path_hash(path: &str) -> u64 { + path.bytes().fold(0xcbf2_9ce4_8422_2325_u64, |hash, byte| { + (hash ^ u64::from(byte)).wrapping_mul(0x0100_0000_01b3) + }) +} + +/// A window of `bytes` bytes of a text, from a place its path fixes, at +/// scalar boundaries. +fn window(path: &str, source: &str, bytes: usize) -> String { + let mut start = if source.len() > bytes { + usize::try_from(path_hash(path) % u64::try_from(source.len() - bytes).expect("a length")) + .expect("an offset") + } else { + 0 + }; + while !source.is_char_boundary(start) { + start -= 1; + } + let mut end = (start + bytes).min(source.len()); + while !source.is_char_boundary(end) { + end -= 1; + } + source[start..end].to_owned() +} + +// ------------------------------------------------------------------- plants + +/// A defect planted in the source, and the named vector that catches it. +struct Plant { + label: &'static str, + root: usize, + site: &'static str, + replacement: &'static str, + vector: &'static str, +} + +const PLANTS: &[Plant] = &[ + Plant { + label: "a word ends only at an ASCII space", + root: 0, + site: "(if (0 eprintln!( + "BS-22: plant detected: {}: {}", + plant.label, + first.chars().take(160).collect::() + ), + None => missed.push(plant.label), + } + } + assert!( + missed.is_empty(), + "planted defects their own vector does not detect: {missed:?}" + ); +} + +fn export(lowered: &BTreeMap<&str, super::Lowered>) { + let exported = harness::exported("bootstrap"); + let bundle = exported.bundle(); + let mut realized = 0usize; + for exported_root in &exported.output.result.roots { + let Some(member) = exported_root + .root + .strip_prefix(&format!("{MODULE}.")) + .and_then(|member| lowered.get(member).map(|low| (member, low))) + else { + continue; + }; + let largest = exported_root + .outputs + .iter() + .flat_map(|output| &output.certification.certificates) + .map(|certificate| certificate.file.byte_length) + .max() + .expect("a certificate"); + eprintln!( + "BS-22: {}: largest certificate part {largest} bytes", + exported_root.root + ); + assert!( + largest <= CERTIFICATE_BUDGET, + "{}: a certificate part of {largest} bytes is beyond {CERTIFICATE_BUDGET}", + exported_root.root + ); + for output in &exported_root.outputs { + let package: Json = serde_json::from_slice( + &std::fs::read( + bundle + .join(&output.package.directory) + .join("package.json") + .as_std_path(), + ) + .expect("package.json"), + ) + .expect("package JSON"); + assert_eq!( + package["program"], + serde_json::to_value(&member.1.program).expect("a program is JSON"), + "{}: the exported package realizes the program the differential ran", + member.0 + ); + realized += 1; + } + } + assert_eq!(realized, ROOTS.len(), "one certified package per root"); + let packages = harness::tree(&bundle.join("packages")); + let committed_packages = harness::tree(&harness::fixture_dir("bootstrap").join("packages")); + assert_eq!( + packages.keys().collect::>(), + committed_packages.keys().collect::>(), + "the exported package files are the committed ones" + ); + for (relative, content) in &packages { + assert!( + committed_packages.get(relative) == Some(content), + "packages/{relative} differs from the committed package" + ); + } +} + +#[allow(clippy::too_many_lines)] +pub(super) fn case() { + let started = std::time::Instant::now(); + let project = support::P::bootstrap(); + let checked_units = project.check_ok(); + assert!( + checked_units.units.contains_key("SurfaceLex") + && checked_units.units.contains_key("AxiomOutput"), + "SurfaceLex and AxiomOutput are linked modules of the project" + ); + eligibility_of(&project, "SurfaceLex", MODULE, &ROOTS); + assert_eq!( + sorts().keys().map(String::as_str).collect::>(), + ["Token"], + "the sorts of the module" + ); + + let vectors = named(); + let labels: BTreeSet<&str> = vectors.iter().map(|vector| vector.label.as_str()).collect(); + assert_eq!(labels.len(), vectors.len(), "each vector has its own label"); + let mut messages = Vec::new(); + let mut lexing: Vec = Vec::new(); + let mut quoting: Vec = Vec::new(); + for vector in &vectors { + let (run, refusal) = vector.run(); + messages.extend(refusal); + if vector.quoting { + quoting.push(run); + } else { + lexing.push(run); + } + } + for refusal in REFUSALS { + assert!( + messages.iter().any(|message| message.contains(refusal)), + "no vector reaches the error `{refusal}`" + ); + } + let accepted = lexing.iter().filter(|run| run.1["kind"] == "ok").count(); + + let sources = super::scan::sources(); + let evaluator_windows: Vec = sources + .iter() + .map(|(path, source)| window(path, source, EVALUATOR_WINDOW)) + .collect(); + let rustc_windows: Vec = sources + .iter() + .map(|(path, source)| window(path, source, RUSTC_WINDOW)) + .collect(); + let normative = support::normative_host(); + let (mut evaluator_lexing, mut evaluator_quoting) = (Vec::new(), Vec::new()); + let (mut rustc_lexing, mut rustc_quoting) = (Vec::new(), Vec::new()); + for source in &evaluator_windows { + evaluator_lexing.push(lex_run(source).0); + evaluator_quoting.push(quote_run(source)); + } + for source in &rustc_windows { + rustc_lexing.push(lex_run(source).0); + rustc_quoting.push(quote_run(source)); + } + let bases: Vec = vectors + .iter() + .map(|vector| vector.source.clone()) + .chain(evaluator_windows.iter().cloned()) + .collect(); + let (mut evaluator_mutants, mut evaluator_quoted) = if normative { + mutants(EVALUATOR_MUTANTS, &bases) + } else { + (Vec::new(), Vec::new()) + }; + let (mut rustc_mutants, mut rustc_quoted) = if normative { + mutants(RUSTC_MUTANTS, &bases) + } else { + (Vec::new(), Vec::new()) + }; + eprintln!( + "BS-22: {} vectors ({} lexed, {} accepted, {} quoted), {} committed sources, {} and {} seeded runs ({:?})", + vectors.len(), + lexing.len(), + accepted, + quoting.len(), + sources.len(), + evaluator_mutants.len(), + rustc_mutants.len(), + started.elapsed() + ); + + let checked = support::checked_project(&project); + let lowered: BTreeMap<&str, super::Lowered> = ROOTS + .iter() + .map(|(root, _)| (*root, super::lower(&checked, &project, MODULE, root))) + .collect(); + if normative { + let mut lex_corpus: Vec = lexing.clone(); + lex_corpus.append(&mut evaluator_lexing); + lex_corpus.append(&mut evaluator_mutants); + let mut quote_corpus: Vec = quoting.clone(); + quote_corpus.append(&mut evaluator_quoting); + quote_corpus.append(&mut evaluator_quoted); + for (root, corpus) in [("lexText", &lex_corpus), ("quoteText", "e_corpus)] { + let disagreements = super::evaluate(&lowered[root], corpus, 5); + assert!( + disagreements.is_empty(), + "{root} and the Rust oracle disagree: {disagreements:?}" + ); + eprintln!( + "BS-22: {root}: the evaluator agrees on {} runs ({:?})", + corpus.len(), + started.elapsed() + ); + } + plants(&vectors); + } else { + eprintln!("BS-22: platform-bound host: the evaluator, the seeded runs, and the planted defects run on the normative host"); + } + + let mut compiled_lexing = lexing; + compiled_lexing.append(&mut rustc_lexing); + compiled_lexing.append(&mut rustc_mutants); + let mut compiled_quoting = quoting; + compiled_quoting.append(&mut rustc_quoting); + compiled_quoting.append(&mut rustc_quoted); + let total = rustc_differential( + "BS-22", + &[ + ("lexText", &lowered["lexText"], compiled_lexing.as_slice()), + ( + "quoteText", + &lowered["quoteText"], + compiled_quoting.as_slice(), + ), + ], + "oracle", + ); + eprintln!("BS-22: {total} rustc runs agree ({:?})", started.elapsed()); + + if !support::lean_backed("BS-22") { + return; + } + export(&lowered); + eprintln!("BS-22: passed in {:?}", started.elapsed()); +} diff --git a/crates/conformance/src/cases/repository.rs b/crates/conformance/src/cases/repository.rs index 4237ccd569..2da16b1b48 100644 --- a/crates/conformance/src/cases/repository.rs +++ b/crates/conformance/src/cases/repository.rs @@ -813,7 +813,7 @@ pub(crate) fn run(id: &str) { let model = repo_model::Model::load(&root.join("model").into_std_path_buf()) .expect("the model loads"); let table = spec_table(); - assert_eq!(table.len(), 469, "§31 has 469 rows"); + assert_eq!(table.len(), 470, "§31 has 470 rows"); assert_eq!(model.ids.id.len(), table.len(), "register row count"); for ((spec_id, spec_suite, spec_statement), row) in table.iter().zip(model.ids.id.iter()) diff --git a/crates/conformance/tests/conformance.rs b/crates/conformance/tests/conformance.rs index 587c6360b6..ef5121e0ea 100644 --- a/crates/conformance/tests/conformance.rs +++ b/crates/conformance/tests/conformance.rs @@ -2275,6 +2275,11 @@ fn conformance_bs_21() { repo_conformance::cases::run("BS-21"); } +#[test] +fn conformance_bs_22() { + repo_conformance::cases::run("BS-22"); +} + #[test] fn conformance_pk_01() { repo_conformance::cases::run("PK-01"); diff --git a/features/suites/bootstrap.feature b/features/suites/bootstrap.feature index 33b2fc20d5..13048781be 100644 --- a/features/suites/bootstrap.feature +++ b/features/suites/bootstrap.feature @@ -113,3 +113,10 @@ Feature: bootstrap When the project is checked and built, and the lowered program of the root and its rendered package are run on the atoms the oracle's scanner returns Then the root is eligible for rust-std with exactly its declared effects, no certificate part is beyond 3,145,728 bytes, and the root equals `parse_module` on every run, every refusal of which a vector reaches And every planted defect is detected by its own vector, and the exported packages are byte-identical to the committed packages and realize the program the differential ran + + @BS-22 @build + Scenario: The lexing of the authoring notation of SPEC.md §17.21 (`surface::lex::lex`: a text into its delimiters, `:=` and `::`, quoted strings with their escapes decoded, and words, which end at white space in the sense of `char::is_whitespace` or at a delimiter, each with its byte range, or the offset and message of the first lexical error; and `surface::lex::quote`: a text as the notation quotes it, with the quotation mark, the backslash, and every control character escaped) is language-1.2 source in the `bootstrap/` project, as two production roots of `SurfaceLex`: `lexText` and `quoteText`, each eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and with no part of its certificate beyond 3,145,728 bytes; on every host their rendered `rust-std` packages under the pinned rustc, and on the normative host their lowered programs under the calculus evaluator, agree with the oracle to the byte on the hand-built vectors, which reach every lexical error and every escape, on windows of the committed sources, and on seeded edits of those texts; each planted defect in the source is detected by the one vector it names, and the certified packages the production export publishes for the project are the committed ones; the readers of the tokens (the generic notation and the sugar of the closed schemas), the pretty-printing algebra, and canonical JSON are outside the claim. + Given the committed `bootstrap` project, hand-built vectors of every token, escape, and lexical error, windows of the committed sources, and seeded edits of those texts + When the project is checked and built, and the lowered program and the rendered package of each root are run on those texts + Then each root is eligible for rust-std with exactly its declared effects, no certificate part is beyond 3,145,728 bytes, and `lexText` and `quoteText` equal `lex` and `quote` on every run, every lexical error of which a vector reaches + And every planted defect is detected by its own vector, and the exported packages are byte-identical to the committed packages and realize the programs the differential ran diff --git a/model/ids.toml b/model/ids.toml index dfa16cf2dd..d7a6e2041b 100644 --- a/model/ids.toml +++ b/model/ids.toml @@ -2732,6 +2732,12 @@ level = "build" suite = "bootstrap" statement = "The fixed structural grammar of SPEC.md §15.1 and §15.2 (`grammar::structural::parse_module`: the module header, the blocks and declarations, the axiom policies, the sentences, the structured proof environments, and the closed data forms, with the coverage rows of every structural atom it consumes) is language-1.2 source in the `bootstrap/` project: `parseModule` of `StructuralGrammar`, for the atoms `scanSource` of `BS-03` returns, the path, the name the path gives the module, and `max_scope_depth`, is a production root eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and has no part of its certificate beyond 3,145,728 bytes; it returns the module with its rows, or the refusal's code, message, and span, and on 267 hand-built vectors, 146 of them refused, which reach every refusal of `parse_module`, and on the committed modules, whole or by the atoms of both their ends, its lowered program under the calculus evaluator on the normative host and its rendered package under the pinned rustc on every host agree with the oracle to the byte; on the normative host both also agree on 2,000 seeded edits of those texts, each of 44 planted defects in the source is detected by the one vector it names, and the certified package the production export publishes for the project is the committed one; the grammars of mathematical islands, propositions, and proof sentences, which read the lexicon's closure, and the interpretation of the module this root returns are outside the claim." +[[id]] +id = "BS-22" +level = "build" +suite = "bootstrap" +statement = "The lexing of the authoring notation of SPEC.md §17.21 (`surface::lex::lex`: a text into its delimiters, `:=` and `::`, quoted strings with their escapes decoded, and words, which end at white space in the sense of `char::is_whitespace` or at a delimiter, each with its byte range, or the offset and message of the first lexical error; and `surface::lex::quote`: a text as the notation quotes it, with the quotation mark, the backslash, and every control character escaped) is language-1.2 source in the `bootstrap/` project, as two production roots of `SurfaceLex`: `lexText` and `quoteText`, each eligible for `rust-std` with exactly its declared effects, none of them `recursion`, and with no part of its certificate beyond 3,145,728 bytes; on every host their rendered `rust-std` packages under the pinned rustc, and on the normative host their lowered programs under the calculus evaluator, agree with the oracle to the byte on the hand-built vectors, which reach every lexical error and every escape, on windows of the committed sources, and on seeded edits of those texts; each planted defect in the source is detected by the one vector it names, and the certified packages the production export publishes for the project are the committed ones; the readers of the tokens (the generic notation and the sugar of the closed schemas), the pretty-printing algebra, and canonical JSON are outside the claim." + [[id]] id = "PK-01" level = "build" diff --git a/tests/production-export/bootstrap/roots.json b/tests/production-export/bootstrap/roots.json index 692aa4a7bb..918e3ecfef 100644 --- a/tests/production-export/bootstrap/roots.json +++ b/tests/production-export/bootstrap/roots.json @@ -104,6 +104,14 @@ ], "version": "1.0.0" }, + { + "name": "lex_text", + "root": "LexLeanBoot.SurfaceLex.lexText", + "targets": [ + "rust-std" + ], + "version": "1.0.0" + }, { "name": "loop_function", "root": "LexLeanBoot.RustLoops.loopFunction", @@ -176,6 +184,14 @@ ], "version": "1.0.0" }, + { + "name": "quote_text", + "root": "LexLeanBoot.SurfaceLex.quoteText", + "targets": [ + "rust-std" + ], + "version": "1.0.0" + }, { "name": "reject_atoms", "root": "LexLeanBoot.PrimitiveScan.rejectAtoms", diff --git a/xtask/src/spec_links.rs b/xtask/src/spec_links.rs index 6eea6aecad..1957d248c8 100644 --- a/xtask/src/spec_links.rs +++ b/xtask/src/spec_links.rs @@ -61,8 +61,8 @@ fn parse_table(spec: &str) -> Result, Fail> { pub fn validate(root: &Path) -> Result<(), Fail> { let spec = std::fs::read_to_string(root.join("SPEC.md"))?; let rows = parse_table(&spec)?; - if rows.len() != 469 { - return Err(format!("RP-07: the §31 table has {} rows, not 469", rows.len()).into()); + if rows.len() != 470 { + return Err(format!("RP-07: the §31 table has {} rows, not 470", rows.len()).into()); } // The prose total under the table is part of the contract a reader // relies on; a row added without it would leave §31 contradicting itself.