Skip to content

BS-21: the structural grammar as language-1.2 source (first slice of #269) - #395

Merged
humuhumu33 merged 22 commits into
release/v0.4from
claude/issue-269-frontend-grammar
Nov 10, 2026
Merged

humuhumu33 merged 22 commits into
release/v0.4from
claude/issue-269-frontend-grammar

Conversation

@humuhumu33

@humuhumu33 humuhumu33 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

grammar::structural::parse_module is now parseModule of bootstrap/src/StructuralGrammar.lex.tex, one production root for rust-std, certified, with the Rust kept as the stage-0 oracle. It reads the atoms scanSource of BS-03 returns and imports PrimitiveScan for Atom and Rejection, so the host composes the two without a conversion.

Part of #269. The rest of grammar/ reads the lexicon's closure at every leaf, so it follows the lexicon and is not in this PR.

Design

The root admits allocation and overflow and no recursion. Where the oracle descends, it keeps an explicit stack of frames under iterate_until, so it owes no row in tests/stack-bounds.json and avoids the recursive shapes of #310 and #314. The first refusal stands in the state. A recursive part of the module is a chain inside its own type, as in BS-04; a step keeps ordinary lists and folds one into the chain when it pops a frame, so no definition recurses over a chain.

Evidence

Measured on the normative host at 1f470cda7, on release/v0.4 at 6d47f88fa:

Leg Runs Agreed
named vectors, evaluator 267, 146 refused, reaching all 30 listed refusals all
committed modules, evaluator 975, 917 accepted all
seeded edits, evaluator 2,000, 377 accepted all
vectors, modules and edits, pinned rustc 3,242 all
planted defects, each on its own named vector 44 all detected

Each plant names the vector that catches it, so growing the seeded corpus cannot cost a plant its witness, which is how BS-05 and BS-11 each lost one. Committed modules larger than the window are read by both ends of their atoms, because a prefix of a data-form module is a single refusal.

The largest certificate part is 2,879,430 bytes against the row's 3,145,728, and lexlean verify passes. Both were measured on a two-entrypoint copy of the project (PrimitiveScan, StructuralGrammar), not on the whole project.

The case failed first, at 6283ac8f3, before the module was an entrypoint.

Regenerated and ready

The member is merged with the tree of release/v0.4 at 4dc5b803c (after #415, SL-13), register 479, and the regeneration has run on the normative host (WSL Ubuntu, Rust 1.97.1, Lean 4.32.1) at verify width 1:

  • bootstrap/lexlean.lock, the production export with the bootstrap/parse_module package, the build oracles, and 482 verification records for bootstrap (UOR Atlas skipped on this host, as before).
  • conformance_bs_21 passed whole on that tree with the stack measured (3,006 s); the root does not recurse.
  • The one new package moves the counts of BS-05 to BS-08 and BS-13 to BS-15: 18 constants, their §31 rows, statements, scenarios and §17.23 prose, measured under LEXLEAN_COUNTS=fast with the writer of Bootstrap count cases: report every moved count in one pass, and write them #442 and written by it. BS-11, BS-12 and every mutant draw are unchanged. VERIFICATION.md carries the table.

Gates at the head

Gate Result
cargo fmt --check passed
validate-model passed, 479 IDs
validate-spec-links passed, 479 rows
clippy, workspace, all targets, -D warnings passed
RP cases passed, 17
nine bootstrap count parts, fast mode passed

The nine count parts run whole on CI; they are not run whole here.

🤖 Generated with Claude Code

Closes #412.

humuhumu33 and others added 6 commits November 2, 2026 17:56
… grammar

The row is the first slice of #269: `grammar::structural::parse_module` as
language-1.2 source. The case fails because the module is no entrypoint of the
project yet held by this commit; the source follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`parseModule` is a pushdown machine under `iterate_until`: errors stay in the
state, the first standing, so a definition reads as a pipeline, and accumulators
are lists folded into a chain when a frame is popped, so nothing recurses over a
chain and the root admits no `recursion`.

The case draws both ends of a committed module rather than a prefix: a prefix of
a data-form module is one refusal, and both ends are a module the oracle accepts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The register is 469 with SL-10 and this row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The rustc leg took 48 seconds on 1,742 runs, so it reads all the mutants and
whole modules of up to 4,096 atoms. The lock is this tree's only: the packages
and the records are the regeneration's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SPEC section 31 row had no sections cell, so the table parser skipped it and
counted 468 rows. CONFORMANCE.md follows the statement with its counts. The case
is as `cargo fmt` writes it, without the `drop` of a closure clippy refuses.
VERIFICATION and CHANGELOG record the measured run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RP-11 holds a range to the register's first and last ID of its prefix, so
`BS-01`..`BS-15` with `BS-21` beside it fails once BS-21 is the last BS row.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

afflom commented Nov 3, 2026

Copy link
Copy Markdown
Member

Read-only readiness audit of final head ac2db12218002cbb5bedc51d9096a4beea14800c: the BS-21 structural-grammar slice is now tracked by #412, which this PR closes when complete; #269 remains open for the rest of the front end. The existing PR description and its disclosed missing regeneration are preserved.

Current-head honesty and reproducibility workflows pass. CI 41542854707, rest job 125611339368, fails golden with R10: bootstrap: missing oracle expected/build/coverage/LexLeanBoot/StructuralGrammar.coverage.json. This draft run executes only the rest leg; its 283 passing conformance cases do not establish the new BS-21 differential/export leg or the complete host/Atlas gate.

Before landing, the coordinated regeneration owner must integrate this slice with the agreed wave, use the official writers for the bootstrap lock, build/verification records and bootstrap/parse_module package, recount the affected BS-05/BS-06 corpora and merged register/count documents, then run BS-21 through its actual exported-program comparison and certificate budget check. Compile/Clippy and the complete acceptance gate must pass on that final head, including the whole bootstrap project's axiom/certificate checks. The earlier two-entrypoint measurement is useful partial evidence and does not replace these final-tree checks. No duplicate standalone regeneration is requested; keep the draft until the coordinated work supplies that evidence.

humuhumu33 and others added 8 commits November 9, 2026 00:03
BS-21 stacks on SL-13 (#415), whose stdlib digest every bootstrap export
records, so its regeneration runs once on the tree both will land in.
The conflicts are taken from that tree with this member's own edits put
back on them: the StructuralGrammar prose of SPEC.md §17.23 and the BS-21
row after BS-20, the register at 479 IDs (SPEC.md, the README, the RP-07
check and its case), the README's bootstrap row through BS-21, the
StructuralGrammar entrypoint of bootstrap/lexlean.toml and its note in
bootstrap/README.md, the case's wiring and named test, the scenario, the
CHANGELOG entry, and the BS-21 subsection of VERIFICATION.md. The lock
and CONFORMANCE.md are the tree's and are rewritten next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cargo xtask check-production-export --write` at verify width 1 on BS-21
merged with SL-13 (release/v0.4 at 4dc5b80): the bootstrap export gains
the `parse_grammar` packages of the structural grammar and every package
records the standard library's new digest; the lock follows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`cargo xtask check-golden --write` on the regenerated export.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`LEXLEAN_EXAMPLES_PART=without:uor-atlas cargo xtask verify-examples
--write` at width 1: 482 normalized records for bootstrap (attestation
e7c027e2422e88511a077ff190af086ce79c7bbfcf7c38a90a04a02aed2d322e); the
UOR Atlas example is skipped on this host, as before.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Written from the model at the regeneration; the merge had taken the
tree's copy, which predates the row.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d BS-13 to BS-15

The committed export gains one package, so the committed manifests, packages,
programs and crates each grow by one and the corpora the cases draw from them
follow: 18 constants in seven cases, their §31 rows, register statements and
scenarios, and the §17.23 prose that repeats them. Measured under
LEXLEAN_COUNTS=fast with the count writer of #442 applied and then removed
again; the writer placed every number. BS-11, BS-12 and every mutant draw are
unchanged. VERIFICATION.md carries the landing's table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The merge kept the standard-library row from before #415 claimed its
row; RP-11 reads the row from release/v0.4 again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@humuhumu33
humuhumu33 marked this pull request as ready for review November 9, 2026 15:56
@afflom afflom mentioned this pull request Nov 9, 2026
The one conflict is the README's bootstrap row, where this member's row,
which claims BS-21 and the range through BS-21, is kept; the standard
library row is the line's, as BS-21 already had it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@afflom afflom left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head 8b9715b against release/v0.4 4dc5b80. Its tree equals prior 933ab; the new ancestry incorporates the actual release. The bootstrap products previously missing at ac2db are now present, so that old missing-product finding is superseded. Scoped source/R3/eligibility and product-closure review found two remaining count discrepancies:

  1. P2 — BS-15 combined total: model/ids.toml:2775 still says “all 3,190 runs”, mirrored in SPEC §31, the bootstrap scenario and generated CONFORMANCE. At this head, rust_package.rs:838–843 constructs 319 + 182 + 2,000 = 2,501 header runs; lines 869–911 construct 300 + 137 + 300 = 737 render runs. Lines 965–973 send these to distinct roots, and rust_validate.rs:2635–2761 executes every run and sums their outcomes. The combined total is 3,238; truncating oversized library arguments does not remove runs.

  2. P2 — BS-07 prose total: SPEC.md:8420 says “1,392 vectors (351 per root …)”. rust_lower.rs:919–961 asserts 351 entries and passes all of them to four roots through the cardinality-preserving runs mapping (lines 570–575). The every-host total is 1,404. The separate 6,000 normative seeded runs are unaffected.

Please reconcile these current normative counts on this existing branch, using the established model writer for generated projections. These findings are independently derived from the actual cases; they do not presume the count tool caused the retained numerals. Preserve historical measurements. A count-prose correction alone does not call for repeating completed lock/export/verification producers.

This COMMENT is not full acceptance: current-head full CI42677680086 is still running, with reproducibility passed; the separately pending manual42677419330 must not be aggregated into it. Final corrected-head acceptance remains required. #412 may close after accepted delivery through this PR; broader #269 remains open.

#394 changed bootstrap/src/TokenAudit.lex.tex and regenerated the bootstrap
export and records, which every package of the project depends on. The
conflicts are the generated files, taken from the line here and regenerated
on the merged tree next, and VERIFICATION.md, where both added sections are
kept, the line's first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

afflom commented Nov 10, 2026

Copy link
Copy Markdown
Member

Separate, non-blocking efficiency observation for coordination with the existing BS15/count work: a read-only audit of accepted release cf76090d found that crates/conformance/src/cases/bootstrap/rust_package.rs constructs and deep-clones the evaluator-only within corpus at lines 879–887 on every host. Outside the normative branch, consumers use only its cardinality/log value; evaluation and planted-defect consumers use its contents inside the normative branch. Separating that allocation from the all-host count could avoid unused preparation.

This is scoped to the accepted source, not a proposed patch to this PR. Its cost is unmeasured and it does not explain the reported 10,304.13-second BS15 bottleneck. Header/render mutant corpora still feed native execution on every host, so they cannot be moved behind the guard as the earlier BS07 preparation was. Repeated check/build/linked-project calls also serve distinct obligations; no safe removal was established.

I have not started a patch, benchmark, test or gate, and this adds no acceptance requirement to #395 or #451. Please retain your source/count ownership and #442’s writer coordination if this small opportunity is taken up. The existing exact-head count findings remain separate.

humuhumu33 and others added 5 commits November 10, 2026 02:06
`cargo xtask check-production-export --write` at verify width 1 on BS-21
merged with release/v0.4 at 4318b17, whose TokenAudit change (#394)
re-issues every package of the bootstrap project; the lock follows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cargo xtask check-golden --write` on the regenerated export.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`LEXLEAN_EXAMPLES_PART=without:uor-atlas cargo xtask verify-examples
--write` at width 1; the UOR Atlas example is skipped on this host, as
before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No bootstrap source changes, so nothing regenerates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#450 maps every BDD part to one Intel lane; bootstrap-structural-grammar
joins the print lane with the other light bootstrap parts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@afflom afflom left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked the two existing count findings against current head f10cfc4 on accepted release cf76090. Both remain unresolved:

  • BS-15 constructs (319 + 182 + 2000) + (300 + 137 + 300) = 3238 runs; model/ids.toml, SPEC §31, the bootstrap scenario and CONFORMANCE still describe 3190.
  • BS-07 retains 351 vectors per native root, hence 1404 across four roots; SPEC still states 1392. The separate normative seeded corpus is unaffected.

Please retain the existing owner repair and official projection workflow described in review5831477654. #451 has its own larger totals and must not inherit these parent-specific numbers. This is a bounded exact-head source review, not final acceptance: current CI42741897776 is incomplete; repro42741897510 passed. Old8b971 acceptance and separately triggered runs are not acceptance for this head. No duplicate patch or producer was started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants