Repository navigation
BS-02: the lexer's scans run only in the branch that reads them (#373) - #394
Conversation
stepLexAt, stepLexToken, stepLexIdent and scanSegmentsStep bound every candidate scan with a strict let before branching, so block comment, string, raw string, line comment and quoted name scans ran at positions that never used them. Each scan now sits in the branch that reads it. The packages, records and PERFORMANCE.md row are regenerated in a later commit. Part of #373. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Maintenance review of 0a36f1e: guarding the scans preserves the branch order, errors and token results while removing the demonstrated eager rescanning. No additional functional blocker emerged in static review. The draft run 41530726883 reaches golden after passing the rest BDD partition (283 cases), then fails on For readiness, update to the landing release/v0.4 head, finish BS-02 and the bootstrap verify/export comparison, update the |
…-lazy-lexer-scans
The scans in TokenAudit.lex.tex moved into the branches that read them, so the rendered lex and audit_tokens packages, bootstrap's build oracles and coverage maps, and its normalized verification records change with them. Every rewrite is under bootstrap/ and tests/production-export/bootstrap/; no language identity moves and bootstrap/lexlean.lock is unchanged. Regenerated on WSL2 with check-golden --write, check-production-export --write and verify-examples --write (all projects but uor-atlas), then check-golden and check-production-export pass without --write. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bootstrap_perf rows for lex and audit_tokens at 343,104 B, measured through the harness at d34c1fe with LEXLEAN_PERF_MAX=400000: lex takes 28.8 ms against 680,088 ms before and grows at 1.01 against 2.04; audit_tokens takes 25.8 ms and grows at 0.94. The sections that called the lexer quadratic, the stage 1 projection and the "faster first" list now record it as fixed. Closes #373 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
afflom
left a comment
There was a problem hiding this comment.
Fresh review at d80997c supersedes the earlier d34 status findings. The new commit changes only bootstrap/PERFORMANCE.md: it adds the d34 harness rows and updates the former quadratic-status prose. TokenAudit's implementation remains byte-identical to the previously reviewed0a36 source; the generated-product refresh is already in d34, and the branch retains actual Host c613 ancestry. No new implementation or weakened plant is introduced by this documentation follow-up.
The new rows report lex28.8ms/growth1.01 and audit_tokens25.8ms/growth0.94 at343104bytes, with comparison/host/measurement caveats retained. The body now reports the full BS02 including export comparison passed1839s, plus the official writers and nonwrite checks. Thus the previously missing PERFORMANCE/full-BS02-report items are addressed; these local outcomes are author-provided evidence, not independently reproduced executions. Please retain exact command/host/source/terminal logs for the final evidence review; no repeated measurement or completed writer is requested.
The current complete changed-head CI, same-run honesty and reproducibility remain mandatory. Leave those runs intact; no merge or #373 closure follows from the reported local pass. Coordinate the eventual source/product integration with the concurrent #393 carrier on release/v0.4 so both accepted deltas and actual merge ancestry survive. A later combined tree requires its own complete final acceptance; prior-tree green jobs cannot accept it.
|
While the current final-head gates run, the minimal source integration after #393 has been prepared and independently reviewed without executing producers. Provisional external treee3403347 adopts only this head's exact TokenAudit and PERFORMANCE blobs onto unlanded050e; all other10,637 entries remain exact. The two authored paths do not intersect the905 embedded identity inputs. The existing generated bootstrap products are explicitly stale in that provisional tree; it is not a publishable or accepted merge tree. This avoids an unjustified second identity/model/blanket-lock/fixture wave. The reviewed plan reuses source-compatible frozen executables only after actual landing/head/root/input checks. Existing official tools have no bootstrap-only export/golden/whole-verification selector: the necessary later producers are one all-five export, one golden writer, and the supported22-project non-Atlas verification writer, with actual unrelated outputs required to remain exact. Post-golden calculus work is conditional on a real change in its compiler-manifest dependency, not an automatic repeated phase. No new selector, hand-normalized oracle, repeated performance run or extra broad local gate is proposed. No writer has started or been admitted by this source preparation. Leave both current hosted runs intact. Actual #393 acceptance/landing, current ownership/source refresh, resource/provenance review and complete final combined-tree CI remain prerequisites to the later restack through this existing PR. Its original contributor ancestry and d34/WSL performance provenance must survive; prior-tree passes do not accept the union. |
|
BS02 now has independent exact-head hosted evidence at d80997c, run41923680898, synthetic13a0e7ba/tree4e4a8427. Job126904369839 recorded This is hosted functional/export acceptance for the named case on this head, separate from the author's WSL performance measurements and any future combined393 tree. At23:34 the overall run still had seven pending jobs and no failures; honesty/reproducibility passed. No measurement, writer or local BS02 run was repeated. Full current-tree acceptance and later integration/actual-tree review remain mandatory before #373 closes. |
|
Release/v0.4 has now accepted #393 as55be5514cf4fe377c186059260d7666c7c9c2304, after all45 final jobs/honesty/repro passed and actual/tested/reviewed tree306551 equality was verified. I restored this PR to draft because its current d809 products and gate describe the prior c613 base. Its current run41923680898 remains untouched to collect the final package result; no rerun or cancellation is requested. I am taking the already reviewed minimal integration through this existing PR: retain d809's exact TokenAudit/PERFORMANCE source and measurement provenance, accepted #393 source/products as the baseline, and actual contributor/release ancestry. The source union has no905-input/identity change, so no speculative pin/model/blanket-lock/fixture wave or repeated performance/BS02 run is warranted. The independently reviewed first phase will use the established physical root and frozen binaries for one official all-five export after final live/resource/receipt checks. Later golden/normalized verification phases and final combined-head CI remain separately required. No duplicate implementation or PR is being started; #373 stays open. |
|
The post-#393 product refresh has completed its first two official phases once each, with independent actual-artifact reviews:
All recorded source/input/helper/index/905-input/error/frozen-executable guards passed. The compiler manifest and four shipped calculus module pairs stayed byte-identical, so no calculus writer or immediate comparison was needed or run. No identity/model/blanket-lock/fixture wave or repeated BS02/performance run was introduced. After separate exact driver review and fresh unchanged head/ownership/release checks, one supported 22-project non-Atlas verification writer started at 02:31 UTC using the same frozen executables and whole-phase lock. Its entry provenance passed; terminal normalized-record/axiom review is pending. This is not combined-tree acceptance. The old d809/c613 hosted run has now finished 44/44 successfully plus honesty/repro, but the eventual combined head still needs its complete final hosted gate and actual-tree review before #373 closes. This PR remains draft. |
Retain the contributor lexer changes and actual #393 landing. Refresh the bootstrap products with the official export, golden and verification writers. Record independently reviewed producer evidence; final combined CI is pending.
afflom
left a comment
There was a problem hiding this comment.
Final source/artifact/history review is complete on 281de5b, tree a194c5ce81eeaf4623524d8f3884f000e406444f, with ordered actual parents d80997c + accepted release55be5514. Independent tree review20acdda0 binds all10,639 captured files, actual completed export/golden/verification inventories and the additive honest VI record; actual-commit reviewb4f6e09f confirms contributor and release ancestry. Normal fast-forward push retained the existing PR.
The official producers each ran once and passed independent actual artifact reviews: all-five export3015.36s, golden59.07s, supported22-project verification3879.17s. Verification changed only bootstrap records; all278 bootstrap process records exited0, source axiom audit stdout stayed exact, and genuine lex/audit_tokens program/certificate changes match the reviewed source. Compiler/calculus dependencies did not change, so no redundant calculus, extra comparison, BS02 run or performance measurement was added.
This establishes source/product readiness for the complete new-head hosted gate, not merge acceptance. Prior d809/c613 green44-job CI and WSL/d34 measurements retain their original scopes. Full changed-head CI, same-run honesty, reproducibility, fresh final-head/base review and actual merged-tree equality remain mandatory before #373 closes. No other issue closure is claimed.
|
Live refresh on 2026-11-09: the Nov5 surviving run41964177567 finished successfully only for its captured draft event. It ran plan, honesty and the rest partition; seven required host/Atlas job groups were skipped. The original ready run41964083701 was cancelled by native concurrency. Repro41964177626 passed, but these results do not establish full combined-head acceptance. Release/v0.4 has since advanced through #363, #414 and #415 to4dc5b803c5b0771f9f19563373515d0a2c2f313a. This PR remains281de5b5 and now conflicts with the live release. I restored draft status and assigned a bounded source/product reconciliation review against that actual base. The completed Nov5 export, golden and22-project verification phases and their independent reviews remain valid for their recorded tree; none is being repeated by this review. Existing contributor history and d34/WSL measurement provenance remain required. No obsolete gate rerun, source push or identity writer is started. The current #444 scheduling gate remains untouched. The next publishable combination needs reviewed conflict resolution, only demonstrably necessary official product work, and complete final-head CI before merge and #373 closure. |
|
The current-base reconciliation is now independently clear for static product reuse. Against actual release4dc, the external candidate retains all823 release-only changes and738 PR-only changes, including the exact64 export and668 normalized verification changes already produced and reviewed on281. Its22 selected module sources/import edges are identical to281; the new Json/Map/JsonRead modules are outside that closure. All905 compiler inputs and the compiler implementation remain exact. This supports retaining those products without repeating the export, verification, BS02 or performance phases. The unresolved producer work is23 bootstrap golden files: the manifest and22 source maps. The current lock changes their source/build envelopes while preserving the selected semantics. Neither side's envelope can simply be copied as the final answer; the existing official golden writer must produce the combined result, with its entire write footprint guarded. The candidate also preserves the complete current release verification history plus this PR's exact prior appendix. This is a source/dependency review, not a final tree or new runtime result. The candidate still has23 unresolved entries. A concrete guarded golden-only procedure is being prepared for separate review; no producer is running, and the active #444 gate is untouched. Final actual history and complete combined-head hosted acceptance remain mandatory. |
|
The necessary current-release golden reconciliation has now completed once against actual release4dc5b803 and this PR's281de5b5 source/product union. The official frozen The complete before/after inventory confirms exactly23 changes:22 bootstrap source maps changed only their source_id, and the manifest changed only the source/build IDs, current lock hash and those22 map hashes. The selected semantic_id remains0bb16d7a. All current release-only changes and the previously reviewed64 export,668 normalized verification and4 other golden deltas remain retained. The905 compiler inputs, errors, source/helper/frozen-executable guards and physical HEAD/index were preserved. No export,22-project verification, BS02, performance, identity/model/lock/fixture or calculus phase was repeated. Independent review1b9bd717 binds terminal receiptac776f08 and the complete footprint. Final source/product capture, honest verification-history append and actual281+4dc ancestry are now being prepared for separate review; this PR remains draft. Full final-head hosted acceptance and actual merged-tree equality remain required before #373 closes. #444's cancelled final host job is not accepted or incorporated. |
Preserve actual281 contributor/history and actual4dc release ancestry. Reconcile only demonstrated golden source/build envelopes and retain reviewed export/verification products with additive scoped evidence.
afflom
left a comment
There was a problem hiding this comment.
Final source, actual products and ordered history are independently reviewed on1062e3951990a8ce9f9885085834fd898eef6c4c, tree897999743ac4348711b86058832a1666dc88f572. Live GitHub commit headers match the reviewed actual parents281de5b5 first and4dc5b803 second. Normal fast-forward publication preserves the existing contributor branch and all intervening accepted release history.
Independent final reviewb4b8ede6 binds10,900 captured files/eight symlinks, all823 current-release-only entries, the exact retained64 export/668 normalized verification/four other golden deltas, and the two contributor-authored files. The only new producer was one official golden write, exit0 in77.579seconds, independently reviewed in1b9bd717:22 map source_id envelopes and the manifest's admitted IDs/lock/map hashes. The additive verification appendix preserves the full current4dc record and exact earlier281 appendix with honest historical measurement/proof scopes.
No export,22-project verification, BS02/performance measurement, identity/model/lock/fixture or calculus phase was repeated. #444's cancelled gate/source is not accepted or incorporated. This is source/product readiness for the complete new-head hosted gate, not merge acceptance. All required jobs, same-run honesty and reproducibility must pass on this final head, followed by fresh head/base/review and actual merged-tree equality. Only #373 is within closure scope; it remains open.
Preserve the exact #394 source and generated products together with #450's Atlas scheduling, BS-07 host preparation and Intel construction changes. Retain both contributor histories and historical verification scopes. No producer regeneration is needed: the only shared path is the additive VERIFICATION record, and the CI changes do not alter compiler or project identity inputs. Complete combined-head hosted acceptance remains required.
|
The intended release union is now published through existing #450 at fddf2e8, retaining this exact head 1062e39 as its second parent. All lexer source, PERFORMANCE caveats and generated products remain byte-for-byte intact. The only overlap is the additive VERIFICATION appendix; independent full-inventory and actual-commit review passed. #450 is now ready for one complete gate on the intended combined tree, including the Intel scheduling changes. This PR's current run 42688365239 is left intact, with no rerun or cancellation. Keep this history and #373 open until complete combined proof/BS02 acceptance, final-head review and actual landing-tree equality are verified. No source producer or performance phase was repeated. |
|
Maintainer coordination: acknowledged the combined #450 landing route in #394 (comment). Live #450 is ready at fddf2e8; its ordered parents retain a06facb and this exact 1062e39 head. The GitHub test merge bf0ed1a has the same 0c24b969 tree as the published union. The existing #394 run remains untouched and is watched only as evidence for its original tree. I will not independently merge #394 or close #373 while this combined route is active. Existing #450 implementation and qualification ownership remains with its current owner; no competing patch, producer run, readiness change or CI rerun is initiated. Final combined-head review, complete same-run acceptance/reproducibility and actual landing-tree equality still govern delivery and closures. |
|
Merged as 4318b17 after exact-head run 42688365239 passed all 45 expected jobs, including BS02's compiler partition, every Atlas shard/join and all four supported hosts; reproducibility 42688244759 passed. Fresh head/base/review checks found no unresolved thread or changes-requested review. The actual landing has ordered parents 4dc5b80 then 1062e39, and tree 897999743ac4348711b86058832a1666dc88f572 exactly equals the independently reviewed source/products and tested synthetic merge. No force or protection bypass was used. The merge response timed out, so I verified the closed/merged PR, branch ref and actual commit rather than retrying it. #450 already contains exact 1062e39 and retains its own final-union qualification in progress. This landing does not substitute for that gate. Historical performance measurements retain their documented scopes; no new timing claim is made. |
#394 changed bootstrap/src/TokenAudit.lex.tex and regenerated the bootstrap export and records, which every package of the project depends on. The conflicts are the generated files, taken from the line here and regenerated on the merged tree next, and VERIFICATION.md, where both added sections are kept, the line's first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`cargo xtask check-golden --write` on the regenerated export. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`LEXLEAN_EXAMPLES_PART=without:uor-atlas cargo xtask verify-examples --write` at width 1; the UOR Atlas example is skipped on this host, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The conflicts are generated bootstrap files only, taken from the line and regenerated in the commits that follow. The three surface parts join the Intel print lane that #450 requires of every BDD part. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #373.
stepLexAt,stepLexToken,stepLexIdentandscanSegmentsStepinbootstrap/src/TokenAudit.lex.texbound every candidate scan with a strictletbefore branching. Block comment, string, raw string, line comment and quoted name scans therefore ran at positions that never used them, which madelexquadratic. Each scan now sits in the branch that reads it. The results are unchanged; only when the scans run moves.isBangedandisQuoteAfterstay as bindings because BS-02's interpolated-string plant needs that site verbatim.Measurements
On
TargetSemantics.lean, at100f42097plus this change:The committed package and the one rendered from this source were built into one binary, so the times compare directly, with three caveats:
bootstrap/PERFORMANCE.md, a different build from the rest of the table. In this binary the exponent from 32,645 B to 343,104 B is 2.42 before and 1.02 after. The 0.95 is a fit over eight sizes from 929 B to 343,104 B.bootstrap_perfharness than in one binary, so these milliseconds are not the row's numbers.Through the
bootstrap_perfharness, atd34c1feb3withLEXLEAN_PERF_MAX=400000, the rows now inbootstrap/PERFORMANCE.md:lexaudit_tokensThe sections of
PERFORMANCE.mdthat called the lexer quadratic, the stage 1 projection and the "faster first" list now record it as fixed.Original contributor regeneration (prior c613 base)
On the branch merged with
release/v0.4atc613290b8(1170e50fe), in WSL2.TokenAudit.lex.texis the only source change, so no language identity moves andbootstrap/lexlean.lockis unchanged.bootstrap:fmt --check(18 modules canonical) andlock --check(current).cargo xtask check-golden --write,check-production-export --write, andverify-examples --writeover every project butuor-atlas. All 576 rewritten files are underbootstrap/andtests/production-export/bootstrap/; nothing else changed.--write:check-goldenandcheck-production-exportpass.conformance_bs_02passes in full, including its export comparison (1 passed, 1,839 s).just modelandjust spec-linkspass.The original d809/c613 hosted gate subsequently passed all44 jobs, same-run honesty and reproducibility. That prior-tree result does not accept the combined tree after actual #393 landing.
🤖 Generated with Claude Code
Integration after actual #393 landing
This existing branch retains the contributor d809/d34/1170/0a36 history and actual release/v0.4 landing55be5514cf4fe377c186059260d7666c7c9c2304. Exact TokenAudit and PERFORMANCE authored bytes are retained on accepted #393 source/products. Language identities, selected905 compiled inputs, errors, pins, locks, negative inputs, Atlas and unrelated source/products remain unchanged.
The original frozen source-compatible CLI/xtask performed one official all-five production export (3015.363s), one golden write (59.073s), and one supported
verify-examples --writewithwithout:uor-atlas(3879.170s, all22 projects). Independent actual artifact/input reviews are ca6b56f7,8f1e24a6,3db13148. Only official bootstrap export/golden/verification outputs changed. Compiler-manifest/shipped calculus dependencies remained exact, so no calculus rewrite or immediate comparison was run.Verification changed668 bootstrap paths (329 additions,329 removals,10 modifications). Bootstrap has463 normalized records and278 process records with exit_code0;278 is not the total over22 projects. Source axiom audits remain exact [Quot.sound,propext]; pre-existing preservation-library Classical.choice remains unchanged. Genuine lex/audit_tokens program/certificate changes were independently classified; source-ID renaming alone is not a semantic-equivalence argument.
No BS02 or performance measurement was repeated. The prior exact-d809 hosted BS02 named pass and complete compiler partition are retained as prior-tree functional evidence. The d34 harness and original author WSL/host/comparison caveats above remain the measurement scope; no combined-tree timing is claimed.
Final source/artifact/history tree
a194c5ce81eeaf4623524d8f3884f000e406444fand the additive VI appendix passed independent review20acdda0. Actual head281de5b5d41adda58e152ab49227c99f0f1fa24aretains ordered parents d809 then actual55be, independently reviewed inb4f6e09f. One complete changed-head hosted gate, same-run honesty, reproducibility and eventual actual landing/tree equality remain pending. This PR closes only #373 after those requirements are satisfied.Current release reconciliation (2026-11-09)
Published commit
1062e3951990a8ce9f9885085834fd898eef6c4c, tree897999743ac4348711b86058832a1666dc88f572, integrates actual PR head281de5b5with actual release4dc5b803, preserving both complete histories and all current base changes. The lexer source and PERFORMANCE bytes remain exactly the reviewed contributor bytes. #444 was cancelled and not merged; its source is not part of this candidate.Independent static closure review 195005d1 supports retaining the 64 existing bootstrap export and 668 normalized verification deltas without repeating their completed Nov 5 producers. Their prior durations and proof/process records above retain their actual 55be/281 scope. The only new producer was one official current base golden write: exit0 in 77.579 s, with independent actual-artifact review 1b9bd717. Exactly 22 maps and one manifest reconciled source/build envelopes; selected semantics and every other guarded product stayed unchanged.
An additive verification record preserves the complete current 4dc history and exact prior 281 appendix. Final source/artifact/history review passed independently in b4b8ede6 and root COMMENT review5831839979; live published commit/tree/ordered parents match exactly. No new BS02 or timing measurement was run; original d34/WSL caveats and PERFORMANCE evidence remain unchanged. Complete new-head CI, same-run honesty, reproducibility, combined proof/BS02 acceptance and actual landing/tree equality remain required. Historical draft-only skipped jobs are not acceptance. This PR closes only #373 after those requirements pass.