From c9502c5c8f768ec10a19a9ea2e3e5ce367b2ebfb Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 06:20:15 +0200 Subject: [PATCH 01/26] chore: prepare release 0.5.1 --- CHANGELOG.md | 12 ++++++++++++ peglib-core/pom.xml | 2 +- peglib-formatter/pom.xml | 2 +- peglib-incremental/pom.xml | 2 +- peglib-maven-plugin/pom.xml | 2 +- peglib-playground/pom.xml | 2 +- pom.xml | 2 +- 7 files changed, 18 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f4ffb45..ea1f547 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.5.1] - 2026-05-09 + +_Unreleased — patch cycle following 0.5.0. Use this section to log fixes / minor perf tweaks / docs updates accumulated post-release._ + +### Added + +### Changed + +### Fixed + +### Removed + ## [0.5.0] - 2026-05-06 _Work in progress — incremental-native architectural rework. See `docs/incremental/ARCHITECTURE-0.5.0.md`._ diff --git a/peglib-core/pom.xml b/peglib-core/pom.xml index 61c47a9..2eb24f0 100644 --- a/peglib-core/pom.xml +++ b/peglib-core/pom.xml @@ -7,7 +7,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 ../pom.xml diff --git a/peglib-formatter/pom.xml b/peglib-formatter/pom.xml index b65d86a..6cfce8b 100644 --- a/peglib-formatter/pom.xml +++ b/peglib-formatter/pom.xml @@ -7,7 +7,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 ../pom.xml diff --git a/peglib-incremental/pom.xml b/peglib-incremental/pom.xml index 543a424..f7e95e0 100644 --- a/peglib-incremental/pom.xml +++ b/peglib-incremental/pom.xml @@ -7,7 +7,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 ../pom.xml diff --git a/peglib-maven-plugin/pom.xml b/peglib-maven-plugin/pom.xml index a296caf..55d4b96 100644 --- a/peglib-maven-plugin/pom.xml +++ b/peglib-maven-plugin/pom.xml @@ -7,7 +7,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 ../pom.xml diff --git a/peglib-playground/pom.xml b/peglib-playground/pom.xml index f62a340..f69c920 100644 --- a/peglib-playground/pom.xml +++ b/peglib-playground/pom.xml @@ -7,7 +7,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 ../pom.xml diff --git a/pom.xml b/pom.xml index e4a3c20..5a4a993 100644 --- a/pom.xml +++ b/pom.xml @@ -6,7 +6,7 @@ org.pragmatica-lite peglib-parent - 0.5.0 + 0.5.1 pom Peglib Parent From d2cc6be53f9b5f8e7793c0afa2240b12fd3ead43 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 06:44:06 +0200 Subject: [PATCH 02/26] test: trivia adversarial corpus + findings (Step 2 of context-independent attribution investigation) --- .../TRIVIA-ADVERSARIAL-FINDINGS.md | 95 +++ .../peg/perf/TriviaAdversarialTest.java | 739 ++++++++++++++++++ 2 files changed, 834 insertions(+) create mode 100644 docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java diff --git a/docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md b/docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md new file mode 100644 index 0000000..8d44a1b --- /dev/null +++ b/docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md @@ -0,0 +1,95 @@ +# Trivia Adversarial Corpus — Step 2 Findings + +Step 2 of the trivia attribution context-dependency investigation. +Tests at `peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java`. +17 tests added — 16 enabled passing, 1 `@Disabled`. Production code untouched. + +## Bottom line + +| Severity | Count | Items | +| ---------------- | ----- | ---------------------------------------------------- | +| Definite bug | 0 | | +| Latent bug | 2 | Bug 1 (gen size-only restore), Bug 2 (Optional cut) | +| Robust-by-luck | 2 | Cache hit parity, Choice backtrack same-position | +| Robust-by-design | 3 | Bug C' nested, Predicate symmetry, %whitespace pure | + +**Did Target #1 surface as a definite bug?** No. Two probes +(`generator_choiceBacktrackAfterDrain_parityProbe`, +`generator_choiceBacktrackDifferentStructure_parityProbe`) confirm +interpreter and generator produce byte-identical trivia text. The +size-only restore IS structurally weaker than `List.copyOf` snapshot +(`ParserGenerator.java:6768-6776` vs `ParsingContext.java:526-528`), but +the second alternative re-runs `skipWhitespace` at the rolled-back +position and rebuilds pending equivalently. Bug 1 reclassified as +**robust-by-luck**. + +## Per target + +### #1 Generator size-only restore (3 tests passing) +- Interpreter and generator agree on backtracking-choice trivia. +- Code-level asymmetry is real; black-box impact is masked. + +### #2 Bug C' nested boundary (4 tests passing) +- Empty-ZoM tail with multi-sibling prefix — orphan WS attached. +- Doubly-nested empty ZoM, comment+empty-ZoM, empty Optional tail. +- All round-trip. Robust-by-design. Pinned as regression net. + +### #3 parseRuleAt context-loss (3 tests passing) +- Documents narrowing of wrapper.leadingTrivia from fresh-context entry. +- Item span text from parseRuleAt equals full-parse Item span text. +- Behaviour as documented; tests pin current contract. + +### #4 Cache hit/miss parity (2 tests passing) +- Two call sites, predicate-seeded cache hit — both produce correct + leading. Robust-by-luck (relies on Target #7). + +### #5 Predicate symmetry (2 tests passing) +- And/Not save+restore via `savePendingLeadingTrivia` (full snapshot). +- Robust-by-design. + +### #6 Optional CutFailure (1 test `@Disabled`) +- `parseOptionalWithMode` line 1936 returns CutFailure without restoring + `entryPendingSnapshot`. +- **Bug 2 latent.** No black-box surface — when cut fires the outer + parse fails and pending is unobservable through the returned CST. + Observability needs instrumented `ParsingContext` or upstream + recoverable cut-failure-from-Optional. +- Fix is trivial: restore snapshot at line 1937. Risk-free even masked. + +### #7 %whitespace purity (1 test passing) +- Sub-rule-using whitespace; cache hit reattach matches cache miss. +- Robust-by-design for the tested fixture. Bug-B correctness *requires* + whitespace purity. Step 3 should formalise as grammar-validation rule. + +### Fuzz (1 test passing) +- 100 random nested-pair inputs. parseRuleAt subtree span text equals + full-parse subtree span text for every Value/Pair node visited. +- No new mismatches beyond Target #3's documented context-loss. + +## Reproducers + +**Bug 1 (latent in generator):** +``` +Start <- Triple / Pair +Pair <- 'x' 'y' +Triple <- 'x' 'y' 'z' +%whitespace <- [ ]+ +``` +Input `"x y"`. Generator's `restorePendingLeading(int)` (line 6772) +truncates only — drained items not recovered. Masked because the second +alt re-runs skipWhitespace. + +**Bug 2 (Optional CutFailure):** `PegEngine.java:1936`. Add +`ctx.restorePendingLeadingTrivia(entryPendingSnapshot);` before +`return result;` on line 1937. + +## Suggested next direction + +1. Fix Bug 2 — one-line change at `PegEngine.java:1937`. +2. Step 3 design: treat the size-only-vs-list-snapshot asymmetry as a + semantic invariant relying on `skipWhitespace` re-skip after + rollback. Document this in `docs/TRIVIA-ATTRIBUTION.md` rather than + "fix" the generator. +3. Search for a grammar where Bug 1 manifests black-box (a Choice whose + second alt skips the WS-bearing region entirely). If none exists, + that confirms size-only path is sufficient. diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java new file mode 100644 index 0000000..f57be53 --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java @@ -0,0 +1,739 @@ +package org.pragmatica.peg.perf; + +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.pragmatica.lang.Result; +import org.pragmatica.peg.PegParser; +import org.pragmatica.peg.action.RuleId; +import org.pragmatica.peg.tree.CstNode; +import org.pragmatica.peg.tree.Trivia; + +import javax.tools.ToolProvider; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Random; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Step-2 adversarial corpus for the trivia-attribution context-dependency + * investigation. + * + *

Each {@code @Nested} class targets one of the seven divergence points + * catalogued in Step 1. Tests fall into three groups: + *

+ * + *

Production code is not modified by Step 2. These tests serve as both bug + * discovery and regression baseline. + */ +class TriviaAdversarialTest { + + /** + * Target #1 — generator size-only restore vs interpreter list-snapshot + * restore. Highest-priority target. The interpreter saves a {@code + * List.copyOf(pendingLeadingTrivia)} so a branch that drains pending then + * fails recovers the drained items. The generator emits {@code int + * savePendingLeading()} / {@code restorePendingLeading(int)} which only + * truncates back to the recorded size — drained items are not recovered. + */ + @Nested + class GeneratorSizeOnlyRestoreAsymmetry { + + /** + * Grammar: {@code Choice <- ('a' WS 'b' WS 'c' / 'a' WS 'b' WS 'd')} + * with {@code %whitespace <- [ ]+}. + * + *

Input {@code "a b c"} succeeds in the first alternative. + * + *

Critical scenario: input {@code "a b d"} — the first alternative + * starts; pending whitespace gets drained when {@code 'b'} consumes + * leading. Then {@code 'c'} fails, the choice rolls back. The + * interpreter restores the drained items via list-snapshot; the + * generator keeps them dropped. The test then re-enters via the second + * alternative which reconsumes whitespace from the input, so total + * trivia text is identical — but the Trivia object identities (and + * grouping) for the second alt differ on the two paths. + * + *

Currently runs against the interpreter only and pins the correct + * (post Bug-A fix) behaviour. The corresponding generator-parity test + * is below. + */ + @Test + void interpreter_choiceBacktrack_drainsAndRestoresPendingTrivia() { + var grammar = """ + Start <- Item + Item <- 'a' 'b' 'd' / 'a' 'b' 'c' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + + var result = parser.parseCst("a b c"); + assertThat(result.isSuccess()) + .as("backtracking choice should succeed via second alt") + .isTrue(); + // Walk the tree — every space chunk in the input must be + // accounted for in some leading/trailing trivia somewhere. + var totalTriviaText = collectAllTriviaText(result.unwrap()); + assertThat(totalTriviaText).isEqualTo(" "); + } + + /** + * Bug A reproducer (now fixed in interpreter, latent in generator). + * + *

Grammar exercises the case where a failed alternative drains + * pending trivia AFTER the snapshot point. With size-only restore the + * drained items are gone for good even though the size is restored. + * + *

Disabled because it surfaces the latent generator bug. To make + * it run as a verification: cross-compare interpreter output against + * generated parser output on the same input — the trivia-text totals + * differ. + * + *

See {@code docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md} + * §"Bug 1 — generator size-only restore" for the minimum reproducer. + */ + @Test + void generator_choiceBacktrackAfterDrain_parityProbe() throws Exception { + // Probe: black-box compare interpreter vs generated parser on a + // backtracking-choice grammar where the failed branch drains + // pending. PASSES — parity holds because the second alternative's + // skipWhitespace re-runs at the same input offset and rebuilds + // pending equivalently. The size-only restore IS structurally + // weaker than list-snapshot (Step-1 catalogue, target #1) but + // the weakness is not observable through CST text/trivia content + // for this construction. Test pinned as a regression baseline: + // if rewrite changes generator save/restore semantics, this must + // continue to hold. Bug 1 is therefore reclassified as + // robust-by-luck (latent) rather than definite/manifest. See + // TRIVIA-ADVERSARIAL-FINDINGS.md. + var grammar = """ + Start <- Item + Item <- 'a' 'b' 'd' / 'a' 'b' 'c' + %whitespace <- [ ]+ + """; + var input = "a b c"; + + var interp = PegParser.fromGrammar(grammar).unwrap() + .parseCst(input).unwrap(); + var interpText = collectAllTriviaText(interp); + + var src = PegParser.generateCstParser(grammar, "test.adv.szbug", "SzBugParser").unwrap(); + var cls = compileAndLoad(src, "test.adv.szbug.SzBugParser"); + var node = invokeParse(cls, input); + var genText = collectAllTriviaTextReflective(node); + + assertThat(genText).isEqualTo(interpText); + } + + /** + * Stronger adversarial probe: backtrack from a deeply-drained choice + * alternative into a second alternative that consumes the SAME input + * but via different rule structure (so the rule-wrapping differs and + * trivia attachment may diverge if pending state carries over). + * Pinned passing — confirms parity holds for this shape too. + */ + @Test + void generator_choiceBacktrackDifferentStructure_parityProbe() throws Exception { + // Triple first so the long form is tried; if input is "x y" we + // backtrack to Pair after Triple drains pending and then fails + // on the missing 'z'. + var grammar = """ + Start <- Triple / Pair + Pair <- 'x' 'y' + Triple <- 'x' 'y' 'z' + %whitespace <- [ ]+ + """; + var input = "x y"; + + var interp = PegParser.fromGrammar(grammar).unwrap() + .parseCst(input).unwrap(); + var interpText = collectAllTriviaText(interp); + + var src = PegParser.generateCstParser(grammar, "test.adv.szbug2", "SzBug2Parser").unwrap(); + var cls = compileAndLoad(src, "test.adv.szbug2.SzBug2Parser"); + var node = invokeParse(cls, input); + var genText = collectAllTriviaTextReflective(node); + + assertThat(genText).isEqualTo(interpText); + } + } + + /** + * Target #2 — Bug C' (rule-exit trailing attribution) boundary cases. + * Bug C' attaches orphan trivia consumed before a zero-width tail element + * (empty ZoM/Optional) to the last child's trailingTrivia. Current corpus + * exercises the simple enum-trailing-comma case; here we hit nested ZoM + * and trivia-at-end-of-rule-body. + */ + @Nested + class BugCprimeNestedBoundary { + + /** + * Grammar: rule body ends with a zero-or-more whose body is itself + * empty for this input, after multiple sibling literals. Input has + * trailing whitespace before EOF that must attach to last child's + * trailing — not vanish. + */ + @Test + void emptyZomTail_attributesOrphanWsToLastChildTrailing() { + var grammar = """ + Item <- 'a' 'b' 'c' Tail* + Tail <- 'x' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + + // " a b c " — trailing 2 spaces with empty Tail*. + var result = parser.parseCst(" a b c "); + assertThat(result.isSuccess()).isTrue(); + var node = result.unwrap(); + + // Trivia text must round-trip: 2 leading + 1+1 inter-word + 2 trailing = 6 spaces. + assertThat(collectAllTriviaText(node)).isEqualTo(" "); + } + + /** + * Doubly-nested empty ZoM at rule end. Inner Tail* and outer Wrapper* + * are both empty. Whitespace consumed before the first Tail iteration + * attempt must surface somewhere in the tree. + */ + @Test + void doublyNestedEmptyZom_noWhitespaceLost() { + var grammar = """ + Outer <- 'a' Wrapper* + Wrapper <- 'b' Tail* + Tail <- 'c' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var input = "a "; // 3 trailing spaces, both ZoMs empty + var result = parser.parseCst(input); + assertThat(result.isSuccess()).isTrue(); + assertThat(collectAllTriviaText(result.unwrap())).isEqualTo(" "); + } + + /** + * Comments mixed with whitespace at empty-ZoM site. Verifies that + * trivia type classification (LineComment/BlockComment/Whitespace) + * survives the orphan-attach path. + */ + @Test + void commentBeforeEmptyZom_survivesAsTrivia() { + var grammar = """ + Item <- 'a' Tail* + Tail <- 'x' + %whitespace <- ([ ]+ / '//' [^\\n]* '\\n')+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var input = "a // comment\n"; + var result = parser.parseCst(input); + assertThat(result.isSuccess()).isTrue(); + // The comment must appear somewhere in trivia. + var totalText = collectAllTriviaText(result.unwrap()); + assertThat(totalText).contains("// comment"); + } + + /** + * Optional tail (rather than ZoM) at rule end with whitespace before. + * Optional that didn't match should still give the consumed + * whitespace a home. + */ + @Test + void emptyOptionalTail_attributesOrphanWs() { + var grammar = """ + Item <- 'a' Tail? + Tail <- 'x' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var input = "a "; + var result = parser.parseCst(input); + assertThat(result.isSuccess()).isTrue(); + assertThat(collectAllTriviaText(result.unwrap())).isEqualTo(" "); + } + } + + /** + * Target #3 — parseRuleAt context-loss. + * parseRuleAt initializes a fresh ParsingContext with empty pending. So + * the wrapper's leadingTrivia is whatever skipWhitespace produces from + * the offset onwards — narrower than what the same rule sees during a + * full parse, where carriedLeading from prior siblings flows in. + */ + @Nested + class ParseRuleAtContextLoss { + + record Item() implements RuleId {} + record Outer() implements RuleId {} + + /** + * Full parse of {@code " 42 "} attributes the leading 2 spaces to + * the {@code Item} wrapper. parseRuleAt(Item, " 42 ", 0) does too — + * but parseRuleAt(Item, " 42 ", 2) sees offset 2 (already past the + * leading spaces) and returns Item with empty leadingTrivia. The + * test pins this as the documented behaviour. + */ + @Test + void parseRuleAt_atOffsetPastLeading_returnsEmptyLeading() { + var grammar = """ + Item <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + + var partial = parser.parseRuleAt(Item.class, " 42 ", 2); + assertThat(partial.isSuccess()).isTrue(); + var node = partial.unwrap().node(); + // Item invoked at offset 2 — leading is empty (no whitespace at + // positions 2..2, "42" starts here). Documented context-loss. + assertThat(node.leadingTrivia()).isEmpty(); + } + + /** + * Now the more subtle case: parseRuleAt at offset 0 should reproduce + * what a full parse sees for the same rule — both contexts start with + * empty pending and skipWhitespace runs from position 0. + */ + @Test + void parseRuleAt_atOffsetZero_matchesFullParseLeading() { + var grammar = """ + Item <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + + var fullCst = parser.parseCst(" 42 ").unwrap(); + var partial = parser.parseRuleAt(Item.class, " 42 ", 0).unwrap(); + + // Top-level Item wrapper from full parse: + var fullLeadingText = trailingTextOf(fullCst.leadingTrivia()); + var partialLeadingText = trailingTextOf(partial.node().leadingTrivia()); + assertThat(partialLeadingText).isEqualTo(fullLeadingText); + } + + /** + * Surfaces the context-loss for a NESTED rule. {@code Item} appears + * inside a parent context with leading whitespace carried from the + * parent. parseRuleAt invoked at the same offset as the full parse's + * Item span sees fresh pending, so its leading is whatever + * skipWhitespace produces locally. + * + *

This test exercises the documented context-loss: the Item's + * leading from a full parse may differ from the leading from + * parseRuleAt at the same offset. + */ + @Test + void parseRuleAt_nestedRule_pinsContextLossDifference() { + var grammar = """ + Outer <- 'X' Item + Item <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + + // Find any Item node in the full parse. + var full = parser.parseCst("X 42").unwrap(); + CstNode fullItem = findRule(full, "Item"); + assertThat(fullItem).as("full-parse Item must exist").isNotNull(); + + int itemStart = fullItem.span().start().offset(); + var partial = parser.parseRuleAt(Item.class, "X 42", itemStart).unwrap(); + + // parseRuleAt enters Item with empty pending — its wrapper sees + // local skipWhitespace from itemStart only. If itemStart already + // sits at the digits, leading is empty; if itemStart points at + // whitespace, leading captures it. Either way, the test + // documents what happens — current Step-1 catalogue says the + // wrapper.leadingTrivia is narrower than the full parse here + // because the full parse threaded carriedLeading through Outer. + // We assert structural property: Item subtree text matches. + String fullText = "X 42".substring( + fullItem.span().start().offset(), + fullItem.span().end().offset() + ); + String partialText = "X 42".substring( + partial.node().span().start().offset(), + partial.node().span().end().offset() + ); + assertThat(partialText).isEqualTo(fullText); + } + } + + /** + * Target #4 — cache hit vs miss path divergence. + * Bug B fix: cache-hit drains pending + reskips ws + reattaches. Test + * that the same rule called from two different sites with different + * carried-leading produces leadingTrivia consistent with each call's + * context. + */ + @Nested + class PackratCacheHitMissParity { + + /** + * Grammar invokes Number from two sites with different prior-sibling + * trivia. The packrat cache may seed on the first call (e.g. via a + * predicate's lookahead) and serve the second call from cache. Both + * resulting Number nodes should have leading trivia consistent with + * their own call site. + */ + @Test + void sameRuleTwoCallSites_eachLeadingMatchesItsContext() { + var grammar = """ + Pair <- A B + A <- &Number Number + B <- ',' Number + Number <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + // " 42 , 7" — Number is called at A (lookahead first, then real) + // and at B (after comma). + var input = " 42 , 7"; + var result = parser.parseCst(input); + assertThat(result.isSuccess()).isTrue(); + + // Total trivia text must round-trip the input's whitespace. + // Input has 4 spaces total (2 leading + 1 around comma left + 1 right). + var triviaText = collectAllTriviaText(result.unwrap()); + assertThat(triviaText.replace(" ", "").length()).isZero(); + assertThat(triviaText.length()).isEqualTo(4); + } + + /** + * Predicate-driven cache seeding then real call. The {@code &Number} + * lookahead seeds the packrat cache for Number at the post-leading + * position. The subsequent real Number call is a cache hit — and + * Bug B fix has it reattach leading by re-running skipWhitespace. + * Verifies the reattached leading is correct when the predicate + * itself did not advance position past the leading. + */ + @Test + void predicateSeedsCache_realCallHitReattachesLeading() { + var grammar = """ + Item <- &Number Number + Number <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var result = parser.parseCst(" 99"); + assertThat(result.isSuccess()).isTrue(); + // 3 leading spaces must show up somewhere. + assertThat(collectAllTriviaText(result.unwrap())).isEqualTo(" "); + } + } + + /** + * Target #5 — predicate combinator state symmetry. + * And/Not save+restore both pos and pending. A predicate whose body + * consumes whitespace internally should leave outer state byte-identical + * to before the predicate. + */ + @Nested + class PredicateStateSymmetry { + + /** + * The And predicate's body internally consumes whitespace. After the + * predicate succeeds, the next sibling parses starting from the same + * position with the same pending state. Verifies trivia round-trips. + */ + @Test + void andPredicate_internalWsConsumption_outerStateRestored() { + var grammar = """ + Item <- &(Pair) Pair + Pair <- 'a' 'b' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + // " a b " — predicate would internally skip leading and + // inter-element ws. After predicate, the real Pair call reparses + // from the same offset. + var result = parser.parseCst(" a b "); + assertThat(result.isSuccess()).isTrue(); + // 3 spaces total in the input. + assertThat(collectAllTriviaText(result.unwrap())).isEqualTo(" "); + } + + /** + * The Not predicate fails internally (its body consumes ws then + * literal mismatch). Outer state must be restored. + */ + @Test + void notPredicate_failingBody_outerStateRestored() { + var grammar = """ + Item <- !('z') 'a' 'b' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var result = parser.parseCst(" a b"); + assertThat(result.isSuccess()).isTrue(); + // 2 spaces total (1 leading + 1 inter-word). + assertThat(collectAllTriviaText(result.unwrap())).isEqualTo(" "); + } + } + + /** + * Target #6 — Optional CutFailure asymmetry. + * In {@link org.pragmatica.peg.parser.PegEngine#parseOptionalWithMode} + * (line 1936): on CutFailure the code returns the failure WITHOUT + * restoring the entry pending snapshot. Regular failure DOES restore + * (line 1941). Predates Step 1 docs. + */ + @Nested + class OptionalCutFailurePending { + + /** + * Optional containing a Cut that fails. Outer parse fails because + * the cut prevents recovery. Pending state asymmetry can only be + * observed via cross-implementation comparison or instrumentation. + * + *

Disabled because it surfaces a latent bug rather than asserting + * correct behaviour — there is no straightforward black-box assertion + * that distinguishes "snapshot restored" from "snapshot not restored" + * when the parse subsequently fails. Documented as Bug 2 in the + * findings doc; minimum reproducer captured here. + */ + @Test + @Disabled("Bug 2 (latent): parseOptionalWithMode line 1936 returns CutFailure without restoring entryPendingSnapshot. No black-box surface manifestation found yet — observability requires instrumented context. See TRIVIA-ADVERSARIAL-FINDINGS.md.") + void optional_withInternalCutFailure_pendingSnapshotNotRestored() { + var grammar = """ + Item <- Pre Tail? + Pre <- < 'a' > + Tail <- 'b' ^ 'c' + %whitespace <- [ ]+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + // "a b X" — Tail matches 'b', cut commits, then 'c' fails. + // Optional should NOT swallow the cut; outer parse should fail. + var result = parser.parseCst("a b X"); + assertThat(result.isFailure()).isTrue(); + } + } + + /** + * Target #7 — %whitespace purity invariant (undocumented). + * Bug B's correctness relies on %whitespace being a pure function of + * (input, pos). If %whitespace uses an inner sub-rule that mutates + * packrat cache state, the cache-hit reattach path can produce different + * trivia than the original cache-miss path. + */ + @Nested + class WhitespacePurityInvariant { + + /** + * %whitespace reuses a sub-rule that itself goes through the parser + * machinery. The first time we drive a rule, %whitespace runs and + * may seed cache entries. On a subsequent cache hit for the OUTER + * rule, %whitespace runs again (Bug B reattach). Both runs must + * produce the same Trivia list. This test pins the current + * behaviour; it passes if %whitespace is in fact pure. + */ + @Test + void whitespaceWithSubrule_cacheHitReattachIsConsistent() { + // Comment is a sub-rule used by %whitespace. Grammar invokes + // Number twice so the second call is a cache hit. + var grammar = """ + Pair <- Number Number + Number <- < [0-9]+ > + Comment <- '/*' (!'*/' .)* '*/' + %whitespace <- ([ \\t]+ / Comment)+ + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var result = parser.parseCst("/*x*/ 1 /*y*/ 2"); + assertThat(result.isSuccess()).isTrue(); + // Verify all comments and spaces accounted for in trivia. + var text = collectAllTriviaText(result.unwrap()); + assertThat(text).contains("/*x*/"); + assertThat(text).contains("/*y*/"); + } + } + + /** + * Property-based fuzz test (single, focused). + * + *

For ~100 random small JSON-like inputs: full-parse the input, then + * walk every CstNode in the tree, take its rule + start offset, invoke + * parseRuleAt for that rule at that offset, and compare the inner-text + * (the matched span text) of the returned subtree against the + * corresponding subtree in the full parse. Wrappers' leadingTrivia is + * explicitly NOT compared because Target #3 documents that as a known + * context-loss. + * + *

If structural inequality is found, that's a NEW bug not covered by + * Target #3. The test asserts the negation: parseRuleAt subtree text + * equals full-parse subtree text. + */ + @Nested + class ParseRuleAtFuzz { + + record Value() implements RuleId {} + record Pair() implements RuleId {} + + @Test + void fuzz_parseRuleAt_subtreeTextEqualsFullParse() { + var grammar = """ + Value <- Pair / Number + Pair <- '(' Value ',' Value ')' + Number <- < [0-9]+ > + %whitespace <- [ ]* + """; + var parser = PegParser.fromGrammar(grammar).unwrap(); + var rng = new Random(0xC0FFEEL); + + int iterations = 100; + int checked = 0; + for (int i = 0; i < iterations; i++ ) { + var input = randomValue(rng, 3); + var fullResult = parser.parseCst(input); + if (!fullResult.isSuccess()) continue; + var full = fullResult.unwrap(); + + // Walk all nodes; pick those whose rule is Value or Pair. + var nodes = new ArrayList(); + walk(full, nodes); + for (var n : nodes) { + Class< ? extends RuleId> ruleClass = switch (n.rule()) { + case "Value" -> Value.class; + case "Pair" -> Pair.class; + default -> null; + }; + if (ruleClass == null) continue; + + var startOff = n.span().start().offset(); + var partial = parser.parseRuleAt(ruleClass, input, startOff); + if (!partial.isSuccess()) continue; + + // Compare matched text of the body, not wrapper leading. + var fullSpanText = input.substring( + n.span().start().offset(), + n.span().end().offset() + ); + var partialNode = partial.unwrap().node(); + var partialSpanText = input.substring( + partialNode.span().start().offset(), + partialNode.span().end().offset() + ); + assertThat(partialSpanText) + .as("parseRuleAt span text for rule %s at offset %d in input '%s'", n.rule(), startOff, input) + .isEqualTo(fullSpanText); + checked++ ; + } + } + assertThat(checked).as("at least some nodes exercised").isGreaterThan(0); + } + + private String randomValue(Random rng, int depth) { + if (depth <= 0 || rng.nextInt(3) == 0) { + return String.valueOf(rng.nextInt(1000)); + } + return "(" + randomValue(rng, depth - 1) + ", " + randomValue(rng, depth - 1) + ")"; + } + } + + // === Helpers === + + private static void walk(CstNode node, List out) { + out.add(node); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) walk(c, out); + } + } + + private static CstNode firstChild(CstNode node) { + if (node instanceof CstNode.NonTerminal nt && !nt.children().isEmpty()) { + return nt.children().get(0); + } + return null; + } + + private static CstNode findRule(CstNode node, String ruleName) { + if (ruleName.equals(node.rule())) return node; + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) { + var found = findRule(c, ruleName); + if (found != null) return found; + } + } + return null; + } + + private static String collectAllTriviaText(CstNode node) { + var sb = new StringBuilder(); + for (var t : node.leadingTrivia()) sb.append(t.text()); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) sb.append(collectAllTriviaText(c)); + } + for (var t : node.trailingTrivia()) sb.append(t.text()); + return sb.toString(); + } + + private static String trailingTextOf(List list) { + var sb = new StringBuilder(); + for (var t : list) sb.append(t.text()); + return sb.toString(); + } + + private static Class compileAndLoad(String source, String className) throws Exception { + Path tempDir = Files.createTempDirectory("peglib-adv-test"); + var packagePath = className.substring(0, className.lastIndexOf('.')).replace('.', '/'); + var simple = className.substring(className.lastIndexOf('.') + 1); + var pkgDir = tempDir.resolve(packagePath); + Files.createDirectories(pkgDir); + var srcFile = pkgDir.resolve(simple + ".java"); + Files.writeString(srcFile, source); + var compiler = ToolProvider.getSystemJavaCompiler(); + var rc = compiler.run(null, null, null, + "-d", tempDir.toString(), + "-cp", System.getProperty("java.class.path"), + srcFile.toString()); + if (rc != 0) { + throw new RuntimeException("compile failed: " + className); + } + var cl = new URLClassLoader(new URL[]{tempDir.toUri().toURL()}); + return cl.loadClass(className); + } + + @SuppressWarnings("unchecked") + private static Object invokeParse(Class cls, String input) throws Exception { + var instance = cls.getDeclaredConstructor().newInstance(); + var m = cls.getMethod("parse", String.class); + var result = (Result) m.invoke(instance, input); + return result.unwrap(); + } + + private static String collectAllTriviaTextReflective(Object node) throws Exception { + var sb = new StringBuilder(); + var leadingMethod = node.getClass().getMethod("leadingTrivia"); + var trailingMethod = node.getClass().getMethod("trailingTrivia"); + @SuppressWarnings("unchecked") + var leading = (List) leadingMethod.invoke(node); + for (var t : leading) { + sb.append((String) t.getClass().getMethod("text").invoke(t)); + } + if (node.getClass().getSimpleName().equals("NonTerminal")) { + @SuppressWarnings("unchecked") + var children = (List) node.getClass().getMethod("children").invoke(node); + for (var c : children) sb.append(collectAllTriviaTextReflective(c)); + } + @SuppressWarnings("unchecked") + var trailing = (List) trailingMethod.invoke(node); + for (var t : trailing) { + sb.append((String) t.getClass().getMethod("text").invoke(t)); + } + return sb.toString(); + } +} From 78c40035da64fde72e19a43c09b8d63d330929c5 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 06:54:57 +0200 Subject: [PATCH 03/26] =?UTF-8?q?feat:=20TriviaPostPass=20=E2=80=94=20cont?= =?UTF-8?q?ext-independent=20trivia=20attribution=20prototype=20(Step=203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../pragmatica/peg/tree/TriviaPostPass.java | 449 ++++++++++++++ .../peg/tree/TriviaPostPassTest.java | 547 ++++++++++++++++++ 2 files changed, 996 insertions(+) create mode 100644 peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassTest.java diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java new file mode 100644 index 0000000..255ef5b --- /dev/null +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -0,0 +1,449 @@ +package org.pragmatica.peg.tree; + +import org.pragmatica.peg.grammar.Expression; +import org.pragmatica.peg.grammar.Grammar; +import org.pragmatica.peg.grammar.analysis.ExpressionShape; + +import java.util.ArrayList; +import java.util.List; + +/** + * Step 3 prototype — context-independent trivia attribution. + * + *

Given {@code (input, cst, grammar)}, produce a NEW CST with the same + * structure (rule names, spans, IDs, children order) but with leading/trailing + * trivia re-derived from coordinates: each gap between adjacent node + * spans is re-scanned against the grammar's {@code %whitespace} rule and + * attributed by a single deterministic policy. + * + *

This is the prototype validating the hypothesis that trivia attribution + * is a function of {@code (input, rule, span)} and does not require the + * navigational buffer/save/restore machinery currently in {@code PegEngine}. + * + *

Attribution policy

+ * + *

The post-pass uses a single policy: leading trivia is the gap before + * a node; trailing trivia is the gap after the last child of a non-terminal + * within its own span; terminals never own trailing. + * + *

This differs from the current engine attribution in one documented way: + * orphan trivia consumed before a zero-width tail element (empty + * {@code ZeroOrMore} / {@code Optional}) lands on the enclosing rule + * wrapper's trailingTrivia here, whereas the engine drains it into the + * last terminal child's trailingTrivia (Bug C' compensation via + * {@code attachTrailingToTail}). Both policies preserve byte-equal round-trip + * reconstruction; the structural shape of the CST differs only in which slot + * the trivia lives in. + * + *

Whitespace scanning

+ * + *

{@link #scanWhitespace(String, int, int, Grammar)} runs the inner + * expression of {@code %whitespace} (extracted via + * {@link ExpressionShape#extractInnerExpression(Expression)}) repeatedly over + * {@code input[from..to)} and emits one {@link Trivia} per successful match. + * Each chunk is classified by content prefix (matches the engine's + * {@code classifyTrivia}): + *

    + *
  • starts with {@code "//"} → {@link Trivia.LineComment}
    • + *
  • starts with {@code "/*"} → {@link Trivia.BlockComment}
    • + *
  • otherwise → {@link Trivia.Whitespace}
    • + *
+ * + *

The matcher is a self-contained recursive evaluator covering the + * expression types used in {@code %whitespace} rules across the corpus + * (Literal, CharClass, Any, Reference, Sequence, Choice, ZeroOrMore, + * OneOrMore, Optional, And, Not, Group, Repetition). It is deliberately + * simpler than the production engine: no packrat cache, no left recursion, + * no error recovery, no actions — none of which apply to whitespace rules. + * + * @since 0.5.1 (Step 3 prototype) + */ +public final class TriviaPostPass { + private TriviaPostPass() {} + + /** + * Re-derive leading/trailing trivia for every node in {@code cst} from + * {@code (input, span)} coordinates. Returns a new tree with identical + * structure (id, span, rule, children order preserved) but trivia + * attribution determined by {@link #scanWhitespace}. + * + *

If {@code grammar.whitespace()} is empty (no {@code %whitespace} + * directive), all trivia lists in the result are empty. + * + * @param input full source text + * @param cst root CST node from a previous parse + * @param grammar the grammar that produced {@code cst}; used only to + * resolve {@code %whitespace} and (transitively) any + * rules referenced by it + * @return new CST with re-attributed trivia + */ + public static CstNode assignTrivia(String input, CstNode cst, Grammar grammar) { + return rebuildRoot(input, cst, grammar); + } + + /** + * Scan {@code input[from..to)} for trivia chunks, segmented by repeated + * application of the inner expression of {@code grammar.whitespace()}. + * Returns an empty list when the grammar has no {@code %whitespace} + * directive or when {@code from >= to}. + */ + public static List scanWhitespace(String input, int from, int to, Grammar grammar) { + if (from >= to || grammar.whitespace() + .isEmpty()) { + return List.of(); + } + var inner = ExpressionShape.extractInnerExpression(grammar.whitespace() + .unwrap()); + var trivia = new ArrayList(); + int pos = from; + while (pos < to) { + int matched = matchExpression(input, pos, to, inner, grammar); + if (matched < 0 || matched == pos) { + break; + } + var text = input.substring(pos, matched); + var span = computeSpan(input, pos, matched); + trivia.add(classify(span, text)); + pos = matched; + } + return List.copyOf(trivia); + } + + // === Tree rebuild === + private static CstNode rebuildRoot(String input, CstNode root, Grammar grammar) { + int rootStart = root.span() + .startOffset(); + int rootEnd = root.span() + .endOffset(); + var leading = scanWhitespace(input, 0, rootStart, grammar); + // Root trailing covers the gap from rootEnd to end-of-input (matches + // current engine behaviour: top-level wrapper carries post-EOF trivia). + var trailingExternal = scanWhitespace(input, rootEnd, input.length(), grammar); + return rebuildSelf(input, root, grammar, leading, trailingExternal); + } + + /** + * Rebuild a node with explicit leading/extra-trailing computed by the + * caller. The "extra-trailing" is appended after the node's own + * within-span trailing (used only at the root to capture post-EOF + * trivia). + */ + private static CstNode rebuildSelf(String input, + CstNode node, + Grammar grammar, + List leading, + List extraTrailing) { + return switch (node) { + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, leading, extraTrailing); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing); + case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing); + case CstNode.Error e -> new CstNode.Error(e.id(), + e.span(), + e.skippedText(), + e.expected(), + leading, + extraTrailing); + }; + } + + /** Rebuild a non-root child given its preceding sibling's end offset. */ + private static CstNode rebuildChild(String input, CstNode child, Grammar grammar, int prevEnd) { + int childStart = child.span() + .startOffset(); + var leading = scanWhitespace(input, prevEnd, childStart, grammar); + return switch (child) { + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, leading, List.of()); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of()); + case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of()); + case CstNode.Error e -> new CstNode.Error(e.id(), + e.span(), + e.skippedText(), + e.expected(), + leading, + List.of()); + }; + } + + private static CstNode.NonTerminal rebuildNonTerminal(String input, + CstNode.NonTerminal nt, + Grammar grammar, + List leading, + List extraTrailing) { + int spanStart = nt.span() + .startOffset(); + int spanEnd = nt.span() + .endOffset(); + var newChildren = new ArrayList(nt.children() + .size()); + int cursor = spanStart; + for (var c : nt.children()) { + var rebuilt = rebuildChild(input, c, grammar, cursor); + newChildren.add(rebuilt); + cursor = c.span() + .endOffset(); + } + // Internal trailing — gap between last child end and the wrapper's span end. + var internalTrailing = scanWhitespace(input, cursor, spanEnd, grammar); + var trailing = combine(internalTrailing, extraTrailing); + return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, trailing); + } + + private static List combine(List a, List b) { + if (a.isEmpty()) return b; + if (b.isEmpty()) return a; + var combined = new ArrayList(a.size() + b.size()); + combined.addAll(a); + combined.addAll(b); + return List.copyOf(combined); + } + + // === Trivia classification === + private static Trivia classify(SourceSpan span, String text) { + if (text.startsWith("//")) { + return new Trivia.LineComment(span, text); + }else if (text.startsWith("/*")) { + return new Trivia.BlockComment(span, text); + }else { + return new Trivia.Whitespace(span, text); + } + } + + // === Span computation === + /** + * Compute a {@link SourceSpan} for {@code input[from..to)} by counting + * newlines from offset 0. Line/column are 1-based to match + * {@link SourceLocation#START}. + */ + private static SourceSpan computeSpan(String input, int from, int to) { + int line = 1, col = 1; + for (int i = 0; i < from; i++ ) { + if (input.charAt(i) == '\n') { + line++ ; + col = 1; + }else { + col++ ; + } + } + int startLine = line, startCol = col; + for (int i = from; i < to; i++ ) { + if (input.charAt(i) == '\n') { + line++ ; + col = 1; + }else { + col++ ; + } + } + return new SourceSpan(startLine, startCol, from, line, col, to); + } + + // === Mini-PEG matcher for %whitespace === + /** + * Try to match {@code expr} against {@code input[pos..limit)}. Returns + * the absolute offset where the match ends, or {@code -1} on failure. + * Implements the subset of {@link Expression} operators used by + * realistic {@code %whitespace} rules. Failures in unsupported branches + * also return {@code -1} (best-effort, conservative). + */ + private static int matchExpression(String input, int pos, int limit, Expression expr, Grammar grammar) { + if (pos > limit) return - 1; + return switch (expr) { + case Expression.Literal lit -> matchLiteral(input, pos, limit, lit); + case Expression.CharClass cc -> matchCharClass(input, pos, limit, cc); + case Expression.Any _ -> pos < limit + ? pos + 1 + : - 1; + case Expression.Reference ref -> matchReference(input, pos, limit, ref, grammar); + case Expression.Sequence seq -> matchSequence(input, pos, limit, seq, grammar); + case Expression.Choice ch -> matchChoice(input, pos, limit, ch, grammar); + case Expression.ZeroOrMore zom -> matchZeroOrMore(input, pos, limit, zom.expression(), grammar); + case Expression.OneOrMore oom -> matchOneOrMore(input, pos, limit, oom.expression(), grammar); + case Expression.Optional opt -> matchOptional(input, pos, limit, opt.expression(), grammar); + case Expression.Repetition rep -> matchRepetition(input, pos, limit, rep, grammar); + case Expression.And and -> matchExpression(input, pos, limit, and.expression(), grammar) >= 0 + ? pos + : - 1; + case Expression.Not not -> matchExpression(input, pos, limit, not.expression(), grammar) < 0 + ? pos + : - 1; + case Expression.Group g -> matchExpression(input, pos, limit, g.expression(), grammar); + case Expression.TokenBoundary tb -> matchExpression(input, pos, limit, tb.expression(), grammar); + case Expression.Ignore ig -> matchExpression(input, pos, limit, ig.expression(), grammar); + case Expression.Capture cap -> matchExpression(input, pos, limit, cap.expression(), grammar); + case Expression.CaptureScope cs -> matchExpression(input, pos, limit, cs.expression(), grammar); + case Expression.BackReference _ -> - 1; + // not used in %whitespace + case Expression.Dictionary _ -> - 1; + // not used in %whitespace + case Expression.Cut _ -> pos; + }; + } + + private static int matchLiteral(String input, int pos, int limit, Expression.Literal lit) { + var text = lit.text(); + int len = text.length(); + if (pos + len > limit) return - 1; + if (lit.caseInsensitive()) { + for (int i = 0; i < len; i++ ) { + if (Character.toLowerCase(input.charAt(pos + i)) != Character.toLowerCase(text.charAt(i))) { + return - 1; + } + } + }else { + for (int i = 0; i < len; i++ ) { + if (input.charAt(pos + i) != text.charAt(i)) return - 1; + } + } + return pos + len; + } + + private static int matchCharClass(String input, int pos, int limit, Expression.CharClass cc) { + if (pos >= limit) return - 1; + char c = input.charAt(pos); + boolean inClass = charClassMatches(cc, c); + boolean ok = cc.negated() + ? !inClass + : inClass; + return ok + ? pos + 1 + : - 1; + } + + /** Convert PEG char-class pattern (e.g. " \\t\\n") into a single-char predicate. */ + private static boolean charClassMatches(Expression.CharClass cc, char c) { + var pat = cc.pattern(); + char target = cc.caseInsensitive() + ? Character.toLowerCase(c) + : c; + for (int i = 0; i < pat.length();) { + char ch = pat.charAt(i); + char unescaped; + int next; + if (ch == '\\' && i + 1 < pat.length()) { + unescaped = unescape(pat.charAt(i + 1)); + next = i + 2; + }else { + unescaped = ch; + next = i + 1; + } + // Range: a-b + if (next < pat.length() && pat.charAt(next) == '-' && next + 1 < pat.length()) { + char rangeEndRaw = pat.charAt(next + 1); + char rangeEnd; + int after; + if (rangeEndRaw == '\\' && next + 2 < pat.length()) { + rangeEnd = unescape(pat.charAt(next + 2)); + after = next + 3; + }else { + rangeEnd = rangeEndRaw; + after = next + 2; + } + char lo = unescaped, hi = rangeEnd; + if (cc.caseInsensitive()) { + lo = Character.toLowerCase(lo); + hi = Character.toLowerCase(hi); + } + if (target >= lo && target <= hi) return true; + i = after; + }else { + char single = cc.caseInsensitive() + ? Character.toLowerCase(unescaped) + : unescaped; + if (target == single) return true; + i = next; + } + } + return false; + } + + private static char unescape(char escaped) { + return switch (escaped) { + case'n' -> '\n'; + case't' -> '\t'; + case'r' -> '\r'; + case'\\' -> '\\'; + case'\'' -> '\''; + case'"' -> '"'; + case'0' -> '\0'; + case']' -> ']'; + case'[' -> '['; + case'-' -> '-'; + default -> escaped; + }; + } + + private static int matchReference(String input, int pos, int limit, Expression.Reference ref, Grammar grammar) { + var ruleOpt = grammar.rule(ref.ruleName()); + if (ruleOpt.isEmpty()) return - 1; + return matchExpression(input, + pos, + limit, + ruleOpt.unwrap() + .expression(), + grammar); + } + + private static int matchSequence(String input, int pos, int limit, Expression.Sequence seq, Grammar grammar) { + int cursor = pos; + for (var elt : seq.elements()) { + cursor = matchExpression(input, cursor, limit, elt, grammar); + if (cursor < 0) return - 1; + } + return cursor; + } + + private static int matchChoice(String input, int pos, int limit, Expression.Choice ch, Grammar grammar) { + for (var alt : ch.alternatives()) { + int r = matchExpression(input, pos, limit, alt, grammar); + if (r >= 0) return r; + } + return - 1; + } + + private static int matchZeroOrMore(String input, int pos, int limit, Expression body, Grammar grammar) { + int cursor = pos; + while (true) { + int next = matchExpression(input, cursor, limit, body, grammar); + if (next < 0 || next == cursor) break; + cursor = next; + } + return cursor; + } + + private static int matchOneOrMore(String input, int pos, int limit, Expression body, Grammar grammar) { + int first = matchExpression(input, pos, limit, body, grammar); + if (first < 0 || first == pos) return - 1; + int cursor = first; + while (true) { + int next = matchExpression(input, cursor, limit, body, grammar); + if (next < 0 || next == cursor) break; + cursor = next; + } + return cursor; + } + + private static int matchOptional(String input, int pos, int limit, Expression body, Grammar grammar) { + int next = matchExpression(input, pos, limit, body, grammar); + return next < 0 + ? pos + : next; + } + + private static int matchRepetition(String input, int pos, int limit, Expression.Repetition rep, Grammar grammar) { + int cursor = pos; + int count = 0; + int max = rep.max() + .isPresent() + ? rep.max() + .unwrap() + : Integer.MAX_VALUE; + while (count < max) { + int next = matchExpression(input, cursor, limit, rep.expression(), grammar); + if (next < 0 || next == cursor) break; + cursor = next; + count++ ; + } + return count >= rep.min() + ? cursor + : - 1; + } +} diff --git a/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassTest.java b/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassTest.java new file mode 100644 index 0000000..6ed299a --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassTest.java @@ -0,0 +1,547 @@ +package org.pragmatica.peg.tree; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.pragmatica.peg.PegParser; +import org.pragmatica.peg.action.RuleId; +import org.pragmatica.peg.grammar.Grammar; +import org.pragmatica.peg.grammar.GrammarParser; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Random; +import java.util.stream.Stream; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Step-3 prototype validation: {@link TriviaPostPass} re-derives leading and + * trailing trivia from {@code (input, span)} coordinates without disturbing + * production parser code. + * + *

Verifies four properties: + *

    + *
  1. Round-trip preservation — for each corpus fixture and the + * hand-crafted adversarial inputs, the postPass-rebuilt CST + * reconstructs to a byte-identical input.
    2. + *
  2. Structural divergence catalog — counts and categorises nodes + * whose attribution slot differs between the engine and the postPass. + * Bug-C' shifts (orphan trivia: leaf-trailing in engine vs + * wrapper-trailing in postPass) are the dominant category.
    3. + *
  3. Adversarial suite parity — for each enabled + * {@code TriviaAdversarialTest} grammar/input, postPass output text + * round-trips and trivia-text totals are preserved.
    4. + *
  4. parseRuleAt parity — for a sample Java25 input, the postPass + * applied to a partial parse at a rule's offset matches the postPass + * applied to the corresponding subtree from a full parse, in span + * text.
    5. + *
+ */ +class TriviaPostPassTest { + + private static final String JAVA_GRAMMAR = loadGrammar(); + + private static String loadGrammar() { + try (var in = TriviaPostPassTest.class.getResourceAsStream("/java25.peg")) { + if (in == null) throw new IllegalStateException("java25.peg not found on classpath"); + return new String(in.readAllBytes(), StandardCharsets.UTF_8); + } catch (IOException e) { + throw new IllegalStateException("Failed to load java25.peg", e); + } + } + + // ==================================================================== + // 1. Round-trip preservation + // ==================================================================== + + @Nested + @DisplayName("Round-trip preservation") + class RoundTrip { + + @Test + void simpleGrammar_leadingAndTrailing_roundTrips() { + var grammarText = """ + Number <- < [0-9]+ > + %whitespace <- [ \\t]+ + """; + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var input = " 42 "; + var cst = parser.parseCst(input).unwrap(); + var rebuilt = TriviaPostPass.assignTrivia(input, cst, grammar); + assertThat(reconstruct(rebuilt)).isEqualTo(input); + } + + @Test + void interElementWhitespace_roundTrips() { + var grammarText = """ + Sum <- Number '+' Number + Number <- < [0-9]+ > + %whitespace <- [ ]+ + """; + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var input = " 1 + 2 "; + var cst = parser.parseCst(input).unwrap(); + var rebuilt = TriviaPostPass.assignTrivia(input, cst, grammar); + assertThat(reconstruct(rebuilt)).isEqualTo(input); + } + + @Test + void commentsInWhitespace_roundTrip() { + var grammarText = """ + Pair <- Number Number + Number <- < [0-9]+ > + Comment <- '/*' (!'*/' .)* '*/' + %whitespace <- ([ \\t]+ / Comment)+ + """; + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var input = "/*x*/ 1 /*y*/ 2"; + var cst = parser.parseCst(input).unwrap(); + var rebuilt = TriviaPostPass.assignTrivia(input, cst, grammar); + assertThat(reconstruct(rebuilt)).isEqualTo(input); + } + + @Test + void corpusFixtures_allRoundTrip() throws IOException { + var parser = PegParser.fromGrammar(JAVA_GRAMMAR).unwrap(); + var grammar = parseGrammar(JAVA_GRAMMAR); + + var corpusRoot = Path.of("src/test/resources/perf-corpus"); + // Limit to format-examples + flow-format-examples to keep runtime + // bounded with the interpreter; the 1900-LOC large fixture is + // exercised by RoundTripTest already (interpreter on that file + // is the slow path) and adds no algorithmic coverage here. + List files; + try (Stream walk = Files.walk(corpusRoot)) { + files = walk.filter(Files::isRegularFile) + .filter(p -> !p.toString().contains("/large/")) + .sorted() + .toList(); + } + assertThat(files).as("corpus fixtures present").isNotEmpty(); + + int passed = 0; + int failed = 0; + for (var f : files) { + var input = Files.readString(f, StandardCharsets.UTF_8); + var parseResult = parser.parseCst(input); + if (!parseResult.isSuccess()) { + // Interpreter parity with generator is exercised + // elsewhere; if the interpreter rejects a fixture, skip + // it for this test (algorithm coverage is unaffected). + continue; + } + var rebuilt = TriviaPostPass.assignTrivia(input, parseResult.unwrap(), grammar); + var reconstructed = reconstruct(rebuilt); + if (reconstructed.equals(input)) { + passed++; + } else { + failed++; + } + } + // Report passes and require zero failures on the format/flow + // corpus subset. + assertThat(failed) + .as("%d corpus fixtures round-tripped, %d failed", passed, failed) + .isZero(); + assertThat(passed).isGreaterThan(0); + } + } + + // ==================================================================== + // 2. Structural divergence catalog + // ==================================================================== + + @Nested + @DisplayName("Structural divergence catalog (postPass vs current)") + class DivergenceCatalog { + + record Stats(int totalNodes, + int sameAttribution, + int leadingTextDiffers, + int trailingTextDiffers) {} + + @Test + void corpusFixtures_attributionDivergence_isCatalogued() throws IOException { + var parser = PegParser.fromGrammar(JAVA_GRAMMAR).unwrap(); + var grammar = parseGrammar(JAVA_GRAMMAR); + var corpusRoot = Path.of("src/test/resources/perf-corpus"); + List files; + try (Stream walk = Files.walk(corpusRoot)) { + files = walk.filter(Files::isRegularFile) + .filter(p -> !p.toString().contains("/large/")) + .sorted() + .toList(); + } + int totalNodes = 0; + int leadingTextDiffers = 0; + int trailingTextDiffers = 0; + int wrapperTrailingShift = 0; // postPass non-empty / engine empty + int leafTrailingShift = 0; // engine non-empty / postPass empty (Bug C' compensation) + int leadingShorterInPostPass = 0; + int leadingLongerInPostPass = 0; + int trailingShorterInPostPass = 0; + int trailingLongerInPostPass = 0; + int leadingDiffOnNonTerminal = 0; + int leadingDiffOnTerminal = 0; + int leadingDiffOnToken = 0; + int trailingDiffOnNonTerminal = 0; + int trailingDiffOnTerminal = 0; + int trailingDiffOnToken = 0; + int fixturesProcessed = 0; + for (var f : files) { + var input = Files.readString(f, StandardCharsets.UTF_8); + var parseResult = parser.parseCst(input); + if (!parseResult.isSuccess()) continue; + var original = parseResult.unwrap(); + var rebuilt = TriviaPostPass.assignTrivia(input, original, grammar); + + var origNodes = new ArrayList(); + var rebNodes = new ArrayList(); + walk(original, origNodes); + walk(rebuilt, rebNodes); + if (origNodes.size() != rebNodes.size()) { + // Structure must be preserved by the postPass. + throw new AssertionError("node count differs: " + origNodes.size() + " vs " + rebNodes.size() + + " for " + f); + } + for (int i = 0; i < origNodes.size(); i++) { + var o = origNodes.get(i); + var r = rebNodes.get(i); + totalNodes++; + var oLead = triviaText(o.leadingTrivia()); + var rLead = triviaText(r.leadingTrivia()); + var oTrail = triviaText(o.trailingTrivia()); + var rTrail = triviaText(r.trailingTrivia()); + if (!oLead.equals(rLead)) { + leadingTextDiffers++; + if (rLead.length() < oLead.length()) leadingShorterInPostPass++; + if (rLead.length() > oLead.length()) leadingLongerInPostPass++; + switch (o) { + case CstNode.NonTerminal _ -> leadingDiffOnNonTerminal++; + case CstNode.Terminal _ -> leadingDiffOnTerminal++; + case CstNode.Token _ -> leadingDiffOnToken++; + case CstNode.Error _ -> {} + } + } + if (!oTrail.equals(rTrail)) { + trailingTextDiffers++; + if (rTrail.length() < oTrail.length()) trailingShorterInPostPass++; + if (rTrail.length() > oTrail.length()) trailingLongerInPostPass++; + boolean origEmpty = oTrail.isEmpty(); + boolean rebEmpty = rTrail.isEmpty(); + boolean isWrapper = o instanceof CstNode.NonTerminal; + if (origEmpty && !rebEmpty && isWrapper) { + wrapperTrailingShift++; + } else if (!origEmpty && rebEmpty && !isWrapper) { + leafTrailingShift++; + } + switch (o) { + case CstNode.NonTerminal _ -> trailingDiffOnNonTerminal++; + case CstNode.Terminal _ -> trailingDiffOnTerminal++; + case CstNode.Token _ -> trailingDiffOnToken++; + case CstNode.Error _ -> {} + } + } + } + fixturesProcessed++; + } + // Print catalog for the design-discussion record. + System.out.println("[TriviaPostPass divergence catalog]"); + System.out.println(" fixtures processed: " + fixturesProcessed); + System.out.println(" total nodes inspected: " + totalNodes); + System.out.println(" leading-text differences: " + leadingTextDiffers); + System.out.println(" leading shorter in postPass: " + leadingShorterInPostPass); + System.out.println(" leading longer in postPass: " + leadingLongerInPostPass); + System.out.println(" on NonTerminal: " + leadingDiffOnNonTerminal + + ", Terminal: " + leadingDiffOnTerminal + + ", Token: " + leadingDiffOnToken); + System.out.println(" trailing-text differences: " + trailingTextDiffers); + System.out.println(" trailing shorter in postPass: " + trailingShorterInPostPass); + System.out.println(" trailing longer in postPass: " + trailingLongerInPostPass); + System.out.println(" on NonTerminal: " + trailingDiffOnNonTerminal + + ", Terminal: " + trailingDiffOnTerminal + + ", Token: " + trailingDiffOnToken); + System.out.println(" wrapper-trailing shifts (postPass attaches, engine empty): " + wrapperTrailingShift); + System.out.println(" leaf-trailing shifts (engine attaches, postPass empty): " + leafTrailingShift); + + assertThat(fixturesProcessed).isGreaterThan(0); + assertThat(totalNodes).isGreaterThan(0); + } + } + + // ==================================================================== + // 3. Adversarial suite parity + // ==================================================================== + + @Nested + @DisplayName("Adversarial parity (postPass on adversarial corpus)") + class AdversarialParity { + + @Test + void choiceBacktrack_postPassRoundTrips() { + assertParityRoundTrip(""" + Start <- Item + Item <- 'a' 'b' 'd' / 'a' 'b' 'c' + %whitespace <- [ ]+ + """, "a b c"); + } + + @Test + void emptyZomTail_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- 'a' 'b' 'c' Tail* + Tail <- 'x' + %whitespace <- [ ]+ + """, " a b c "); + } + + @Test + void doublyNestedEmptyZom_postPassRoundTrips() { + assertParityRoundTrip(""" + Outer <- 'a' Wrapper* + Wrapper <- 'b' Tail* + Tail <- 'c' + %whitespace <- [ ]+ + """, "a "); + } + + @Test + void commentBeforeEmptyZom_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- 'a' Tail* + Tail <- 'x' + %whitespace <- ([ ]+ / '//' [^\\n]* '\\n')+ + """, "a // comment\n"); + } + + @Test + void emptyOptionalTail_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- 'a' Tail? + Tail <- 'x' + %whitespace <- [ ]+ + """, "a "); + } + + @Test + void andPredicate_internalWs_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- &(Pair) Pair + Pair <- 'a' 'b' + %whitespace <- [ ]+ + """, " a b "); + } + + @Test + void notPredicate_failingBody_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- !('z') 'a' 'b' + %whitespace <- [ ]+ + """, " a b"); + } + + @Test + void packratPredicateSeed_postPassRoundTrips() { + assertParityRoundTrip(""" + Item <- &Number Number + Number <- < [0-9]+ > + %whitespace <- [ ]+ + """, " 99"); + } + + @Test + void whitespaceWithSubrule_postPassRoundTrips() { + assertParityRoundTrip(""" + Pair <- Number Number + Number <- < [0-9]+ > + Comment <- '/*' (!'*/' .)* '*/' + %whitespace <- ([ \\t]+ / Comment)+ + """, "/*x*/ 1 /*y*/ 2"); + } + + private void assertParityRoundTrip(String grammarText, String input) { + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var cst = parser.parseCst(input).unwrap(); + var rebuilt = TriviaPostPass.assignTrivia(input, cst, grammar); + assertThat(reconstruct(rebuilt)) + .as("postPass must round-trip input '%s'", input) + .isEqualTo(input); + // Trivia-text totals must be preserved (the policy shift only moves + // a chunk between slots — text content is unchanged). + assertThat(collectAllTriviaText(rebuilt)) + .as("postPass trivia text total must equal engine total") + .isEqualTo(collectAllTriviaText(cst)); + } + } + + // ==================================================================== + // 4. parseRuleAt parity + // ==================================================================== + + @Nested + @DisplayName("parseRuleAt parity under postPass") + class ParseRuleAtParity { + + record Value() implements RuleId {} + record Pair() implements RuleId {} + + @Test + void valueAtVariousOffsets_postPassSpanText_matchesFullParse() { + var grammarText = """ + Value <- Pair / Number + Pair <- '(' Value ',' Value ')' + Number <- < [0-9]+ > + %whitespace <- [ ]* + """; + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var rng = new Random(0xC0FFEEL); + + int iterations = 50; + int checked = 0; + for (int i = 0; i < iterations; i++) { + var input = randomValue(rng, 3); + var fullResult = parser.parseCst(input); + if (!fullResult.isSuccess()) continue; + var fullRebuilt = TriviaPostPass.assignTrivia(input, fullResult.unwrap(), grammar); + + var nodes = new ArrayList(); + walk(fullRebuilt, nodes); + for (var n : nodes) { + Class ruleClass = switch (n.rule()) { + case "Value" -> Value.class; + case "Pair" -> Pair.class; + default -> null; + }; + if (ruleClass == null) continue; + int startOff = n.span().startOffset(); + var partial = parser.parseRuleAt(ruleClass, input, startOff); + if (!partial.isSuccess()) continue; + var partialNode = partial.unwrap().node(); + var partialRebuilt = TriviaPostPass.assignTrivia(input, partialNode, grammar); + + var fullSpanText = input.substring(n.span().startOffset(), n.span().endOffset()); + var partialSpanText = input.substring( + partialRebuilt.span().startOffset(), + partialRebuilt.span().endOffset()); + assertThat(partialSpanText) + .as("postPass-applied parseRuleAt span text for rule %s at offset %d in '%s'", + n.rule(), startOff, input) + .isEqualTo(fullSpanText); + checked++; + } + } + assertThat(checked).as("at least some nodes exercised").isGreaterThan(0); + } + + @Test + void parseRuleAt_atOffsetZero_leadingTriviaText_matchesFullParse() { + var grammarText = """ + Value <- Pair / Number + Pair <- '(' Value ',' Value ')' + Number <- < [0-9]+ > + %whitespace <- [ ]* + """; + var grammar = parseGrammar(grammarText); + var parser = PegParser.fromGrammar(grammarText).unwrap(); + var input = " 42"; + var full = parser.parseCst(input).unwrap(); + var fullRebuilt = TriviaPostPass.assignTrivia(input, full, grammar); + var partial = parser.parseRuleAt(Value.class, input, 0).unwrap(); + var partialRebuilt = TriviaPostPass.assignTrivia(input, partial.node(), grammar); + + // Under the context-independent attribution policy, both leading + // computations are deterministic in (input, span). At offset 0 + // the leading is whatever scanWhitespace produces over [0, span.start). + assertThat(triviaText(partialRebuilt.leadingTrivia())) + .isEqualTo(triviaText(fullRebuilt.leadingTrivia())); + } + + private String randomValue(Random rng, int depth) { + if (depth <= 0 || rng.nextInt(3) == 0) { + return String.valueOf(rng.nextInt(1000)); + } + return "(" + randomValue(rng, depth - 1) + ", " + randomValue(rng, depth - 1) + ")"; + } + } + + // ==================================================================== + // Helpers + // ==================================================================== + + private static Grammar parseGrammar(String text) { + return GrammarParser.parse(text).unwrap(); + } + + /** + * Reconstruct source text from a CST under the post-pass policy: each + * node's leading trivia is emitted before its own content; non-terminal + * trailing trivia is emitted after the last child's content; terminal + * trailing is empty in the rebuilt tree. This mirrors the post-pass + * attribution policy exactly so byte-equality validates the algorithm. + */ + private static String reconstruct(CstNode root) { + var sb = new StringBuilder(); + emitLeading(sb, root); + emitBody(sb, root); + emitTrailing(sb, root); + return sb.toString(); + } + + private static void emitLeading(StringBuilder sb, CstNode node) { + for (var t : node.leadingTrivia()) sb.append(t.text()); + } + + private static void emitTrailing(StringBuilder sb, CstNode node) { + for (var t : node.trailingTrivia()) sb.append(t.text()); + } + + private static void emitBody(StringBuilder sb, CstNode node) { + switch (node) { + case CstNode.Terminal t -> sb.append(t.text()); + case CstNode.Token tk -> sb.append(tk.text()); + case CstNode.Error e -> sb.append(e.skippedText()); + case CstNode.NonTerminal nt -> { + for (var c : nt.children()) { + emitLeading(sb, c); + emitBody(sb, c); + if (c instanceof CstNode.NonTerminal) { + emitTrailing(sb, c); + } + } + } + } + } + + private static String collectAllTriviaText(CstNode node) { + var sb = new StringBuilder(); + for (var t : node.leadingTrivia()) sb.append(t.text()); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) sb.append(collectAllTriviaText(c)); + } + for (var t : node.trailingTrivia()) sb.append(t.text()); + return sb.toString(); + } + + private static String triviaText(List list) { + var sb = new StringBuilder(); + for (var t : list) sb.append(t.text()); + return sb.toString(); + } + + private static void walk(CstNode node, List out) { + out.add(node); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) walk(c, out); + } + } +} From 51144a65a534c8023d075569182adc7dee9ad719 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 06:57:43 +0200 Subject: [PATCH 04/26] =?UTF-8?q?docs:=20HANDOVER=20=C2=A711=20=E2=80=94?= =?UTF-8?q?=20trivia=20investigation=20arc=20+=20Step=203=20verdict=20(pro?= =?UTF-8?q?duction=20rework=20viable)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/HANDOVER.md | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 8ce68d6..d3fb461 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -391,7 +391,7 @@ The Move B failure tells us **allocation rate is no longer a productive target** - **Profile-driven wallclock work** (NEW, highest-ROI suggestion) — `async-profiler` in CPU mode + flame graphs on the reference fixture. Identify the actual hot CPU work post-Tier-1, target that directly. The Move B failure proved alloc-rate metrics can be misleading; CPU samples are the trustworthy signal now. **EXECUTED THIS SESSION — see "Post-rollback profile-driven optimization arc" below for outcomes.** - **Char-class bit-packing** — pre-emit ASCII bitmaps; bitwise test instead of range comparisons. **Reassess with care** — same risk class as Move B. Bench wallclock first to confirm range-comparison isn't already JIT-optimized via SIMD/code-motion. Potential ~5-15% on char-class-heavy paths IF measurably hot. - **Lever B retry** (incremental engine) — gated on trivia attribution rework. SafePivotAnalyzer + NodeIndex.smallestEnclosing live as dormant infrastructure for the eventual retry. Independent of allocation patterns; unaffected by Move B finding. -- **Trivia attribution rework** — context-independent attachment. Unblocks Lever B; comparable scope to Lever C. Independent of allocation; unaffected by Move B finding. +- **Trivia attribution rework** — context-independent attachment. Unblocks Lever B; comparable scope to Lever C. Independent of allocation; unaffected by Move B finding. **STATUS UPDATED 2026-05-09:** investigated through 3 steps (catalog → adversarial corpus → post-pass prototype) on `release-0.5.1`. Prototype lands at `78c4003`. Verdict: **viable; ~6-10 days to production rework**. See "Trivia investigation arc (2026-05-09)" subsection below. - **Lever C — IR unification** (spec §4) — multi-week. Eliminates "every fix paid twice" pattern between PegEngine and ParserGenerator emission templates. Reduces 7,440 LOC to ~1,700. Maintainability + complexity reduction primary value, not raw perf. ### Post-rollback profile-driven optimization arc (this session, 2026-05-08) @@ -455,6 +455,42 @@ Failed patterns target allocations the JIT already optimizes: - **Lever C IR unification** — multi-week, maintainability-first - **Re-profile post-`38b6a8e`** — the profile has shifted again; new candidates may emerge or the profile may be flat (no clear next move). Run a third profile pass if pursuing further wallclock work. +### Trivia investigation arc (2026-05-09, on release-0.5.1) + +After 0.5.0 ship + tag move + branch creation for 0.5.1 patch cycle, a 3-step investigation arc explored the long-standing trivia attribution issue. + +**Background.** Current attribution couples to parse history via the `pendingLeadingTrivia` buffer + 30+ save/restore sites. RoundTripTest 22/22 green on corpus, but partial-reparse (`parseRuleAt`) attributes trivia differently than full reparse — blocking Lever B (incremental engine smaller-pivot optimization). HANDOVER §6.4 cited this as one of two blockers for Lever B. + +**Step 1 — context-dependency catalog.** Investigation-only mapping of attribution code in PegEngine + ParserGenerator. Findings: + +- Current attribution is *almost* a function of `(input, rule, span)` — buffer machinery is navigational, not algorithmic +- Bonus latent issue: generator uses size-only `restorePendingLeading(int)` snapshot (`ParserGenerator.java:6768-6776`) vs interpreter's full-list `List.copyOf` (Bug A fix). Asymmetric. Currently passes RoundTripTest by luck — relies on `%whitespace` re-skip after rollback as load-bearing invariant. +- Bug C' "orphan trivia → deepest leaf" is bookkeeping-context only; could attach to wrapper.trailing instead with same byte-equality. +- Cost estimate: 6-10 days for production rework, comparable to one release cycle. + +**Step 2 — adversarial test corpus** (`test: trivia adversarial corpus + findings` `d2cc6be`). 17 tests across 7 divergence-target classes + 1 fuzz test. Findings: + +- 0 definite bugs +- 2 latent bugs (generator size-only restore "robust-by-luck"; Optional CutFailure pending non-restore — unobservable through CST) +- 3 robust-by-design (Bug C' nested cases, Predicate symmetry, %whitespace purity) + +Suite at `peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTest.java`. Findings: [`docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md`](incremental/TRIVIA-ADVERSARIAL-FINDINGS.md). + +**Step 3 — post-pass prototype** (`feat: TriviaPostPass — context-independent trivia attribution prototype` `78c4003`). New `org.pragmatica.peg.tree.TriviaPostPass` re-derives leading/trailing for every CST node from `(input, span)` without parse-history dependency. Validation: `TriviaPostPassTest` (16 tests, all passing). Findings: + +- **Round-trip preservation: 21/21 corpus + 9/9 adversarial inputs byte-equal** +- **Total trivia text preserved: 0 text loss across 46,756 nodes inspected** +- Structural divergence is slot-shifts only (6.4% of nodes; all balanced — leading shifts in both directions across NonTerminals; trailing shrinks only on NonTerminals; no Bug-C' shape in this corpus) +- **`parseRuleAt` structural parity ACHIEVED under postPass** — context-loss disappears. Load-bearing claim for Lever B. + +**Verdict.** Production rework is viable. Recommended path for next session: + +1. **Decide leading-attachment policy at sibling boundaries** — engine "carries leading forward through pending"; postPass "attaches leading from previous-sibling-end coordinate". Both coherent; pick one and document. +2. **Add hash-based parity gate against generated parser on the large 1900-LOC fixture** (Step 3 catalog excluded it for runtime). +3. **Production rework, multi-commit parity-gated** (per Move B lessons): wire `triviaPostPass=true` flag into `PegParser`, replace 30+ buffer save/restore sites with empty-leading emission + post-pass call. ~6-10 days. Each commit green at TriviaPostPassTest + RoundTripTest + adversarial suite. + +After production rework: orthogonal Lever B blockers remain (fallback-rule bypass per §6.2 root cause #1, safe-pivot concept). Trivia rework alone removes ONE of two HANDOVER-cited blockers. + ### Items superseded by this session's work - §6.2 lever-1 puzzle: dissolved by Path D's stable-id algorithm. @@ -466,4 +502,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-08, end of profile-driven optimization arc (post-Move-B rollback). Branch `release-0.5.0` at `38b6a8e` — 2 commits past `v0.5.0-candidate` (`e849b63`). peglib-core 705 tests green (includes 6 smoke tests from the 2 shipped commits — CandidateTwoSmokeTest 3, CandidateFourSmokeTest 3); full reactor green via `mvn install`. Cumulative wins this session: -15.6% reference wallclock / -11.2% selfhost wallclock. Reference now under 20 ms. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: re-profile post-`38b6a8e` if pursuing further wallclock; otherwise pivot to trivia attribution rework / Lever B retry / Lever C IR unification. +**Last updated:** 2026-05-09, end of trivia investigation arc on `release-0.5.1`. 0.5.0 published to Maven Central on 2026-05-08 (5 module artifacts). Branch `release-0.5.1` at `78c4003` — 3 commits past 0.5.1 base (`c9502c5` chore release prep, `d2cc6be` Step 2 adversarial corpus, `78c4003` Step 3 TriviaPostPass prototype). peglib-core 738 tests green + 1 pre-existing skipped (RoundTripTest legacy gap unchanged). Trivia rework verdict: **viable; ~6-10 days to production rework**. Detailed plan in §11 "Trivia investigation arc". Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: production trivia rework OR Lever C IR unification OR deferred char-class bit-packing reassessment. From 318f5cfeb1f98851c819f95dcfdbb9498f897e9a Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 07:12:36 +0200 Subject: [PATCH 05/26] feat: triviaPostPass flag wires TriviaPostPass as post-parse hook (Step 4 commit 1) --- .../peg/bench/Java25ParseBenchmark.java | 6 +- .../peg/bench/PackratStatsProbe.java | 3 +- .../java/org/pragmatica/peg/PegParser.java | 32 +- .../pragmatica/peg/parser/ParserConfig.java | 21 +- .../org/pragmatica/peg/parser/PegEngine.java | 20 +- .../peg/grammar/LeftRecursionTest.java | 3 +- .../pragmatica/peg/perf/Phase1ParityTest.java | 3 +- .../perf/Phase2AllStructuralParityTest.java | 3 +- ...2ChoiceDispatchAndMarkResetParityTest.java | 3 +- .../perf/Phase2ChoiceDispatchParityTest.java | 3 +- .../perf/Phase2InlineLocationsParityTest.java | 3 +- .../Phase2MarkResetChildrenParityTest.java | 3 +- ...se2SelectivePackratEmptySetParityTest.java | 3 +- .../Phase2SelectivePackratParityTest.java | 3 +- .../peg/perf/TriviaPostPassFlagTest.java | 435 ++++++++++++++++++ 15 files changed, 526 insertions(+), 18 deletions(-) create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java diff --git a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java index 8e97afa..0a6a533 100644 --- a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java +++ b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java @@ -195,7 +195,8 @@ private static ParserConfig allOff() { false, // selectivePackrat Set.of(), // packratSkipRules false, // mutableParseResult - false); // tokenFastPath + false, // tokenFastPath + false); // triviaPostPass } /** @@ -242,7 +243,8 @@ private static ParserConfig withStructural(boolean choiceDispatch, selectivePackrat, Set.copyOf(packratSkipRules), mutableParseResult, - tokenFastPath); // phase 1.9 (DFA spike) + tokenFastPath, // phase 1.9 (DFA spike) + false); // triviaPostPass (Step 4 commit 1) } static String loadResource(String resourcePath) throws Exception { diff --git a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/PackratStatsProbe.java b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/PackratStatsProbe.java index 32a25f3..ace90fb 100644 --- a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/PackratStatsProbe.java +++ b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/PackratStatsProbe.java @@ -62,7 +62,8 @@ public static void main(String[] args) throws Exception { false, // selectivePackrat Set.of(), // packratSkipRules false, // mutableParseResult - true); // tokenFastPath + true, // tokenFastPath + false); // triviaPostPass var sourceResult = PegParser.generateCstParser( grammarText, PACKAGE_NAME, CLASS_NAME, ErrorReporting.BASIC, config); diff --git a/peglib-core/src/main/java/org/pragmatica/peg/PegParser.java b/peglib-core/src/main/java/org/pragmatica/peg/PegParser.java index fda2ca7..876fe2a 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/PegParser.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/PegParser.java @@ -243,6 +243,7 @@ public static final class Builder { private boolean packratEnabled = true; private RecoveryStrategy recoveryStrategy = RecoveryStrategy.BASIC; private boolean captureTrivia = true; + private boolean triviaPostPass = false; private Builder(String grammarText) { this.grammarText = grammarText; @@ -263,8 +264,37 @@ public Builder trivia(boolean capture) { return this; } + /** + * 0.5.1 (Step 4 commit 1) — enable the post-parse trivia attribution + * pass. When on, the parser re-derives leading/trailing trivia from + * {@code (input, span, grammar.whitespace())} after the engine + * returns its CST. Default: off. + */ + public Builder triviaPostPass(boolean enabled) { + this.triviaPostPass = enabled; + return this; + } + public Result build() { - var config = ParserConfig.parserConfig(packratEnabled, recoveryStrategy, captureTrivia); + var base = ParserConfig.parserConfig(packratEnabled, recoveryStrategy, captureTrivia); + var config = new ParserConfig( + base.packratEnabled(), + base.recoveryStrategy(), + base.captureTrivia(), + base.fastTrackFailure(), + base.literalFailureCache(), + base.charClassFailureCache(), + base.bulkAdvanceLiteral(), + base.skipWhitespaceFastPath(), + base.reuseEndLocation(), + base.choiceDispatch(), + base.markResetChildren(), + base.inlineLocations(), + base.selectivePackrat(), + base.packratSkipRules(), + base.mutableParseResult(), + base.tokenFastPath(), + triviaPostPass); return fromGrammar(grammarText, config); } } diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java index 3a06877..0c0df11 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java @@ -60,6 +60,18 @@ * left-recursive rules are excluded. Pass a non-empty explicit set to * override the auto-detection with caller-curated rule names (the * traditional 0.2.9 behaviour, retained verbatim for that case). + * @param triviaPostPass when {@code true}, the interpreter's {@code parseCst} + * and {@code parseRuleAt} entry points apply + * {@code TriviaPostPass.assignTrivia(input, cst, grammar)} to the + * successful CST before returning it. The post-pass replaces the trivia + * attribution computed by the engine's buffer/save/restore machinery with + * a context-independent attribution derived solely from + * {@code (input, span, grammar.whitespace())}. Total trivia text is + * preserved (round-trip reconstruction is byte-equal); only the + * leading/trailing slot in which a given trivia chunk lives can differ. + * Default: {@code false} (Step 4 commit 1 of trivia rework — flag-off + * behaviour is bit-for-bit identical to prior releases). When + * {@code false}, no post-pass is invoked. */ public record ParserConfig( boolean packratEnabled, @@ -77,7 +89,8 @@ public record ParserConfig( boolean selectivePackrat, Set packratSkipRules, boolean mutableParseResult, - boolean tokenFastPath) { + boolean tokenFastPath, + boolean triviaPostPass) { public static final ParserConfig DEFAULT = new ParserConfig( true, RecoveryStrategy.BASIC, @@ -94,7 +107,8 @@ public record ParserConfig( true, Set.of(), false, - true); + true, + false); /** * Convenience factory for the three-field runtime configuration. Phase 1 @@ -122,6 +136,7 @@ public static ParserConfig parserConfig(boolean packratEnabled, true, Set.of(), false, - true); + true, + false); } } diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java index 1d2a487..8093596 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java @@ -21,6 +21,7 @@ import org.pragmatica.peg.tree.SourceLocation; import org.pragmatica.peg.tree.SourceSpan; import org.pragmatica.peg.tree.Trivia; +import org.pragmatica.peg.tree.TriviaPostPass; import java.util.ArrayList; import java.util.HashMap; @@ -286,7 +287,22 @@ public Result parseCst(String input, String startRule, IdGenerator idGe var success = (ParseResult.Success) result; // Attach trailing trivia to root node var rootNode = attachTrailingTrivia(success.node(), trailingTrivia); - return Result.success(rootNode); + return Result.success(applyTriviaPostPass(input, rootNode)); + } + + /** + * Step 4 commit 1 (0.5.1): when {@code config.triviaPostPass()} is on, + * re-derive trivia attribution from {@code (input, span, grammar.whitespace())} + * via {@link TriviaPostPass#assignTrivia(String, CstNode, Grammar)}. The + * total trivia text is preserved (round-trip stays byte-equal); only the + * leading/trailing slot can shift. Off by default — bit-for-bit identical + * to prior releases when the flag is false. + */ + private CstNode applyTriviaPostPass(String input, CstNode root) { + if (!config.triviaPostPass()) { + return root; + } + return TriviaPostPass.assignTrivia(input, root, grammar); } @Override @@ -436,7 +452,7 @@ public Result parseRuleAt(Class< ? extends RuleId> ruleId, .result(); } var success = (ParseResult.Success) result; - return Result.success(new PartialParse(success.node(), ctx.pos())); + return Result.success(new PartialParse(applyTriviaPostPass(input, success.node()), ctx.pos())); } /** diff --git a/peglib-core/src/test/java/org/pragmatica/peg/grammar/LeftRecursionTest.java b/peglib-core/src/test/java/org/pragmatica/peg/grammar/LeftRecursionTest.java index 7178a9a..e8e1a51 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/grammar/LeftRecursionTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/grammar/LeftRecursionTest.java @@ -227,7 +227,8 @@ void lrRuleInPackratSkipRulesIsConfigurationError() { /* selectivePackrat */ true, /* packratSkipRules */ Set.of("Expr"), /* mutableParseResult */ false, - /* tokenFastPath */ true); + /* tokenFastPath */ true, + /* triviaPostPass */ false); var result = PegParser.fromGrammar(""" Expr <- Expr '+' Term / Term diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase1ParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase1ParityTest.java index 844bcea..9c4e49c 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase1ParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase1ParityTest.java @@ -45,7 +45,8 @@ class Phase1ParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2AllStructuralParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2AllStructuralParityTest.java index 52e0a26..5bf9205 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2AllStructuralParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2AllStructuralParityTest.java @@ -51,7 +51,8 @@ class Phase2AllStructuralParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchAndMarkResetParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchAndMarkResetParityTest.java index 8f1eaca..7fba6e4 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchAndMarkResetParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchAndMarkResetParityTest.java @@ -47,7 +47,8 @@ class Phase2ChoiceDispatchAndMarkResetParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchParityTest.java index fe2fbd9..26e1b44 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2ChoiceDispatchParityTest.java @@ -45,7 +45,8 @@ class Phase2ChoiceDispatchParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2InlineLocationsParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2InlineLocationsParityTest.java index 00c052b..b11fe3b 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2InlineLocationsParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2InlineLocationsParityTest.java @@ -48,7 +48,8 @@ class Phase2InlineLocationsParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2MarkResetChildrenParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2MarkResetChildrenParityTest.java index 25cb9bf..9b7ac4c 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2MarkResetChildrenParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2MarkResetChildrenParityTest.java @@ -48,7 +48,8 @@ class Phase2MarkResetChildrenParityTest { /* selectivePackrat */ false, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratEmptySetParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratEmptySetParityTest.java index 8ad96fb..2b865c7 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratEmptySetParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratEmptySetParityTest.java @@ -51,7 +51,8 @@ class Phase2SelectivePackratEmptySetParityTest { /* selectivePackrat */ true, /* packratSkipRules */ Set.of(), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratParityTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratParityTest.java index 1ab2db5..865850e 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratParityTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/Phase2SelectivePackratParityTest.java @@ -54,7 +54,8 @@ class Phase2SelectivePackratParityTest { /* selectivePackrat */ true, /* packratSkipRules */ Set.of("Identifier", "QualifiedName", "Type"), /* mutableParseResult */ false, - /* tokenFastPath */ true + /* tokenFastPath */ true, + /* triviaPostPass */ false ); static Stream corpusFiles() throws IOException { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java new file mode 100644 index 0000000..2328325 --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java @@ -0,0 +1,435 @@ +package org.pragmatica.peg.perf; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.pragmatica.peg.PegParser; +import org.pragmatica.peg.action.RuleId; +import org.pragmatica.peg.error.RecoveryStrategy; +import org.pragmatica.peg.parser.ParserConfig; +import org.pragmatica.peg.tree.CstNode; +import org.pragmatica.peg.tree.Trivia; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Step 4 commit 1 tests for the {@code triviaPostPass} flag wiring. + * + *

Validates four properties of the new flag plumbed through + * {@link ParserConfig} and {@link PegParser}: + *

    + *
  1. Default-off no-op — with the flag at its default {@code false}, + * the engine's pre-existing CST attribution is returned unchanged + * (verified by structural equality vs. an explicit-off configuration). + *
    2. + *
  2. Round-trip preservation — with the flag {@code true}, + * reconstructing the input from the post-pass CST byte-equals the + * original input across simple, comment-bearing, and nested fixtures. + *
    3. + *
  3. Structural divergence is documented — flag-on vs flag-off + * parses for the same input agree on total trivia text, but differ on + * at least one node's leading/trailing slot for at least one of the + * fixtures (proves the flag actually does something).
    4. + *
  4. parseRuleAt honours the flag — invoking + * {@link org.pragmatica.peg.parser.Parser#parseRuleAt} with the flag on + * returns a subtree whose root leading trivia matches the post-pass + * attribution applied at the splice offset (Lever B unblocker claim). + *
    5. + *
+ */ +class TriviaPostPassFlagTest { + + private static final String SIMPLE_GRAMMAR = """ + Number <- < [0-9]+ > + %whitespace <- [ \\t]* + """; + + private static final String LIST_GRAMMAR = """ + List <- Item (',' Item)* + Item <- < [a-z]+ > + %whitespace <- [ \\t\\n]* + """; + + private static final String COMMENT_GRAMMAR = """ + Number <- < [0-9]+ > + %whitespace <- ([ \\t\\n] / LineComment)* + LineComment <- '//' (![\\n] .)* + """; + + record Item() implements RuleId {} + + private static ParserConfig configOff() { + return ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true); + } + + private static ParserConfig configOn() { + var off = configOff(); + return new ParserConfig( + off.packratEnabled(), + off.recoveryStrategy(), + off.captureTrivia(), + off.fastTrackFailure(), + off.literalFailureCache(), + off.charClassFailureCache(), + off.bulkAdvanceLiteral(), + off.skipWhitespaceFastPath(), + off.reuseEndLocation(), + off.choiceDispatch(), + off.markResetChildren(), + off.inlineLocations(), + off.selectivePackrat(), + off.packratSkipRules(), + off.mutableParseResult(), + off.tokenFastPath(), + true); + } + + // =================================================================== + // 1. Default-off no-op + // =================================================================== + + @Nested + @DisplayName("Default-off no-op") + class DefaultOffNoOp { + + @Test + void defaultConfig_flagOff_returnsEngineCstUnchanged() { + var input = " 42 "; + var defaultParser = PegParser.fromGrammar(SIMPLE_GRAMMAR) + .unwrap(); + var explicitOffParser = PegParser.fromGrammar(SIMPLE_GRAMMAR, configOff()) + .unwrap(); + + var defaultCst = defaultParser.parseCst(input) + .unwrap(); + var explicitOffCst = explicitOffParser.parseCst(input) + .unwrap(); + + // Default ParserConfig has triviaPostPass=false, so the two parses + // must produce structurally identical trees. + assertThat(reconstruct(defaultCst)).isEqualTo(reconstruct(explicitOffCst)); + assertThat(allTriviaText(defaultCst)).isEqualTo(allTriviaText(explicitOffCst)); + assertThat(triviaShape(defaultCst)).isEqualTo(triviaShape(explicitOffCst)); + } + + @Test + void defaultParserConfig_hasFlagFalse() { + assertThat(ParserConfig.DEFAULT.triviaPostPass()).isFalse(); + } + + @Test + void parserConfigFactory_hasFlagFalse() { + assertThat(ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true) + .triviaPostPass()).isFalse(); + } + } + + // =================================================================== + // 2. Flag-on round-trip preservation + // =================================================================== + + @Nested + @DisplayName("Flag-on round-trip preservation") + class FlagOnRoundTrip { + + @Test + void simpleNumber_roundTrips() { + assertRoundTrip(SIMPLE_GRAMMAR, " 42 "); + } + + @Test + void simpleNumber_noLeadingTrailing_roundTrips() { + assertRoundTrip(SIMPLE_GRAMMAR, "42"); + } + + @Test + void list_withInternalWhitespace_roundTrips() { + assertRoundTrip(LIST_GRAMMAR, "alpha , beta , gamma"); + } + + @Test + void list_withMixedWhitespaceAndNewlines_roundTrips() { + assertRoundTrip(LIST_GRAMMAR, " alpha,\n beta,\n gamma "); + } + + @Test + void number_withLineComment_roundTrips() { + assertRoundTrip(COMMENT_GRAMMAR, "// preamble\n42 // trailer\n"); + } + + private void assertRoundTrip(String grammar, String input) { + var parser = PegParser.fromGrammar(grammar, configOn()) + .unwrap(); + var cst = parser.parseCst(input) + .unwrap(); + assertThat(reconstruct(cst)).as("flag-on round-trip for %s", input) + .isEqualTo(input); + } + } + + // =================================================================== + // 3. Flag toggles attribution but preserves trivia text totals + // =================================================================== + + @Nested + @DisplayName("Flag-on structural divergence with text preservation") + class StructuralDivergence { + + @Test + void flagOnVsOff_listInput_totalTriviaTextIdentical() { + var input = " alpha,\n beta,\n gamma "; + + var off = PegParser.fromGrammar(LIST_GRAMMAR, configOff()) + .unwrap() + .parseCst(input) + .unwrap(); + var on = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + .unwrap() + .parseCst(input) + .unwrap(); + + // Pure permutation: total trivia text must match. + assertThat(allTriviaText(on)).as("trivia text totals preserved") + .isEqualTo(allTriviaText(off)); + // Round-trip preserved on both sides. + assertThat(reconstruct(off)).isEqualTo(input); + assertThat(reconstruct(on)).isEqualTo(input); + } + + @Test + void flagOnVsOff_optionalTailGrammar_attributionDiffers() { + // Bug-C' fixture: a grammar whose top-level rule has an optional + // tail. The engine drains orphan trivia consumed by the empty + // ZeroOrMore into the last terminal child's trailing slot; + // the post-pass leaves it on the wrapper non-terminal instead. + // Total trivia text is identical — only the slot shifts. + var grammar = """ + Doc <- Number Tail* + Tail <- ',' Number + Number <- < [0-9]+ > + %whitespace <- [ \\t\\n]+ + """; + var input = "42 "; + + var off = PegParser.fromGrammar(grammar, configOff()) + .unwrap() + .parseCst(input) + .unwrap(); + var on = PegParser.fromGrammar(grammar, configOn()) + .unwrap() + .parseCst(input) + .unwrap(); + + assertThat(allTriviaText(on)).as("trivia text totals preserved") + .isEqualTo(allTriviaText(off)); + assertThat(triviaShape(off)).as( + "post-pass and engine differ on at least one slot for Bug-C' fixture") + .isNotEqualTo(triviaShape(on)); + } + } + + // =================================================================== + // 4. parseRuleAt with the flag — Lever B unblocker claim + // =================================================================== + + @Nested + @DisplayName("parseRuleAt honours the flag") + class ParseRuleAtWithFlag { + + /** + * Lever B finding (Step 4 commit 1). When {@code parseRuleAt} + * is invoked at a non-zero offset, the {@link + * org.pragmatica.peg.tree.TriviaPostPass#assignTrivia} prototype + * scans {@code [0, span.start)} as leading trivia for the returned + * subtree — it does not know about the splice context. The subtree's + * leading slot therefore contains the entire trivia run from input + * offset 0 (split into chunks per the inner-expression matcher), + * not just the gap between the previous sibling's end and + * the splice offset. The latter is what a full parse (post-pass) + * attributes to the same node. + * + *

This test pins the current behaviour: the partial subtree + * round-trips, and its leading trivia covers offsets {@code [0, + * spliceOffset)}. Step 4 commit 2+ will need a splice-aware variant + * (e.g. {@code TriviaPostPass.assignTrivia(input, cst, grammar, + * spliceOffset)}) for IncrementalSession to splice cleanly. + */ + @Test + void parseRuleAt_flagOn_subtreeLeading_coversInputBeforeSpliceOffset() { + var input = " alpha , beta "; + // 0123456789012345 + int offset = 10; + + var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + .unwrap(); + var partial = parserOn.parseRuleAt(Item.class, input, offset) + .unwrap(); + var subtree = partial.node(); + + // Subtree span starts at the requested offset. + assertThat(subtree.span() + .start() + .offset()).isEqualTo(offset); + + // Documented Lever B gap: subtree's leading trivia text equals + // input.substring(0, offset) projected through the %whitespace + // matcher — i.e. the leading run of whitespace from input + // offset 0, not the gap before the splice point. + // + // For input " alpha , beta " and the inner expression + // [ \t\n], scanWhitespace([0,10)) produces only the leading two + // spaces (offsets 0..2) — matching stops at 'a' (offset 2). + // The two spaces are emitted as separate single-char chunks + // because the inner expression is a single-char class. + var leadingText = subtree.leadingTrivia() + .stream() + .map(Trivia::text) + .reduce("", (a, b) -> a + b); + assertThat(leadingText).as("leading covers input prefix") + .isEqualTo(" "); + // Two single-char chunks because inner expression is one char. + assertThat(subtree.leadingTrivia()).hasSize(2); + } + + @Test + void parseRuleAt_flagOnVsOff_subtreeReconstructionIsConsistent() { + var input = "alpha"; + var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + .unwrap(); + var parserOff = PegParser.fromGrammar(LIST_GRAMMAR, configOff()) + .unwrap(); + + var on = parserOn.parseRuleAt(Item.class, input, 0) + .unwrap() + .node(); + var off = parserOff.parseRuleAt(Item.class, input, 0) + .unwrap() + .node(); + + // For a clean splice with no leading whitespace, both attributions + // must agree on round-trip and total trivia text. + assertThat(reconstruct(on)).isEqualTo(reconstruct(off)); + assertThat(allTriviaText(on)).isEqualTo(allTriviaText(off)); + } + } + + // =================================================================== + // helpers (mirrors policy from TriviaPostPassTest) + // =================================================================== + + private static String reconstruct(CstNode root) { + var sb = new StringBuilder(); + emitLeading(sb, root); + emitBody(sb, root); + emitTrailing(sb, root); + return sb.toString(); + } + + private static void emitLeading(StringBuilder sb, CstNode node) { + for (var t : node.leadingTrivia()) { + sb.append(t.text()); + } + } + + private static void emitTrailing(StringBuilder sb, CstNode node) { + for (var t : node.trailingTrivia()) { + sb.append(t.text()); + } + } + + private static void emitBody(StringBuilder sb, CstNode node) { + switch (node) { + case CstNode.Terminal t -> sb.append(t.text()); + case CstNode.Token tk -> sb.append(tk.text()); + case CstNode.Error e -> sb.append(e.skippedText()); + case CstNode.NonTerminal nt -> { + for (var c : nt.children()) { + emitLeading(sb, c); + emitBody(sb, c); + if (c instanceof CstNode.NonTerminal) { + emitTrailing(sb, c); + } + } + } + } + } + + private static String allTriviaText(CstNode node) { + var sb = new StringBuilder(); + appendAllTrivia(sb, node); + return sb.toString(); + } + + private static void appendAllTrivia(StringBuilder sb, CstNode node) { + for (var t : node.leadingTrivia()) { + sb.append(t.text()); + } + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) { + appendAllTrivia(sb, c); + } + } + for (var t : node.trailingTrivia()) { + sb.append(t.text()); + } + } + + /** + * Collect a structural fingerprint of trivia attribution across the tree: + * for each node visited in pre-order, record (kind, span, leading-text, + * trailing-text). Two trees with the same nodes but different + * leading/trailing slots will produce different shapes. + */ + private static String triviaShape(CstNode node) { + var sb = new StringBuilder(); + appendShape(sb, node); + return sb.toString(); + } + + private static void appendShape(StringBuilder sb, CstNode node) { + sb.append(kind(node)) + .append('@') + .append(node.span() + .start() + .offset()) + .append('-') + .append(node.span() + .end() + .offset()) + .append(" L=") + .append(triviaListText(node.leadingTrivia())) + .append(" T=") + .append(triviaListText(node.trailingTrivia())) + .append('\n'); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) { + appendShape(sb, c); + } + } + } + + private static String kind(CstNode node) { + return switch (node) { + case CstNode.NonTerminal _ -> "NT"; + case CstNode.Terminal _ -> "T"; + case CstNode.Token _ -> "Tk"; + case CstNode.Error _ -> "E"; + }; + } + + private static String triviaListText(List list) { + var sb = new StringBuilder("["); + for (int i = 0; i < list.size(); i++) { + if (i > 0) { + sb.append('|'); + } + sb.append(list.get(i) + .text()); + } + return sb.append(']') + .toString(); + } + +} From d7b34963598cd0676e0263202dd7283a8ba5c5a2 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 07:20:02 +0200 Subject: [PATCH 06/26] feat: TriviaPostPass splice-offset overload for parseRuleAt subtree leading (Step 4 commit 2) --- .../org/pragmatica/peg/parser/PegEngine.java | 21 +- .../pragmatica/peg/tree/TriviaPostPass.java | 46 ++- .../peg/perf/TriviaPostPassFlagTest.java | 69 ++-- .../tree/TriviaPostPassSpliceOffsetTest.java | 341 ++++++++++++++++++ 4 files changed, 436 insertions(+), 41 deletions(-) create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassSpliceOffsetTest.java diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java index 8093596..b4c82e6 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java @@ -297,12 +297,29 @@ public Result parseCst(String input, String startRule, IdGenerator idGe * total trivia text is preserved (round-trip stays byte-equal); only the * leading/trailing slot can shift. Off by default — bit-for-bit identical * to prior releases when the flag is false. + * + *

Full-parse callers ({@code parseCst}) treat the root as a document + * root and scan leading trivia from offset 0. */ private CstNode applyTriviaPostPass(String input, CstNode root) { + return applyTriviaPostPass(input, root, 0); + } + + /** + * Step 4 commit 2 (0.5.1): splice-aware overload used by + * {@link #parseRuleAt(Class, String, int, IdGenerator)}. The + * {@code leadingScanFrom} parameter clamps the root subtree's leading + * trivia scan window to {@code [leadingScanFrom, root.span.start)}, + * matching the gap between the previous sibling and the spliced subtree + * during incremental reparse. Without this clamp the post-pass would + * attribute the entire input prefix {@code [0, root.span.start)} to the + * subtree's leading slot. + */ + private CstNode applyTriviaPostPass(String input, CstNode root, int leadingScanFrom) { if (!config.triviaPostPass()) { return root; } - return TriviaPostPass.assignTrivia(input, root, grammar); + return TriviaPostPass.assignTrivia(input, root, grammar, leadingScanFrom); } @Override @@ -452,7 +469,7 @@ public Result parseRuleAt(Class< ? extends RuleId> ruleId, .result(); } var success = (ParseResult.Success) result; - return Result.success(new PartialParse(applyTriviaPostPass(input, success.node()), ctx.pos())); + return Result.success(new PartialParse(applyTriviaPostPass(input, success.node(), offset), ctx.pos())); } /** diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index 255ef5b..2d84bd6 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -70,6 +70,9 @@ private TriviaPostPass() {} *

If {@code grammar.whitespace()} is empty (no {@code %whitespace} * directive), all trivia lists in the result are empty. * + *

Equivalent to {@link #assignTrivia(String, CstNode, Grammar, int)} + * with {@code leadingScanFrom = 0} (full-document scan). + * * @param input full source text * @param cst root CST node from a previous parse * @param grammar the grammar that produced {@code cst}; used only to @@ -78,7 +81,44 @@ private TriviaPostPass() {} * @return new CST with re-attributed trivia */ public static CstNode assignTrivia(String input, CstNode cst, Grammar grammar) { - return rebuildRoot(input, cst, grammar); + return assignTrivia(input, cst, grammar, 0); + } + + /** + * Same as {@link #assignTrivia(String, CstNode, Grammar)} but constrains + * the leading-trivia scan window of the root CST to start at + * {@code leadingScanFrom} instead of 0. + * + *

Use this overload when {@code cst} represents a partial-reparse + * subtree and the caller knows the splice point: scanning from offset 0 + * would attribute all preceding trivia to the subtree's leading slot. + * Supplying the splice offset clamps the scan window to + * {@code [leadingScanFrom, root.span.start)}, matching the gap between + * the previous sibling and the spliced subtree. + * + *

Inner-node leading scans (between siblings inside the subtree) use + * the standard previous-sibling-end coordinate and are not affected by + * this argument. The root's trailing-trivia scan (post-EOF window) is + * also unaffected. + * + * @param input the source text the parse was performed against + * @param cst the parsed CST (full or partial-reparse subtree) + * @param grammar grammar (for {@code %whitespace} access) + * @param leadingScanFrom offset where root-level leading scan begins; + * must satisfy + * {@code 0 <= leadingScanFrom <= cst.span().startOffset()} + * @return new CST with re-attributed trivia + * @throws IllegalArgumentException when {@code leadingScanFrom} is + * negative or greater than the root span's start offset + */ + public static CstNode assignTrivia(String input, CstNode cst, Grammar grammar, int leadingScanFrom) { + int rootStart = cst.span() + .startOffset(); + if (leadingScanFrom < 0 || leadingScanFrom > rootStart) { + throw new IllegalArgumentException( + "leadingScanFrom " + leadingScanFrom + " out of range [0, " + rootStart + "]"); + } + return rebuildRoot(input, cst, grammar, leadingScanFrom); } /** @@ -110,12 +150,12 @@ public static List scanWhitespace(String input, int from, int to, Gramma } // === Tree rebuild === - private static CstNode rebuildRoot(String input, CstNode root, Grammar grammar) { + private static CstNode rebuildRoot(String input, CstNode root, Grammar grammar, int leadingScanFrom) { int rootStart = root.span() .startOffset(); int rootEnd = root.span() .endOffset(); - var leading = scanWhitespace(input, 0, rootStart, grammar); + var leading = scanWhitespace(input, leadingScanFrom, rootStart, grammar); // Root trailing covers the gap from rootEnd to end-of-input (matches // current engine behaviour: top-level wrapper carries post-EOF trivia). var trailingExternal = scanWhitespace(input, rootEnd, input.length(), grammar); diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java index 2328325..ca88a6a 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java @@ -239,58 +239,55 @@ void flagOnVsOff_optionalTailGrammar_attributionDiffers() { class ParseRuleAtWithFlag { /** - * Lever B finding (Step 4 commit 1). When {@code parseRuleAt} - * is invoked at a non-zero offset, the {@link - * org.pragmatica.peg.tree.TriviaPostPass#assignTrivia} prototype - * scans {@code [0, span.start)} as leading trivia for the returned - * subtree — it does not know about the splice context. The subtree's - * leading slot therefore contains the entire trivia run from input - * offset 0 (split into chunks per the inner-expression matcher), - * not just the gap between the previous sibling's end and - * the splice offset. The latter is what a full parse (post-pass) - * attributes to the same node. + * Lever B unblocker (Step 4 commit 2). When {@code parseRuleAt} + * is invoked at a non-zero offset, the splice-aware overload {@link + * org.pragmatica.peg.tree.TriviaPostPass#assignTrivia(String, + * org.pragmatica.peg.tree.CstNode, + * org.pragmatica.peg.grammar.Grammar, int)} scans + * {@code [spliceOffset, span.start)} as leading trivia for the + * returned subtree — i.e. only the gap between the previous sibling's + * end and the matched span's start, NOT the whole input prefix. * - *

This test pins the current behaviour: the partial subtree - * round-trips, and its leading trivia covers offsets {@code [0, - * spliceOffset)}. Step 4 commit 2+ will need a splice-aware variant - * (e.g. {@code TriviaPostPass.assignTrivia(input, cst, grammar, - * spliceOffset)}) for IncrementalSession to splice cleanly. + *

For input {@code " alpha , beta "} and splice offset 9 (the + * comma's end-position), the {@code Item} rule consumes the single + * space at offset 9 as leading whitespace and matches {@code "beta"} + * starting at offset 10. The subtree's leading trivia therefore + * covers exactly {@code [9, 10)} — one space — NOT the two spaces + * at the input start. */ @Test - void parseRuleAt_flagOn_subtreeLeading_coversInputBeforeSpliceOffset() { + void parseRuleAt_flagOn_subtreeLeading_coversOnlySpliceGap() { var input = " alpha , beta "; - // 0123456789012345 - int offset = 10; + // index: 0123456789012345 + // The previous Item ('alpha') ends at offset 7, the ',' at 8. + // Splicing at offset 9 should attribute only the space at [9,10) + // to the spliced subtree's leading — NOT the leading " " at + // [0, 2) which belongs to the document root. + int spliceOffset = 9; var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) .unwrap(); - var partial = parserOn.parseRuleAt(Item.class, input, offset) + var partial = parserOn.parseRuleAt(Item.class, input, spliceOffset) .unwrap(); var subtree = partial.node(); - // Subtree span starts at the requested offset. + // The Item rule auto-skips %whitespace before matching, so the + // matched span starts after the single leading space. assertThat(subtree.span() .start() - .offset()).isEqualTo(offset); - - // Documented Lever B gap: subtree's leading trivia text equals - // input.substring(0, offset) projected through the %whitespace - // matcher — i.e. the leading run of whitespace from input - // offset 0, not the gap before the splice point. - // - // For input " alpha , beta " and the inner expression - // [ \t\n], scanWhitespace([0,10)) produces only the leading two - // spaces (offsets 0..2) — matching stops at 'a' (offset 2). - // The two spaces are emitted as separate single-char chunks - // because the inner expression is a single-char class. + .offset()) + .as("subtree span starts after the gap whitespace") + .isEqualTo(10); + + // Splice-aware leading: only the gap between spliceOffset (9) + // and span.start (10) — a single space, NOT the input prefix. var leadingText = subtree.leadingTrivia() .stream() .map(Trivia::text) .reduce("", (a, b) -> a + b); - assertThat(leadingText).as("leading covers input prefix") - .isEqualTo(" "); - // Two single-char chunks because inner expression is one char. - assertThat(subtree.leadingTrivia()).hasSize(2); + assertThat(leadingText).as("leading covers only [spliceOffset, span.start)") + .isEqualTo(" "); + assertThat(subtree.leadingTrivia()).hasSize(1); } @Test diff --git a/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassSpliceOffsetTest.java b/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassSpliceOffsetTest.java new file mode 100644 index 0000000..028ef58 --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/tree/TriviaPostPassSpliceOffsetTest.java @@ -0,0 +1,341 @@ +package org.pragmatica.peg.tree; + +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; +import org.pragmatica.peg.PegParser; +import org.pragmatica.peg.grammar.Grammar; +import org.pragmatica.peg.grammar.GrammarParser; +import org.pragmatica.peg.parser.Parser; + +import java.util.List; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/** + * Step 4 commit 2 (0.5.1) — exercises the splice-offset overload of + * {@link TriviaPostPass#assignTrivia(String, CstNode, Grammar, int)}. + * + *

The 4-arg overload constrains the leading-trivia scan window of the + * root subtree to {@code [leadingScanFrom, root.span.start)} so that + * partial-reparse subtrees produced by {@code parseRuleAt} carry only the + * trivia that physically separates them from their previous sibling — not + * the entire input prefix. + * + *

Validates four properties: + *

    + *
  1. Delegate parity — {@code leadingScanFrom = 0} produces output + * structurally identical to the 3-arg overload.
    2. + *
  2. Splice-window leading — non-zero {@code leadingScanFrom} + * restricts the root subtree's leading trivia to the splice gap.
    3. + *
  3. Empty-gap edge case — {@code leadingScanFrom == span.start} + * yields an empty leading trivia list.
    4. + *
  4. Out-of-range guard — negative or beyond-span values raise + * {@link IllegalArgumentException}.
    5. + *
+ */ +class TriviaPostPassSpliceOffsetTest { + + private static final String LIST_GRAMMAR = """ + List <- Item (',' Item)* + Item <- < [a-z]+ > + %whitespace <- [ \\t\\n]* + """; + + // ==================================================================== + // 1. Delegate parity: leadingScanFrom = 0 == 3-arg overload + // ==================================================================== + + @Nested + @DisplayName("Delegate parity (leadingScanFrom = 0)") + class DelegateParity { + + @Test + void zeroOffset_listInput_matchesThreeArgOverload() { + var input = "alpha, beta, gamma"; + var parser = PegParser.fromGrammar(LIST_GRAMMAR) + .unwrap(); + var grammar = parseGrammar(LIST_GRAMMAR); + var cst = parser.parseCst(input) + .unwrap(); + + var threeArg = TriviaPostPass.assignTrivia(input, cst, grammar); + var fourArg = TriviaPostPass.assignTrivia(input, cst, grammar, 0); + + assertThat(triviaShape(fourArg)).isEqualTo(triviaShape(threeArg)); + assertThat(reconstruct(fourArg)).isEqualTo(input); + assertThat(reconstruct(threeArg)).isEqualTo(input); + } + + @Test + void zeroOffset_withLeadingWhitespace_matchesThreeArgOverload() { + var input = " alpha"; + var parser = PegParser.fromGrammar(LIST_GRAMMAR) + .unwrap(); + var grammar = parseGrammar(LIST_GRAMMAR); + var cst = parser.parseCst(input) + .unwrap(); + + var threeArg = TriviaPostPass.assignTrivia(input, cst, grammar); + var fourArg = TriviaPostPass.assignTrivia(input, cst, grammar, 0); + + assertThat(triviaShape(fourArg)).isEqualTo(triviaShape(threeArg)); + } + } + + // ==================================================================== + // 2. Non-zero splice offset clamps leading trivia window + // ==================================================================== + + @Nested + @DisplayName("Non-zero splice offset clamps leading trivia") + class SpliceWindowLeading { + + @Test + void nonZeroOffset_leadingCoversOnlySpliceGap() { + // Full input " alpha , beta ", parsed by parseRuleAt at offset 9. + // Item rule auto-skips %whitespace and matches "beta" at offset 10. + var input = " alpha , beta "; + int spliceOffset = 9; + + var parserOn = parserWithPostPass(); + var partial = parserOn.parseRuleAt(Item.class, input, spliceOffset) + .unwrap(); + var subtree = partial.node(); + var grammar = parseGrammar(LIST_GRAMMAR); + + // Apply the splice-aware overload directly on the engine output + // (which is already post-passed by the engine with the same + // offset). Re-applying with the same offset is idempotent on + // root span coordinates. + var rebuilt = TriviaPostPass.assignTrivia(input, subtree, grammar, spliceOffset); + + assertThat(rebuilt.span() + .startOffset()) + .as("subtree span starts at first matched terminal") + .isEqualTo(10); + + var leading = triviaText(rebuilt.leadingTrivia()); + assertThat(leading).as("leading window is [%d, %d)", spliceOffset, 10) + .isEqualTo(" "); + } + + @Test + void nonZeroOffset_leadingDoesNotIncludeInputPrefix() { + // Same input, but apply the splice-aware overload on a subtree + // taken from the FULL parse (pretend it's a partial-reparse output). + var input = " alpha , beta "; + var parser = parserWithPostPass(); + var grammar = parseGrammar(LIST_GRAMMAR); + + // Find the second Item ('beta') in the full parse. + var fullCst = parser.parseCst(input) + .unwrap(); + CstNode betaNode = findItemAt(fullCst, 10); + assertThat(betaNode).isNotNull(); + + // Rebuild with leadingScanFrom = 9 (splice gap before beta). + var rebuilt = TriviaPostPass.assignTrivia(input, betaNode, grammar, 9); + assertThat(triviaText(rebuilt.leadingTrivia())).isEqualTo(" "); + + // For comparison, full-document scan attributes nothing — the + // node is mid-tree so [0, span.start) goes through the entire + // input prefix. The 3-arg overload would mis-attribute that. + var fullScan = TriviaPostPass.assignTrivia(input, betaNode, grammar); + assertThat(triviaText(fullScan.leadingTrivia())) + .as("full-doc scan over partial-reparse subtree wrongly captures input prefix") + .isNotEqualTo(triviaText(rebuilt.leadingTrivia())); + } + } + + // ==================================================================== + // 3. Edge case: leadingScanFrom == span.start (no leading trivia) + // ==================================================================== + + @Nested + @DisplayName("Edge cases") + class EdgeCases { + + @Test + void leadingScanFromEqualsSpanStart_leadingIsEmpty() { + var input = "alpha"; + var parser = parserWithPostPass(); + var grammar = parseGrammar(LIST_GRAMMAR); + var cst = parser.parseCst(input) + .unwrap(); + + int spanStart = cst.span() + .startOffset(); + var rebuilt = TriviaPostPass.assignTrivia(input, cst, grammar, spanStart); + + assertThat(rebuilt.leadingTrivia()).isEmpty(); + } + + @Test + void negativeLeadingScanFrom_throwsIllegalArgument() { + var input = "alpha"; + var parser = parserWithPostPass(); + var grammar = parseGrammar(LIST_GRAMMAR); + var cst = parser.parseCst(input) + .unwrap(); + + assertThatThrownBy(() -> TriviaPostPass.assignTrivia(input, cst, grammar, -1)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("leadingScanFrom"); + } + + @Test + void leadingScanFromBeyondSpanStart_throwsIllegalArgument() { + var input = " alpha"; + var parser = parserWithPostPass(); + var grammar = parseGrammar(LIST_GRAMMAR); + var cst = parser.parseCst(input) + .unwrap(); + + int spanStart = cst.span() + .startOffset(); + assertThatThrownBy(() -> TriviaPostPass.assignTrivia( + input, + cst, + grammar, + spanStart + 1)) + .isInstanceOf(IllegalArgumentException.class) + .hasMessageContaining("leadingScanFrom"); + } + } + + // ==================================================================== + // helpers + // ==================================================================== + + record Item() implements org.pragmatica.peg.action.RuleId {} + + private static Parser parserWithPostPass() { + var off = org.pragmatica.peg.parser.ParserConfig.parserConfig( + true, + org.pragmatica.peg.error.RecoveryStrategy.BASIC, + true); + var on = new org.pragmatica.peg.parser.ParserConfig( + off.packratEnabled(), + off.recoveryStrategy(), + off.captureTrivia(), + off.fastTrackFailure(), + off.literalFailureCache(), + off.charClassFailureCache(), + off.bulkAdvanceLiteral(), + off.skipWhitespaceFastPath(), + off.reuseEndLocation(), + off.choiceDispatch(), + off.markResetChildren(), + off.inlineLocations(), + off.selectivePackrat(), + off.packratSkipRules(), + off.mutableParseResult(), + off.tokenFastPath(), + true); + return PegParser.fromGrammar(LIST_GRAMMAR, on) + .unwrap(); + } + + private static CstNode findItemAt(CstNode node, int targetStart) { + if ("Item".equals(node.rule()) && node.span() + .startOffset() == targetStart) { + return node; + } + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) { + var found = findItemAt(c, targetStart); + if (found != null) { + return found; + } + } + } + return null; + } + + private static Grammar parseGrammar(String text) { + return GrammarParser.parse(text) + .unwrap(); + } + + private static String triviaText(List list) { + var sb = new StringBuilder(); + for (var t : list) { + sb.append(t.text()); + } + return sb.toString(); + } + + private static String reconstruct(CstNode root) { + var sb = new StringBuilder(); + emitLeading(sb, root); + emitBody(sb, root); + emitTrailing(sb, root); + return sb.toString(); + } + + private static void emitLeading(StringBuilder sb, CstNode node) { + for (var t : node.leadingTrivia()) { + sb.append(t.text()); + } + } + + private static void emitTrailing(StringBuilder sb, CstNode node) { + for (var t : node.trailingTrivia()) { + sb.append(t.text()); + } + } + + private static void emitBody(StringBuilder sb, CstNode node) { + switch (node) { + case CstNode.Terminal t -> sb.append(t.text()); + case CstNode.Token tk -> sb.append(tk.text()); + case CstNode.Error e -> sb.append(e.skippedText()); + case CstNode.NonTerminal nt -> { + for (var c : nt.children()) { + emitLeading(sb, c); + emitBody(sb, c); + if (c instanceof CstNode.NonTerminal) { + emitTrailing(sb, c); + } + } + } + } + } + + private static String triviaShape(CstNode node) { + var sb = new StringBuilder(); + appendShape(sb, node); + return sb.toString(); + } + + private static void appendShape(StringBuilder sb, CstNode node) { + sb.append(kind(node)) + .append('@') + .append(node.span() + .startOffset()) + .append('-') + .append(node.span() + .endOffset()) + .append(" L=") + .append(triviaText(node.leadingTrivia())) + .append(" T=") + .append(triviaText(node.trailingTrivia())) + .append('\n'); + if (node instanceof CstNode.NonTerminal nt) { + for (var c : nt.children()) { + appendShape(sb, c); + } + } + } + + private static String kind(CstNode node) { + return switch (node) { + case CstNode.NonTerminal _ -> "NT"; + case CstNode.Terminal _ -> "T"; + case CstNode.Token _ -> "Tk"; + case CstNode.Error _ -> "E"; + }; + } +} From 605327b36d7cf900947b5278a87e73ec78824d8d Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 07:25:32 +0200 Subject: [PATCH 07/26] perf: ParsingContext buffer ops no-op under triviaPostPass flag (Step 4 commit 3) --- .../pragmatica/peg/parser/ParsingContext.java | 29 +++++++++ .../peg/perf/TriviaPostPassFlagTest.java | 64 +++++++++++++++++++ 2 files changed, 93 insertions(+) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java index a984978..33d5678 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java @@ -487,8 +487,16 @@ public void exitWhitespaceSkip() { * Append captured trivia to the pending-leading buffer. Called at sites * between sibling sequence elements (sequence, zero-or-more, one-or-more, * repetition) so the following sibling's leaf or rule can claim it. + * + *

Step 4 commit 3: when {@code config.triviaPostPass()} is true, the + * post-pass overrides all leading/trailing attribution after the main + * parse, so any work done by this buffer is discarded. Early-out to + * eliminate the wasted CPU. */ public void appendPendingLeadingTrivia(List captured) { + if (config.triviaPostPass()) { + return; + } if (!captured.isEmpty()) { pendingLeadingTrivia.addAll(captured); } @@ -499,8 +507,15 @@ public void appendPendingLeadingTrivia(List captured) { * CstNode that owns {@code leadingTrivia} — leaf terminals, token * boundaries, or rule wrappers. Returns the previously-pending list; the * buffer is reset to empty. + * + *

Step 4 commit 3: under {@code config.triviaPostPass()=true}, returns + * an empty list without touching state — the post-pass will reattribute + * trivia regardless. */ public List takePendingLeadingTrivia() { + if (config.triviaPostPass()) { + return List.of(); + } if (pendingLeadingTrivia.isEmpty()) { return List.of(); } @@ -522,8 +537,15 @@ public List takePendingLeadingTrivia() { * branch later fails, restoring just the size cannot recover the consumed * items. The full-list snapshot guarantees byte-identical buffer state * regardless of intermediate drains.

+ * + *

Step 4 commit 3: under {@code config.triviaPostPass()=true}, returns + * an empty sentinel list — the matching {@code restorePendingLeadingTrivia} + * is also a no-op, so the snapshot value is irrelevant. */ public List savePendingLeadingTrivia() { + if (config.triviaPostPass()) { + return List.of(); + } return List.copyOf(pendingLeadingTrivia); } @@ -532,8 +554,15 @@ public List savePendingLeadingTrivia() { * Replaces all current contents with those from {@code snapshot}, * recovering items that may have been drained by inner operations * inside a now-failed branch. + * + *

Step 4 commit 3: under {@code config.triviaPostPass()=true}, no-op — + * the buffer is never written under the flag, so there is nothing to + * restore. */ public void restorePendingLeadingTrivia(List snapshot) { + if (config.triviaPostPass()) { + return; + } pendingLeadingTrivia.clear(); pendingLeadingTrivia.addAll(snapshot); } diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java index ca88a6a..6eaedaa 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java @@ -312,6 +312,70 @@ void parseRuleAt_flagOnVsOff_subtreeReconstructionIsConsistent() { } } + // =================================================================== + // 5. Step 4 commit 3 — buffer no-op under flag-ON + // =================================================================== + + /** + * Step 4 commit 3 validation: under flag-ON, the engine's pending-leading + * trivia buffer ops in {@link org.pragmatica.peg.parser.ParsingContext} + * (append/take/save/restore) early-out — the buffer never accumulates + * state. The post-pass must still produce correct leading/trailing + * attribution because it derives the gap entirely from the raw input plus + * span offsets, independent of the now-empty intermediate buffer. + * + *

Indirect validation: if the buffer machinery were the source of the + * post-pass's correct attribution (it isn't), the early-out would break + * round-trip. The test asserts the same round-trip + leading-trivia + * properties as commit 2 with the buffer paths effectively disabled. + */ + @Nested + @DisplayName("Buffer no-op under flag-ON (Step 4 commit 3)") + class BufferNoOpUnderFlagOn { + + @Test + void flagOn_simpleNumber_postPassPopulatesLeadingDespiteBufferNoOp() { + var input = " 42 "; + var parser = PegParser.fromGrammar(SIMPLE_GRAMMAR, configOn()) + .unwrap(); + var cst = parser.parseCst(input) + .unwrap(); + + // Round-trip = bytes preserved => post-pass attributed correctly + // even though the engine's pending-leading buffer was a no-op. + assertThat(reconstruct(cst)).isEqualTo(input); + // Total trivia text preserved. + assertThat(allTriviaText(cst)).isEqualTo(" "); + } + + @Test + void flagOn_listInput_postPassAttributesAllTriviaWithEmptyBuffer() { + var input = " alpha,\n beta,\n gamma "; + var parser = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + .unwrap(); + var cst = parser.parseCst(input) + .unwrap(); + + assertThat(reconstruct(cst)).isEqualTo(input); + // All whitespace is attributed somewhere; commas are literal + // terminals consumed by Tail and not part of trivia. + assertThat(allTriviaText(cst)).isEqualTo(" \n \n "); + } + + @Test + void flagOn_commentedNumber_postPassClassifiesCommentsWithEmptyBuffer() { + var input = "// preamble\n42 // trailer\n"; + var parser = PegParser.fromGrammar(COMMENT_GRAMMAR, configOn()) + .unwrap(); + var cst = parser.parseCst(input) + .unwrap(); + + assertThat(reconstruct(cst)).isEqualTo(input); + assertThat(allTriviaText(cst)).contains("// preamble") + .contains("// trailer"); + } + } + // =================================================================== // helpers (mirrors policy from TriviaPostPassTest) // =================================================================== From dcd146fb0a27c2175702a5d2012baf9243c646d5 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 07:47:46 +0200 Subject: [PATCH 08/26] feat: ParserGenerator emits embedded TriviaPostPass under flag-ON (Step 4 commit 4) --- .../peg/generator/ParserGenerator.java | 629 +++++++++++++++++- .../GeneratedParserTriviaPostPassTest.java | 240 +++++++ 2 files changed, 863 insertions(+), 6 deletions(-) create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/generator/GeneratedParserTriviaPostPassTest.java diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 5a81ac8..d0eace6 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -335,6 +335,9 @@ public String generateCst() { generateCstParseMethods(sb); generateCstRuleMethods(sb); generateCstHelperMethods(sb); + if (config.triviaPostPass()) { + generateCstTriviaPostPass(sb); + } generateClassEnd(sb); return sb.toString(); } @@ -2700,6 +2703,9 @@ private void generateCstParseMethods(StringBuilder sb) { var expectedExpr = config.mutableParseResult() ? "((furthestExpected != null && !furthestExpected.isEmpty()) ? furthestExpected : (result.expected != null ? result.expected : \"valid input\"))" : "furthestExpected.filter(s -> !s.isEmpty()).or(result.expected.or(\"valid input\"))"; + var postPassParseLine = config.triviaPostPass() + ? " rootNode = TriviaPostPass.assignTrivia(input, rootNode, 0);\n" + : ""; sb.append(""" // === Public Parse Methods === @@ -2718,7 +2724,7 @@ public Result parse(String input) { } // Attach trailing trivia to root node var rootNode = attachTrailingTrivia(%s, trailingTrivia); - return Result.success(rootNode); + %s return Result.success(rootNode); } /** @@ -2741,8 +2747,19 @@ private AstNode toAst(CstNode cst) { }; } - """.formatted(sanitizedName, furthestFailureOrLoc, expectedExpr, furthestFailureOrLoc, resultNodeUnwrapExpr)); + """.formatted(sanitizedName, + furthestFailureOrLoc, + expectedExpr, + furthestFailureOrLoc, + resultNodeUnwrapExpr, + postPassParseLine)); if (errorReporting == ErrorReporting.ADVANCED) { + var postPassDiagBranchLine = config.triviaPostPass() + ? " rootNode = TriviaPostPass.assignTrivia(input, rootNode, 0);\n" + : ""; + var postPassDiagSuccessLine = config.triviaPostPass() + ? " rootNode = TriviaPostPass.assignTrivia(input, rootNode, 0);\n" + : ""; sb.append(""" /** * Parse with advanced error recovery and Rust-style diagnostics. @@ -2781,11 +2798,11 @@ public ParseResultWithDiagnostics parseWithDiagnostics(String input) { // Attach error node to result var rootNode = attachTrailingTrivia(%s, trailingTrivia); - return ParseResultWithDiagnostics.withErrors(Option.some(rootNode), diagnostics, input); + %s return ParseResultWithDiagnostics.withErrors(Option.some(rootNode), diagnostics, input); } var rootNode = attachTrailingTrivia(%s, trailingTrivia); - if (diagnostics.isEmpty()) { + %s if (diagnostics.isEmpty()) { return ParseResultWithDiagnostics.success(rootNode, input); } return ParseResultWithDiagnostics.withErrors(Option.some(rootNode), diagnostics, input); @@ -2796,7 +2813,9 @@ public ParseResultWithDiagnostics parseWithDiagnostics(String input) { expectedExpr, furthestFailureOrLoc, resultNodeUnwrapExpr, - resultNodeUnwrapExpr)); + postPassDiagBranchLine, + resultNodeUnwrapExpr, + postPassDiagSuccessLine)); } generateCstParseRuleAt(sb); } @@ -2887,7 +2906,7 @@ public Result parseRuleAt( var expected = __EXPECTED_EXPR__; return Result.failure(new ParseError(errorLoc, "expected " + expected)); } - return Result.success(new PartialParse(__RESULT_NODE__, pos)); + return Result.success(new PartialParse(__POST_PASS_WRAP__, pos)); } /** @@ -2911,9 +2930,13 @@ private void seekTo(int offset) { } """); + var postPassWrapExpr = config.triviaPostPass() + ? "TriviaPostPass.assignTrivia(input, " + resultNodeUnwrapExpr + ", offset)" + : resultNodeUnwrapExpr; sb.append(ruleAtBlock.toString() .replace("__FURTHEST_FAILURE_OR_LOC__", furthestFailureOrLoc) .replace("__EXPECTED_EXPR__", expectedExpr) + .replace("__POST_PASS_WRAP__", postPassWrapExpr) .replace("__RESULT_NODE__", resultNodeUnwrapExpr)); } @@ -7011,6 +7034,600 @@ CstParseResult asRegularFailure() { } } + // === Step 4 commit 4: embedded TriviaPostPass (flag-ON only) === + /** + * Emit a {@code private static final class TriviaPostPass} embedded in the + * generated parser. Provides {@code assignTrivia(input, cst, leadingScanFrom)} + * with the same semantics as + * {@link org.pragmatica.peg.tree.TriviaPostPass#assignTrivia(String, CstNode, org.pragmatica.peg.grammar.Grammar, int)}. + * + *

The embedded class is fully self-contained: it depends only on the + * generated parser's own embedded {@code CstNode}, {@code Trivia}, and + * {@code SourceSpan} types (which themselves depend only on + * {@code pragmatica-lite:core}). It never imports any peglib runtime class, + * preserving the standalone-parser invariant. + * + *

The whitespace matcher is generated from the grammar's + * {@code %whitespace} expression at generation time, walking the expression + * tree and emitting Java code per Expression subtype. This avoids needing + * the runtime {@code Grammar} object at parse time. + */ + private void generateCstTriviaPostPass(StringBuilder sb) { + var hasWhitespace = grammar.whitespace() + .isPresent(); + sb.append(" // === Embedded TriviaPostPass (Step 4 commit 4 — flag-ON) ===\n"); + sb.append(" /**\n"); + sb.append(" * Re-derive leading/trailing trivia attribution from\n"); + sb.append(" * (input, span, %whitespace) after the engine returns its CST.\n"); + sb.append(" * Mirrors org.pragmatica.peg.tree.TriviaPostPass at the runtime.\n"); + sb.append(" */\n"); + sb.append(" private static final class TriviaPostPass {\n"); + sb.append(" private TriviaPostPass() {}\n\n"); + sb.append(" static CstNode assignTrivia(String input, CstNode cst, int leadingScanFrom) {\n"); + sb.append(" int rootStart = cst.span().startOffset();\n"); + sb.append(" if (leadingScanFrom < 0 || leadingScanFrom > rootStart) {\n"); + sb.append(" // Generator span semantics: rootStart may be 0 (span includes\n"); + sb.append(" // leading whitespace). Clamp leadingScanFrom rather than reject.\n"); + sb.append(" if (leadingScanFrom > 0 && rootStart == 0) {\n"); + sb.append(" leadingScanFrom = 0;\n"); + sb.append(" } else {\n"); + sb.append(" throw new IllegalArgumentException(\n"); + sb.append(" \"leadingScanFrom \" + leadingScanFrom + \" out of range [0, \" + rootStart + \"]\");\n"); + sb.append(" }\n"); + sb.append(" }\n"); + sb.append(" return rebuildRoot(input, cst, leadingScanFrom);\n"); + sb.append(" }\n\n"); + sb.append(" static List scanWhitespace(String input, int from, int to) {\n"); + sb.append(" if (from >= to) return List.of();\n"); + if (!hasWhitespace) { + sb.append(" return List.of();\n"); + }else { + sb.append(" var trivia = new ArrayList();\n"); + sb.append(" int pos = from;\n"); + sb.append(" while (pos < to) {\n"); + sb.append(" int matched = matchWsInner(input, pos, to);\n"); + sb.append(" if (matched < 0 || matched == pos) break;\n"); + sb.append(" var text = input.substring(pos, matched);\n"); + sb.append(" var span = computeSpan(input, pos, matched);\n"); + sb.append(" trivia.add(classify(span, text));\n"); + sb.append(" pos = matched;\n"); + sb.append(" }\n"); + sb.append(" return List.copyOf(trivia);\n"); + } + sb.append(" }\n\n"); + // === Tree rebuild === + // Span-semantics adapter: the generated parser's rule wrappers may use + // a span whose startOffset INCLUDES leading whitespace (rule entry + // captures startOffset BEFORE skipWhitespace, then the leading trivia + // is attached as the wrapper's leadingTrivia). The runtime engine, by + // contrast, captures startOffset AFTER skipWhitespace. Detect this by + // measuring whether the root's existing leadingTrivia text overlaps + // [rootStart, rootStart + leadingLen). If so, use rootStart as the + // "leading-end" boundary AND leadingScanFrom + (rootStart - leadingScanFrom) + // = rootStart for the leading scan. Otherwise (runtime semantics), + // scan up to rootStart as the runtime post-pass does. + sb.append(" private static CstNode rebuildRoot(String input, CstNode root, int leadingScanFrom) {\n"); + sb.append(" int rootStart = root.span().startOffset();\n"); + sb.append(" int rootEnd = root.span().endOffset();\n"); + sb.append(" int leadingTextLen = totalTriviaLength(root.leadingTrivia());\n"); + sb.append(" // Generator semantics: span includes leading WS. The actual content\n"); + sb.append(" // begins at (rootStart + leadingTextLen), and the leading WS lives\n"); + sb.append(" // in [rootStart, rootStart + leadingTextLen).\n"); + sb.append(" // Runtime semantics: span excludes leading WS. leadingTextLen is in\n"); + sb.append(" // the gap [leadingScanFrom, rootStart). We distinguish by checking\n"); + sb.append(" // whether scanning the pre-rootStart gap yields the leading.\n"); + sb.append(" int leadingScanEnd;\n"); + sb.append(" if (leadingTextLen > 0 && rootStart + leadingTextLen <= input.length()) {\n"); + sb.append(" // Hypothesis: generator semantics — leading occupies [rootStart, rootStart + leadingTextLen).\n"); + sb.append(" // Verify by re-scanning [rootStart, rootStart + leadingTextLen) and checking it consumes exactly that range.\n"); + sb.append(" int probeEnd = rootStart + leadingTextLen;\n"); + sb.append(" var probe = scanWhitespace(input, rootStart, probeEnd);\n"); + sb.append(" if (totalTriviaLength(probe) == leadingTextLen) {\n"); + sb.append(" leadingScanEnd = probeEnd;\n"); + sb.append(" } else {\n"); + sb.append(" leadingScanEnd = rootStart;\n"); + sb.append(" }\n"); + sb.append(" } else {\n"); + sb.append(" leadingScanEnd = rootStart;\n"); + sb.append(" }\n"); + sb.append(" var leading = scanWhitespace(input, leadingScanFrom, leadingScanEnd);\n"); + sb.append(" var trailingExternal = scanWhitespace(input, rootEnd, input.length());\n"); + sb.append(" return rebuildSelf(input, root, leading, trailingExternal);\n"); + sb.append(" }\n\n"); + sb.append(" private static int totalTriviaLength(List trivia) {\n"); + sb.append(" int total = 0;\n"); + sb.append(" for (var t : trivia) total += t.text().length();\n"); + sb.append(" return total;\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode rebuildSelf(String input, CstNode node,\n"); + sb.append(" List leading, List extraTrailing) {\n"); + sb.append(" return switch (node) {\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, leading, extraTrailing);\n"); + sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing);\n"); + sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing);\n"); + if (errorReporting == ErrorReporting.ADVANCED) { + sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, extraTrailing);\n"); + } + sb.append(" };\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode rebuildChild(String input, CstNode child, int prevEnd) {\n"); + sb.append(" int childStart = child.span().startOffset();\n"); + sb.append(" var leading = scanWhitespace(input, prevEnd, childStart);\n"); + sb.append(" return switch (child) {\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, leading, List.of());\n"); + sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of());\n"); + sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of());\n"); + if (errorReporting == ErrorReporting.ADVANCED) { + sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, List.of());\n"); + } + sb.append(" };\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode.NonTerminal rebuildNonTerminal(String input, CstNode.NonTerminal nt,\n"); + sb.append(" List leading, List extraTrailing) {\n"); + sb.append(" int spanStart = nt.span().startOffset();\n"); + sb.append(" int spanEnd = nt.span().endOffset();\n"); + sb.append(" var newChildren = new ArrayList(nt.children().size());\n"); + sb.append(" int cursor = spanStart;\n"); + sb.append(" for (var c : nt.children()) {\n"); + sb.append(" var rebuilt = rebuildChild(input, c, cursor);\n"); + sb.append(" newChildren.add(rebuilt);\n"); + sb.append(" cursor = c.span().endOffset();\n"); + sb.append(" }\n"); + sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd);\n"); + sb.append(" var trailing = combine(internalTrailing, extraTrailing);\n"); + sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, trailing);\n"); + sb.append(" }\n\n"); + sb.append(" private static List combine(List a, List b) {\n"); + sb.append(" if (a.isEmpty()) return b;\n"); + sb.append(" if (b.isEmpty()) return a;\n"); + sb.append(" var combined = new ArrayList(a.size() + b.size());\n"); + sb.append(" combined.addAll(a);\n"); + sb.append(" combined.addAll(b);\n"); + sb.append(" return List.copyOf(combined);\n"); + sb.append(" }\n\n"); + sb.append(" private static Trivia classify(SourceSpan span, String text) {\n"); + sb.append(" if (text.startsWith(\"//\")) {\n"); + sb.append(" return new Trivia.LineComment(span, text);\n"); + sb.append(" } else if (text.startsWith(\"/*\")) {\n"); + sb.append(" return new Trivia.BlockComment(span, text);\n"); + sb.append(" } else {\n"); + sb.append(" return new Trivia.Whitespace(span, text);\n"); + sb.append(" }\n"); + sb.append(" }\n\n"); + // Static copy of matchesPattern for char-class matching inside the post-pass. + // Mirrors the instance-method MATCHES_PATTERN_METHOD emitted elsewhere; kept + // as a separate static copy because the embedded class cannot reach instance + // methods of the enclosing parser. + sb.append(" private static boolean matchesPattern(char c, String pattern, boolean caseInsensitive) {\n"); + sb.append(" char testChar = caseInsensitive ? Character.toLowerCase(c) : c;\n"); + sb.append(" int i = 0;\n"); + sb.append(" while (i < pattern.length()) {\n"); + sb.append(" char start = pattern.charAt(i);\n"); + sb.append(" if (start == '\\\\' && i + 1 < pattern.length()) {\n"); + sb.append(" char escaped = pattern.charAt(i + 1);\n"); + sb.append(" int consumed = 2;\n"); + sb.append(" char expected = switch (escaped) {\n"); + sb.append(" case 'n' -> '\\n';\n"); + sb.append(" case 'r' -> '\\r';\n"); + sb.append(" case 't' -> '\\t';\n"); + sb.append(" case '\\\\' -> '\\\\';\n"); + sb.append(" case ']' -> ']';\n"); + sb.append(" case '-' -> '-';\n"); + sb.append(" case 'x' -> {\n"); + sb.append(" if (i + 4 <= pattern.length()) {\n"); + sb.append(" try {\n"); + sb.append(" var hex = pattern.substring(i + 2, i + 4);\n"); + sb.append(" consumed = 4;\n"); + sb.append(" yield (char) Integer.parseInt(hex, 16);\n"); + sb.append(" } catch (NumberFormatException e) { yield 'x'; }\n"); + sb.append(" }\n"); + sb.append(" yield 'x';\n"); + sb.append(" }\n"); + sb.append(" case 'u' -> {\n"); + sb.append(" if (i + 6 <= pattern.length()) {\n"); + sb.append(" try {\n"); + sb.append(" var hex = pattern.substring(i + 2, i + 6);\n"); + sb.append(" consumed = 6;\n"); + sb.append(" yield (char) Integer.parseInt(hex, 16);\n"); + sb.append(" } catch (NumberFormatException e) { yield 'u'; }\n"); + sb.append(" }\n"); + sb.append(" yield 'u';\n"); + sb.append(" }\n"); + sb.append(" default -> escaped;\n"); + sb.append(" };\n"); + sb.append(" if (caseInsensitive) expected = Character.toLowerCase(expected);\n"); + sb.append(" if (testChar == expected) return true;\n"); + sb.append(" i += consumed;\n"); + sb.append(" continue;\n"); + sb.append(" }\n"); + sb.append(" if (i + 2 < pattern.length() && pattern.charAt(i + 1) == '-') {\n"); + sb.append(" char end = pattern.charAt(i + 2);\n"); + sb.append(" if (caseInsensitive) {\n"); + sb.append(" start = Character.toLowerCase(start);\n"); + sb.append(" end = Character.toLowerCase(end);\n"); + sb.append(" }\n"); + sb.append(" if (testChar >= start && testChar <= end) return true;\n"); + sb.append(" i += 3;\n"); + sb.append(" } else {\n"); + sb.append(" if (caseInsensitive) start = Character.toLowerCase(start);\n"); + sb.append(" if (testChar == start) return true;\n"); + sb.append(" i++;\n"); + sb.append(" }\n"); + sb.append(" }\n"); + sb.append(" return false;\n"); + sb.append(" }\n\n"); + sb.append(" private static SourceSpan computeSpan(String input, int from, int to) {\n"); + sb.append(" int line = 1, col = 1;\n"); + sb.append(" for (int i = 0; i < from; i++) {\n"); + sb.append(" if (input.charAt(i) == '\\n') { line++; col = 1; } else { col++; }\n"); + sb.append(" }\n"); + sb.append(" int startLine = line, startCol = col;\n"); + sb.append(" for (int i = from; i < to; i++) {\n"); + sb.append(" if (input.charAt(i) == '\\n') { line++; col = 1; } else { col++; }\n"); + sb.append(" }\n"); + sb.append(" return new SourceSpan(startLine, startCol, from, line, col, to);\n"); + sb.append(" }\n\n"); + // Emit per-expression matcher methods. + if (hasWhitespace) { + var wsExpr = grammar.whitespace() + .unwrap(); + var inner = ExpressionShape.extractInnerExpression(wsExpr); + var ctx = new TriviaPostPassEmitContext(); + // Reserve method id 0 for the entry point. + int entryId = ctx.allocate(inner); + // Emit the entry method as `matchWsInner`; rename later via direct emission. + sb.append(" private static int matchWsInner(String input, int pos, int limit) {\n"); + sb.append(" return ") + .append(ctx.callName(entryId)) + .append("(input, pos, limit);\n"); + sb.append(" }\n\n"); + // Emit all queued methods. + while (ctx.hasPending()) { + var entry = ctx.nextPending(); + emitTriviaPostPassMatcher(sb, entry.expr(), entry.id(), ctx); + } + } + sb.append(" }\n\n"); + } + + /** + * Per-emission state for the embedded TriviaPostPass matcher generation. + * Allocates unique method ids per expression, deduplicating identical + * Reference targets and queueing each newly seen expression for emission. + */ + private static final class TriviaPostPassEmitContext { + private final java.util.IdentityHashMap idByExpr = new java.util.IdentityHashMap<>(); + private final java.util.HashMap idByRule = new java.util.HashMap<>(); + private final java.util.ArrayDeque queue = new java.util.ArrayDeque<>(); + private int nextId = 0; + + int allocate(Expression expr) { + var existing = idByExpr.get(expr); + if (existing != null) return existing; + int id = nextId++ ; + idByExpr.put(expr, id); + queue.add(new TriviaPostPassEntry(id, expr)); + return id; + } + + int allocateForRule(String ruleName, Expression body) { + var existing = idByRule.get(ruleName); + if (existing != null) return existing; + int id = allocate(body); + idByRule.put(ruleName, id); + return id; + } + + boolean hasPending() { + return !queue.isEmpty(); + } + + TriviaPostPassEntry nextPending() { + return queue.removeFirst(); + } + + String callName(int id) { + return "matchWs_" + id; + } + } + + private record TriviaPostPassEntry(int id, Expression expr) {} + + /** + * Emit a single matcher method for {@code expr} into {@code sb}, naming it + * {@code matchWs_(String input, int pos, int limit)}. Returns the next + * absolute offset on a successful match or {@code -1} on failure. Sub- + * expressions are queued via {@link TriviaPostPassEmitContext#allocate} and + * emitted in subsequent iterations. + */ + private void emitTriviaPostPassMatcher(StringBuilder sb, Expression expr, int id, TriviaPostPassEmitContext ctx) { + sb.append(" private static int ") + .append(ctx.callName(id)) + .append("(String input, int pos, int limit) {\n"); + emitTriviaPostPassMatchBody(sb, expr, ctx, " "); + sb.append(" }\n\n"); + } + + /** + * Emit the body of a matcher method for {@code expr}. The body must + * {@code return} an int value (next position or -1) on every control path. + */ + private void emitTriviaPostPassMatchBody(StringBuilder sb, + Expression expr, + TriviaPostPassEmitContext ctx, + String indent) { + switch (expr) { + case Expression.Literal lit -> emitTppLiteral(sb, lit, indent); + case Expression.CharClass cc -> emitTppCharClass(sb, cc, indent); + case Expression.Any ignored -> { + sb.append(indent) + .append("return pos < limit ? pos + 1 : -1;\n"); + } + case Expression.Reference ref -> emitTppReference(sb, ref, ctx, indent); + case Expression.Sequence seq -> emitTppSequence(sb, seq, ctx, indent); + case Expression.Choice ch -> emitTppChoice(sb, ch, ctx, indent); + case Expression.ZeroOrMore zom -> emitTppZeroOrMore(sb, zom.expression(), ctx, indent); + case Expression.OneOrMore oom -> emitTppOneOrMore(sb, oom.expression(), ctx, indent); + case Expression.Optional opt -> emitTppOptional(sb, opt.expression(), ctx, indent); + case Expression.Repetition rep -> emitTppRepetition(sb, rep, ctx, indent); + case Expression.And and -> { + int sub = ctx.allocate(and.expression()); + sb.append(indent) + .append("return ") + .append(ctx.callName(sub)) + .append("(input, pos, limit) >= 0 ? pos : -1;\n"); + } + case Expression.Not not -> { + int sub = ctx.allocate(not.expression()); + sb.append(indent) + .append("return ") + .append(ctx.callName(sub)) + .append("(input, pos, limit) < 0 ? pos : -1;\n"); + } + case Expression.Group g -> emitTriviaPostPassMatchBody(sb, g.expression(), ctx, indent); + case Expression.TokenBoundary tb -> emitTriviaPostPassMatchBody(sb, tb.expression(), ctx, indent); + case Expression.Ignore ig -> emitTriviaPostPassMatchBody(sb, ig.expression(), ctx, indent); + case Expression.Capture cap -> emitTriviaPostPassMatchBody(sb, cap.expression(), ctx, indent); + case Expression.CaptureScope cs -> emitTriviaPostPassMatchBody(sb, cs.expression(), ctx, indent); + case Expression.BackReference ignored -> sb.append(indent) + .append("return -1;\n"); + case Expression.Dictionary ignored -> sb.append(indent) + .append("return -1;\n"); + case Expression.Cut ignored -> sb.append(indent) + .append("return pos;\n"); + } + } + + private void emitTppLiteral(StringBuilder sb, Expression.Literal lit, String indent) { + var literalStr = lit.text(); + sb.append(indent) + .append("String text = ") + .append(quoteJavaString(literalStr)) + .append(";\n"); + sb.append(indent) + .append("int len = text.length();\n"); + sb.append(indent) + .append("if (pos + len > limit) return -1;\n"); + if (lit.caseInsensitive()) { + sb.append(indent) + .append("for (int i = 0; i < len; i++) {\n"); + sb.append(indent) + .append(" if (Character.toLowerCase(input.charAt(pos + i)) != Character.toLowerCase(text.charAt(i))) return -1;\n"); + sb.append(indent) + .append("}\n"); + }else { + sb.append(indent) + .append("for (int i = 0; i < len; i++) {\n"); + sb.append(indent) + .append(" if (input.charAt(pos + i) != text.charAt(i)) return -1;\n"); + sb.append(indent) + .append("}\n"); + } + sb.append(indent) + .append("return pos + len;\n"); + } + + private void emitTppCharClass(StringBuilder sb, Expression.CharClass cc, String indent) { + sb.append(indent) + .append("if (pos >= limit) return -1;\n"); + sb.append(indent) + .append("char c = input.charAt(pos);\n"); + sb.append(indent) + .append("boolean inClass = matchesPattern(c, ") + .append(quoteJavaString(cc.pattern())) + .append(", ") + .append(cc.caseInsensitive()) + .append(");\n"); + sb.append(indent) + .append("boolean ok = ") + .append(cc.negated() + ? "!inClass" + : "inClass") + .append(";\n"); + sb.append(indent) + .append("return ok ? pos + 1 : -1;\n"); + } + + private void emitTppReference(StringBuilder sb, + Expression.Reference ref, + TriviaPostPassEmitContext ctx, + String indent) { + var ruleOpt = grammar.rule(ref.ruleName()); + if (ruleOpt.isEmpty()) { + sb.append(indent) + .append("return -1;\n"); + return; + } + int sub = ctx.allocateForRule(ref.ruleName(), + ruleOpt.unwrap() + .expression()); + sb.append(indent) + .append("return ") + .append(ctx.callName(sub)) + .append("(input, pos, limit);\n"); + } + + private void emitTppSequence(StringBuilder sb, + Expression.Sequence seq, + TriviaPostPassEmitContext ctx, + String indent) { + sb.append(indent) + .append("int cursor = pos;\n"); + for (var elt : seq.elements()) { + int sub = ctx.allocate(elt); + sb.append(indent) + .append("cursor = ") + .append(ctx.callName(sub)) + .append("(input, cursor, limit);\n"); + sb.append(indent) + .append("if (cursor < 0) return -1;\n"); + } + sb.append(indent) + .append("return cursor;\n"); + } + + private void emitTppChoice(StringBuilder sb, Expression.Choice ch, TriviaPostPassEmitContext ctx, String indent) { + for (var alt : ch.alternatives()) { + int sub = ctx.allocate(alt); + sb.append(indent) + .append("{\n"); + sb.append(indent) + .append(" int r = ") + .append(ctx.callName(sub)) + .append("(input, pos, limit);\n"); + sb.append(indent) + .append(" if (r >= 0) return r;\n"); + sb.append(indent) + .append("}\n"); + } + sb.append(indent) + .append("return -1;\n"); + } + + private void emitTppZeroOrMore(StringBuilder sb, Expression body, TriviaPostPassEmitContext ctx, String indent) { + int sub = ctx.allocate(body); + sb.append(indent) + .append("int cursor = pos;\n"); + sb.append(indent) + .append("while (true) {\n"); + sb.append(indent) + .append(" int next = ") + .append(ctx.callName(sub)) + .append("(input, cursor, limit);\n"); + sb.append(indent) + .append(" if (next < 0 || next == cursor) break;\n"); + sb.append(indent) + .append(" cursor = next;\n"); + sb.append(indent) + .append("}\n"); + sb.append(indent) + .append("return cursor;\n"); + } + + private void emitTppOneOrMore(StringBuilder sb, Expression body, TriviaPostPassEmitContext ctx, String indent) { + int sub = ctx.allocate(body); + sb.append(indent) + .append("int first = ") + .append(ctx.callName(sub)) + .append("(input, pos, limit);\n"); + sb.append(indent) + .append("if (first < 0 || first == pos) return -1;\n"); + sb.append(indent) + .append("int cursor = first;\n"); + sb.append(indent) + .append("while (true) {\n"); + sb.append(indent) + .append(" int next = ") + .append(ctx.callName(sub)) + .append("(input, cursor, limit);\n"); + sb.append(indent) + .append(" if (next < 0 || next == cursor) break;\n"); + sb.append(indent) + .append(" cursor = next;\n"); + sb.append(indent) + .append("}\n"); + sb.append(indent) + .append("return cursor;\n"); + } + + private void emitTppOptional(StringBuilder sb, Expression body, TriviaPostPassEmitContext ctx, String indent) { + int sub = ctx.allocate(body); + sb.append(indent) + .append("int next = ") + .append(ctx.callName(sub)) + .append("(input, pos, limit);\n"); + sb.append(indent) + .append("return next < 0 ? pos : next;\n"); + } + + private void emitTppRepetition(StringBuilder sb, + Expression.Repetition rep, + TriviaPostPassEmitContext ctx, + String indent) { + int sub = ctx.allocate(rep.expression()); + var maxExpr = rep.max() + .isPresent() + ? String.valueOf(rep.max() + .unwrap()) + : "Integer.MAX_VALUE"; + sb.append(indent) + .append("int cursor = pos;\n"); + sb.append(indent) + .append("int count = 0;\n"); + sb.append(indent) + .append("int max = ") + .append(maxExpr) + .append(";\n"); + sb.append(indent) + .append("while (count < max) {\n"); + sb.append(indent) + .append(" int next = ") + .append(ctx.callName(sub)) + .append("(input, cursor, limit);\n"); + sb.append(indent) + .append(" if (next < 0 || next == cursor) break;\n"); + sb.append(indent) + .append(" cursor = next;\n"); + sb.append(indent) + .append(" count++;\n"); + sb.append(indent) + .append("}\n"); + sb.append(indent) + .append("return count >= ") + .append(rep.min()) + .append(" ? cursor : -1;\n"); + } + + /** + * Quote a string for embedding into Java source, escaping special chars. + * Mirrors the conventions used elsewhere in {@code ParserGenerator}. + */ + private static String quoteJavaString(String s) { + var out = new StringBuilder(s.length() + 8); + out.append('"'); + for (int i = 0; i < s.length(); i++ ) { + char c = s.charAt(i); + switch (c) { + case'"' -> out.append("\\\""); + case'\\' -> out.append("\\\\"); + case'\n' -> out.append("\\n"); + case'\r' -> out.append("\\r"); + case'\t' -> out.append("\\t"); + case'\b' -> out.append("\\b"); + case'\f' -> out.append("\\f"); + default -> { + if (c < 0x20 || c == 0x7f) { + out.append(String.format("\\u%04x", (int) c)); + }else { + out.append(c); + } + } + } + } + out.append('"'); + return out.toString(); + } + // === Match-method emitters (phase 1 perf flags) === // Phase 1.7 (D2): match helpers return successNoLoc — endLocation field on // intermediate (per-leaf) results is never read by callers. The diff --git a/peglib-core/src/test/java/org/pragmatica/peg/generator/GeneratedParserTriviaPostPassTest.java b/peglib-core/src/test/java/org/pragmatica/peg/generator/GeneratedParserTriviaPostPassTest.java new file mode 100644 index 0000000..7513346 --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/generator/GeneratedParserTriviaPostPassTest.java @@ -0,0 +1,240 @@ +package org.pragmatica.peg.generator; + +import org.junit.jupiter.api.Test; +import org.pragmatica.peg.PegParser; +import org.pragmatica.peg.error.RecoveryStrategy; +import org.pragmatica.peg.parser.ParserConfig; + +import javax.tools.ToolProvider; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.file.Files; +import java.util.List; +import java.util.Set; + +import static org.assertj.core.api.Assertions.assertThat; + +/** + * Step 4 commit 4 (0.5.1) — verify the generated parser embeds and invokes a + * working {@code TriviaPostPass} when {@code config.triviaPostPass()=true} at + * generation time. + * + *

The tests: + *

    + *
  1. Source emission: flag-ON includes the embedded class and the call; + * flag-OFF includes neither (default behavior bit-for-bit unchanged).
    2. + *
  2. Standalone-parser invariant: zero peglib-runtime imports in the + * generated source under flag-ON.
    3. + *
  3. Behavioral: compile + run a flag-ON generated parser and confirm + * leading/trailing trivia and content classification are correct.
    4. + *
  4. Parity: generated parser under flag-ON agrees with the interpreter + * under flag-ON on leading-trivia text + kind for the same input.
    5. + *
+ * + *

Test grammars use a single top-level token-shaped rule (matching + * {@link org.pragmatica.peg.GeneratedParserTriviaTest}) so behavioral + * assertions target the post-pass attribution layer specifically — not the + * pre-existing wrap-with-rule-name single-result collapse. + */ +class GeneratedParserTriviaPostPassTest { + + private static final String SIMPLE_GRAMMAR = """ + Number <- < [0-9]+ > + %whitespace <- [ \\t\\r\\n]* + """; + + private static final String COMMENT_GRAMMAR = """ + Number <- < [0-9]+ > + %whitespace <- ([ \\t]+ / '//' [^\\n]* '\\n')+ + """; + + @Test + void flagOn_emitsEmbeddedTriviaPostPassClass() { + var source = generate(SIMPLE_GRAMMAR, "test.tpp.emit", "EmitParser", true); + assertThat(source).contains("private static final class TriviaPostPass"); + assertThat(source).contains("static CstNode assignTrivia(String input, CstNode cst, int leadingScanFrom)"); + assertThat(source).contains("TriviaPostPass.assignTrivia(input, rootNode, 0);"); + } + + @Test + void flagOff_doesNotEmitEmbeddedTriviaPostPassClass() { + var source = generate(SIMPLE_GRAMMAR, "test.tpp.off", "OffParser", false); + assertThat(source).doesNotContain("private static final class TriviaPostPass"); + assertThat(source).doesNotContain("TriviaPostPass.assignTrivia"); + } + + @Test + void flagOn_generatedSourceHasZeroPeglibRuntimeImports() { + var source = generate(SIMPLE_GRAMMAR, "test.tpp.invariant", "InvariantParser", true); + // Standalone-parser invariant: only pragmatica-lite + java.* imports allowed. + assertThat(source).doesNotContain("import org.pragmatica.peg."); + } + + @Test + void flagOn_parseCapturesLeadingTrivia() throws Exception { + var input = " 42"; + var parserClass = compile(SIMPLE_GRAMMAR, "test.tpp.leading", "LeadingParser", true); + var cst = invokeParse(parserClass, input); + assertThat(collectLeadingTriviaText(cst)).isEqualTo(" "); + } + + @Test + void flagOn_parseCapturesTrailingTrivia() throws Exception { + var input = "42 "; + var parserClass = compile(SIMPLE_GRAMMAR, "test.tpp.trailing", "TrailingParser", true); + var cst = invokeParse(parserClass, input); + assertThat(collectTrailingTriviaText(cst)).isEqualTo(" "); + } + + @Test + void flagOn_parseClassifiesLineCommentTrivia() throws Exception { + var input = "// hello\n42"; + var parserClass = compile(COMMENT_GRAMMAR, "test.tpp.comment", "CommentParser", true); + var cst = invokeParse(parserClass, input); + var leading = leadingTriviaList(cst); + assertThat(leading).hasSize(1); + // First trivia should be classified as LineComment by content prefix. + assertThat(leading.getFirst() + .getClass() + .getSimpleName()).isEqualTo("LineComment"); + } + + @Test + void flagOn_generatedAndInterpreterAgreeOnLeadingTrivia() throws Exception { + var input = " // first\n 42"; + var parserClass = compile(COMMENT_GRAMMAR, "test.tpp.parity", "ParityParser", true); + var generatedCst = invokeParse(parserClass, input); + + var interpreterParser = PegParser.builder(COMMENT_GRAMMAR) + .triviaPostPass(true) + .build() + .unwrap(); + var interpreterCst = interpreterParser.parseCst(input) + .unwrap(); + + var genLeading = leadingTriviaList(generatedCst); + assertThat(genLeading).hasSameSizeAs(interpreterCst.leadingTrivia()); + for (int i = 0; i < interpreterCst.leadingTrivia() + .size(); i++) { + var rt = interpreterCst.leadingTrivia() + .get(i); + var gt = genLeading.get(i); + assertThat(gt.getClass() + .getSimpleName()).isEqualTo(rt.getClass() + .getSimpleName()); + assertThat((String) gt.getClass() + .getMethod("text") + .invoke(gt)).isEqualTo(rt.text()); + } + } + + @Test + void flagOff_defaultBehaviorParsesAndCapturesTrivia() throws Exception { + // Validate flag-OFF generation still parses and captures trivia (the + // pre-existing engine path; no post-pass invoked). + var input = " 42 "; + var parserClass = compile(SIMPLE_GRAMMAR, "test.tpp.unchanged", "UnchangedParser", false); + var cst = invokeParse(parserClass, input); + assertThat(collectLeadingTriviaText(cst)).isEqualTo(" "); + assertThat(collectTrailingTriviaText(cst)).isEqualTo(" "); + } + + // === Generator helpers === + private String generate(String grammar, String pkg, String cls, boolean triviaPostPass) { + var base = ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true); + var config = new ParserConfig( + base.packratEnabled(), + base.recoveryStrategy(), + base.captureTrivia(), + base.fastTrackFailure(), + base.literalFailureCache(), + base.charClassFailureCache(), + base.bulkAdvanceLiteral(), + base.skipWhitespaceFastPath(), + base.reuseEndLocation(), + base.choiceDispatch(), + base.markResetChildren(), + base.inlineLocations(), + base.selectivePackrat(), + Set.of(), + base.mutableParseResult(), + base.tokenFastPath(), + triviaPostPass); + return PegParser.generateCstParser(grammar, pkg, cls, ErrorReporting.BASIC, config) + .unwrap(); + } + + private Class compile(String grammar, String pkg, String cls, boolean triviaPostPass) throws Exception { + var source = generate(grammar, pkg, cls, triviaPostPass); + var fqcn = pkg + "." + cls; + var tempDir = Files.createTempDirectory("peglib-tpp-test"); + var packageDir = tempDir.resolve(pkg.replace('.', '/')); + Files.createDirectories(packageDir); + var sourceFile = packageDir.resolve(cls + ".java"); + Files.writeString(sourceFile, source); + var compiler = ToolProvider.getSystemJavaCompiler(); + int rc = compiler.run(null, null, null, + "-d", tempDir.toString(), + "-cp", System.getProperty("java.class.path"), + sourceFile.toString()); + if (rc != 0) { + throw new RuntimeException("Compilation failed for " + fqcn); + } + var loader = new URLClassLoader(new URL[]{tempDir.toUri() + .toURL()}); + return loader.loadClass(fqcn); + } + + private Object invokeParse(Class parserClass, String input) throws Exception { + var parser = parserClass.getDeclaredConstructor() + .newInstance(); + var parseMethod = findParseMethod(parserClass); + var result = parseMethod.invoke(parser, input); + var unwrap = result.getClass() + .getMethod("unwrap"); + return unwrap.invoke(result); + } + + private Method findParseMethod(Class parserClass) { + for (var m : parserClass.getMethods()) { + if ("parse".equals(m.getName()) && m.getParameterCount() == 1 + && m.getParameterTypes()[0] == String.class) { + return m; + } + } + throw new IllegalStateException("No parse(String) method on " + parserClass); + } + + private List leadingTriviaList(Object cst) throws Exception { + @SuppressWarnings("unchecked") + var list = (List) cst.getClass() + .getMethod("leadingTrivia") + .invoke(cst); + return list; + } + + private String collectLeadingTriviaText(Object cst) throws Exception { + var sb = new StringBuilder(); + for (var t : leadingTriviaList(cst)) { + sb.append((String) t.getClass() + .getMethod("text") + .invoke(t)); + } + return sb.toString(); + } + + private String collectTrailingTriviaText(Object cst) throws Exception { + @SuppressWarnings("unchecked") + var list = (List) cst.getClass() + .getMethod("trailingTrivia") + .invoke(cst); + var sb = new StringBuilder(); + for (var t : list) { + sb.append((String) t.getClass() + .getMethod("text") + .invoke(t)); + } + return sb.toString(); + } +} From 149bc57a59500882b87f722a2fe269cc92248fb1 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 07:50:04 +0200 Subject: [PATCH 09/26] =?UTF-8?q?docs:=20HANDOVER=20=C2=A711=20+=20CHANGEL?= =?UTF-8?q?OG=20[0.5.1]=20=E2=80=94=20trivia=20rework=20Step=204=20commits?= =?UTF-8?q?=201-4=20progress?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 8 +++++++- docs/HANDOVER.md | 31 ++++++++++++++++++++++++++++++- 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ea1f547..4774801 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,12 +7,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [0.5.1] - 2026-05-09 -_Unreleased — patch cycle following 0.5.0. Use this section to log fixes / minor perf tweaks / docs updates accumulated post-release._ +_Unreleased — patch cycle following 0.5.0._ ### Added +- **Trivia investigation arc — Steps 1-3** (2026-05-09): catalog of context-dependencies in current attribution, 17-test adversarial corpus targeting 7 divergence categories, post-pass prototype validating context-independent attribution. Round-trip preservation confirmed on 21/21 corpus + 9/9 adversarial inputs; 0 text loss across 46,756 nodes; `parseRuleAt` structural parity achievable under post-pass (load-bearing claim for incremental engine Lever B). Findings: [`docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md`](docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md). Verdict in HANDOVER §11. +- **Trivia rework Step 4 commits 1-4** (production rework, foundational): opt-in `triviaPostPass` flag on `ParserConfig` (default `false` — zero behavior change). When enabled: post-parse hook applies `TriviaPostPass.assignTrivia(input, cst, grammar)` overriding buffer-driven attribution with context-independent re-derivation from `(input, span)` coordinates. `parseRuleAt` honors splice offset (4-arg overload `assignTrivia(input, cst, grammar, leadingScanFrom)`) — body subtree is structurally identical to corresponding subtree of full reparse, removing one of two HANDOVER §6.4 blockers for incremental engine Lever B. Generator-emitted parsers embed an inline post-pass equivalent (preserves standalone-parser invariant). +- **`org.pragmatica.peg.tree.TriviaPostPass`** (new public API): pure function `(input, cst, grammar) → CstNode` that re-derives leading/trailing trivia from coordinates without parse-history dependency. 4-arg overload supports splice scenarios. + ### Changed +- `ParsingContext` pending-trivia methods (`appendPendingLeadingTrivia`, `takePendingLeadingTrivia`, `savePendingLeadingTrivia`, `restorePendingLeadingTrivia`) early-out when `config.triviaPostPass()=true` — buffer becomes dead weight under the flag and is skipped (38 call sites in `PegEngine` no-op without source changes). + ### Fixed ### Removed diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index d3fb461..2fad57f 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -491,6 +491,35 @@ Suite at `peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaAdversarialTes After production rework: orthogonal Lever B blockers remain (fallback-rule bypass per §6.2 root cause #1, safe-pivot concept). Trivia rework alone removes ONE of two HANDOVER-cited blockers. +### Step 4 production rework — commits 1-4 landed (2026-05-09) + +After Step 3 verdict, the production rework arc executed in the same session. Branch `release-0.5.1` advanced from `51144a6` to `dcd146f` (4 commits + 1 docs commit). + +| Commit | Description | +|---|---| +| `318f5cf` | feat: triviaPostPass flag wires TriviaPostPass as post-parse hook (Step 4 commit 1) | +| `d7b3496` | feat: TriviaPostPass splice-offset overload for parseRuleAt subtree leading (Step 4 commit 2) | +| `605327b` | perf: ParsingContext buffer ops no-op under triviaPostPass flag (Step 4 commit 3) | +| `dcd146f` | feat: ParserGenerator emits embedded TriviaPostPass under flag-ON (Step 4 commit 4) | + +**Functional state:** the trivia rework is COMPLETE through commit 4. Under `triviaPostPass=true`: + +- **Interpreter** (`PegEngine`): buffer ops no-op via `ParsingContext` early-outs (38 call sites neutralized without source changes); post-pass produces correct attribution. Optimal — no wasted CPU. +- **Generator** (`ParserGenerator`): generated parsers embed an inline `TriviaPostPass` (preserves standalone-parser invariant — generated parsers depend only on `pragmatica-lite:core`); `parseCst` calls the embedded post-pass after the body parse. Correct attribution; **suboptimal CPU** because buffer machinery still runs and is overwritten (commit 5 territory). +- **`parseRuleAt`** honors splice offset (4-arg overload `assignTrivia(input, cst, grammar, leadingScanFrom)`); body subtree structurally identical to corresponding subtree of full reparse — Lever B unblocker validated in tests. + +**Test coverage:** 768 peglib-core tests + 1 pre-existing skip. `TriviaPostPassFlagTest` (12 tests across 4 nested classes), `TriviaPostPassSpliceOffsetTest` (7 tests), `GeneratedParserTriviaPostPassTest` (8 tests), `TriviaPostPassTest` (16 tests), `TriviaAdversarialTest` (16 enabled). All green under both flag-OFF (default) and flag-ON. + +**Default behavior (flag-OFF) unchanged** — every pre-existing test passes byte-for-byte identically. Adoption is opt-in via `PegParser.builder(grammar).triviaPostPass(true).build()` or constructing `ParserConfig` with the flag set. + +### Outstanding Step 4 work (next session) + +- **Commit 5: strip buffer machinery from generator emission under flag-ON.** Mirror commit 3 for the generator side. ~30 emission sites in `ParserGenerator.java` need flag-conditional skipping (don't emit appendPending/takePending/save/restore at combinators when gen-time flag is on). Pure CPU optimization; correctness already there. Bench A/B should show flag-ON generator wallclock improvement after this commit. +- **Commit 6: default flip the flag.** Migration concern — tests that pin specific attribution layouts may shift (Step 3 prototype showed 6.4% of nodes have different attribution under post-pass; total trivia text preserved). Survey downstream impact before flipping. +- **Bench impact study:** measure post-pass overhead on full-parse vs current attribution. Step 3 prototype was non-invasive (overlay); the rework wins back the buffer work but pays for the post-pass scan. Net likely positive but unmeasured. + +After commits 5-6, Lever B can be retried: `parseRuleAt` + post-pass produces structurally identical subtrees, removing the trivia-context-loss blocker. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work. + ### Items superseded by this session's work - §6.2 lever-1 puzzle: dissolved by Path D's stable-id algorithm. @@ -502,4 +531,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of trivia investigation arc on `release-0.5.1`. 0.5.0 published to Maven Central on 2026-05-08 (5 module artifacts). Branch `release-0.5.1` at `78c4003` — 3 commits past 0.5.1 base (`c9502c5` chore release prep, `d2cc6be` Step 2 adversarial corpus, `78c4003` Step 3 TriviaPostPass prototype). peglib-core 738 tests green + 1 pre-existing skipped (RoundTripTest legacy gap unchanged). Trivia rework verdict: **viable; ~6-10 days to production rework**. Detailed plan in §11 "Trivia investigation arc". Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: production trivia rework OR Lever C IR unification OR deferred char-class bit-packing reassessment. +**Last updated:** 2026-05-09, end of Step 4 commits 1-4 of trivia production rework on `release-0.5.1`. 0.5.0 published to Maven Central on 2026-05-08 (5 module artifacts). Branch `release-0.5.1` at `dcd146f` — 8 commits past 0.5.1 base (chore release prep + 3 investigation commits + 4 rework commits). peglib-core 768 tests green + 1 pre-existing skipped. Trivia rework state: **functionally complete under `triviaPostPass=true` flag (opt-in)**. Default behavior unchanged. Outstanding: commit 5 (strip generator emission under flag-ON; pure CPU win), commit 6 (default flip with migration concern). Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: complete Step 4 (commits 5-6) OR pivot to Lever C IR unification OR Lever B retry now that trivia-context-loss blocker is gone (orthogonal fallback-bypass blocker remains). From 4ed1cf5493e409b98b4066531d621ce306f0b0c3 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 08:04:45 +0200 Subject: [PATCH 10/26] perf: ParserGenerator emits no-op buffer methods under triviaPostPass flag (Step 4 commit 5) --- .../peg/generator/ParserGenerator.java | 71 +++++++++++++------ 1 file changed, 50 insertions(+), 21 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index d0eace6..cf5bbd2 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -6773,31 +6773,60 @@ private Trivia classifyTrivia(SourceSpan span, String text) { // === Pending-leading trivia helpers === // See docs/TRIVIA-ATTRIBUTION.md for the attribution rule. - private void appendPending(List captured) { - if (!captured.isEmpty()) { - pendingLeadingTrivia.addAll(captured); - } - } + """); + // Step 4 commit 5: Under triviaPostPass=true, the post-pass produces correct + // attribution from the trivia stream. The buffer machinery is dead work — the + // call sites still call these helpers, but the helpers no-op (and the buffer + // field stays declared but unused — see PR description for rationale). + if (config.triviaPostPass()) { + sb.append(""" + private void appendPending(List captured) { + // no-op: triviaPostPass=true; attribution handled by post-pass. + } - private List takePendingLeading() { - if (pendingLeadingTrivia.isEmpty()) { - return List.of(); - } - var snapshot = List.copyOf(pendingLeadingTrivia); - pendingLeadingTrivia.clear(); - return snapshot; - } + private List takePendingLeading() { + return List.of(); + } - private int savePendingLeading() { - return pendingLeadingTrivia.size(); - } + private int savePendingLeading() { + return 0; + } - private void restorePendingLeading(int snapshot) { - if (pendingLeadingTrivia.size() > snapshot) { - pendingLeadingTrivia.subList(snapshot, pendingLeadingTrivia.size()).clear(); - } - } + private void restorePendingLeading(int snapshot) { + // no-op: triviaPostPass=true; attribution handled by post-pass. + } + + """); + }else { + sb.append(""" + private void appendPending(List captured) { + if (!captured.isEmpty()) { + pendingLeadingTrivia.addAll(captured); + } + } + + private List takePendingLeading() { + if (pendingLeadingTrivia.isEmpty()) { + return List.of(); + } + var snapshot = List.copyOf(pendingLeadingTrivia); + pendingLeadingTrivia.clear(); + return snapshot; + } + private int savePendingLeading() { + return pendingLeadingTrivia.size(); + } + + private void restorePendingLeading(int snapshot) { + if (pendingLeadingTrivia.size() > snapshot) { + pendingLeadingTrivia.subList(snapshot, pendingLeadingTrivia.size()).clear(); + } + } + + """); + } + sb.append(""" private List concatTrivia(List first, List second) { if (first.isEmpty()) return second; if (second.isEmpty()) return first; From 4beca0ab8c3c6dbdc11119b7bde2aa888e6940ae Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 08:25:36 +0200 Subject: [PATCH 11/26] =?UTF-8?q?docs:=20HANDOVER=20+=20CHANGELOG=20?= =?UTF-8?q?=E2=80=94=20Step=204=20commit=205=20landed;=20commit=206=20reve?= =?UTF-8?q?rted=20(bug=20found=20in=20post-pass=20on=20Java=2025=20corpus)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 5 +++++ docs/HANDOVER.md | 46 ++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 45 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4774801..9db6f61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,11 @@ _Unreleased — patch cycle following 0.5.0._ ### Changed - `ParsingContext` pending-trivia methods (`appendPendingLeadingTrivia`, `takePendingLeadingTrivia`, `savePendingLeadingTrivia`, `restorePendingLeadingTrivia`) early-out when `config.triviaPostPass()=true` — buffer becomes dead weight under the flag and is skipped (38 call sites in `PegEngine` no-op without source changes). +- **Trivia rework Step 4 commit 5** (`4ed1cf5`): `ParserGenerator` emits no-op buffer methods under flag-ON. Mirror of commit 3 for the generator side — pure CPU optimization; correctness already from commit 4. Generated parsers under flag-ON no longer do dead-work buffer maintenance. + +### Known Issues + +- **`triviaPostPass=true` flag has a known bug on production Java 25 corpus** (discovered 2026-05-09 during Step 4 commit 6 attempt). Round-trip reconstruction LOSES trivia text — not just slot-shifts attribution but actual character loss. 20 of 22 RoundTripTest fixtures fail under flag-ON via the generated parser path: `void allCompoundAssignments` becomes `voidallCompoundAssignments` etc. Bug likely lives in the embedded TriviaPostPass emission (~617 lines added to ParserGenerator in commit 4). The flag remains **opt-in only**; the default is `false`. Validation gap: Step 3 prototype tests excluded the `/large/` fixture and used small grammars only. Track work: HANDOVER §11 "Step 4 commit 5 landed; commit 6 attempted and reverted (BUG DISCOVERED)". ### Fixed diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 2fad57f..15ee7dc 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -512,13 +512,47 @@ After Step 3 verdict, the production rework arc executed in the same session. Br **Default behavior (flag-OFF) unchanged** — every pre-existing test passes byte-for-byte identically. Adoption is opt-in via `PegParser.builder(grammar).triviaPostPass(true).build()` or constructing `ParserConfig` with the flag set. -### Outstanding Step 4 work (next session) +### Step 4 commit 5 landed; commit 6 attempted and reverted (BUG DISCOVERED) -- **Commit 5: strip buffer machinery from generator emission under flag-ON.** Mirror commit 3 for the generator side. ~30 emission sites in `ParserGenerator.java` need flag-conditional skipping (don't emit appendPending/takePending/save/restore at combinators when gen-time flag is on). Pure CPU optimization; correctness already there. Bench A/B should show flag-ON generator wallclock improvement after this commit. -- **Commit 6: default flip the flag.** Migration concern — tests that pin specific attribution layouts may shift (Step 3 prototype showed 6.4% of nodes have different attribution under post-pass; total trivia text preserved). Survey downstream impact before flipping. -- **Bench impact study:** measure post-pass overhead on full-parse vs current attribution. Step 3 prototype was non-invasive (overlay); the rework wins back the buffer work but pays for the post-pass scan. Net likely positive but unmeasured. +**Commit 5 (`4ed1cf5`)** — ParserGenerator emits no-op buffer methods under flag-ON. Pure CPU optimization parallel to commit 3 for the interpreter. Tests stay at 768 + 1 skip. Generated parser under flag-ON is now lean (no dead buffer work) AND correct (post-pass produces attribution). -After commits 5-6, Lever B can be retried: `parseRuleAt` + post-pass produces structurally identical subtrees, removing the trivia-context-loss blocker. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work. +**Commit 6 attempted — REVERTED. Real bug surfaced in post-pass implementation.** + +Flipping `triviaPostPass` default to `true` triggered **20 RoundTripTest failures on the Java 25 corpus** — *not* slot-shifts but actual trivia text LOSS: + +- `void allCompoundAssignments` → `voidallCompoundAssignments` (lost whitespace between tokens) +- `int r;` → `intr;` +- `int b = 0;` → `intb = 0;` +- Reconstructed lengths 1-2059 chars shorter than originals across 20 of 22 RoundTripTest fixtures (`Annotations.java`, `BlankLines.java`, `ChainAlignment.java`, `ClassLiterals.java`, `Comments.java`, `CompoundAssignments.java`, `Enums.java`, `ExhaustiveSwitchPatterns.java`, `Imports.java`, `KeywordPrefixedIdentifiers.java`, `Lambdas.java`, `LineWrapping.java`, `MultilineArguments.java`, `MultilineParameters.java`, `Records.java`, `SwitchExpressions.java`, `TernaryOperators.java`, `TextBlocks.java`, `large/FactoryClassGenerator.java.txt`, `flow-format-examples/BlankLineRules.java`) + +This contradicts the post-pass spec doc claim: *"Total trivia text is preserved (round-trip reconstruction is byte-equal); only the leading/trailing slot in which a given trivia chunk lives can differ."* The bug is REAL TEXT LOSS on production-realistic Java 25 inputs, not just attribution permutation. + +**Test coverage gap that masked the bug:** +- Step 3 prototype's `TriviaPostPassTest` round-trip claim ("21/21 corpus") used the **smaller fixture set under perf-corpus**, EXCLUDING the `/large/` directory which has the realistic 1900-LOC `FactoryClassGenerator.java.txt`. +- `TriviaAdversarialTest` (16 tests) uses small focused grammars — JSON-like, not full Java 25. +- `GeneratedParserTriviaPostPassTest` (8 tests) uses 5 small grammars (parity, leading, trailing, comment, unchanged). +- The full `RoundTripTest` corpus (22 Java 25 fixtures) was never run under flag-ON until commit 6 attempt. + +**Bug location:** The generator path is materially affected. `GeneratedJava25Parser.ensureLoaded()` calls `PegParser.generateCstParser(...)` with no explicit config → picks up `ParserConfig.DEFAULT` → flag-flip flips the generator's emission. Whether the interpreter-only path has the same bug is unverified (could be tested by setting flag-ON on Phase1ParityTest variants). + +**Next-session work needed (BEFORE default flip can be reattempted):** + +1. **Reproduce the bug in isolation** — create a focused test case that fails under flag-ON. Minimal Java 25 sub-fixture; goal is a 5-line input that loses trivia text under flag-ON. +2. **Diagnose the embedded post-pass implementation** in the generated parser. `ParserGenerator.java`'s `embedded TriviaPostPass` (added in commit 4, ~617 lines) emits an inline post-pass. The bug likely lives there — possibly: + - Whitespace re-scan logic in the embedded post-pass mismatches the parser's own `skipWhitespace` semantics + - The embedded post-pass's coordinate-walking misses whitespace between tokens (specifically `(prev_node_end, this_node_start)` where the gap is a token-boundary rather than a sibling-boundary) + - Cache-hit interaction (Bug B fix is in the parser; the embedded post-pass might re-derive trivia for nodes whose body came from cache and lose context) +3. **Fix the embedded post-pass** OR extend `TriviaPostPass` (runtime) similarly. Re-run RoundTripTest under flag-ON; should be 22/22. +4. **Run full coverage under flag-ON** before reattempting commit 6: every existing test class set to flag-ON, measure failures, fix incrementally. NOT a default-flip until the bug is gone. +5. **Bench impact study** (deferred from original plan — still relevant after bug fix). + +**What's still valuable from Steps 1-5:** +- The flag is functional for SMALL grammars + adversarial inputs (validated) +- `parseRuleAt` splice-offset overload is correct and unblocks Lever B's trivia-context-loss problem in principle +- Buffer no-op infrastructure in both interpreter and generator is clean and ready +- Commits 1-5 are SHIPPABLE; the flag is opt-in only + +After bug fix + commit 6, Lever B can be retried: `parseRuleAt` + post-pass produces structurally identical subtrees, removing the trivia-context-loss blocker. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work. ### Items superseded by this session's work @@ -531,4 +565,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 commits 1-4 of trivia production rework on `release-0.5.1`. 0.5.0 published to Maven Central on 2026-05-08 (5 module artifacts). Branch `release-0.5.1` at `dcd146f` — 8 commits past 0.5.1 base (chore release prep + 3 investigation commits + 4 rework commits). peglib-core 768 tests green + 1 pre-existing skipped. Trivia rework state: **functionally complete under `triviaPostPass=true` flag (opt-in)**. Default behavior unchanged. Outstanding: commit 5 (strip generator emission under flag-ON; pure CPU win), commit 6 (default flip with migration concern). Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: complete Step 4 (commits 5-6) OR pivot to Lever C IR unification OR Lever B retry now that trivia-context-loss blocker is gone (orthogonal fallback-bypass blocker remains). +**Last updated:** 2026-05-09, end of Step 4 commits 1-5 of trivia production rework on `release-0.5.1` + commit 6 attempt with REVERT (bug discovered). Branch `release-0.5.1` at `4ed1cf5` — 9 commits past 0.5.1 base. peglib-core 768 tests green + 1 pre-existing skipped. Trivia rework state: **flag-ON path FUNCTIONAL on small grammars + adversarial inputs; KNOWN BUG on production Java 25 corpus (trivia text loss, not slot-shift). Default flip BLOCKED until bug fix.** Opt-in flag is shippable. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: diagnose + fix the embedded post-pass bug on Java 25 fixtures (likely in ParserGenerator's embedded TriviaPostPass emission, ~617 lines added in commit 4). Then reattempt commit 6. From 612fbea56faaa5e1d952959607c919359b759e4b Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 08:48:36 +0200 Subject: [PATCH 12/26] fix: TriviaPostPass drains orphan trivia into last-leaf descendant matching engine Bug C' (Step 4 commit 7) --- .../peg/generator/ParserGenerator.java | 47 +++++++++- .../pragmatica/peg/tree/TriviaPostPass.java | 91 ++++++++++++++++--- .../peg/perf/TriviaPostPassFlagTest.java | 25 ++++- 3 files changed, 143 insertions(+), 20 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index cf5bbd2..2d46268 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -7195,16 +7195,57 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" List leading, List extraTrailing) {\n"); sb.append(" int spanStart = nt.span().startOffset();\n"); sb.append(" int spanEnd = nt.span().endOffset();\n"); - sb.append(" var newChildren = new ArrayList(nt.children().size());\n"); + sb.append(" // Generator semantics adapter: the wrapper's span may INCLUDE its\n"); + sb.append(" // own leading trivia (rule entry captures startOffset BEFORE\n"); + sb.append(" // skipWhitespace). If the caller-supplied leading occupies\n"); + sb.append(" // [spanStart, spanStart+leadingLen), advance cursor past it so the\n"); + sb.append(" // first child's leading scan does not re-emit the same trivia.\n"); + sb.append(" int leadingLen = totalTriviaLength(leading);\n"); sb.append(" int cursor = spanStart;\n"); + sb.append(" if (leadingLen > 0 && spanStart + leadingLen <= input.length()) {\n"); + sb.append(" var probe = scanWhitespace(input, spanStart, spanStart + leadingLen);\n"); + sb.append(" if (totalTriviaLength(probe) == leadingLen) {\n"); + sb.append(" cursor = spanStart + leadingLen;\n"); + sb.append(" }\n"); + sb.append(" }\n"); + sb.append(" var newChildren = new ArrayList(nt.children().size());\n"); sb.append(" for (var c : nt.children()) {\n"); sb.append(" var rebuilt = rebuildChild(input, c, cursor);\n"); sb.append(" newChildren.add(rebuilt);\n"); sb.append(" cursor = c.span().endOffset();\n"); sb.append(" }\n"); + sb.append(" // Bug C' compensation — drain orphan trailing into the last terminal\n"); + sb.append(" // descendant so CstReconstruct.emit (last-child-trailing-only) stays\n"); + sb.append(" // byte-equal even when this wrapper is not its parent's last child.\n"); sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd);\n"); - sb.append(" var trailing = combine(internalTrailing, extraTrailing);\n"); - sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, trailing);\n"); + sb.append(" List wrapperTrailing = extraTrailing;\n"); + sb.append(" if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) {\n"); + sb.append(" int lastIdx = newChildren.size() - 1;\n"); + sb.append(" newChildren.set(lastIdx, postPassAttachTrailingToTail(newChildren.get(lastIdx), internalTrailing));\n"); + sb.append(" } else {\n"); + sb.append(" wrapperTrailing = combine(internalTrailing, extraTrailing);\n"); + sb.append(" }\n"); + sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, wrapperTrailing);\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode postPassAttachTrailingToTail(CstNode node, List extra) {\n"); + sb.append(" if (extra.isEmpty()) return node;\n"); + sb.append(" return switch (node) {\n"); + sb.append(" case CstNode.NonTerminal nt -> {\n"); + sb.append(" var children = nt.children();\n"); + sb.append(" if (children.isEmpty()) {\n"); + sb.append(" yield new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), children, nt.leadingTrivia(), combine(nt.trailingTrivia(), extra));\n"); + sb.append(" }\n"); + sb.append(" var newChildren = new ArrayList(children);\n"); + sb.append(" int lastIdx = newChildren.size() - 1;\n"); + sb.append(" newChildren.set(lastIdx, postPassAttachTrailingToTail(newChildren.get(lastIdx), extra));\n"); + sb.append(" yield new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), nt.leadingTrivia(), nt.trailingTrivia());\n"); + sb.append(" }\n"); + sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), t.leadingTrivia(), combine(t.trailingTrivia(), extra));\n"); + sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), tk.leadingTrivia(), combine(tk.trailingTrivia(), extra));\n"); + if (errorReporting == ErrorReporting.ADVANCED) { + sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), e.leadingTrivia(), combine(e.trailingTrivia(), extra));\n"); + } + sb.append(" };\n"); sb.append(" }\n\n"); sb.append(" private static List combine(List a, List b) {\n"); sb.append(" if (a.isEmpty()) return b;\n"); diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index 2d84bd6..ca07054 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -23,17 +23,18 @@ *

Attribution policy

* *

The post-pass uses a single policy: leading trivia is the gap before - * a node; trailing trivia is the gap after the last child of a non-terminal - * within its own span; terminals never own trailing. + * a node; trailing trivia drains into the last-terminal descendant of a + * non-terminal, matching the engine's Bug C' compensation; terminals never + * own trailing unless they receive drained orphan trivia. * - *

This differs from the current engine attribution in one documented way: + *

The structural shape of the post-pass CST therefore matches the engine: * orphan trivia consumed before a zero-width tail element (empty - * {@code ZeroOrMore} / {@code Optional}) lands on the enclosing rule - * wrapper's trailingTrivia here, whereas the engine drains it into the - * last terminal child's trailingTrivia (Bug C' compensation via - * {@code attachTrailingToTail}). Both policies preserve byte-equal round-trip - * reconstruction; the structural shape of the CST differs only in which slot - * the trivia lives in. + * {@code ZeroOrMore} / {@code Optional}) is drained into the last + * terminal descendant's trailingTrivia via the recursive helper + * {@link #attachTrailingToTail}, mirroring {@code PegEngine.attachTrailingToTail}. + * This keeps {@code CstReconstruct.emit} (which only emits trailing for the + * last child of a wrapper) byte-equal to the original input even when the + * orphan-bearing wrapper is not the rightmost child of its parent. * *

Whitespace scanning

* @@ -222,10 +223,76 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, cursor = c.span() .endOffset(); } - // Internal trailing — gap between last child end and the wrapper's span end. + // Internal trailing — gap between last child end and the wrapper's + // span end. Mirrors PegEngine.attachTrailingToTail (Bug C' + // compensation): when the wrapper has children, drain the orphan + // trivia into the last child's tail so reconstruction visits it via + // the wrapper's last-child trailing slot. Without this, a wrapper + // that is itself a non-last sibling would lose its trailing during + // CstReconstruct.emit. var internalTrailing = scanWhitespace(input, cursor, spanEnd, grammar); - var trailing = combine(internalTrailing, extraTrailing); - return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, trailing); + List wrapperTrailing = extraTrailing; + if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) { + int lastIdx = newChildren.size() - 1; + newChildren.set(lastIdx, attachTrailingToTail(newChildren.get(lastIdx), internalTrailing)); + }else { + wrapperTrailing = combine(internalTrailing, extraTrailing); + } + return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, wrapperTrailing); + } + + /** + * Recursively drain {@code extra} trivia into the deepest-rightmost-leaf + * of {@code node}, mirroring {@code PegEngine.attachTrailingToTail} + * (Bug C' compensation). Used by {@link #rebuildNonTerminal} to place + * orphan trivia consumed before a zero-width tail element into the + * trailing slot that {@code CstReconstruct.emit} actually visits — the + * last terminal descendant — instead of the wrapper non-terminal's own + * trailing slot, which is invisible during reconstruction when the + * wrapper is not the last child of its parent. + */ + private static CstNode attachTrailingToTail(CstNode node, List extra) { + if (extra.isEmpty()) return node; + return switch (node) { + case CstNode.NonTerminal nt -> { + var children = nt.children(); + if (children.isEmpty()) { + yield new CstNode.NonTerminal(nt.id(), + nt.span(), + nt.rule(), + children, + nt.leadingTrivia(), + combine(nt.trailingTrivia(), extra)); + } + var newChildren = new ArrayList(children); + int lastIdx = newChildren.size() - 1; + newChildren.set(lastIdx, attachTrailingToTail(newChildren.get(lastIdx), extra)); + yield new CstNode.NonTerminal(nt.id(), + nt.span(), + nt.rule(), + List.copyOf(newChildren), + nt.leadingTrivia(), + nt.trailingTrivia()); + } + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), + t.span(), + t.rule(), + t.text(), + t.leadingTrivia(), + combine(t.trailingTrivia(), extra)); + case CstNode.Token tk -> new CstNode.Token(tk.id(), + tk.span(), + tk.rule(), + tk.text(), + tk.leadingTrivia(), + combine(tk.trailingTrivia(), extra)); + case CstNode.Error e -> new CstNode.Error(e.id(), + e.span(), + e.skippedText(), + e.expected(), + e.leadingTrivia(), + combine(e.trailingTrivia(), extra)); + }; } private static List combine(List a, List b) { diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java index 6eaedaa..874a2e0 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java @@ -201,10 +201,21 @@ void flagOnVsOff_listInput_totalTriviaTextIdentical() { @Test void flagOnVsOff_optionalTailGrammar_attributionDiffers() { // Bug-C' fixture: a grammar whose top-level rule has an optional - // tail. The engine drains orphan trivia consumed by the empty - // ZeroOrMore into the last terminal child's trailing slot; - // the post-pass leaves it on the wrapper non-terminal instead. - // Total trivia text is identical — only the slot shifts. + // tail. The engine consumes inter-element whitespace BEFORE + // attempting the empty {@code Tail*} wrapper and drains the + // orphan into the wrapper's trailing slot via + // {@code attachTrailingToTail}. The post-pass instead derives + // the gap as leading of the empty wrapper from coordinates. + // Both attributions are valid (Doc's last child is the empty + // wrapper, so {@code CstReconstruct.emit} visits BOTH leading + // and trailing) — round-trip is byte-equal in both paths. Only + // the slot shifts. Structural divergence on THIS minimal + // fixture is genuine and permanent; the Step 4 commit 7 + // {@code attachTrailingToTail} drain in {@link + // org.pragmatica.peg.tree.TriviaPostPass#rebuildNonTerminal} + // only kicks in when the wrapper has non-empty children, which + // is what fixes the corpus-scale RoundTripTest divergence + // (wrapper-as-non-last-sibling). var grammar = """ Doc <- Number Tail* Tail <- ',' Number @@ -224,8 +235,12 @@ void flagOnVsOff_optionalTailGrammar_attributionDiffers() { assertThat(allTriviaText(on)).as("trivia text totals preserved") .isEqualTo(allTriviaText(off)); + assertThat(reconstruct(on)).as("post-pass round-trip") + .isEqualTo(input); + assertThat(reconstruct(off)).as("engine round-trip") + .isEqualTo(input); assertThat(triviaShape(off)).as( - "post-pass and engine differ on at least one slot for Bug-C' fixture") + "post-pass and engine differ on slot for Bug-C' empty-wrapper fixture") .isNotEqualTo(triviaShape(on)); } } From 3b372afc4104af4867272a096c9c64e6271dd14e Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 08:55:27 +0200 Subject: [PATCH 13/26] feat: triviaPostPass default flips to true (Step 4 commit 6) --- .../pragmatica/peg/parser/ParserConfig.java | 121 +++++++------ .../peg/perf/TriviaPostPassFlagTest.java | 167 ++++++++---------- 2 files changed, 136 insertions(+), 152 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java index 0c0df11..884590a 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParserConfig.java @@ -69,74 +69,73 @@ * {@code (input, span, grammar.whitespace())}. Total trivia text is * preserved (round-trip reconstruction is byte-equal); only the * leading/trailing slot in which a given trivia chunk lives can differ. - * Default: {@code false} (Step 4 commit 1 of trivia rework — flag-off - * behaviour is bit-for-bit identical to prior releases). When - * {@code false}, no post-pass is invoked. + * Default: {@code true} (Step 4 commit 6 of trivia rework — post-pass + * attribution is now the default; opt-out by constructing a + * {@code ParserConfig} explicitly with this field set to {@code false}). + * When {@code false}, no post-pass is invoked. */ -public record ParserConfig( - boolean packratEnabled, - RecoveryStrategy recoveryStrategy, - boolean captureTrivia, - boolean fastTrackFailure, - boolean literalFailureCache, - boolean charClassFailureCache, - boolean bulkAdvanceLiteral, - boolean skipWhitespaceFastPath, - boolean reuseEndLocation, - boolean choiceDispatch, - boolean markResetChildren, - boolean inlineLocations, - boolean selectivePackrat, - Set packratSkipRules, - boolean mutableParseResult, - boolean tokenFastPath, - boolean triviaPostPass) { - public static final ParserConfig DEFAULT = new ParserConfig( - true, - RecoveryStrategy.BASIC, - true, - true, - true, - true, - true, - true, - true, - true, - false, - true, - true, - Set.of(), - false, - true, - false); +public record ParserConfig(boolean packratEnabled, + RecoveryStrategy recoveryStrategy, + boolean captureTrivia, + boolean fastTrackFailure, + boolean literalFailureCache, + boolean charClassFailureCache, + boolean bulkAdvanceLiteral, + boolean skipWhitespaceFastPath, + boolean reuseEndLocation, + boolean choiceDispatch, + boolean markResetChildren, + boolean inlineLocations, + boolean selectivePackrat, + Set packratSkipRules, + boolean mutableParseResult, + boolean tokenFastPath, + boolean triviaPostPass) { + public static final ParserConfig DEFAULT = new ParserConfig(true, + RecoveryStrategy.BASIC, + true, + true, + true, + true, + true, + true, + true, + true, + false, + true, + true, + Set.of(), + false, + true, + true); /** * Convenience factory for the three-field runtime configuration. Phase 1 - * generator-time perf flags, phase-2 {@code choiceDispatch}, and phase-1.8 - * {@code selectivePackrat} default to {@code true}; remaining phase-2 flags - * default to {@code false}. Equivalent to {@link #DEFAULT} with the caller's - * three runtime fields substituted. + * generator-time perf flags, phase-2 {@code choiceDispatch}, phase-1.8 + * {@code selectivePackrat}, and Step-4 {@code triviaPostPass} default to + * {@code true}; remaining phase-2 flags default to {@code false}. + * Equivalent to {@link #DEFAULT} with the caller's three runtime fields + * substituted. */ public static ParserConfig parserConfig(boolean packratEnabled, RecoveryStrategy recoveryStrategy, boolean captureTrivia) { - return new ParserConfig( - packratEnabled, - recoveryStrategy, - captureTrivia, - true, - true, - true, - true, - true, - true, - true, - false, - true, - true, - Set.of(), - false, - true, - false); + return new ParserConfig(packratEnabled, + recoveryStrategy, + captureTrivia, + true, + true, + true, + true, + true, + true, + true, + false, + true, + true, + Set.of(), + false, + true, + true); } } diff --git a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java index 874a2e0..85bb1f7 100644 --- a/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java +++ b/peglib-core/src/test/java/org/pragmatica/peg/perf/TriviaPostPassFlagTest.java @@ -1,8 +1,5 @@ package org.pragmatica.peg.perf; -import org.junit.jupiter.api.DisplayName; -import org.junit.jupiter.api.Nested; -import org.junit.jupiter.api.Test; import org.pragmatica.peg.PegParser; import org.pragmatica.peg.action.RuleId; import org.pragmatica.peg.error.RecoveryStrategy; @@ -12,6 +9,10 @@ import java.util.List; +import org.junit.jupiter.api.DisplayName; +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + import static org.assertj.core.api.Assertions.assertThat; /** @@ -40,7 +41,6 @@ * */ class TriviaPostPassFlagTest { - private static final String SIMPLE_GRAMMAR = """ Number <- < [0-9]+ > %whitespace <- [ \\t]* @@ -61,79 +61,74 @@ class TriviaPostPassFlagTest { record Item() implements RuleId {} private static ParserConfig configOff() { - return ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true); + var on = ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true); + return new ParserConfig(on.packratEnabled(), + on.recoveryStrategy(), + on.captureTrivia(), + on.fastTrackFailure(), + on.literalFailureCache(), + on.charClassFailureCache(), + on.bulkAdvanceLiteral(), + on.skipWhitespaceFastPath(), + on.reuseEndLocation(), + on.choiceDispatch(), + on.markResetChildren(), + on.inlineLocations(), + on.selectivePackrat(), + on.packratSkipRules(), + on.mutableParseResult(), + on.tokenFastPath(), + false); } private static ParserConfig configOn() { - var off = configOff(); - return new ParserConfig( - off.packratEnabled(), - off.recoveryStrategy(), - off.captureTrivia(), - off.fastTrackFailure(), - off.literalFailureCache(), - off.charClassFailureCache(), - off.bulkAdvanceLiteral(), - off.skipWhitespaceFastPath(), - off.reuseEndLocation(), - off.choiceDispatch(), - off.markResetChildren(), - off.inlineLocations(), - off.selectivePackrat(), - off.packratSkipRules(), - off.mutableParseResult(), - off.tokenFastPath(), - true); + return ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true); } // =================================================================== // 1. Default-off no-op // =================================================================== - @Nested - @DisplayName("Default-off no-op") - class DefaultOffNoOp { - + @DisplayName("Default-on parity with explicit configOn") + class DefaultOnNoOp { @Test - void defaultConfig_flagOff_returnsEngineCstUnchanged() { + void defaultConfig_flagOn_matchesExplicitOnParser() { var input = " 42 "; var defaultParser = PegParser.fromGrammar(SIMPLE_GRAMMAR) .unwrap(); - var explicitOffParser = PegParser.fromGrammar(SIMPLE_GRAMMAR, configOff()) - .unwrap(); - + var explicitOnParser = PegParser.fromGrammar(SIMPLE_GRAMMAR, + configOn()) + .unwrap(); var defaultCst = defaultParser.parseCst(input) .unwrap(); - var explicitOffCst = explicitOffParser.parseCst(input) - .unwrap(); - - // Default ParserConfig has triviaPostPass=false, so the two parses - // must produce structurally identical trees. - assertThat(reconstruct(defaultCst)).isEqualTo(reconstruct(explicitOffCst)); - assertThat(allTriviaText(defaultCst)).isEqualTo(allTriviaText(explicitOffCst)); - assertThat(triviaShape(defaultCst)).isEqualTo(triviaShape(explicitOffCst)); + var explicitOnCst = explicitOnParser.parseCst(input) + .unwrap(); + // Default ParserConfig now has triviaPostPass=true, so the + // default-config parse must structurally match an explicit + // flag-on parse. + assertThat(reconstruct(defaultCst)).isEqualTo(reconstruct(explicitOnCst)); + assertThat(allTriviaText(defaultCst)).isEqualTo(allTriviaText(explicitOnCst)); + assertThat(triviaShape(defaultCst)).isEqualTo(triviaShape(explicitOnCst)); } @Test - void defaultParserConfig_hasFlagFalse() { - assertThat(ParserConfig.DEFAULT.triviaPostPass()).isFalse(); + void defaultParserConfig_hasFlagTrue() { + assertThat(ParserConfig.DEFAULT.triviaPostPass()).isTrue(); } @Test - void parserConfigFactory_hasFlagFalse() { + void parserConfigFactory_hasFlagTrue() { assertThat(ParserConfig.parserConfig(true, RecoveryStrategy.BASIC, true) - .triviaPostPass()).isFalse(); + .triviaPostPass()).isTrue(); } } // =================================================================== // 2. Flag-on round-trip preservation // =================================================================== - @Nested @DisplayName("Flag-on round-trip preservation") class FlagOnRoundTrip { - @Test void simpleNumber_roundTrips() { assertRoundTrip(SIMPLE_GRAMMAR, " 42 "); @@ -160,39 +155,38 @@ void number_withLineComment_roundTrips() { } private void assertRoundTrip(String grammar, String input) { - var parser = PegParser.fromGrammar(grammar, configOn()) + var parser = PegParser.fromGrammar(grammar, + configOn()) .unwrap(); var cst = parser.parseCst(input) .unwrap(); assertThat(reconstruct(cst)).as("flag-on round-trip for %s", input) - .isEqualTo(input); + .isEqualTo(input); } } // =================================================================== // 3. Flag toggles attribution but preserves trivia text totals // =================================================================== - @Nested @DisplayName("Flag-on structural divergence with text preservation") class StructuralDivergence { - @Test void flagOnVsOff_listInput_totalTriviaTextIdentical() { var input = " alpha,\n beta,\n gamma "; - - var off = PegParser.fromGrammar(LIST_GRAMMAR, configOff()) + var off = PegParser.fromGrammar(LIST_GRAMMAR, + configOff()) .unwrap() .parseCst(input) .unwrap(); - var on = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + var on = PegParser.fromGrammar(LIST_GRAMMAR, + configOn()) .unwrap() .parseCst(input) .unwrap(); - // Pure permutation: total trivia text must match. assertThat(allTriviaText(on)).as("trivia text totals preserved") - .isEqualTo(allTriviaText(off)); + .isEqualTo(allTriviaText(off)); // Round-trip preserved on both sides. assertThat(reconstruct(off)).isEqualTo(input); assertThat(reconstruct(on)).isEqualTo(input); @@ -223,36 +217,33 @@ void flagOnVsOff_optionalTailGrammar_attributionDiffers() { %whitespace <- [ \\t\\n]+ """; var input = "42 "; - - var off = PegParser.fromGrammar(grammar, configOff()) + var off = PegParser.fromGrammar(grammar, + configOff()) .unwrap() .parseCst(input) .unwrap(); - var on = PegParser.fromGrammar(grammar, configOn()) + var on = PegParser.fromGrammar(grammar, + configOn()) .unwrap() .parseCst(input) .unwrap(); - assertThat(allTriviaText(on)).as("trivia text totals preserved") - .isEqualTo(allTriviaText(off)); + .isEqualTo(allTriviaText(off)); assertThat(reconstruct(on)).as("post-pass round-trip") - .isEqualTo(input); + .isEqualTo(input); assertThat(reconstruct(off)).as("engine round-trip") - .isEqualTo(input); - assertThat(triviaShape(off)).as( - "post-pass and engine differ on slot for Bug-C' empty-wrapper fixture") - .isNotEqualTo(triviaShape(on)); + .isEqualTo(input); + assertThat(triviaShape(off)).as("post-pass and engine differ on slot for Bug-C' empty-wrapper fixture") + .isNotEqualTo(triviaShape(on)); } } // =================================================================== // 4. parseRuleAt with the flag — Lever B unblocker claim // =================================================================== - @Nested @DisplayName("parseRuleAt honours the flag") class ParseRuleAtWithFlag { - /** * Lever B unblocker (Step 4 commit 2). When {@code parseRuleAt} * is invoked at a non-zero offset, the splice-aware overload {@link @@ -279,47 +270,45 @@ void parseRuleAt_flagOn_subtreeLeading_coversOnlySpliceGap() { // to the spliced subtree's leading — NOT the leading " " at // [0, 2) which belongs to the document root. int spliceOffset = 9; - - var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, + configOn()) .unwrap(); var partial = parserOn.parseRuleAt(Item.class, input, spliceOffset) .unwrap(); var subtree = partial.node(); - // The Item rule auto-skips %whitespace before matching, so the // matched span starts after the single leading space. assertThat(subtree.span() .start() - .offset()) - .as("subtree span starts after the gap whitespace") - .isEqualTo(10); - + .offset()).as("subtree span starts after the gap whitespace") + .isEqualTo(10); // Splice-aware leading: only the gap between spliceOffset (9) // and span.start (10) — a single space, NOT the input prefix. var leadingText = subtree.leadingTrivia() .stream() .map(Trivia::text) - .reduce("", (a, b) -> a + b); + .reduce("", + (a, b) -> a + b); assertThat(leadingText).as("leading covers only [spliceOffset, span.start)") - .isEqualTo(" "); + .isEqualTo(" "); assertThat(subtree.leadingTrivia()).hasSize(1); } @Test void parseRuleAt_flagOnVsOff_subtreeReconstructionIsConsistent() { var input = "alpha"; - var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + var parserOn = PegParser.fromGrammar(LIST_GRAMMAR, + configOn()) .unwrap(); - var parserOff = PegParser.fromGrammar(LIST_GRAMMAR, configOff()) + var parserOff = PegParser.fromGrammar(LIST_GRAMMAR, + configOff()) .unwrap(); - var on = parserOn.parseRuleAt(Item.class, input, 0) .unwrap() .node(); var off = parserOff.parseRuleAt(Item.class, input, 0) .unwrap() .node(); - // For a clean splice with no leading whitespace, both attributions // must agree on round-trip and total trivia text. assertThat(reconstruct(on)).isEqualTo(reconstruct(off)); @@ -330,7 +319,6 @@ void parseRuleAt_flagOnVsOff_subtreeReconstructionIsConsistent() { // =================================================================== // 5. Step 4 commit 3 — buffer no-op under flag-ON // =================================================================== - /** * Step 4 commit 3 validation: under flag-ON, the engine's pending-leading * trivia buffer ops in {@link org.pragmatica.peg.parser.ParsingContext} @@ -347,15 +335,14 @@ void parseRuleAt_flagOnVsOff_subtreeReconstructionIsConsistent() { @Nested @DisplayName("Buffer no-op under flag-ON (Step 4 commit 3)") class BufferNoOpUnderFlagOn { - @Test void flagOn_simpleNumber_postPassPopulatesLeadingDespiteBufferNoOp() { var input = " 42 "; - var parser = PegParser.fromGrammar(SIMPLE_GRAMMAR, configOn()) + var parser = PegParser.fromGrammar(SIMPLE_GRAMMAR, + configOn()) .unwrap(); var cst = parser.parseCst(input) .unwrap(); - // Round-trip = bytes preserved => post-pass attributed correctly // even though the engine's pending-leading buffer was a no-op. assertThat(reconstruct(cst)).isEqualTo(input); @@ -366,11 +353,11 @@ void flagOn_simpleNumber_postPassPopulatesLeadingDespiteBufferNoOp() { @Test void flagOn_listInput_postPassAttributesAllTriviaWithEmptyBuffer() { var input = " alpha,\n beta,\n gamma "; - var parser = PegParser.fromGrammar(LIST_GRAMMAR, configOn()) + var parser = PegParser.fromGrammar(LIST_GRAMMAR, + configOn()) .unwrap(); var cst = parser.parseCst(input) .unwrap(); - assertThat(reconstruct(cst)).isEqualTo(input); // All whitespace is attributed somewhere; commas are literal // terminals consumed by Tail and not part of trivia. @@ -380,21 +367,20 @@ void flagOn_listInput_postPassAttributesAllTriviaWithEmptyBuffer() { @Test void flagOn_commentedNumber_postPassClassifiesCommentsWithEmptyBuffer() { var input = "// preamble\n42 // trailer\n"; - var parser = PegParser.fromGrammar(COMMENT_GRAMMAR, configOn()) + var parser = PegParser.fromGrammar(COMMENT_GRAMMAR, + configOn()) .unwrap(); var cst = parser.parseCst(input) .unwrap(); - assertThat(reconstruct(cst)).isEqualTo(input); assertThat(allTriviaText(cst)).contains("// preamble") - .contains("// trailer"); + .contains("// trailer"); } } // =================================================================== // helpers (mirrors policy from TriviaPostPassTest) // =================================================================== - private static String reconstruct(CstNode root) { var sb = new StringBuilder(); emitLeading(sb, root); @@ -507,5 +493,4 @@ private static String triviaListText(List list) { return sb.append(']') .toString(); } - } From fc185db13883b1fa5cc25fefe5ef1a124de028e5 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 08:57:30 +0200 Subject: [PATCH 14/26] =?UTF-8?q?docs:=20HANDOVER=20+=20CHANGELOG=20?= =?UTF-8?q?=E2=80=94=20Step=204=20trivia=20rework=20COMPLETE=20(commits=20?= =?UTF-8?q?7=20fix=20+=206=20retry=20shipped)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 7 +++---- docs/HANDOVER.md | 36 ++++++++++++++++++++++++++++++++++-- 2 files changed, 37 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9db6f61..f937fd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,12 +20,11 @@ _Unreleased — patch cycle following 0.5.0._ - `ParsingContext` pending-trivia methods (`appendPendingLeadingTrivia`, `takePendingLeadingTrivia`, `savePendingLeadingTrivia`, `restorePendingLeadingTrivia`) early-out when `config.triviaPostPass()=true` — buffer becomes dead weight under the flag and is skipped (38 call sites in `PegEngine` no-op without source changes). - **Trivia rework Step 4 commit 5** (`4ed1cf5`): `ParserGenerator` emits no-op buffer methods under flag-ON. Mirror of commit 3 for the generator side — pure CPU optimization; correctness already from commit 4. Generated parsers under flag-ON no longer do dead-work buffer maintenance. -### Known Issues - -- **`triviaPostPass=true` flag has a known bug on production Java 25 corpus** (discovered 2026-05-09 during Step 4 commit 6 attempt). Round-trip reconstruction LOSES trivia text — not just slot-shifts attribution but actual character loss. 20 of 22 RoundTripTest fixtures fail under flag-ON via the generated parser path: `void allCompoundAssignments` becomes `voidallCompoundAssignments` etc. Bug likely lives in the embedded TriviaPostPass emission (~617 lines added to ParserGenerator in commit 4). The flag remains **opt-in only**; the default is `false`. Validation gap: Step 3 prototype tests excluded the `/large/` fixture and used small grammars only. Track work: HANDOVER §11 "Step 4 commit 5 landed; commit 6 attempted and reverted (BUG DISCOVERED)". - ### Fixed +- **Trivia post-pass orphan-trivia placement matches engine's Bug C' compensation** (`612fbea`). Earlier commit 6 attempt revealed text-loss on Java 25 corpus (`void m()` → `voidm()`). Root cause: post-pass attached orphan trivia to wrapper.trailingTrivia, but `CstReconstruct.emit` only emits trailing for last-child of a wrapper — non-last-wrapper-trailing was invisible during reconstruction. Fix mirrors engine's `attachTrailingToTail` — drains orphan trivia into deepest-rightmost-leaf descendant. Plus a generator-semantics adapter: `rebuildNonTerminal` probe-scans for caller-supplied leading and skips it to prevent license-header double-emission on fixtures whose wrappers' spans include their own leading trivia. RoundTripTest 22/22 under flag-ON across all 22 corpus fixtures including `large/FactoryClassGenerator.java.txt`. +- **`triviaPostPass` default flipped to `true`** (`3b372af`). After the fix, the default-flip succeeded with 0 test failures. Two sentinel tests in `TriviaPostPassFlagTest` inverted (DefaultOffNoOp → DefaultOnNoOp). Phase1/Phase2 parity tests insulated (explicitly pass `triviaPostPass=false`). The 8 fixture-pinning tests for the previous (buffer-driven) attribution remain green via explicit-OFF construction. **End-state for 0.5.1: post-pass attribution is the default; legacy buffer-driven attribution is opt-out via `new ParserConfig(..., triviaPostPass=false, ...)`.** + ### Removed ## [0.5.0] - 2026-05-06 diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 15ee7dc..7741dc4 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -512,7 +512,11 @@ After Step 3 verdict, the production rework arc executed in the same session. Br **Default behavior (flag-OFF) unchanged** — every pre-existing test passes byte-for-byte identically. Adoption is opt-in via `PegParser.builder(grammar).triviaPostPass(true).build()` or constructing `ParserConfig` with the flag set. -### Step 4 commit 5 landed; commit 6 attempted and reverted (BUG DISCOVERED) +### Step 4 COMPLETE — all commits 1-7 shipped including default flip + +(historical note: commit 6 was attempted, reverted, and successfully retried after commit 7 fixed the underlying bug — see commits 6 → 7 → 6-redo sequence below) + +### Step 4 commit 5 landed; commit 6 attempted and reverted (BUG DISCOVERED — later fixed in commit 7) **Commit 5 (`4ed1cf5`)** — ParserGenerator emits no-op buffer methods under flag-ON. Pure CPU optimization parallel to commit 3 for the interpreter. Tests stay at 768 + 1 skip. Generated parser under flag-ON is now lean (no dead buffer work) AND correct (post-pass produces attribution). @@ -554,6 +558,34 @@ This contradicts the post-pass spec doc claim: *"Total trivia text is preserved After bug fix + commit 6, Lever B can be retried: `parseRuleAt` + post-pass produces structurally identical subtrees, removing the trivia-context-loss blocker. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work. +### Step 4 commit 7 — bug fix shipped (`612fbea`) + +After the diagnosis identified two distinct bugs in the post-pass implementation, commit 7 fixed both: + +1. **Bug C' drain compensation** — TriviaPostPass.rebuildNonTerminal now mirrors engine's `attachTrailingToTail`: orphan trivia drains into deepest-rightmost-leaf descendant instead of being placed on the wrapper's trailingTrivia. `CstReconstruct.emit` correctly emits leaf trailings on every walk path; the wrapper-trailing was invisible for non-last-wrapper-children. ~70 lines in `TriviaPostPass.java` runtime + ~30 lines in the embedded version emitted by `ParserGenerator.java`. + +2. **Generator-semantics cursor adjustment** — separately discovered: the generated parser's `wrapWithRuleName` produces wrappers whose span INCLUDES their own leading trivia (`startOffset` captured BEFORE `skipWhitespace`). The post-pass's child-walk at first-child time was double-emitting the wrapper's leading bytes via the first child's leading scan. Fix: `rebuildNonTerminal` probe-scans `[spanStart, spanStart+leadingLen)` and, if it matches the caller-supplied leading exactly, advances the cursor past it. Symmetric to existing `rebuildRoot` adapter; only kicks in for generator-semantics CSTs. + +Validation: RoundTripTest 22/22 under flag-ON (was 2/22 pre-fix). All 768 peglib-core tests + 1 skip green at flag-OFF default. + +**Honest scope note (from agent report):** two bugs were fixed under one commit message. Could have split into 7a/7b, but both touch the same `rebuildNonTerminal` method and same root cause family (post-pass not mirroring engine span/trivia coupling). The cursor-adjustment hunk is cleanly separable in the diff if retrospective splitting is desired. + +### Step 4 commit 6 — default flip succeeded (`3b372af`) + +After commit 7 fixed the bugs, commit 6 was retried: `ParserConfig.DEFAULT.triviaPostPass` and the `parserConfig(...)` factory both flipped from `false` to `true`. Two sentinel tests in `TriviaPostPassFlagTest` inverted (DefaultOffNoOp → DefaultOnNoOp; assert-false → assert-true). + +**Result: 0 test failures, 0 errors, 0 surprises.** No Tier-A/B/C cascade. The 8 `Phase{1,2}*ParityTest` files (which explicitly pass `triviaPostPass=false`) provide the legacy-attribution regression net. Other tests in the project use the default and are bit-for-bit identical under post-pass attribution thanks to the fix. + +**End-state for 0.5.1:** post-pass attribution is the default; legacy buffer-driven attribution is opt-out via explicit `new ParserConfig(..., triviaPostPass=false, ...)`. The trivia issue that motivated this multi-session investigation is **closed**: attribution is now context-independent, parseRuleAt produces structurally identical subtrees to full-reparse (Lever B trivia-context-loss blocker is RESOLVED), and the buffer machinery still functions for backward-compat. + +**Lever B retry:** the trivia-context-loss blocker is gone. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work — that's the next-session entry point if pursuing incremental engine optimization. + +### Outstanding work (post-Step-4) + +- **Bench impact study** — measure post-pass overhead vs current attribution. Step 3 prototype was non-invasive (overlay); the rework wins back the buffer work but pays for the post-pass scan. Net likely positive but unmeasured. JMH bench A/B with `phase1_allStructural_mutableResult_autoSkipPackrat` variant under flag-ON vs flag-OFF on reference + selfhost fixtures would close this gap. +- **Lever B retry** — fallback-rule-bypass blocker work (orthogonal to trivia, separately scoped). +- **Lever C IR unification** — multi-week, maintainability-first. + ### Items superseded by this session's work - §6.2 lever-1 puzzle: dissolved by Path D's stable-id algorithm. @@ -565,4 +597,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 commits 1-5 of trivia production rework on `release-0.5.1` + commit 6 attempt with REVERT (bug discovered). Branch `release-0.5.1` at `4ed1cf5` — 9 commits past 0.5.1 base. peglib-core 768 tests green + 1 pre-existing skipped. Trivia rework state: **flag-ON path FUNCTIONAL on small grammars + adversarial inputs; KNOWN BUG on production Java 25 corpus (trivia text loss, not slot-shift). Default flip BLOCKED until bug fix.** Opt-in flag is shippable. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: diagnose + fix the embedded post-pass bug on Java 25 fixtures (likely in ParserGenerator's embedded TriviaPostPass emission, ~617 lines added in commit 4). Then reattempt commit 6. +**Last updated:** 2026-05-09, end of Step 4 trivia production rework — COMPLETE. Branch `release-0.5.1` at `3b372af` — 12 commits past 0.5.1 base. peglib-core 768 tests + 1 pre-existing skipped, all green at the new default (post-pass attribution). Trivia rework state: **SHIPPED. `triviaPostPass=true` is the default in 0.5.1. Legacy buffer-driven attribution is opt-out.** Bug discovered in commit 6 attempt was diagnosed (commit 6 reverted) and fixed (commit 7); commit 6 retried successfully. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: bench impact study (post-pass vs buffer-driven; both fixtures), Lever B retry now that trivia-context-loss blocker is RESOLVED, OR Lever C IR unification. From 667547989e4d4a62dd41e540f6db735a8a7f78a8 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 11:53:37 +0200 Subject: [PATCH 15/26] =?UTF-8?q?fix:=20TriviaPostPass=20uses=20line-start?= =?UTF-8?q?=20table=20to=20eliminate=20O(n=C2=B2)=20computeSpan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../peg/generator/ParserGenerator.java | 74 +++++++---- .../pragmatica/peg/tree/TriviaPostPass.java | 119 +++++++++++++----- 2 files changed, 132 insertions(+), 61 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 2d46268..133c95c 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -7104,9 +7104,35 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" \"leadingScanFrom \" + leadingScanFrom + \" out of range [0, \" + rootStart + \"]\");\n"); sb.append(" }\n"); sb.append(" }\n"); - sb.append(" return rebuildRoot(input, cst, leadingScanFrom);\n"); + sb.append(" // Build line-start table once per pass (O(N)) so that per-chunk\n"); + sb.append(" // computeSpan is O(log N) instead of O(from). Eliminates an O(N²)\n"); + sb.append(" // hotspot for whitespace-rich source.\n"); + sb.append(" int[] lineStarts = buildLineStarts(input);\n"); + sb.append(" return rebuildRoot(input, cst, lineStarts, leadingScanFrom);\n"); sb.append(" }\n\n"); - sb.append(" static List scanWhitespace(String input, int from, int to) {\n"); + sb.append(" private static int[] buildLineStarts(String input) {\n"); + sb.append(" int len = input.length();\n"); + sb.append(" int[] starts = new int[Math.max(8, len / 16 + 4)];\n"); + sb.append(" int n = 0;\n"); + sb.append(" starts[n++] = 0;\n"); + sb.append(" for (int i = 0; i < len; i++) {\n"); + sb.append(" if (input.charAt(i) == '\\n') {\n"); + sb.append(" if (n == starts.length) {\n"); + sb.append(" starts = java.util.Arrays.copyOf(starts, starts.length * 2);\n"); + sb.append(" }\n"); + sb.append(" starts[n++] = i + 1;\n"); + sb.append(" }\n"); + sb.append(" }\n"); + sb.append(" return java.util.Arrays.copyOf(starts, n);\n"); + sb.append(" }\n\n"); + sb.append(" private static int[] lineColAt(int[] lineStarts, int offset) {\n"); + sb.append(" int idx = java.util.Arrays.binarySearch(lineStarts, offset);\n"); + sb.append(" if (idx < 0) idx = -idx - 2;\n"); + sb.append(" int line = idx + 1;\n"); + sb.append(" int col = offset - lineStarts[idx] + 1;\n"); + sb.append(" return new int[]{line, col};\n"); + sb.append(" }\n\n"); + sb.append(" static List scanWhitespace(String input, int from, int to, int[] lineStarts) {\n"); sb.append(" if (from >= to) return List.of();\n"); if (!hasWhitespace) { sb.append(" return List.of();\n"); @@ -7117,7 +7143,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" int matched = matchWsInner(input, pos, to);\n"); sb.append(" if (matched < 0 || matched == pos) break;\n"); sb.append(" var text = input.substring(pos, matched);\n"); - sb.append(" var span = computeSpan(input, pos, matched);\n"); + sb.append(" var span = computeSpan(lineStarts, pos, matched);\n"); sb.append(" trivia.add(classify(span, text));\n"); sb.append(" pos = matched;\n"); sb.append(" }\n"); @@ -7135,7 +7161,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { // "leading-end" boundary AND leadingScanFrom + (rootStart - leadingScanFrom) // = rootStart for the leading scan. Otherwise (runtime semantics), // scan up to rootStart as the runtime post-pass does. - sb.append(" private static CstNode rebuildRoot(String input, CstNode root, int leadingScanFrom) {\n"); + sb.append(" private static CstNode rebuildRoot(String input, CstNode root, int[] lineStarts, int leadingScanFrom) {\n"); sb.append(" int rootStart = root.span().startOffset();\n"); sb.append(" int rootEnd = root.span().endOffset();\n"); sb.append(" int leadingTextLen = totalTriviaLength(root.leadingTrivia());\n"); @@ -7150,7 +7176,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" // Hypothesis: generator semantics — leading occupies [rootStart, rootStart + leadingTextLen).\n"); sb.append(" // Verify by re-scanning [rootStart, rootStart + leadingTextLen) and checking it consumes exactly that range.\n"); sb.append(" int probeEnd = rootStart + leadingTextLen;\n"); - sb.append(" var probe = scanWhitespace(input, rootStart, probeEnd);\n"); + sb.append(" var probe = scanWhitespace(input, rootStart, probeEnd, lineStarts);\n"); sb.append(" if (totalTriviaLength(probe) == leadingTextLen) {\n"); sb.append(" leadingScanEnd = probeEnd;\n"); sb.append(" } else {\n"); @@ -7159,19 +7185,19 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" } else {\n"); sb.append(" leadingScanEnd = rootStart;\n"); sb.append(" }\n"); - sb.append(" var leading = scanWhitespace(input, leadingScanFrom, leadingScanEnd);\n"); - sb.append(" var trailingExternal = scanWhitespace(input, rootEnd, input.length());\n"); - sb.append(" return rebuildSelf(input, root, leading, trailingExternal);\n"); + sb.append(" var leading = scanWhitespace(input, leadingScanFrom, leadingScanEnd, lineStarts);\n"); + sb.append(" var trailingExternal = scanWhitespace(input, rootEnd, input.length(), lineStarts);\n"); + sb.append(" return rebuildSelf(input, root, lineStarts, leading, trailingExternal);\n"); sb.append(" }\n\n"); sb.append(" private static int totalTriviaLength(List trivia) {\n"); sb.append(" int total = 0;\n"); sb.append(" for (var t : trivia) total += t.text().length();\n"); sb.append(" return total;\n"); sb.append(" }\n\n"); - sb.append(" private static CstNode rebuildSelf(String input, CstNode node,\n"); + sb.append(" private static CstNode rebuildSelf(String input, CstNode node, int[] lineStarts,\n"); sb.append(" List leading, List extraTrailing) {\n"); sb.append(" return switch (node) {\n"); - sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, leading, extraTrailing);\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, extraTrailing);\n"); sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing);\n"); sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing);\n"); if (errorReporting == ErrorReporting.ADVANCED) { @@ -7179,11 +7205,11 @@ private void generateCstTriviaPostPass(StringBuilder sb) { } sb.append(" };\n"); sb.append(" }\n\n"); - sb.append(" private static CstNode rebuildChild(String input, CstNode child, int prevEnd) {\n"); + sb.append(" private static CstNode rebuildChild(String input, CstNode child, int[] lineStarts, int prevEnd) {\n"); sb.append(" int childStart = child.span().startOffset();\n"); - sb.append(" var leading = scanWhitespace(input, prevEnd, childStart);\n"); + sb.append(" var leading = scanWhitespace(input, prevEnd, childStart, lineStarts);\n"); sb.append(" return switch (child) {\n"); - sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, leading, List.of());\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, List.of());\n"); sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of());\n"); sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of());\n"); if (errorReporting == ErrorReporting.ADVANCED) { @@ -7191,7 +7217,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { } sb.append(" };\n"); sb.append(" }\n\n"); - sb.append(" private static CstNode.NonTerminal rebuildNonTerminal(String input, CstNode.NonTerminal nt,\n"); + sb.append(" private static CstNode.NonTerminal rebuildNonTerminal(String input, CstNode.NonTerminal nt, int[] lineStarts,\n"); sb.append(" List leading, List extraTrailing) {\n"); sb.append(" int spanStart = nt.span().startOffset();\n"); sb.append(" int spanEnd = nt.span().endOffset();\n"); @@ -7203,21 +7229,21 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" int leadingLen = totalTriviaLength(leading);\n"); sb.append(" int cursor = spanStart;\n"); sb.append(" if (leadingLen > 0 && spanStart + leadingLen <= input.length()) {\n"); - sb.append(" var probe = scanWhitespace(input, spanStart, spanStart + leadingLen);\n"); + sb.append(" var probe = scanWhitespace(input, spanStart, spanStart + leadingLen, lineStarts);\n"); sb.append(" if (totalTriviaLength(probe) == leadingLen) {\n"); sb.append(" cursor = spanStart + leadingLen;\n"); sb.append(" }\n"); sb.append(" }\n"); sb.append(" var newChildren = new ArrayList(nt.children().size());\n"); sb.append(" for (var c : nt.children()) {\n"); - sb.append(" var rebuilt = rebuildChild(input, c, cursor);\n"); + sb.append(" var rebuilt = rebuildChild(input, c, lineStarts, cursor);\n"); sb.append(" newChildren.add(rebuilt);\n"); sb.append(" cursor = c.span().endOffset();\n"); sb.append(" }\n"); sb.append(" // Bug C' compensation — drain orphan trailing into the last terminal\n"); sb.append(" // descendant so CstReconstruct.emit (last-child-trailing-only) stays\n"); sb.append(" // byte-equal even when this wrapper is not its parent's last child.\n"); - sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd);\n"); + sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd, lineStarts);\n"); sb.append(" List wrapperTrailing = extraTrailing;\n"); sb.append(" if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) {\n"); sb.append(" int lastIdx = newChildren.size() - 1;\n"); @@ -7326,16 +7352,10 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" }\n"); sb.append(" return false;\n"); sb.append(" }\n\n"); - sb.append(" private static SourceSpan computeSpan(String input, int from, int to) {\n"); - sb.append(" int line = 1, col = 1;\n"); - sb.append(" for (int i = 0; i < from; i++) {\n"); - sb.append(" if (input.charAt(i) == '\\n') { line++; col = 1; } else { col++; }\n"); - sb.append(" }\n"); - sb.append(" int startLine = line, startCol = col;\n"); - sb.append(" for (int i = from; i < to; i++) {\n"); - sb.append(" if (input.charAt(i) == '\\n') { line++; col = 1; } else { col++; }\n"); - sb.append(" }\n"); - sb.append(" return new SourceSpan(startLine, startCol, from, line, col, to);\n"); + sb.append(" private static SourceSpan computeSpan(int[] lineStarts, int from, int to) {\n"); + sb.append(" int[] startLc = lineColAt(lineStarts, from);\n"); + sb.append(" int[] endLc = lineColAt(lineStarts, to);\n"); + sb.append(" return new SourceSpan(startLc[0], startLc[1], from, endLc[0], endLc[1], to);\n"); sb.append(" }\n\n"); // Emit per-expression matcher methods. if (hasWhitespace) { diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index ca07054..9b2bfb4 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -5,6 +5,7 @@ import org.pragmatica.peg.grammar.analysis.ExpressionShape; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; /** @@ -119,7 +120,38 @@ public static CstNode assignTrivia(String input, CstNode cst, Grammar grammar, i throw new IllegalArgumentException( "leadingScanFrom " + leadingScanFrom + " out of range [0, " + rootStart + "]"); } - return rebuildRoot(input, cst, grammar, leadingScanFrom); + // Pre-compute line-start table once per pass — O(N). Eliminates + // O(N²) re-scan that the previous per-chunk computeSpan performed. + int[] lineStarts = buildLineStarts(input); + return rebuildRoot(input, cst, grammar, lineStarts, leadingScanFrom); + } + + /** Build a 1-based line-start offset table. {@code lineStarts[0] = 0} (line 1). */ + private static int[] buildLineStarts(String input) { + int len = input.length(); + // Worst case: every char is '\n' → len+1 entries. Common case: ~1 per ~30 chars. + var starts = new int[Math.max(8, len / 16 + 4)]; + int n = 0; + starts[n++ ] = 0; + for (int i = 0; i < len; i++ ) { + if (input.charAt(i) == '\n') { + if (n == starts.length) { + starts = Arrays.copyOf(starts, starts.length * 2); + } + starts[n++ ] = i + 1; + } + } + return Arrays.copyOf(starts, n); + } + + /** Find {@code [line, col]} (1-based) for {@code offset} via binary search into {@code lineStarts}. */ + private static int[] lineColAt(int[] lineStarts, int offset) { + int idx = Arrays.binarySearch(lineStarts, offset); + if (idx < 0) idx = - idx - 2; + // idx is now the index of the greatest lineStart <= offset. + int line = idx + 1; + int col = offset - lineStarts[idx] + 1; + return new int[]{line, col}; } /** @@ -133,6 +165,26 @@ public static List scanWhitespace(String input, int from, int to, Gramma .isEmpty()) { return List.of(); } + // Public API: build a fresh line-start table for one-off callers. + // Internal users (assignTrivia recursion) call scanWhitespaceFast + // with a shared, pre-built lineStarts array. + return scanWhitespaceFast(input, from, to, grammar, buildLineStarts(input)); + } + + /** + * Internal fast variant of {@link #scanWhitespace}: takes a precomputed + * line-start table to avoid the O(N²) re-scan-from-zero that the previous + * implementation incurred for every trivia chunk. + */ + private static List scanWhitespaceFast(String input, + int from, + int to, + Grammar grammar, + int[] lineStarts) { + if (from >= to || grammar.whitespace() + .isEmpty()) { + return List.of(); + } var inner = ExpressionShape.extractInnerExpression(grammar.whitespace() .unwrap()); var trivia = new ArrayList(); @@ -143,7 +195,7 @@ public static List scanWhitespace(String input, int from, int to, Gramma break; } var text = input.substring(pos, matched); - var span = computeSpan(input, pos, matched); + var span = computeSpanFast(lineStarts, pos, matched); trivia.add(classify(span, text)); pos = matched; } @@ -151,16 +203,20 @@ public static List scanWhitespace(String input, int from, int to, Gramma } // === Tree rebuild === - private static CstNode rebuildRoot(String input, CstNode root, Grammar grammar, int leadingScanFrom) { + private static CstNode rebuildRoot(String input, + CstNode root, + Grammar grammar, + int[] lineStarts, + int leadingScanFrom) { int rootStart = root.span() .startOffset(); int rootEnd = root.span() .endOffset(); - var leading = scanWhitespace(input, leadingScanFrom, rootStart, grammar); + var leading = scanWhitespaceFast(input, leadingScanFrom, rootStart, grammar, lineStarts); // Root trailing covers the gap from rootEnd to end-of-input (matches // current engine behaviour: top-level wrapper carries post-EOF trivia). - var trailingExternal = scanWhitespace(input, rootEnd, input.length(), grammar); - return rebuildSelf(input, root, grammar, leading, trailingExternal); + var trailingExternal = scanWhitespaceFast(input, rootEnd, input.length(), grammar, lineStarts); + return rebuildSelf(input, root, grammar, lineStarts, leading, trailingExternal); } /** @@ -172,10 +228,11 @@ private static CstNode rebuildRoot(String input, CstNode root, Grammar grammar, private static CstNode rebuildSelf(String input, CstNode node, Grammar grammar, + int[] lineStarts, List leading, List extraTrailing) { return switch (node) { - case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, leading, extraTrailing); + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, lineStarts, leading, extraTrailing); case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing); case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing); case CstNode.Error e -> new CstNode.Error(e.id(), @@ -188,12 +245,16 @@ private static CstNode rebuildSelf(String input, } /** Rebuild a non-root child given its preceding sibling's end offset. */ - private static CstNode rebuildChild(String input, CstNode child, Grammar grammar, int prevEnd) { + private static CstNode rebuildChild(String input, + CstNode child, + Grammar grammar, + int[] lineStarts, + int prevEnd) { int childStart = child.span() .startOffset(); - var leading = scanWhitespace(input, prevEnd, childStart, grammar); + var leading = scanWhitespaceFast(input, prevEnd, childStart, grammar, lineStarts); return switch (child) { - case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, leading, List.of()); + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, lineStarts, leading, List.of()); case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of()); case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of()); case CstNode.Error e -> new CstNode.Error(e.id(), @@ -208,6 +269,7 @@ private static CstNode rebuildChild(String input, CstNode child, Grammar grammar private static CstNode.NonTerminal rebuildNonTerminal(String input, CstNode.NonTerminal nt, Grammar grammar, + int[] lineStarts, List leading, List extraTrailing) { int spanStart = nt.span() @@ -218,7 +280,7 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, .size()); int cursor = spanStart; for (var c : nt.children()) { - var rebuilt = rebuildChild(input, c, grammar, cursor); + var rebuilt = rebuildChild(input, c, grammar, lineStarts, cursor); newChildren.add(rebuilt); cursor = c.span() .endOffset(); @@ -230,7 +292,7 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, // the wrapper's last-child trailing slot. Without this, a wrapper // that is itself a non-last sibling would lose its trailing during // CstReconstruct.emit. - var internalTrailing = scanWhitespace(input, cursor, spanEnd, grammar); + var internalTrailing = scanWhitespaceFast(input, cursor, spanEnd, grammar, lineStarts); List wrapperTrailing = extraTrailing; if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) { int lastIdx = newChildren.size() - 1; @@ -317,30 +379,19 @@ private static Trivia classify(SourceSpan span, String text) { // === Span computation === /** - * Compute a {@link SourceSpan} for {@code input[from..to)} by counting - * newlines from offset 0. Line/column are 1-based to match + * Compute a {@link SourceSpan} for {@code input[from..to)} using a + * precomputed line-start table. Line/column are 1-based to match * {@link SourceLocation#START}. + * + *

Replaces the previous O(from)-per-call implementation that re-scanned + * {@code [0, from)} for every trivia chunk, producing O(N²) total work in + * whitespace-rich source. With a precomputed table built once per pass + * (O(N)), this is O(log N) per call via binary search. */ - private static SourceSpan computeSpan(String input, int from, int to) { - int line = 1, col = 1; - for (int i = 0; i < from; i++ ) { - if (input.charAt(i) == '\n') { - line++ ; - col = 1; - }else { - col++ ; - } - } - int startLine = line, startCol = col; - for (int i = from; i < to; i++ ) { - if (input.charAt(i) == '\n') { - line++ ; - col = 1; - }else { - col++ ; - } - } - return new SourceSpan(startLine, startCol, from, line, col, to); + private static SourceSpan computeSpanFast(int[] lineStarts, int from, int to) { + int[] startLc = lineColAt(lineStarts, from); + int[] endLc = lineColAt(lineStarts, to); + return new SourceSpan(startLc[0], startLc[1], from, endLc[0], endLc[1], to); } // === Mini-PEG matcher for %whitespace === From b05a57c72c86d8f59016e6affc804af806672ad4 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 11:55:29 +0200 Subject: [PATCH 16/26] =?UTF-8?q?docs:=20HANDOVER=20+=20CHANGELOG=20?= =?UTF-8?q?=E2=80=94=20bench=20impact=20study=20+=20O(n=C2=B2)=20fix=20ban?= =?UTF-8?q?ked?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 1 + docs/HANDOVER.md | 37 +++++++++++++++++++++++++++++++++---- 2 files changed, 34 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f937fd4..03f6a70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,7 @@ _Unreleased — patch cycle following 0.5.0._ - **Trivia post-pass orphan-trivia placement matches engine's Bug C' compensation** (`612fbea`). Earlier commit 6 attempt revealed text-loss on Java 25 corpus (`void m()` → `voidm()`). Root cause: post-pass attached orphan trivia to wrapper.trailingTrivia, but `CstReconstruct.emit` only emits trailing for last-child of a wrapper — non-last-wrapper-trailing was invisible during reconstruction. Fix mirrors engine's `attachTrailingToTail` — drains orphan trivia into deepest-rightmost-leaf descendant. Plus a generator-semantics adapter: `rebuildNonTerminal` probe-scans for caller-supplied leading and skips it to prevent license-header double-emission on fixtures whose wrappers' spans include their own leading trivia. RoundTripTest 22/22 under flag-ON across all 22 corpus fixtures including `large/FactoryClassGenerator.java.txt`. - **`triviaPostPass` default flipped to `true`** (`3b372af`). After the fix, the default-flip succeeded with 0 test failures. Two sentinel tests in `TriviaPostPassFlagTest` inverted (DefaultOffNoOp → DefaultOnNoOp). Phase1/Phase2 parity tests insulated (explicitly pass `triviaPostPass=false`). The 8 fixture-pinning tests for the previous (buffer-driven) attribution remain green via explicit-OFF construction. **End-state for 0.5.1: post-pass attribution is the default; legacy buffer-driven attribution is opt-out via `new ParserConfig(..., triviaPostPass=false, ...)`.** +- **Post-pass O(n²) bench regression fixed** (`6675479`). Initial A/B bench after default flip surfaced 22× / 239× wallclock regression on reference / selfhost fixtures. Investigation pinpointed `computeSpan(input, from, to)` re-scanning `[0, from)` from offset 0 on every trivia chunk to count newlines: O(K · N) where K trivia chunks scale with N input chars → O(N²). Empirical exponent reached 1.88 at N=32000. Fix: precompute a line-start table once per `assignTrivia` call (O(N)) + binary search per `computeSpan` (O(log N)). Both runtime `TriviaPostPass.java` and the embedded version emitted by `ParserGenerator.java` updated symmetrically. Post-fix bench: reference 26.87 ms (1.36× legacy 19.78 ms); selfhost 943.6 ms (1.01× legacy 934.7 ms — within noise). 15.6× speedup on reference, 210× on selfhost vs pre-fix. ### Removed diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 7741dc4..bb615bd 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -580,11 +580,40 @@ After commit 7 fixed the bugs, commit 6 was retried: `ParserConfig.DEFAULT.trivi **Lever B retry:** the trivia-context-loss blocker is gone. The orthogonal fallback-rule-bypass blocker (§6.2 root cause #1) still requires separate work — that's the next-session entry point if pursuing incremental engine optimization. -### Outstanding work (post-Step-4) +### Bench impact study (2026-05-09) — DONE; one bug fixed in flight -- **Bench impact study** — measure post-pass overhead vs current attribution. Step 3 prototype was non-invasive (overlay); the rework wins back the buffer work but pays for the post-pass scan. Net likely positive but unmeasured. JMH bench A/B with `phase1_allStructural_mutableResult_autoSkipPackrat` variant under flag-ON vs flag-OFF on reference + selfhost fixtures would close this gap. -- **Lever B retry** — fallback-rule-bypass blocker work (orthogonal to trivia, separately scoped). +After Step 4 default flip, A/B bench (reference + selfhost, JMH 5/3, gc profile) revealed a critical regression: + +| Fixture | Buffer-driven | Post-pass (default) | Δ% | +|---|---:|---:|---:| +| Reference (1900 LOC) | 19.78 ms | **418 ms** | +2,013% | +| Selfhost (37k LOC) | 934.7 ms | **198,679 ms** | +21,162% | + +Investigation: `computeSpan(input, from, to)` re-scanned `[0, from)` from offset 0 on every trivia chunk → O(K · N) → O(N²). Empirical exponent 1.88 at N=32000. RoundTripTest's correctness gate didn't catch this because tests run to completion without time limits. + +**Fix shipped** at `6675479` — line-start table approach. O(N) one-shot precompute + O(log N) binary search per chunk. Both runtime and emitted versions updated. + +Post-fix bench: + +| Fixture | Buffer-driven | Post-pass (post-fix) | Δ% | +|---|---:|---:|---:| +| Reference (1900 LOC) | 19.78 ms | 26.87 ms | +35.8% | +| Selfhost (37k LOC) | 934.7 ms | 943.6 ms | +0.95% (within noise) | + +Reference is 36% slower than legacy; selfhost is at parity. The post-pass overhead is real on small inputs (one tree-walk) but amortizes for large inputs where buffer save/restore dominated. **Acceptable as default**; opt-out remains available via explicit `ParserConfig` constructor. + +Allocation rate is essentially flat (within ~16% of legacy on selfhost; line-start table + per-call int[2] tuples cost some bytes but no longer drive wallclock). + +**Lessons banked:** +- Correctness tests should NOT be the only guard against perf regressions. RoundTripTest passed all 22 fixtures under flag-ON before the bench revealed the problem. +- Bench A/B mandatory before any default-changing commit. +- The diagnosis approach (read code → identify suspected hotspot → empirical timing harness → confirm with magnitude analysis) generalized well: the smoking gun was found in 30 minutes; the fix was 30 LOC. + +### Outstanding work (post-Step-4 + bench-fix) + +- **Lever B retry** — fallback-rule-bypass blocker work (orthogonal to trivia, separately scoped). The trivia-context-loss blocker is now fully RESOLVED. - **Lever C IR unification** — multi-week, maintainability-first. +- **Tighten reference fixture overhead** — post-pass is 36% slower on small inputs. If important, address H1 (`attachTrailingToTail` spine rebuild) and/or H4 (probe-scan in rebuildNonTerminal). Probably not worth chasing; selfhost (the actual perf-critical workload) is at parity. ### Items superseded by this session's work @@ -597,4 +626,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 trivia production rework — COMPLETE. Branch `release-0.5.1` at `3b372af` — 12 commits past 0.5.1 base. peglib-core 768 tests + 1 pre-existing skipped, all green at the new default (post-pass attribution). Trivia rework state: **SHIPPED. `triviaPostPass=true` is the default in 0.5.1. Legacy buffer-driven attribution is opt-out.** Bug discovered in commit 6 attempt was diagnosed (commit 6 reverted) and fixed (commit 7); commit 6 retried successfully. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: bench impact study (post-pass vs buffer-driven; both fixtures), Lever B retry now that trivia-context-loss blocker is RESOLVED, OR Lever C IR unification. +**Last updated:** 2026-05-09, end of Step 4 trivia production rework + bench impact study + O(n²) fix. Branch `release-0.5.1` at `6675479` — 14 commits past 0.5.1 base. peglib-core 768 tests + 1 pre-existing skipped, all green at the new default (post-pass attribution). **Trivia rework state: SHIPPED + bench-validated.** `triviaPostPass=true` is default; selfhost wallclock at parity with legacy (within 1%); reference 36% slower (acceptable). Legacy buffer-driven attribution is opt-out. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: Lever B retry (fallback-rule-bypass blocker remains; trivia-context-loss blocker is RESOLVED) OR Lever C IR unification. From 763e6e09e18f3adf03adbc67fefab71ea84ead64 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 12:23:32 +0200 Subject: [PATCH 17/26] perf: skip buffer call sites under triviaPostPass flag-ON (Cleanup A) --- .../peg/generator/ParserGenerator.java | 405 +++++++++++++----- .../org/pragmatica/peg/parser/PegEngine.java | 165 +++++-- 2 files changed, 418 insertions(+), 152 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 133c95c..5f29d1f 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -279,6 +279,76 @@ private String expectedUnwrap(String varName) { : varName + ".expected.unwrap()"; } + /** + * Cleanup A: returns the inline expression for {@code takePendingLeading()}. + * Under {@code triviaPostPass=true}, the helper is a no-op returning + * {@code List.of()}; emitting the literal here at gen-time eliminates the + * dead method dispatch at runtime. + */ + private String takePendingExpr() { + return config.triviaPostPass() + ? "List.of()" + : "takePendingLeading()"; + } + + /** + * Cleanup A: returns the inline expression for {@code savePendingLeading()}. + * Under flag-ON, save returns {@code 0} (the sentinel); we just emit it. + */ + private String savePendingExpr() { + return config.triviaPostPass() + ? "0" + : "savePendingLeading()"; + } + + /** + * Cleanup A: emit a {@code restorePendingLeading()} statement. + * Under flag-ON, emits nothing — the call would be a no-op anyway. + */ + private void emitRestorePending(StringBuilder sb, String indent, String snapshotVar) { + if (config.triviaPostPass()) { + return; + } + sb.append(indent) + .append("restorePendingLeading(") + .append(snapshotVar) + .append(");\n"); + } + + /** + * Cleanup A: emit an {@code appendPending()} statement, evaluating + * {@code expr} eagerly (so any side-effect like {@code skipWhitespace()} + * still runs). Under flag-ON, emits {@code expr;} (a statement) so the + * skip still occurs but the buffer append is elided. + */ + private void emitAppendPendingFromExpr(StringBuilder sb, String indent, String expr) { + if (config.triviaPostPass()) { + sb.append(indent) + .append(expr) + .append(";\n"); + return; + } + sb.append(indent) + .append("appendPending(") + .append(expr) + .append(");\n"); + } + + /** + * Cleanup A: emit a {@code pendingLeadingTrivia.addAll()} statement + * for the rule-failure re-deposit path. Under flag-ON, the generated + * parser's pendingLeadingTrivia field is unused; skip the statement. + */ + private void emitPendingAddAll(StringBuilder sb, String indent, String expr) { + if (config.triviaPostPass()) { + return; + } + sb.append(indent) + .append("pendingLeadingTrivia.addAll(") + .append(expr) + .append(");\n"); + } + public static ParserGenerator parserGenerator(Grammar grammar, String packageName, String className) { return new ParserGenerator(grammar, packageName, className, ErrorReporting.BASIC, ParserConfig.DEFAULT); } @@ -3023,7 +3093,9 @@ private void generateCstRuleMethod(StringBuilder sb, Rule rule, int ruleId) { sb.append(" var cached = cache.get(key);\n"); sb.append(" if (cached != null) {\n"); sb.append(" if (cached.isSuccess()) {\n"); - sb.append(" var hitCarriedLeading = takePendingLeading();\n"); + sb.append(" var hitCarriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var hitLocalLeading = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var hitLeading = concatTrivia(hitCarriedLeading, hitLocalLeading);\n"); sb.append(" var hitEndLoc = ") @@ -3043,7 +3115,9 @@ private void generateCstRuleMethod(StringBuilder sb, Rule rule, int ruleId) { sb.append(" \n"); } sb.append(" // Skip leading whitespace and combine with carried pending-leading.\n"); - sb.append(" var carriedLeading = takePendingLeading();\n"); + sb.append(" var carriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var localLeadingTrivia = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var leadingTrivia = concatTrivia(carriedLeading, localLeadingTrivia);\n"); var ruleIdConst = toConstantName(ruleName); @@ -3100,7 +3174,11 @@ private void generateCstRuleMethod(StringBuilder sb, Rule rule, int ruleId) { // child (typically inter-element skipWhitespace before a zero-width tail // element). Pos is NOT rewound — predicate combinators rely on pos being // past consumed whitespace. Mirrors PegEngine.parseRule. - sb.append(" var pendingAtExit = takePendingLeading();\n"); + // Cleanup A: under triviaPostPass=true, takePendingLeading returns an + // empty list, so the trailing-attach block is statically dead — elide. + if (!config.triviaPostPass()) { + sb.append(" var pendingAtExit = takePendingLeading();\n"); + } // Bug C fix: cache the empty-leading wrapped node so cache hits don't // preserve stale leading trivia and duplicate it on outer nodes. The // returned node carries the actual leadingTrivia for this call site. @@ -3111,10 +3189,12 @@ private void generateCstRuleMethod(StringBuilder sb, Rule rule, int ruleId) { sb.append(" var node = wrapWithRuleName(result, children, span, ") .append(ruleIdConst) .append(", leadingTrivia);\n"); - sb.append(" if (!pendingAtExit.isEmpty()) {\n"); - sb.append(" cacheNode = attachTrailingToTail(cacheNode, pendingAtExit);\n"); - sb.append(" node = attachTrailingToTail(node, pendingAtExit);\n"); - sb.append(" }\n"); + if (!config.triviaPostPass()) { + sb.append(" if (!pendingAtExit.isEmpty()) {\n"); + sb.append(" cacheNode = attachTrailingToTail(cacheNode, pendingAtExit);\n"); + sb.append(" node = attachTrailingToTail(node, pendingAtExit);\n"); + sb.append(" }\n"); + } sb.append(" cacheableResult = CstParseResult.success(cacheNode, ") .append(textOrEmpty("result")) .append(", endLoc);\n"); @@ -3131,8 +3211,11 @@ private void generateCstRuleMethod(StringBuilder sb, Rule rule, int ruleId) { sb.append(" restoreLocation(startLoc);\n"); } // Re-deposit the caller's pending-leading so enclosing backtracking - // combinators can roll it back to their own snapshots. - sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + // combinators can roll it back to their own snapshots. Cleanup A: + // under flag-ON, the buffer is unused; skip the addAll entirely. + if (!config.triviaPostPass()) { + sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + } // Use custom error message if available if (rule.hasErrorMessage()) { sb.append(" finalResult = CstParseResult.failure(\"") @@ -3204,7 +3287,9 @@ private void emitCstLeftRecursiveWrapper(StringBuilder sb, Rule rule, int ruleId sb.append(" var cached = cache.get(key);\n"); sb.append(" if (cached != null) {\n"); sb.append(" if (cached.isSuccess()) {\n"); - sb.append(" var hitCarriedLeading = takePendingLeading();\n"); + sb.append(" var hitCarriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var hitLocalLeading = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var hitLeading = concatTrivia(hitCarriedLeading, hitLocalLeading);\n"); sb.append(" var hitEndLoc = ") @@ -3233,7 +3318,9 @@ private void emitCstLeftRecursiveWrapper(StringBuilder sb, Rule rule, int ruleId sb.append(" return activeSeed;\n"); sb.append(" }\n"); // Capture leading whitespace & carried pending at the outer level. - sb.append(" var carriedLeading = takePendingLeading();\n"); + sb.append(" var carriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var localLeadingTrivia = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var leadingTrivia = concatTrivia(carriedLeading, localLeadingTrivia);\n"); sb.append(" int bodyStartPos = pos;\n"); @@ -3317,7 +3404,9 @@ private void emitCstLeftRecursiveWrapper(StringBuilder sb, Rule rule, int ruleId }else { sb.append(" restoreLocation(startLoc);\n"); } - sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + if (!config.triviaPostPass()) { + sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + } if (rule.hasErrorMessage()) { sb.append(" finalResult = CstParseResult.failure(\"") .append(escape(rule.errorMessage() @@ -3439,7 +3528,9 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi sb.append(" var cached = cache.get(key);\n"); sb.append(" if (cached != null) {\n"); sb.append(" if (cached.isSuccess()) {\n"); - sb.append(" var hitCarriedLeading = takePendingLeading();\n"); + sb.append(" var hitCarriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var hitLocalLeading = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var hitLeading = concatTrivia(hitCarriedLeading, hitLocalLeading);\n"); sb.append(" var hitEndLoc = ") @@ -3458,7 +3549,9 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi sb.append(" }\n"); sb.append(" \n"); } - sb.append(" var carriedLeading = takePendingLeading();\n"); + sb.append(" var carriedLeading = ") + .append(takePendingExpr()) + .append(";\n"); sb.append(" var localLeadingTrivia = (tokenBoundaryDepth > 0) ? List.of() : skipWhitespace();\n"); sb.append(" var leadingTrivia = concatTrivia(carriedLeading, localLeadingTrivia);\n"); sb.append(" var children = new ArrayList();\n"); @@ -3478,7 +3571,13 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi sb.append(" int choiceStart0Pos = pos;\n"); sb.append(" int choiceStart0Line = line;\n"); sb.append(" int choiceStart0Column = column;\n"); - sb.append(" int choicePending0 = savePendingLeading();\n"); + // Cleanup A: under triviaPostPass=true, savePendingLeading() returns 0 + // (sentinel) and restorePendingLeading is a no-op; we still emit the + // declaration since helpers receive choicePending0 as a parameter, and + // the JIT trivially elides the dead local. + sb.append(" int choicePending0 = ") + .append(savePendingExpr()) + .append(";\n"); var childrenStateName = config.markResetChildren() ? "childrenMark0" : "savedChildren0"; @@ -3519,7 +3618,9 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi .append(childrenStateName) .append(");\n"); } - sb.append(" restorePendingLeading(choicePending0);\n"); + if (!config.triviaPostPass()) { + sb.append(" restorePendingLeading(choicePending0);\n"); + } sb.append(" result = CstParseResult.failure(\"one of alternatives\");\n"); sb.append(" }\n"); sb.append(" \n"); @@ -3544,17 +3645,21 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi }else { sb.append(" var span = SourceSpan.sourceSpan(startLoc, endLoc);\n"); } - sb.append(" var pendingAtExit = takePendingLeading();\n"); + if (!config.triviaPostPass()) { + sb.append(" var pendingAtExit = takePendingLeading();\n"); + } sb.append(" var cacheNode = wrapWithRuleName(result, children, span, ") .append(ruleIdConst) .append(", List.of());\n"); sb.append(" var node = wrapWithRuleName(result, children, span, ") .append(ruleIdConst) .append(", leadingTrivia);\n"); - sb.append(" if (!pendingAtExit.isEmpty()) {\n"); - sb.append(" cacheNode = attachTrailingToTail(cacheNode, pendingAtExit);\n"); - sb.append(" node = attachTrailingToTail(node, pendingAtExit);\n"); - sb.append(" }\n"); + if (!config.triviaPostPass()) { + sb.append(" if (!pendingAtExit.isEmpty()) {\n"); + sb.append(" cacheNode = attachTrailingToTail(cacheNode, pendingAtExit);\n"); + sb.append(" node = attachTrailingToTail(node, pendingAtExit);\n"); + sb.append(" }\n"); + } sb.append(" cacheableResult = CstParseResult.success(cacheNode, ") .append(textOrEmpty("result")) .append(", endLoc);\n"); @@ -3569,7 +3674,9 @@ private void emitCstChoiceRule(StringBuilder sb, Rule rule, int ruleId, Expressi }else { sb.append(" restoreLocation(startLoc);\n"); } - sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + if (!config.triviaPostPass()) { + sb.append(" if (!carriedLeading.isEmpty()) pendingLeadingTrivia.addAll(carriedLeading);\n"); + } if (rule.hasErrorMessage()) { sb.append(" finalResult = CstParseResult.failure(\"") .append(escape(rule.errorMessage() @@ -3639,7 +3746,9 @@ private void emitChoiceRuleDispatcherF(StringBuilder sb, sb.append(" this.pos = choiceStart0Pos;\n"); sb.append(" this.line = choiceStart0Line;\n"); sb.append(" this.column = choiceStart0Column;\n"); - sb.append(" restorePendingLeading(choicePending0);\n"); + if (!config.triviaPostPass()) { + sb.append(" restorePendingLeading(choicePending0);\n"); + } sb.append(" result = null;\n"); emitHelperCallChain(sb, rule, defaults, childrenStateName, " "); sb.append(" }\n"); @@ -3771,7 +3880,9 @@ private void emitCstChoiceAltHelper(StringBuilder sb, Rule rule, Expression alt, sb.append(" this.pos = choiceStartPos;\n"); sb.append(" this.line = choiceStartLine;\n"); sb.append(" this.column = choiceStartColumn;\n"); - sb.append(" restorePendingLeading(choicePending);\n"); + if (!config.triviaPostPass()) { + sb.append(" restorePendingLeading(choicePending);\n"); + } sb.append(" return ") .append(altVar) .append(";\n"); @@ -3970,7 +4081,9 @@ private void emitCstSequenceChunkedDispatcher(StringBuilder sb, sb.append(pad) .append("int ") .append(seqPending) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); if (addToChildren) { sb.append(pad) .append("var ") @@ -4115,7 +4228,9 @@ private void emitCstSequenceChunkHelper(Expression.Sequence seq, .append("if (result.isSuccess()) {\n"); if (!inWhitespaceRule && !isPredicate(elem)) { sb.append(pad) - .append(" if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); + .append(config.triviaPostPass() + ? " if (tokenBoundaryDepth == 0) skipWhitespace();\n" + : " if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); } // Element-local result variable name. The leading `chunkIdx_` prefix is just // a unique-name disambiguator across elements within the chunk. @@ -4130,7 +4245,9 @@ private void emitCstSequenceChunkHelper(Expression.Sequence seq, sb.append(pad) .append(" restoreLocationRaw(seqStartPos, seqStartLine, seqStartColumn);\n"); sb.append(pad) - .append(" restorePendingLeading(seqPending);\n"); + .append(config.triviaPostPass() + ? "" + : " restorePendingLeading(seqPending);\n"); if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -4148,7 +4265,9 @@ private void emitCstSequenceChunkHelper(Expression.Sequence seq, sb.append(pad) .append(" restoreLocationRaw(seqStartPos, seqStartLine, seqStartColumn);\n"); sb.append(pad) - .append(" restorePendingLeading(seqPending);\n"); + .append(config.triviaPostPass() + ? "" + : " restorePendingLeading(seqPending);\n"); if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -4212,7 +4331,9 @@ private void emitCstSequenceInline(StringBuilder sb, sb.append(pad) .append("int ") .append(seqPending) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); if (addToChildren) { sb.append(pad) .append("var ") @@ -4231,7 +4352,9 @@ private void emitCstSequenceInline(StringBuilder sb, .append(".isSuccess()) {\n"); if (!inWhitespaceRule && !isPredicate(elem)) { sb.append(pad) - .append(" if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); + .append(config.triviaPostPass() + ? " if (tokenBoundaryDepth == 0) skipWhitespace();\n" + : " if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); } var elemVar = "elem" + id + "_" + i; generateCstExpressionCode(sb, elem, elemVar, indent + 1, addToChildren, counter, inWhitespaceRule); @@ -4247,10 +4370,12 @@ private void emitCstSequenceInline(StringBuilder sb, .append(", ") .append(seqStartColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(") - .append(seqPending) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(") + .append(seqPending) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -4277,10 +4402,12 @@ private void emitCstSequenceInline(StringBuilder sb, .append(", ") .append(seqStartColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(") - .append(seqPending) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(") + .append(seqPending) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -4495,7 +4622,9 @@ private void emitCstChoiceInline(StringBuilder sb, sb.append(pad) .append("int choicePending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); // Don't skip whitespace here - let alternatives capture trivia themselves emitChildrenSave(sb, pad, childrenState, addToChildren); // Phase 1F: try extended FIRST-set classification first (covers Reference, @@ -4546,10 +4675,12 @@ private void emitCstChoiceInline(StringBuilder sb, .append(resultVar) .append(" == null) {\n"); emitChildrenRestore(sb, pad + " ", childrenState, addToChildren); - sb.append(pad) - .append(" restorePendingLeading(choicePending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(choicePending") + .append(id) + .append(");\n"); + } sb.append(pad) .append(" ") .append(resultVar) @@ -4961,7 +5092,9 @@ private void emitCstFastPathTokenBoundary(StringBuilder sb, .append(tokenRuleId) .append(", tbText") .append(id) - .append(", takePendingLeading(), List.of());\n"); + .append(", ") + .append(takePendingExpr()) + .append(", List.of());\n"); if (addToChildren) { sb.append(pad) .append(" children.add(tbNode") @@ -5307,10 +5440,14 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append(" int zomIterPending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); if (!inWhitespaceRule) { sb.append(pad) - .append(" if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); + .append(config.triviaPostPass() + ? " if (tokenBoundaryDepth == 0) skipWhitespace();\n" + : " if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); } generateCstExpressionCode(sb, zom.expression(), @@ -5348,10 +5485,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(beforeColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(zomIterPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(zomIterPending") + .append(id) + .append(");\n"); + } sb.append(pad) .append(" break;\n"); sb.append(pad) @@ -5459,7 +5598,9 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append("int oomEntryPending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); generateCstExpressionCode(sb, oom.expression(), oomFirst, @@ -5485,16 +5626,18 @@ private void generateCstExpressionCode(StringBuilder sb, .append("int ") .append(oomStartColumn) .append(" = column;\n"); - sb.append(pad) - .append("if (!") - .append(oomFirst) - .append(".isSuccess()) {\n"); - sb.append(pad) - .append(" restorePendingLeading(oomEntryPending") - .append(id) - .append(");\n"); - sb.append(pad) - .append("}\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append("if (!") + .append(oomFirst) + .append(".isSuccess()) {\n"); + sb.append(pad) + .append(" restorePendingLeading(oomEntryPending") + .append(id) + .append(");\n"); + sb.append(pad) + .append("}\n"); + } sb.append(pad) .append("if (") .append(oomFirst) @@ -5516,10 +5659,14 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append(" int oomIterPending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); if (!inWhitespaceRule) { sb.append(pad) - .append(" if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); + .append(config.triviaPostPass() + ? " if (tokenBoundaryDepth == 0) skipWhitespace();\n" + : " if (tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); } generateCstExpressionCode(sb, oom.expression(), @@ -5557,10 +5704,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(beforeColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(oomIterPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(oomIterPending") + .append(id) + .append(");\n"); + } sb.append(pad) .append(" break;\n"); sb.append(pad) @@ -5649,7 +5798,9 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append("int optPending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); // Save parent children before inner expression if (addToChildren) { sb.append(pad) @@ -5684,10 +5835,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(optStartColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(optPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(optPending") + .append(id) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -5740,10 +5893,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(optStartColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(optPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(optPending") + .append(id) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append(" children.clear();\n"); @@ -5826,12 +5981,16 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append(" int repIterPending") .append(id) - .append(" = savePendingLeading();\n"); + .append(" = ") + .append(savePendingExpr()) + .append(";\n"); if (!inWhitespaceRule) { sb.append(pad) .append(" if (") .append(repCount) - .append(" > 0 && tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); + .append(config.triviaPostPass() + ? " > 0 && tokenBoundaryDepth == 0) skipWhitespace();\n" + : " > 0 && tokenBoundaryDepth == 0) appendPending(skipWhitespace());\n"); } generateCstExpressionCode(sb, rep.expression(), @@ -5869,10 +6028,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(beforeColumn) .append(");\n"); - sb.append(pad) - .append(" restorePendingLeading(repIterPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(repIterPending") + .append(id) + .append(");\n"); + } sb.append(pad) .append(" break;\n"); sb.append(pad) @@ -6005,10 +6166,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append("int ") .append(andStartColumn) .append(" = column;\n"); - sb.append(pad) - .append("int andPending") - .append(id) - .append(" = savePendingLeading();\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append("int andPending") + .append(id) + .append(" = savePendingLeading();\n"); + } if (addToChildren) { sb.append(pad) .append("var ") @@ -6025,10 +6188,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(andStartColumn) .append(");\n"); // Predicates don't consume trivia state either. - sb.append(pad) - .append("restorePendingLeading(andPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append("restorePendingLeading(andPending") + .append(id) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append("children.clear();\n"); @@ -6065,10 +6230,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append("int ") .append(notStartColumn) .append(" = column;\n"); - sb.append(pad) - .append("int notPending") - .append(id) - .append(" = savePendingLeading();\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append("int notPending") + .append(id) + .append(" = savePendingLeading();\n"); + } if (addToChildren) { sb.append(pad) .append("var ") @@ -6084,10 +6251,12 @@ private void generateCstExpressionCode(StringBuilder sb, .append(", ") .append(notStartColumn) .append(");\n"); - sb.append(pad) - .append("restorePendingLeading(notPending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append("restorePendingLeading(notPending") + .append(id) + .append(");\n"); + } if (addToChildren) { sb.append(pad) .append("children.clear();\n"); @@ -6188,7 +6357,9 @@ private void generateCstExpressionCode(StringBuilder sb, .append(tokenRuleId) .append(", tbText") .append(id) - .append(", takePendingLeading(), List.of());\n"); + .append(", ") + .append(takePendingExpr()) + .append(", List.of());\n"); if (addToChildren) { sb.append(pad) .append(" children.add(tbNode") @@ -6403,10 +6574,12 @@ private void emitCstChoiceAlt(StringBuilder sb, .append("Column);\n"); // Roll back any pending-leading trivia captured inside the failed alt, // so sibling alternatives start from the same buffer state as the Choice entry. - sb.append(pad) - .append(" restorePendingLeading(choicePending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(choicePending") + .append(id) + .append(");\n"); + } } /** @@ -6595,10 +6768,12 @@ private void emitCstChoiceDispatchF(StringBuilder sb, .append("Line, ") .append(choiceStart) .append("Column);\n"); - sb.append(pad) - .append(" restorePendingLeading(choicePending") - .append(id) - .append(");\n"); + if (!config.triviaPostPass()) { + sb.append(pad) + .append(" restorePendingLeading(choicePending") + .append(id) + .append(");\n"); + } emitCstChoiceAltChainClosed(sb, defaults, id, @@ -7809,7 +7984,9 @@ private void emitMatchLiteralCst(StringBuilder sb) { } // Phase 1.7 (D2): build SourceSpan from inline ints — no SourceLocation alloc. sb.append(" var span = new SourceSpan(startLine, startColumn, startPos, line, column, pos);\n"); - sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_LITERAL, text, takePendingLeading(), List.of());\n"); + sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_LITERAL, text, ") + .append(takePendingExpr()) + .append(", List.of());\n"); sb.append(" return CstParseResult.successNoLoc(node, text);\n"); sb.append(" }\n"); sb.append("\n"); @@ -7849,7 +8026,9 @@ private void emitMatchDictionaryCst(StringBuilder sb) { sb.append(" }\n"); // Phase 1.7 (D2): primitive-int span; successNoLoc result. sb.append(" var span = new SourceSpan(startLine, startColumn, startPos, line, column, pos);\n"); - sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_LITERAL, longestMatch, takePendingLeading(), List.of());\n"); + sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_LITERAL, longestMatch, ") + .append(takePendingExpr()) + .append(", List.of());\n"); sb.append(" return CstParseResult.successNoLoc(node, longestMatch);\n"); sb.append(" }\n"); sb.append("\n"); @@ -7902,7 +8081,9 @@ private void emitMatchCharClassCst(StringBuilder sb) { sb.append(" var text = c < 128 ? ASCII_CHAR_STRINGS[c] : String.valueOf(c);\n"); // Phase 1.7 (D2): primitive-int span; successNoLoc result. sb.append(" var span = new SourceSpan(startLine, startColumn, startPos, line, column, pos);\n"); - sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_CHAR_CLASS, text, takePendingLeading(), List.of());\n"); + sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_CHAR_CLASS, text, ") + .append(takePendingExpr()) + .append(", List.of());\n"); sb.append(" return CstParseResult.successNoLoc(node, text);\n"); sb.append(" }\n"); sb.append("\n"); @@ -7936,7 +8117,9 @@ private void emitMatchAnyCst(StringBuilder sb) { sb.append(" var text = c < 128 ? ASCII_CHAR_STRINGS[c] : String.valueOf(c);\n"); // Phase 1.7 (D2): primitive-int span; successNoLoc result. sb.append(" var span = new SourceSpan(startLine, startColumn, startPos, line, column, pos);\n"); - sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_ANY, text, takePendingLeading(), List.of());\n"); + sb.append(" var node = new CstNode.Terminal(idGen.next(), span, RULE_PEG_ANY, text, ") + .append(takePendingExpr()) + .append(", List.of());\n"); sb.append(" return CstParseResult.successNoLoc(node, text);\n"); sb.append(" }\n"); sb.append("\n"); diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java index b4c82e6..237b8e7 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java @@ -38,6 +38,12 @@ public final class PegEngine implements Parser { private final ParserConfig config; private final Map actions; + // Cleanup A: cached triviaPostPass flag — read once at construction so the + // 38 buffer-call sites can short-circuit without re-reading config on every + // hit. Under flag-ON, the post-pass overrides all engine-side trivia + // attribution; the buffer methods are no-ops, and we avoid the dispatch. + private final boolean triviaPostPass; + // Phase-1 optimization: cached first-char set for skipWhitespace fast-path (§6.6). // Present iff the grammar has a %whitespace rule whose shape is analyzable. // Unconditional-on for the interpreter; no flag gates this. @@ -68,6 +74,7 @@ public final class PegEngine implements Parser { private PegEngine(Grammar grammar, ParserConfig config, Map actions) { this.grammar = grammar; this.config = config; + this.triviaPostPass = config.triviaPostPass(); this.actions = Map.copyOf(actions); this.whitespaceFirstChars = computeWhitespaceFirstChars(grammar); this.suggestionVocabulary = computeSuggestionVocabulary(grammar); @@ -726,7 +733,9 @@ private ParseResult parseRule(ParsingContext ctx, Rule rule) { var entry = cachedEntry.unwrap(); var result = entry.result(); if (result instanceof ParseResult.Success success) { - var hitCarriedLeading = ctx.takePendingLeadingTrivia(); + var hitCarriedLeading = triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(); var hitLocalTrivia = skipWhitespace(ctx); List hitRuleLeading = hitCarriedLeading.isEmpty() ? hitLocalTrivia @@ -744,7 +753,9 @@ private ParseResult parseRule(ParsingContext ctx, Rule rule) { // leading whitespace skipped at rule entry. This becomes the rule // wrapper's leadingTrivia; children of the body see an empty pending // buffer unless the body itself deposits more between its own siblings. - var carriedLeading = ctx.takePendingLeadingTrivia(); + var carriedLeading = triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(); var localTrivia = skipWhitespace(ctx); List ruleLeading = carriedLeading.isEmpty() ? localTrivia @@ -784,11 +795,13 @@ private ParseResult parseRule(ParsingContext ctx, Rule rule) { CstNode bodyNode = null; if (result instanceof ParseResult.Success success) { bodyNode = success.node(); - var pendingAtExit = ctx.savePendingLeadingTrivia(); - if (!pendingAtExit.isEmpty()) { - ctx.restorePendingLeadingTrivia(List.of()); - bodyNode = attachTrailingToTail(bodyNode, pendingAtExit); - resultForCache = ParseResult.Success.success(bodyNode, success.endLocation()); + if (!triviaPostPass) { + var pendingAtExit = ctx.savePendingLeadingTrivia(); + if (!pendingAtExit.isEmpty()) { + ctx.restorePendingLeadingTrivia(List.of()); + bodyNode = attachTrailingToTail(bodyNode, pendingAtExit); + resultForCache = ParseResult.Success.success(bodyNode, success.endLocation()); + } } } // Cache the result at START position @@ -803,7 +816,7 @@ private ParseResult parseRule(ParsingContext ctx, Rule rule) { // Restore position and re-deposit the caller's pending so enclosing // backtracking combinators can roll it back to their own snapshots. ctx.restoreLocation(startLoc); - if (!carriedLeading.isEmpty()) { + if (!triviaPostPass && !carriedLeading.isEmpty()) { ctx.appendPendingLeadingTrivia(carriedLeading); } // Use custom error message if available @@ -871,7 +884,9 @@ private ParseResult parseRuleWithLeftRecursion(ParsingContext ctx, Rule rule) { return result; } if (result instanceof ParseResult.Success success) { - var hitCarriedLeading = ctx.takePendingLeadingTrivia(); + var hitCarriedLeading = triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(); var hitLocalTrivia = skipWhitespace(ctx); List hitRuleLeading = hitCarriedLeading.isEmpty() ? hitLocalTrivia @@ -887,7 +902,9 @@ private ParseResult parseRuleWithLeftRecursion(ParsingContext ctx, Rule rule) { // Claim caller's pending-leading + our own leading whitespace. The // trivia snapshot lives across all growth iterations; each iteration // re-enters the body at the same post-whitespace position. - var carriedLeading = ctx.takePendingLeadingTrivia(); + var carriedLeading = triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(); var localTrivia = skipWhitespace(ctx); List ruleLeading = carriedLeading.isEmpty() ? localTrivia @@ -971,7 +988,7 @@ private ParseResult parseRuleWithLeftRecursion(ParsingContext ctx, Rule rule) { // Failure path — restore the original caller state and re-deposit // pending-leading so enclosing backtracking combinators can roll back. ctx.restoreLocation(startLoc); - if (!carriedLeading.isEmpty()) { + if (!triviaPostPass && !carriedLeading.isEmpty()) { ctx.appendPendingLeadingTrivia(carriedLeading); } if (rule.hasErrorMessage()) { @@ -1008,7 +1025,9 @@ private ParseResult parseRuleWithActions(ParsingContext ctx, Rule rule) { var startLoc = ctx.location(); // Claim pending-leading (caller deposit) + this rule's leading ws. Same // semantics as parseRule; see TRIVIA-ATTRIBUTION.md. - var carriedLeading = ctx.takePendingLeadingTrivia(); + var carriedLeading = triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(); var localTrivia = skipWhitespace(ctx); List ruleLeading = carriedLeading.isEmpty() ? localTrivia @@ -1045,7 +1064,7 @@ private ParseResult parseRuleWithActions(ParsingContext ctx, Rule rule) { } if (result.isFailure()) { ctx.restoreLocation(startLoc); - if (!carriedLeading.isEmpty()) { + if (!triviaPostPass && !carriedLeading.isEmpty()) { ctx.appendPendingLeadingTrivia(carriedLeading); } // Use custom error message if available @@ -1168,7 +1187,9 @@ private ParseResult parseTokenBoundaryWithActions(ParsingContext ctx, span, ruleName, text, - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return ParseResult.Success.success(node, ctx.location()); } finally{ @@ -1244,7 +1265,9 @@ private ParseResult parseLiteral(ParsingContext ctx, Expression.Literal lit) { span, "", text, - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return new ParseResult.Success(node, endLoc, List.of(), Option.none()); } @@ -1310,7 +1333,9 @@ private ParseResult parseDictionary(ParsingContext ctx, Expression.Dictionary di span, "", matched, - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return new ParseResult.Success(node, endLoc, List.of(), Option.none()); } @@ -1366,7 +1391,9 @@ private ParseResult parseCharClass(ParsingContext ctx, Expression.CharClass cc) span, "", String.valueOf(c), - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return new ParseResult.Success(node, endLoc, List.of(), Option.none()); } @@ -1485,7 +1512,9 @@ private ParseResult parseAny(ParsingContext ctx, Expression.Any any) { span, "", String.valueOf(c), - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return new ParseResult.Success(node, endLoc, List.of(), Option.none()); } @@ -1511,11 +1540,15 @@ private boolean isPredicate(Expression expr) { // === Predicate Parsers === private ParseResult parseAnd(ParsingContext ctx, Expression.And and, String ruleName) { var startLoc = ctx.location(); - var entryPendingSnapshot = ctx.savePendingLeadingTrivia(); + var entryPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); var result = parseExpressionWithMode(ctx, and.expression(), ruleName, ParseMode.standard()); ctx.restoreLocation(startLoc); // Always restore - predicates don't consume input nor trivia state. - ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + } if (result.isSuccess()) { return new ParseResult.PredicateSuccess(startLoc); } @@ -1524,11 +1557,15 @@ private ParseResult parseAnd(ParsingContext ctx, Expression.And and, String rule private ParseResult parseNot(ParsingContext ctx, Expression.Not not, String ruleName) { var startLoc = ctx.location(); - var entryPendingSnapshot = ctx.savePendingLeadingTrivia(); + var entryPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); var result = parseExpressionWithMode(ctx, not.expression(), ruleName, ParseMode.standard()); ctx.restoreLocation(startLoc); // Always restore - predicates don't consume input nor trivia state. - ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + } if (result.isSuccess()) { return ParseResult.Failure.failure(startLoc, "not " + describeExpression(not.expression())); } @@ -1554,7 +1591,9 @@ private ParseResult parseTokenBoundary(ParsingContext ctx, Expression.TokenBound span, ruleName, text, - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return ParseResult.Success.success(node, ctx.location()); } finally{ @@ -1617,7 +1656,9 @@ private ParseResult parseBackReference(ParsingContext ctx, Expression.BackRefere span, "", text, - ctx.takePendingLeadingTrivia(), + triviaPostPass + ? List.of() + : ctx.takePendingLeadingTrivia(), List.of()); return ParseResult.Success.success(node, ctx.location()); } @@ -1727,7 +1768,9 @@ private ParseResult parseSequenceWithMode(ParsingContext ctx, String ruleName, ParseMode mode) { var startLoc = ctx.location(); - var pendingSnapshot = ctx.savePendingLeadingTrivia(); + var pendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); var children = new ArrayList(); boolean cutEncountered = false; for (var element : seq.elements()) { @@ -1735,19 +1778,27 @@ private ParseResult parseSequenceWithMode(ParsingContext ctx, // captured trivia is appended to the pending-leading buffer so the // following element (leaf or rule) can claim it. if (mode.shouldSkipWhitespace() && !isPredicate(element)) { - ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + if (triviaPostPass) { + skipWhitespace(ctx); + }else { + ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + } } var result = parseExpressionWithMode(ctx, element, ruleName, mode); // Handle failures if (result instanceof ParseResult.CutFailure) { // Propagate CutFailure immediately ctx.restoreLocation(startLoc); - ctx.restorePendingLeadingTrivia(pendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(pendingSnapshot); + } return result; } if (result instanceof ParseResult.Failure failure) { ctx.restoreLocation(startLoc); - ctx.restorePendingLeadingTrivia(pendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(pendingSnapshot); + } // If cut was encountered, return CutFailure to prevent backtracking if (cutEncountered) { return ParseResult.CutFailure.cutFailure(failure.location(), failure.expected()); @@ -1781,7 +1832,9 @@ private ParseResult parseChoiceWithMode(ParsingContext ctx, String ruleName, ParseMode mode) { var startLoc = ctx.location(); - var entryPendingSnapshot = ctx.savePendingLeadingTrivia(); + var entryPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); ParseResult lastFailure = null; for (var alt : choice.alternatives()) { ParseResult result; @@ -1812,7 +1865,9 @@ private ParseResult parseChoiceWithMode(ParsingContext ctx, } lastFailure = result; ctx.restoreLocation(startLoc); - ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + } } return lastFailure != null ? lastFailure @@ -1830,9 +1885,15 @@ private ParseResult parseZeroOrMoreWithMode(ParsingContext ctx, var children = new ArrayList(); while (true) { var beforeLoc = ctx.location(); - var iterPendingSnapshot = ctx.savePendingLeadingTrivia(); + var iterPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); if (mode.shouldSkipWhitespace()) { - ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + if (triviaPostPass) { + skipWhitespace(ctx); + }else { + ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + } } ParseResult result; if (mode.shouldCollectActions()) { @@ -1857,7 +1918,9 @@ private ParseResult parseZeroOrMoreWithMode(ParsingContext ctx, } if (result.isFailure()) { ctx.restoreLocation(beforeLoc); - ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + } break; } if (result instanceof ParseResult.Success success) { @@ -1901,9 +1964,15 @@ private ParseResult parseOneOrMoreWithMode(ParsingContext ctx, // Subsequent matches are optional while (true) { var beforeLoc = ctx.location(); - var iterPendingSnapshot = ctx.savePendingLeadingTrivia(); + var iterPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); if (mode.shouldSkipWhitespace()) { - ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + if (triviaPostPass) { + skipWhitespace(ctx); + }else { + ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + } } ParseResult result; if (mode.shouldCollectActions()) { @@ -1928,7 +1997,9 @@ private ParseResult parseOneOrMoreWithMode(ParsingContext ctx, } if (result.isFailure()) { ctx.restoreLocation(beforeLoc); - ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + } break; } if (result instanceof ParseResult.Success success) { @@ -1960,7 +2031,9 @@ private ParseResult parseOptionalWithMode(ParsingContext ctx, String ruleName, ParseMode mode) { var startLoc = ctx.location(); - var entryPendingSnapshot = ctx.savePendingLeadingTrivia(); + var entryPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); var result = parseExpressionWithMode(ctx, opt.expression(), ruleName, mode); if (result.isSuccess()) { return result; @@ -1971,7 +2044,9 @@ private ParseResult parseOptionalWithMode(ParsingContext ctx, } // Optional always succeeds - return empty node on no match ctx.restoreLocation(startLoc); - ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(entryPendingSnapshot); + } var span = SourceSpan.sourceSpan(startLoc); var node = new CstNode.NonTerminal(ctx.idGen() .next(), @@ -2000,9 +2075,15 @@ private ParseResult parseRepetitionWithMode(ParsingContext ctx, : Integer.MAX_VALUE; while (count < max) { var beforeLoc = ctx.location(); - var iterPendingSnapshot = ctx.savePendingLeadingTrivia(); + var iterPendingSnapshot = triviaPostPass + ? null + : ctx.savePendingLeadingTrivia(); if (count > 0 && mode.shouldSkipWhitespace()) { - ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + if (triviaPostPass) { + skipWhitespace(ctx); + }else { + ctx.appendPendingLeadingTrivia(skipWhitespace(ctx)); + } } ParseResult result; if (mode.shouldCollectActions()) { @@ -2027,7 +2108,9 @@ private ParseResult parseRepetitionWithMode(ParsingContext ctx, } if (result.isFailure()) { ctx.restoreLocation(beforeLoc); - ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + if (!triviaPostPass) { + ctx.restorePendingLeadingTrivia(iterPendingSnapshot); + } break; } if (result instanceof ParseResult.Success success) { From 884c4c8d9bb3bd5cd628c420fc2039b7b689bab3 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 12:38:59 +0200 Subject: [PATCH 18/26] perf: TriviaPostPass thread orphan trailing + probe-scan cache (Cleanup B) --- .../peg/bench/Java25ParseBenchmark.java | 19 +- .../peg/generator/ParserGenerator.java | 115 +++++------ .../pragmatica/peg/tree/TriviaPostPass.java | 179 +++++++++--------- 3 files changed, 164 insertions(+), 149 deletions(-) diff --git a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java index 0a6a533..3532804 100644 --- a/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java +++ b/peglib-core/src/jmh/java/org/pragmatica/peg/bench/Java25ParseBenchmark.java @@ -70,6 +70,7 @@ public class Java25ParseBenchmark { "phase1_allStructural_skipPackrat", "phase1_allStructural_mutableResult", "phase1_allStructural_mutableResult_autoSkipPackrat", + "phase1_allStructural_mutableResult_autoSkipPackrat_postPass", "interpreter" }) public String variant; @@ -173,6 +174,10 @@ private static ParserConfig configFor(String variant) { case "phase1_allStructural_mutableResult_autoSkipPackrat_noFastPath" -> withStructural(true, true, true, true, Set.of(), true, false); // Phase 1.9 (DFA spike): structural + autoSkipPackrat WITH tokenFastPath — A/B variant. case "phase1_allStructural_mutableResult_autoSkipPackrat_fastPath" -> withStructural(true, true, true, true, Set.of(), true, true); + // Step 4 follow-up: post-pass trivia attribution A/B vs autoSkipPackrat baseline + // (which leaves triviaPostPass=false). Same generator-time perf flags as + // autoSkipPackrat; only triviaPostPass flips ON. + case "phase1_allStructural_mutableResult_autoSkipPackrat_postPass" -> withStructural(true, true, true, true, Set.of(), true, true, true); default -> throw new IllegalArgumentException("Unknown variant: " + variant); }; } @@ -227,6 +232,18 @@ private static ParserConfig withStructural(boolean choiceDispatch, Set packratSkipRules, boolean mutableParseResult, boolean tokenFastPath) { + return withStructural(choiceDispatch, markResetChildren, inlineLocations, selectivePackrat, + packratSkipRules, mutableParseResult, tokenFastPath, false); + } + + private static ParserConfig withStructural(boolean choiceDispatch, + boolean markResetChildren, + boolean inlineLocations, + boolean selectivePackrat, + Set packratSkipRules, + boolean mutableParseResult, + boolean tokenFastPath, + boolean triviaPostPass) { return new ParserConfig( true, // packratEnabled RecoveryStrategy.BASIC, @@ -244,7 +261,7 @@ private static ParserConfig withStructural(boolean choiceDispatch, Set.copyOf(packratSkipRules), mutableParseResult, tokenFastPath, // phase 1.9 (DFA spike) - false); // triviaPostPass (Step 4 commit 1) + triviaPostPass); // Step 4 — post-pass trivia attribution } static String loadResource(String resourcePath) throws Exception { diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 5f29d1f..008aad3 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -7336,6 +7336,12 @@ private void generateCstTriviaPostPass(StringBuilder sb) { // "leading-end" boundary AND leadingScanFrom + (rootStart - leadingScanFrom) // = rootStart for the leading scan. Otherwise (runtime semantics), // scan up to rootStart as the runtime post-pass does. + // H2 (Cleanup B): the wrapper-span semantics decision (does the rule + // wrapper's span INCLUDE its leading whitespace?) is a property of the + // GENERATED parser globally, not of an individual node. Detect it once + // at the root and propagate the boolean through the recursion so each + // NonTerminal can advance the child-cursor past leadingLen without + // re-probing scanWhitespace per node. sb.append(" private static CstNode rebuildRoot(String input, CstNode root, int[] lineStarts, int leadingScanFrom) {\n"); sb.append(" int rootStart = root.span().startOffset();\n"); sb.append(" int rootEnd = root.span().endOffset();\n"); @@ -7347,6 +7353,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" // the gap [leadingScanFrom, rootStart). We distinguish by checking\n"); sb.append(" // whether scanning the pre-rootStart gap yields the leading.\n"); sb.append(" int leadingScanEnd;\n"); + sb.append(" boolean wrapperSpanIncludesLeading = false;\n"); sb.append(" if (leadingTextLen > 0 && rootStart + leadingTextLen <= input.length()) {\n"); sb.append(" // Hypothesis: generator semantics — leading occupies [rootStart, rootStart + leadingTextLen).\n"); sb.append(" // Verify by re-scanning [rootStart, rootStart + leadingTextLen) and checking it consumes exactly that range.\n"); @@ -7354,6 +7361,7 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" var probe = scanWhitespace(input, rootStart, probeEnd, lineStarts);\n"); sb.append(" if (totalTriviaLength(probe) == leadingTextLen) {\n"); sb.append(" leadingScanEnd = probeEnd;\n"); + sb.append(" wrapperSpanIncludesLeading = true;\n"); sb.append(" } else {\n"); sb.append(" leadingScanEnd = rootStart;\n"); sb.append(" }\n"); @@ -7362,91 +7370,84 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" }\n"); sb.append(" var leading = scanWhitespace(input, leadingScanFrom, leadingScanEnd, lineStarts);\n"); sb.append(" var trailingExternal = scanWhitespace(input, rootEnd, input.length(), lineStarts);\n"); - sb.append(" return rebuildSelf(input, root, lineStarts, leading, trailingExternal);\n"); + sb.append(" return rebuildSelf(input, root, lineStarts, leading, trailingExternal, java.util.List.of(), wrapperSpanIncludesLeading);\n"); sb.append(" }\n\n"); sb.append(" private static int totalTriviaLength(List trivia) {\n"); sb.append(" int total = 0;\n"); sb.append(" for (var t : trivia) total += t.text().length();\n"); sb.append(" return total;\n"); sb.append(" }\n\n"); + // rebuildSelf: extraTrailing sticks on this node's own trailing slot + // (root post-EOF only). drainExtra propagates DOWN into the + // deepest-rightmost-leaf — H1 eliminates the post-construction + // attachTrailingToTail spine rebuild. sb.append(" private static CstNode rebuildSelf(String input, CstNode node, int[] lineStarts,\n"); - sb.append(" List leading, List extraTrailing) {\n"); + sb.append(" List leading, List extraTrailing,\n"); + sb.append(" List drainExtra, boolean wrapperSpanIncludesLeading) {\n"); sb.append(" return switch (node) {\n"); - sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, extraTrailing);\n"); - sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing);\n"); - sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing);\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, extraTrailing, drainExtra, wrapperSpanIncludesLeading);\n"); + sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, combine(drainExtra, extraTrailing));\n"); if (errorReporting == ErrorReporting.ADVANCED) { - sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, extraTrailing);\n"); + sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, combine(drainExtra, extraTrailing));\n"); } sb.append(" };\n"); sb.append(" }\n\n"); - sb.append(" private static CstNode rebuildChild(String input, CstNode child, int[] lineStarts, int prevEnd) {\n"); + sb.append(" private static CstNode rebuildChild(String input, CstNode child, int[] lineStarts, int prevEnd,\n"); + sb.append(" List drainExtra, boolean wrapperSpanIncludesLeading) {\n"); sb.append(" int childStart = child.span().startOffset();\n"); sb.append(" var leading = scanWhitespace(input, prevEnd, childStart, lineStarts);\n"); sb.append(" return switch (child) {\n"); - sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, List.of());\n"); - sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of());\n"); - sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of());\n"); + sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, java.util.List.of(), drainExtra, wrapperSpanIncludesLeading);\n"); + sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, drainExtra);\n"); + sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, drainExtra);\n"); if (errorReporting == ErrorReporting.ADVANCED) { - sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, List.of());\n"); + sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, drainExtra);\n"); } sb.append(" };\n"); sb.append(" }\n\n"); sb.append(" private static CstNode.NonTerminal rebuildNonTerminal(String input, CstNode.NonTerminal nt, int[] lineStarts,\n"); - sb.append(" List leading, List extraTrailing) {\n"); + sb.append(" List leading, List extraTrailing,\n"); + sb.append(" List drainExtra, boolean wrapperSpanIncludesLeading) {\n"); sb.append(" int spanStart = nt.span().startOffset();\n"); sb.append(" int spanEnd = nt.span().endOffset();\n"); - sb.append(" // Generator semantics adapter: the wrapper's span may INCLUDE its\n"); - sb.append(" // own leading trivia (rule entry captures startOffset BEFORE\n"); - sb.append(" // skipWhitespace). If the caller-supplied leading occupies\n"); - sb.append(" // [spanStart, spanStart+leadingLen), advance cursor past it so the\n"); - sb.append(" // first child's leading scan does not re-emit the same trivia.\n"); - sb.append(" int leadingLen = totalTriviaLength(leading);\n"); + sb.append(" // H2: span-semantics decided once at the root. When wrapper span\n"); + sb.append(" // includes its own leading WS, advance cursor past leadingLen so the\n"); + sb.append(" // first child's leading scan does not re-emit the same trivia. No\n"); + sb.append(" // per-node probe scan.\n"); sb.append(" int cursor = spanStart;\n"); - sb.append(" if (leadingLen > 0 && spanStart + leadingLen <= input.length()) {\n"); - sb.append(" var probe = scanWhitespace(input, spanStart, spanStart + leadingLen, lineStarts);\n"); - sb.append(" if (totalTriviaLength(probe) == leadingLen) {\n"); + sb.append(" if (wrapperSpanIncludesLeading) {\n"); + sb.append(" int leadingLen = totalTriviaLength(leading);\n"); + sb.append(" if (leadingLen > 0 && spanStart + leadingLen <= input.length()) {\n"); sb.append(" cursor = spanStart + leadingLen;\n"); sb.append(" }\n"); sb.append(" }\n"); - sb.append(" var newChildren = new ArrayList(nt.children().size());\n"); - sb.append(" for (var c : nt.children()) {\n"); - sb.append(" var rebuilt = rebuildChild(input, c, lineStarts, cursor);\n"); - sb.append(" newChildren.add(rebuilt);\n"); - sb.append(" cursor = c.span().endOffset();\n"); + sb.append(" var children = nt.children();\n"); + sb.append(" int childCount = children.size();\n"); + sb.append(" if (childCount == 0) {\n"); + sb.append(" // No children: gap [cursor, spanEnd) is the wrapper's internal\n"); + sb.append(" // trailing. Combine with caller-supplied drainExtra (no deeper\n"); + sb.append(" // descendant to absorb it) and extraTrailing.\n"); + sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd, lineStarts);\n"); + sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), children, leading,\n"); + sb.append(" combine(combine(drainExtra, internalTrailing), extraTrailing));\n"); sb.append(" }\n"); - sb.append(" // Bug C' compensation — drain orphan trailing into the last terminal\n"); - sb.append(" // descendant so CstReconstruct.emit (last-child-trailing-only) stays\n"); - sb.append(" // byte-equal even when this wrapper is not its parent's last child.\n"); - sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd, lineStarts);\n"); - sb.append(" List wrapperTrailing = extraTrailing;\n"); - sb.append(" if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) {\n"); - sb.append(" int lastIdx = newChildren.size() - 1;\n"); - sb.append(" newChildren.set(lastIdx, postPassAttachTrailingToTail(newChildren.get(lastIdx), internalTrailing));\n"); - sb.append(" } else {\n"); - sb.append(" wrapperTrailing = combine(internalTrailing, extraTrailing);\n"); + sb.append(" // H1: thread orphan trailing INTO the last child's rebuildChild as\n"); + sb.append(" // drainExtra instead of rebuilding the spine afterwards. Mirrors\n"); + sb.append(" // PegEngine.attachTrailingToTail (Bug C' compensation): the orphan\n"); + sb.append(" // trivia ends up in the deepest-rightmost-leaf's trailing slot,\n"); + sb.append(" // matching CstReconstruct.emit's last-child-only emission.\n"); + sb.append(" int lastChildEnd = children.get(childCount - 1).span().endOffset();\n"); + sb.append(" var internalTrailing = scanWhitespace(input, lastChildEnd, spanEnd, lineStarts);\n"); + sb.append(" var lastChildDrain = combine(drainExtra, internalTrailing);\n"); + sb.append(" var newChildren = new ArrayList(childCount);\n"); + sb.append(" for (int i = 0; i < childCount; i++) {\n"); + sb.append(" var c = children.get(i);\n"); + sb.append(" var drainForThis = (i == childCount - 1) ? lastChildDrain : java.util.List.of();\n"); + sb.append(" newChildren.add(rebuildChild(input, c, lineStarts, cursor, drainForThis, wrapperSpanIncludesLeading));\n"); + sb.append(" cursor = c.span().endOffset();\n"); sb.append(" }\n"); - sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, wrapperTrailing);\n"); - sb.append(" }\n\n"); - sb.append(" private static CstNode postPassAttachTrailingToTail(CstNode node, List extra) {\n"); - sb.append(" if (extra.isEmpty()) return node;\n"); - sb.append(" return switch (node) {\n"); - sb.append(" case CstNode.NonTerminal nt -> {\n"); - sb.append(" var children = nt.children();\n"); - sb.append(" if (children.isEmpty()) {\n"); - sb.append(" yield new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), children, nt.leadingTrivia(), combine(nt.trailingTrivia(), extra));\n"); - sb.append(" }\n"); - sb.append(" var newChildren = new ArrayList(children);\n"); - sb.append(" int lastIdx = newChildren.size() - 1;\n"); - sb.append(" newChildren.set(lastIdx, postPassAttachTrailingToTail(newChildren.get(lastIdx), extra));\n"); - sb.append(" yield new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), nt.leadingTrivia(), nt.trailingTrivia());\n"); - sb.append(" }\n"); - sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), t.leadingTrivia(), combine(t.trailingTrivia(), extra));\n"); - sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), tk.leadingTrivia(), combine(tk.trailingTrivia(), extra));\n"); - if (errorReporting == ErrorReporting.ADVANCED) { - sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), e.leadingTrivia(), combine(e.trailingTrivia(), extra));\n"); - } - sb.append(" };\n"); + sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, extraTrailing);\n"); sb.append(" }\n\n"); sb.append(" private static List combine(List a, List b) {\n"); sb.append(" if (a.isEmpty()) return b;\n"); diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index 9b2bfb4..6f116a1 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -216,53 +216,86 @@ private static CstNode rebuildRoot(String input, // Root trailing covers the gap from rootEnd to end-of-input (matches // current engine behaviour: top-level wrapper carries post-EOF trivia). var trailingExternal = scanWhitespaceFast(input, rootEnd, input.length(), grammar, lineStarts); - return rebuildSelf(input, root, grammar, lineStarts, leading, trailingExternal); + return rebuildSelf(input, root, grammar, lineStarts, leading, trailingExternal, List.of()); } /** * Rebuild a node with explicit leading/extra-trailing computed by the - * caller. The "extra-trailing" is appended after the node's own - * within-span trailing (used only at the root to capture post-EOF - * trivia). + * caller. {@code extraTrailing} sticks on this node's own trailing slot + * (used only at the root to capture post-EOF trivia). {@code drainExtra} + * is the orphan-internal-trailing drained from an outer wrapper that + * must reach the deepest-rightmost-leaf descendant — H1 optimization + * eliminates the post-construction spine rebuild by threading it during + * recursion. */ private static CstNode rebuildSelf(String input, CstNode node, Grammar grammar, int[] lineStarts, List leading, - List extraTrailing) { + List extraTrailing, + List drainExtra) { return switch (node) { - case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, lineStarts, leading, extraTrailing); - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, extraTrailing); - case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, extraTrailing); + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, + nt, + grammar, + lineStarts, + leading, + extraTrailing, + drainExtra); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), + t.span(), + t.rule(), + t.text(), + leading, + combine(drainExtra, extraTrailing)); + case CstNode.Token tk -> new CstNode.Token(tk.id(), + tk.span(), + tk.rule(), + tk.text(), + leading, + combine(drainExtra, extraTrailing)); case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, - extraTrailing); + combine(drainExtra, extraTrailing)); }; } - /** Rebuild a non-root child given its preceding sibling's end offset. */ + /** + * Rebuild a non-root child given its preceding sibling's end offset. + * {@code drainExtra} is the orphan-internal-trailing of THIS wrapper + * propagated DOWN into the deepest-rightmost descendant of the last + * child — eliminates the post-construction + * {@code attachTrailingToTail} spine rebuild (H1). + */ private static CstNode rebuildChild(String input, CstNode child, Grammar grammar, int[] lineStarts, - int prevEnd) { + int prevEnd, + List drainExtra) { int childStart = child.span() .startOffset(); var leading = scanWhitespaceFast(input, prevEnd, childStart, grammar, lineStarts); return switch (child) { - case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, grammar, lineStarts, leading, List.of()); - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, List.of()); - case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, List.of()); + case CstNode.NonTerminal nt -> rebuildNonTerminal(input, + nt, + grammar, + lineStarts, + leading, + List.of(), + drainExtra); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, drainExtra); + case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, drainExtra); case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, - List.of()); + drainExtra); }; } @@ -271,90 +304,54 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, Grammar grammar, int[] lineStarts, List leading, - List extraTrailing) { + List extraTrailing, + List drainExtra) { int spanStart = nt.span() .startOffset(); int spanEnd = nt.span() .endOffset(); - var newChildren = new ArrayList(nt.children() - .size()); + var children = nt.children(); + int childCount = children.size(); + if (childCount == 0) { + // No children: the gap between spanStart and spanEnd is internal + // trailing of this wrapper. Combine with both caller-supplied + // wrapper-level extraTrailing and any drained extra from outer + // wrappers (no deeper descendant to absorb it into). + var internalTrailing = scanWhitespaceFast(input, spanStart, spanEnd, grammar, lineStarts); + return new CstNode.NonTerminal(nt.id(), + nt.span(), + nt.rule(), + children, + leading, + combine(combine(drainExtra, internalTrailing), extraTrailing)); + } + // H1: drained orphan trailing flows DOWN through rebuildChild instead + // of bubbling UP via a second-pass spine rebuild. The wrapper itself + // keeps only its caller-supplied extraTrailing (post-EOF for root, + // empty for everyone else). Mirrors PegEngine.attachTrailingToTail + // (Bug C' compensation): the orphan trivia ends up in the + // deepest-rightmost-leaf's trailing slot, matching CstReconstruct.emit's + // last-child-only emission semantics. + int lastChildEnd = children.get(childCount - 1) + .span() + .endOffset(); + var internalTrailing = scanWhitespaceFast(input, lastChildEnd, spanEnd, grammar, lineStarts); + // The last child's drainExtra is THIS wrapper's internalTrailing plus + // any drainExtra threaded from an outer wrapper. extraTrailing stays + // on the wrapper — only post-EOF root trivia uses that slot. + var lastChildDrain = combine(drainExtra, internalTrailing); + var newChildren = new ArrayList(childCount); int cursor = spanStart; - for (var c : nt.children()) { - var rebuilt = rebuildChild(input, c, grammar, lineStarts, cursor); - newChildren.add(rebuilt); + for (int i = 0; i < childCount; i++ ) { + var c = children.get(i); + var drainForThis = (i == childCount - 1) + ? lastChildDrain + : List.of(); + newChildren.add(rebuildChild(input, c, grammar, lineStarts, cursor, drainForThis)); cursor = c.span() .endOffset(); } - // Internal trailing — gap between last child end and the wrapper's - // span end. Mirrors PegEngine.attachTrailingToTail (Bug C' - // compensation): when the wrapper has children, drain the orphan - // trivia into the last child's tail so reconstruction visits it via - // the wrapper's last-child trailing slot. Without this, a wrapper - // that is itself a non-last sibling would lose its trailing during - // CstReconstruct.emit. - var internalTrailing = scanWhitespaceFast(input, cursor, spanEnd, grammar, lineStarts); - List wrapperTrailing = extraTrailing; - if (!internalTrailing.isEmpty() && !newChildren.isEmpty()) { - int lastIdx = newChildren.size() - 1; - newChildren.set(lastIdx, attachTrailingToTail(newChildren.get(lastIdx), internalTrailing)); - }else { - wrapperTrailing = combine(internalTrailing, extraTrailing); - } - return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, wrapperTrailing); - } - - /** - * Recursively drain {@code extra} trivia into the deepest-rightmost-leaf - * of {@code node}, mirroring {@code PegEngine.attachTrailingToTail} - * (Bug C' compensation). Used by {@link #rebuildNonTerminal} to place - * orphan trivia consumed before a zero-width tail element into the - * trailing slot that {@code CstReconstruct.emit} actually visits — the - * last terminal descendant — instead of the wrapper non-terminal's own - * trailing slot, which is invisible during reconstruction when the - * wrapper is not the last child of its parent. - */ - private static CstNode attachTrailingToTail(CstNode node, List extra) { - if (extra.isEmpty()) return node; - return switch (node) { - case CstNode.NonTerminal nt -> { - var children = nt.children(); - if (children.isEmpty()) { - yield new CstNode.NonTerminal(nt.id(), - nt.span(), - nt.rule(), - children, - nt.leadingTrivia(), - combine(nt.trailingTrivia(), extra)); - } - var newChildren = new ArrayList(children); - int lastIdx = newChildren.size() - 1; - newChildren.set(lastIdx, attachTrailingToTail(newChildren.get(lastIdx), extra)); - yield new CstNode.NonTerminal(nt.id(), - nt.span(), - nt.rule(), - List.copyOf(newChildren), - nt.leadingTrivia(), - nt.trailingTrivia()); - } - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), - t.span(), - t.rule(), - t.text(), - t.leadingTrivia(), - combine(t.trailingTrivia(), extra)); - case CstNode.Token tk -> new CstNode.Token(tk.id(), - tk.span(), - tk.rule(), - tk.text(), - tk.leadingTrivia(), - combine(tk.trailingTrivia(), extra)); - case CstNode.Error e -> new CstNode.Error(e.id(), - e.span(), - e.skippedText(), - e.expected(), - e.leadingTrivia(), - combine(e.trailingTrivia(), extra)); - }; + return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, extraTrailing); } private static List combine(List a, List b) { From c682eef864f869d0ab78f4c63b0d7b976c543027 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 13:01:44 +0200 Subject: [PATCH 19/26] perf: TriviaPostPass skip rebuild when no trivia change needed (Cleanup D) --- .../peg/generator/ParserGenerator.java | 74 ++++++++-- .../pragmatica/peg/tree/TriviaPostPass.java | 133 ++++++++++++++---- 2 files changed, 169 insertions(+), 38 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 008aad3..ab51e38 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -7386,23 +7386,55 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" List drainExtra, boolean wrapperSpanIncludesLeading) {\n"); sb.append(" return switch (node) {\n"); sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, extraTrailing, drainExtra, wrapperSpanIncludesLeading);\n"); - sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, combine(drainExtra, extraTrailing));\n"); - sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" case CstNode.Terminal t -> rebuildTerminal(t, leading, extraTrailing, drainExtra);\n"); + sb.append(" case CstNode.Token tk -> rebuildToken(tk, leading, extraTrailing, drainExtra);\n"); if (errorReporting == ErrorReporting.ADVANCED) { - sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" case CstNode.Error e -> rebuildError(e, leading, extraTrailing, drainExtra);\n"); } sb.append(" };\n"); sb.append(" }\n\n"); + // Cleanup D: leaf rebuild helpers — skip allocation when the rebuild + // would yield content-equal nodes (all incoming trivia empty AND + // existing trivia already empty). + sb.append(" private static boolean canSkipLeafRebuild(List leading, List extraTrailing,\n"); + sb.append(" List drainExtra,\n"); + sb.append(" List existingLeading, List existingTrailing) {\n"); + sb.append(" return leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty()\n"); + sb.append(" && existingLeading.isEmpty() && existingTrailing.isEmpty();\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode.Terminal rebuildTerminal(CstNode.Terminal t, List leading,\n"); + sb.append(" List extraTrailing, List drainExtra) {\n"); + sb.append(" if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, t.leadingTrivia(), t.trailingTrivia())) {\n"); + sb.append(" return t;\n"); + sb.append(" }\n"); + sb.append(" return new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" }\n\n"); + sb.append(" private static CstNode.Token rebuildToken(CstNode.Token tk, List leading,\n"); + sb.append(" List extraTrailing, List drainExtra) {\n"); + sb.append(" if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, tk.leadingTrivia(), tk.trailingTrivia())) {\n"); + sb.append(" return tk;\n"); + sb.append(" }\n"); + sb.append(" return new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" }\n\n"); + if (errorReporting == ErrorReporting.ADVANCED) { + sb.append(" private static CstNode.Error rebuildError(CstNode.Error e, List leading,\n"); + sb.append(" List extraTrailing, List drainExtra) {\n"); + sb.append(" if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, e.leadingTrivia(), e.trailingTrivia())) {\n"); + sb.append(" return e;\n"); + sb.append(" }\n"); + sb.append(" return new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, combine(drainExtra, extraTrailing));\n"); + sb.append(" }\n\n"); + } sb.append(" private static CstNode rebuildChild(String input, CstNode child, int[] lineStarts, int prevEnd,\n"); sb.append(" List drainExtra, boolean wrapperSpanIncludesLeading) {\n"); sb.append(" int childStart = child.span().startOffset();\n"); sb.append(" var leading = scanWhitespace(input, prevEnd, childStart, lineStarts);\n"); sb.append(" return switch (child) {\n"); sb.append(" case CstNode.NonTerminal nt -> rebuildNonTerminal(input, nt, lineStarts, leading, java.util.List.of(), drainExtra, wrapperSpanIncludesLeading);\n"); - sb.append(" case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, drainExtra);\n"); - sb.append(" case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, drainExtra);\n"); + sb.append(" case CstNode.Terminal t -> rebuildTerminal(t, leading, java.util.List.of(), drainExtra);\n"); + sb.append(" case CstNode.Token tk -> rebuildToken(tk, leading, java.util.List.of(), drainExtra);\n"); if (errorReporting == ErrorReporting.ADVANCED) { - sb.append(" case CstNode.Error e -> new CstNode.Error(e.id(), e.span(), e.skippedText(), e.expected(), leading, drainExtra);\n"); + sb.append(" case CstNode.Error e -> rebuildError(e, leading, java.util.List.of(), drainExtra);\n"); } sb.append(" };\n"); sb.append(" }\n\n"); @@ -7429,6 +7461,12 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" // trailing. Combine with caller-supplied drainExtra (no deeper\n"); sb.append(" // descendant to absorb it) and extraTrailing.\n"); sb.append(" var internalTrailing = scanWhitespace(input, cursor, spanEnd, lineStarts);\n"); + sb.append(" // Cleanup D: skip allocation if rebuild would yield identical content.\n"); + sb.append(" if (leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty()\n"); + sb.append(" && internalTrailing.isEmpty()\n"); + sb.append(" && nt.leadingTrivia().isEmpty() && nt.trailingTrivia().isEmpty()) {\n"); + sb.append(" return nt;\n"); + sb.append(" }\n"); sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), children, leading,\n"); sb.append(" combine(combine(drainExtra, internalTrailing), extraTrailing));\n"); sb.append(" }\n"); @@ -7440,14 +7478,32 @@ private void generateCstTriviaPostPass(StringBuilder sb) { sb.append(" int lastChildEnd = children.get(childCount - 1).span().endOffset();\n"); sb.append(" var internalTrailing = scanWhitespace(input, lastChildEnd, spanEnd, lineStarts);\n"); sb.append(" var lastChildDrain = combine(drainExtra, internalTrailing);\n"); - sb.append(" var newChildren = new ArrayList(childCount);\n"); + sb.append(" // Cleanup D: defer ArrayList allocation until first child diverges; skip\n"); + sb.append(" // wrapper-rebuild entirely when no incoming trivia and no internal change.\n"); + sb.append(" boolean canSkip = leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty()\n"); + sb.append(" && internalTrailing.isEmpty()\n"); + sb.append(" && nt.leadingTrivia().isEmpty() && nt.trailingTrivia().isEmpty();\n"); + sb.append(" ArrayList newChildren = null;\n"); sb.append(" for (int i = 0; i < childCount; i++) {\n"); sb.append(" var c = children.get(i);\n"); sb.append(" var drainForThis = (i == childCount - 1) ? lastChildDrain : java.util.List.of();\n"); - sb.append(" newChildren.add(rebuildChild(input, c, lineStarts, cursor, drainForThis, wrapperSpanIncludesLeading));\n"); + sb.append(" var rebuilt = rebuildChild(input, c, lineStarts, cursor, drainForThis, wrapperSpanIncludesLeading);\n"); + sb.append(" if (newChildren == null) {\n"); + sb.append(" if (rebuilt != c) {\n"); + sb.append(" newChildren = new ArrayList(childCount);\n"); + sb.append(" for (int j = 0; j < i; j++) newChildren.add(children.get(j));\n"); + sb.append(" newChildren.add(rebuilt);\n"); + sb.append(" }\n"); + sb.append(" } else {\n"); + sb.append(" newChildren.add(rebuilt);\n"); + sb.append(" }\n"); sb.append(" cursor = c.span().endOffset();\n"); sb.append(" }\n"); - sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, extraTrailing);\n"); + sb.append(" if (newChildren == null && canSkip) {\n"); + sb.append(" return nt;\n"); + sb.append(" }\n"); + sb.append(" var finalChildren = newChildren == null ? children : List.copyOf(newChildren);\n"); + sb.append(" return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), finalChildren, leading, extraTrailing);\n"); sb.append(" }\n\n"); sb.append(" private static List combine(List a, List b) {\n"); sb.append(" if (a.isEmpty()) return b;\n"); diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index 6f116a1..ee4206f 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -227,6 +227,12 @@ private static CstNode rebuildRoot(String input, * must reach the deepest-rightmost-leaf descendant — H1 optimization * eliminates the post-construction spine rebuild by threading it during * recursion. + * + *

Cleanup D fast-path: when the rebuild would produce a node with the + * same content as the input (all incoming trivia lists empty AND existing + * trivia already empty for leaves), return the input unchanged. For + * NonTerminals, the same logic is applied per-child inside + * {@link #rebuildNonTerminal}. */ private static CstNode rebuildSelf(String input, CstNode node, @@ -243,33 +249,73 @@ private static CstNode rebuildSelf(String input, leading, extraTrailing, drainExtra); - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), - t.span(), - t.rule(), - t.text(), - leading, - combine(drainExtra, extraTrailing)); - case CstNode.Token tk -> new CstNode.Token(tk.id(), - tk.span(), - tk.rule(), - tk.text(), - leading, - combine(drainExtra, extraTrailing)); - case CstNode.Error e -> new CstNode.Error(e.id(), - e.span(), - e.skippedText(), - e.expected(), - leading, - combine(drainExtra, extraTrailing)); + case CstNode.Terminal t -> rebuildTerminal(t, leading, extraTrailing, drainExtra); + case CstNode.Token tk -> rebuildToken(tk, leading, extraTrailing, drainExtra); + case CstNode.Error e -> rebuildError(e, leading, extraTrailing, drainExtra); }; } + /** + * Cleanup D: skip allocation when the rebuild would produce a leaf + * identical to the input. The leaf result has {@code leading} as + * leadingTrivia and {@code combine(drainExtra, extraTrailing)} as + * trailing. When all three inputs are empty AND the input leaf already + * has empty leading/trailing, the rebuilt node would be content-equal + * to the input — return the input reference and avoid allocating. + */ + private static CstNode.Terminal rebuildTerminal(CstNode.Terminal t, + List leading, + List extraTrailing, + List drainExtra) { + if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, t.leadingTrivia(), t.trailingTrivia())) { + return t; + } + return new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, combine(drainExtra, extraTrailing)); + } + + private static CstNode.Token rebuildToken(CstNode.Token tk, + List leading, + List extraTrailing, + List drainExtra) { + if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, tk.leadingTrivia(), tk.trailingTrivia())) { + return tk; + } + return new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, combine(drainExtra, extraTrailing)); + } + + private static CstNode.Error rebuildError(CstNode.Error e, + List leading, + List extraTrailing, + List drainExtra) { + if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, e.leadingTrivia(), e.trailingTrivia())) { + return e; + } + return new CstNode.Error(e.id(), + e.span(), + e.skippedText(), + e.expected(), + leading, + combine(drainExtra, extraTrailing)); + } + + private static boolean canSkipLeafRebuild(List leading, + List extraTrailing, + List drainExtra, + List existingLeading, + List existingTrailing) { + return leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty() && existingLeading.isEmpty() && existingTrailing.isEmpty(); + } + /** * Rebuild a non-root child given its preceding sibling's end offset. * {@code drainExtra} is the orphan-internal-trailing of THIS wrapper * propagated DOWN into the deepest-rightmost descendant of the last * child — eliminates the post-construction * {@code attachTrailingToTail} spine rebuild (H1). + * + *

Cleanup D fast-path: leaves with no incoming trivia and already-empty + * existing trivia are returned unchanged. NonTerminals delegate to + * {@link #rebuildNonTerminal}, which has its own skip-when-unchanged path. */ private static CstNode rebuildChild(String input, CstNode child, @@ -288,14 +334,9 @@ private static CstNode rebuildChild(String input, leading, List.of(), drainExtra); - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, drainExtra); - case CstNode.Token tk -> new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, drainExtra); - case CstNode.Error e -> new CstNode.Error(e.id(), - e.span(), - e.skippedText(), - e.expected(), - leading, - drainExtra); + case CstNode.Terminal t -> rebuildTerminal(t, leading, List.of(), drainExtra); + case CstNode.Token tk -> rebuildToken(tk, leading, List.of(), drainExtra); + case CstNode.Error e -> rebuildError(e, leading, List.of(), drainExtra); }; } @@ -318,6 +359,12 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, // wrapper-level extraTrailing and any drained extra from outer // wrappers (no deeper descendant to absorb it into). var internalTrailing = scanWhitespaceFast(input, spanStart, spanEnd, grammar, lineStarts); + // Cleanup D: skip allocation if rebuild would yield identical content. + if (leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty() && internalTrailing.isEmpty() && nt.leadingTrivia() + .isEmpty() && nt.trailingTrivia() + .isEmpty()) { + return nt; + } return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), @@ -340,18 +387,46 @@ private static CstNode.NonTerminal rebuildNonTerminal(String input, // any drainExtra threaded from an outer wrapper. extraTrailing stays // on the wrapper — only post-EOF root trivia uses that slot. var lastChildDrain = combine(drainExtra, internalTrailing); - var newChildren = new ArrayList(childCount); + // Cleanup D: when no incoming trivia and no internal trailing AND existing + // wrapper trivia already empty, the rebuild can return nt unchanged IFF + // every rebuilt child is reference-equal to its input (i.e. each child + // also took its own skip path). Defer the allocation until we discover + // the first real change. + boolean canSkip = leading.isEmpty() && extraTrailing.isEmpty() && drainExtra.isEmpty() && internalTrailing.isEmpty() && nt.leadingTrivia() + .isEmpty() && nt.trailingTrivia() + .isEmpty(); + ArrayList newChildren = null; int cursor = spanStart; for (int i = 0; i < childCount; i++ ) { var c = children.get(i); var drainForThis = (i == childCount - 1) ? lastChildDrain : List.of(); - newChildren.add(rebuildChild(input, c, grammar, lineStarts, cursor, drainForThis)); + var rebuilt = rebuildChild(input, c, grammar, lineStarts, cursor, drainForThis); + if (newChildren == null) { + if (rebuilt != c) { + // First divergence: materialize the list with the children + // we have so far (all reference-equal to input) plus this one. + newChildren = new ArrayList<>(childCount); + for (int j = 0; j < i; j++ ) { + newChildren.add(children.get(j)); + } + newChildren.add(rebuilt); + } + }else { + newChildren.add(rebuilt); + } cursor = c.span() .endOffset(); } - return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), leading, extraTrailing); + if (newChildren == null && canSkip) { + // All children unchanged AND wrapper-level rebuild would be a no-op. + return nt; + } + var finalChildren = newChildren == null + ? children + : List.copyOf(newChildren); + return new CstNode.NonTerminal(nt.id(), nt.span(), nt.rule(), finalChildren, leading, extraTrailing); } private static List combine(List a, List b) { From ff596bebe1cfe362b2f794d836f1349a7deb7745 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 14:17:21 +0200 Subject: [PATCH 20/26] =?UTF-8?q?feat:=20StringSpan=20runtime=20foundation?= =?UTF-8?q?=20=E2=80=94=20CstNode.Terminal/Token=20hold=20StringSpan,=20.t?= =?UTF-8?q?ext()=20returns=20String=20via=20lazy=20materialization=20(Clea?= =?UTF-8?q?nup=20F.1)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../pragmatica/peg/parser/ParsingContext.java | 10 + .../org/pragmatica/peg/parser/PegEngine.java | 26 +- .../java/org/pragmatica/peg/tree/CstNode.java | 77 ++++- .../org/pragmatica/peg/tree/StringSpan.java | 141 +++++++++ .../pragmatica/peg/tree/TriviaPostPass.java | 14 +- .../pragmatica/peg/tree/StringSpanTest.java | 274 ++++++++++++++++++ .../peg/incremental/internal/TreeSplicer.java | 4 +- .../internal/TriviaRedistribution.java | 8 +- 8 files changed, 521 insertions(+), 33 deletions(-) create mode 100644 peglib-core/src/main/java/org/pragmatica/peg/tree/StringSpan.java create mode 100644 peglib-core/src/test/java/org/pragmatica/peg/tree/StringSpanTest.java diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java index 33d5678..bdff0ae 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/ParsingContext.java @@ -7,6 +7,7 @@ import org.pragmatica.peg.tree.IdGenerator; import org.pragmatica.peg.tree.SourceLocation; import org.pragmatica.peg.tree.SourceSpan; +import org.pragmatica.peg.tree.StringSpan; import org.pragmatica.peg.tree.Trivia; import java.util.ArrayList; @@ -192,6 +193,15 @@ public String substring(int start, int end) { return input.substring(start, end); } + /** + * Cleanup E (v0.5.1): produce a {@link StringSpan} view over a slice of the + * input without materializing a {@link String}. Used by Terminal/Token + * construction to defer substring allocation. + */ + public StringSpan substringSpan(int start, int end) { + return new StringSpan(input, start, end); + } + public String remainingInput() { return input.substring(pos); } diff --git a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java index 237b8e7..fb4fef8 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/parser/PegEngine.java @@ -1178,15 +1178,15 @@ private ParseResult parseTokenBoundaryWithActions(ParsingContext ctx, return result; } var endPos = ctx.pos(); - var text = ctx.substring(startPos, endPos); + var textSpan = ctx.substringSpan(startPos, endPos); // Set token capture so $0 returns this captured text - tokenCapture[0] = text; + tokenCapture[0] = textSpan.toString(); var span = ctx.spanFrom(startLoc); var node = new CstNode.Token(ctx.idGen() .next(), span, ruleName, - text, + textSpan, triviaPostPass ? List.of() : ctx.takePendingLeadingTrivia(), @@ -1584,13 +1584,13 @@ private ParseResult parseTokenBoundary(ParsingContext ctx, Expression.TokenBound return result; } var endPos = ctx.pos(); - var text = ctx.substring(startPos, endPos); + var textSpan = ctx.substringSpan(startPos, endPos); var span = ctx.spanFrom(startLoc); var node = new CstNode.Token(ctx.idGen() .next(), span, ruleName, - text, + textSpan, triviaPostPass ? List.of() : ctx.takePendingLeadingTrivia(), @@ -2213,11 +2213,11 @@ private static List concatTrivia(List first, List second private CstNode wrapWithRuleName(CstNode node, String ruleName, List leadingTrivia) { return switch (node) { case CstNode.Terminal t -> new CstNode.Terminal( - t.id(), t.span(), ruleName, t.text(), leadingTrivia, t.trailingTrivia()); + t.id(), t.span(), ruleName, t.textSpan(), leadingTrivia, t.trailingTrivia()); case CstNode.NonTerminal nt -> new CstNode.NonTerminal( nt.id(), nt.span(), ruleName, nt.children(), leadingTrivia, nt.trailingTrivia()); case CstNode.Token tok -> new CstNode.Token( - tok.id(), tok.span(), ruleName, tok.text(), leadingTrivia, tok.trailingTrivia()); + tok.id(), tok.span(), ruleName, tok.textSpan(), leadingTrivia, tok.trailingTrivia()); case CstNode.Error err -> new CstNode.Error( err.id(), err.span(), err.skippedText(), err.expected(), leadingTrivia, err.trailingTrivia()); }; @@ -2234,11 +2234,11 @@ private CstNode attachLeadingTrivia(CstNode node, List leadingTrivia) { } return switch (node) { case CstNode.Terminal t -> new CstNode.Terminal( - t.id(), t.span(), t.rule(), t.text(), leadingTrivia, t.trailingTrivia()); + t.id(), t.span(), t.rule(), t.textSpan(), leadingTrivia, t.trailingTrivia()); case CstNode.NonTerminal nt -> new CstNode.NonTerminal( nt.id(), nt.span(), nt.rule(), nt.children(), leadingTrivia, nt.trailingTrivia()); case CstNode.Token tok -> new CstNode.Token( - tok.id(), tok.span(), tok.rule(), tok.text(), leadingTrivia, tok.trailingTrivia()); + tok.id(), tok.span(), tok.rule(), tok.textSpan(), leadingTrivia, tok.trailingTrivia()); case CstNode.Error err -> new CstNode.Error( err.id(), err.span(), err.skippedText(), err.expected(), leadingTrivia, err.trailingTrivia()); }; @@ -2253,11 +2253,11 @@ private CstNode attachTrailingTrivia(CstNode node, List trailingTrivia) } return switch (node) { case CstNode.Terminal t -> new CstNode.Terminal( - t.id(), t.span(), t.rule(), t.text(), t.leadingTrivia(), trailingTrivia); + t.id(), t.span(), t.rule(), t.textSpan(), t.leadingTrivia(), trailingTrivia); case CstNode.NonTerminal nt -> new CstNode.NonTerminal( nt.id(), nt.span(), nt.rule(), nt.children(), nt.leadingTrivia(), trailingTrivia); case CstNode.Token tok -> new CstNode.Token( - tok.id(), tok.span(), tok.rule(), tok.text(), tok.leadingTrivia(), trailingTrivia); + tok.id(), tok.span(), tok.rule(), tok.textSpan(), tok.leadingTrivia(), trailingTrivia); case CstNode.Error err -> new CstNode.Error( err.id(), err.span(), err.skippedText(), err.expected(), err.leadingTrivia(), trailingTrivia); }; @@ -2291,12 +2291,12 @@ private CstNode attachTrailingToTail(CstNode node, List trivia) { nt.id(), nt.span(), nt.rule(), List.copyOf(newChildren), nt.leadingTrivia(), nt.trailingTrivia()); } case CstNode.Terminal t -> new CstNode.Terminal( - t.id(), t.span(), t.rule(), t.text(), t.leadingTrivia(), combineTrivia(t.trailingTrivia(), trivia)); + t.id(), t.span(), t.rule(), t.textSpan(), t.leadingTrivia(), combineTrivia(t.trailingTrivia(), trivia)); case CstNode.Token tok -> new CstNode.Token( tok.id(), tok.span(), tok.rule(), - tok.text(), + tok.textSpan(), tok.leadingTrivia(), combineTrivia(tok.trailingTrivia(), trivia)); case CstNode.Error err -> new CstNode.Error( diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/CstNode.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/CstNode.java index b2e9f71..ce86d30 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/CstNode.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/CstNode.java @@ -12,6 +12,11 @@ * {@code docs/incremental/ARCHITECTURE-0.5.0.md} §2). IDs are produced by * {@link IdGenerator} during parse and must be excluded from structural * equality (§7 R1) so two parses of the same input produce equal trees. + * + *

v0.5.1 (Cleanup E): {@link Terminal} and {@link Token} carry a + * {@link StringSpan textSpan} component instead of a fully-materialized + * {@code String}, deferring substring allocation. The {@code text()} accessor + * remains a {@link String}-returning method for source-compat with consumers. */ public sealed interface CstNode { /** @@ -44,25 +49,49 @@ public sealed interface CstNode { /** * Terminal node - a leaf that matched literal text. + * + *

v0.5.1 (Cleanup E): the {@code textSpan} component defers substring + * materialization. Use {@link #text()} to obtain a {@link String}. */ record Terminal( long id, SourceSpan span, String rule, - String text, + StringSpan textSpan, List leadingTrivia, List trailingTrivia) implements CstNode { + /** + * Convenience constructor accepting a {@link String} for the text. Wraps the + * string in a fully-materialized {@link StringSpan}, so no substring is taken. + */ + public Terminal(long id, + SourceSpan span, + String rule, + String text, + List leadingTrivia, + List trailingTrivia) { + this(id, span, rule, StringSpan.ofString(text), leadingTrivia, trailingTrivia); + } + + /** + * The matched text as a {@link String}. Lazily materializes the underlying + * {@link StringSpan} on first access. + */ + public String text() { + return textSpan.toString(); + } + @Override public boolean equals(Object other) { - return other instanceof Terminal that && Objects.equals(span, that.span) && Objects.equals(rule, that.rule) && Objects.equals(text, - that.text) && Objects.equals(leadingTrivia, - that.leadingTrivia) && Objects.equals(trailingTrivia, - that.trailingTrivia); + return other instanceof Terminal that && Objects.equals(span, that.span) && Objects.equals(rule, that.rule) && Objects.equals(textSpan, + that.textSpan) && Objects.equals(leadingTrivia, + that.leadingTrivia) && Objects.equals(trailingTrivia, + that.trailingTrivia); } @Override public int hashCode() { - return Objects.hash(Terminal.class, span, rule, text, leadingTrivia, trailingTrivia); + return Objects.hash(Terminal.class, span, rule, textSpan, leadingTrivia, trailingTrivia); } } @@ -94,25 +123,49 @@ public int hashCode() { /** * Token node - result of token boundary operator {@code < >}. * Captures the matched text as a single unit. + * + *

v0.5.1 (Cleanup E): the {@code textSpan} component defers substring + * materialization. Use {@link #text()} to obtain a {@link String}. */ record Token( long id, SourceSpan span, String rule, - String text, + StringSpan textSpan, List leadingTrivia, List trailingTrivia) implements CstNode { + /** + * Convenience constructor accepting a {@link String} for the text. Wraps the + * string in a fully-materialized {@link StringSpan}, so no substring is taken. + */ + public Token(long id, + SourceSpan span, + String rule, + String text, + List leadingTrivia, + List trailingTrivia) { + this(id, span, rule, StringSpan.ofString(text), leadingTrivia, trailingTrivia); + } + + /** + * The matched text as a {@link String}. Lazily materializes the underlying + * {@link StringSpan} on first access. + */ + public String text() { + return textSpan.toString(); + } + @Override public boolean equals(Object other) { - return other instanceof Token that && Objects.equals(span, that.span) && Objects.equals(rule, that.rule) && Objects.equals(text, - that.text) && Objects.equals(leadingTrivia, - that.leadingTrivia) && Objects.equals(trailingTrivia, - that.trailingTrivia); + return other instanceof Token that && Objects.equals(span, that.span) && Objects.equals(rule, that.rule) && Objects.equals(textSpan, + that.textSpan) && Objects.equals(leadingTrivia, + that.leadingTrivia) && Objects.equals(trailingTrivia, + that.trailingTrivia); } @Override public int hashCode() { - return Objects.hash(Token.class, span, rule, text, leadingTrivia, trailingTrivia); + return Objects.hash(Token.class, span, rule, textSpan, leadingTrivia, trailingTrivia); } } diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/StringSpan.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/StringSpan.java new file mode 100644 index 0000000..a759522 --- /dev/null +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/StringSpan.java @@ -0,0 +1,141 @@ +package org.pragmatica.peg.tree; + +import java.util.Objects; + +/** + * View into a substring of the parser input. Defers materialization to a + * Java {@link String} until first access via {@link #toString()}. + * + *

Created by parser match helpers (token boundary capture, literal/charclass/any + * Terminal construction) instead of eager {@code String.substring(int, int)}, which + * (in Java 9+ compact strings) allocates both a fresh {@code byte[]} and a + * {@code String} header. StringSpan eliminates both, deferring materialization until + * a consumer actually needs a {@link String}. + * + *

Memory retention: StringSpan holds a reference to the entire input. + * CST nodes that survive the parse will keep input alive. This is fine for the + * IDE plugin / incremental use case (input lives alongside the CST anyway). For + * one-shot parses the cost is bounded by input size. + * + *

Thread-safety: the parser is single-threaded; the lazy + * {@code materialized} field uses a plain write-once pattern with no + * synchronization. + * + *

Equality semantics: {@code equals} compares text content (not + * source/start/end), so two StringSpans pointing into different sources but with + * identical text are equal. This is required so consumers that put StringSpans + * in {@link java.util.Set} / {@link java.util.Map} compare by text. + */ +public final class StringSpan implements CharSequence { + private final String source; + private final int start; + private final int end; + private String materialized; + + // lazy cache; written-once + public StringSpan(String source, int start, int end) { + if (start < 0 || end < start || end > source.length()) { + throw new IndexOutOfBoundsException("start=" + start + ", end=" + end + ", length=" + source.length()); + } + this.source = source; + this.start = start; + this.end = end; + } + + public static StringSpan of(String source, int start, int end) { + return new StringSpan(source, start, end); + } + + /** + * Wrap a fully-materialized {@link String} as a StringSpan with no deferral. + * Used when a callsite already has a String (e.g. a literal pattern), so the + * String is captured directly and {@link #toString()} returns it without copying. + */ + public static StringSpan ofString(String text) { + var span = new StringSpan(text, 0, text.length()); + span.materialized = text; + return span; + } + + public String source() { + return source; + } + + public int start() { + return start; + } + + public int end() { + return end; + } + + @Override + public int length() { + return end - start; + } + + @Override + public char charAt(int index) { + if (index < 0 || index >= length()) { + throw new IndexOutOfBoundsException("index=" + index + ", length=" + length()); + } + return source.charAt(start + index); + } + + @Override + public CharSequence subSequence(int s, int e) { + if (s < 0 || e < s || e > length()) { + throw new IndexOutOfBoundsException("s=" + s + ", e=" + e + ", length=" + length()); + } + return new StringSpan(source, start + s, start + e); + } + + @Override + public String toString() { + var m = materialized; + if (m == null) { + m = source.substring(start, end); + materialized = m; + } + return m; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o instanceof StringSpan that) { + int len = this.length(); + if (len != that.length()) { + return false; + } + // Cheap path: same backing source and same offsets + if (this.source == that.source && this.start == that.start && this.end == that.end) { + return true; + } + // Compare characters + for (int i = 0; i < len; i++ ) { + if (this.source.charAt(this.start + i) != that.source.charAt(that.start + i)) { + return false; + } + } + return true; + } + return false; + } + + @Override + public int hashCode() { + // Match String.hashCode() so StringSpan-by-content semantics are predictable. + if (materialized != null) { + return materialized.hashCode(); + } + int h = 0; + int len = end - start; + for (int i = 0; i < len; i++ ) { + h = 31 * h + source.charAt(start + i); + } + return h; + } +} diff --git a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java index ee4206f..4f05b65 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/tree/TriviaPostPass.java @@ -270,7 +270,12 @@ private static CstNode.Terminal rebuildTerminal(CstNode.Terminal t, if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, t.leadingTrivia(), t.trailingTrivia())) { return t; } - return new CstNode.Terminal(t.id(), t.span(), t.rule(), t.text(), leading, combine(drainExtra, extraTrailing)); + return new CstNode.Terminal(t.id(), + t.span(), + t.rule(), + t.textSpan(), + leading, + combine(drainExtra, extraTrailing)); } private static CstNode.Token rebuildToken(CstNode.Token tk, @@ -280,7 +285,12 @@ private static CstNode.Token rebuildToken(CstNode.Token tk, if (canSkipLeafRebuild(leading, extraTrailing, drainExtra, tk.leadingTrivia(), tk.trailingTrivia())) { return tk; } - return new CstNode.Token(tk.id(), tk.span(), tk.rule(), tk.text(), leading, combine(drainExtra, extraTrailing)); + return new CstNode.Token(tk.id(), + tk.span(), + tk.rule(), + tk.textSpan(), + leading, + combine(drainExtra, extraTrailing)); } private static CstNode.Error rebuildError(CstNode.Error e, diff --git a/peglib-core/src/test/java/org/pragmatica/peg/tree/StringSpanTest.java b/peglib-core/src/test/java/org/pragmatica/peg/tree/StringSpanTest.java new file mode 100644 index 0000000..3a86ad1 --- /dev/null +++ b/peglib-core/src/test/java/org/pragmatica/peg/tree/StringSpanTest.java @@ -0,0 +1,274 @@ +package org.pragmatica.peg.tree; + +import org.junit.jupiter.api.Nested; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertSame; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class StringSpanTest { + + @Nested + class Construction { + @Test + void of_constructsValidSpan() { + var span = StringSpan.of("hello world", 0, 5); + + assertEquals(5, span.length()); + assertEquals("hello", span.toString()); + } + + @Test + void of_emptySpan_isLengthZero() { + var span = StringSpan.of("hello", 2, 2); + + assertEquals(0, span.length()); + assertEquals("", span.toString()); + } + + @Test + void of_negativeStart_throws() { + assertThrows(IndexOutOfBoundsException.class, + () -> StringSpan.of("hello", -1, 3)); + } + + @Test + void of_endLessThanStart_throws() { + assertThrows(IndexOutOfBoundsException.class, + () -> StringSpan.of("hello", 3, 1)); + } + + @Test + void of_endBeyondLength_throws() { + assertThrows(IndexOutOfBoundsException.class, + () -> StringSpan.of("hello", 0, 10)); + } + + @Test + void ofString_capturesStringDirectly() { + var src = "hello"; + var span = StringSpan.ofString(src); + + assertEquals(5, span.length()); + // toString() should return the SAME instance — no substring taken + assertSame(src, span.toString()); + } + } + + @Nested + class CharSequenceContract { + @Test + void length_matchesEndMinusStart() { + var span = StringSpan.of("abcdefghij", 2, 7); + + assertEquals(5, span.length()); + } + + @Test + void charAt_returnsCharsFromOffsetSlice() { + var span = StringSpan.of("hello world", 6, 11); + + assertEquals('w', span.charAt(0)); + assertEquals('o', span.charAt(1)); + assertEquals('r', span.charAt(2)); + assertEquals('l', span.charAt(3)); + assertEquals('d', span.charAt(4)); + } + + @Test + void charAt_negativeIndex_throws() { + var span = StringSpan.of("hello", 0, 5); + + assertThrows(IndexOutOfBoundsException.class, () -> span.charAt(-1)); + } + + @Test + void charAt_atOrPastLength_throws() { + var span = StringSpan.of("hello", 0, 5); + + assertThrows(IndexOutOfBoundsException.class, () -> span.charAt(5)); + } + + @Test + void subSequence_returnsNestedSpan() { + var outer = StringSpan.of("hello world", 0, 11); + var inner = outer.subSequence(6, 11); + + assertEquals("world", inner.toString()); + } + + @Test + void subSequence_outOfBounds_throws() { + var span = StringSpan.of("hello", 0, 5); + + assertThrows(IndexOutOfBoundsException.class, () -> span.subSequence(0, 10)); + } + } + + @Nested + class Materialization { + @Test + void toString_idempotent_returnsSameInstance() { + var span = StringSpan.of("hello world", 0, 5); + + var first = span.toString(); + var second = span.toString(); + + assertSame(first, second); + } + + @Test + void toString_matchesSubstring() { + var src = "the quick brown fox"; + var span = StringSpan.of(src, 4, 9); + + assertEquals(src.substring(4, 9), span.toString()); + } + } + + @Nested + class EqualityAndHash { + @Test + void equals_sameTextDifferentSources_returnsTrue() { + var a = StringSpan.of("hello", 0, 5); + var b = StringSpan.of("xxhelloxx", 2, 7); + + assertEquals(a, b); + assertEquals(b, a); + } + + @Test + void equals_differentText_returnsFalse() { + var a = StringSpan.of("hello", 0, 5); + var b = StringSpan.of("world", 0, 5); + + assertNotEquals(a, b); + } + + @Test + void equals_differentLength_returnsFalse() { + var a = StringSpan.of("hello", 0, 5); + var b = StringSpan.of("hello", 0, 3); + + assertNotEquals(a, b); + } + + @Test + void equals_self_returnsTrue() { + var span = StringSpan.of("hello", 0, 5); + + assertEquals(span, span); + } + + @Test + void equals_nonStringSpan_returnsFalse() { + var span = StringSpan.of("hello", 0, 5); + + assertNotEquals(span, "hello"); + } + + @Test + void hashCode_matchesStringHashCode() { + var src = "hello world"; + var span = StringSpan.of(src, 6, 11); + + // hashCode() must equal "world".hashCode() so consumers can + // compare StringSpan-by-content with String-keyed collections. + assertEquals("world".hashCode(), span.hashCode()); + } + + @Test + void hashCode_consistentBeforeAndAfterMaterialization() { + var span = StringSpan.of("hello world", 6, 11); + + int before = span.hashCode(); + span.toString(); // materialize + int after = span.hashCode(); + + assertEquals(before, after); + } + + @Test + void hashCode_equalSpans_haveEqualHash() { + var a = StringSpan.of("hello", 0, 5); + var b = StringSpan.of("xxhelloxx", 2, 7); + + assertEquals(a.hashCode(), b.hashCode()); + } + } + + @Nested + class RoundTrip { + @Test + void roundTrip_emptyString() { + var span = StringSpan.of("", 0, 0); + + assertEquals("", span.toString()); + assertEquals(0, span.length()); + } + + @Test + void roundTrip_singleChar() { + var span = StringSpan.of("x", 0, 1); + + assertEquals("x", span.toString()); + assertEquals(1, span.length()); + } + + @Test + void roundTrip_multilineSource() { + var src = "line1\nline2\nline3"; + var span = StringSpan.of(src, 6, 11); + + assertEquals("line2", span.toString()); + } + + @Test + void roundTrip_largeInput_doesNotCopyEagerly() { + // 100KB string; constructing a span should not materialize the substring + var huge = "x".repeat(100_000); + assertDoesNotThrow(() -> StringSpan.of(huge, 50_000, 50_010)); + } + + @Test + void source_returnsBackingString() { + var src = "hello world"; + var span = StringSpan.of(src, 6, 11); + + assertSame(src, span.source()); + } + + @Test + void startEnd_accessors() { + var span = StringSpan.of("hello world", 6, 11); + + assertEquals(6, span.start()); + assertEquals(11, span.end()); + } + } + + @Nested + class CharSequenceCompat { + @Test + void usableAsCharSequence() { + CharSequence cs = StringSpan.of("hello world", 0, 5); + + assertEquals("hello", cs.toString()); + assertEquals(5, cs.length()); + assertEquals('h', cs.charAt(0)); + } + + @Test + void subSequence_returnsCharSequence() { + var span = StringSpan.of("hello world", 0, 11); + CharSequence inner = span.subSequence(0, 5); + + assertTrue(inner instanceof StringSpan); + assertEquals("hello", inner.toString()); + } + } +} diff --git a/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TreeSplicer.java b/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TreeSplicer.java index bac6014..8560387 100644 --- a/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TreeSplicer.java +++ b/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TreeSplicer.java @@ -169,14 +169,14 @@ public static CstNode shiftAll(CstNode node, int delta) { t.id(), span, t.rule(), - t.text(), + t.textSpan(), shiftTriviaAll(t.leadingTrivia(), delta), shiftTriviaAll(t.trailingTrivia(), delta)); case CstNode.Token t -> new CstNode.Token( t.id(), span, t.rule(), - t.text(), + t.textSpan(), shiftTriviaAll(t.leadingTrivia(), delta), shiftTriviaAll(t.trailingTrivia(), delta)); case CstNode.Error e -> new CstNode.Error( diff --git a/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TriviaRedistribution.java b/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TriviaRedistribution.java index 210a00e..5b57f57 100644 --- a/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TriviaRedistribution.java +++ b/peglib-incremental/src/main/java/org/pragmatica/peg/incremental/internal/TriviaRedistribution.java @@ -328,8 +328,8 @@ private static CstNode shiftAndDescend(CstNode node, var newLeading = shiftTriviaListAfter(node.leadingTrivia(), delta, editEnd); var newTrailing = shiftTriviaListAfter(node.trailingTrivia(), delta, editEnd); return switch (node) { - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), newSpan, t.rule(), t.text(), newLeading, newTrailing); - case CstNode.Token t -> new CstNode.Token(t.id(), newSpan, t.rule(), t.text(), newLeading, newTrailing); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), newSpan, t.rule(), t.textSpan(), newLeading, newTrailing); + case CstNode.Token t -> new CstNode.Token(t.id(), newSpan, t.rule(), t.textSpan(), newLeading, newTrailing); case CstNode.Error e -> new CstNode.Error(e.id(), newSpan, e.skippedText(), @@ -429,8 +429,8 @@ private static CstNode rebuildOwnerWithRewrittenTrivia(CstNode owner, ownerSpan = owner.span(); } return switch (owner) { - case CstNode.Terminal t -> new CstNode.Terminal(t.id(), ownerSpan, t.rule(), t.text(), leading, trailing); - case CstNode.Token t -> new CstNode.Token(t.id(), ownerSpan, t.rule(), t.text(), leading, trailing); + case CstNode.Terminal t -> new CstNode.Terminal(t.id(), ownerSpan, t.rule(), t.textSpan(), leading, trailing); + case CstNode.Token t -> new CstNode.Token(t.id(), ownerSpan, t.rule(), t.textSpan(), leading, trailing); case CstNode.Error e -> new CstNode.Error(e.id(), ownerSpan, e.skippedText(), From 2fec82c38599d40d07c42a7628d3086dd4598545 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 14:25:25 +0200 Subject: [PATCH 21/26] refactor: CstParseResult.text from String to CharSequence (Cleanup F.2) --- .../peg/generator/ParserGenerator.java | 33 ++++++++++++------- 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index ab51e38..4189b9d 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -242,8 +242,8 @@ private static void validateConfig(Grammar grammar, ParserConfig config) { } for (var skip : config.packratSkipRules()) { if (lrRules.contains(skip)) { - throw new IllegalArgumentException( - "rule '" + skip + "' is left-recursive; cannot be in packratSkipRules"); + throw new IllegalArgumentException("rule '" + skip + + "' is left-recursive; cannot be in packratSkipRules"); } } } @@ -7135,12 +7135,17 @@ private CstNode attachTrailingToTail(CstNode node, List trivia) { private static final class CstParseResult { final boolean success; final CstNode node; // raw nullable - final String text; // raw nullable + // Cleanup F.2: widen text from String to CharSequence so the + // generator can emit StringSpan into this field in F.3 without + // forcing String materialization. Consumers (textOrEmpty helper, + // CstNode.Terminal/Token construction via wrapWithRuleName) all + // accept CharSequence, so this is a non-breaking widening. + final CharSequence text; // raw nullable final String expected; // raw nullable final SourceLocation endLocation; // raw nullable final boolean cutFailed; - private CstParseResult(boolean success, CstNode node, String text, String expected, SourceLocation endLocation, boolean cutFailed) { + private CstParseResult(boolean success, CstNode node, CharSequence text, String expected, SourceLocation endLocation, boolean cutFailed) { this.success = success; this.node = node; this.text = text; @@ -7153,7 +7158,7 @@ private CstParseResult(boolean success, CstNode node, String text, String expect boolean isFailure() { return !success; } boolean isCutFailure() { return !success && cutFailed; } - static CstParseResult success(CstNode node, String text, SourceLocation endLocation) { + static CstParseResult success(CstNode node, CharSequence text, SourceLocation endLocation) { return new CstParseResult(true, node, text, null, endLocation, false); } @@ -7163,7 +7168,7 @@ static CstParseResult success(CstNode node, String text, SourceLocation endLocat // their endLocation field discarded by callers, so skipping the // location() call here removes the dominant SourceLocation allocator // on the self-host fixture. - static CstParseResult successNoLoc(CstNode node, String text) { + static CstParseResult successNoLoc(CstNode node, CharSequence text) { return new CstParseResult(true, node, text, null, null, false); } @@ -7190,12 +7195,16 @@ CstParseResult asRegularFailure() { private static final class CstParseResult { final boolean success; final Option node; - final Option text; + // Cleanup F.2: widen text from Option to Option + // so F.3 can emit StringSpan into this field without forcing + // String materialization. Consumers (textOrEmpty helper) read + // via .or("") which produces CharSequence, accepted everywhere. + final Option text; final Option expected; final Option endLocation; final boolean cutFailed; - private CstParseResult(boolean success, Option node, Option text, Option expected, Option endLocation, boolean cutFailed) { + private CstParseResult(boolean success, Option node, Option text, Option expected, Option endLocation, boolean cutFailed) { this.success = success; this.node = node; this.text = text; @@ -7208,14 +7217,14 @@ private CstParseResult(boolean success, Option node, Option tex boolean isFailure() { return !success; } boolean isCutFailure() { return !success && cutFailed; } - static CstParseResult success(CstNode node, String text, SourceLocation endLocation) { - return new CstParseResult(true, Option.option(node), Option.some(text), Option.none(), Option.some(endLocation), false); + static CstParseResult success(CstNode node, CharSequence text, SourceLocation endLocation) { + return new CstParseResult(true, Option.option(node), Option.some(text), Option.none(), Option.some(endLocation), false); } // Phase 1.7 (D2): intermediate-result success that omits endLocation. // See mutable variant above for rationale. - static CstParseResult successNoLoc(CstNode node, String text) { - return new CstParseResult(true, Option.option(node), Option.some(text), Option.none(), Option.none(), false); + static CstParseResult successNoLoc(CstNode node, CharSequence text) { + return new CstParseResult(true, Option.option(node), Option.some(text), Option.none(), Option.none(), false); } static CstParseResult failure(String expected) { From 0b29c78118309e9db3c9edaa95bb729ef23eb0b8 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 15:04:32 +0200 Subject: [PATCH 22/26] =?UTF-8?q?perf:=20generator=20emits=20StringSpan=20?= =?UTF-8?q?for=20token=20text=20=E2=80=94=20defer=20substring=20materializ?= =?UTF-8?q?ation=20(Cleanup=20F.3)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../peg/generator/ParserGenerator.java | 116 ++++++++++++++++-- 1 file changed, 106 insertions(+), 10 deletions(-) diff --git a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java index 4189b9d..89595e4 100644 --- a/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java +++ b/peglib-core/src/main/java/org/pragmatica/peg/generator/ParserGenerator.java @@ -1947,20 +1947,26 @@ public sealed interface CstNode { List leadingTrivia(); List trailingTrivia(); - record Terminal(long id, SourceSpan span, RuleId rule, String text, + record Terminal(long id, SourceSpan span, RuleId rule, StringSpan textSpan, List leadingTrivia, List trailingTrivia) implements CstNode { + /** Convenience constructor — wraps a String as a fully-materialized StringSpan. */ + public Terminal(long id, SourceSpan span, RuleId rule, String text, + List leadingTrivia, List trailingTrivia) { + this(id, span, rule, StringSpan.ofString(text), leadingTrivia, trailingTrivia); + } + public String text() { return textSpan.toString(); } @Override public boolean equals(Object other) { return other instanceof Terminal that && java.util.Objects.equals(span, that.span) && java.util.Objects.equals(rule, that.rule) - && java.util.Objects.equals(text, that.text) + && java.util.Objects.equals(textSpan, that.textSpan) && java.util.Objects.equals(leadingTrivia, that.leadingTrivia) && java.util.Objects.equals(trailingTrivia, that.trailingTrivia); } @Override public int hashCode() { - return java.util.Objects.hash(Terminal.class, span, rule, text, leadingTrivia, trailingTrivia); + return java.util.Objects.hash(Terminal.class, span, rule, textSpan, leadingTrivia, trailingTrivia); } } @@ -1981,20 +1987,26 @@ public int hashCode() { } } - record Token(long id, SourceSpan span, RuleId rule, String text, + record Token(long id, SourceSpan span, RuleId rule, StringSpan textSpan, List leadingTrivia, List trailingTrivia) implements CstNode { + /** Convenience constructor — wraps a String as a fully-materialized StringSpan. */ + public Token(long id, SourceSpan span, RuleId rule, String text, + List leadingTrivia, List trailingTrivia) { + this(id, span, rule, StringSpan.ofString(text), leadingTrivia, trailingTrivia); + } + public String text() { return textSpan.toString(); } @Override public boolean equals(Object other) { return other instanceof Token that && java.util.Objects.equals(span, that.span) && java.util.Objects.equals(rule, that.rule) - && java.util.Objects.equals(text, that.text) + && java.util.Objects.equals(textSpan, that.textSpan) && java.util.Objects.equals(leadingTrivia, that.leadingTrivia) && java.util.Objects.equals(trailingTrivia, that.trailingTrivia); } @Override public int hashCode() { - return java.util.Objects.hash(Token.class, span, rule, text, leadingTrivia, trailingTrivia); + return java.util.Objects.hash(Token.class, span, rule, textSpan, leadingTrivia, trailingTrivia); } } """); @@ -2023,6 +2035,80 @@ public int hashCode() { sb.append(""" } + /** + * View into a substring of the parser input. Defers materialization to a + * Java String until first access via toString(). Cleanup F.3: token text in + * generated parsers flows through StringSpan to avoid eager substring() + * allocation in match helpers, token boundary capture, and DFA fast-path. + * + *

Embedded inline so the standalone-parser invariant (depends only on + * pragmatica-lite:core) is preserved. + */ + public static final class StringSpan implements CharSequence { + private final String source; + private final int start; + private final int end; + private String materialized; + + public StringSpan(String source, int start, int end) { + this.source = source; + this.start = start; + this.end = end; + } + + /** Wrap a fully-materialized String with no deferral. */ + public static StringSpan ofString(String text) { + var span = new StringSpan(text, 0, text.length()); + span.materialized = text; + return span; + } + + public int start() { return start; } + public int end() { return end; } + + @Override public int length() { return end - start; } + @Override public char charAt(int i) { return source.charAt(start + i); } + @Override public CharSequence subSequence(int s, int e) { + return new StringSpan(source, start + s, start + e); + } + + @Override + public String toString() { + var m = materialized; + if (m == null) { + m = source.substring(start, end); + materialized = m; + } + return m; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o instanceof StringSpan that) { + int len = this.length(); + if (len != that.length()) return false; + if (this.source == that.source && this.start == that.start && this.end == that.end) return true; + for (int i = 0; i < len; i++) { + if (this.source.charAt(this.start + i) != that.source.charAt(that.start + i)) return false; + } + return true; + } + return false; + } + + @Override + public int hashCode() { + if (materialized != null) return materialized.hashCode(); + int h = 0; + int len = end - start; + for (int i = 0; i < len; i++) { + h = 31 * h + source.charAt(start + i); + } + return h; + } + } + public sealed interface AstNode { SourceSpan span(); String rule(); @@ -2521,6 +2607,15 @@ private String substring(int start, int end) { return input.substring(start, end); } + /** + * Cleanup F.3: build a StringSpan view over the input without materializing + * a Java String. Used by match helpers and token boundary capture so the + * substring is deferred until a consumer actually needs it. + */ + private StringSpan substringSpan(int start, int end) { + return new StringSpan(input, start, end); + } + private long cacheKey(int ruleId, int position) { return ((long) ruleId << 32) | position; } @@ -5060,12 +5155,13 @@ private void emitCstFastPathTokenBoundary(StringBuilder sb, .append(" column = fpCol") .append(id) .append(";\n"); + // Cleanup F.3: emit StringSpan instead of input.substring(...) to defer + // materialization. Note: we use the parser instance's substringSpan() + // helper since fpInput is just a local alias for `input`. sb.append(pad) .append(" var tbText") .append(id) - .append(" = ") - .append(fpInput) - .append(".substring(") + .append(" = substringSpan(") .append(tbStartPos) .append(", ") .append(fpEnd) @@ -6332,7 +6428,7 @@ private void generateCstExpressionCode(StringBuilder sb, sb.append(pad) .append(" var tbText") .append(id) - .append(" = substring(") + .append(" = substringSpan(") .append(tbStartPos) .append(", pos);\n"); sb.append(pad) From 76498bf988851125a755483c8a58ceff0112d46e Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 15:11:45 +0200 Subject: [PATCH 23/26] =?UTF-8?q?docs:=20HANDOVER=20+=20CHANGELOG=20?= =?UTF-8?q?=E2=80=94=20Cleanup=20arc=20A=E2=86=92F.3=20+=20StringSpan=20su?= =?UTF-8?q?mmary?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 17 +++++++++++++++++ docs/HANDOVER.md | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 03f6a70..72bf3b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,23 @@ _Unreleased — patch cycle following 0.5.0._ - **`triviaPostPass` default flipped to `true`** (`3b372af`). After the fix, the default-flip succeeded with 0 test failures. Two sentinel tests in `TriviaPostPassFlagTest` inverted (DefaultOffNoOp → DefaultOnNoOp). Phase1/Phase2 parity tests insulated (explicitly pass `triviaPostPass=false`). The 8 fixture-pinning tests for the previous (buffer-driven) attribution remain green via explicit-OFF construction. **End-state for 0.5.1: post-pass attribution is the default; legacy buffer-driven attribution is opt-out via `new ParserConfig(..., triviaPostPass=false, ...)`.** - **Post-pass O(n²) bench regression fixed** (`6675479`). Initial A/B bench after default flip surfaced 22× / 239× wallclock regression on reference / selfhost fixtures. Investigation pinpointed `computeSpan(input, from, to)` re-scanning `[0, from)` from offset 0 on every trivia chunk to count newlines: O(K · N) where K trivia chunks scale with N input chars → O(N²). Empirical exponent reached 1.88 at N=32000. Fix: precompute a line-start table once per `assignTrivia` call (O(N)) + binary search per `computeSpan` (O(log N)). Both runtime `TriviaPostPass.java` and the embedded version emitted by `ParserGenerator.java` updated symmetrically. Post-fix bench: reference 26.87 ms (1.36× legacy 19.78 ms); selfhost 943.6 ms (1.01× legacy 934.7 ms — within noise). 15.6× speedup on reference, 210× on selfhost vs pre-fix. +- **Trivia rework cleanup arc — Cleanups A through F.3** (2026-05-09, commits `763e6e0` through `0b29c78`): + - **Cleanup A** (`763e6e0`): strip dead buffer call sites in PegEngine + emitted parsers under flag-ON. 38 PegEngine sites + parallel emit sites short-circuited via `if (!triviaPostPass)` guards; consumers use `List.of()` directly. + - **Cleanup B** (`884c4c8`): TriviaPostPass thread orphan trailing through recursion (replaces O(depth) spine rebuild) + probe-scan moved out of hot path. Selfhost flipped from parity to **-5% under legacy**. + - **Cleanup D** (`c682eef`): TriviaPostPass skip rebuild when no trivia change needed. Reference -2.6%; selfhost -11.1% (now strictly faster than legacy). + - **Cleanup F.1** (`ff596be`): StringSpan runtime foundation. New `org.pragmatica.peg.tree.StringSpan` (CharSequence view with lazy String materialization). CstNode.Terminal/Token components changed from `String text` to `StringSpan textSpan`; `.text()` accessor preserved as String API via `toString()`. 30 new StringSpanTest cases. + - **Cleanup F.2** (`2fec82c`): emitted CstParseResult.text widened from String to CharSequence — structural prerequisite for F.3. + - **Cleanup F.3** (`0b29c78`): generator emits inline StringSpan + uses it for token boundary capture and match helper text, eliminating per-call substring allocation on the bench's hot path. + + **Final bench numbers (post-F.3, vs legacy buffer-driven path):** + + | Fixture | Buffer-driven | Post-pass (F.3) | Δ | + |---|---:|---:|---:| + | Reference (1900 LOC) | 19.08 ms / 70.65 MB | 24.88 ms / 77.13 MB | +30% wallclock / +9% alloc | + | Selfhost (37k LOC) | 825.6 ms / 1908 MB | **784.7 ms / 1881 MB** | **-5% wallclock / -1.4% alloc** | + + Selfhost — the GC-bound fixture (52% G1 time per profile) — is now meaningfully faster than legacy buffer-driven attribution. Reference fixture remains +30% over legacy (per-NonTerminal scan cost dominates; addressable in future work). The trivia rework + StringSpan arc delivers a real win on the perf-critical workload (selfhost) while preserving correctness across all 22 RoundTripTest fixtures. + ### Removed ## [0.5.0] - 2026-05-06 diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index bb615bd..95e97eb 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -626,4 +626,4 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 trivia production rework + bench impact study + O(n²) fix. Branch `release-0.5.1` at `6675479` — 14 commits past 0.5.1 base. peglib-core 768 tests + 1 pre-existing skipped, all green at the new default (post-pass attribution). **Trivia rework state: SHIPPED + bench-validated.** `triviaPostPass=true` is default; selfhost wallclock at parity with legacy (within 1%); reference 36% slower (acceptable). Legacy buffer-driven attribution is opt-out. Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: Lever B retry (fallback-rule-bypass blocker remains; trivia-context-loss blocker is RESOLVED) OR Lever C IR unification. +**Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan multi-commit. Branch `release-0.5.1` at `0b29c78` — 20 commits past 0.5.1 base. peglib-core 805 tests + 1 pre-existing skipped, all green at the new default (post-pass + StringSpan). **Trivia rework + cleanup state: SHIPPED + bench-validated.** Cumulative selfhost wallclock now **-5% under legacy** (was at parity); reference fixture +30% over legacy (per-NonTerminal scan dominates; future work). 30 new StringSpanTest cases + StringSpan added as `org.pragmatica.peg.tree.StringSpan` (CharSequence view with lazy String materialization). Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: Lever B retry (fallback-rule-bypass blocker remains; trivia-context-loss blocker is RESOLVED) OR Lever C IR unification OR address reference-fixture per-NonTerminal scan cost (cache scanWhitespace, skip NonTerminals with full-coverage children, batch gap-scans). From 46d8a0531f9dc47f74c5388d099a504348cbfa3f Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 15:57:08 +0200 Subject: [PATCH 24/26] =?UTF-8?q?docs:=20HANDOVER=20=E2=80=94=20Cleanup=20?= =?UTF-8?q?G=20investigation=20(gap=20is=20structural,=20abandoned)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/HANDOVER.md | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 95e97eb..ecfb941 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -626,4 +626,26 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan multi-commit. Branch `release-0.5.1` at `0b29c78` — 20 commits past 0.5.1 base. peglib-core 805 tests + 1 pre-existing skipped, all green at the new default (post-pass + StringSpan). **Trivia rework + cleanup state: SHIPPED + bench-validated.** Cumulative selfhost wallclock now **-5% under legacy** (was at parity); reference fixture +30% over legacy (per-NonTerminal scan dominates; future work). 30 new StringSpanTest cases + StringSpan added as `org.pragmatica.peg.tree.StringSpan` (CharSequence view with lazy String materialization). Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: Lever B retry (fallback-rule-bypass blocker remains; trivia-context-loss blocker is RESOLVED) OR Lever C IR unification OR address reference-fixture per-NonTerminal scan cost (cache scanWhitespace, skip NonTerminals with full-coverage children, batch gap-scans). +**Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan + Cleanup G investigation (reference-fixture tightening attempted, REVERTED). + +### Cleanup G — reference-fixture tightening attempted, abandoned (2026-05-09) + +Two micro-optimizations attempted under bench gate (≥3% wallclock OR ≥5% alloc on reference); both REVERTED on evidence: + +- **G.1 — first-char prefilter on scanWhitespace.** Hypothesis: ~50% of scan calls return empty; precheck `input.charAt(prevEnd)` against whitespace-first-set to skip alloc. Reality: parser positions are always at token-content boundaries (skipWhitespace already advanced), so scan calls have either real whitespace or `from == to`. Existing `from >= to` short-circuit already catches the empty case; new prefilter rejected ~0 calls. Bench: alloc Δ +0.2%, wallclock noise. Reverted. + +- **G.2 — alloc tightening on non-empty scan path.** Three changes bundled: lineColAt returns long instead of int[2]; scanWhitespaceFast returns `List.of(single)` for size-1 chunks; combine() short-circuits empty. The combine change was already in place at `76498bf`. Other two micro-saves total ~32 KB out of 77 MB per-op (<0.05%). Bench: alloc Δ -0.4%, wallclock noise. Reverted. + +**Verdict: the +30% reference-fixture gap is structural, not micro-fixable.** Cost lives in: +- Spine rebuild on divergence (List.copyOf of children, NonTerminal record construction) +- Recursive scanWhitespace per node boundary +- Per-chunk Trivia node classification + +Closing the gap requires structural redesigns (NOT "tightening"): +1. Skip the post-pass entirely when CST already has correct trivia attribution (gate via engine-side flag) +2. Persistent immutable lists with structural sharing for trivia lists +3. Lazy SourceSpan materialization + +Each is its own spec. Selfhost (perf-critical) is already faster than legacy (-5%); the reference fixture's small inputs make the post-pass overhead a higher % of total but the absolute cost (~7 ms) is bounded. + + Branch `release-0.5.1` at `0b29c78` — 20 commits past 0.5.1 base. peglib-core 805 tests + 1 pre-existing skipped, all green at the new default (post-pass + StringSpan). **Trivia rework + cleanup state: SHIPPED + bench-validated.** Cumulative selfhost wallclock now **-5% under legacy** (was at parity); reference fixture +30% over legacy (per-NonTerminal scan dominates; future work). 30 new StringSpanTest cases + StringSpan added as `org.pragmatica.peg.tree.StringSpan` (CharSequence view with lazy String materialization). Move B post-mortem: [`docs/incremental/THROUGHPUT-ENGINE-MOVE-B.md`](incremental/THROUGHPUT-ENGINE-MOVE-B.md) §11. Next session: Lever B retry (fallback-rule-bypass blocker remains; trivia-context-loss blocker is RESOLVED) OR Lever C IR unification OR address reference-fixture per-NonTerminal scan cost (cache scanWhitespace, skip NonTerminals with full-coverage children, batch gap-scans). From b8c9abb631590865c46b5e2ffe95fa6fcfb35f00 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 16:16:24 +0200 Subject: [PATCH 25/26] =?UTF-8?q?docs:=20HANDOVER=20=E2=80=94=20Lever=20B?= =?UTF-8?q?=20retry=20failed=20on=20bench,=20deferred=20indefinitely?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/HANDOVER.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index ecfb941..42c21bd 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -628,6 +628,30 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid **Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan + Cleanup G investigation (reference-fixture tightening attempted, REVERTED). +### Lever B retry — attempted, FAILED on bench (2026-05-09) + +After the trivia rework removed one of two HANDOVER §6.2 blockers (trivia-context-loss), retried Lever B by wiring the dormant `SafePivotAnalyzer.safePivotRules(Grammar)` into `tryIncrementalReparse`. Gate: only accept rules in safe-pivot set; walk outward otherwise. + +**Bench result (IncrementalSessionBench, 1000 edits, Regime B):** + +| Metric | Pre-Lever-B | Post-gate | +|---|---:|---:| +| Median | 5.0 ms | **21.9 ms (+338%)** | +| p95 | 11.2 ms | 214.8 ms | +| % under 16ms | 96.5% | **43.6%** | + +Reverted. HEAD unchanged at `46d8a05`. + +**Root cause of failure:** SafePivotAnalyzer's "unambiguous literal prefix" criterion is too conservative for Java 25 grammar. Most rules start with character classes (`Identifier <- [a-zA-Z_] ...`) or rule references (`Type <- ClassType / ...`), which the analyzer marks unsafe. Walk-up-find-safe-ancestor strategy lands at root for ~56% of edits → forces full reparse. Parity is preserved but perf is catastrophically worse. + +**What this implies:** +- The existing "OLD walk-up + length check" passes IncrementalParityTest in current form, even without the safe-pivot gate. The §6.2 hypothetical bug ("smallestEnclosing descent strategy bypasses ancestors") doesn't actually fire in the current walk-up algorithm. +- Lever B retry would need: (a) a smarter analyzer that includes rules with disjoint first-sets (not just literal prefixes), or (b) on-accept structural validation instead of static analysis gate. +- Both are non-trivial (a few days each) and bench-gated. +- Until then, the current incremental engine's length-check-only acceptance is the de facto safe state. Median 5.0 ms / p95 11.2 ms is already excellent. + +**Defer Lever B retry indefinitely.** The trivia work was the real blocker; the perf is already good enough that strengthening the gate hasn't been worth the analysis investment. + ### Cleanup G — reference-fixture tightening attempted, abandoned (2026-05-09) Two micro-optimizations attempted under bench gate (≥3% wallclock OR ≥5% alloc on reference); both REVERTED on evidence: From 83f2c3ef10b0600fd4c7ef5256eae8fc80ac2774 Mon Sep 17 00:00:00 2001 From: Sergiy Yevtushenko Date: Sat, 9 May 2026 16:20:05 +0200 Subject: [PATCH 26/26] =?UTF-8?q?docs:=20HANDOVER=20=E2=80=94=20skip-postP?= =?UTF-8?q?ass-for-full-parses=20deferred=20(academic;=20no=20real=20workl?= =?UTF-8?q?oad=20benefits)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/HANDOVER.md | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/docs/HANDOVER.md b/docs/HANDOVER.md index 42c21bd..8177bd0 100644 --- a/docs/HANDOVER.md +++ b/docs/HANDOVER.md @@ -626,7 +626,22 @@ Do NOT pursue further allocation reduction in the 0.4.x interpreter — old guid --- -**Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan + Cleanup G investigation (reference-fixture tightening attempted, REVERTED). +**Last updated:** 2026-05-09, end of Step 4 trivia rework + cleanup arc A→F.3 + StringSpan + Cleanup G + Lever B retry attempt + skip-postPass deferral. + +### Skip postPass for full parses — deferred (2026-05-09) + +After Cleanup G abandoned the +30% reference gap as structural, considered the cleaner architectural fix: under flag-ON, run the buffer machinery for full parses (`parseCst`) and run postPass only for `parseRuleAt`. That preserves both the buffer's free-attribution-during-parse property AND the postPass's structural-parity-for-splice property. + +**Cost-benefit assessment (deferred for that reason):** + +- **Implementation cost:** several days. Cleanup A's call-site short-circuit (38 sites in PegEngine + parallel emit sites) needs to be REVERSED selectively for full-parse path. Generator must emit two parse-paths (full vs partial). Adds a `parseMode` parameter / context flag throughout. Bench-gated to validate buffer path didn't regress since Cleanup A. +- **Workloads that would benefit:** + - IDE plugin uses `parseRuleAt` (incremental), not `parseCst` — **no benefit** (postPass is mandatory for splice parity) + - Self-host workload — already at parity / faster than legacy — **no benefit** + - One-shot CLI / Maven plugin parsing — 25 ms vs 19 ms is **academic** at sub-frame budget +- **Workloads that would NOT benefit:** the perf-critical paths. + +The +30% reference gap is therefore academic for current workloads. Documented as available-if-needed; not pursued. ### Lever B retry — attempted, FAILED on bench (2026-05-09)