Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c9502c5
chore: prepare release 0.5.1
siy May 9, 2026
d2cc6be
test: trivia adversarial corpus + findings (Step 2 of context-indepen…
siy May 9, 2026
78c4003
feat: TriviaPostPass — context-independent trivia attribution prototy…
siy May 9, 2026
51144a6
docs: HANDOVER §11 — trivia investigation arc + Step 3 verdict (produ…
siy May 9, 2026
318f5cf
feat: triviaPostPass flag wires TriviaPostPass as post-parse hook (St…
siy May 9, 2026
d7b3496
feat: TriviaPostPass splice-offset overload for parseRuleAt subtree l…
siy May 9, 2026
605327b
perf: ParsingContext buffer ops no-op under triviaPostPass flag (Step…
siy May 9, 2026
dcd146f
feat: ParserGenerator emits embedded TriviaPostPass under flag-ON (St…
siy May 9, 2026
149bc57
docs: HANDOVER §11 + CHANGELOG [0.5.1] — trivia rework Step 4 commits…
siy May 9, 2026
4ed1cf5
perf: ParserGenerator emits no-op buffer methods under triviaPostPass…
siy May 9, 2026
4beca0a
docs: HANDOVER + CHANGELOG — Step 4 commit 5 landed; commit 6 reverte…
siy May 9, 2026
612fbea
fix: TriviaPostPass drains orphan trivia into last-leaf descendant ma…
siy May 9, 2026
3b372af
feat: triviaPostPass default flips to true (Step 4 commit 6)
siy May 9, 2026
fc185db
docs: HANDOVER + CHANGELOG — Step 4 trivia rework COMPLETE (commits 7…
siy May 9, 2026
6675479
fix: TriviaPostPass uses line-start table to eliminate O(n²) computeSpan
siy May 9, 2026
b05a57c
docs: HANDOVER + CHANGELOG — bench impact study + O(n²) fix banked
siy May 9, 2026
763e6e0
perf: skip buffer call sites under triviaPostPass flag-ON (Cleanup A)
siy May 9, 2026
884c4c8
perf: TriviaPostPass thread orphan trailing + probe-scan cache (Clean…
siy May 9, 2026
c682eef
perf: TriviaPostPass skip rebuild when no trivia change needed (Clean…
siy May 9, 2026
ff596be
feat: StringSpan runtime foundation — CstNode.Terminal/Token hold Str…
siy May 9, 2026
2fec82c
refactor: CstParseResult.text from String to CharSequence (Cleanup F.2)
siy May 9, 2026
0b29c78
perf: generator emits StringSpan for token text — defer substring mat…
siy May 9, 2026
76498bf
docs: HANDOVER + CHANGELOG — Cleanup arc A→F.3 + StringSpan summary
siy May 9, 2026
46d8a05
docs: HANDOVER — Cleanup G investigation (gap is structural, abandoned)
siy May 9, 2026
b8c9abb
docs: HANDOVER — Lever B retry failed on bench, deferred indefinitely
siy May 9, 2026
83f2c3e
docs: HANDOVER — skip-postPass-for-full-parses deferred (academic; no…
siy May 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,46 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.5.1] - 2026-05-09

_Unreleased — patch cycle following 0.5.0._
Comment on lines +8 to +10

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Remove the “Unreleased” label from the dated 0.5.1 release section.

Line 10 contradicts the dated heading at Line 8 (2026-05-09) and may confuse consumers parsing release state from the changelog.

- _Unreleased — patch cycle following 0.5.0._
+ _Patch cycle following 0.5.0._
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
## [0.5.1] - 2026-05-09
_Unreleased — patch cycle following 0.5.0._
## [0.5.1] - 2026-05-09
_Patch cycle following 0.5.0._
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@CHANGELOG.md` around lines 8 - 10, The changelog entry for the 0.5.1 release
shows a dated header "## [0.5.1] - 2026-05-09" but still contains the line
"_Unreleased — patch cycle following 0.5.0._"; remove or replace that line so
the dated section no longer includes the "Unreleased" label (edit the paragraph
under the header "## [0.5.1] - 2026-05-09" to reflect a released state or remove
the sentence entirely).


### Added

- **Trivia investigation arc — Steps 1-3** (2026-05-09): catalog of context-dependencies in current attribution, 17-test adversarial corpus targeting 7 divergence categories, post-pass prototype validating context-independent attribution. Round-trip preservation confirmed on 21/21 corpus + 9/9 adversarial inputs; 0 text loss across 46,756 nodes; `parseRuleAt` structural parity achievable under post-pass (load-bearing claim for incremental engine Lever B). Findings: [`docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md`](docs/incremental/TRIVIA-ADVERSARIAL-FINDINGS.md). Verdict in HANDOVER §11.
- **Trivia rework Step 4 commits 1-4** (production rework, foundational): opt-in `triviaPostPass` flag on `ParserConfig` (default `false` — zero behavior change). When enabled: post-parse hook applies `TriviaPostPass.assignTrivia(input, cst, grammar)` overriding buffer-driven attribution with context-independent re-derivation from `(input, span)` coordinates. `parseRuleAt` honors splice offset (4-arg overload `assignTrivia(input, cst, grammar, leadingScanFrom)`) — body subtree is structurally identical to corresponding subtree of full reparse, removing one of two HANDOVER §6.4 blockers for incremental engine Lever B. Generator-emitted parsers embed an inline post-pass equivalent (preserves standalone-parser invariant).
- **`org.pragmatica.peg.tree.TriviaPostPass`** (new public API): pure function `(input, cst, grammar) → CstNode` that re-derives leading/trailing trivia from coordinates without parse-history dependency. 4-arg overload supports splice scenarios.

### Changed

- `ParsingContext` pending-trivia methods (`appendPendingLeadingTrivia`, `takePendingLeadingTrivia`, `savePendingLeadingTrivia`, `restorePendingLeadingTrivia`) early-out when `config.triviaPostPass()=true` — buffer becomes dead weight under the flag and is skipped (38 call sites in `PegEngine` no-op without source changes).
- **Trivia rework Step 4 commit 5** (`4ed1cf5`): `ParserGenerator` emits no-op buffer methods under flag-ON. Mirror of commit 3 for the generator side — pure CPU optimization; correctness already from commit 4. Generated parsers under flag-ON no longer do dead-work buffer maintenance.

### Fixed

- **Trivia post-pass orphan-trivia placement matches engine's Bug C' compensation** (`612fbea`). Earlier commit 6 attempt revealed text-loss on Java 25 corpus (`void m()` → `voidm()`). Root cause: post-pass attached orphan trivia to wrapper.trailingTrivia, but `CstReconstruct.emit` only emits trailing for last-child of a wrapper — non-last-wrapper-trailing was invisible during reconstruction. Fix mirrors engine's `attachTrailingToTail` — drains orphan trivia into deepest-rightmost-leaf descendant. Plus a generator-semantics adapter: `rebuildNonTerminal` probe-scans for caller-supplied leading and skips it to prevent license-header double-emission on fixtures whose wrappers' spans include their own leading trivia. RoundTripTest 22/22 under flag-ON across all 22 corpus fixtures including `large/FactoryClassGenerator.java.txt`.
- **`triviaPostPass` default flipped to `true`** (`3b372af`). After the fix, the default-flip succeeded with 0 test failures. Two sentinel tests in `TriviaPostPassFlagTest` inverted (DefaultOffNoOp → DefaultOnNoOp). Phase1/Phase2 parity tests insulated (explicitly pass `triviaPostPass=false`). The 8 fixture-pinning tests for the previous (buffer-driven) attribution remain green via explicit-OFF construction. **End-state for 0.5.1: post-pass attribution is the default; legacy buffer-driven attribution is opt-out via `new ParserConfig(..., triviaPostPass=false, ...)`.**
- **Post-pass O(n²) bench regression fixed** (`6675479`). Initial A/B bench after default flip surfaced 22× / 239× wallclock regression on reference / selfhost fixtures. Investigation pinpointed `computeSpan(input, from, to)` re-scanning `[0, from)` from offset 0 on every trivia chunk to count newlines: O(K · N) where K trivia chunks scale with N input chars → O(N²). Empirical exponent reached 1.88 at N=32000. Fix: precompute a line-start table once per `assignTrivia` call (O(N)) + binary search per `computeSpan` (O(log N)). Both runtime `TriviaPostPass.java` and the embedded version emitted by `ParserGenerator.java` updated symmetrically. Post-fix bench: reference 26.87 ms (1.36× legacy 19.78 ms); selfhost 943.6 ms (1.01× legacy 934.7 ms — within noise). 15.6× speedup on reference, 210× on selfhost vs pre-fix.

- **Trivia rework cleanup arc — Cleanups A through F.3** (2026-05-09, commits `763e6e0` through `0b29c78`):
- **Cleanup A** (`763e6e0`): strip dead buffer call sites in PegEngine + emitted parsers under flag-ON. 38 PegEngine sites + parallel emit sites short-circuited via `if (!triviaPostPass)` guards; consumers use `List.of()` directly.
- **Cleanup B** (`884c4c8`): TriviaPostPass thread orphan trailing through recursion (replaces O(depth) spine rebuild) + probe-scan moved out of hot path. Selfhost flipped from parity to **-5% under legacy**.
- **Cleanup D** (`c682eef`): TriviaPostPass skip rebuild when no trivia change needed. Reference -2.6%; selfhost -11.1% (now strictly faster than legacy).
- **Cleanup F.1** (`ff596be`): StringSpan runtime foundation. New `org.pragmatica.peg.tree.StringSpan` (CharSequence view with lazy String materialization). CstNode.Terminal/Token components changed from `String text` to `StringSpan textSpan`; `.text()` accessor preserved as String API via `toString()`. 30 new StringSpanTest cases.
- **Cleanup F.2** (`2fec82c`): emitted CstParseResult.text widened from String to CharSequence — structural prerequisite for F.3.
- **Cleanup F.3** (`0b29c78`): generator emits inline StringSpan + uses it for token boundary capture and match helper text, eliminating per-call substring allocation on the bench's hot path.

**Final bench numbers (post-F.3, vs legacy buffer-driven path):**

| Fixture | Buffer-driven | Post-pass (F.3) | Δ |
|---|---:|---:|---:|
| Reference (1900 LOC) | 19.08 ms / 70.65 MB | 24.88 ms / 77.13 MB | +30% wallclock / +9% alloc |
| Selfhost (37k LOC) | 825.6 ms / 1908 MB | **784.7 ms / 1881 MB** | **-5% wallclock / -1.4% alloc** |

Selfhost — the GC-bound fixture (52% G1 time per profile) — is now meaningfully faster than legacy buffer-driven attribution. Reference fixture remains +30% over legacy (per-NonTerminal scan cost dominates; addressable in future work). The trivia rework + StringSpan arc delivers a real win on the perf-critical workload (selfhost) while preserving correctness across all 22 RoundTripTest fixtures.

### Removed

## [0.5.0] - 2026-05-06

_Work in progress — incremental-native architectural rework. See `docs/incremental/ARCHITECTURE-0.5.0.md`._
Expand Down
Loading
Loading