Release 0.6.2

CHANGELOG.md

@@ -5,6 +5,36 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.6.2] - 2026-06-06
+
+### Fixed
+
+- Shift operators (`<<`, `>>`, `>>>`) in field and local-variable initializer context
+  failed at `CompilationUnit` level. Root cause: `LShift`/`RShift`/`URShift` lexer rules
+  contain negative lookaheads (`!'='`), which the DFA builder cannot compile — the rules
+  were silently skipped while their assigned token kinds stayed live in the generated
+  parser, so the shift alternatives could never match. Fixed by inline expansion:
+  DFA-skipped LEXER rules of shape `literal+ (!literal)*` now expand in the generated
+  parser as adjacent constituent-token matches (byte-contiguity checked, so `1 < < 2`
+  does not fuse) with trailing negative lookaheads. Selfhost fixture (40K LOC) now
+  parses with 0 diagnostics; both previously `@Disabled` `Java25SelfHostDiagTest`
+  assertions re-enabled.
+- Per-iteration `%whitespace` tokenization: an unsplit folded `%whitespace <- (ws / comment / ...)*`
+  rule coalesced an entire trivia run into a single token classified only by prefix-sniffing.
+  The DFA builder now absorbs each closure alternative as its own structurally-classified trivia
+  kind, so the lexer emits one correctly-kinded token per iteration. The canonical `java25.peg`
+  `%whitespace` grammar-split workaround is reverted; the empty-match lexer warning no longer
+  fires for the folded form. Two previously `@Disabled` block-comment trivia tests re-enabled.
+
+### Added
+
+- Loud `fromGrammar` failure (`SkippedRuleReferenced`) when a parser rule references a
+  lexer rule whose token kind was assigned but whose DFA compilation was skipped without
+  an inline-expansion path — broken parsers are no longer generated silently.
+- `ShiftOperatorInlineExpansionTest` — 16 regression tests covering shift operators in
+  field/local-var/parenthesized/chained contexts, compound shift assignments, adjacency
+  non-fusing, nested generics, and the loud guard.
+
 ## [0.6.1] - 2026-05-12
 
 Patch release closing gaps from the 0.6.0 ship. Adds doc-comment trivia, completes

README.md

@@ -3,7 +3,7 @@
 A PEG (Parsing Expression Grammar) parser library for Java. Tokens-first lex-then-parse
 architecture, flat int[] CST, visitor pattern, true incremental reparse.
 
-Maven Central: `org.pragmatica-lite:peglib:0.6.1`
+Maven Central: `org.pragmatica-lite:peglib:0.6.2`
 
 Migrating from 0.5.x? See [`docs/MIGRATION-0.5-TO-0.6.md`](docs/MIGRATION-0.5-TO-0.6.md).
 Design rationale: [`docs/ARCHITECTURE-0.6.0.md`](docs/ARCHITECTURE-0.6.0.md).
@@ -29,7 +29,7 @@ Design rationale: [`docs/ARCHITECTURE-0.6.0.md`](docs/ARCHITECTURE-0.6.0.md).
 <dependency>
     <groupId>org.pragmatica-lite</groupId>
     <artifactId>peglib</artifactId>
-    <version>0.6.1</version>
+    <version>0.6.2</version>
 </dependency>
 ```
 
@@ -284,7 +284,7 @@ pre-compiled classes — no `fromGrammar` cost at runtime:
 <plugin>
     <groupId>org.pragmatica-lite</groupId>
     <artifactId>peglib-maven-plugin</artifactId>
-    <version>0.6.1</version>
+    <version>0.6.2</version>
     <executions>
         <execution>
             <goals><goal>generate-v6</goal></goals>
@@ -343,6 +343,7 @@ Full history in [`CHANGELOG.md`](CHANGELOG.md).
 
 | Version | Date | What |
 |---|---|---|
+| **0.6.2** | 2026-06-06 | Patch release. Shift operators (`<<`/`>>`/`>>>`) in field/local-var initializer context fixed via inline expansion of DFA-skipped lexer rules; loud `SkippedRuleReferenced` guard. Per-iteration `%whitespace` tokenization (folded form emits per-kind trivia; grammar-split workaround reverted). Selfhost fixture now parses with 0 diagnostics. |
 | **0.6.1** | 2026-05-12 | Patch release. Doc-comment trivia kinds (`KIND_DOC_LINE_COMMENT`, `KIND_DOC_BLOCK_COMMENT`), per-rule `%recover` runtime, `%checkpoint` directive parsing, named captures and back-references restored, `MIXED`-rule char-level fallback, diagnostic cap honored. |
 | **0.6.0** | 2026-05-11 | Clean-slate redesign. Tokens-first lex-then-parse, flat `int[]` CST, visitor pattern, always-on recovery, true partial reparse. ~12x faster than 0.5.x; parity-class with `javac`. **BREAKING** — see [migration guide](docs/MIGRATION-0.5-TO-0.6.md). |
 | **0.5.1** | 2026-05-08 | Final 0.5.x — selfhost stability and minor fixes. |

docs/ARCHITECTURE-0.6.0.md

@@ -807,7 +807,7 @@ Reference machine: same Apple Silicon used for 0.5.x bench session.
 | First-call (cold compile) | n/a | **≤ 600 ms** (one-time) | — |
 | Subsequent parse (warm cache) | n/a | reference parse target | — |
 
-For comparison: javac parses 1900-LOC Java in ~9 ms (parse-only mode). 0.6.0 target of ~10 ms for reference fixture brings peglib to **parity-with-or-faster-than javac** on Java parsing — while emitting strictly more output (CST + trivia tokens, vs javac's parse-only AST).
+For comparison: measured javac parse-only for the 1900-LOC reference is ≈2.2 ms; peglib's warm reference parse (≈2.68 ms as of 0.6.2) is ≈1.2× javac — **parity-class with javac** on Java parsing while emitting strictly more output (CST + trivia tokens, vs javac's parse-only AST).
 
 ---
 

docs/GRAMMAR-DSL.md

@@ -198,39 +198,33 @@ These appear at the top level of the grammar, alongside `%whitespace`.
 
 ### `%whitespace` shape and v6 per-kind trivia classification
 
-In the v6 (0.6.x) tokens-first lexer, trivia is classified by inspecting the
-matched span's leading prefix: `//` → `LINE_COMMENT`, `///` → `DOC_LINE_COMMENT`,
-`/*` → `BLOCK_COMMENT`, `/**` (followed by non-`/`) → `DOC_BLOCK_COMMENT`,
-otherwise `WHITESPACE`. The check only fires when the entire matched token IS
-the comment.
-
-This means **the shape of the `%whitespace` rule matters**. Two equivalent-looking
-forms produce very different token streams:
+In the v6 (0.6.x) tokens-first lexer, each `%whitespace` Choice alternative is
+absorbed into the DFA at its own trivia kind, decided **structurally** by the
+alternative's leading literal: a whitespace char-class alternative → `WHITESPACE`,
+an alternative beginning with `'//'` → `LINE_COMMENT`, with `'/*'` →
+`BLOCK_COMMENT`. The doc variants (`///` → `DOC_LINE_COMMENT`, `/**` followed by
+non-`/` → `DOC_BLOCK_COMMENT`) cannot be told apart from their regular form by
+the grammar alone (both share the same `'//' …` / `'/*' …` alternative), so they
+are refined from the matched span text after the structural base kind is set.
+
+As of 0.6.2 **both shapes below produce the same per-kind token stream** — the
+folded `(...)*` form no longer collapses a mixed-trivia run into a single
+`WHITESPACE` token:
 
 ```peg
-# Folds runs into ONE WHITESPACE-kind token — comment classification never fires:
+# Folded form — DfaBuilder.absorbWhitespace absorbs each alternative at its kind:
 %whitespace <- ([ \t\r\n] / '//' [^\n]* / '/*' (!'*/' .)* '*/')*
 
-# Emits one token per chunk — classification works:
+# Split form — equivalent token stream:
 %whitespace <- [ \t\r\n]+ / '//' [^\n]* / '/*' (!'*/' .)* '*/'
 ```
 
-The outer `*` in the first form causes the lexer's longest-match scan to consume
-multi-chunk runs (whitespace + comment + whitespace) as a single token whose
-first char is whitespace — the prefix check fails and the entire run stays
-`KIND_WHITESPACE`. Dropping the outer `*` and making the inner whitespace
-alternative `[ \t\r\n]+` lets each invocation match exactly one chunk; the
-lexer's outer maximal-munch loop iterates and emits separate tokens, each
-classified correctly.
-
-A side benefit: the split form does NOT match the empty string, so the
-"`LEXER rule '%whitespace' matches the empty string`" warning emitted by
-`peglib:generate-v6` goes away.
-
-A 0.6.2 follow-up will add per-iteration trivia tokenization so the folded
-form also produces per-kind tokens, but until then the split form is the
-canonical shape. The example grammar at
-`peglib-core/src/test/resources/java25.peg` uses the split form.
+The folded form's outer `*` is dropped during absorption (the lexer's own
+maximal-munch loop supplies the repetition), and the whitespace char-class
+alternative is absorbed as one-or-more, so the DFA start state never accepts the
+empty string — the `"LEXER rule '%whitespace' matches the empty string"` warning
+does **not** fire for either shape. The canonical example grammar at
+`peglib-core/src/test/resources/java25.peg` uses the folded form.
 
 ### `%suggest`
 

docs/HANDOVER.md

@@ -1,6 +1,28 @@
 # peglib — Handover
 
-**Last updated:** 2026-05-14 — **0.6.1 SHIPPING to Maven Central.** Closing checkpoint after Session 6.
+**Last updated:** 2026-06-06 — 0.6.2 release branch (release-0.6.2)
+
+---
+
+## Session 7 — 0.6.2 (2026-06-06)
+
+### State at a glance
+
+| | |
+|---|---|
+| **Release** | **v0.6.2** on `release-0.6.2` (patch) |
+| **Tests** | **1420 passing** across 7 modules, 0 failures, 2 pre-existing skips |
+| **Java25 corpus** | 20/20 clean parse |
+| **Selfhost (40K LOC)** | **0 diagnostics**, 1.26M CST nodes |
+| **Real-world Java** | FactoryClassGenerator (1900 LOC): 0 diagnostics |
+| **JMH** | reference 2.68 ms, selfhost 71.9 ms — flat vs 0.6.1 |
+
+### What landed in session 7 (0.6.2 patch items)
+
+1. **Shift operators (`<<`/`>>`/`>>>`) in field/local-var initializer context** — fixed. Root cause was NOT the 0.6.1 rollback hypothesis (`Type`/`Relational`/`TypeArgs` `<` rollback); it was a **dead token kind from a lexer rule silently skipped by the DFA**. Fix: inline expansion of DFA-skipped lexer rules of the `literal+ (!literal)*` shape, plus a loud `SkippedRuleReferenced` guard at `fromGrammar`. Both `Java25SelfHostDiagTest` assertions re-enabled; selfhost fixture (40K LOC) now parses with **0 diagnostics**, 1,261,302 CST nodes.
+2. **Per-iteration `%whitespace` tokenization** — folded `%whitespace <- (...)*` now emits per-kind trivia tokens via per-alternative DFA absorption. The c4169b6 canonical-grammar split workaround is **REVERTED** in `java25.peg`; the empty-match warning is gone for the folded form; 2 previously-disabled trivia tests re-enabled.
+
+Skips moved 4 → 2 (remaining: `LexerGeneratorTest` parity 1, `TriviaAdversarialTest$OptionalCutFailurePending` 1).
 
 ---
 
@@ -43,18 +65,18 @@
 - BASIC/ADVANCED recovery split; inline `{...}` action blocks; `AstNode` type; packrat memoization
 
 **Deferred to 0.6.2 or 0.7**:
-- **Shift-in-FieldDecl bug** (0.6.2 target): shift operators in field/local-var init context fail at `CompilationUnit` level. Hypothesis: `Type`/`Relational`/`TypeArgs` `<` literal rollback corrupts state. 2 `Java25SelfHostDiagTest` assertions `@Disabled` pending fix.
-- **Per-iteration `%whitespace` tokenization** (Item A harder part): ZeroOrMore loop itself should drive per-iteration token emission rather than relying on grammar split. Deferred.
-- **jbct v6 API migration** (0.6.2): jbct `46ac5e993` applied the `%whitespace` grammar split fix; full peglib v6 API migration remains.
+- **Shift-in-FieldDecl bug** (0.6.2 target): shift operators in field/local-var init context fail at `CompilationUnit` level. Hypothesis: `Type`/`Relational`/`TypeArgs` `<` literal rollback corrupts state. 2 `Java25SelfHostDiagTest` assertions `@Disabled` pending fix. ✅ RESOLVED in 0.6.2
+- **Per-iteration `%whitespace` tokenization** (Item A harder part): ZeroOrMore loop itself should drive per-iteration token emission rather than relying on grammar split. Deferred. ✅ RESOLVED in 0.6.2
+- **jbct v6 API migration** (0.6.2): jbct `46ac5e993` applied the `%whitespace` grammar split fix; full peglib v6 API migration remains. (split workaround obsolete since 0.6.2 — folded form now emits per-kind trivia; full jbct v6 API migration still pending)
 - **JBCT `<skip>true</skip>`** in `peglib-core/pom.xml`: formatter convergence bug on 5 v6 files. Lint passes cleanly; tracking upstream.
 - **Items H (token pool), I (lexer modes), K (JBCT plugin bump)**: 0.7+ backlog.
 
 ### Quick-start for next session
 
 1. **Read this file**, then `docs/ARCHITECTURE-0.6.0.md` if going architectural.
 2. **Read `CLAUDE.md`** — banked lessons (parser-domain + collaboration notes).
-3. **Verify**: `mvn install -Djbct.skip=true` → 1440 tests pass.
-4. **0.6.2 target**: fix shift-in-FieldDecl bug (see `Java25SelfHostDiagTest` disabled assertions), then re-enable the gate.
+3. **Verify**: `mvn install -Djbct.skip=true` → 1420 tests pass.
+4. **0.6.2 target**: fix shift-in-FieldDecl bug (see `Java25SelfHostDiagTest` disabled assertions), then re-enable the gate. (done in 0.6.2)
 5. **Agents**: `jbct-coder` for ALL coding, `build-runner` for `mvn`, `chore-runner` for git/changelog.
 
 ---
@@ -108,7 +130,7 @@
 - Per-iteration trivia tokens for `%whitespace` ZeroOrMore (currently coalesces into single token)
 - Named captures + back-references runtime (rejects at `fromGrammar` with helpful error)
 - JBCT `<skip>true</skip>` in `peglib-core/pom.xml` (lint passes cleanly; only the upstream formatter has a convergence bug on 5 v6 files)
-- True partial parse on `selfhost` (40K LOC) fixture (parses with some diagnostics — grammar gaps in specific Java patterns)
+- True partial parse on `selfhost` (40K LOC) fixture (parses with some diagnostics — grammar gaps in specific Java patterns) (resolved: 0 diagnostics as of 0.6.2)
 
 ### Possible next-session targets
 
@@ -662,7 +684,7 @@ Reference machine: same Apple Silicon used for 0.5.x bench session. Numbers from
 | Incremental edit median (Regime B) | 5.0 ms | **≤ 3 ms** | ≤ 1 ms |
 | First-call (cold compile) | n/a | **≤ 600 ms** (one-time) | — |
 
-For comparison: javac parses 1900-LOC Java in ~9 ms (parse-only). 0.6.0 reference target of ~10 ms = parity with javac while emitting strictly more output.
+For comparison: measured javac parse-only for the 1900-LOC reference is ≈2.2 ms; peglib's warm reference parse (≈2.68 ms as of 0.6.2) is ≈1.2× javac while emitting strictly more output.
 
 ---
 

docs/VISITOR-TUTORIAL.md

@@ -93,7 +93,7 @@ Configure `peglib-maven-plugin` in your `pom.xml`:
 <plugin>
     <groupId>org.pragmatica-lite</groupId>
     <artifactId>peglib-maven-plugin</artifactId>
-    <version>0.6.1</version>
+    <version>0.6.2</version>
     <executions>
         <execution>
             <goals><goal>generate-v6</goal></goals>
@@ -483,7 +483,7 @@ emitting either the original token text or your transformed version per node.
   for dropping actions in favor of `GVisitor<T>`.
 - [`peglib-core/src/main/java/org/pragmatica/peg/v6/generator/VisitorGenerator.java`](../peglib-core/src/main/java/org/pragmatica/peg/v6/generator/VisitorGenerator.java)
   — the codegen source. Read it once; it's short.
-- [`peglib-core/src/main/java/org/pragmatica/peg/v6/cst/CstArray.java`](../peglib-core/src/main/java/org/pragmatica/peg/v6/cst/CstArray.java)
+- [`peglib-core/src/main/java/org/pragmatica/peg/v6/cst/CstArray.java`](../peglib-runtime/src/main/java/org/pragmatica/peg/v6/cst/CstArray.java)
   — full CST API surface: `children`, `descendants`, `viewAt`, `kindAt`,
   `kindNameAt`, `firstChildAt`, `nextSiblingAt`, `textAt`, `spanStart`,
   `spanEnd`, `leadingTriviaTokens`, `trailingTriviaTokens`.

peglib-core/pom.xml

@@ -7,7 +7,7 @@
     <parent>
         <groupId>org.pragmatica-lite</groupId>
         <artifactId>peglib-parent</artifactId>
-        <version>0.6.1</version>
+        <version>0.6.2</version>
         <relativePath>../pom.xml</relativePath>
     </parent>
 

peglib-core/src/main/java/org/pragmatica/peg/v6/PegParser.java

@@ -105,7 +105,12 @@ private static Result<CompiledParser> compileParser(Grammar grammar,
                                                         RuleClassifier.Classification classification,
                                                         DfaBuilder.Built built,
                                                         String className) {
-        return ParserGenerator.generate(grammar, classification, built.kinds(), GENERATED_PACKAGE, className)
+        var skippedReasons = built.skipped()
+                                  .stream()
+                                  .collect(java.util.stream.Collectors.toMap(DfaBuilder.SkippedRule::ruleName,
+                                                                             DfaBuilder.SkippedRule::reason,
+                                                                             (a, __) -> a));
+        return ParserGenerator.generate(grammar, classification, built.kinds(), skippedReasons, GENERATED_PACKAGE, className)
         .flatMap(ParserCompiler::compile);
     }
 }

peglib-core/src/main/java/org/pragmatica/peg/v6/generator/LexerGenerator.java

@@ -297,12 +297,18 @@ private static void renderLexMethod(StringBuilder sb, int alphabet, boolean hasR
             sb.append("                }\n");
             sb.append("            }\n");
         }
-        // Phase A.6 / 0.6.1 — content-based trivia classification (mirrors LexerEngine).
+        // Phase A.6 / 0.6.1 / 0.6.2 — content-based trivia refinement (mirrors LexerEngine).
+        // Fires for the legacy coalesced WHITESPACE token AND for structurally-assigned
+        // LINE_COMMENT / BLOCK_COMMENT base kinds whose doc variant can only be told
+        // apart by the matched text.
         // //         → LINE_COMMENT          (also // without third '/')
         // ///        → DOC_LINE_COMMENT      (3+ slashes)
         // /* ... */  → BLOCK_COMMENT
         // /** ... */ → DOC_BLOCK_COMMENT     (NOT '/**/' — smallest empty block)
-        sb.append("            if (lastAcceptKind == TokenArray.KIND_WHITESPACE && lastAcceptEnd > pos + 1) {\n");
+        sb.append("            if ((lastAcceptKind == TokenArray.KIND_WHITESPACE\n");
+        sb.append("                 || lastAcceptKind == TokenArray.KIND_LINE_COMMENT\n");
+        sb.append("                 || lastAcceptKind == TokenArray.KIND_BLOCK_COMMENT)\n");
+        sb.append("                && lastAcceptEnd > pos + 1) {\n");
         sb.append("                char c0 = input.charAt(pos);\n");
         sb.append("                char c1 = input.charAt(pos + 1);\n");
         sb.append("                if (c0 == '/') {\n");