Merge pull request #90 from Aaronontheweb/feature/v03-bash-heredoc-substitution

Discover Bash heredoc substitutions

Author: Aaronontheweb

IMPLEMENTATION_PLAN.md

@@ -265,8 +265,18 @@ priorities.
       depth limits, comment-safe delimiter scanning, and fail-closed command
       identities, background lists, assignment prefixes, backticks, heredocs,
       and malformed interiors.
-- [ ] Extend Bash substitution discovery to iterables and expanding heredoc
-      bodies, then add the corresponding Netclaw approval-matrix cases.
+- [x] Extend Bash substitution discovery to expanding heredoc bodies. The
+      bounded slice recognizes quoted, escaped, mixed, and tab-stripping
+      delimiters; preserves exact body/terminator provenance; surfaces nested
+      substitutions in authored order with isolated state; and rejects header
+      tails, queued heredocs, backticks, arithmetic, continuations, incomplete
+      interiors, and depth overflow atomically. Direct and executable-corpus
+      cases pin exact syntax, command ancestry, spans, completeness, and
+      literal-versus-expanding behavior; real-Bash output and parse-only
+      oracles independently pin the bounded semantic boundary.
+- [ ] Extend Bash substitution discovery to iterables with the complete
+      `for ... in` vertical slice, then add the Bash substitution cases to the
+      Netclaw approval matrix.
 - [ ] Complete PowerShell `$()` discovery in `foreach` expressions and add the
       Netclaw approval-matrix cases. The simple-command slice is delivered for
       ordinary, adjacent, quoted, here-string, redirect, standalone,
@@ -276,9 +286,9 @@ priorities.
 - [ ] Deliver Bash `for ... in` and PowerShell `foreach` as the first two
       language-specific vertical slices, then extract only the shared analysis
       proven by both implementations.
-- [ ] Preserve the existing Bash heredoc grammar, fix quoted-delimiter
-      adjacency, expose body/delimiter/expansion/completeness facts, and add a
-      separately tested Bash `<<<` here-string redirect slice.
+- [ ] Build on the delivered bounded Bash heredoc grammar and quoted-delimiter
+      adjacency by exposing public body/delimiter/expansion/completeness facts,
+      then add a separately tested Bash `<<<` here-string redirect slice.
 - [ ] Near the end of v0.3 delivery, expand the Web sample with curated complex
       Bash and PowerShell inputs and deterministic Mermaid views of syntax,
       occurrences, compatibility clauses, ancestry, redirects, and fail-closed

SPEC.md

@@ -978,6 +978,17 @@ quoted_string   := single-quoted | double-quoted
 - v0.2 recognizes heredocs (`<<EOF ... EOF`) as redirect syntax while the
   body is skipped. Stable v0.3 preserves delimiter, body, expansion mode,
   tab-stripping mode, and completeness through `HereDocumentAnalysis`.
+  The bounded grammar accepts one terminal `<<` / `<<-` redirect on a command
+  header, with optional whitespace or a trailing comment after the delimiter.
+  Additional header tokens, pipelines, and queued heredocs are unparseable
+  until their body-association grammar is modeled. Quote removal determines
+  the delimiter spelling; any quoted or escaped delimiter fragment makes the
+  body literal. In an expanding body, unescaped `$()` substitutions are
+  executable even when their spelling is surrounded by quote characters,
+  because heredoc body quotes are data rather than shell quoting syntax.
+  Escaped substitutions remain literal. Legacy backticks, arithmetic
+  expansion, line continuations that could hide a substitution boundary, and
+  incomplete substitutions make the whole result unparseable.
 - Redirect targets matching the POSIX fd-dup / fd-close shorthand —
   `&N`, `&N-`, or `&-` (where `N` is one or more decimal digits) — are
   NOT path-resolved. The parser carries the raw token (e.g. `&1`) on
@@ -1068,7 +1079,8 @@ The lexer produces tokens consumed by the parser. Token kinds:
 - **OPERATOR** — `&&`, `||`, `;`, `|`, `>`, `>>`, `<`, `2>`, `2>>`,
   `(`, `)`, `<<`, `<<-`.
 - **WHITESPACE** — one or more spaces, tabs, or newlines (newlines inside
-  a skipped heredoc body are not tokenized). A whitespace run that
+  a heredoc body are not emitted as ordinary tokens; the delimiter token
+  retains the body's resolver fragments and authored extent). A whitespace run that
   contains a newline — including the newline after a heredoc terminator —
   is flagged as a **statement separator**; the parser retains those
   tokens past `FilterSignificant` and splits clauses on them per §4. A
@@ -1081,6 +1093,9 @@ The lexer produces tokens consumed by the parser. Token kinds:
   and `\X` escapes via a shared opaque-region scanner. The parser
   consumes this token as `Arg{ Kind=DynamicSkip, IsPath=false,
   Resolved=null }` per locked interpretation #2.
+  Expanding-heredoc substitutions use the same opaque fragment semantics but
+  remain attached to the delimiter token rather than entering the ordinary
+  command-token stream.
 - **UNPARSEABLE_SENTINEL** — `$((expr))` arithmetic expansion or
   `${var//pat/repl}` complex parameter expansion. The lexer skips past
   the matching close (`))` or `}` respectively) and emits a sentinel

TOOLING.md

@@ -40,12 +40,15 @@ and parser slice exists. The PII audit scans both corpus trees.
 ### PwshCorpusTool
 
 `tools/PwshCorpusTool` is the PowerShell corpus authoring aid
-(SPEC.POWERSHELL.md §13). It is a dev-only console app (not packed).
+(SPEC.POWERSHELL.md §13). Its shared serializer also exposes an ad-hoc Bash
+check so structural Bash entries do not need hand-authored spans. It is a
+dev-only console app (not packed).
 
 | Command | Purpose |
 |---|---|
 | `dotnet run --project tools/PwshCorpusTool -- generate` | Regenerate every `Corpus/powershell/NNN_slug.json` from the curated `CorpusManifest`. Run after any parser change that shifts PowerShell AST output. |
 | `dotnet run --project tools/PwshCorpusTool -- check "<command>"` | Print the parser's expected-AST JSON block for a command beside the real-`pwsh` oracle verdict — the fastest way to author or debug a binding-category entry. |
+| `dotnet run --project tools/PwshCorpusTool -- check-bash "<command>"` | Print exact Bash `clauses`, `syntax`, and `commands` expectations using the corpus resolver settings. |
 
 The curated inputs live in `tools/PwshCorpusTool/CorpusManifest.cs`; the
 `expected` AST is generated from `PwshParser`, and `PwshOracleTests`

openspec/changes/v0-3-structured-shell-analysis/tasks.md

@@ -33,14 +33,17 @@
 - [x] 3.9 Add corpus expectations for syntax shape, occurrences, roles, and completeness for existing constructs.
 - [ ] 3.10 Implement Bash `$()` discovery in supported argument words, redirect values, iterables, and expanding heredoc bodies; retain literal/escaped spellings and fail closed on command-name substitutions, legacy backticks, or incomplete interiors.
   - [x] 3.10a Implement the simple-command argument and redirect-target slice, including comment-safe boundaries and fail-closed unsupported interiors.
+  - [x] 3.10b Implement the bounded expanding-heredoc slice with quote-removed delimiters, literal quoted/escaped bodies, tab stripping, exact provenance, and fail-closed unsupported header/body forms.
 - [ ] 3.11 Implement PowerShell `$()` discovery in supported words, redirect values, foreach expressions, call-operator dynamic identities, standalone expression statements, double-quoted strings, and expandable here-strings; never invent invocation from standalone output, retain literal/escaped spellings, and fail closed on trailing command-style arguments, call-operator script blocks, or unsupported execution-bearing `@()` / `@{}` forms.
   - [x] 3.11a Implement words, redirect values, call-operator dynamic identities, standalone statements, expandable strings/here-strings, and parent-versus-child host payload provenance; fail closed on arbitrary expression values and unsupported execution-bearing `@()` / `@{}` forms.
 - [ ] 3.12 Pin substitution parentage, authored sibling indices, innermost-first ordering, Bash-isolated versus PowerShell-current-scope state, unknown-state propagation, nesting/depth limits, and incomplete dynamic identities in direct tests.
   - [x] 3.12a Pin the Bash argument/redirect slice, isolated cwd behavior, wrapper provenance, and the shared structural-depth budget.
   - [x] 3.12b Pin the PowerShell simple-command slice, current-scope exact and unknown cwd propagation, parent/child wrapper provenance, expression boundaries, and the shared structural-depth budget.
+  - [x] 3.12c Pin expanding-heredoc sibling/nested ordering, exact spans, isolated state, delimiter modes, escape parity, depth limits, and atomic failure.
 - [ ] 3.13 Promote ordinary, multiple, nested, iterator, redirect, quoted, escaped, stateful, malformed, and hidden-execution substitution cases into both executable corpora and the Netclaw approval matrix.
   - [x] 3.13a Promote the Bash ordinary, multiple, nested, redirect, quoted, escaped, stateful, malformed, and hidden-execution cases into its executable corpus.
   - [x] 3.13b Promote the PowerShell ordinary, multiple, nested, redirect, quoted, escaped, stateful, malformed, expression-boundary, and hidden-execution cases into its executable corpus.
+  - [x] 3.13c Promote expanding, literal, tab-stripped, multiple, and malformed Bash heredoc cases with full structural expectations into the executable corpus.
 
 ## 4. Explicit Redirect Semantics
 
@@ -106,10 +109,11 @@
 ## 10. Heredoc / Here-String Slice and Separately Gated Follow-ups
 
 - [x] 10.1 Specify heredoc delimiter adjacency and quoting, expansion mode, body provenance, substitutions, tab stripping, completeness, and Bash here-string semantics.
-- [ ] 10.2 Preserve existing `<<` / `<<-` behavior and fix quoted-delimiter adjacency without regressing the v0.2 compatibility redirect.
+- [x] 10.2 Preserve existing `<<` / `<<-` behavior and fix quoted-delimiter adjacency without regressing the v0.2 compatibility redirect.
 - [ ] 10.3 Add explicit heredoc delimiter/body/expansion/completeness facts and surface every supported substitution command.
 - [ ] 10.4 Add Bash `<<<` here-string tokenization, explicit redirect facts, bounded operand analysis, and trailing-newline semantics.
 - [ ] 10.5 Add direct, malformed, quoted/unquoted, tab-stripped, dynamic, and substitution-bearing corpus cases plus real-Bash parse-only validation.
+  - [x] 10.5a Add direct, executable-corpus, real-Bash output, and real-Bash parse-only coverage for the bounded substitution-discovery slice; explicit redirect facts and the full heredoc matrix remain pending.
 - [ ] 10.6 After stable v0.3, specify process-substitution command discovery and the unknown produced descriptor/path value before enabling it.
 - [ ] 10.7 After stable v0.3, specify background-list concurrency, ordering, and shell-state boundaries before enabling single `&`.
 - [ ] 10.8 Specify C-style loop and arithmetic hidden-execution behavior before enabling either construct.