From fc532b4a886dbb10ff77c68cd859f7d276eaf526 Mon Sep 17 00:00:00 2001 From: Aaron Stannard Date: Sun, 9 Aug 2026 00:40:36 +0000 Subject: [PATCH] Implement bounded Bash heredoc analysis --- IMPLEMENTATION_PLAN.md | 18 +- SPEC.md | 56 +++-- .../v0-3-structured-shell-analysis/design.md | 24 ++ .../proposal.md | 5 + .../specs/bounded-shell-analysis/spec.md | 56 +++++ .../specs/explicit-redirect-semantics/spec.md | 5 + .../specs/structured-shell-syntax/spec.md | 5 + .../v0-3-structured-shell-analysis/tasks.md | 13 +- .../Internal/Bash/Lexing/BashLexer.cs | 146 ++++++++---- .../Internal/Bash/Lexing/BashToken.cs | 19 ++ .../Bash/Parsing/BashCommandParser.cs | 2 +- .../Bash/Parsing/BashCwdInvocationGrammar.cs | 146 ++++++++++-- .../Bash/Parsing/BashRedirectAnalysis.cs | 149 ++++++++++-- .../Bash/Parsing/BashStructuralCoordinator.cs | 108 ++++++++- .../ShellSyntaxTree.Tests/Corpus/AstAssert.cs | 55 +++++ .../Corpus/CorpusRunnerTests.cs | 40 +++- ...00_v03_expanding_heredoc_substitution.json | 26 ++- ...literal_heredoc_substitution_spelling.json | 26 ++- ...03_tab_stripping_heredoc_substitution.json | 26 ++- ...03_v03_heredoc_multiple_substitutions.json | 26 ++- ...v03_for_wrapper_export_state_rejected.json | 5 +- ...03_for_post_binding_mutation_rejected.json | 4 +- .../269_v03_nameref_heredoc_expansion.json | 10 + .../270_v03_wrapped_execution_builtin.json | 10 + ...271_v03_unknown_named_parameter_state.json | 11 + .../bash/272_v03_attached_printf_v.json | 10 + .../Lexing/BashLexerTests.cs | 31 +++ .../Parsing/BashExecutionBoundaryTests.cs | 162 +++++++++++++ .../Parsing/BashForInStructuralTests.cs | 23 +- .../Parsing/BashStructuralProjectionTests.cs | 215 +++++++++++++++++- .../Parsing/ClauseElementTests.cs | 1 + .../Parsing/ResolverProvenanceTests.cs | 2 + .../Parsing/ShellValueOracleTests.cs | 96 ++++++++ 33 files changed, 1396 insertions(+), 135 deletions(-) create mode 100644 tests/ShellSyntaxTree.Tests/Corpus/bash/269_v03_nameref_heredoc_expansion.json create mode 100644 tests/ShellSyntaxTree.Tests/Corpus/bash/270_v03_wrapped_execution_builtin.json create mode 100644 tests/ShellSyntaxTree.Tests/Corpus/bash/271_v03_unknown_named_parameter_state.json create mode 100644 tests/ShellSyntaxTree.Tests/Corpus/bash/272_v03_attached_printf_v.json create mode 100644 tests/ShellSyntaxTree.Tests/Parsing/BashExecutionBoundaryTests.cs diff --git a/IMPLEMENTATION_PLAN.md b/IMPLEMENTATION_PLAN.md index 45849f3..edc3e43 100644 --- a/IMPLEMENTATION_PLAN.md +++ b/IMPLEMENTATION_PLAN.md @@ -530,9 +530,21 @@ priorities. plus equal-record hash consistency, demonstrate that default JSON is not a polymorphic round-trip contract, and make every policy-sensitive unknown numeric enum value detectable so consumers can reject it. -- [ ] Build on the delivered bounded Bash heredoc grammar and quoted-delimiter - adjacency by exposing public body/delimiter/expansion/completeness facts, - then add a separately tested Bash `<<<` here-string redirect slice. +- [x] Expose public Bash heredoc body, delimiter, expansion, tab-stripping, and + completeness facts from the delivered bounded grammar. Direct tests pin + literal and expanding delimiters, every supported substitution command, + exact empty/LF/CRLF/tabbed body provenance, the retained v0.2 redirect, + and nullable source offsets for decoded `bash -c` payloads. +- [x] Close the Bash variable-attribute hidden-execution boundary. Named + parameter dereferences now require a caller-proved isolated initial + state, with that proof propagated into substitutions and subshells. + Direct and recursively dispatch-wrapped `eval`, source/dot, trap, + variable-attribute and variable-mutating builtins, plus `printf -v`, + fail atomically until their state effects are modeled. Native Bash + oracles pin nameref-deferred and integer-assignment execution, and the + executable corpus carries isolated, unknown-state, and wrapper cases. +- [ ] Add the separately tested Bash `<<<` here-string redirect slice with + bounded operand analysis and trailing-newline semantics. --- diff --git a/SPEC.md b/SPEC.md index 12fbcfe..dc85b9a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -280,6 +280,16 @@ mode for every later region that can observe it. In particular, a decoded the option is not blindly copied into the child. Cwd-only state changes retain the caller's initial-variable assertion. +The same attribute-state proof governs every simple named-parameter +dereference. With `BashInitialStateMode.Unknown`, `$name` and `${name}` are +unparseable because an ambient nameref can evaluate an arithmetic array +subscript and execute authored command text. In isolated mode, a fresh-process +variable before reachable source mutation may remain an unknown value while +still being proved free of recursive variable attributes. A modeled ordinary +loop binding retains its explicit proof. Positional and special parameters +that cannot carry variable attributes keep their existing typed cardinality +rules. + Even in isolated mode, the v0.3 bounded loop grammar accepts only ordinary lowercase scalar binding names matching `[a-z][a-z0-9_]*`, excluding `auto_resume` and `histchars`. `_`, uppercase names, and every name outside @@ -791,12 +801,17 @@ variable values to `Unknown` rather than selecting one path. `cd` and `chdir` use the effective argument vector for the current visit. `pushd` and `popd` may be recognized only with unknown success cwd until the -directory stack is modeled. Variable mutators (`read`, `unset`, `printf -v`, -`export`, `declare`, and equivalents), `eval`, `source` / `.`, and -execution-bearing `trap` make the complete loop region unparseable. The same is -true for `break`, `continue`, `return`, `exit`, and `exec` until their transfers -are implemented. Recognition recursively unwraps statically proved `command` -and `builtin` dispatch; a wrapper must not bypass the rejection. +directory stack is modeled. Unmodeled execution-bearing or +attribute-mutating builtins fail the complete parse closed globally, not only +inside loops. The stable catalog is `eval`, `source` / `.`, `trap`, `let`, +`declare`, `typeset`, `local`, `readonly`, `export`, `unset`, `read`, +`readarray`, `mapfile`, `getopts`, and `set`, plus `printf -v`. These forms can +evaluate argument text, install deferred execution, or assign through +unproved integer, nameref, or array attributes. Recognition recursively +unwraps statically proved `command` and `builtin` dispatch; dynamic or invalid +wrapper grammar fails closed. Ordinary `printf` without `-v` remains +supported. `break`, `continue`, `return`, `exit`, and `exec` remain loop-region +failures until their transfers are implemented. Substitutions and subshells inherit the current variable/cwd state but discard their state changes on exit. Decoded Bash command wrappers inherit invocation @@ -1305,17 +1320,20 @@ quoted_string := single-quoted | double-quoted - v0.2 recognizes heredocs (`<`, `>>`, `<`, numeric-descriptor - forms such as `2>`, `3>>`, and `10<`, `&>`, `&>>`, + forms such as `2>`, `3>>`, `10<`, `3<<`, and `4<<-`, `&>`, `&>>`, `(`, `)`, `<<`, `<<-`. - **WHITESPACE** — one or more spaces, tabs, or newlines (newlines inside a heredoc body are not emitted as ordinary tokens; the delimiter token @@ -1419,9 +1440,9 @@ The lexer produces tokens consumed by the parser. Token kinds: Expanding-heredoc substitutions use the same opaque fragment semantics but remain attached to the delimiter token rather than entering the ordinary command-token stream. -- **UNPARSEABLE_SENTINEL** — `$((expr))` arithmetic expansion or any - operator-bearing parameter expansion such as `${var:-$(cmd)}` or - `${var//pat/repl}`. The lexer skips past +- **UNPARSEABLE_SENTINEL** — `$((expr))` or obsolete `$[expr]` arithmetic + expansion, or any operator-bearing parameter expansion such as + `${var:-$(cmd)}`, `${var//pat/repl}`, or `${var@P}`. The lexer skips past the matching close (`))` or `}` respectively) and emits a sentinel whose reason names the rejected construct. The parser consumes this token by setting outer `ParsedCommand.IsUnparseable = true` (see §11). @@ -2307,6 +2328,11 @@ Each file: } ``` +An entry may set `bashInitialStateMode` to `Unknown` or +`IsolatedNonInteractive` when the expected result depends on the caller-proved +Bash variable-state contract. When omitted, the Bash corpus runner uses +`IsolatedNonInteractive`. The field is rejected outside the Bash corpus. + An entry may add an `elements` list to a clause to pin the complete `Clause.Elements` projection (`raw`, `value`, `role`, `sourceStart`, `sourceLength`, `precedingVerbElementCount`, `kind`, `isFlag`, `isPath`, and diff --git a/openspec/changes/v0-3-structured-shell-analysis/design.md b/openspec/changes/v0-3-structured-shell-analysis/design.md index eb7c882..c7f9c07 100644 --- a/openspec/changes/v0-3-structured-shell-analysis/design.md +++ b/openspec/changes/v0-3-structured-shell-analysis/design.md @@ -543,6 +543,30 @@ later scope that can observe it. A decoded `bash -c` after `export` therefore enters with `Unknown` initial variable state, while a cwd-only transfer retains the caller's variable-state assertion. +This state proof applies to every simple Bash parameter dereference, not only +to loop assignment. A syntactically simple `$name` or `${name}` can execute +when `name` is a nameref whose target contains an arithmetic array subscript. +For example, a prior `declare -n x='a[$(hidden)0]'` makes `${x}` evaluate the +subscript and execute `hidden`, even though the declaration operand was quoted +data. `Unknown` initial state therefore cannot classify a variable +dereference as fully accounted for. `IsolatedNonInteractive` may do so only +before any reachable unmodeled variable mutation, or for a binding whose +ordinary-scalar attributes are modeled explicitly. The produced value may +still be `Unknown`; variable-attribute safety and value exactness remain +independent facts. + +Stable v0.3 does not add a partial expression evaluator for Bash builtins. +Direct or statically wrapped `eval`, `source` / `.`, `trap`, `let`, `declare`, +`typeset`, `local`, `readonly`, `export`, `unset`, `read`, `readarray`, +`mapfile`, `getopts`, and `set` forms fail the complete parse closed, as does +`printf -v`. These commands can execute argument text directly, evaluate +arithmetic, install deferred execution, or assign through attributes whose +state is not proved. Exact `command` and `builtin` dispatch wrappers are +recursively unwrapped; dynamic or invalid wrapper grammar fails closed rather +than bypassing the catalog. Ordinary `printf` without `-v` remains supported. +A later additive grammar may admit individually proved query or assignment +forms without weakening this stable boundary. + PowerShell needs the same explicit boundary for different reasons. `PwshParserOptions.InitialStateMode` defaults to `Unknown`, which permits structural discovery but withholds exact or finite `foreach` binding proofs. diff --git a/openspec/changes/v0-3-structured-shell-analysis/proposal.md b/openspec/changes/v0-3-structured-shell-analysis/proposal.md index cc6d5b6..111a6f5 100644 --- a/openspec/changes/v0-3-structured-shell-analysis/proposal.md +++ b/openspec/changes/v0-3-structured-shell-analysis/proposal.md @@ -29,6 +29,11 @@ fail-closed behavior for incomplete analysis. - Add conservative value and shell-state analysis that distinguishes exact, finite, bounded-symbolic, and unknown facts without executing commands or enumerating the filesystem. +- Require proved Bash variable-attribute state before treating a simple + parameter dereference as non-executable. Stable v0.3 fails closed on + unmodeled shell builtins that can evaluate, assign through, or defer + argument text, because nameref and arithmetic attributes can otherwise turn + quoted data into hidden execution. - Preserve resolver-relevant lexical fragments, typed expansion identity and cardinality, operation-specific transform eligibility, opaque cause, and consumer/binding context through decoding so escaped or diff --git a/openspec/changes/v0-3-structured-shell-analysis/specs/bounded-shell-analysis/spec.md b/openspec/changes/v0-3-structured-shell-analysis/specs/bounded-shell-analysis/spec.md index bd4f058..e765602 100644 --- a/openspec/changes/v0-3-structured-shell-analysis/specs/bounded-shell-analysis/spec.md +++ b/openspec/changes/v0-3-structured-shell-analysis/specs/bounded-shell-analysis/spec.md @@ -168,6 +168,62 @@ the independently proved variable-state mode. - **THEN** the decoded child enters with unknown initial variable state - **THEN** the complete result is unparseable rather than publishing an isolated scalar proof +### Requirement: Bash parameter dereferences require proved attribute state +A syntactically simple Bash `$name` or `${name}` dereference SHALL be treated +as fully execution-accounted only when the incoming variable-attribute state +proves that the binding cannot be an integer, nameref, array reference, or +other recursively evaluated form. Value exactness SHALL remain independent: +an attribute-safe variable MAY still produce an `Unknown` value. + +`BashInitialStateMode.Unknown` SHALL make a simple named-variable dereference +unparseable because an ambient nameref can evaluate an authored array +subscript. `IsolatedNonInteractive` MAY establish safe initial attributes for +a fresh-process variable before reachable source mutation. A modeled ordinary +loop binding MAY retain that proof. Any reachable unmodeled variable mutation +SHALL invalidate the proof in later regions that can observe it. Positional +and special parameters that cannot carry variable attributes remain governed +by their existing typed cardinality rules. + +#### Scenario: Unknown ambient nameref fails closed +- **WHEN** default-mode Bash parses `printf '%s' "$x"` +- **THEN** the whole result is unparseable because ambient `x` may be a nameref +- **THEN** the parser does not report the outer `printf` complete while hiding recursive execution + +#### Scenario: Fresh isolated scalar may remain unknown data +- **WHEN** isolated-mode Bash parses `printf '%s' "$x"` before any source mutation +- **THEN** the variable value remains unknown +- **THEN** the dereference does not by itself make the result unparseable because fresh-process attributes are proved safe + +#### Scenario: Nameref mutation hides execution in a later heredoc +- **WHEN** isolated-mode Bash parses `declare -a a; declare -n x='a[$(hidden)0]'; cat <