Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions IMPLEMENTATION_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,21 @@ priorities.
transfers, and occurrence-specific redirect values remain fail closed.
Next add the Netclaw approval matrix before calling the Bash consumer
integration complete.
- [x] Deliver occurrence-level Bash explicit redirect facts for ordinary file
input/output/append, static descriptor duplicate/close/move, computed
descriptor targets, and combined `&>` / `&>>` output. Static descriptor
operations are complete non-path facts even though the v0.2 compatibility
redirect remains `IsDynamicSkip`; computed targets remain incomplete and
cannot be exempted by raw prefix. Arbitrary numeric source descriptors are
retained only when authored at a token boundary; overflow sources remain
incomplete rather than being truncated, and word suffixes such as
`command3>file` keep `3` in the command name. Unquoted LF/CRLF line
continuations are removed before descriptor recognition, including when
they join multi-digit sources. Exact file targets now complete their
containing occurrence, while cwd or value uncertainty still downgrades
the redirect and occurrence after abstract-state joins. Direct lexer and
parser tests plus executable corpus cases pin the boundary. Next map the
PowerShell stream model, then prove the paired Netclaw redirect matrix.
- [ ] Complete PowerShell `foreach` integration and add the
Netclaw approval-matrix cases. The structural slice now preserves literal
scalar/array and executable iterator forms, recursively parses bodies,
Expand Down
22 changes: 16 additions & 6 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -967,8 +967,9 @@ public sealed record Clause
public IReadOnlyList<Arg> Args { get; init; } = [];

/// <summary>
/// Redirect operators on this clause (>, >>, <, 2>, 2>>). Each entry
/// includes direction and target path.
/// Compatibility redirects on this clause (>, >>, <, 2>, 2>>, &>,
/// &>>). Each entry retains the v0.2 direction and target projection;
/// v0.3 consumers use CommandOccurrence.Redirects for exact semantics.
/// </summary>
public IReadOnlyList<Redirect> Redirects { get; init; } = [];

Expand Down Expand Up @@ -1270,7 +1271,8 @@ verb_like_word := static word satisfying §6.1; the initial command-name
arg := word | flag | quoted_string | supported_substitution
flag := "-" letter+ | "--" word
redirect := redirect_op target
redirect_op := ">" | ">>" | "<" | "2>" | "2>>"
redirect_op := descriptor? (">" | ">>" | "<") | "&>" | "&>>"
descriptor := digit+
target := word | quoted_string | supported_substitution
supported_substitution := "$(" command ")"
word := non-whitespace, non-operator fragments; may contain
Expand Down Expand Up @@ -1396,7 +1398,8 @@ The lexer produces tokens consumed by the parser. Token kinds:
- **QUOTED_STRING** — single- or double-quoted string. The lexer strips
the quote delimiters from the token value. Example: `"hello world"`
becomes the token value `hello world`.
- **OPERATOR** — `&&`, `||`, `;`, `|`, `>`, `>>`, `<`, `2>`, `2>>`,
- **OPERATOR** — `&&`, `||`, `;`, `|`, `>`, `>>`, `<`, numeric-descriptor
forms such as `2>`, `3>>`, and `10<`, `&>`, `&>>`,
`(`, `)`, `<<`, `<<-`.
- **WHITESPACE** — one or more spaces, tabs, or newlines (newlines inside
a heredoc body are not emitted as ordinary tokens; the delimiter token
Expand Down Expand Up @@ -1456,15 +1459,22 @@ The lexer produces tokens consumed by the parser. Token kinds:

Operators terminate the current token. `cd /tmp&&ls` lexes as
`[cd, /tmp, &&, ls]` — no whitespace required around operators. The lexer
must handle this.
must handle this. A numeric descriptor is an operator prefix only when its
digits begin at a shell-token boundary and become adjacent to `<`, `>`, or
`>>` after Bash removes unquoted line continuations. Continuations may join
digit fragments or the descriptor and operator; LF and CRLF spellings retain
their authored span while producing the same descriptor. Digits joined to an
ordinary, quoted, or escaped word remain part of that word; `command3>file`
therefore uses command name `command3` and a default-source `>` redirect.

### Comment handling

- An unquoted `#` that appears at a **word boundary** starts a comment
that runs to (but does not include) the next newline. A word boundary
is: start of input, or the position immediately after a whitespace
run, a newline, an operator (`&&`, `||`, `;`, `|`, `>`, `>>`, `<`,
`2>`, `2>>`, `(`, `)`, `<<`, `<<-`), a quoted string, or an opaque
a numeric descriptor adjacent to `>`, `>>`, or `<`, `&>`, `&>>`, `(`,
`)`, `<<`, `<<-`), a quoted string, or an opaque
substitution. Equivalently: `#` is comment-start everywhere the
outer lexer dispatch loop sits, because every other lexer rule has
already consumed its territory before `#` is considered.
Expand Down
15 changes: 12 additions & 3 deletions openspec/changes/v0-3-structured-shell-analysis/tasks.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,20 @@
## 4. Explicit Redirect Semantics

- [x] 4.1 Add the locked redirect operation and target-analysis types while retaining compatibility redirect members.
- [ ] 4.2 Classify Bash descriptor duplication, close, and move as static only for the complete literal descriptor grammar.
- [ ] 4.3 Keep variable-driven and otherwise computed Bash descriptor targets unknown or incomplete.
- [ ] 4.4 Lex and classify Bash `&>` and `&>>` independently from background-list operators.
- [x] 4.2 Classify Bash descriptor duplication, close, and move as static only for the complete literal descriptor grammar.
- [x] 4.3 Keep variable-driven and otherwise computed Bash descriptor targets unknown or incomplete.
- [x] 4.4 Lex and classify Bash `&>` and `&>>` independently from background-list operators.
- [ ] 4.5 Map existing PowerShell stream redirects into the shared explicit model without losing shell-specific stream identity.
- [ ] 4.6 Add paired direct tests and corpus cases for static, dynamic, malformed, multiple, combined, and file redirects.
- [x] 4.6a Add Bash direct and executable-corpus cases for static duplicate,
close, and move; computed descriptor targets; combined output overwrite and
append; ordinary file redirects; arbitrary numeric source descriptors; and
overflow/malformed fail-closed boundaries. LF/CRLF continuations at the
descriptor boundary and inside multi-digit sources follow Bash's pre-token
removal semantics. Executable-corpus cases pin malformed atomic failure and
independent occurrence facts for multiple redirects.
- [ ] 4.6b Add the paired PowerShell direct and executable-corpus cases when
task 4.5 maps its stream model.
- [ ] 4.7 Verify the explicit model removes the need for raw-prefix inference in a Netclaw integration test.

## 5. Consumer Migration Baseline
Expand Down
147 changes: 128 additions & 19 deletions src/ShellSyntaxTree/Internal/Bash/Lexing/BashLexer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,16 @@ internal static IReadOnlyList<BashToken> Tokenize(string input)
}

// ---- operators (longer-match first) ----
// Order matters: `&&` before `&`, `||` before `|`, `>>` before `>`,
// `2>>` before `2>`, `<<-` before `<<`, `<<` before `<`. We don't
// recognize a bare `&` in v0.1 (no background-job support; SPEC §1).
if (TryReadOperator(src, i, out var opLen, out var opText))
// Order matters: a token-boundary numeric descriptor precedes its
// redirect, `&&` precedes `&`, `||` precedes `|`, `>>` precedes
// `>`, and `<<-` precedes `<<` and `<`. Bare `&` background jobs
// remain unsupported.
if (TryReadOperator(
src,
i,
CanStartNumericDescriptor(tokens),
out var opLen,
out var opText))
{
var operatorTok = new BashToken(
BashTokenKind.Operator, "", opText, i, opLen, null);
Expand Down Expand Up @@ -268,7 +274,7 @@ private static bool TryConsumeFileDescriptorTarget(
}

if (previousIndex < 0 || tokens[previousIndex].Kind != BashTokenKind.Operator ||
tokens[previousIndex].OperatorText is not (">" or ">>" or "<" or "2>" or "2>>"))
!CanTakeDescriptorTarget(tokens[previousIndex].OperatorText))
{
return false;
}
Expand Down Expand Up @@ -323,16 +329,89 @@ tokens[previousIndex].OperatorText is not (">" or ">>" or "<" or "2>" or "2>>"))
return true;
}

private static bool CanTakeDescriptorTarget(string? operatorText)
{
if (string.IsNullOrEmpty(operatorText))
{
return false;
}

var operatorStart = 0;
while (operatorStart < operatorText!.Length &&
operatorText[operatorStart] is >= '0' and <= '9')
{
operatorStart++;
}

var redirect = operatorText.Substring(operatorStart);
return redirect is ">" or ">>" or "<";
}

// ---------------------------------------------------------------- operators

private static bool TryReadOperator(
ReadOnlySpan<char> src, int i, out int length, out string? text)
ReadOnlySpan<char> src,
int i,
bool canStartNumericDescriptor,
out int length,
out string? text)
{
var descriptor = new StringBuilder();
var descriptorEnd = i;
while (descriptorEnd < src.Length)
{
if (src[descriptorEnd] is >= '0' and <= '9')
{
descriptor.Append(src[descriptorEnd]);
descriptorEnd++;
continue;
}

if (TrySkipLineContinuation(src, descriptorEnd, out var afterContinuation))
{
descriptorEnd = afterContinuation;
continue;
}

break;
}

if (canStartNumericDescriptor &&
descriptor.Length > 0 &&
descriptorEnd < src.Length)
{
if (src[descriptorEnd] == '>')
{
var append = descriptorEnd + 1 < src.Length &&
src[descriptorEnd + 1] == '>';
length = descriptorEnd - i + (append ? 2 : 1);
text = descriptor.ToString() + (append ? ">>" : ">");
return true;
}

if (src[descriptorEnd] == '<')
{
length = descriptorEnd - i + 1;
text = descriptor.ToString() + "<";
return true;
}
}

// Multi-char operators first.
if (i + 1 < src.Length)
{
var c0 = src[i];
var c1 = src[i + 1];
if (c0 == '&' && c1 == '>')
{
if (i + 2 < src.Length && src[i + 2] == '>')
{
length = 3; text = "&>>"; return true;
}

length = 2; text = "&>"; return true;
}

if (c0 == '&' && c1 == '&') { length = 2; text = "&&"; return true; }
if (c0 == '|' && c1 == '|') { length = 2; text = "||"; return true; }
if (c0 == '>' && c1 == '>') { length = 2; text = ">>"; return true; }
Expand All @@ -346,15 +425,6 @@ private static bool TryReadOperator(
length = 2; text = "<<"; return true;
}

if (c0 == '2' && c1 == '>')
{
if (i + 2 < src.Length && src[i + 2] == '>')
{
length = 3; text = "2>>"; return true;
}

length = 2; text = "2>"; return true;
}
}

// Single-char operators.
Expand All @@ -371,6 +441,49 @@ private static bool TryReadOperator(
}
}

private static bool TrySkipLineContinuation(
ReadOnlySpan<char> source,
int index,
out int afterContinuation)
{
afterContinuation = index;
if (index + 1 >= source.Length || source[index] != '\\' ||
source[index + 1] is not ('\n' or '\r'))
{
return false;
}

afterContinuation = source[index + 1] == '\r' &&
index + 2 < source.Length && source[index + 2] == '\n'
? index + 3
: index + 2;
return true;
}

private static bool CanStartNumericDescriptor(IReadOnlyList<BashToken> tokens)
{
if (tokens.Count == 0)
{
return true;
}

var index = tokens.Count - 1;
while (index >= 0 && tokens[index].Kind == BashTokenKind.Continuation)
{
index--;
}

if (index < 0)
{
return true;
}

return tokens[index].Kind is
BashTokenKind.Whitespace or
BashTokenKind.Operator or
BashTokenKind.Comment;
}

// ---------------------------------------------------------------- quoted

private static int ReadSingleQuoted(
Expand Down Expand Up @@ -1219,10 +1332,6 @@ private static bool IsOperatorStart(ReadOnlySpan<char> src, int i)
// Both `&&` and unsupported bare `&` terminate a word. The
// tokenizer emits a sentinel for the latter on its next pass.
return true;
case '2':
// `2>` and `2>>` start with '2' — only treat them as operator
// starts when the immediate next char is '>'.
return i + 1 < src.Length && src[i + 1] == '>';
default:
return false;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1207,14 +1207,15 @@ private SimpleCommandSyntax RewriteSimple(
}

var clause = RewriteClause(simple.Clause, input, sourceFacts.CwdPathDependencies);
var redirects = RewriteRedirectFacts(sourceFacts.Redirects, clause);
facts.Add(clause, new CommandOccurrenceFacts
{
EffectiveArguments = CreateEffectiveArguments(simple.Clause),
WorkingDirectory = input.ToDomain(),
Redirects = RewriteRedirectFacts(sourceFacts.Redirects, clause),
Redirects = redirects,
CwdPathDependencies = sourceFacts.CwdPathDependencies,
ValueProvenance = sourceFacts.ValueProvenance,
IsComplete = sourceFacts.IsComplete,
IsComplete = sourceFacts.IsComplete && AreRedirectsComplete(redirects),
});
return simple with
{
Expand Down Expand Up @@ -1463,12 +1464,26 @@ private static IReadOnlyList<RedirectAnalysis> RewriteRedirectFacts(
Kind = ShellValueDomainKind.Exact,
Values = new[] { redirect.Target },
},
IsComplete = fact.IsComplete && !redirect.IsDynamicSkip,
};
}

return rewritten;
}

private static bool AreRedirectsComplete(IReadOnlyList<RedirectAnalysis> redirects)
{
foreach (var redirect in redirects)
{
if (!redirect.IsComplete)
{
return false;
}
}

return true;
}

private string? RebaseResolution(
string? resolved,
bool isPath,
Expand Down
Loading