Skip to content

fix(scripts): stop treating command failure as a pattern match - #160

Merged
mkoura merged 1 commit into
masterfrom
fix/pipefail_sigpipe_grep
Aug 20, 2026
Merged

fix(scripts): stop treating command failure as a pattern match#160
mkoura merged 1 commit into
masterfrom
fix/pipefail_sigpipe_grep

Conversation

@mkoura

@mkoura mkoura commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Three related ways a shell test reported the wrong answer, all by letting a producer's exit status stand in for -- or be mistaken for -- the result of the match.

grep -q exits on first match and closes the pipe while the producer may still have a pending write. The producer dies on SIGPIPE with 141, and pipefail promotes that to the pipeline's exit status, so the test reads as "no match" even when the pattern is present. It is a race on syscall interleaving, not a size threshold, so it fires intermittently and the odds worsen as the scanned output grows. The three _wait_for_tx_*_tx helpers hung a full 7200s timeout each, then failed cluster start, because a healthy tx tool fills the last 100 log lines with the very pattern being matched -- so the match lands on line 1, leaving tail with the most left to write. The healthier the tool, the likelier the false negative. Keep the producers out of the pipelines.

check_spend_success folded the query and the match into one condition, so a failing query utxo produced no output, no lovelace match, and the negation returned 0 -- inputs reported as spent, retry loop skipped. Reachable both through the SIGPIPE above and through any ordinary query error. Capture the query and test its status separately.

grep -c exits 1 when it counts zero, so the CC registration check aborted at the ERR trap instead of reaching its own error message.

Three related ways a shell test reported the wrong answer, all by
letting a producer's exit status stand in for -- or be mistaken for --
the result of the match.

`grep -q` exits on first match and closes the pipe while the producer
may still have a pending write. The producer dies on SIGPIPE with 141,
and `pipefail` promotes that to the pipeline's exit status, so the test
reads as "no match" even when the pattern is present. It is a race on
syscall interleaving, not a size threshold, so it fires intermittently
and the odds worsen as the scanned output grows. The three
`_wait_for_tx_*_tx` helpers hung a full 7200s timeout each, then failed
cluster start, because a healthy tx tool fills the last 100 log lines
with the very pattern being matched -- so the match lands on line 1,
leaving `tail` with the most left to write. The healthier the tool, the
likelier the false negative. Keep the producers out of the pipelines.

`check_spend_success` folded the query and the match into one condition,
so a failing `query utxo` produced no output, no `lovelace` match, and
the negation returned 0 -- inputs reported as spent, retry loop skipped.
Reachable both through the SIGPIPE above and through any ordinary query
error. Capture the query and test its status separately.

`grep -c` exits 1 when it counts zero, so the CC registration check
aborted at the ERR trap instead of reaching its own error message.
@mkoura
mkoura merged commit e0de84d into master Aug 20, 2026
4 checks passed
@mkoura
mkoura deleted the fix/pipefail_sigpipe_grep branch August 20, 2026 20:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant