Repository navigation
tests: read the whole pipe instead of grep -q under pipefail - #575
Conversation
AGENTS.md's Userspace gotcha (and review lesson L-15): grep -q exits at its first match, the command still writing into the pipe gets SIGPIPE, and pipefail fails the pipeline. Issue derekross#457 is this race in the modem NV-overlay test, reproduced every time with PYTHONUNBUFFERED=1. The same pattern sits in other test scripts that set pipefail. A "|| fail" check then fails at random, and a "&& fail" or "! ... || fail" check passes when it should fail. Replace each grep -q with grep -c ... >/dev/null, which reads all of its input; the patterns, -x, -E, -v, negations and messages stay as they were, so no check asserts anything new. packages/omamobile-modem/test/nv-overlay is left alone: the maintainer offered its fix to the reporter of derekross#457. Two kinds of line can't race and change only so that no test pipes into grep -q: the audio filter check reads one line from grep -c, and the three omamobile-remote checks run in a child bash without pipefail. Tested: each racing pattern run 20 times with a writer that keeps writing after the match (old: a false fail or a false pass 20 of 20; new: the right result 20 of 20); tests/packages for every touched package, where omamobile-audio stops at two host-only checks that fail the same way on main; and tests/all, where only release-upload fails, on a checkout path containing "/.". Test-only, so no package versions change (derekross#457). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A small guard so the previous commit's fix holds: it git-greps the test scripts that set pipefail (tests/ and packages/*/test) for a pipe into grep -q, --quiet or --silent, and fails with the lines it found. It first checks its own pattern against lines it must and must not match (a "||" before grep, a here-string, grep -c). nv-overlay is left out until derekross#457's fix lands. Tested: passes on this branch (46 scripts); on origin/main's tree it fails and lists the lines the previous commit changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you, Victor. This is a tidy one: every conversion keeps its pattern, I'll merge it as it is. |
|
Note 🤖 Opus 5.5 replying on behalf of Victor. The shipped-tools side is in #587: |
* bin/omamobile, tools, rootfs: no pipe into grep -q under pipefail The follow-up to #575, which fixed the test scripts: the same pattern in the laptop-side scripts that set pipefail. grep -q exits at the first match, a writer still writing gets SIGPIPE, and pipefail fails the pipeline (AGENTS.md section 4, Userspace). Each site now uses grep -c ... >/dev/null, which reads all its input, with the same pattern, flags and truth value. The one that loses in practice is tools/fingerprint-capture's wait loop: sed prints the whole log from the step's start, and with the end line early in a verbose kmsg stream the old check never saw it and the step ran to its timeout. The others have short writers (adb get-state, android-info.txt, 64 bytes of the TA, the keyring, the image's passwd) that fit the pipe buffer before grep exits; they change so that no script keeps the pattern. Two of them failed open: build-inside's user-account check and signing.sh's revoked-key check would have passed on a SIGPIPE. tests/grep-q-pipes now scans bin/, tools/ and rootfs/ as well, plus tools/lib (sourced by pipefail scripts), and skips comment lines. It leaves out five stage tools whose remaining hits are command strings that tools/phone-ssh runs in the phone's shell, which has no pipefail. Nothing under packages/ or initramfs/ changes. Tested: the race harness (old 0/200, new 200/200 per form; the fingerprint-capture loop old 0/100, new 100/100), tests/grep-q-pipes with an injected pipe (fails, names the line) and a commented one (passes), tools/pr-check, tests/all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests/grep-q-pipes: scan the package tests too, and the two pipes they held The scan's 'packages/*/test' pathspec matched no file, so no package test was ever scanned (91 scripts). With 'packages/*/test/*' it reads 164, and found two pipes into grep -q that landed since: device-husky's speaker_protection check (#453) and omamobile-waydroid's render-node check (#541). Both now read the whole pipe. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Derek Ross <derekross@gmail.com>
…ss#575) * tests: read the whole pipe instead of grep -q under pipefail AGENTS.md's Userspace gotcha (and review lesson L-15): grep -q exits at its first match, the command still writing into the pipe gets SIGPIPE, and pipefail fails the pipeline. Issue derekross#457 is this race in the modem NV-overlay test, reproduced every time with PYTHONUNBUFFERED=1. The same pattern sits in other test scripts that set pipefail. A "|| fail" check then fails at random, and a "&& fail" or "! ... || fail" check passes when it should fail. Replace each grep -q with grep -c ... >/dev/null, which reads all of its input; the patterns, -x, -E, -v, negations and messages stay as they were, so no check asserts anything new. packages/omamobile-modem/test/nv-overlay is left alone: the maintainer offered its fix to the reporter of derekross#457. Two kinds of line can't race and change only so that no test pipes into grep -q: the audio filter check reads one line from grep -c, and the three omamobile-remote checks run in a child bash without pipefail. Tested: each racing pattern run 20 times with a writer that keeps writing after the match (old: a false fail or a false pass 20 of 20; new: the right result 20 of 20); tests/packages for every touched package, where omamobile-audio stops at two host-only checks that fail the same way on main; and tests/all, where only release-upload fails, on a checkout path containing "/.". Test-only, so no package versions change (derekross#457). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests/grep-q-pipes: no test script under pipefail pipes into grep -q A small guard so the previous commit's fix holds: it git-greps the test scripts that set pipefail (tests/ and packages/*/test) for a pipe into grep -q, --quiet or --silent, and fails with the lines it found. It first checks its own pattern against lines it must and must not match (a "||" before grep, a here-string, grep -c). nv-overlay is left out until derekross#457's fix lands. Tested: passes on this branch (46 scripts); on origin/main's tree it fails and lists the lines the previous commit changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…rekross#587) * bin/omamobile, tools, rootfs: no pipe into grep -q under pipefail The follow-up to derekross#575, which fixed the test scripts: the same pattern in the laptop-side scripts that set pipefail. grep -q exits at the first match, a writer still writing gets SIGPIPE, and pipefail fails the pipeline (AGENTS.md section 4, Userspace). Each site now uses grep -c ... >/dev/null, which reads all its input, with the same pattern, flags and truth value. The one that loses in practice is tools/fingerprint-capture's wait loop: sed prints the whole log from the step's start, and with the end line early in a verbose kmsg stream the old check never saw it and the step ran to its timeout. The others have short writers (adb get-state, android-info.txt, 64 bytes of the TA, the keyring, the image's passwd) that fit the pipe buffer before grep exits; they change so that no script keeps the pattern. Two of them failed open: build-inside's user-account check and signing.sh's revoked-key check would have passed on a SIGPIPE. tests/grep-q-pipes now scans bin/, tools/ and rootfs/ as well, plus tools/lib (sourced by pipefail scripts), and skips comment lines. It leaves out five stage tools whose remaining hits are command strings that tools/phone-ssh runs in the phone's shell, which has no pipefail. Nothing under packages/ or initramfs/ changes. Tested: the race harness (old 0/200, new 200/200 per form; the fingerprint-capture loop old 0/100, new 100/100), tests/grep-q-pipes with an injected pipe (fails, names the line) and a commented one (passes), tools/pr-check, tests/all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests/grep-q-pipes: scan the package tests too, and the two pipes they held The scan's 'packages/*/test' pathspec matched no file, so no package test was ever scanned (91 scripts). With 'packages/*/test/*' it reads 164, and found two pipes into grep -q that landed since: device-husky's speaker_protection check (derekross#453) and omamobile-waydroid's render-node check (derekross#541). Both now read the whole pipe. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Derek Ross <derekross@gmail.com>
What and why
AGENTS.md (Userspace) and review lesson L-15:
grep -qexits at its first match, the command still writing into the pipe gets SIGPIPE, andpipefailfails the pipeline. #457 is this race in the modem NV-overlay test (reproduced 10 of 10 withPYTHONUNBUFFERED=1, L-511). The same pattern was in other test scripts that set pipefail: a|| failcheck fails at random, and a&& failor! … || failcheck can pass when it should fail.The first commit replaces each
… | grep -qwithgrep -c … >/dev/null, which reads all of its input, in 16 test files (the device packages'bootctl dumpand[Network]Gateway=checks, audio, fingerprint, gnss, mirror, modem, remote,tests/initramfs-reproducible,tests/install-phone,tests/rootfs-smoke). Patterns,-x/-E/-v, negations and messages are unchanged, so no check asserts anything new. The audio filter check (one line fromgrep -c) and the three omamobile-remote checks (a childbash -cwithout pipefail) can't race; they change only so that no test pipes intogrep -q.Not changed:
packages/omamobile-modem/test/nv-overlaylines 104, 194 and 254. On #457 the maintainer offered that fix to the reporter (or to make it), so this leaves it to them.The second commit adds
tests/grep-q-pipestotests/all: it git-greps the pipefail test scripts for a pipe intogrep -q(--quiet,--silent) and fails with the lines it finds, leaving nv-overlay out until #457's fix lands. Open PRs that add the pattern would need the same one-word change after a rebase: #539, #510, #483, #541, #453, #436, #429, #346, #25. It can be dropped without touching the first commit. Per L-549, thetests/allloop line conflicts with other open PRs that edit it (#565, #562, #559, #543): the resolution keeps both entries, withtests/grep-q-pipesaftertests/usb-product.Test-only, so no package versions change (#457).
Old form against new, each run 20 times under
set -euo pipefailwith a writer that keeps writing after the match:P | grep -q X || failexists, install-phone:139! P | grep -q X || failGateway=presentP | grep -q X && failDevice and build
uname -r): n/aChecks (no phone needed)
tools/pr-check origin/mainpassestests/allpasses (479 ok, 0 not ok)tests/packagespasses for the packages I touched (audio, device-akita, -cheetah, -felix, -husky, -lynx, -panther, -raven, fingerprint, gnss, mirror, modem, remote), apart from two omamobile-audio checks that fail the same way on main on this build machine (the alsaucm HiFi device count at line 42, and the/dev/snd/controlC0check at line 1013, which use the host's own sound card); the speaker-filter section with line 1030 passesCo-Authored-By:orAssisted-by:trailer, never aSigned-off-by:On the phone
tests/rootfs-smokeagainst a real root image (none built here); its lines were checked withbash -nand the stubbed runs aboveuserdata, weakens the storage guard, or reads registers from userspace (AGENTS.md §1)tools/phone-smokereportNot applicable: a test-only change (CONTRIBUTING.md §6.5).
Generated with Claude Opus 5.5 in T3 Code Nightly