Repository navigation
fix(validate): refuse a recipe that silences its own %check - #197
Conversation
Issue projectbluefin#132 has one obvious wrong answer. Fedora's audio recipes end %check with %{!?tests_nonfatal:exit $TESTS_ERROR} so a single `%global tests_nonfatal 1` above it turns pipewire's hanging pw-test-endpoint into a green build, while the log still says "test failed" and nothing downstream notices. For pipewire that is not a cosmetic shortcut. Hummingbird's repository ships no pipewire packages at all, so this factory is Utah's only source of the audio stack, and pipewire-libs-extra -- which config/bluefin-packages.toml installs -- carries Requires: pipewire >= %{version} so the recipe cannot be dropped either. The one-line fix would have published a desktop image with a known hang in its audio stack. tools/check_suppressed_tests.py rejects any definition of tests_nonfatal under packages/, and tools/validate.py runs it, so the existing validate job gates it with no new workflow. pulseaudio is recorded as the single inherited case: Fedora's own arch- and release-gated FIXMEs, imported verbatim rather than added here. The allowlist is checked in both directions -- an entry whose recipe stops defining the macro fails the gate, and one whose recipe is gone fails the unit test -- so the exception cannot outlive its reason. An absent recipe is deliberately not an error: validate.py runs against synthetic trees in tests/test_validate.py, and a gate that depended on which tree it was handed would be the wrong kind of gate. The build-failure-triage skill takes the three findings that cost the measurements: * SIGALRM with no output at a constant wall-clock second is a hang, not starvation. Serializing %check was measured at the same 5.00s failure while pw-test-filter was handed 18s and passed, so neither serializing nor raising the alarm is a fix. * `changed` is tested before `published` in the prepare matrix, so restoring a recipe byte-for-byte does not restore its skip. Only leaving it alone does. * Why tests_nonfatal is now refused, and what the one exception is. Assisted-by: Claude Opus 5 via hive Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
There was a problem hiding this comment.
correctness
INHERITEDexempts the whole package, not the two definitions it documents:offenders()doesif package in INHERITED: continuebefore scanning (tools/check_suppressed_tests.py:311-313), andstale()only requires that at least one definition remain. So a future Fedora import bump of pulseaudio could add a new, unconditional%global tests_nonfatal 1— beyond the two arch/release-gated FIXMEs at packages/pulseaudio/pulseaudio.spec:384,388 — without the gate noticing. The docstring says the entry is "not permission to add more," but mechanically, within pulseaudio, it is. Low; pinning the allowlist to a definition count (or the gating conditions) would close it.
style
- On a gate failure with everything else clean,
tools/validate.pystill printsvalidated N source RPMsto stdout before exiting 1 (tools/validate.py:62-63) — the failure detail is on stderr from the gate, but the last stdout line reads as success. Minor CI-log confusion, agent-fixable.
docs-currency
- The Justfile's
validaterecipe comment still reads "provenance, source locks, Packit coverage" (Justfile:9) and now understates what the gate checks. Info.
Verified while reading: pipewire.spec:587 carries only the guard, not a definition; all 349 specs sit at packages/<name>/<name>.spec so the */*.spec glob covers the tree; test_validate.py assertions are substring-based, so the new checked N recipes stdout line breaks nothing; validate.yml's existing validate job runs tools/validate.py, so the gate is wired into CI as the body claims.
No findings from: security, intent-alignment.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
Review on projectbluefin#197 found the allowlist exempted the whole package rather than the two definitions it documents. `offenders()` did `if package in INHERITED: continue` before scanning, and `stale()` only required that at least one definition remain, so a future Fedora import bump could add a third, unconditional `%global tests_nonfatal 1` to pulseaudio -- beyond the two arch- and release-gated FIXMEs -- and sail past the gate. The docstring said the entry was "not permission to add more"; the code said otherwise. Reproduced against the pre-fix tree by adding exactly that third definition: checked 349 recipes: no new tests_nonfatal (inherited: pulseaudio) validated 349 source RPMs exit=0 INHERITED now maps package -> (count, reason), and `drifted()` replaces `stale()` by comparing the definition count in both directions. Too few -- down to none -- still means the entry outlived its import. Too many now means a definition was added after it, which is the thing the gate exists to refuse. The same tree now reports: pulseaudio: defines tests_nonfatal 3 time(s), allowlist records 2; a definition was added after the import (...:380, :385, :389) exit=1 An absent recipe is still deliberately not drift, for the same reason it was not stale: validate.py runs against synthetic trees. Also from the same review: * tools/validate.py printed `validated N source RPMs` to stdout before returning the gate's non-zero status, so a failing run's last stdout line read as success. The success line now prints only on exit 0; the gate's detail is already on stderr. * The Justfile `validate` recipe comment still listed only "provenance, source locks, Packit coverage" and understated what it now checks. tests/test_check_suppressed_tests.py covers both drift directions, and test_recorded_counts_match_the_real_tree names the recipe to revisit when an import bump changes the count. 383 tests pass; the 2 collection errors in test_atomic_publish and test_factory_witness are a missing PyYAML in this environment and are present on the base commit too. Assisted-by: Claude Opus 5 via hive Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Thanks — all three findings were real. Fixed in correctness:
|
castrojo
left a comment
There was a problem hiding this comment.
Reviewed: Adds tools/check_suppressed_tests.py and wires it into tools/validate.py and Justfile check to forbid recipes from silencing %check with tests_nonfatal. Adds unit tests and skill documentation. Clean trial merge and passes all factory validation checks. Awaiting second independent review to satisfy merge policy.
There was a problem hiding this comment.
security
- (low) The gate matches only
%global/%define tests_nonfatalinpackages/*/*.spec(tools/check_suppressed_tests.py:361—DEFINITIONregex, and theoffenders()glob attools/check_suppressed_tests.py:392). A recipe can still silence its own%checkby deleting the%{!?tests_nonfatal:exit $TESTS_ERROR}guard line, appending|| :to%meson_test, or defining the macro in an%included file outside the*.specglob. The SKILL.md "Never" prose covers those, but the mechanical gate does not — worth a docstring caveat so nobody reads the gate as complete coverage of the PR title's claim.
Verified against head ad47b0d: packages/pulseaudio/pulseaudio.spec:384,388 matches the allowlisted count of 2; drifted()'s relative_to(root) works with the Path(".") root CI uses; synthetic trees in tests/test_validate.py are unaffected (no pulseaudio recipe → entry skipped, per drifted()'s absent-recipe rule).
No findings from: correctness, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
Review on projectbluefin#197 asked for this, and it is a fair reading of the title: "refuse a recipe that silences its own %check" claims more than the code does. The gate greps one line -- a %global/%define tests_nonfatal in a packages/<name>/<name>.spec. That is the evasion projectbluefin#132 was about to use, not the only one available. Three routes still pass it, now named in the module docstring and in the skill rather than left for the next author to discover: * deleting the %{!?tests_nonfatal:exit $TESTS_ERROR} guard, which leaves a %check that prints "test failed" and exits 0 with no macro at all; * appending `|| :` to the test command, which never reaches TESTS_ERROR; * defining the macro outside the packages/*/*.spec glob. Not a hypothetical -- packages/grub2/grub2.spec:1136 already does `%include %{SOURCE11}`, so a source file is a real place to hide one. Catching those mechanically means parsing %check bodies instead of grepping for a line, which is a different tool and a larger one. Until it exists the honest framing is a tripwire on the likeliest route, with AGENTS.md and the build-failure-triage skill carrying the actual rule, and a reviewer still reading the %check diff. Docstring and SKILL.md only; no behaviour change. The gate still passes 349 recipes and tests/test_check_suppressed_tests.py is 14/14. Assisted-by: Claude Opus 5 (1M context) via hive Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
Fair finding, and the fix is the docstring caveat you asked for — You're right that the title claims more than the code does. The gate greps one line, and I verified each of your three evasions against this tree rather than just repeating them:
(Checked the glob claim too — all 349 specs are at Catching these mechanically means parsing I added the same caveat to Verification:
|
|
All four findings across these two reviews are closed at this head (fedf992). Walking each, since they landed in the same commit and the reviews are spread out: 1. INHERITED = {
"pulseaudio": (2, "inherited from Fedora dist-git, arch and release gated"),
}
2. 3. Justfile comment understated the gate (docs-currency). Updated to "provenance, source locks, Packit coverage, and no recipe silencing its own 4. The gate is narrower than the PR title claims (security). This was the most useful one, and I've put the caveat in the module docstring rather than leaving the gate to be read as complete coverage. It now has a "What this gate does not catch" section naming all three routes you listed: deleting the On that last one — it isn't hypothetical, which I only found by checking:
|
hanthor
left a comment
There was a problem hiding this comment.
Reviewed: skill-doc additions on alarm-hang diagnosis and rebuild scheduling. CI 3/3 green, CLEAN, docs-only, low risk.
Refs #132.
#132 has three listed resolutions, and two of them need something this
repository does not have: interactive reproduction in a build root to
root-cause the hang, and a freedesktop.org GitLab account to report it
upstream. What it does have is a fourth, unlisted option that the issue
names as explicitly unacceptable, and nothing in the tree stopped anyone
taking it. This closes that.
The shortcut
Fedora's audio recipes end
%checkwithso one
%global tests_nonfatal 1above it turnspipewire's hangingpw-test-endpointinto a green build. The log still printstest failed;the build still succeeds; nothing downstream says otherwise.
For
pipewirespecifically that ships a desktop image with a known hang inits audio stack, and there is no other source to fall back on — the issue
measured that
public-hummingbird-x86_64-rpmscarries nopipewirepackagesat all, and
packages/pipewire-libs-extra/pipewire-libs-extra.spec:30carriesRequires: pipewire >= %{version}whileconfig/bluefin-packages.toml:84installs it.
The gate
tools/check_suppressed_tests.pyrefuses any%global/%defineoftests_nonfatalunderpackages/, andtools/validate.pycalls it — so theexisting
validatejob gates it and no new workflow,Justfilerecipe orpre-commit hook enters the catalog.
One thing turned up while writing it, which is the reason the allowlist
exists:
packages/pulseaudio/pulseaudio.specalready defines it, twice.Both are Fedora's own FIXMEs, imported verbatim — one arch-gated (i686
cpu-remap-test, s390xcore-util-test) and one release-gated%if 0%{?fedora} > 27, which is therefore always on for the Fedora 44 buildroot. PulseAudio's suite is effectively advisory in this factory today. That
is recorded rather than changed here: it is inherited, it is not what #132 is
about, and reverting Fedora's decision is a separate call. This pull request
only stops the list growing.
The entry is checked in both directions, so it cannot rot:
pulseaudiostops defining the macro, the gate fails and says to dropthe entry;
tests/test_check_suppressed_tests.pyfails instead.
An absent recipe is deliberately not a gate failure:
tools/validate.pyrunsagainst synthetic trees in
tests/test_validate.py, and a gate whose resultdepends on which tree it was pointed at would be the wrong kind of gate.
The triage that is worth not repeating
.agents/skills/build-failure-triage/SKILL.mdtakes three findings from theissue's measurements, per the same-pull-request learning mandate:
SIGALRMwith no output at aconstant wall-clock second is a hang. The starvation hypothesis was tested
and disproved: serialized with
--num-processes 1,pw-test-endpointstillfailed at
5.00swhilepw-test-filterin the same run was handed 18seconds and passed. Neither serializing
%checknor raising the test's ownalarm(5)is a fix, and serializing belongs in a dispatch, not a recipe.preparein.github/workflows/rebuild-rpms.ymlevaluatesfull or name in changed or name not in published or ..., sochangedwinsbefore the published repository is consulted. Restoring a recipe
byte-for-byte does not restore its skip — the pull request still touched
packages/<name>/. Only leaving it alone does, which is what option 3 inthe issue actually requires.
tests_nonfatalis refused, and what the one inherited exception is.Plus rows in the classification table,
NeverandCommon Rationalizations.Not in scope
No recipe under
packages/is touched, by design — a changed recipe is ascheduled rebuild, and
pipewireis still the failure it was. The hang itselfstill needs interactive reproduction, and the upstream report still needs a
freedesktop.org account. This makes the wrong answer impossible and writes
down what the two measured runs bought.
Validation
pytestandpre-commitare unavailable in this environment; the new testsare plain
unittest, whichpytestcollects unchanged. Two pre-existingloader errors (
test_atomic_publish,test_factory_witness) come from amissing local
PyYAMLand are identical with and without this branch.Supersedes #165, which the same content was split out of, on the maintainer's
note there to resubmit rebased once #178 established the baseline. Rebased on
mainat 3d38215, and therebuild_plan.plan()reference in the original hasbeen corrected — that module does not exist on
main; the logic is thepreparematrix step quoted above.— hive: backend=claude model=claude-opus-5