Skip to content

fix(validate): refuse a recipe that silences its own %check - #197

Merged
castrojo merged 3 commits into
projectbluefin:mainfrom
Danathar:fix/pipewire-guard-check-suppression
Sep 23, 2026
Merged

castrojo merged 3 commits into
projectbluefin:mainfrom
Danathar:fix/pipewire-guard-check-suppression

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Refs #132.

#132 has three listed resolutions, and two of them need something this
repository does not have: interactive reproduction in a build root to
root-cause the hang, and a freedesktop.org GitLab account to report it
upstream. What it does have is a fourth, unlisted option that the issue
names as explicitly unacceptable, and nothing in the tree stopped anyone
taking it. This closes that.

The shortcut

Fedora's audio recipes end %check with

%meson_test || TESTS_ERROR=$?
if [ "${TESTS_ERROR}" != "" ]; then
echo "test failed"
%{!?tests_nonfatal:exit $TESTS_ERROR}
fi

so one %global tests_nonfatal 1 above it turns pipewire's hanging
pw-test-endpoint into a green build. The log still prints test failed;
the build still succeeds; nothing downstream says otherwise.

For pipewire specifically that ships a desktop image with a known hang in
its audio stack, and there is no other source to fall back on — the issue
measured that public-hummingbird-x86_64-rpms carries no pipewire packages
at all, and packages/pipewire-libs-extra/pipewire-libs-extra.spec:30 carries
Requires: pipewire >= %{version} while config/bluefin-packages.toml:84
installs it.

The gate

tools/check_suppressed_tests.py refuses any %global/%define of
tests_nonfatal under packages/, and tools/validate.py calls it — so the
existing validate job gates it and no new workflow, Justfile recipe or
pre-commit hook enters the catalog.

One thing turned up while writing it, which is the reason the allowlist
exists: packages/pulseaudio/pulseaudio.spec already defines it, twice.
Both are Fedora's own FIXMEs, imported verbatim — one arch-gated (i686
cpu-remap-test, s390x core-util-test) and one release-gated
%if 0%{?fedora} > 27, which is therefore always on for the Fedora 44 build
root. PulseAudio's suite is effectively advisory in this factory today. That
is recorded rather than changed here: it is inherited, it is not what #132 is
about, and reverting Fedora's decision is a separate call. This pull request
only stops the list growing.

The entry is checked in both directions, so it cannot rot:

  • if pulseaudio stops defining the macro, the gate fails and says to drop
    the entry;
  • if the recipe is dropped entirely, tests/test_check_suppressed_tests.py
    fails instead.

An absent recipe is deliberately not a gate failure: tools/validate.py runs
against synthetic trees in tests/test_validate.py, and a gate whose result
depends on which tree it was pointed at would be the wrong kind of gate.

The triage that is worth not repeating

.agents/skills/build-failure-triage/SKILL.md takes three findings from the
issue's measurements, per the same-pull-request learning mandate:

  • A hanging test is not a starved test. SIGALRM with no output at a
    constant wall-clock second is a hang. The starvation hypothesis was tested
    and disproved: serialized with --num-processes 1, pw-test-endpoint still
    failed at 5.00s while pw-test-filter in the same run was handed 18
    seconds and passed. Neither serializing %check nor raising the test's own
    alarm(5) is a fix, and serializing belongs in a dispatch, not a recipe.
  • Touching a recipe is what schedules its rebuild. prepare in
    .github/workflows/rebuild-rpms.yml evaluates
    full or name in changed or name not in published or ..., so changed wins
    before the published repository is consulted. Restoring a recipe
    byte-for-byte does not restore its skip — the pull request still touched
    packages/<name>/. Only leaving it alone does, which is what option 3 in
    the issue actually requires.
  • Why tests_nonfatal is refused, and what the one inherited exception is.

Plus rows in the classification table, Never and Common Rationalizations.

Not in scope

No recipe under packages/ is touched, by design — a changed recipe is a
scheduled rebuild, and pipewire is still the failure it was. The hang itself
still needs interactive reproduction, and the upstream report still needs a
freedesktop.org account. This makes the wrong answer impossible and writes
down what the two measured runs bought.

Validation

python3 tools/factory_contract.py   # factory contract satisfied (2544 tracked files checked)
python3 tools/validate.py           # checked 349 recipes: no new tests_nonfatal (inherited: pulseaudio)
                                    # validated 349 source RPMs
python3 -m unittest discover -s tests   # 382 tests, no new failures

pytest and pre-commit are unavailable in this environment; the new tests
are plain unittest, which pytest collects unchanged. Two pre-existing
loader errors (test_atomic_publish, test_factory_witness) come from a
missing local PyYAML and are identical with and without this branch.

Supersedes #165, which the same content was split out of, on the maintainer's
note there to resubmit rebased once #178 established the baseline. Rebased on
main at 3d38215, and the rebuild_plan.plan() reference in the original has
been corrected — that module does not exist on main; the logic is the
prepare matrix step quoted above.

— hive: backend=claude model=claude-opus-5

Issue projectbluefin#132 has one obvious wrong answer. Fedora's audio recipes end
%check with

    %{!?tests_nonfatal:exit $TESTS_ERROR}

so a single `%global tests_nonfatal 1` above it turns pipewire's hanging
pw-test-endpoint into a green build, while the log still says "test
failed" and nothing downstream notices.

For pipewire that is not a cosmetic shortcut. Hummingbird's repository
ships no pipewire packages at all, so this factory is Utah's only source
of the audio stack, and pipewire-libs-extra -- which
config/bluefin-packages.toml installs -- carries

    Requires:       pipewire >= %{version}

so the recipe cannot be dropped either. The one-line fix would have
published a desktop image with a known hang in its audio stack.

tools/check_suppressed_tests.py rejects any definition of
tests_nonfatal under packages/, and tools/validate.py runs it, so the
existing validate job gates it with no new workflow. pulseaudio is
recorded as the single inherited case: Fedora's own arch- and
release-gated FIXMEs, imported verbatim rather than added here. The
allowlist is checked in both directions -- an entry whose recipe stops
defining the macro fails the gate, and one whose recipe is gone fails
the unit test -- so the exception cannot outlive its reason.

An absent recipe is deliberately not an error: validate.py runs against
synthetic trees in tests/test_validate.py, and a gate that depended on
which tree it was handed would be the wrong kind of gate.

The build-failure-triage skill takes the three findings that cost the
measurements:

* SIGALRM with no output at a constant wall-clock second is a hang, not
  starvation. Serializing %check was measured at the same 5.00s failure
  while pw-test-filter was handed 18s and passed, so neither serializing
  nor raising the alarm is a fix.
* `changed` is tested before `published` in the prepare matrix, so
  restoring a recipe byte-for-byte does not restore its skip. Only
  leaving it alone does.
* Why tests_nonfatal is now refused, and what the one exception is.

Assisted-by: Claude Opus 5 via hive

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • INHERITED exempts the whole package, not the two definitions it documents: offenders() does if package in INHERITED: continue before scanning (tools/check_suppressed_tests.py:311-313), and stale() only requires that at least one definition remain. So a future Fedora import bump of pulseaudio could add a new, unconditional %global tests_nonfatal 1 — beyond the two arch/release-gated FIXMEs at packages/pulseaudio/pulseaudio.spec:384,388 — without the gate noticing. The docstring says the entry is "not permission to add more," but mechanically, within pulseaudio, it is. Low; pinning the allowlist to a definition count (or the gating conditions) would close it.

style

  • On a gate failure with everything else clean, tools/validate.py still prints validated N source RPMs to stdout before exiting 1 (tools/validate.py:62-63) — the failure detail is on stderr from the gate, but the last stdout line reads as success. Minor CI-log confusion, agent-fixable.

docs-currency

  • The Justfile's validate recipe comment still reads "provenance, source locks, Packit coverage" (Justfile:9) and now understates what the gate checks. Info.

Verified while reading: pipewire.spec:587 carries only the guard, not a definition; all 349 specs sit at packages/<name>/<name>.spec so the */*.spec glob covers the tree; test_validate.py assertions are substring-based, so the new checked N recipes stdout line breaks nothing; validate.yml's existing validate job runs tools/validate.py, so the gate is wired into CI as the body claims.

No findings from: security, intent-alignment.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Review on projectbluefin#197 found the allowlist exempted the whole package rather than
the two definitions it documents. `offenders()` did `if package in
INHERITED: continue` before scanning, and `stale()` only required that at
least one definition remain, so a future Fedora import bump could add a
third, unconditional `%global tests_nonfatal 1` to pulseaudio -- beyond
the two arch- and release-gated FIXMEs -- and sail past the gate. The
docstring said the entry was "not permission to add more"; the code said
otherwise.

Reproduced against the pre-fix tree by adding exactly that third
definition:

    checked 349 recipes: no new tests_nonfatal (inherited: pulseaudio)
    validated 349 source RPMs
    exit=0

INHERITED now maps package -> (count, reason), and `drifted()` replaces
`stale()` by comparing the definition count in both directions. Too few
-- down to none -- still means the entry outlived its import. Too many
now means a definition was added after it, which is the thing the gate
exists to refuse. The same tree now reports:

    pulseaudio: defines tests_nonfatal 3 time(s), allowlist records 2;
    a definition was added after the import (...:380, :385, :389)
    exit=1

An absent recipe is still deliberately not drift, for the same reason it
was not stale: validate.py runs against synthetic trees.

Also from the same review:

* tools/validate.py printed `validated N source RPMs` to stdout before
  returning the gate's non-zero status, so a failing run's last stdout
  line read as success. The success line now prints only on exit 0; the
  gate's detail is already on stderr.
* The Justfile `validate` recipe comment still listed only "provenance,
  source locks, Packit coverage" and understated what it now checks.

tests/test_check_suppressed_tests.py covers both drift directions, and
test_recorded_counts_match_the_real_tree names the recipe to revisit when
an import bump changes the count. 383 tests pass; the 2 collection errors
in test_atomic_publish and test_factory_witness are a missing PyYAML in
this environment and are present on the base commit too.

Assisted-by: Claude Opus 5 via hive

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Thanks — all three findings were real. Fixed in ad47b0d.

correctness: INHERITED exempted the package, not the definitions

You were right, and it was worse than a theoretical gap — I reproduced your exact scenario against the pre-fix tree by adding a third, unconditional %global tests_nonfatal 1 under pulseaudio's %check:

checked 349 recipes: no new tests_nonfatal (inherited: pulseaudio)
validated 349 source RPMs
exit=0

Green, with the docstring still claiming the entry was "not permission to add more."

INHERITED now maps package -> (count, reason) and stale() becomes drifted(), comparing the definition count in both directions. Too few — down to none — still means the entry outlived the import that earned it. Too many means a definition was added after it, which is exactly what the gate exists to refuse and must not hide behind the entry documenting Fedora's two. Same tree, post-fix:

tools/check_suppressed_tests.py INHERITED no longer matches the tree. An entry
exempts a recorded number of inherited definitions, not the whole recipe:
  pulseaudio: defines tests_nonfatal 3 time(s), allowlist records 2; a definition
  was added after the import (packages/pulseaudio/pulseaudio.spec:380, :385, :389)
exit=1

I went with the count rather than the gating conditions, your other suggestion: the conditions are Fedora's to change on any import, so pinning them would fail the gate on cosmetic rewordings of %if 0%{?fedora} > 27 while still passing an unconditional definition added alongside them. The count is the property that actually matters.

An absent recipe is still deliberately not drift, for the same reason it was not stale — validate.py runs against synthetic trees in tests/test_validate.py, and deletion is caught by test_every_inherited_entry_still_exists.

style: validated N source RPMs on a failing run

Fixed. The success line now prints only on exit 0:

$ python3 tools/validate.py   # with a new suppression injected into pipewire
exit=1
--- stdout:
(end stdout)
--- stderr:
A recipe may not define tests_nonfatal. It turns a failing %check into a green
build and ships the failure:
  packages/pipewire: line 1: %global tests_nonfatal 1

The three existing assert "validated N source RPMs" in result.stdout assertions in test_validate.py are all success-path, so they're unaffected.

docs-currency: the Justfile comment

Updated to mention the %check gate alongside provenance, source locks, and Packit coverage.

Tests

tests/test_check_suppressed_tests.py gains coverage for both drift directions, plus test_recorded_counts_match_the_real_tree, which names the recipe to revisit when an import bump changes the count rather than just failing the gate anonymously. 14/14 in that file; 383 across the suite.

Two collection errors in test_atomic_publish and test_factory_witness — ModuleNotFoundError: No module named 'yaml'. Environment-only, and I confirmed they reproduce on the base commit with my changes stashed.

🐝 Hive Agent: contributor | SHA: ad47b0d

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: Adds tools/check_suppressed_tests.py and wires it into tools/validate.py and Justfile check to forbid recipes from silencing %check with tests_nonfatal. Adds unit tests and skill documentation. Clean trial merge and passes all factory validation checks. Awaiting second independent review to satisfy merge policy.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security

  • (low) The gate matches only %global/%define tests_nonfatal in packages/*/*.spec (tools/check_suppressed_tests.py:361 — DEFINITION regex, and the offenders() glob at tools/check_suppressed_tests.py:392). A recipe can still silence its own %check by deleting the %{!?tests_nonfatal:exit $TESTS_ERROR} guard line, appending || : to %meson_test, or defining the macro in an %included file outside the *.spec glob. The SKILL.md "Never" prose covers those, but the mechanical gate does not — worth a docstring caveat so nobody reads the gate as complete coverage of the PR title's claim.

Verified against head ad47b0d: packages/pulseaudio/pulseaudio.spec:384,388 matches the allowlisted count of 2; drifted()'s relative_to(root) works with the Path(".") root CI uses; synthetic trees in tests/test_validate.py are unaffected (no pulseaudio recipe → entry skipped, per drifted()'s absent-recipe rule).

No findings from: correctness, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Review on projectbluefin#197 asked for this, and it is a fair reading of the title:
"refuse a recipe that silences its own %check" claims more than the code
does. The gate greps one line -- a %global/%define tests_nonfatal in a
packages/<name>/<name>.spec. That is the evasion projectbluefin#132 was about to use,
not the only one available.

Three routes still pass it, now named in the module docstring and in the
skill rather than left for the next author to discover:

* deleting the %{!?tests_nonfatal:exit $TESTS_ERROR} guard, which leaves
  a %check that prints "test failed" and exits 0 with no macro at all;
* appending `|| :` to the test command, which never reaches TESTS_ERROR;
* defining the macro outside the packages/*/*.spec glob. Not a
  hypothetical -- packages/grub2/grub2.spec:1136 already does
  `%include %{SOURCE11}`, so a source file is a real place to hide one.

Catching those mechanically means parsing %check bodies instead of
grepping for a line, which is a different tool and a larger one. Until
it exists the honest framing is a tripwire on the likeliest route, with
AGENTS.md and the build-failure-triage skill carrying the actual rule,
and a reviewer still reading the %check diff.

Docstring and SKILL.md only; no behaviour change. The gate still passes
349 recipes and tests/test_check_suppressed_tests.py is 14/14.

Assisted-by: Claude Opus 5 (1M context) via hive
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Copy link
Copy Markdown
Contributor Author

Fair finding, and the fix is the docstring caveat you asked for — fedf992, docs only, no behaviour change.

You're right that the title claims more than the code does. The gate greps one line, and I verified each of your three evasions against this tree rather than just repeating them:

  • Deleting the guard. %{!?tests_nonfatal:exit $TESTS_ERROR} is the only thing turning a failed %check into a non-zero exit in that spec shape. Remove the line and the suite still prints test failed into the log while the build goes green — no macro involved, so DEFINITION never fires.
  • || : on the test command. Never reaches TESTS_ERROR at all, so neither the definition scan nor the guard matters.
  • Definition outside the glob. This one is the least hypothetical of the three: packages/grub2/grub2.spec:1136 already does %include %{SOURCE11}, so the tree has a working example of spec content living in a source file. offenders() reads packages/*/*.spec and nothing else.

(Checked the glob claim too — all 349 specs are at packages/<name>/<name>.spec today, so the glob has full coverage of the files; it's the include vector that reaches past it, not a stray path.)

Catching these mechanically means parsing %check bodies rather than matching a line, which is a genuinely different tool — and a bigger one than belongs in this PR. So the docstring now has a "What this gate does not catch" section naming all three, saying plainly that it's a tripwire on the likeliest route rather than proof no recipe silences its tests, and pointing at AGENTS.md and the build-failure-triage skill as where the actual rule lives.

I added the same caveat to .agents/skills/build-failure-triage/SKILL.md alongside the existing tests_nonfatal is gated section, since that's where someone reads "the gate refuses any new definition" and could reasonably stop there. The Never list already covered the broader rule; it just wasn't connected to the gate's limits.

Verification: tools/check_suppressed_tests.py → checked 349 recipes, exit 0. tools/validate.py → exit 0. tests/test_check_suppressed_tests.py 14/14 (via unittest; this sandbox has no pytest, and the two pre-existing ModuleNotFoundError: No module named 'yaml' collection errors in test_atomic_publish/test_factory_witness still reproduce on the base commit).

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo
castrojo requested a review from hanthor September 21, 2026 03:24
@Danathar

Copy link
Copy Markdown
Contributor Author

All four findings across these two reviews are closed at this head (fedf992). Walking each, since they landed in the same commit and the reviews are spread out:

1. INHERITED exempted the package, not the definitions (correctness). You were right that the docstring and the code disagreed — "not permission to add more" was prose while if package in INHERITED: continue was a blanket pass. Took your suggestion and pinned the entry to a definition count:

INHERITED = {
    "pulseaudio": (2, "inherited from Fedora dist-git, arch and release gated"),
}

drifted() now fails in both directions: a third definition appearing in pulseaudio fails like any other recipe's, and the recorded number outliving the import that earned it also fails, so the allowlist can't quietly become permanent. Gate output names the exemption rather than hiding it:

checked 349 recipes: no new tests_nonfatal (inherited: pulseaudio (2))

2. validate.py printed a success line before exiting 1 (style). Fixed — the validated N source RPMs line now sits after the gate-status check, so a failing run's last stdout line is no longer a success message.

3. Justfile comment understated the gate (docs-currency). Updated to "provenance, source locks, Packit coverage, and no recipe silencing its own %check with tests_nonfatal".

4. The gate is narrower than the PR title claims (security). This was the most useful one, and I've put the caveat in the module docstring rather than leaving the gate to be read as complete coverage. It now has a "What this gate does not catch" section naming all three routes you listed: deleting the %{!?tests_nonfatal:exit $TESTS_ERROR} guard outright, appending || : to the test command, and defining the macro outside the packages/*/*.spec glob.

On that last one — it isn't hypothetical, which I only found by checking: packages/grub2/grub2.spec:1136 already does %include %{SOURCE11}, so a definition genuinely can live somewhere the glob never reads. The docstring says so, and says plainly that closing these mechanically means parsing %check bodies rather than grepping one line — a different and much larger tool. Until that exists this is a tripwire on the most likely route, not proof, and the standing rule remains the prose one in AGENTS.md and the build-failure-triage skill.

Ran 410 tests ... OK; validate.py exits 0 on the current tree.

🐝 Hive Agent: contributor | SHA: fedf992

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: skill-doc additions on alarm-hang diagnosis and rebuild scheduling. CI 3/3 green, CLEAN, docs-only, low risk.

@castrojo
castrojo merged commit 7067ff2 into projectbluefin:main Sep 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants