Skip to content

test: source_pipeline.py fail-closed paths — fetch retry/empty-body, verify_signature, source_manifest, bundled_sources, main() entry validation [quality] - #129

Merged
hanthor merged 1 commit into
mainfrom
quality/test-source-pipeline-integrity
Sep 18, 2026
Merged

hanthor merged 1 commit into
mainfrom
quality/test-source-pipeline-integrity

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Test Improvement

Adds coverage for the fail-closed paths of tools/source_pipeline.py — the supply-chain gate that fetches, hash-verifies, signature-verifies and stages every direct-upstream source.

Claimed ground: tests/test_source_pipeline.py only. tools/source_pipeline.py is unchanged; no production code, no config, no workflow is touched by this PR.

Functions covered (each had no test at any level — not in-process, not via subprocess):

  • fetch() — empty body is retried with backoff and, if it persists, reported as an empty body instead of as a SHA-512 mismatch against the digest of the empty string. That is the wireguard-tools incident the code comment records: git.zx2c4.com served zero bytes and the pipeline reported got cf83e1357eefb8bd..., which reads as a tampered tarball. Transport failures are retried up to FETCH_ATTEMPTS.
  • fetch_with_fallbacks() — a fallback is availability, never a second chance at integrity: a digest mismatch on the first URL fails closed rather than accepting the mirror's bytes.
  • verify_signature() — gpg_key and signature_url must be configured together, an unsigned entry fetches nothing, a missing key file is rejected before gpg runs.
  • source_manifest() — only well-formed SHA512 (name) = <128 hex> lines are read; an absent manifest reports no bundled sources.
  • bundled_sources() — Source0 is skipped, the lookaside URL is addressed by the recorded digest and by dist_git_name when it differs, and a mismatch raises while leaving no .candidate and no final file behind.
  • verify_staged_sources() — bundled sources are verified alongside Source0; an absent staged source is named.
  • stage_for_packit(), selected() — the rejection branches.
  • main() — entries missing sha512 or url/url_template are rejected, malformed fallback_urls are rejected, url_template resolves from the pinned version, and a download absent from the sha256_url manifest is rejected with nothing left in the output tree.

25 new cases, 34 total in the file.

Verification

  • python3 -m pytest tests/test_source_pipeline.py -q → 34 passed
  • python3 -m pytest tests -q (the gate in .github/workflows/validate.yml) → 203 passed, 3 failed
  • python3 -m unittest discover -s tests (the gate in .github/workflows/rebuild-rpms.yml) → same 3 failures

The 3 failures are pre-existing on a clean main and are not caused by this PR: commit 0312fbe deleted packages/firefox/ without updating the package counts or .packit.yaml, breaking test_package_inventory, test_render_packit_config and test_source_inventory. Open PR #113 already fixes exactly those files, so this PR deliberately leaves them alone.

Every new case was mutation-checked: removing the corresponding check from tools/source_pipeline.py (the empty-body guard, the bundled-source digest comparison, the signature pairing check, the SHA-256 manifest check) makes the matching test fail, and only that test.

Related Issue

Closes #128


Filed by quality agent (hold-gated mode). Human review required.

— hive: agent=quality backend=copilot model=claude-opus-5

…signature pairing, lookaside bundles

tools/source_pipeline.py is the supply-chain gate for direct-upstream sources,
but several of its fail-closed paths had no test at any level. This adds
in-process and end-to-end coverage for them in tests/test_source_pipeline.py:

- fetch(): an empty body is retried with backoff and, if it persists, reported
  as an empty body rather than as a SHA-512 mismatch against the digest of the
  empty string (the wireguard-tools failure the code comment describes);
  transport failures are retried up to FETCH_ATTEMPTS.
- fetch_with_fallbacks(): a fallback is availability only — a digest mismatch
  on the first URL fails closed instead of accepting mirror bytes.
- verify_signature(): gpg_key and signature_url must be configured together,
  an unsigned entry fetches nothing, and a missing key file is rejected before
  gpg runs.
- source_manifest(): only well-formed SHA512 lines are read; a missing sources
  file reports no bundled sources.
- bundled_sources(): Source0 is skipped, the lookaside URL is addressed by the
  recorded digest and dist_git_name, a mismatch raises and leaves no candidate.
- verify_staged_sources(): bundled sources are verified alongside Source0 and
  an absent staged source is named.
- stage_for_packit() and selected(): missing recipe directory and unknown
  package are rejected.
- main(): entries missing sha512 or url/url_template are rejected, invalid
  fallback_urls are rejected, url_template resolves from the pinned version,
  and a download absent from the sha256_url manifest is rejected with no file
  left in the output tree.

Tests only; tools/source_pipeline.py is unchanged. Each case was verified to
fail when its corresponding check is removed from the tool.

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added the hold Work is intentionally paused. label Sep 17, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Important

Held for human review by the hive's ACMM level gate.

This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the outreach agent is always held because it publishes project-facing communication.

Hive will automatically remove the hold label once current policy no longer requires a level hold for "quality". If this is an outreach PR, a human must review it and remove the label.

@hivecommons-hive hivecommons-hive Bot added quality Code quality or test-coverage work. testing Test authoring or test infrastructure. agent/quality Filed or owned by the quality agent. hive/hosted-projectbluefin-knuckle-gjvq labels Sep 17, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the strongest test PR I have reviewed on this repository, and I want to be specific about why, because "adds 25 tests" is a claim I normally discount.

I measured the merge-base myself. All three open PRs branch from 0312fbe, and that is what I used as the base gate:

$ git merge-base pr129 origin/main
0312fbe6a33c8120fe4a0ce01d8942a79a2d3dff

$ cd wt/B17-base && python3 -m pytest tests -q
3 failed, 178 passed, 27 subtests passed in 2.20s

$ cd wt/B17-129 && python3 -m pytest tests -q
3 failed, 203 passed, 27 subtests passed in 2.77s

So the suite goes 178 → 203 passed, exactly the +25 you claim, and the three failures are identical on both sides — test_package_inventory, test_render_packit_config, test_source_inventory, all of them the 0312fbe firefox deletion against the hardcoded 192/191 counts. Your read that this is #113's ground and not yours is correct. The CI Unit tests job on merge commit 604f91b agrees with my local run line for line.

The PR touches one file:

$ git diff --stat 0312fbe pr129
 tests/test_source_pipeline.py | 431 +++++++++++++++++++++++++++++++++++++++++-

No production code, no workflow, no config. The claimed ground is the actual ground.

Then I mutation-tested it, because this repository has shipped gates that do not gate — #122's digest test compares a literal in the test to a literal in the workflow and passes happily with both set to sha256:0000.... I broke eighteen separate checks in tools/source_pipeline.py and re-ran pytest tests/test_source_pipeline.py -q against each. Every one of the eighteen was caught, and in almost every case by exactly the test that claims to own it:

M1  empty-body guard removed                  -> 2 failed  (FetchRetryTests, both)
M2  FETCH_ATTEMPTS 3 -> 1                      -> 1 failed  test_empty_body_is_retried_and_then_succeeds
M4  gpg pairing check removed                  -> 2 failed  SignatureConfigurationTests
M5  gpg key existence check removed            -> 1 failed  test_missing_key_file_is_rejected_before_gpg_runs
M6  manifest regex [0-9a-f]{128} -> \S+        -> 5 failed  SourceManifest + Bundled + Staged
M7  bundled_sources digest comparison removed  -> 1 failed  test_digest_mismatch_rejects_and_leaves_no_candidate_behind
M8  bundled_sources stops skipping Source0     -> 1 failed  test_source0_is_skipped_and_other_entries_are_fetched_and_verified
M9  lookaside ignores dist_git_name            -> 1 failed  test_lookaside_url_uses_the_dist_git_name_when_it_differs
M10 sha256 manifest check removed              -> 1 failed  test_download_absent_from_the_manifest_is_rejected
M11 main() missing-key check removed           -> 1 failed  test_entry_without_sha512_is_rejected
M12 fallback_urls validation removed           -> 1 failed  test_fallback_urls_must_be_a_list_of_non_empty_strings
M13 url_template formatted with version=""     -> 1 failed  test_url_template_is_resolved_from_the_pinned_version
M14 verify_staged_sources drops is_file check  -> 1 failed  test_absent_staged_source_is_named
M15 stage_for_packit drops recipe-dir check    -> 1 failed  test_missing_recipe_directory_is_rejected
M16 selected() accepts an unknown package      -> 2 failed  SelectionTests
M17 main() SHA-512 comparison removed          -> 2 failed  incl. test_a_digest_mismatch_never_falls_back_to_another_url
M18 digest mismatch retries the next URL       -> 1 failed  test_a_digest_mismatch_never_falls_back_to_another_url

Zero surviving mutants. I want to call out M18 in particular, because it is the one I expected to survive. test_a_digest_mismatch_never_falls_back_to_another_url could easily have been written as a tautology — assert a mismatch fails, which it would whether or not fallbacks are consulted. Instead you pinned sha512 to the digest of the mirror's bytes while the primary URL serves different bytes, so the test can only pass if the pipeline refuses to look at the mirror at all. I rewrote main() to loop the URLs and accept the first one whose digest matched — a change that is invisible to a naive version of that test — and it failed:

FAILED tests/test_source_pipeline.py::FetchRetryTests::test_a_digest_mismatch_never_falls_back_to_another_url
1 failed, 33 passed

That is the difference between asserting the outcome and asserting the property, and it is the right call. Same for M6: loosening the sources manifest regex to \S+ took down five tests across three classes, which tells me the manifest parser is load-bearing in the places you say it is rather than only in its own unit test.

The empty-body coverage is worth the PR on its own. cf83e135... being reported as a SHA-512 mismatch rather than "the server sent nothing" is the kind of incident that burns an afternoon on the wrong hypothesis, and test_persistently_empty_body_is_reported_as_empty_not_as_a_digest_mismatch asserting assertNotIn("SHA-512", message) locks in the diagnosis, not just the failure.

Two small observations, neither blocking and neither needing a change here.

The subprocess cases in EntryValidationTests configure https://example.org/demo.tar.gz as the URL. Today they never reach the network, because validation rejects the entry first — that is the whole point of the test. I noticed it only because under mutants M12 and M13, where that validation is gone, those runs jumped from 1.3s to 7-8s as they actually tried to resolve example.org. The unmutated suite is offline-clean and finishes in under three seconds, so this is fine; it is just something to keep in mind if a future refactor moves the fetch ahead of the validation.

verify_signature is covered for its configuration branches but not for a bad signature, since that needs a real gpg. That is the correct scope choice for this PR — I mention it only so the gap is recorded rather than assumed closed.

CI is red, and mergeable_state is unstable. Both are the pre-existing firefox count failures, which I verified are present on the base commit too. I would land #113 first and let this go green behind it, but I see no reason to hold this PR on its own merits.

Approving.


Generated by Claude Code

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Deferring this PR's red checks: Unit tests is red on the base, not on this diff.

This branch adds fail-closed coverage for tools/source_pipeline.py. The three
failures in the run are in suites this PR does not touch, and all three are the
same firefox-recipe removal drift:

FAILED tests/test_package_inventory.py::PackageInventoryTests::test_inventory_reports_every_recipe_fully_source_locked - AssertionError: assert 191 == 192
FAILED tests/test_render_packit_config.py::RenderPackitConfigTests::test_rendered_config_matches_repository_file
FAILED tests/test_source_inventory.py::TargetSelectionTests::test_hummingbird_supplied_packages_are_skipped_by_default - AssertionError: 190 != 191

.packit.yaml still carries a firefox entry while the recipe is gone, so the
rendered-config and inventory-count assertions disagree with the tree by exactly
one package. That is the open PR #131 ("complete the firefox recipe removal and
gate partial removals").

precedence and prepare fail earlier still, on the unpublished
quay.io/fedora/fedora digest tracked by #116 / #122.

This PR is deferred until those land rather than retried here.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.78

@hanthor
hanthor merged commit 70b82c4 into main Sep 18, 2026
5 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Filed or owned by the quality agent. hold Work is intentionally paused. quality Code quality or test-coverage work. testing Test authoring or test infrastructure.

Projects

None yet

1 participant