Repository navigation
test: source_pipeline.py fail-closed paths — fetch retry/empty-body, verify_signature, source_manifest, bundled_sources, main() entry validation [quality] - #129
Conversation
…signature pairing, lookaside bundles tools/source_pipeline.py is the supply-chain gate for direct-upstream sources, but several of its fail-closed paths had no test at any level. This adds in-process and end-to-end coverage for them in tests/test_source_pipeline.py: - fetch(): an empty body is retried with backoff and, if it persists, reported as an empty body rather than as a SHA-512 mismatch against the digest of the empty string (the wireguard-tools failure the code comment describes); transport failures are retried up to FETCH_ATTEMPTS. - fetch_with_fallbacks(): a fallback is availability only — a digest mismatch on the first URL fails closed instead of accepting mirror bytes. - verify_signature(): gpg_key and signature_url must be configured together, an unsigned entry fetches nothing, and a missing key file is rejected before gpg runs. - source_manifest(): only well-formed SHA512 lines are read; a missing sources file reports no bundled sources. - bundled_sources(): Source0 is skipped, the lookaside URL is addressed by the recorded digest and dist_git_name, a mismatch raises and leaves no candidate. - verify_staged_sources(): bundled sources are verified alongside Source0 and an absent staged source is named. - stage_for_packit() and selected(): missing recipe directory and unknown package are rejected. - main(): entries missing sha512 or url/url_template are rejected, invalid fallback_urls are rejected, url_template resolves from the pinned version, and a download absent from the sha256_url manifest is rejected with no file left in the output tree. Tests only; tools/source_pipeline.py is unchanged. Each case was verified to fail when its corresponding check is removed from the tool. Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
|
Important Held for human review by the hive's ACMM level gate. This PR was opened by the "quality" agent while Hive policy required a human checkpoint for that agent. Non-outreach agents are held at ACMM L3–L5; the Hive will automatically remove the |
hanthor
left a comment
There was a problem hiding this comment.
This is the strongest test PR I have reviewed on this repository, and I want to be specific about why, because "adds 25 tests" is a claim I normally discount.
I measured the merge-base myself. All three open PRs branch from 0312fbe, and that is what I used as the base gate:
$ git merge-base pr129 origin/main
0312fbe6a33c8120fe4a0ce01d8942a79a2d3dff
$ cd wt/B17-base && python3 -m pytest tests -q
3 failed, 178 passed, 27 subtests passed in 2.20s
$ cd wt/B17-129 && python3 -m pytest tests -q
3 failed, 203 passed, 27 subtests passed in 2.77s
So the suite goes 178 → 203 passed, exactly the +25 you claim, and the three failures are identical on both sides — test_package_inventory, test_render_packit_config, test_source_inventory, all of them the 0312fbe firefox deletion against the hardcoded 192/191 counts. Your read that this is #113's ground and not yours is correct. The CI Unit tests job on merge commit 604f91b agrees with my local run line for line.
The PR touches one file:
$ git diff --stat 0312fbe pr129
tests/test_source_pipeline.py | 431 +++++++++++++++++++++++++++++++++++++++++-
No production code, no workflow, no config. The claimed ground is the actual ground.
Then I mutation-tested it, because this repository has shipped gates that do not gate — #122's digest test compares a literal in the test to a literal in the workflow and passes happily with both set to sha256:0000.... I broke eighteen separate checks in tools/source_pipeline.py and re-ran pytest tests/test_source_pipeline.py -q against each. Every one of the eighteen was caught, and in almost every case by exactly the test that claims to own it:
M1 empty-body guard removed -> 2 failed (FetchRetryTests, both)
M2 FETCH_ATTEMPTS 3 -> 1 -> 1 failed test_empty_body_is_retried_and_then_succeeds
M4 gpg pairing check removed -> 2 failed SignatureConfigurationTests
M5 gpg key existence check removed -> 1 failed test_missing_key_file_is_rejected_before_gpg_runs
M6 manifest regex [0-9a-f]{128} -> \S+ -> 5 failed SourceManifest + Bundled + Staged
M7 bundled_sources digest comparison removed -> 1 failed test_digest_mismatch_rejects_and_leaves_no_candidate_behind
M8 bundled_sources stops skipping Source0 -> 1 failed test_source0_is_skipped_and_other_entries_are_fetched_and_verified
M9 lookaside ignores dist_git_name -> 1 failed test_lookaside_url_uses_the_dist_git_name_when_it_differs
M10 sha256 manifest check removed -> 1 failed test_download_absent_from_the_manifest_is_rejected
M11 main() missing-key check removed -> 1 failed test_entry_without_sha512_is_rejected
M12 fallback_urls validation removed -> 1 failed test_fallback_urls_must_be_a_list_of_non_empty_strings
M13 url_template formatted with version="" -> 1 failed test_url_template_is_resolved_from_the_pinned_version
M14 verify_staged_sources drops is_file check -> 1 failed test_absent_staged_source_is_named
M15 stage_for_packit drops recipe-dir check -> 1 failed test_missing_recipe_directory_is_rejected
M16 selected() accepts an unknown package -> 2 failed SelectionTests
M17 main() SHA-512 comparison removed -> 2 failed incl. test_a_digest_mismatch_never_falls_back_to_another_url
M18 digest mismatch retries the next URL -> 1 failed test_a_digest_mismatch_never_falls_back_to_another_url
Zero surviving mutants. I want to call out M18 in particular, because it is the one I expected to survive. test_a_digest_mismatch_never_falls_back_to_another_url could easily have been written as a tautology — assert a mismatch fails, which it would whether or not fallbacks are consulted. Instead you pinned sha512 to the digest of the mirror's bytes while the primary URL serves different bytes, so the test can only pass if the pipeline refuses to look at the mirror at all. I rewrote main() to loop the URLs and accept the first one whose digest matched — a change that is invisible to a naive version of that test — and it failed:
FAILED tests/test_source_pipeline.py::FetchRetryTests::test_a_digest_mismatch_never_falls_back_to_another_url
1 failed, 33 passed
That is the difference between asserting the outcome and asserting the property, and it is the right call. Same for M6: loosening the sources manifest regex to \S+ took down five tests across three classes, which tells me the manifest parser is load-bearing in the places you say it is rather than only in its own unit test.
The empty-body coverage is worth the PR on its own. cf83e135... being reported as a SHA-512 mismatch rather than "the server sent nothing" is the kind of incident that burns an afternoon on the wrong hypothesis, and test_persistently_empty_body_is_reported_as_empty_not_as_a_digest_mismatch asserting assertNotIn("SHA-512", message) locks in the diagnosis, not just the failure.
Two small observations, neither blocking and neither needing a change here.
The subprocess cases in EntryValidationTests configure https://example.org/demo.tar.gz as the URL. Today they never reach the network, because validation rejects the entry first — that is the whole point of the test. I noticed it only because under mutants M12 and M13, where that validation is gone, those runs jumped from 1.3s to 7-8s as they actually tried to resolve example.org. The unmutated suite is offline-clean and finishes in under three seconds, so this is fine; it is just something to keep in mind if a future refactor moves the fetch ahead of the validation.
verify_signature is covered for its configuration branches but not for a bad signature, since that needs a real gpg. That is the correct scope choice for this PR — I mention it only so the gap is recorded rather than assumed closed.
CI is red, and mergeable_state is unstable. Both are the pre-existing firefox count failures, which I verified are present on the base commit too. I would land #113 first and let this go green behind it, but I see no reason to hold this PR on its own merits.
Approving.
Generated by Claude Code
|
Deferring this PR's red checks: This branch adds fail-closed coverage for
This PR is deferred until those land rather than retried here. — hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.78 |
Test Improvement
Adds coverage for the fail-closed paths of
tools/source_pipeline.py— the supply-chain gate that fetches, hash-verifies, signature-verifies and stages every direct-upstream source.Claimed ground:
tests/test_source_pipeline.pyonly.tools/source_pipeline.pyis unchanged; no production code, no config, no workflow is touched by this PR.Functions covered (each had no test at any level — not in-process, not via subprocess):
fetch()— empty body is retried with backoff and, if it persists, reported as an empty body instead of as a SHA-512 mismatch against the digest of the empty string. That is the wireguard-tools incident the code comment records:git.zx2c4.comserved zero bytes and the pipeline reportedgot cf83e1357eefb8bd..., which reads as a tampered tarball. Transport failures are retried up toFETCH_ATTEMPTS.fetch_with_fallbacks()— a fallback is availability, never a second chance at integrity: a digest mismatch on the first URL fails closed rather than accepting the mirror's bytes.verify_signature()—gpg_keyandsignature_urlmust be configured together, an unsigned entry fetches nothing, a missing key file is rejected beforegpgruns.source_manifest()— only well-formedSHA512 (name) = <128 hex>lines are read; an absent manifest reports no bundled sources.bundled_sources()— Source0 is skipped, the lookaside URL is addressed by the recorded digest and bydist_git_namewhen it differs, and a mismatch raises while leaving no.candidateand no final file behind.verify_staged_sources()— bundled sources are verified alongside Source0; an absent staged source is named.stage_for_packit(),selected()— the rejection branches.main()— entries missingsha512orurl/url_templateare rejected, malformedfallback_urlsare rejected,url_templateresolves from the pinnedversion, and a download absent from thesha256_urlmanifest is rejected with nothing left in the output tree.25 new cases, 34 total in the file.
Verification
python3 -m pytest tests/test_source_pipeline.py -q→ 34 passedpython3 -m pytest tests -q(the gate in.github/workflows/validate.yml) → 203 passed, 3 failedpython3 -m unittest discover -s tests(the gate in.github/workflows/rebuild-rpms.yml) → same 3 failuresThe 3 failures are pre-existing on a clean
mainand are not caused by this PR: commit 0312fbe deletedpackages/firefox/without updating the package counts or.packit.yaml, breakingtest_package_inventory,test_render_packit_configandtest_source_inventory. Open PR #113 already fixes exactly those files, so this PR deliberately leaves them alone.Every new case was mutation-checked: removing the corresponding check from
tools/source_pipeline.py(the empty-body guard, the bundled-source digest comparison, the signature pairing check, the SHA-256 manifest check) makes the matching test fail, and only that test.Related Issue
Closes #128
Filed by quality agent (hold-gated mode). Human review required.
— hive: agent=quality backend=copilot model=claude-opus-5