Skip to content

[quality] test: unit coverage for tools/validate.py and tools/recalculate_hummingbird_gaps.py - #71

Closed
castrojo wants to merge 1 commit into
mainfrom
quality/test-validate-and-hummingbird-gaps
Closed

castrojo wants to merge 1 commit into
mainfrom
quality/test-validate-and-hummingbird-gaps

Conversation

@castrojo

Copy link
Copy Markdown
Contributor

Summary

Adds unit coverage for tools/validate.py and tools/recalculate_hummingbird_gaps.py.

Closes #60

What this adds

  • tests/test_validate.py: 15 cases testing tools/validate.py across synthetic factory trees (rawhide imports with required commit/tree, upstream imports with no snapshot, invalid schema, missing locks, missing Packit config, and validated count output).
  • tests/test_hummingbird_gaps.py: 16 cases testing tools/recalculate_hummingbird_gaps.py (whitespace/comment stripping in lines(), duplicate collapsing, contract partitioning across image, repo-only, and missing, excluded section handling, ISO-8601 timestamps, output directory creation, and real manifest validation).

Verification

PYTHONPATH=. pytest tests/test_validate.py tests/test_hummingbird_gaps.py
============================== 31 passed in 0.51s ==============================

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI root cause for the red srpm jobs: pinned packit image digest was pruned from quay (exit 125, manifest unknown). One-line fix at .github/workflows/packit-srpm-pilot.yml:90 — details and verified replacement digest: #74. Agent cannot push workflow-file changes (App lacks workflows permission).

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI root cause identified: packit-srpm-pilot.yml pins a packit image digest pruned from quay.io — every srpm (*) job dies with manifest unknown (exit 125), on main as well as this branch. Fix (repin to live digest 8a178425…, verified via quay API) is ready as commit 3017fe3 but the hive App lacks the workflows permission to push workflow edits. Tracked in the issue filed on this repo; needs a human push, then rebase this branch.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI red is systemic, two clusters: (1) all srpm jobs fail exit 125 — the pinned quay.io/packit/packit digest was pruned upstream, filed #84 (fix ground occupied by #70 which rewrites packit-srpm-pilot.yml); (2) rebuild* / build failures (samba/gpgme hummingbird dep mismatches, fish %check, firefox) are the cluster tracked by #30. No in-branch fix available without colliding with those PRs.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] rebuild failures share the same root cause as #56/#57/#58: ruby3.3/3.4/4.0 default-gems file conflicts in the buildroot transaction. Blocked on utah-packages#83; rebase after it merges.


scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean suites, but both files this PR adds are also added by other open PRs. Full matrix in #59; the collisions involving this branch, from three-dot diffs against main (54cc8d1):

61 71 ['tests/test_validate.py']
63 71 ['tests/test_hummingbird_gaps.py']

tests/test_validate.py is also added by #61, and tests/test_hummingbird_gaps.py by #63. Four PRs (#59, #61, #63, #71) between them write four test files, each one twice.

They form a cycle, so two of them cover everything with zero overlap: #59+#71 or #61+#63. Either pair is complete and needs no rebasing; the other two can be closed as duplicates.

This PR is in good shape. Merged into main locally (baseline 72 passed):

PR#71 fc=0 val=0 pytest=0 | validated 193 source RPMs | 103 passed in 1.47s
PR#71 quoting=0 runtime_contract=0 unittest=0 | validated runtime contract: 72 packages

+31 tests, every other gate unchanged. The body's 31 passed for the two files in isolation reproduces as part of the +31 total.

Comparing the overlapping halves, since the pairs are otherwise equivalent:

  • test_validate.py — #61's version drives validate.py as a subprocess over a synthetic factory tree, which is exactly how .github/workflows/validate.yml invokes it (python3 tools/validate.py is the whole step). This PR's 15 cases cover the same ground at a similar depth. Close to a wash; #61's harness is marginally more faithful to CI.
  • test_hummingbird_gaps.py — #63's version carries 14 cases plus a guard that no [excluded] package leaks into the contract. That one is worth keeping whichever pair wins: reports/hummingbird-gap.json is consumed by projectbluefin/utah, so a leak there misreports coverage to a downstream repo, not just locally. This PR's 16 cases cover the ISO-8601 measured_at and output-directory creation, which #63 also has.

So on content the pairs are close, with #63 holding the single most valuable individual case. Either choice is defensible; the important thing is making it once rather than merging three of the four and discovering the collisions at merge time.

One note on the test_validate.py interaction with #73: #73 refactors tools/validate.py into check_provenance() / main(root) with a __main__ guard. I merged #73 and #61 together and the suite stays green (97 passed), because these tests go through the CLI rather than the internals — the same should hold for this PR's version. Landing #73 first would let a later round of tests call the functions directly instead of shelling out.

Recommendation: pick a pair, close the other two with a pointer to which PR carries their coverage, and have the quality agent re-check disjointness against open PRs at review time rather than only at filing time.


Generated by Claude Code

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI root cause: both failure classes here are systemic, not from this PR.

  1. srpm jobs fail with docker: manifest unknown (exit 125) — the pinned quay.io/packit/packit@sha256:149e6e06... digest was garbage-collected upstream when latest was republished. Fix already open: ci: repin Packit image to live digest #87 (repins to the live digest 8a178425..., verified still current as of 2026-09-12).
  2. rebuild jobs fail on ruby3.3-default-gems / ruby3.4-default-gems / ruby4.0-default-gems file conflicts in the Hummingbird repo transaction. Fix already open: fix(buildroot): exclude legacy ruby default-gems from Hummingbird repo #83 (excludes the legacy default-gems from the Hummingbird repo).

Once #83 and #87 merge and this branch is rebased, CI should validate this PRs actual changes. Note: the hive App token currently lacks the workflows permission, so agent-authored pushes touching .github/workflows/ are rejected — the repin must land via a human-pushed PR (#87).


scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: 860e512

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] The mass red here is environmental, not this PR: (1) every srpm (*) job dies at docker run — the pinned quay.io/packit/packit@sha256:149e6e06… manifest was deleted upstream (404 verified); repin fixes are open and green in #85/#87 (duplicates — close one). (2) rebuild* failures come from Hummingbird repo content bugs with fixes already open: ruby3.4+ruby4.0 default-gems transaction conflict (#83), missing closure members like libwebrtc-audio-processing (#81), atomic publish gating (#89). Merging those + re-running should clear this PR.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

@castrojo castrojo added the lgtm Approved by a Hive merger/owner for auto-merge on green CI label Sep 13, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The lgtm here resolves the one thing my earlier review was waiting on — which of the two disjoint pairs to take. The branch is unchanged since I verified it (still 2c45477), so that verification stands: merged into main locally, 103 passed against a 72 passed baseline, +31 tests, every other gate unchanged.

This approval picks the #59 + #71 pair, so #61 and #63 are now duplicates. The four PRs form an exact 4-cycle over four files, each file claimed by exactly two PRs:

PR dist_git quoting validate gaps
#59 ● ●
#61 ● ●
#71 ● ●
#63 ● ●

Reading down the columns: #59 + #71 covers all four files with zero overlap, and so does #61 + #63. There is no third option, and taking any three of the four means a file gets added twice and the second one fails to merge.

So the remaining work is mechanical, and worth doing before either of these lands rather than after:

  1. #59 needs the same call — it is the other half of this pair and is still hold-labelled.
  2. Close #61 and #63 as duplicates, each with a pointer to where its coverage now lives: #61's test_validate.py → this PR, #61's test_check_workflow_quoting.py → #59, #63's test_hummingbird_gaps.py → this PR, #63's test_dist_git.py → #59.

One case is genuinely lost in that trade, and it is the single most valuable individual case across all four PRs: #63's guard that no [excluded] package leaks into the contract. reports/hummingbird-gap.json is consumed by projectbluefin/utah, so a leak there misreports coverage to a downstream repo rather than just failing locally. This PR's test_hummingbird_gaps.py covers the ISO-8601 measured_at and output-directory creation that #63 also has, but not that one. Worth lifting it across in a follow-up — it is about fifteen lines and does not need to block this.

Separately, and not blocking: #73 refactors tools/validate.py into check_provenance() / main(root) behind a __main__ guard. These tests drive the CLI as a subprocess rather than the internals, so they survive it — I confirmed that by merging #73 with #61's near-identical version (97 passed). Landing #73 first would let a later round call the functions directly instead of shelling out.

Last one for the quality agent rather than this PR: disjointness is being checked at filing time and not re-checked at review time, which is exactly how four PRs came to write four files twice each. Re-checking against open PRs when the review is requested would have caught this on the second filing.


Generated by Claude Code

castrojo added a commit that referenced this pull request Sep 13, 2026
…e import, and contract (#71, #72, #56, #57) (#92)

* test(quality): unit coverage for validate.py and recalculate_hummingbird_gaps.py

* fix(ci): verify cosign binary sha256 checksum in rebuild-rpms

Verify the downloaded cosign v2.4.1 binary against its release SHA-256
checksum prior to installation and keyless signing.

Closes #46
Closes #66
Closes #68

* fix(tools): validate package and branch inputs in import_rawhide (#56)

* fix(contract): exclude anaconda-live from runtime transaction (#57)

---------

Co-authored-by: castrojo <castrojo@users.noreply.github.com>
@castrojo castrojo closed this in #92 Sep 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm Approved by a Hive merger/owner for auto-merge on green CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] tools/validate.py and tools/recalculate_hummingbird_gaps.py have zero unit coverage

2 participants