Skip to content

test: coverage for scripts/check-download-integrity.py and scripts/check_workflow_outputs.py — new tests/unit/test_ci_guards.py - #80

Closed
hivecommons-hive[bot] wants to merge 2 commits into
mainfrom
quality/test-ci-guards
Closed

hivecommons-hive[bot] wants to merge 2 commits into
mainfrom
quality/test-ci-guards

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Claimed ground

  • New file: tests/unit/test_ci_guards.py — the only file this PR adds or edits
  • Scripts under test: scripts/check-download-integrity.py (check() / is_allowed() / main()) and scripts/check_workflow_outputs.py (module-level guard)
  • Cluster: utah#79 — the two just check CI guard scripts

Checked against every open utah PR: #40 claims scripts/flavors.py + a Justfile test recipe, #44 and #64 claim flavors.py, #74 claims scripts/check-repo-availability.py. None of them touch either script covered here, and this PR edits no Justfile, no workflow, and no production code.

Test improvement

Both scripts are pure-verdict gates run on every PR via just check (Justfile:62, Justfile:83) and neither had a single test. Nothing asserted they still say no:

  • check-download-integrity.py is a supply-chain control — it rejects mutable releases/latest/download URLs and unverified executable fetches in the 7 committed composition recipes. A dropped suffix in its regex or a widened allowlist would let an unverified executable into a shipped image with just check still green.
  • check_workflow_outputs.py exists because a job output wired to a missing step id once made the image build silently produce no jobs at all.

35 black-box tests, each running the real script in a synthetic repository tree (both scripts resolve inputs relative to cwd, so a tmp_path tree is enough) and asserting exit status plus the reported location.

Download guard:

  • every one of the 7 inspected recipe paths rejects a mutable latest release, with path:line
  • every guarded executable suffix (.run, .tar.gz, .tgz, .rpm, .flatpak) and both curl and wget
  • each accepted verifier token (sha256sum, sha512sum, --check, cosign, gpg --verify) clears that file's downloads
  • clearance is per file, not repository-wide — a verified install-nvidia.sh does not excuse an unverified Containerfile
  • comment lines are ignored; the Flathub flatpakrepo/appstream descriptors stay allowed without a digest
  • a non-executable download (config.json) is not flagged
  • a mutable latest is still reported even in a file that verifies digests
  • empty tree passes and reports checked 7 build recipes

Workflow guard:

  • an output wired to a defined step passes; one reading an undefined step id fails, naming job, output and step id
  • step ids do not leak across jobs in the same workflow
  • jobs with no outputs block are skipped even when a step body mentions steps.*.outputs
  • outputs built from inputs.* / github.* are accepted
  • every dangling reference in every workflow file is reported, not just the first
  • a tree with no .github/workflows passes

One test per script runs it against the shipped tree, so a real regression in either fails here too.

Verification

$ python3 -m pytest tests/unit -q
35 passed

Mutation-checked — each introduced, caught, reverted:

Mutation Tests that failed
drop flatpak from the guarded-suffix regex test_unverified_executable_download_is_rejected_for_every_guarded_suffix[flatpak]
verifier lookup token in text → token in stripped (per-line instead of per-file) 6 tests, incl. test_verification_is_per_file_not_repository_wide and test_shipped_build_recipes_pass_the_integrity_guard
defined step ids collected across all jobs instead of the current job test_step_ids_do_not_leak_across_jobs

CI wiring

None added, deliberately — utah has no tests/ directory yet, and #40 already proposes the just test recipe (pytest tests/unit) called from just check. These tests sit under tests/unit/ and are collected by that recipe unchanged, so this PR stays a pure test addition with no Justfile conflict. If #40 is closed rather than merged, wiring for the whole suite should be added once, in one place, not twice.

Related issue

Closes #79.


Filed by quality agent (hold-gated mode). Human review required. Do not merge without review; the hold label stays on.

— hive: agent=quality backend=copilot model=claude-opus-5 copilot=1.0.78

…ty.py and scripts/check_workflow_outputs.py

Both scripts are gates run by 'just check' (Justfile:62 and Justfile:83) and
had zero tests. The supply-chain guard's rejection of mutable
releases/latest URLs and unverified executable downloads, and the workflow
guard's detection of job outputs reading undefined step ids, were asserted by
nothing.

Adds tests/unit/test_ci_guards.py: 35 black-box tests that run each real
script against synthetic repository trees, plus one pass over the shipped
tree for each.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added hold Indicates that a PR should not merge because someone has issued a /hold command. quality agent/quality Filed or owned by the quality agent. labels Sep 14, 2026
@hivecommons-hive hivecommons-hive Bot changed the title [quality] test: coverage for scripts/check-download-integrity.py and scripts/check_workflow_outputs.py — new tests/unit/test_ci_guards.py test: coverage for scripts/check-download-integrity.py and scripts/check_workflow_outputs.py — new tests/unit/test_ci_guards.py Sep 17, 2026
@hivecommons-hive hivecommons-hive Bot added the agent/scanner Filed or owned by the scanner agent. label Sep 17, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same note as on #40, since this is the same shape: the repo's tests live in tests/test_*.py as unittest cases and are discovered by just check with python3 -m unittest discover -s tests -p 'test_*.py'. A tests/unit/test_ci_guards.py written for pytest is never collected by that gate — nothing in just check runs it today — and adding pytest as a second runner for one file isn't worth the split.

The content is good: the INSPECTED list assertion and the mutable-releases/latest cases are exactly what check-download-integrity.py lacked, and #93's review found a real hole in that script that these tests would have caught. Please port it to tests/test_ci_guards.py as unittest.TestCases (tmp_path → tempfile.TemporaryDirectory, parametrised cases → subTest) so it runs under the existing gate.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing rather than waiting on the port (see the review above). Still wanted as tests/test_ci_guards.py in the repo's unittest layout — and #93's review shows the whole-file verified hole in check-download-integrity.py that these tests should pin.


Generated by Claude Code

@hanthor hanthor closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Filed or owned by the quality agent. agent/scanner Filed or owned by the scanner agent. hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] CI guard scripts check-download-integrity.py and check_workflow_outputs.py have zero tests

1 participant