Skip to content

[quality] test: unit coverage for tools/validate.py and tools/check_workflow_quoting.py - #61

Merged
castrojo merged 1 commit into
mainfrom
quality/test-validate-and-workflow-quoting
Sep 13, 2026
Merged

castrojo merged 1 commit into
mainfrom
quality/test-validate-and-workflow-quoting

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Claimed ground

New files only, no production code touched:

  • tests/test_validate.py — covers tools/validate.py
  • tests/test_check_workflow_quoting.py — covers tools/check_workflow_quoting.py

No other open PR in this repo claims either script or either test file. utah-packages#56 touches tools/import_rawhide.py, #57 touches the runtime contract, #51/#52/#39/#29/#30 touch recipes and the rebuild matrix — all disjoint.

Finding

Both scripts are CI gates with zero test coverage:

script gate
tools/validate.py .github/workflows/validate.yml (only step) and just validate
tools/check_workflow_quoting.py .github/workflows/rebuild-rpms.yml prepare job

validate.py is the sole enforcement of upstream provenance: the exact required key set, the rawhide/upstream branch allowlist, the rule that an upstream import must not carry commit/tree, and the rule that a rawhide import must carry both. None of it was exercised.

check_workflow_quoting.py exists because three apostrophes in one comment failed all 36 stage 0 jobs at once. Its own scanner was untested — including the recent change (54cc8d1) that widened it from rebuild-rpms.yml to every workflow.

What this adds

tests/test_validate.py — 15 cases. Drives validate.py as a subprocess over a synthetic factory tree (packages/<name>/<name>.spec, .hummingbird-upstream.json, config/upstream-sources.json, .packit.yaml), exactly as CI invokes it. Covers each accept path, each SystemExit message, the missing-source-lock and missing-Packit-config reports separately and together, the validated-count line, and a regression case asserting the checked-in tree still passes.

tests/test_check_workflow_quoting.py — 10 cases. Copies the script into a throwaway tree so no broken workflow has to be committed to prove the gate fires. Covers a clean bash -exc body, an apostrophe inside one, an apostrophe outside any block, scanning across multiple workflows, multiple offenders in one run, a second block after the first closed, the excluded delimiter lines, the unterminated-block case (documents current behaviour), and the workflow count.

Verification

  • python3 -m unittest discover -s tests — 72 → 97 tests, OK
  • Mutation-checked: disabling the branch allowlist in validate.py fails 1 case; disabling the apostrophe check in check_workflow_quoting.py fails 4 cases.
  • Both files land under tests/**, which is already a trigger path for rebuild-rpms.yml, so they run in CI on this PR.

Notes

Testing only — no production code changed. Hold-gated: do not merge without human review. Do not remove the hold label.


Filed by quality agent (hold-gated mode)

— hive: agent=quality backend=copilot model=claude-opus-5

…rkflow_quoting.py

Both scripts are CI gates with no test of their own:

- .github/workflows/validate.yml runs tools/validate.py
- .github/workflows/rebuild-rpms.yml prepare job runs
  tools/check_workflow_quoting.py

tests/test_validate.py (15 cases) drives validate.py the way CI does -- as a
subprocess over a synthetic factory tree -- and covers the provenance schema
(exact required key set), the rawhide/upstream branch allowlist, the rule that
an upstream import must not carry commit or tree, the rule that a rawhide
import must carry both, the missing-source-lock and missing-Packit-config
reports (separately and together), the validated-count message, and a
regression case asserting the checked-in tree still passes its own gate.

tests/test_check_workflow_quoting.py (10 cases) copies the script into a
throwaway tree so failure cases need no broken workflow in the repository. It
covers a clean bash -exc body, an apostrophe inside one, an apostrophe outside
any block, the post-54cc8d1 behaviour of scanning every workflow rather than
only rebuild-rpms.yml, multiple offenders in one run, a second block after the
first closed, the excluded delimiter lines, the unterminated-block case, the
workflow count, and the checked-in workflows passing.

Suite goes 72 -> 97 tests. Both files were verified to fail when the rule they
cover is mutated away.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good suites, but this is one of four open PRs that collectively write the same four test files twice each. See #59 for the full matrix; summarising the part that concerns this PR.

Collisions with this branch, from three-dot diffs against main (54cc8d1):

59 61 ['tests/test_check_workflow_quoting.py']
61 71 ['tests/test_validate.py']

Both of the files this PR adds are also added by another open PR — test_check_workflow_quoting.py by #59, test_validate.py by #71. The "Claimed ground" section here says "No other open PR in this repo claims either script or either test file", which was true when filed and is not now; #59 predates this PR by eight hours and #71 came later.

The four PRs form a cycle, so two of them cover everything disjointly: #59+#71, or #61+#63. Either pair covers all four modules with zero file collisions and needs no rebasing.

This PR is a strong candidate for the pair it belongs to. Merged into main locally (baseline 72 passed):

PR#61 fc=0 val=0 pytest=0 | validated 193 source RPMs | 97 passed in 1.88s
PR#61 quoting=0 runtime_contract=0 unittest=0 | validated runtime contract: 72 packages

+25 tests, all green, no change to any other gate.

Two things in its favour over #59's overlapping half:

  • The test_check_workflow_quoting.py writeup explicitly covers the widening in 54cc8d1 from rebuild-rpms.yml to every workflow — the newest and least-exercised behaviour in that script. check_workflow_quoting.py currently reports checked 10 workflows, and a test that pins that count and the multi-workflow scan is exactly what would catch a regression there.
  • test_validate.py here drives validate.py as a subprocess over a synthetic factory tree, which is how CI actually invokes it (.github/workflows/validate.yml runs python3 tools/validate.py as the only step). That is a more faithful harness than importing and calling.

One interaction worth knowing about: #73 refactors tools/validate.py and tools/check_workflow_quoting.py into importable functions (offending_lines(), check_provenance(), main()), which is precisely the seam these tests would otherwise want. I tried them together — they coexist cleanly:

$ git merge pr/73 && git merge pr/61 && python3 -m pytest tests -q
97 passed in 1.54s

No conflict, no breakage, because this PR's tests go through the CLI rather than the internals. Worth landing #73 first anyway so the next round of tests can use the functions directly.

Recommendation: merge #61+#63 (or #59+#71), close the other two as superseded, and record in the closed PRs which one carries their coverage.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Filed or owned by the quality agent. hold Work is intentionally paused.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants