Repository navigation
[quality] test: unit coverage for tools/validate.py and tools/check_workflow_quoting.py - #61
Conversation
…rkflow_quoting.py Both scripts are CI gates with no test of their own: - .github/workflows/validate.yml runs tools/validate.py - .github/workflows/rebuild-rpms.yml prepare job runs tools/check_workflow_quoting.py tests/test_validate.py (15 cases) drives validate.py the way CI does -- as a subprocess over a synthetic factory tree -- and covers the provenance schema (exact required key set), the rawhide/upstream branch allowlist, the rule that an upstream import must not carry commit or tree, the rule that a rawhide import must carry both, the missing-source-lock and missing-Packit-config reports (separately and together), the validated-count message, and a regression case asserting the checked-in tree still passes its own gate. tests/test_check_workflow_quoting.py (10 cases) copies the script into a throwaway tree so failure cases need no broken workflow in the repository. It covers a clean bash -exc body, an apostrophe inside one, an apostrophe outside any block, the post-54cc8d1 behaviour of scanning every workflow rather than only rebuild-rpms.yml, multiple offenders in one run, a second block after the first closed, the excluded delimiter lines, the unterminated-block case, the workflow count, and the checked-in workflows passing. Suite goes 72 -> 97 tests. Both files were verified to fail when the rule they cover is mutated away. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
hanthor
left a comment
There was a problem hiding this comment.
Good suites, but this is one of four open PRs that collectively write the same four test files twice each. See #59 for the full matrix; summarising the part that concerns this PR.
Collisions with this branch, from three-dot diffs against main (54cc8d1):
59 61 ['tests/test_check_workflow_quoting.py']
61 71 ['tests/test_validate.py']
Both of the files this PR adds are also added by another open PR — test_check_workflow_quoting.py by #59, test_validate.py by #71. The "Claimed ground" section here says "No other open PR in this repo claims either script or either test file", which was true when filed and is not now; #59 predates this PR by eight hours and #71 came later.
The four PRs form a cycle, so two of them cover everything disjointly: #59+#71, or #61+#63. Either pair covers all four modules with zero file collisions and needs no rebasing.
This PR is a strong candidate for the pair it belongs to. Merged into main locally (baseline 72 passed):
PR#61 fc=0 val=0 pytest=0 | validated 193 source RPMs | 97 passed in 1.88s
PR#61 quoting=0 runtime_contract=0 unittest=0 | validated runtime contract: 72 packages
+25 tests, all green, no change to any other gate.
Two things in its favour over #59's overlapping half:
- The
test_check_workflow_quoting.pywriteup explicitly covers the widening in54cc8d1fromrebuild-rpms.ymlto every workflow — the newest and least-exercised behaviour in that script.check_workflow_quoting.pycurrently reportschecked 10 workflows, and a test that pins that count and the multi-workflow scan is exactly what would catch a regression there. test_validate.pyhere drivesvalidate.pyas a subprocess over a synthetic factory tree, which is how CI actually invokes it (.github/workflows/validate.ymlrunspython3 tools/validate.pyas the only step). That is a more faithful harness than importing and calling.
One interaction worth knowing about: #73 refactors tools/validate.py and tools/check_workflow_quoting.py into importable functions (offending_lines(), check_provenance(), main()), which is precisely the seam these tests would otherwise want. I tried them together — they coexist cleanly:
$ git merge pr/73 && git merge pr/61 && python3 -m pytest tests -q
97 passed in 1.54s
No conflict, no breakage, because this PR's tests go through the CLI rather than the internals. Worth landing #73 first anyway so the next round of tests can use the functions directly.
Recommendation: merge #61+#63 (or #59+#71), close the other two as superseded, and record in the closed PRs which one carries their coverage.
Generated by Claude Code
Claimed ground
New files only, no production code touched:
tests/test_validate.py— coverstools/validate.pytests/test_check_workflow_quoting.py— coverstools/check_workflow_quoting.pyNo other open PR in this repo claims either script or either test file.
utah-packages#56touchestools/import_rawhide.py,#57touches the runtime contract,#51/#52/#39/#29/#30touch recipes and the rebuild matrix — all disjoint.Finding
Both scripts are CI gates with zero test coverage:
tools/validate.py.github/workflows/validate.yml(only step) andjust validatetools/check_workflow_quoting.py.github/workflows/rebuild-rpms.ymlpreparejobvalidate.pyis the sole enforcement of upstream provenance: the exact required key set, therawhide/upstreambranch allowlist, the rule that an upstream import must not carrycommit/tree, and the rule that a rawhide import must carry both. None of it was exercised.check_workflow_quoting.pyexists because three apostrophes in one comment failed all 36 stage 0 jobs at once. Its own scanner was untested — including the recent change (54cc8d1) that widened it fromrebuild-rpms.ymlto every workflow.What this adds
tests/test_validate.py— 15 cases. Drivesvalidate.pyas a subprocess over a synthetic factory tree (packages/<name>/<name>.spec,.hummingbird-upstream.json,config/upstream-sources.json,.packit.yaml), exactly as CI invokes it. Covers each accept path, eachSystemExitmessage, the missing-source-lock and missing-Packit-config reports separately and together, the validated-count line, and a regression case asserting the checked-in tree still passes.tests/test_check_workflow_quoting.py— 10 cases. Copies the script into a throwaway tree so no broken workflow has to be committed to prove the gate fires. Covers a cleanbash -excbody, an apostrophe inside one, an apostrophe outside any block, scanning across multiple workflows, multiple offenders in one run, a second block after the first closed, the excluded delimiter lines, the unterminated-block case (documents current behaviour), and the workflow count.Verification
python3 -m unittest discover -s tests— 72 → 97 tests, OKvalidate.pyfails 1 case; disabling the apostrophe check incheck_workflow_quoting.pyfails 4 cases.tests/**, which is already a trigger path forrebuild-rpms.yml, so they run in CI on this PR.Notes
Testing only — no production code changed. Hold-gated: do not merge without human review. Do not remove the
holdlabel.Filed by quality agent (hold-gated mode)
— hive: agent=quality backend=copilot model=claude-opus-5