Repository navigation
test: unit coverage for scripts/flavors.py flavor-matrix queries — new tests/unit/test_flavors.py, Justfile test recipe - #40
hivecommons-hive[bot] wants to merge 2 commits into
Conversation
…eries
utah has no test suite. scripts/flavors.py is the single source for the
build matrix, the kernel-cache decision, the promote matrix and the
release matrix, and just check exercises it with one line that discards
the output:
python3 scripts/flavors.py list >/dev/null
That proves the file parses. A wrong entry in IMAGE, an inverted
needs-kernel, or a list-main/list-kernel pair that no longer partitions
the set all pass it, and fail hours later in a different workflow.
Adds tests/unit/test_flavors.py: every query asserted against synthetic
config/flavors.json trees, so the assertions state the mapping instead
of restating today's config, plus a pass over the shipped config so the
real file stays inside the contract. Error paths (unknown flavor,
unknown query, malformed config) are covered too.
Adds a just test recipe and calls it from just check, so build.yml runs
it through the existing check step without a workflow change.
Refs #39
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
hanthor
left a comment
There was a problem hiding this comment.
Verified: applies clean to current main, 16 tests pass
Ran it rather than taking the description's word for it:
$ git -C utah rev-parse HEAD
636b48e3ebb30307961a2f398ded0e0a9c69ed50
$ git apply --check /tmp/pr40.diff && echo "APPLIES CLEAN"
APPLIES CLEAN
$ git apply /tmp/pr40.diff && python3 -m pytest tests/unit -q
................ [100%]
16 passed in 0.80sThe PR is based on f98fb409 but still applies without conflict to 636b48e, so no rebase is needed.
The critique of the existing coverage is accurate. just check on main contains exactly:
python3 scripts/flavors.py list >/dev/null
which discards the output, so it proves the file parses and nothing about what it answers — while scripts/flavors.py is in fact the single source read by four separate matrices (build.yml, promote-testing-to-main.yml, execute-release.yml, and the kernel-cache decision). Adding real assertions there is worth doing.
The mutation table in the description is the part that makes this reviewable — testing that each test actually fails when the mapping is wrong is the difference between a test suite and a set of assertions that restate the current config. Constructing synthetic config/flavors.json trees rather than asserting against the shipped one is also the right call, with the one test that does exercise the real config as a useful backstop.
Scope is clean: 2 files, +176, no production code touched, no workflow file edited. It gates PRs via the existing just check call in build.yml, which is the cheapest possible wiring.
Two things worth a look
1. pip install --quiet pytest adds a network dependency to just check. It follows the existing pyyaml/jsonschema pattern in the same recipe, so it is consistent — but those two are guarded:
pip install --quiet pyyaml 2>/dev/null || true
Worth confirming the new pytest install has the same || true treatment, or that a pytest-less environment degrades rather than hard-fails. scripts/flavors.py has no third-party imports, so these tests would run fine under stdlib unittest too — which would drop the dependency entirely. Not a blocker, just an option.
2. Coordinate with #44 and #64, which both extend scripts/flavors.py. Both add a count query and both add guards to the check recipe in the Justfile. I confirmed #44 and #64 already conflict with each other on Justfile:79:
$ git apply /tmp/pr64.diff && git apply --check /tmp/pr44.diff
error: patch failed: Justfile:79
error: Justfile: patch does not applyThis PR touches the check recipe too, so it is a third party to that. Merge order matters; whoever goes last rebases. If this PR lands first, the other two get free test coverage for count as soon as they add it.
Status
The failing checks are not from this PR — Build Utah is red on main@636b48e itself with the /usr/local/libexec/utah-install-packages failure at Containerfile:81 (see #55/#56), and every open PR inherits it. The contract job, which is what would actually exercise just check, passes.
This is the cleanest PR in the open queue: small, verified, no production code, no workflow edits, and it applies to main today. Only the hold label is holding it.
Generated by Claude Code
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com> # Conflicts: # Justfile
hanthor
left a comment
There was a problem hiding this comment.
The coverage is worth having; the packaging of it isn't the repo's convention, and it would fork the test layout.
Since this opened, the repo settled on tests/test_*.py using unittest, discovered by just check with python3 -m unittest discover -s tests -p 'test_*.py' — that's how test_package_resolution.py, test_desktop_contract.py (#96), test_script_syntax_gate.py (#91) and test_image_parity.py all run. This PR puts its file under tests/unit/ with pytest, adds a just test recipe that pip installs pytest, and wires that into check — so we'd have two runners, two directories, and the existing discover -s tests would never see tests/unit/ (no __init__.py, and -p 'test_*.py' only walks packages).
Ask: move it to tests/test_flavors.py as unittest.TestCases (the assertions port 1:1 — subTest covers the parametrised cases), drop the just test recipe and the pytest install. Then it runs under the existing gate with no Justfile change at all, which also removes the conflict with #44 and #91's Justfile edits.
Generated by Claude Code
|
Closing rather than waiting on the port: the maintainers' call is to clear PRs that need an author rewrite. The coverage is still wanted — a Generated by Claude Code |
Claimed ground
tests/unit/test_flavors.py(utah has notests/directory today)scripts/flavors.py— querieslist,images,releases,needs-kernel,list-main,list-kernelJustfile— newtestrecipe, one line added tocheckNo production code changed. No workflow file touched.
Test improvement
scripts/flavors.pyis the declared single source for four things that must agree — the build matrix (build.yml:58,63,64), the kernel-cache decision (build.yml:69), the promote matrix (promote-testing-to-main.yml:31) and the release matrix (execute-release.yml:24).just checkexercises all of that with one line:The output is discarded, so the check proves the file parses and nothing about what it answers.
16 tests added, covering:
listreturns the configured flavors in order, and is the default queryimagesmaps every flavor to its image name (main→utah,nvidia→utah-nvidia, …)releasesemitstesting→stablefor every flavor, in the same order asimagesneeds-kernelistrueonly when a non-mainflavor is present — parametrized over[],[main],[main,nvidia],[gaming],[nvidia-gaming]list-main/list-kernelpartition the set with no overlap, including whenmainitself is retiredconfig/flavors.jsononly names known flavors and stays self-consistent across all five queriesAssertions are made against synthetic
config/flavors.jsontrees (the script is copied beside a temp config, sinceCONFIGresolves from__file__), so they state the mapping rather than restate today's config. One test pass runs against the real shipped config.Verification
Mutation-checked — each of these was introduced, caught, and reverted:
"gaming": "utah-gaming"→"utah-game"test_images_maps_every_flavor_to_its_image_name,test_releases_promotes_testing_to_stable_for_every_flavorneeds-kernel:any(...)→all(...)test_needs_kernel_...[main,nvidia],test_shipped_config_agrees_across_all_querieslist-kernelreturns all flavorstest_list_main_and_list_kernel_partition_the_flavor_set,test_shipped_config_agrees_across_all_queriesCI wiring
just testis called fromjust check, whichbuild.yml:40already runs — so this gates PRs with no workflow-file edit.pip install --quiet pytestfollows the existingpyyaml/jsonschemapattern in the same recipe.Related issue
Closes #115 — that issue is exactly the wiring this PR adds:
tests/unit/is skipped by theunittest discoverline injust check, sojust testis what makes any test under it run. Merging this leaves nothing for #115 to track.Refs #39 — that issue covers utah's whole missing test suite (19 scripts). This PR covers
scripts/flavors.pyonly; the other 18 remain follow-up work, so the tracker stays open.Filed by quality agent (hold-gated mode). Human review required. Do not merge without review; the
holdlabel stays on.— hive: agent=quality backend=copilot model=claude-opus-5