Skip to content

test: unit coverage for scripts/flavors.py flavor-matrix queries — new tests/unit/test_flavors.py, Justfile test recipe - #40

Closed
hivecommons-hive[bot] wants to merge 2 commits into
mainfrom
quality/test-flavors-cli
Closed

hivecommons-hive[bot] wants to merge 2 commits into
mainfrom
quality/test-flavors-cli

Conversation

@hivecommons-hive

@hivecommons-hive hivecommons-hive Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Claimed ground

  • New file: tests/unit/test_flavors.py (utah has no tests/ directory today)
  • Script under test: scripts/flavors.py — queries list, images, releases, needs-kernel, list-main, list-kernel
  • Edited file: Justfile — new test recipe, one line added to check
  • Cluster: utah#39 (utah test-suite bootstrap / flavor-matrix coverage)

No production code changed. No workflow file touched.

Test improvement

scripts/flavors.py is the declared single source for four things that must agree — the build matrix (build.yml:58,63,64), the kernel-cache decision (build.yml:69), the promote matrix (promote-testing-to-main.yml:31) and the release matrix (execute-release.yml:24).

just check exercises all of that with one line:

python3 scripts/flavors.py list >/dev/null

The output is discarded, so the check proves the file parses and nothing about what it answers.

16 tests added, covering:

  • list returns the configured flavors in order, and is the default query
  • images maps every flavor to its image name (main→utah, nvidia→utah-nvidia, …)
  • releases emits testing→stable for every flavor, in the same order as images
  • needs-kernel is true only when a non-main flavor is present — parametrized over [], [main], [main,nvidia], [gaming], [nvidia-gaming]
  • list-main / list-kernel partition the set with no overlap, including when main itself is retired
  • unknown flavor in config fails every query, naming the offender
  • unknown query name exits non-zero and prints nothing on stdout
  • malformed JSON config fails loudly
  • the shipped config/flavors.json only names known flavors and stays self-consistent across all five queries

Assertions are made against synthetic config/flavors.json trees (the script is copied beside a temp config, since CONFIG resolves from __file__), so they state the mapping rather than restate today's config. One test pass runs against the real shipped config.

Verification

$ python3 -m pytest tests/unit -q
16 passed

Mutation-checked — each of these was introduced, caught, and reverted:

Mutation Tests that failed
"gaming": "utah-gaming" → "utah-game" test_images_maps_every_flavor_to_its_image_name, test_releases_promotes_testing_to_stable_for_every_flavor
needs-kernel: any(...) → all(...) test_needs_kernel_...[main,nvidia], test_shipped_config_agrees_across_all_queries
list-kernel returns all flavors test_list_main_and_list_kernel_partition_the_flavor_set, test_shipped_config_agrees_across_all_queries

CI wiring

just test is called from just check, which build.yml:40 already runs — so this gates PRs with no workflow-file edit. pip install --quiet pytest follows the existing pyyaml/jsonschema pattern in the same recipe.

Related issue

Closes #115 — that issue is exactly the wiring this PR adds: tests/unit/ is skipped by the unittest discover line in just check, so just test is what makes any test under it run. Merging this leaves nothing for #115 to track.

Refs #39 — that issue covers utah's whole missing test suite (19 scripts). This PR covers scripts/flavors.py only; the other 18 remain follow-up work, so the tracker stays open.


Filed by quality agent (hold-gated mode). Human review required. Do not merge without review; the hold label stays on.

— hive: agent=quality backend=copilot model=claude-opus-5

…eries

utah has no test suite. scripts/flavors.py is the single source for the
build matrix, the kernel-cache decision, the promote matrix and the
release matrix, and just check exercises it with one line that discards
the output:

    python3 scripts/flavors.py list >/dev/null

That proves the file parses. A wrong entry in IMAGE, an inverted
needs-kernel, or a list-main/list-kernel pair that no longer partitions
the set all pass it, and fail hours later in a different workflow.

Adds tests/unit/test_flavors.py: every query asserted against synthetic
config/flavors.json trees, so the assertions state the mapping instead
of restating today's config, plus a pass over the shipped config so the
real file stays inside the contract. Error paths (unknown flavor,
unknown query, malformed config) are covered too.

Adds a just test recipe and calls it from just check, so build.yml runs
it through the existing check step without a workflow change.

Refs #39

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added hold Indicates that a PR should not merge because someone has issued a /hold command. quality agent/quality Filed or owned by the quality agent. labels Sep 6, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified: applies clean to current main, 16 tests pass

Ran it rather than taking the description's word for it:

$ git -C utah rev-parse HEAD
636b48e3ebb30307961a2f398ded0e0a9c69ed50
$ git apply --check /tmp/pr40.diff && echo "APPLIES CLEAN"
APPLIES CLEAN
$ git apply /tmp/pr40.diff && python3 -m pytest tests/unit -q
................                                                         [100%]
16 passed in 0.80s

The PR is based on f98fb409 but still applies without conflict to 636b48e, so no rebase is needed.

The critique of the existing coverage is accurate. just check on main contains exactly:

python3 scripts/flavors.py list >/dev/null

which discards the output, so it proves the file parses and nothing about what it answers — while scripts/flavors.py is in fact the single source read by four separate matrices (build.yml, promote-testing-to-main.yml, execute-release.yml, and the kernel-cache decision). Adding real assertions there is worth doing.

The mutation table in the description is the part that makes this reviewable — testing that each test actually fails when the mapping is wrong is the difference between a test suite and a set of assertions that restate the current config. Constructing synthetic config/flavors.json trees rather than asserting against the shipped one is also the right call, with the one test that does exercise the real config as a useful backstop.

Scope is clean: 2 files, +176, no production code touched, no workflow file edited. It gates PRs via the existing just check call in build.yml, which is the cheapest possible wiring.

Two things worth a look

1. pip install --quiet pytest adds a network dependency to just check. It follows the existing pyyaml/jsonschema pattern in the same recipe, so it is consistent — but those two are guarded:

pip install --quiet pyyaml 2>/dev/null || true

Worth confirming the new pytest install has the same || true treatment, or that a pytest-less environment degrades rather than hard-fails. scripts/flavors.py has no third-party imports, so these tests would run fine under stdlib unittest too — which would drop the dependency entirely. Not a blocker, just an option.

2. Coordinate with #44 and #64, which both extend scripts/flavors.py. Both add a count query and both add guards to the check recipe in the Justfile. I confirmed #44 and #64 already conflict with each other on Justfile:79:

$ git apply /tmp/pr64.diff && git apply --check /tmp/pr44.diff
error: patch failed: Justfile:79
error: Justfile: patch does not apply

This PR touches the check recipe too, so it is a third party to that. Merge order matters; whoever goes last rebases. If this PR lands first, the other two get free test coverage for count as soon as they add it.

Status

The failing checks are not from this PR — Build Utah is red on main@636b48e itself with the /usr/local/libexec/utah-install-packages failure at Containerfile:81 (see #55/#56), and every open PR inherits it. The contract job, which is what would actually exercise just check, passes.

This is the cleanest PR in the open queue: small, verified, no production code, no workflow edits, and it applies to main today. Only the hold label is holding it.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot changed the title [quality] test: unit coverage for scripts/flavors.py flavor-matrix queries — new tests/unit/test_flavors.py, Justfile test recipe test: unit coverage for scripts/flavors.py flavor-matrix queries — new tests/unit/test_flavors.py, Justfile test recipe Sep 17, 2026
@hivecommons-hive hivecommons-hive Bot added the agent/scanner Filed or owned by the scanner agent. label Sep 17, 2026
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

# Conflicts:
#	Justfile

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The coverage is worth having; the packaging of it isn't the repo's convention, and it would fork the test layout.

Since this opened, the repo settled on tests/test_*.py using unittest, discovered by just check with python3 -m unittest discover -s tests -p 'test_*.py' — that's how test_package_resolution.py, test_desktop_contract.py (#96), test_script_syntax_gate.py (#91) and test_image_parity.py all run. This PR puts its file under tests/unit/ with pytest, adds a just test recipe that pip installs pytest, and wires that into check — so we'd have two runners, two directories, and the existing discover -s tests would never see tests/unit/ (no __init__.py, and -p 'test_*.py' only walks packages).

Ask: move it to tests/test_flavors.py as unittest.TestCases (the assertions port 1:1 — subTest covers the parametrised cases), drop the just test recipe and the pytest install. Then it runs under the existing gate with no Justfile change at all, which also removes the conflict with #44 and #91's Justfile edits.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing rather than waiting on the port: the maintainers' call is to clear PRs that need an author rewrite. The coverage is still wanted — a tests/test_flavors.py in the repo's unittest layout would be merged on sight.


Generated by Claude Code

@hanthor hanthor closed this Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent/quality Filed or owned by the quality agent. agent/scanner Filed or owned by the scanner agent. hold Indicates that a PR should not merge because someone has issued a /hold command.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[quality] tests/unit/ is never executed — unittest discover in just check skips it silently

1 participant