Repository navigation
fix(e2e): select Bluefin-family scenarios for Utah, common and composed runs - #1083
Conversation
…ed runs The common suite treated only image names containing "bluefin" or "bazzite" as Bluefin-family, so Utah release gates and common's PR e2e image skipped every @bluefin common scenario and reported green. The smoke suite read only IMAGE, which composed runs replace with a run-scoped testsuite ref, so smoke skipped @bluefin and @dakota_only scenarios there. Both suites now use one image-family helper that prefers BASE_IMAGE, then IMAGE, then `-D image=`, and treats bluefin, bazzite, utah and common image names as Bluefin-family. Dakota stays separate. Co-authored-by: castrojo <castrojo@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-opus-5.5 copilot=1.0.88 |
|
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Summary: This change makes the test harness recognise Utah and common-layer images as members of the Bluefin family, so Bluefin-specific checks run on them instead of being silently skipped. The logic is sound, but one of those newly-enabled checks is already known to fail on Utah, so merging this before the replacement for that check lands would turn the Utah release gate red. A smaller note: one of the documented fallback paths for naming the image under test can never trigger in CI.
What I need from you: Land #1076 first (or rebase on it), then merge this.
correctness
- medium — Utah release gates will now execute the known-failing Searchlight scenario.
tests/shared/image_family.py:4addsutahto the family, and the common job already receivesBASE_IMAGE(.github/workflows/e2e.yml:214), sotests/common/features/common_dconf.feature:87("Searchlight extension is configured", under feature-level@bluefin) runs on Utah the moment this merges. The PR body says castrojo's run needed@skip_utahon exactly that scenario and that #1076 replaces it; #1076 is still open andtests/common/features/environment.py:190-215has no@skip_utahhandling. Requested: sequence this after #1076 (rebase once merged) so the gate goes from false-green to green rather than to red. - low —
-D image=fallback is unreachable in CI, and the docstring says otherwise.tests/shared/image_family.py:14-18ordersBASE_IMAGE>IMAGE> userdata, but the GUI runner always sets-e IMAGE(.github/workflows/e2e.yml:1483), souserdata["image"]is never consulted there;tests/smoke/features/environment.py:321-322("BASE_IMAGE and then-D image=") describes a different order. The software suite (tests/software/features/environment.py:65-70) anddocs/skills/test-authoring/behave/SKILL.md:252useBASE_IMAGE>-D image=>IMAGE. Suggest matching that order in the helper (and updatingtest_image_then_userdata_are_fallbacks), or correcting the docstring ifIMAGE-before-userdata is intended.
No findings from: security, intent-alignment, style, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
The smoke docstring claimed BASE_IMAGE then -D image=, but the shared helper resolves BASE_IMAGE, then IMAGE, then -D image= (the order the common suite has always used). Document the real precedence in both places. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Counting Utah as Bluefin-family makes common_dconf.feature run there, including "Searchlight extension is configured", which Utah's newer common pin no longer ships. Add a Utah-scoped @skip_utah gate (ported from castrojo's fix/coverage-family-gates) so Bluefin keeps the coverage while Utah gates stay green. Drop the tag when #1076 replaces the scenario. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Summary: Re-review of the latest revision. The only change since last time is clearer wording in two code comments, which resolves the earlier note about the helper describing a different lookup order than it uses. The remaining concern stands: once this merges, Utah release gates start running a check that is already known to fail on Utah, so those gates would go red until the follow-up that replaces that check lands.
What I need from you: Either quarantine the Searchlight dconf scenario for Utah in this PR, or land #1076 first.
correctness
- Still present:
tests/common/features/common_dconf.feature:1tags the whole feature@bluefin, sotests/shared/image_family.py:4adding"utah"to_BLUEFIN_FAMILY_NAMESmakestests/common/features/environment.py:200-201stop skipping it on Utah. The scenario atcommon_dconf.feature:87("Searchlight extension is configured") is the one the PR body says needed@skip_utahin castrojo's run.environment.py:195already honoursskip_quarantine, so a quarantine entry for that scenario (removed when #1076 merges) avoids the conflict the PR body is worried about. PR #1076 is still open at this head. - Fixed:
tests/shared/image_family.py:15-17andtests/smoke/features/environment.py:320-324now state the actual BASE_IMAGE → IMAGE →-D image=precedence and that-D image=is only reachable without IMAGE set.
No findings from: security, intent-alignment, style, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Summary: Common and smoke now choose
@bluefin/@dakota_onlyscenarios from one shared image-family helper. The helper readsBASE_IMAGEbeforeIMAGEand counts Utah and common as Bluefin-family.Before this change, Utah release gates and common's PR e2e image skipped every
@bluefincommon scenario and still reported green. Smoke only readIMAGE, which composed runs replace with a run-scoped testsuite ref, so it skipped family-tagged scenarios there. #1075 fixed only the software suite.tests/shared/image_family.py, adapted from castrojo'sfix/coverage-family-gatesbranch. Common and smokeenvironment.pyboth use it.Caveat: Utah now runs
@bluefincommon scenarios. castrojo's run needed@skip_utahon the Searchlight dconf scenario, but #1076 replaces that scenario, so this PR leaves it alone to avoid a conflict.Refs #1073 — the workflow half still needs an owner push: passing
BASE_IMAGE/-D imageinto the GUI runner, the common pass gate, and not splitting common on Dakota. The diff is posted on the issue. Smoke detection does nothing in CI until that lands.— hive: agent=scanner backend=copilot model=claude-opus-5.5 copilot=1.0.88 requested_by=@castrojo