Repository navigation
feat(ci): list which real test-machine runs a change needs before it ships - #1042
Conversation
Add scripts/vm_validation_plan.py, which maps a PR's changed paths to the minimal representative e2e.yml suites (GUI smoke, SSH common, bootc lifecycle, plus suite-specific lanes) that must boot against a real candidate image before v1 distributes the change. Docs, unit tests and non-distributed tooling need no boot; unknown paths fail closed to the full representative set; suites with no hosted lane are reported instead of dropped. Document the policy, how to run it against the PR's own workflow and test revision, and the audited state of existing signals (no commit statuses on main; no check run boots a VM). Wiring it into required checks or gating update-v1-tag.yml is left for maintainer approval. Refs projectbluefin#1027 Hive-Run: projectbluefin#1027 Hive-Plan: projectbluefin#1021 rollout plan — work package projectbluefin#1027 Hive-Spec: testsuite#1027#acceptance-1-change-path-policy Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: codebymarshall <codebymarshall@users.noreply.github.com>
There was a problem hiding this comment.
Summary: This change adds a small tool that reads which files a pull request touched and says which real test-machine runs must happen before the change is shipped to other projects. Two things need fixing before it lands: for changes to the shared "run GNOME tests" action, the tool points at machine runs that never execute that action, so the check would pass without testing the thing that changed; and the pull request is marked to close the design issue even though it only completes one of six items.
What I need from you: Apply the two fixes below (or say why not).
correctness
Composite-action and extension-gate changes map to e2e.yml lanes that never execute them (medium, in-scope). scripts/vm_validation_plan.py:98 maps .github/actions/gnome-e2e/* to (GUI, "extensions") and :106 maps scripts/extension_validation.py to ("extensions",); plan() at :135 then emits those as e2e.yml suites. But e2e.yml at head never references the action (no gnome-e2e anywhere in .github/workflows/e2e.yml), and its sparse-checkout (e2e.yml:235-243) does not include scripts/extension_validation.py; that script runs only inside the action (.github/actions/gnome-e2e/action.yml:23-25), which consumers call directly (action.yml:17). So the plan for an action.yml or extension-gate change says "boot smoke/extensions via e2e.yml", which exercises none of the changed code — the opposite of the fail-closed intent, and acceptance item 1 of #1027 names composite-action changes explicitly. (HOSTED_SUITES:38-52 also lists extensions, which e2e.yml:29 does not document as an input value; it runs generically via tests/extensions, but without the validation gate.)
Proposed fix — report the action as its own boot path instead of folding it into e2e.yml suites:
--- a/scripts/vm_validation_plan.py
+++ b/scripts/vm_validation_plan.py
@@
LIFECYCLE = "lifecycle"
FULL = (GUI, SSH, LIFECYCLE)
+# The gnome-e2e composite action is a separate boot path from e2e.yml:
+# consumers call it directly, and only it runs scripts/extension_validation.py.
+ACTION = "gnome-e2e"
@@
- (".github/actions/gnome-e2e/*", (GUI, "extensions")),
+ (".github/actions/gnome-e2e/*", (ACTION, GUI, "extensions")),
@@
- ("scripts/extension_validation.py", ("extensions",)),
+ ("scripts/extension_validation.py", (ACTION, "extensions")),
@@ def plan(paths: list[str]) -> dict:
return {
"vm_required": bool(required),
+ "composite_action": ACTION in required,
"suites": sorted(required & HOSTED_SUITES),
- "no_hosted_lane": sorted(required - HOSTED_SUITES),
+ "no_hosted_lane": sorted(required - HOSTED_SUITES - {ACTION}),
"reasons": reasons,
}Then update test_every_mapped_suite_is_known / test_composite_action_change_requires_gui_lanes in tests/unit/test_vm_validation_plan.py:75-77,112-121, the docstring at scripts/vm_validation_plan.py:19-20, and the policy table row at docs/skills/ci-ops/ops/references/candidate-vm-validation.md:35 to say the action must be dispatched directly (a workflow calling ./.github/actions/gnome-e2e with suite: extensions).
intent-alignment
Closes #1027 will close a six-item design issue when one item lands (medium, in-scope). The PR body says "Refs #1027. Partial: this covers acceptance item 1" and lists the remaining five items as still open, yet the footer carries Closes #1027, which GitHub honours on merge to main. Change the footer to Refs #1027 so the issue stays open for the remaining acceptance items.
No findings from: security, style, docs-currency.
Confidence: 3/5 (needs attention) — 2 medium findings
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@codebymarshall this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity. |
|
@codebymarshall the hive has adopted this PR ( |
e2e.yml never calls the gnome-e2e composite action and its sparse checkout excludes scripts/extension_validation.py, so mapping those paths to smoke/extensions prescribed runs that exercise none of the changed code. Map them to a gnome-e2e-action pseudo-lane reported under a new composite_action key, excluded from suites and no_hosted_lane. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
The hive adopted this PR ( Blocker: 1 commit(s) on Moving to |
There was a problem hiding this comment.
Summary: This change adds a small tool that tells a contributor which real test-machine runs their change needs before it is released to the image projects that depend on this test suite. One mapping is off: changes to the GNOME extension tests are told to use the standard virtual-machine workflow, but that workflow does not run the extension suite or its mandatory validation gate — only the separate composite action does. Those changes would be sent to a lane that cannot prove them.
What I need from you: Move the extension suite out of the "hosted workflow" set so it is reported alongside the composite-action lane.
correctness
scripts/vm_validation_plan.py:47-61 puts "extensions" in HOSTED_SUITES, so tests/extensions/** (via the tests/<suite>/** branch at :122-129) and tests/shared/extension_* / gnome_extensions_* (:88-89) land in plan()["suites"] at :145, i.e. "boot this via e2e.yml". But e2e.yml does not own that suite: its suites input (.github/workflows/e2e.yml:29) lists smoke,developer,dx,software,vanilla-gnome,bazzite,common,lifecycle,installer,kde-smoke only, and the extension suite's mandatory pass/fail gate runs solely inside the composite action (.github/actions/gnome-e2e/action.yml:8, :24-25, :609-612 — if: always() && inputs.suite == 'extensions' → scripts/extension_validation.py). The PR's own reference already states e2e.yml cannot validate action-only code (docs/skills/ci-ops/ops/references/candidate-vm-validation.md:44-49), so an extensions change currently gets a plan that reads as a hosted lane while the gate that would catch the regression never executes. Note test_every_mapped_suite_is_known (tests/unit/test_vm_validation_plan.py:143-151) only checks membership in HOSTED_SUITES | ACTION_LANES, so it does not catch this.
Smallest fix — treat extensions as an action lane; existing tests still pass (extension_session.py still maps to ("extensions",)):
--- a/scripts/vm_validation_plan.py
+++ b/scripts/vm_validation_plan.py
@@
-# Pseudo-lane: needs a run of the gnome-e2e composite action, not e2e.yml.
+# Lanes that need a run of the gnome-e2e composite action, not e2e.yml.
+# "extensions" is only runnable there: e2e.yml's suites input omits it and the
+# mandatory extension-validation gate lives in the action.
ACTION = "gnome-e2e-action"
-ACTION_LANES = frozenset({ACTION})
+ACTION_LANES = frozenset({ACTION, "extensions"})
@@
"kde-smoke",
- "extensions",
}
)and in candidate-vm-validation.md move the tests/shared/extension_*, gnome_extensions_* row (and tests/extensions/**) under the composite_action wording at :34-35 so the table matches.
No findings from: security, intent-alignment, style, docs-currency.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
@codebymarshall this PR has a new head without confirmed hive push evidence, so the hive has handed this PR back to you ( |
|
@codebymarshall this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (4 waiting): fixes are handed out before new work as soon as you have capacity. |
|
Held at The hive selects a merge commit for this shape by itself only on a |
Summary: When someone changes this project's tests or its shared test workflow, the change goes out to every image project that relies on it, and nothing first boots a real test machine to prove the change works. This PR adds the first piece of a fix: a written rule, backed by a small checking tool, that looks at which files a change touches and lists the smallest set of real virtual-machine test runs that change needs before it goes out. Documentation-only changes need none. Anything the rule doesn't recognise defaults to the full set, so nothing slips through. It does not turn on any new required check yet, because that needs a maintainer to decide.
What
scripts/vm_validation_plan.py: reads changed paths (stdin or args), prints a JSON plan:vm_required,suites(hostede2e.ymlsuites to boot),no_hosted_lane(affected suites with no GHA lane, such ashardware,security,nvidiaandflatcar, reported instead of dropped), and per-pathreasons.tests/<suite>/**→ that suite;tests/shared/*→smoke+common(withssh_*→common+lifecycle,kde_*→kde-smoke,extension_*→extensions)..github/actions/gnome-e2e/**andscripts/extension_validation.py→gnome-e2e-actionpseudo-lane under a separatecomposite_actionkey (e2e.yml never runs that code, so no e2e.yml suite can validate it);e2e.yml/manual.yml→smoke+common+lifecycle(GUI, SSH, bootc reboot lanes).e2e.ymlsparse-checks-out map to their lanes.tests/unit/,Justfile,plans/and non-distributed workflows (pr-validate,unit-tests,docs-validate,publish-to-pages,update-v1-tag) → no boot.tests/unit/test_vm_validation_plan.py: 27 tests covering every rule class, the fail-closed default, union/sorting, the no-hosted-lane report and both CLI modes.docs/skills/ci-ops/ops/references/candidate-vm-validation.md(linked from the ops skill): the policy table, how to run it against the PR's own workflow and test revision (dispatchmanual.ymlon the fork branch, digest-pinnedimage, head-SHAtest_ref, never thetest_ref: maindefault), and the audit below.Audit of existing signals (
main3e5d45a)GET /commits/3e5d45ad/status:{"state":"pending","statuses":[]}, so no ghost-lab or other external lab status is posted. There is nothing external to reuse or reconcile.arc-ghost-runners-local-dev-routing.md), not a PR or merge gate.Scope / what remains
Refs #1027. Partial: this covers acceptance item 1 (the change-path policy) and the status audit. Still open, and all of it needs maintainer approval under the issue's decision boundary: running the plan as a required check, gating
update-v1-tag.ymlon an observed good VM verdict (reusing #1022's fail-closed contract), splitting read-only VM jobs from trusted publication, and the real-PR demonstration with retained VM evidence.I changed no workflow, required check, permission or secret.
Validation
ruff check tests/ scripts/ --select E,F,W --ignore E501: passedpython -m pytest -n auto tests/unit/ -q: 2233 passed, 1 xfailedpython3 scripts/validate_docs.py: passed;generate_skill_index.py --check: in sync;update_coverage_snapshot.py --check: up to date.featurefiles changed. podman was not installed on this host, so I ran the checks in a Python venv with theunit-tests.ymldependency set.e2e-workflow/SKILL.md, so I put the reference underops/to avoid overlapping with them).🤖 Generated with Claude Code
— hive: backend=claude model=claude-opus-5-5
Refs #1027
🐝 Hive Agent:
contributor| SHA:c50d3b25