Skip to content

feat(ci): list which real test-machine runs a change needs before it ships - #1042

Merged
Danathar merged 3 commits into
projectbluefin:mainfrom
codebymarshall:feat/vm-validation-change-path-policy
Oct 28, 2026
Merged

Danathar merged 3 commits into
projectbluefin:mainfrom
codebymarshall:feat/vm-validation-change-path-policy

Conversation

@codebymarshall

@codebymarshall codebymarshall commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary: When someone changes this project's tests or its shared test workflow, the change goes out to every image project that relies on it, and nothing first boots a real test machine to prove the change works. This PR adds the first piece of a fix: a written rule, backed by a small checking tool, that looks at which files a change touches and lists the smallest set of real virtual-machine test runs that change needs before it goes out. Documentation-only changes need none. Anything the rule doesn't recognise defaults to the full set, so nothing slips through. It does not turn on any new required check yet, because that needs a maintainer to decide.


What

  • scripts/vm_validation_plan.py: reads changed paths (stdin or args), prints a JSON plan: vm_required, suites (hosted e2e.yml suites to boot), no_hosted_lane (affected suites with no GHA lane, such as hardware, security, nvidia and flatcar, reported instead of dropped), and per-path reasons.
    • tests/<suite>/** → that suite; tests/shared/* → smoke+common (with ssh_* → common+lifecycle, kde_* → kde-smoke, extension_* → extensions).
    • .github/actions/gnome-e2e/** and scripts/extension_validation.py → gnome-e2e-action pseudo-lane under a separate composite_action key (e2e.yml never runs that code, so no e2e.yml suite can validate it); e2e.yml/manual.yml → smoke+common+lifecycle (GUI, SSH, bootc reboot lanes).
    • Runner images and the scripts e2e.yml sparse-checks-out map to their lanes.
    • Docs, tests/unit/, Justfile, plans/ and non-distributed workflows (pr-validate, unit-tests, docs-validate, publish-to-pages, update-v1-tag) → no boot.
    • Unknown paths fail closed to the full representative set.
  • tests/unit/test_vm_validation_plan.py: 27 tests covering every rule class, the fail-closed default, union/sorting, the no-hosted-lane report and both CLI modes.
  • docs/skills/ci-ops/ops/references/candidate-vm-validation.md (linked from the ops skill): the policy table, how to run it against the PR's own workflow and test revision (dispatch manual.yml on the fork branch, digest-pinned image, head-SHA test_ref, never the test_ref: main default), and the audit below.

Audit of existing signals (main 3e5d45a)

  • GET /commits/3e5d45ad/status: {"state":"pending","statuses":[]}, so no ghost-lab or other external lab status is posted. There is nothing external to reuse or reconcile.
  • Check runs on that commit: Lint & syntax, Behave dry-run, Quarantine age, Coverage snapshot fresh, pytest, build-and-deploy, Force-push v1 to HEAD. None of them boots a VM.
  • ARC "ghost" runners are a local pre-push developer route (documented in arc-ghost-runners-local-dev-routing.md), not a PR or merge gate.

Scope / what remains

Refs #1027. Partial: this covers acceptance item 1 (the change-path policy) and the status audit. Still open, and all of it needs maintainer approval under the issue's decision boundary: running the plan as a required check, gating update-v1-tag.yml on an observed good VM verdict (reusing #1022's fail-closed contract), splitting read-only VM jobs from trusted publication, and the real-PR demonstration with retained VM evidence.

I changed no workflow, required check, permission or secret.

Validation

🤖 Generated with Claude Code

— hive: backend=claude model=claude-opus-5-5

Refs #1027


🐝 Hive Agent: contributor | SHA: c50d3b25

Add scripts/vm_validation_plan.py, which maps a PR's changed paths to
the minimal representative e2e.yml suites (GUI smoke, SSH common,
bootc lifecycle, plus suite-specific lanes) that must boot against a
real candidate image before v1 distributes the change. Docs, unit
tests and non-distributed tooling need no boot; unknown paths fail
closed to the full representative set; suites with no hosted lane are
reported instead of dropped.

Document the policy, how to run it against the PR's own workflow and
test revision, and the audited state of existing signals (no commit
statuses on main; no check run boots a VM). Wiring it into required
checks or gating update-v1-tag.yml is left for maintainer approval.

Refs projectbluefin#1027

Hive-Run: projectbluefin#1027
Hive-Plan: projectbluefin#1021 rollout plan — work package projectbluefin#1027
Hive-Spec: testsuite#1027#acceptance-1-change-path-policy
Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: codebymarshall <codebymarshall@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/contributor Author holds the contributor tier in this hive's registry; a human must lgtm/approve stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change adds a small tool that reads which files a pull request touched and says which real test-machine runs must happen before the change is shipped to other projects. Two things need fixing before it lands: for changes to the shared "run GNOME tests" action, the tool points at machine runs that never execute that action, so the check would pass without testing the thing that changed; and the pull request is marked to close the design issue even though it only completes one of six items.

What I need from you: Apply the two fixes below (or say why not).


correctness

Composite-action and extension-gate changes map to e2e.yml lanes that never execute them (medium, in-scope). scripts/vm_validation_plan.py:98 maps .github/actions/gnome-e2e/* to (GUI, "extensions") and :106 maps scripts/extension_validation.py to ("extensions",); plan() at :135 then emits those as e2e.yml suites. But e2e.yml at head never references the action (no gnome-e2e anywhere in .github/workflows/e2e.yml), and its sparse-checkout (e2e.yml:235-243) does not include scripts/extension_validation.py; that script runs only inside the action (.github/actions/gnome-e2e/action.yml:23-25), which consumers call directly (action.yml:17). So the plan for an action.yml or extension-gate change says "boot smoke/extensions via e2e.yml", which exercises none of the changed code — the opposite of the fail-closed intent, and acceptance item 1 of #1027 names composite-action changes explicitly. (HOSTED_SUITES:38-52 also lists extensions, which e2e.yml:29 does not document as an input value; it runs generically via tests/extensions, but without the validation gate.)

Proposed fix — report the action as its own boot path instead of folding it into e2e.yml suites:

--- a/scripts/vm_validation_plan.py
+++ b/scripts/vm_validation_plan.py
@@
 LIFECYCLE = "lifecycle"
 FULL = (GUI, SSH, LIFECYCLE)
+# The gnome-e2e composite action is a separate boot path from e2e.yml:
+# consumers call it directly, and only it runs scripts/extension_validation.py.
+ACTION = "gnome-e2e"
@@
-    (".github/actions/gnome-e2e/*", (GUI, "extensions")),
+    (".github/actions/gnome-e2e/*", (ACTION, GUI, "extensions")),
@@
-    ("scripts/extension_validation.py", ("extensions",)),
+    ("scripts/extension_validation.py", (ACTION, "extensions")),
@@ def plan(paths: list[str]) -> dict:
     return {
         "vm_required": bool(required),
+        "composite_action": ACTION in required,
         "suites": sorted(required & HOSTED_SUITES),
-        "no_hosted_lane": sorted(required - HOSTED_SUITES),
+        "no_hosted_lane": sorted(required - HOSTED_SUITES - {ACTION}),
         "reasons": reasons,
     }

Then update test_every_mapped_suite_is_known / test_composite_action_change_requires_gui_lanes in tests/unit/test_vm_validation_plan.py:75-77,112-121, the docstring at scripts/vm_validation_plan.py:19-20, and the policy table row at docs/skills/ci-ops/ops/references/candidate-vm-validation.md:35 to say the action must be dispatched directly (a workflow calling ./.github/actions/gnome-e2e with suite: extensions).

intent-alignment

Closes #1027 will close a six-item design issue when one item lands (medium, in-scope). The PR body says "Refs #1027. Partial: this covers acceptance item 1" and lists the remaining five items as still open, yet the footer carries Closes #1027, which GitHub honours on merge to main. Change the footer to Refs #1027 so the issue stays open for the remaining acceptance items.

No findings from: security, style, docs-currency.

Confidence: 3/5 (needs attention) — 2 medium findings

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@codebymarshall this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the review swarm requested changes

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (2 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@codebymarshall the hive has adopted this PR (stage/074-adopted-by-hive): your hive has been offline since the nudge. The hive will rebase this branch on main here; your commits and sign-off stay. The fix here repairs CI only: if the review swarm's finding is still open on this head after the stage dwell, the hive moves the work to a superseding PR that carries the finding. Push a new head yourself at any time and the hive yields it back to you.

@hivecommons-hive hivecommons-hive Bot added stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 28, 2026
e2e.yml never calls the gnome-e2e composite action and its sparse
checkout excludes scripts/extension_validation.py, so mapping those
paths to smoke/extensions prescribed runs that exercise none of the
changed code. Map them to a gnome-e2e-action pseudo-lane reported under
a new composite_action key, excluded from suites and no_hosted_lane.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

The hive adopted this PR (stage/074-adopted-by-hive) but its head be239f74f6e7 has not moved for 1m and the hive has nothing it can push: adoption-stalled:hive-signoff-attribution.

Blocker: 1 commit(s) on be239f74f6e7 carry the App's Signed-off-by while a squash merge would land them as @codebymarshall, so the hive will not squash-merge this head. The hive cannot rewrite that authorship without the contributor, and a superseding PR would carry the same head, so it opened none. The hive selects a merge commit for this shape by itself only on a role/trusted PR or a head a trusted maintainer has approved; this one is neither yet. Ask (maintainer): every commit already carries a sign-off matching its own author, so DCO holds without rewriting history: approve the current head with a GitHub Approve review (the hive then selects the merge commit itself), or merge it by hand with a merge commit (gh pr merge 1042 --merge, not squash).

Moving to stage/085-needs-decision so a person sees it. This is not a merge-approval wait: the lane has not admitted this head.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/074-adopted-by-hive Contributor PR adopted: author unavailable after red CI; the hive is finishing it (takeover path) labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change adds a small tool that tells a contributor which real test-machine runs their change needs before it is released to the image projects that depend on this test suite. One mapping is off: changes to the GNOME extension tests are told to use the standard virtual-machine workflow, but that workflow does not run the extension suite or its mandatory validation gate — only the separate composite action does. Those changes would be sent to a lane that cannot prove them.

What I need from you: Move the extension suite out of the "hosted workflow" set so it is reported alongside the composite-action lane.


correctness

scripts/vm_validation_plan.py:47-61 puts "extensions" in HOSTED_SUITES, so tests/extensions/** (via the tests/<suite>/** branch at :122-129) and tests/shared/extension_* / gnome_extensions_* (:88-89) land in plan()["suites"] at :145, i.e. "boot this via e2e.yml". But e2e.yml does not own that suite: its suites input (.github/workflows/e2e.yml:29) lists smoke,developer,dx,software,vanilla-gnome,bazzite,common,lifecycle,installer,kde-smoke only, and the extension suite's mandatory pass/fail gate runs solely inside the composite action (.github/actions/gnome-e2e/action.yml:8, :24-25, :609-612 — if: always() && inputs.suite == 'extensions' → scripts/extension_validation.py). The PR's own reference already states e2e.yml cannot validate action-only code (docs/skills/ci-ops/ops/references/candidate-vm-validation.md:44-49), so an extensions change currently gets a plan that reads as a hosted lane while the gate that would catch the regression never executes. Note test_every_mapped_suite_is_known (tests/unit/test_vm_validation_plan.py:143-151) only checks membership in HOSTED_SUITES | ACTION_LANES, so it does not catch this.

Smallest fix — treat extensions as an action lane; existing tests still pass (extension_session.py still maps to ("extensions",)):

--- a/scripts/vm_validation_plan.py
+++ b/scripts/vm_validation_plan.py
@@
-# Pseudo-lane: needs a run of the gnome-e2e composite action, not e2e.yml.
+# Lanes that need a run of the gnome-e2e composite action, not e2e.yml.
+# "extensions" is only runnable there: e2e.yml's suites input omits it and the
+# mandatory extension-validation gate lives in the action.
 ACTION = "gnome-e2e-action"
-ACTION_LANES = frozenset({ACTION})
+ACTION_LANES = frozenset({ACTION, "extensions"})
@@
         "kde-smoke",
-        "extensions",
     }
 )

and in candidate-vm-validation.md move the tests/shared/extension_*, gnome_extensions_* row (and tests/extensions/**) under the composite_action wording at :34-35 so the table matches.

No findings from: security, intent-alignment, style, docs-currency.

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@codebymarshall this PR has a new head without confirmed hive push evidence, so the hive has handed this PR back to you (stage/074-adopted-by-hive → stage/072-returned-to-contributor). It will nudge again if CI stays red.

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@codebymarshall this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • 1 commit is signed off as the hive App instead of you, so a squash merge would misattribute it — set git config user.email to your own address, then run git rebase --exec 'git commit --amend --no-edit --reset-author -s' HEAD~2 && git push --force-with-lease

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (4 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Oct 28, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Held at stage/085-needs-decision for a maintainer. 1 of 2 commits on cfab95c16ee4 are the hive's own, signed off as the hive App, so a squash merge would land them as @codebymarshall with the App's sign-off, and the hive will not squash-merge this head. @codebymarshall did not make those commits and has nothing to fix here.

The hive selects a merge commit for this shape by itself only on a role/trusted PR or a head a trusted maintainer has approved; this one is neither yet. Ask (maintainer): every commit already carries a sign-off matching its own author, so DCO holds without rewriting history: approve the current head with a GitHub Approve review (the hive then selects the merge commit itself), or merge it by hand with a merge commit (gh pr merge 1042 --merge, not squash).

@Danathar
Danathar merged commit ecb4731 into projectbluefin:main Oct 28, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/contributor Author holds the contributor tier in this hive's registry; a human must lgtm/approve stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants