Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/skills/ci-ops/ops/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,3 +201,4 @@ Load these when you hit the specific topic:
- [Workflows without a pull_request trigger break main silently](references/workflows-without-a-pull-request-trigger-break-main-silently.md)
- [qecore-headless restarts GDM — the session bus socket is replaced](references/qecore-headless-restarts-gdm-bus-socket-churn.md)
- [Coverage snapshot is generated — never hand-edit scenario counts](references/coverage-snapshot-generated.md)
- [Which candidate-VM suites a testsuite PR must boot before `v1` distributes it](references/candidate-vm-validation.md)
84 changes: 84 additions & 0 deletions docs/skills/ci-ops/ops/references/candidate-vm-validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
name: candidate-vm-validation
description: "Change-path policy for which candidate-VM suites a testsuite PR must boot before v1 distributes it, and the audited state of today's gates (#1027)."
metadata:
type: reference
audience: agents
maturity: draft
---
# Candidate-VM validation before `v1` distribution

`update-v1-tag.yml` force-pushes `v1` to every new `main` commit, and
consumers call `e2e.yml@v1`. A merged test or workflow change therefore reaches
consumer qualification gates without ever booting. Lint, `behave --dry-run`
and unit tests cannot prove GNOME/AT-SPI startup, runner-side SSH probes,
bootc reboots or result propagation (testsuite#1027, parent #1021).

## Change-path policy

`scripts/vm_validation_plan.py` maps changed paths to the minimal
representative `e2e.yml` suites. Rules live in its `RULES` table; tests in
`tests/unit/test_vm_validation_plan.py`.

```bash
git diff --name-only upstream/main...HEAD | python3 scripts/vm_validation_plan.py
```

| Changed path | Suites to boot |
|---|---|
| `*.md`, `docs/`, `tests/unit/`, `dashboard/`, `Justfile`, `plans/`, non-distributed workflows (`pr-validate`, `unit-tests`, `docs-validate`, `publish-to-pages`, `update-v1-tag`) | none |
| `tests/<suite>/**` (steps, hooks, features) | that suite |
| `tests/shared/kde_*` | `kde-smoke` |
| `tests/shared/extension_*`, `gnome_extensions_*` | `extensions` |
| `tests/shared/ssh_*` | `common`, `lifecycle` |
| other `tests/shared/*`, `container/` runner, `scripts/*` that `e2e.yml` runs | `smoke`, `common` |
| `.github/actions/gnome-e2e/**`, `scripts/extension_validation.py` | `gnome-e2e-action` (reported under `composite_action`, not `suites`) |
| `.github/workflows/e2e.yml`, `manual.yml` | `smoke`, `common`, `lifecycle` |
| `migration-test.yml` / `iso-*.yml` | `lifecycle` / `installer` |
| anything else | `smoke`, `common`, `lifecycle` (fail-closed) |

`smoke` is the GUI lane, `common` the SSH lane and `lifecycle` the bootc reboot
lane. Suites with no hosted `e2e.yml` lane (`hardware`, `security`, `nvidia`,
`flatcar`) are reported under `no_hosted_lane` rather than dropped — the plan
must not read as "nothing to boot" when it cannot validate them.

`e2e.yml` never calls the `gnome-e2e` composite action, and its sparse checkout
excludes `scripts/extension_validation.py`, which only the action runs. Booting
`e2e.yml` suites therefore exercises none of that code. Such changes are
reported under `composite_action` (pseudo-lane `gnome-e2e-action`), excluded
from `suites` and `no_hosted_lane`; validate them with a workflow that uses
`.github/actions/gnome-e2e` at the PR's ref.

When adding a workflow, script or shared helper, add a rule: an unmatched path
silently costs the full set, and a wrong empty rule silently skips validation.

## Running the plan against the PR's own code

Dispatch `manual.yml` **on the PR branch in the fork**. It calls the fork's
`e2e.yml` revision and passes `test_repository: ${{ github.repository }}` and
`test_ref` = the dispatched branch, so the modified workflow and tests execute,
not `main`/`v1`. Pin `image` by digest (`ghcr.io/...@sha256:...`) so the
candidate identity is immutable, and pass `test_ref` as the head SHA rather than
the branch name. Never rely on `e2e.yml`'s `test_ref: main` default for this.

## Audited signal state (2026-10-28, `main` 3e5d45ad)

- Combined commit status: no statuses at all. No ghost-lab or other external
lab context is posted to testsuite commits, so there is no existing external
signal to reuse or reconcile.
- Check runs on `main`: Lint & syntax, Behave dry-run, Quarantine age,
Coverage snapshot fresh, pytest, build-and-deploy, Force-push v1 to HEAD.
None boots a VM.
- ARC "ghost" runners ([arc-ghost-runners-local-dev-routing](arc-ghost-runners-local-dev-routing.md))
are a local pre-push developer routing, not a PR or merge gate.

Re-check before relying on this: `gh api repos/projectbluefin/testsuite/commits/<sha>/status`
and `.../check-runs`.

## Not done here (maintainer decisions, #1027)

Running the plan as a required check, gating `update-v1-tag.yml` on an observed
good VM verdict, and any secret/permission split between read-only VM jobs and
trusted publication all change required checks, `v1` publication policy or CI
permissions. Those need explicit maintainer approval; the verdict must reuse the
fail-closed contract from #1022.
158 changes: 158 additions & 0 deletions scripts/vm_validation_plan.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
#!/usr/bin/env python3
"""Change-path policy: which candidate-VM suites must a PR boot before v1?

Maps the files a pull request changes to the minimal representative set of
e2e.yml suites that must run against a real candidate image before the change
is distributed to consumers through the managed ``v1`` tag (testsuite#1027).

The policy is fail-closed: a path no rule recognises requires the full
representative set (GUI, SSH and bootc-lifecycle lanes) rather than nothing.
Docs, unit tests and repo-local tooling that never reaches a VM need no boot.

This script only computes the plan. It does not dispatch VMs, set statuses or
change required checks; wiring it into CI is a maintainer decision (#1027).

Usage:
git diff --name-only BASE...HEAD | python3 scripts/vm_validation_plan.py
python3 scripts/vm_validation_plan.py tests/smoke/features/steps/x.py

Prints JSON: {"vm_required": bool, "suites": [...], "no_hosted_lane": [...],
"composite_action": [...], "reasons": {path: [suite, ...]}}.

``composite_action`` lists pseudo-lanes for code e2e.yml never executes: the
gnome-e2e composite action and the extension-validation gate it alone runs
(e2e.yml neither uses the action nor sparse-checks-out that script). Booting
e2e.yml suites cannot validate those changes, so they are reported separately
and must be exercised through a workflow that calls the action at the PR ref.
"""

from __future__ import annotations

import fnmatch
import json
import sys

# Representative lanes. GUI exercises GNOME/AT-SPI startup inside the VM,
# SSH exercises runner-side probes, LIFECYCLE exercises bootc reboots.
GUI = "smoke"
SSH = "common"
LIFECYCLE = "lifecycle"
FULL = (GUI, SSH, LIFECYCLE)
# Pseudo-lane: needs a run of the gnome-e2e composite action, not e2e.yml.
ACTION = "gnome-e2e-action"
ACTION_LANES = frozenset({ACTION})

# Suites e2e.yml can boot on hosted runners (its `suites` input).
HOSTED_SUITES = frozenset(
{
"smoke",
"developer",
"dx",
"software",
"vanilla-gnome",
"bazzite",
"common",
"lifecycle",
"installer",
"kde-smoke",
"extensions",
}
)

# Ordered (glob, suites) rules; first match wins. An empty tuple means the
# path never executes inside a candidate VM.
RULES: tuple[tuple[str, tuple[str, ...]], ...] = (
# Never reach a VM: documentation, unit tests, repo-local tooling.
("*.md", ()),
("docs/*", ()),
("dashboard/*", ()),
("argocd/*", ()),
("tests/unit/*", ()),
("LICENSE", ()),
("renovate.json", ()),
("codecov.yml", ()),
("Justfile", ()),
("plans/*", ()),
(".github/workflows/pr-validate.yml", ()),
(".github/workflows/unit-tests.yml", ()),
(".github/workflows/docs-validate.yml", ()),
(".github/workflows/publish-to-pages.yml", ()),
(".github/workflows/update-v1-tag.yml", ()),
("scripts/validate_docs.py", ()),
("scripts/generate_skill_index.py", ()),
("scripts/update_coverage_snapshot.py", ()),
("scripts/generate_badges.py", ()),
# Shared helpers: route by the path that imports them.
("tests/shared/kde_*", ("kde-smoke",)),
("tests/shared/extension_*", ("extensions",)),
("tests/shared/gnome_extensions_*", ("extensions",)),
("tests/shared/ssh_*", (SSH, LIFECYCLE)),
("tests/shared/*", (GUI, SSH)),
# Runner images and the composed e2e overlay.
("container/Containerfile.kde-runner", ("kde-smoke",)),
("container/requirements.kde-runner.txt", ("kde-smoke",)),
("container/*", (GUI, SSH)),
(".github/workflows/build-kde-runner.yml", ("kde-smoke",)),
(".github/workflows/build-runner.yml", (GUI, SSH)),
# Distributed workflow and composite action.
(".github/actions/gnome-e2e/*", (ACTION,)),
(".github/workflows/e2e.yml", FULL),
(".github/workflows/manual.yml", FULL),
(".github/workflows/migration-test.yml", (LIFECYCLE,)),
(".github/workflows/iso-*.yml", ("installer",)),
# Helper scripts e2e.yml sparse-checks-out and runs per suite.
("scripts/install-kde-webdriver.sh", ("kde-smoke",)),
("scripts/assert_kde_passed.py", ("kde-smoke",)),
# Run only inside the gnome-e2e composite action, never by e2e.yml.
("scripts/extension_validation.py", (ACTION,)),
("scripts/*", (GUI, SSH)),
("flatpak-app-list.txt", (GUI, SSH)),
)


def suites_for_path(path: str) -> tuple[str, ...]:
"""Return the suites one changed path requires (fail-closed default)."""
path = path.strip().removeprefix("./")
parts = path.split("/")
# tests/<suite>/** — suite READMEs fall through to the *.md rule first.
if (
len(parts) >= 3
and parts[0] == "tests"
and parts[1] not in ("shared", "unit")
and not path.endswith(".md")
):
return (parts[1],)
for pattern, suites in RULES:
if fnmatch.fnmatchcase(path, pattern):
return suites
return FULL


def plan(paths: list[str]) -> dict:
"""Combine per-path requirements into one minimal plan."""
reasons: dict[str, list[str]] = {}
required: set[str] = set()
for raw in paths:
path = raw.strip()
if not path:
continue
suites = list(suites_for_path(path))
reasons[path] = suites
required.update(suites)
return {
"vm_required": bool(required),
"suites": sorted(required & HOSTED_SUITES),
"no_hosted_lane": sorted(required - HOSTED_SUITES - ACTION_LANES),
"composite_action": sorted(required & ACTION_LANES),
"reasons": reasons,
}


def main(argv: list[str]) -> int:
paths = argv[1:] if len(argv) > 1 else sys.stdin.read().splitlines()
print(json.dumps(plan(paths), indent=2, sort_keys=True))
return 0


if __name__ == "__main__":
sys.exit(main(sys.argv))
Loading
Loading