diff --git a/docs/skills/ci-ops/ops/SKILL.md b/docs/skills/ci-ops/ops/SKILL.md index f84c95dd3..5e00f4382 100644 --- a/docs/skills/ci-ops/ops/SKILL.md +++ b/docs/skills/ci-ops/ops/SKILL.md @@ -201,3 +201,4 @@ Load these when you hit the specific topic: - [Workflows without a pull_request trigger break main silently](references/workflows-without-a-pull-request-trigger-break-main-silently.md) - [qecore-headless restarts GDM — the session bus socket is replaced](references/qecore-headless-restarts-gdm-bus-socket-churn.md) - [Coverage snapshot is generated — never hand-edit scenario counts](references/coverage-snapshot-generated.md) +- [Which candidate-VM suites a testsuite PR must boot before `v1` distributes it](references/candidate-vm-validation.md) diff --git a/docs/skills/ci-ops/ops/references/candidate-vm-validation.md b/docs/skills/ci-ops/ops/references/candidate-vm-validation.md new file mode 100644 index 000000000..acca318e1 --- /dev/null +++ b/docs/skills/ci-ops/ops/references/candidate-vm-validation.md @@ -0,0 +1,84 @@ +--- +name: candidate-vm-validation +description: "Change-path policy for which candidate-VM suites a testsuite PR must boot before v1 distributes it, and the audited state of today's gates (#1027)." +metadata: + type: reference + audience: agents + maturity: draft +--- +# Candidate-VM validation before `v1` distribution + +`update-v1-tag.yml` force-pushes `v1` to every new `main` commit, and +consumers call `e2e.yml@v1`. A merged test or workflow change therefore reaches +consumer qualification gates without ever booting. Lint, `behave --dry-run` +and unit tests cannot prove GNOME/AT-SPI startup, runner-side SSH probes, +bootc reboots or result propagation (testsuite#1027, parent #1021). + +## Change-path policy + +`scripts/vm_validation_plan.py` maps changed paths to the minimal +representative `e2e.yml` suites. Rules live in its `RULES` table; tests in +`tests/unit/test_vm_validation_plan.py`. + +```bash +git diff --name-only upstream/main...HEAD | python3 scripts/vm_validation_plan.py +``` + +| Changed path | Suites to boot | +|---|---| +| `*.md`, `docs/`, `tests/unit/`, `dashboard/`, `Justfile`, `plans/`, non-distributed workflows (`pr-validate`, `unit-tests`, `docs-validate`, `publish-to-pages`, `update-v1-tag`) | none | +| `tests//**` (steps, hooks, features) | that suite | +| `tests/shared/kde_*` | `kde-smoke` | +| `tests/shared/extension_*`, `gnome_extensions_*` | `extensions` | +| `tests/shared/ssh_*` | `common`, `lifecycle` | +| other `tests/shared/*`, `container/` runner, `scripts/*` that `e2e.yml` runs | `smoke`, `common` | +| `.github/actions/gnome-e2e/**`, `scripts/extension_validation.py` | `gnome-e2e-action` (reported under `composite_action`, not `suites`) | +| `.github/workflows/e2e.yml`, `manual.yml` | `smoke`, `common`, `lifecycle` | +| `migration-test.yml` / `iso-*.yml` | `lifecycle` / `installer` | +| anything else | `smoke`, `common`, `lifecycle` (fail-closed) | + +`smoke` is the GUI lane, `common` the SSH lane and `lifecycle` the bootc reboot +lane. Suites with no hosted `e2e.yml` lane (`hardware`, `security`, `nvidia`, +`flatcar`) are reported under `no_hosted_lane` rather than dropped — the plan +must not read as "nothing to boot" when it cannot validate them. + +`e2e.yml` never calls the `gnome-e2e` composite action, and its sparse checkout +excludes `scripts/extension_validation.py`, which only the action runs. Booting +`e2e.yml` suites therefore exercises none of that code. Such changes are +reported under `composite_action` (pseudo-lane `gnome-e2e-action`), excluded +from `suites` and `no_hosted_lane`; validate them with a workflow that uses +`.github/actions/gnome-e2e` at the PR's ref. + +When adding a workflow, script or shared helper, add a rule: an unmatched path +silently costs the full set, and a wrong empty rule silently skips validation. + +## Running the plan against the PR's own code + +Dispatch `manual.yml` **on the PR branch in the fork**. It calls the fork's +`e2e.yml` revision and passes `test_repository: ${{ github.repository }}` and +`test_ref` = the dispatched branch, so the modified workflow and tests execute, +not `main`/`v1`. Pin `image` by digest (`ghcr.io/...@sha256:...`) so the +candidate identity is immutable, and pass `test_ref` as the head SHA rather than +the branch name. Never rely on `e2e.yml`'s `test_ref: main` default for this. + +## Audited signal state (2026-10-28, `main` 3e5d45ad) + +- Combined commit status: no statuses at all. No ghost-lab or other external + lab context is posted to testsuite commits, so there is no existing external + signal to reuse or reconcile. +- Check runs on `main`: Lint & syntax, Behave dry-run, Quarantine age, + Coverage snapshot fresh, pytest, build-and-deploy, Force-push v1 to HEAD. + None boots a VM. +- ARC "ghost" runners ([arc-ghost-runners-local-dev-routing](arc-ghost-runners-local-dev-routing.md)) + are a local pre-push developer routing, not a PR or merge gate. + +Re-check before relying on this: `gh api repos/projectbluefin/testsuite/commits//status` +and `.../check-runs`. + +## Not done here (maintainer decisions, #1027) + +Running the plan as a required check, gating `update-v1-tag.yml` on an observed +good VM verdict, and any secret/permission split between read-only VM jobs and +trusted publication all change required checks, `v1` publication policy or CI +permissions. Those need explicit maintainer approval; the verdict must reuse the +fail-closed contract from #1022. diff --git a/scripts/vm_validation_plan.py b/scripts/vm_validation_plan.py new file mode 100644 index 000000000..645726da5 --- /dev/null +++ b/scripts/vm_validation_plan.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +"""Change-path policy: which candidate-VM suites must a PR boot before v1? + +Maps the files a pull request changes to the minimal representative set of +e2e.yml suites that must run against a real candidate image before the change +is distributed to consumers through the managed ``v1`` tag (testsuite#1027). + +The policy is fail-closed: a path no rule recognises requires the full +representative set (GUI, SSH and bootc-lifecycle lanes) rather than nothing. +Docs, unit tests and repo-local tooling that never reaches a VM need no boot. + +This script only computes the plan. It does not dispatch VMs, set statuses or +change required checks; wiring it into CI is a maintainer decision (#1027). + +Usage: + git diff --name-only BASE...HEAD | python3 scripts/vm_validation_plan.py + python3 scripts/vm_validation_plan.py tests/smoke/features/steps/x.py + +Prints JSON: {"vm_required": bool, "suites": [...], "no_hosted_lane": [...], +"composite_action": [...], "reasons": {path: [suite, ...]}}. + +``composite_action`` lists pseudo-lanes for code e2e.yml never executes: the +gnome-e2e composite action and the extension-validation gate it alone runs +(e2e.yml neither uses the action nor sparse-checks-out that script). Booting +e2e.yml suites cannot validate those changes, so they are reported separately +and must be exercised through a workflow that calls the action at the PR ref. +""" + +from __future__ import annotations + +import fnmatch +import json +import sys + +# Representative lanes. GUI exercises GNOME/AT-SPI startup inside the VM, +# SSH exercises runner-side probes, LIFECYCLE exercises bootc reboots. +GUI = "smoke" +SSH = "common" +LIFECYCLE = "lifecycle" +FULL = (GUI, SSH, LIFECYCLE) +# Pseudo-lane: needs a run of the gnome-e2e composite action, not e2e.yml. +ACTION = "gnome-e2e-action" +ACTION_LANES = frozenset({ACTION}) + +# Suites e2e.yml can boot on hosted runners (its `suites` input). +HOSTED_SUITES = frozenset( + { + "smoke", + "developer", + "dx", + "software", + "vanilla-gnome", + "bazzite", + "common", + "lifecycle", + "installer", + "kde-smoke", + "extensions", + } +) + +# Ordered (glob, suites) rules; first match wins. An empty tuple means the +# path never executes inside a candidate VM. +RULES: tuple[tuple[str, tuple[str, ...]], ...] = ( + # Never reach a VM: documentation, unit tests, repo-local tooling. + ("*.md", ()), + ("docs/*", ()), + ("dashboard/*", ()), + ("argocd/*", ()), + ("tests/unit/*", ()), + ("LICENSE", ()), + ("renovate.json", ()), + ("codecov.yml", ()), + ("Justfile", ()), + ("plans/*", ()), + (".github/workflows/pr-validate.yml", ()), + (".github/workflows/unit-tests.yml", ()), + (".github/workflows/docs-validate.yml", ()), + (".github/workflows/publish-to-pages.yml", ()), + (".github/workflows/update-v1-tag.yml", ()), + ("scripts/validate_docs.py", ()), + ("scripts/generate_skill_index.py", ()), + ("scripts/update_coverage_snapshot.py", ()), + ("scripts/generate_badges.py", ()), + # Shared helpers: route by the path that imports them. + ("tests/shared/kde_*", ("kde-smoke",)), + ("tests/shared/extension_*", ("extensions",)), + ("tests/shared/gnome_extensions_*", ("extensions",)), + ("tests/shared/ssh_*", (SSH, LIFECYCLE)), + ("tests/shared/*", (GUI, SSH)), + # Runner images and the composed e2e overlay. + ("container/Containerfile.kde-runner", ("kde-smoke",)), + ("container/requirements.kde-runner.txt", ("kde-smoke",)), + ("container/*", (GUI, SSH)), + (".github/workflows/build-kde-runner.yml", ("kde-smoke",)), + (".github/workflows/build-runner.yml", (GUI, SSH)), + # Distributed workflow and composite action. + (".github/actions/gnome-e2e/*", (ACTION,)), + (".github/workflows/e2e.yml", FULL), + (".github/workflows/manual.yml", FULL), + (".github/workflows/migration-test.yml", (LIFECYCLE,)), + (".github/workflows/iso-*.yml", ("installer",)), + # Helper scripts e2e.yml sparse-checks-out and runs per suite. + ("scripts/install-kde-webdriver.sh", ("kde-smoke",)), + ("scripts/assert_kde_passed.py", ("kde-smoke",)), + # Run only inside the gnome-e2e composite action, never by e2e.yml. + ("scripts/extension_validation.py", (ACTION,)), + ("scripts/*", (GUI, SSH)), + ("flatpak-app-list.txt", (GUI, SSH)), +) + + +def suites_for_path(path: str) -> tuple[str, ...]: + """Return the suites one changed path requires (fail-closed default).""" + path = path.strip().removeprefix("./") + parts = path.split("/") + # tests//** — suite READMEs fall through to the *.md rule first. + if ( + len(parts) >= 3 + and parts[0] == "tests" + and parts[1] not in ("shared", "unit") + and not path.endswith(".md") + ): + return (parts[1],) + for pattern, suites in RULES: + if fnmatch.fnmatchcase(path, pattern): + return suites + return FULL + + +def plan(paths: list[str]) -> dict: + """Combine per-path requirements into one minimal plan.""" + reasons: dict[str, list[str]] = {} + required: set[str] = set() + for raw in paths: + path = raw.strip() + if not path: + continue + suites = list(suites_for_path(path)) + reasons[path] = suites + required.update(suites) + return { + "vm_required": bool(required), + "suites": sorted(required & HOSTED_SUITES), + "no_hosted_lane": sorted(required - HOSTED_SUITES - ACTION_LANES), + "composite_action": sorted(required & ACTION_LANES), + "reasons": reasons, + } + + +def main(argv: list[str]) -> int: + paths = argv[1:] if len(argv) > 1 else sys.stdin.read().splitlines() + print(json.dumps(plan(paths), indent=2, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/tests/unit/test_vm_validation_plan.py b/tests/unit/test_vm_validation_plan.py new file mode 100644 index 000000000..d109e58c4 --- /dev/null +++ b/tests/unit/test_vm_validation_plan.py @@ -0,0 +1,170 @@ +"""Unit tests for the candidate-VM change-path policy (scripts/vm_validation_plan.py).""" + +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +from scripts.vm_validation_plan import ( + ACTION, + ACTION_LANES, + FULL, + HOSTED_SUITES, + plan, + suites_for_path, +) + +REPO_ROOT = Path(__file__).resolve().parents[2] +SCRIPT_PATH = REPO_ROOT / "scripts" / "vm_validation_plan.py" + + +@pytest.mark.parametrize( + "path", + [ + "README.md", + "docs/skills/ci-ops/e2e-workflow/SKILL.md", + "docs/skills/index.json", + "tests/smoke/README.md", + "tests/unit/test_image_slug.py", + "dashboard/index.html", + "Justfile", + ".github/workflows/unit-tests.yml", + ".github/workflows/update-v1-tag.yml", + "scripts/validate_docs.py", + ], +) +def test_paths_that_never_reach_a_vm_need_no_boot(path): + assert suites_for_path(path) == () + + +def test_docs_only_change_does_not_boot(): + result = plan(["README.md", "docs/runbook.md", "tests/unit/test_x.py"]) + assert result["vm_required"] is False + assert result["suites"] == [] + + +@pytest.mark.parametrize( + "path, suite", + [ + ("tests/smoke/features/steps/shell_steps.py", "smoke"), + ("tests/common/features/environment.py", "common"), + ("tests/lifecycle/features/upgrade.feature", "lifecycle"), + ("tests/kde-smoke/features/steps/kde_steps.py", "kde-smoke"), + ], +) +def test_suite_steps_and_hooks_select_their_own_suite(path, suite): + assert suites_for_path(path) == (suite,) + + +def test_shared_gui_helper_needs_gui_and_ssh_lanes(): + assert suites_for_path("tests/shared/wait_for_shell.py") == ("smoke", "common") + + +def test_shared_ssh_helper_needs_ssh_and_lifecycle_lanes(): + assert suites_for_path("tests/shared/ssh_steps.py") == ("common", "lifecycle") + + +def test_shared_kde_and_extension_helpers_route_to_their_suites(): + assert suites_for_path("tests/shared/kde_webdriver.py") == ("kde-smoke",) + assert suites_for_path("tests/shared/extension_session.py") == ("extensions",) + + +def test_reusable_workflow_change_requires_full_representative_set(): + assert suites_for_path(".github/workflows/e2e.yml") == FULL + assert set(FULL) == {"smoke", "common", "lifecycle"} + + +@pytest.mark.parametrize( + "path", + [".github/actions/gnome-e2e/action.yml", "scripts/extension_validation.py"], +) +def test_composite_action_code_maps_to_action_pseudo_lane(path): + assert suites_for_path(path) == (ACTION,) + + +def test_composite_action_reported_outside_e2e_suites(): + result = plan( + [".github/actions/gnome-e2e/action.yml", "scripts/extension_validation.py"] + ) + assert result["vm_required"] is True + assert result["suites"] == [] + assert result["no_hosted_lane"] == [] + assert result["composite_action"] == [ACTION] + + +def test_composite_action_combines_with_e2e_suites(): + result = plan( + [".github/actions/gnome-e2e/action.yml", "tests/smoke/features/x.feature"] + ) + assert result["suites"] == ["smoke"] + assert result["composite_action"] == [ACTION] + + +def test_runner_image_change_requires_gui_and_ssh(): + assert suites_for_path("container/Containerfile.runner") == ("smoke", "common") + assert suites_for_path("container/Containerfile.kde-runner") == ("kde-smoke",) + + +def test_unknown_path_fails_closed_to_full_set(): + assert suites_for_path("some/new/thing.py") == FULL + + +def test_plan_unions_and_sorts_suites_with_reasons(): + result = plan( + [ + "tests/smoke/features/steps/a.py", + "tests/shared/ssh_config.py", + "docs/x.md", + "", + ] + ) + assert result["vm_required"] is True + assert result["suites"] == ["common", "lifecycle", "smoke"] + assert result["reasons"]["docs/x.md"] == [] + assert "" not in result["reasons"] + + +def test_suite_without_hosted_lane_is_reported_not_dropped(): + result = plan(["tests/hardware/features/steps/udev.py"]) + assert result["vm_required"] is True + assert result["suites"] == [] + assert result["no_hosted_lane"] == ["hardware"] + assert result["composite_action"] == [] + + +def test_every_mapped_suite_is_known(): + for path in [ + ".github/workflows/e2e.yml", + ".github/actions/gnome-e2e/action.yml", + ".github/workflows/iso-validation.yml", + "scripts/extension_validation.py", + "tests/shared/screenshot.py", + ]: + assert set(suites_for_path(path)) <= HOSTED_SUITES | ACTION_LANES + + +def test_cli_reads_stdin_and_prints_json(): + proc = subprocess.run( + [sys.executable, str(SCRIPT_PATH)], + input="tests/common/features/environment.py\nREADME.md\n", + capture_output=True, + text=True, + check=True, + ) + out = json.loads(proc.stdout) + assert out["suites"] == ["common"] + assert out["vm_required"] is True + + +def test_cli_accepts_path_arguments(): + proc = subprocess.run( + [sys.executable, str(SCRIPT_PATH), "docs/runbook.md"], + capture_output=True, + text=True, + check=True, + ) + assert json.loads(proc.stdout)["vm_required"] is False