diff --git a/docs/skills/ci-ops/e2e-workflow/SKILL.md b/docs/skills/ci-ops/e2e-workflow/SKILL.md index f60b519f7..d59bcebc6 100644 --- a/docs/skills/ci-ops/e2e-workflow/SKILL.md +++ b/docs/skills/ci-ops/e2e-workflow/SKILL.md @@ -497,3 +497,4 @@ Load these when you hit the specific topic: - [Installer suite assertions, and why an env-var gate made one unreachable.](references/installer-suite.md) - [Migration inputs `migration-target` / `extra-tags` and the startup_failure they cause when missing.](references/migration-inputs.md) - [GNOME OS guest lane for the developer extension-validation service and its mandatory gate.](references/gnome-extensions-validation.md) +- [Which units the fast e2e lane masks, why, and which still need a startup-qualification lane.](references/startup-service-inventory.md) diff --git a/docs/skills/ci-ops/e2e-workflow/references/startup-service-inventory.md b/docs/skills/ci-ops/e2e-workflow/references/startup-service-inventory.md new file mode 100644 index 000000000..36f4627a4 --- /dev/null +++ b/docs/skills/ci-ops/e2e-workflow/references/startup-service-inventory.md @@ -0,0 +1,50 @@ +--- +name: startup-service-inventory +description: "Which units the fast e2e lane masks, why, and which ones still need a startup-qualification lane." +metadata: + type: reference + audience: agents + maturity: draft +--- +# Startup-service inventory + +The fast hosted lane in `.github/workflows/e2e.yml` masks units before first +boot, through `systemd.mask=` in `KERNEL_ARGS` and through `/dev/null` +symlinks in each deployment's `/etc/systemd/system/`. A green fast-lane run +therefore says nothing about those units. `startup_service_inventory.yaml` +(repo root) classifies every masked unit: + +| Category | Meaning | Fast-lane result counts as qualification? | +|---|---|---| +| `startup_contract` | Shipped first-boot, update or health path that a profile enables | No. Needs a representative startup lane (#1025) | +| `vm_inapplicable` | Needs hardware or a radio hosted QEMU lacks | Not applicable | +| `ci_noise` | Shipped background service outside first boot, masked for speed | No, but not a startup gap | + +`tests/unit/test_startup_service_inventory.py` fails when a unit is masked +without an entry, when an entry is no longer masked, or when an entry's +`mask_sites` disagree with the workflow. Changing a mask in `e2e.yml` means +updating the inventory in the same PR. + +## Gotchas found while building the inventory + +- The symlink loop appends `.service`, `.socket`, `.path` and `.timer` to every + token. A bare token (`cups`) already masks `cups.socket` and `cups.path`; + dotted tokens (`cups.path`, `avahi-daemon.socket`) only create bogus + `*.path.service` names. +- `cups.browsed.service` is not a real unit. Fedora ships + `cups-browsed.service`, which neither mask site touches. +- Bluefin and Bluefin LTS enable `podman-auto-update.timer` with + `systemctl --global`, i.e. for user managers. Both mask sites act on the + system manager, so the shipped user timer is probably still active. Observe + the user unit before claiming it is covered or masked. +- The fast lane replaces `flatpak-preinstall.service` with a runner-cached + sideload of `flatpak-app-list.txt`, and `brew-setup.service` with direct + `brew install`/dnf installs. Those substitutes must not satisfy startup + assertions. + +## Still open (#1025) + +The representative startup lane itself (new job or input, bounded deadlines +per `startup_contract` unit, Flatpak remotes/apps/permissions from the shipped +preinstall, greenboot with a real boot-counting path per #706) is a new CI +lane and needs maintainer design approval before implementation. diff --git a/docs/skills/test-authoring/suite-map/SKILL.md b/docs/skills/test-authoring/suite-map/SKILL.md index b0db1ab8e..a30656c04 100644 --- a/docs/skills/test-authoring/suite-map/SKILL.md +++ b/docs/skills/test-authoring/suite-map/SKILL.md @@ -225,6 +225,10 @@ tag precedence order: `@quarantine` > `@hardware_blocked` > `@future` > `@pendin | `ujust toggle-devmode` group mutation | Medium | Non-interactive contract now exists: `bctl devmode --enable/--disable` (bluefinctl), which `toggle-devmode` execs to when `bctl` is present. Presence + idempotent state-check are covered in `common_devmode.feature`, gated `@requires_bctl` because bluefinctl is a Homebrew preinstall and `brew-setup.service` is masked in QEMU CI (#487). The actual group-mutating branch calls `pkexec usermod`, which requires an authentication agent bound to a real login session — unavailable over plain SSH. Scenario stays `@pending @wip`. Next step: a CI/lab-side non-interactive polkit or session-agent contract for `pkexec`; tracked in `projectbluefin/testsuite#500`. | | uupd conditional suppression | Medium | Battery and metered-network checks are not covered: uupd reads UPower and NetworkManager system-bus properties, while testsuite has no supported isolated state-injection contract. Do not use `/sys/class/power_supply` or GNOME proxy settings as substitutes. Next step: add a lab/image-owned simulation hook, then cover the upstream `/etc/uupd/config.json` contract. | +Fast-lane service masks are classified in `startup_service_inventory.yaml` +(see the [startup-service inventory](../../ci-ops/e2e-workflow/references/startup-service-inventory.md)). +A `startup_contract` unit masked there is not qualified by a green fast-lane run. + ### Skipped-coverage breakdown Why the `@pending`/`@future` scenarios above cannot run today. diff --git a/startup_service_inventory.yaml b/startup_service_inventory.yaml new file mode 100644 index 000000000..39729e15f --- /dev/null +++ b/startup_service_inventory.yaml @@ -0,0 +1,215 @@ +# Startup-service inventory for the fast hosted QEMU lane (.github/workflows/e2e.yml). +# +# e2e.yml masks units twice before first boot: +# - kernel_args: `systemd.mask=` entries in KERNEL_ARGS +# - deployment_symlink: `/dev/null` symlinks written into each deployment's +# /etc/systemd/system/ by the "Mask CI-irrelevant services" +# loop. The loop appends .service/.socket/.path/.timer to +# every token, so a bare token `cups` masks cups.socket +# and cups.path too, while a dotted token such as +# `cups.path` only yields `cups.path.service` and masks +# nothing extra. +# +# Every masked unit must have exactly one entry here, and every entry must still +# be masked; tests/unit/test_startup_service_inventory.py enforces both ways. +# +# Categories: +# startup_contract — a shipped first-boot/update/health path that at least one +# profile enables. The fast lane does NOT qualify it; a +# representative startup lane must run it (#1025). +# vm_inapplicable — needs hardware or a radio that hosted QEMU does not have. +# Masking is deliberate and is not a coverage gap. +# ci_noise — a shipped background service outside the first-boot +# contract, masked only to cut boot time or network use. +# +# Profiles are the image families whose build enables the unit (evidence +# cites the enabling build file). `all` means it is part of the shared base. + +units: + # --- startup_contract ------------------------------------------------------- + brew-setup.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bluefin, bluefin-lts, bazzite] + evidence: + - ublue-os/bluefin build_files/base/17-cleanup.sh (systemctl enable brew-setup.service) + - ublue-os/bluefin-lts build_scripts/40-services.sh (systemctl enable brew-setup.service) + - ublue-os/bazzite Containerfile (systemctl enable brew-setup.service) + fast_lane_substitute: e2e.yml installs brew CLI tools directly, or falls back to dnf + blocks: developer brew/bctl/mise scenarios, common ujust changelogs (#487) + + flatpak-preinstall.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bluefin, bluefin-lts, aurora] + evidence: + - projectbluefin/common system_files/shared/usr/lib/systemd/system/flatpak-preinstall.service + - ublue-os/bluefin build_files/base/17-cleanup.sh (systemctl enable flatpak-preinstall.service) + - ublue-os/aurora system_files/shared/usr/lib/systemd/system-preset/89-aurora.preset + fast_lane_substitute: e2e.yml sideloads flatpak-app-list.txt from a runner cache into the system installation + blocks: smoke flatpak_permissions system-wide installs (x39), common flatpak model/state and services scenarios + + NetworkManager-wait-online.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [all] + evidence: + - flatpak-preinstall.service is ordered After=/Wants=network-online.target, which this unit gates + fast_lane_substitute: none; network-online.target is reached without waiting for a connection + blocks: an honest ordering for flatpak-preinstall.service + + firewalld.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bluefin-lts] + evidence: + - ublue-os/bluefin-lts build_scripts/40-services.sh (systemctl enable firewalld.service) + fast_lane_substitute: none; firewall state is never observed + + podman-auto-update.service: + category: startup_contract + mask_sites: [kernel_args, deployment_symlink] + profiles: [bluefin, bluefin-lts] + evidence: + - ublue-os/bluefin build_files/base/17-cleanup.sh (systemctl --global enable podman-auto-update.timer) + - ublue-os/bluefin-lts build_scripts/40-services.sh (systemctl --global enable podman-auto-update.timer) + fast_lane_substitute: none + notes: >- + The images enable the user-level timer (--global). Both mask sites target + the system manager, so the shipped user timer is probably untouched; the + startup lane must observe the user unit, not assume the mask covers it. + + podman-auto-update.timer: + category: startup_contract + mask_sites: [kernel_args, deployment_symlink] + profiles: [bluefin, bluefin-lts] + evidence: + - see podman-auto-update.service + fast_lane_substitute: none + + greenboot-healthcheck.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bazzite] + evidence: + - ublue-os/bazzite Containerfile (systemctl enable greenboot-healthcheck.service) + fast_lane_substitute: none + notes: >- + Automatic rollback needs a real bootloader boot-counting path; the fast + lane boots a direct kernel, so it cannot stand in for it (#706). + + greenboot-set-rollback-trigger.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bazzite] + evidence: + - ublue-os/bazzite Containerfile (systemctl enable greenboot-set-rollback-trigger.service) + fast_lane_substitute: none + notes: see greenboot-healthcheck.service + + bazzite-hardware-setup.service: + category: startup_contract + mask_sites: [kernel_args] + profiles: [bazzite] + evidence: + - ublue-os/bazzite system_files/desktop/shared/usr/lib/systemd/system/bazzite-hardware-setup.service + - ublue-os/bazzite Containerfile (systemctl enable bazzite-hardware-setup.service) + fast_lane_substitute: none + notes: >- + Runs on every boot, including generic hardware. The startup lane must say + which of its branches are hardware-specific before treating any as inapplicable. + + # --- vm_inapplicable -------------------------------------------------------- + wpa_supplicant.service: + category: vm_inapplicable + mask_sites: [kernel_args] + reason: no Wi-Fi device in hosted QEMU; SLIRP user networking only + + bluetooth.service: + category: vm_inapplicable + mask_sites: [deployment_symlink] + reason: no Bluetooth controller in hosted QEMU + + blueman-mechanism.service: + category: vm_inapplicable + mask_sites: [kernel_args, deployment_symlink] + reason: Bluetooth manager helper; no Bluetooth controller in hosted QEMU + + ModemManager.service: + category: vm_inapplicable + mask_sites: [deployment_symlink] + reason: no cellular modem in hosted QEMU + + fwupd.service: + category: vm_inapplicable + mask_sites: [deployment_symlink] + reason: no updatable device firmware in hosted QEMU (bluefin-lts enables it) + + colord.service: + category: vm_inapplicable + mask_sites: [deployment_symlink] + reason: no calibrated display or printer colour profile to manage + + systemd-udev-settle.service: + category: vm_inapplicable + mask_sites: [kernel_args] + reason: times out (~125s) waiting for hardware that QEMU does not present + + # --- ci_noise --------------------------------------------------------------- + cups.service: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: printing is covered by its own suite, not by first-boot startup + + cups.path: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: see cups.service + + cups.socket: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: see cups.service + + cups.browsed.service: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: network printer discovery; see cups.service + notes: >- + Fedora names this unit cups-browsed.service (hyphen). Neither mask site + matches that name, so the real service is not masked in the fast lane. + + avahi-daemon.service: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: mDNS discovery has no peers on SLIRP networking + + avahi-daemon.socket: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: see avahi-daemon.service + + malcontent-webd.service: + category: ci_noise + mask_sites: [kernel_args] + reason: parental-controls web filter downloads data on boot + + malcontent-webd-update.service: + category: ci_noise + mask_sites: [kernel_args] + reason: see malcontent-webd.service + + malcontent-control.service: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: see malcontent-webd.service + + speech-dispatcherd.service: + category: ci_noise + mask_sites: [kernel_args] + reason: speech synthesis daemon, not part of startup + + gnome-remote-desktop.service: + category: ci_noise + mask_sites: [kernel_args, deployment_symlink] + reason: remote desktop server, not part of startup diff --git a/tests/unit/test_startup_service_inventory.py b/tests/unit/test_startup_service_inventory.py new file mode 100644 index 000000000..d089989d6 --- /dev/null +++ b/tests/unit/test_startup_service_inventory.py @@ -0,0 +1,121 @@ +"""Keep the fast-lane service masks in e2e.yml and the startup inventory in sync. + +The fast QEMU lane masks shipped units before first boot. Each masked unit must +be classified in startup_service_inventory.yaml so a masked startup path is +never mistaken for qualified startup coverage (#1025). +""" + +import re +from pathlib import Path + +import pytest +import yaml + +REPO = Path(__file__).parents[2] +WORKFLOW = REPO / ".github" / "workflows" / "e2e.yml" +INVENTORY = REPO / "startup_service_inventory.yaml" + +CATEGORIES = {"startup_contract", "vm_inapplicable", "ci_noise"} +MASK_SITES = {"kernel_args", "deployment_symlink"} +SYMLINK_SUFFIXES = (".service", ".socket", ".path", ".timer") +UNIT_SUFFIX = re.compile(r"\.(service|socket|path|timer|target|mount)$") + + +def kernel_arg_masks(workflow: str) -> set[str]: + match = re.search(r'^\s*KERNEL_ARGS="([^"]*)"', workflow, re.MULTILINE) + assert match, "KERNEL_ARGS assignment not found in e2e.yml" + return set(re.findall(r"systemd\.mask=(\S+)", match.group(1))) + + +def symlink_loop_tokens(workflow: str) -> list[str]: + match = re.search( + r"for svc in ([^;\n]+); do\n" + r'\s*sudo ln -sf /dev/null "\$\{DEP\}/etc/systemd/system/\$\{svc\}\.service"', + workflow, + ) + assert match, "deployment mask loop not found in e2e.yml" + return match.group(1).split() + + +def symlink_masks(tokens: list[str]) -> set[str]: + """Unit names the loop actually masks: every token gets all four suffixes.""" + return {token + suffix for token in tokens for suffix in SYMLINK_SUFFIXES} + + +@pytest.fixture(scope="module") +def workflow() -> str: + return WORKFLOW.read_text() + + +@pytest.fixture(scope="module") +def inventory() -> dict: + return yaml.safe_load(INVENTORY.read_text())["units"] + + +def test_kernel_arg_parser_reads_masks(): + sample = 'KERNEL_ARGS="rw systemd.mask=a.service systemd.wants=sshd.service systemd.mask=b.timer"' + assert kernel_arg_masks(sample) == {"a.service", "b.timer"} + + +def test_symlink_masks_expand_bare_tokens_only_meaningfully(): + masked = symlink_masks(["cups", "cups.path"]) + assert {"cups.service", "cups.socket", "cups.path", "cups.timer"} <= masked + assert "cups.path.service" in masked # dotted token: no real unit + assert "cups.path" in masked # covered by the bare token, not the dotted one + + +def test_every_masked_unit_is_classified(workflow, inventory): + kernel = kernel_arg_masks(workflow) + tokens = symlink_loop_tokens(workflow) + missing = sorted(kernel - inventory.keys()) + assert not missing, f"masked in KERNEL_ARGS but not inventoried: {missing}" + + for token in tokens: + if UNIT_SUFFIX.search(token): + continue # dotted tokens only add nonexistent *.path.service names + produced = {token + suffix for suffix in SYMLINK_SUFFIXES} + assert produced & inventory.keys(), ( + f"deployment mask loop token {token!r} masks no inventoried unit" + ) + + +def test_inventory_mask_sites_match_workflow(workflow, inventory): + kernel = kernel_arg_masks(workflow) + linked = symlink_masks(symlink_loop_tokens(workflow)) + for unit, entry in inventory.items(): + actual = set() + if unit in kernel: + actual.add("kernel_args") + if unit in linked: + actual.add("deployment_symlink") + assert actual, f"{unit} is inventoried but no longer masked; drop the entry" + assert set(entry["mask_sites"]) == actual, ( + f"{unit}: inventory says {sorted(entry['mask_sites'])}, e2e.yml has {sorted(actual)}" + ) + + +def test_inventory_entries_are_well_formed(inventory): + for unit, entry in inventory.items(): + assert UNIT_SUFFIX.search(unit), f"{unit}: not a systemd unit name" + assert entry["category"] in CATEGORIES, f"{unit}: bad category" + assert set(entry["mask_sites"]) <= MASK_SITES, f"{unit}: bad mask site" + if entry["category"] == "startup_contract": + assert entry.get("profiles"), f"{unit}: startup_contract needs profiles" + assert entry.get("evidence"), f"{unit}: startup_contract needs evidence" + assert "fast_lane_substitute" in entry, ( + f"{unit}: say what the fast lane does instead, or 'none'" + ) + else: + assert entry.get("reason"), f"{unit}: masking needs a reason" + + +def test_first_boot_contract_units_are_not_downgraded(inventory): + """The units #1025 names must stay startup_contract until a lane runs them.""" + for unit in ( + "brew-setup.service", + "flatpak-preinstall.service", + "greenboot-healthcheck.service", + "greenboot-set-rollback-trigger.service", + "podman-auto-update.timer", + ): + assert inventory[unit]["category"] == "startup_contract", unit