Repository navigation
test(e2e): list every service the fast test run switches off and why - #1040
Conversation
Add startup_service_inventory.yaml classifying each unit e2e.yml masks (KERNEL_ARGS systemd.mask= and per-deployment /dev/null symlinks) as startup_contract, vm_inapplicable or ci_noise, with per-profile evidence for shipped first-boot paths. A unit test fails when the workflow masks a unit without an entry, or an entry no longer matches the workflow. Document the inventory and gotchas found (misnamed cups.browsed mask, dotted symlink-loop tokens, --global podman-auto-update timer) in a new e2e-workflow reference, linked from the suite-map skill. Refs projectbluefin#1025 Hive-Run: projectbluefin#1025 Hive-Plan: acceptance-1-per-profile-inventory Hive-Spec: testsuite-1025#acceptance-1 Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: codebymarshall <codebymarshall@users.noreply.github.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
There was a problem hiding this comment.
Summary: This change adds a written list of every background service the fast automated test run switches off, says why each one is off, and adds a check so nobody can switch off a new one without listing it. The list and the check are accurate — I re-ran the check and confirmed each listed service against the product build files. Two small things: the PR is worded so that merging it would mark the whole startup-testing request (#1025) as finished, even though the PR itself says only the first of six steps is done; and the new reference page is not linked from the guide it belongs to.
What I need from you: Change "Closes" to "Part of" for #1025, and add the one-line link below.
intent-alignment (medium, in-scope)
PR body ends with Closes #1025, but the same body's "Acceptance status for #1025" marks 1 of 6 boxes done and calls the PR "Partial". Issue #1025 ("Acceptance" section) requires the representative startup lane, Flatpak assertions, red-run proof and hosted runs — none are in this diff (only startup_service_inventory.yaml, one unit test, two docs). Merging as-is auto-closes #1025 and loses the tracking for the remaining lane work. Fix in the PR body:
-Closes #1025
+Part of #1025 (inventory only; startup lane and scenario re-enables remain open)docs-currency (low, in-scope, advisory)
docs/skills/ci-ops/e2e-workflow/SKILL.md:491-499 is the reference list for this skill; every sibling in references/ is linked there, but the new references/startup-service-inventory.md is not (only suite-map/SKILL.md:228 links it). The deferral because #1035/#1037 touch the file is understandable, but a one-line append at line 499 will merge cleanly:
- [GNOME OS guest lane for the developer extension-validation service and its mandatory gate.](references/gnome-extensions-validation.md)
- [Which units the fast lane masks, why, and which still need a startup-qualification lane.](references/startup-service-inventory.md)
Verified: all 23 systemd.mask= units at .github/workflows/e2e.yml:538 and the 4 symlink-only tokens at :631 map to the 27 inventory entries with correct mask_sites; pytest tests/unit/test_startup_service_inventory.py 6 passed at fe3412f, and the new-thing.service mutation fails as described; 17-cleanup.sh, 40-services.sh and the bazzite Containerfile enable the units exactly as the evidence: fields cite.
No findings from: correctness, security, style.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
@codebymarshall this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (3 waiting): fixes are handed out before new work as soon as you have capacity. |
|
@codebymarshall the hive has adopted this PR ( |
Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
|
The hive adopted this PR ( Blocker: 1 commit(s) on Moving to |
There was a problem hiding this comment.
Summary: This change adds a written list of every background service our fast desktop test run switches off, sorted into "real first-boot step we are skipping", "needs hardware a virtual machine lacks", and "just noise", plus an automatic check that keeps the list and the test workflow in agreement. I read the list, the check, and the workflow it reads, and ran the check including a deliberate break. It works as described. Two small things in the list and the write-up should be tidied before merge; neither needs a maintainer decision.
What I need from you: Fix the two small items below, or say why not.
Reviewed at head 8525a09. Linked contract: #1025 (the handed-down reference #1037 is an open PR, not an issue; judged against #1025 "Reviewed per-profile inventory distinguishes deliberately hardware-inapplicable units from shipped startup paths that must run").
correctness
- Low, in-scope —
startup_service_inventory.yaml:61-66listsfirewalld.servicewithprofiles: [bluefin-lts]only. Fedora's90-default.presetenablesfirewalld.serviceon every Fedora-based image, so unless Bluefin/Aurora/Bazzite builds explicitly disable it, the profile list is incomplete and the future startup lane would skip firewall observation on those profiles. I could not check the image repos within this review, so please confirm; if they do not disable it:firewalld.service: category: startup_contract mask_sites: [kernel_args] profiles: [all] evidence: - fedora-release 90-default.preset (enable firewalld.service); not disabled by any profile build - ublue-os/bluefin-lts build_scripts/40-services.sh (systemctl enable firewalld.service) - Verified:
pytest tests/unit/test_startup_service_inventory.py6 passed; addingsystemd.mask=zz.servicetoKERNEL_ARGS(e2e.yml:538) fails withmasked in KERNEL_ARGS but not inventoried: ['zz.service'].generate_skill_index.py --checkandvalidate_docs.pypass.
intent-alignment
- Info, in-scope — PR body says "I didn't edit
e2e-workflow/SKILL.md", but the diff adds a line atdocs/skills/ci-ops/e2e-workflow/SKILL.md:500. The edit is right (the reference should be linked); the body is stale. Please drop that sentence so the description matches the change.
docs-currency
- Low, out-of-scope — The bogus
cups.browsed.servicename the inventory documents (startup_service_inventory.yamlnotes,references/startup-service-inventory.md) is also hard-coded intests/smoke/features/steps/system_health_steps.py:21andtests/kde-smoke/features/steps/steps.py:35(IGNORED_FAILED_UNITS_IN_VM). Those ignore-lists are a second omission mechanism (tuned.service,bootloader-update.serviceetc. are tolerated as failed) that the inventory does not cover. Backlog: either reference the inventory from those lists or fold "tolerated-failed" units into it.
No findings from: security, style.
Confidence: 3/5 (needs attention) — 2 perspectives requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive filed review findings that do not block this PR as backlog items:
Cap: 10 backlog item(s) per PR per day. |
|
@codebymarshall this PR has a new head without confirmed hive push evidence, so the hive has handed this PR back to you ( |
|
@codebymarshall this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (4 waiting): fixes are handed out before new work as soon as you have capacity. |
Resolve docs/skills/ci-ops/e2e-workflow/SKILL.md: projectbluefin#1040 and this PR each appended a reference link at the end of the list; keep both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnzFs53Qcer1xDySyxv1Vj Signed-off-by: Danathar <berths-purger-0g@icloud.com>
Merging main brought projectbluefin#1040's reference link next to this PR's, taking SKILL.md to 501 lines (cap 500). Join one two-line sentence; same words. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SnzFs53Qcer1xDySyxv1Vj Signed-off-by: Danathar <berths-purger-0g@icloud.com>
Summary: Our automated desktop tests turn off about two dozen background services before the test computer starts, to save time. Some of those are just irrelevant in a virtual machine, but others are real setup steps the product runs on a user's first boot, like installing default apps and the Homebrew package manager. Until now nothing recorded which was which, so a passing run could be mistaken for proof that first-boot setup works. This adds that list, plus a check that fails if someone turns off a new service without listing it.
What I need from you: Review the list. The separate test run that actually exercises these setup steps (the rest of #1025) is a new CI lane, which needs your design decision before anyone builds it.
What changed
startup_service_inventory.yaml(repo root, next toselinux_allowlist.yaml): one entry per unit thate2e.ymlmasks, in eitherKERNEL_ARGSsystemd.mask=or the per-deployment/dev/nullsymlink loop. Each entry has a category:startup_contract(9):brew-setup,flatpak-preinstall,NetworkManager-wait-online,firewalld,podman-auto-update.{service,timer},greenboot-healthcheck,greenboot-set-rollback-trigger,bazzite-hardware-setup. Each lists the profiles that enable it, evidence (the enabling build file in the image repo), and what the fast lane does instead.vm_inapplicable(7): wpa_supplicant, bluetooth, blueman-mechanism, ModemManager, fwupd, colord, systemd-udev-settle.ci_noise(11): cups*, avahi*, malcontent*, speech-dispatcherd, gnome-remote-desktop.tests/unit/test_startup_service_inventory.py: parses both mask sites out ofe2e.ymland fails on any unit that is masked but not listed, listed but no longer masked, or listed with the wrong mask sites. It also checks that each entry is well-formed, and that the units the issue names staystartup_contract.docs/skills/ci-ops/e2e-workflow/references/startup-service-inventory.md, linked fromsuite-map/SKILL.md. I didn't edite2e-workflow/SKILL.mdbecause open PRs feat(gnome-e2e): check the GNOME OS test machine before it starts and record which build was tested #1035 and feat(e2e): keep Linux security protections on during desktop tests and fail on blocked actions #1037 both touch it.No change to
e2e.yml, inputs, or scenario tags.Findings recorded (not fixed here, because
e2e.ymlis human-merge and #1037 is editing it)cups.browsed.serviceis not a real unit name. Fedora shipscups-browsed.service, so that mask does nothing.cups.path,cups.socket,avahi-daemon.socket) only create bogus*.path.servicenames. They are redundant, because the bare tokens already cover them.podman-auto-update.timerwithsystemctl --global(user managers). Both mask sites act on the system manager, so the shipped user timer is probably still active. This needs checking in the startup lane.Acceptance status for #1025
startup_contractunits in hosted QEMU with bounded deadlines; produce Flatpak remotes/apps/permissions from the shipped preinstall and assert them; exercise Homebrew and greenboot with a real boot-counting path (#706); show a red run on a deliberately broken service. This needs maintainer design approval for a new lane or input, as the issue's Boundaries section says.Evidence
pytest tests/unit/test_startup_service_inventory.py: 6 passed.systemd.mask=new-thing.servicetoKERNEL_ARGSfails withmasked in KERNEL_ARGS but not inventoried: ['new-thing.service']. Addinguupdto the symlink loop fails withdeployment mask loop token 'uupd' masks no inventoried unit.ruff check tests/ --select E,F,W --ignore E501clean;pytest tests/unit/2212 passed, 1 xfailed;validate_docs.pypassed;generate_skill_index.py --checkin sync. No.featurefiles or scenario totals changed, so there was no dry-run and no coverage snapshot to regenerate.Note on trailers: commits carry
Assisted-by: Claude Opus 5.5 via Claude Codeand a Claude co-author line instead of the Copilot co-author line in AGENTS.md, because Copilot did not author this.Refs #1025. Partial: the startup-qualification lane and the scenario re-enables remain open, pending a maintainer design decision.
🤖 Generated with Claude Code
— hive: backend=claude model=claude-opus-5-5
Part of #1025
🐝 Hive Agent:
contributor| SHA:13da6ce8