Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 99 additions & 48 deletions .github/workflows/e2e.yml

Large diffs are not rendered by default.

32 changes: 32 additions & 0 deletions docs/skills/test-authoring/bootc/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,38 @@ results, and on non-dakota images without an explicit `MIGRATION_TARGET`
`If bootc upgrade output indicates image was staged, reboot VM and wait for
SSH` shares the same staged markers (`_UPGRADE_STAGED_MARKERS`) as the gate.

## Detecting the bootc backend (ostree vs composefs)

bootc selects its install backend from the image's install config, which is
unchanged at runtime. The two backends are mutually exclusive, so detect the
backend from a **marker in the image**, never from the image name:

| Backend | Marker in the image | Never ships |
|------------|-------------------------------------------------------|-------------|
| ostree | `/usr/lib/ostree/prepare-root.conf` (always present) | — |
| composefs | `/usr/lib/composefs/setup-root.toml` | ostree marker |

The ostree marker is the **reliable discriminator**: composefs images never
ship `prepare-root.conf`, so its absence selects the composefs backend. We key
on the ostree marker rather than betting on the composefs marker's exact path,
which varies across bootc versions. A single `test -f /usr/lib/ostree/
prepare-root.conf` (falling back to `/etc/ostree/prepare-root.conf`) is enough:

```bash
if test -f /usr/lib/ostree/prepare-root.conf || test -f /etc/ostree/prepare-root.conf; then
# ostree backend: default install + bootupd (unchanged behaviour)
else
# composefs backend: install with --composefs-backend --bootloader systemd
fi
```

This is how `.github/workflows/e2e.yml` chooses the install flags and how the
lifecycle `environment.py` gates scenarios (see `_ostree_backend`). When writing
a scenario, tag it with the backend it needs — `@ostree` for `ostree admin
status`/pin, `@composefs` for the composefs deployment-count check — and the
gate skips it on the wrong backend with a stated reason instead of erroring on
a backend-specific command.

## ostree admin status parsing

```
Expand Down
4 changes: 2 additions & 2 deletions docs/skills/test-authoring/suite-map/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ that issue rather than file another, and close it when it no longer applies. Det

<!-- coverage-snapshot:start -->

554 scenarios across 81 feature files: 439 active, 0 quarantined, 115 `@future`/`@pending`/`@hardware_blocked`
555 scenarios across 81 feature files: 434 active, 6 quarantined, 115 `@future`/`@pending`/`@hardware_blocked`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness (still present, medium): The snapshot is still stale. Coverage snapshot fresh (job 134735388368) and pytest test_committed_snapshot_is_current both fail at this head. Running python3 scripts/update_coverage_snapshot.py on the merge ref changes only this line: there are 0 quarantined scenarios, not 6.

Suggested change
555 scenarios across 81 feature files: 434 active, 6 quarantined, 115 `@future`/`@pending`/`@hardware_blocked`
555 scenarios across 81 feature files: 440 active, 0 quarantined, 115 `@future`/`@pending`/`@hardware_blocked`


| Suite | Scenarios | Active | Quarantined | Pending/Future | Notes |
|---|---|---|---|---|---|
Expand All @@ -213,7 +213,7 @@ that issue rather than file another, and close it when it no longer applies. Det
| hardware | 13 | 13 | 0 | 0 | udev rules syntax validation (ZSA, Apple SuperDrive, Framework 16, AMD s2idle, Wooting, VIIA); emulated peripherals driven by shared SSH steps |
| installer | 3 | 3 | 0 | 0 | post-boot assertions for installer-driven installs (UEFI, Flatpak exclusion, LUKS cmdline) |
| kde-smoke | 13 | 13 | 0 | 0 | Plasma session, D-Bus services, AT-SPI tree, KWin output, one KCM, Dolphin, Konsole, Kickoff; all `@informational` |
| lifecycle | 24 | 22 | 0 | 2 | bootc upgrade / rollback / migration; switch is `@future` (needs a valid alternate image ref; pin activated with settled-deployment barrier) |
| lifecycle | 25 | 23 | 0 | 2 | bootc upgrade / rollback / migration; switch is `@future` (needs a valid alternate image ref; pin activated with settled-deployment barrier) |
| nvidia | 12 | 0 | 0 | 12 | `@future` / `@hardware_blocked` until GPU passthrough exists in the lab |
| security | 7 | 7 | 0 | 0 | cosign verify: projectbluefin (bluefin, lts, dakota) + ublue-os (latest, LTS, DX, nvidia, GTS, DX-nvidia, negative) |
| smoke | 221 | 175 | 0 | 46 | 39 `@pending` flatpak-permission audits blocked on CI never seeding system Flatpaks; MIME handler coverage (Firefox/Papers/Loupe/Text Editor/video); GNOME accessibility (AT-SPI daemon, high-contrast toggle, a11y panel); display fractional/integer scaling via Mutter DisplayConfig; Bluefin desktop identity (Wayland, hardware accel, Dash to Dock); GNOME regression guards in gnome_regression.feature; Dakota sudo-rs privilege and PAM checks |
Expand Down
15 changes: 15 additions & 0 deletions tests/lifecycle/features/bootc.feature
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,21 @@ Feature: bootc upgrade and rollback lifecycle
* SSH command return code is "0"
* ostree status shows two deployments

@lifecycle @composefs
Scenario: bootc status reports at least two deployments after upgrade
# composefs equivalent of the ostree scenario: bootc status has no
# deployment list, but after an upgrade + reboot the booted and rollback
# deployments are both present, so counting non-null booted/staged/rollback
# entries gives >=2. Gated to the composefs backend; skipped on ostree.
* VM is restored to the pre-upgrade baseline deployment
* Run long SSH command: "sudo bootc upgrade"
* SSH command return code is "0"
* bootc upgrade output indicates image was staged
* Reboot VM and wait for SSH
* Run SSH command: "sudo bootc status --format=json"
* SSH command return code is "0"
* bootc status shows at least two deployments

@lifecycle @upgrade @version
Scenario: os-release version changes are tracked after upgrade
* VM is restored to the pre-upgrade baseline deployment
Expand Down
64 changes: 63 additions & 1 deletion tests/lifecycle/features/environment.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"""
import os

from tests.shared.ssh_steps import * # noqa: F401,F403
from tests.shared.ssh_steps import run_ssh

try:
from tests.shared.timing import record_end, record_start
Expand Down Expand Up @@ -49,6 +49,50 @@ def homed_migration_skip_reason(environ=None):
return None


def _gate_backend_scenarios(context, scenario):
"""Skip lifecycle scenarios whose backend does not match this run.

The ostree-only scenarios (ostree admin pin/status) only apply to the
ostree backend, and the composefs deployment-count scenario only applies
to the composefs backend. Skip here, before any step runs, so a
backend-specific command never executes on the wrong backend (for example
`ostree admin status` on a composefs-native image, which errors because
the ostree tree is absent). Kept mutually exclusive with utah's ostree
path: utah has no composefs marker, so it keeps every ostree scenario and
skips the composefs one.
"""
tags = scenario.effective_tags
if "ostree" not in tags and "composefs" not in tags:
return
if "composefs" in tags and "ostree" not in tags and _ostree_backend(context):
# composefs-only scenario on an ostree image.
scenario.skip(
"bootc status deployment count is the composefs equivalent of "
"ostree admin status; this run boots the ostree backend"
)
elif "ostree" in tags and "composefs" not in tags and not _ostree_backend(context):
# ostree-only scenario on a composefs-native image.
scenario.skip(
"ostree admin pin/status are ostree-only; this run boots the "
"composefs backend, which has no ostree tree"
)


def _ostree_backend(context) -> bool:
"""True when the running VM booted via bootc's ostree backend.

bootc selects the backend from the image's install config, which is
unchanged at runtime: ostree images ship /usr/lib/ostree/prepare-root.conf,
composefs images never do. The ostree marker is the reliable discriminator
for the same reason the install step keys on it (see e2e.yml) -- the
composefs marker path varies across bootc versions, but prepare-root.conf is
always present for ostree and never for composefs. Detect from the running
image so the lifecycle scenario gating never keys on the image name.
"""
_, rc = run_ssh(context, "test -f /usr/lib/ostree/prepare-root.conf", timeout=15)
return rc == 0


def before_all(context):
"""Set up SSH connection parameters from environment."""
context.vm_ip = os.environ.get("VM_IP", "")
Expand Down Expand Up @@ -76,6 +120,7 @@ def before_scenario(context, scenario):
print(f"SKIP Scenario {scenario.name}: {reason}", flush=True)
scenario.skip(reason)
return
_gate_backend_scenarios(context, scenario)
context.scenario = scenario
context.command_stdout = ""
context.ssh_rc = None
Expand All @@ -91,6 +136,22 @@ def after_scenario(context, scenario):
record_end(context, scenario)


def _is_backend_mismatch_skip(scenario) -> bool:
"""True when a backend-tagged scenario was skipped for a backend mismatch.

A @ostree or @composefs scenario only runs on its matching backend;
before_scenario skips it on the other backend, leaving status 'skipped'
with exactly one backend tag. This distinguishes that skip from a real
failure (status 'failed') so the qualification gate does not flag a
backend-incompatible scenario as missing evidence.
"""
tags = set(getattr(scenario, "effective_tags", getattr(scenario, "tags", ())))
return (
scenario.status.name == "skipped"
and ("ostree" in tags) != ("composefs" in tags)
)


def _assert_required_lifecycle_passed(features):
"""Reject missing or skipped evidence, including tag-filtered scenarios.

Expand All @@ -108,6 +169,7 @@ def _assert_required_lifecycle_passed(features):
for feature in features
for scenario in feature.walk_scenarios()
if not is_non_runnable(scenario)
and not _is_backend_mismatch_skip(scenario)
and any(
step.name == "bootc upgrade output indicates image was staged"
for step in scenario.steps
Expand Down
17 changes: 17 additions & 0 deletions tests/lifecycle/features/steps/steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -599,6 +599,23 @@ def ostree_two_deployments(context):
)


@step("bootc status shows at least two deployments")
def bootc_two_deployments(context):
"""Verify bootc status reports at least two deployments after upgrade.

bootc status --format=json has no deployment list, but after an upgrade +
reboot the booted and rollback deployments are both present. Count the
non-null booted/staged/rollback entries in .status; a healthy upgraded VM
has booted + rollback (>= 2). Composefs equivalent of the ostree step.
"""
status = _parse_bootc_status(context)
deployments = [status.get(key) for key in ("booted", "staged", "rollback")]
present = sum(1 for deployment in deployments if deployment is not None)
assert present >= 2, (
f"Expected at least 2 bootc deployments, found {present}. status={status}"
)


# ---------------------------------------------------------------------------
# systemd-homed migration steps (dakota:testing → dakota:next)
# MIGRATION_TARGET env var — defaults to ghcr.io/projectbluefin/dakota:next
Expand Down
32 changes: 32 additions & 0 deletions tests/shared/image_family.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,35 @@ def is_dakota_family_image(image: str) -> bool:

def is_utah_image(image: str) -> bool:
return "utah" in image_name(image)


def composefs_backend_from_markers(ostree_prepare_root_present: bool) -> bool:
"""Return True when an image installs with the composefs backend.

bootc selects the backend from the image's install config, which is
unchanged at runtime. It ships exactly one backend marker per image:
* ostree backend -> /usr/lib/ostree/prepare-root.conf (always present)
* composefs backend -> /usr/lib/composefs/setup-root.toml (no ostree marker)
The ostree marker is the reliable discriminator: composefs images never
ship ``prepare-root.conf``, so its absence selects the composefs backend.
We key on the ostree marker rather than betting on the composefs marker's
exact path (which varies across bootc versions). Any image that ships
``prepare-root.conf`` installs with the ostree backend.
"""
return not ostree_prepare_root_present


def is_composefs_image(
image: str,
*,
ostree_prepare_root_present: bool = False,
) -> bool:
"""Return True when ``image`` should install with the composefs backend.

The backend is a property of the image, not its name, so the marker is
supplied by the caller (a runtime ``podman run`` probe of the image in
``e2e.yml``). Kept mutually exclusive with the ostree path: an ostree image
ships ``prepare-root.conf`` (``ostree_prepare_root_present`` is True), so
the predicate is False regardless of any composefs marker.
"""
return composefs_backend_from_markers(ostree_prepare_root_present)
7 changes: 4 additions & 3 deletions tests/unit/test_e2e_lifecycle_firmware_boot.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,13 @@ def test_lifecycle_install_keeps_the_images_own_loader_and_needs_it():
firmware = install.index("FIRMWARE_BOOT=true", lifecycle)
override = install.index('BOOTLOADER_FLAG="--bootloader systemd"')
# Firmware boot needs bootupd in the image; the systemd-boot override
# applies only when firmware boot is off (other suites, or an image
# without bootupd such as dakota, run 43525356388).
# applies only when firmware boot is off (other suites, or an ostree
# lifecycle image without bootupd, run 43525356388). Composefs images
# such as dakota get firmware boot automatically (#1215).
assert lifecycle < probe < firmware < override
guard = install.index('if [[ "${FIRMWARE_BOOT}" != "true" ]]')
assert firmware < guard < override
assert "has no bootupctl" in install
assert "has neither a composefs backend nor bootupctl" in install
fatal = install.index('if [[ ${BOOTC_INSTALL_RC} -ne 0 && "${FIRMWARE_BOOT}" == "true" ]]; then')
assert fatal < install.index("continuing with direct kernel boot")
assert "has no -kernel fallback" in install[fatal:fatal + 400]
Expand Down
56 changes: 56 additions & 0 deletions tests/unit/test_image_family.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
import pytest

from tests.shared.image_family import (
composefs_backend_from_markers,
image_name,
image_under_test,
is_bluefin_image,
is_composefs_image,
is_dakota_family_image,
is_dakota_image,
is_utah_image,
Expand Down Expand Up @@ -137,3 +139,57 @@ def test_is_utah_image_matches_image_name_only(image, expected):
)
def test_dakota_family_follows_dakota_defaults_by_image_name(image, expected):
assert is_dakota_family_image(image) is expected


# composefs backend detection (#1215): composefs-native when the image has no
# ostree prepare-root.conf. bootc's two backends are mutually exclusive, so the
# ostree marker's absence is the decisive discriminator (see image_family.py).
@pytest.mark.parametrize(
("ostree_prepare_root_present", "expected"),
[
(True, False), # ostree prepare-root present -> ostree (utah/bluefin)
(False, True), # ostree prepare-root absent -> composefs (dakota)
],
)
def test_composefs_backend_from_markers(
ostree_prepare_root_present,
expected,
):
assert (
composefs_backend_from_markers(ostree_prepare_root_present)
is expected
)


@pytest.mark.parametrize(
("ostree_prepare_root_present", "expected"),
[
(True, False),
(False, True),
],
)
def test_is_composefs_image_matches_markers(
ostree_prepare_root_present,
expected,
):
assert (
is_composefs_image(
"ghcr.io/projectbluefin/dakota:testing",
ostree_prepare_root_present=ostree_prepare_root_present,
)
is expected
)


def test_composefs_and_ostree_are_mutually_exclusive():
# An ostree image (utah) ships prepare-root.conf, so it is never composefs
# regardless of any composefs marker.
assert is_composefs_image(
"ghcr.io/projectbluefin/utah:testing",
ostree_prepare_root_present=True,
) is False
# A composefs image (dakota) is composefs only when prepare-root is absent.
assert is_composefs_image(
"ghcr.io/projectbluefin/dakota:testing",
ostree_prepare_root_present=False,
) is True
11 changes: 8 additions & 3 deletions tests/unit/test_startup_service_inventory.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,14 @@


def kernel_arg_masks(workflow: str) -> set[str]:
match = re.search(r'^\s*KERNEL_ARGS="([^"]*)"', workflow, re.MULTILINE)
assert match, "KERNEL_ARGS assignment not found in e2e.yml"
return set(re.findall(r"systemd\.mask=(\S+)", match.group(1)))
# The fast lane keeps the CI kargs in CI_BOOT_KARGS and combines them into
# KERNEL_ARGS per backend, so read every *_ARGS assignment and union the
# masks rather than only the literal KERNEL_ARGS line (#1215).
masks = set()
for match in re.finditer(r'^\s*[A-Z_]*ARGS="([^"]*)"', workflow, re.MULTILINE):
masks.update(re.findall(r"systemd\.mask=(\S+)", match.group(1)))
assert masks, "no systemd.mask= kargs found in e2e.yml"
return masks


def symlink_loop_tokens(workflow: str) -> list[str]:
Expand Down
Loading