Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions .github/workflows/lifecycle-e2e.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: Bootc Lifecycle E2E

on:
workflow_dispatch:
inputs:
base_image:
description: "Initial Utah image ref or digest (default: ghcr.io/projectbluefin/utah:testing)"
required: false
type: string
default: "ghcr.io/projectbluefin/utah:testing"
candidate_image:
description: "Candidate Utah image ref or digest to upgrade to (optional)"
required: false
type: string
default: ""
workflow_run:
workflows: ["Post-Testing E2E"]
types: [completed]
branches: [testing]

permissions: read-all

concurrency:
group: utah-lifecycle-${{ github.event.workflow_run.head_sha || github.run_id }}
cancel-in-progress: false

jobs:
lifecycle-test:
name: Validate bootc upgrade and rollback lifecycle
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch == 'testing')
runs-on: ubuntu-24.04
# Generous headroom for TCG (no KVM on hosted runners): base install,
# two guest reboots, an upgrade and a rollback. guest_ssh_bounded()
# in bootc-lifecycle-e2e.sh already fails the phase cleanly on a
# hung upgrade/rollback; this is only the outer backstop.
timeout-minutes: 90
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5
with:
submodules: recursive

- name: Install QEMU and virtualization dependencies
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y -qq \
qemu-system-x86 \
qemu-utils \
ovmf \
ffmpeg

- name: Run bootc upgrade and rollback lifecycle E2E
env:
BASE_IMG: ${{ inputs.base_image || 'ghcr.io/projectbluefin/utah:testing' }}
CANDIDATE_IMG: ${{ inputs.candidate_image || '' }}
UTAH_LIFECYCLE_WORK: /tmp/utah-lifecycle-e2e
run: |
set -euo pipefail
bash scripts/bootc-lifecycle-e2e.sh "${BASE_IMG}" "${CANDIDATE_IMG}"

- name: Publish lifecycle report to step summary
if: always()
run: |
if [[ -f /tmp/utah-lifecycle-e2e/lifecycle-report.md ]]; then
cat /tmp/utah-lifecycle-e2e/lifecycle-report.md >> "$GITHUB_STEP_SUMMARY"
fi

- name: Upload lifecycle test artifacts and diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: bootc-lifecycle-diagnostics
path: |
/tmp/utah-lifecycle-e2e/lifecycle-summary.json
/tmp/utah-lifecycle-e2e/lifecycle-report.md
/tmp/utah-lifecycle-e2e/serial.log
/tmp/utah-lifecycle-e2e/screenshots/*.png
/tmp/utah-lifecycle-e2e/diagnostics/
if-no-files-found: warn
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ just check-desktop-contract # in-image verifiers against a built image
just generate-bootable-image testing # bootc install to-disk -> output/bootable.raw
just boot-vm # QEMU/noVNC; confirm GDM + GNOME Shell render
just iso testing && just boot-iso # live ISO build + boot (see docs/skills/local-testing.md)
just lifecycle-e2e # bootc upgrade & rollback lifecycle validation in QEMU
```

`just check` is the gate CI runs first; a change that fails it fails the whole
Expand Down
14 changes: 14 additions & 0 deletions Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ check:
<(grep -m1 '^ARG BASE_IMAGE=' Containerfile.kernel)
python3 -m py_compile scripts/flavors.py
python3 scripts/flavors.py list >/dev/null
test -f scripts/verify-bootc-lifecycle.py
python3 -m py_compile scripts/verify-bootc-lifecycle.py
test -f scripts/bootc-lifecycle-e2e.sh
bash -n scripts/bootc-lifecycle-e2e.sh
python3 -m unittest tests/unit/test_bootc_lifecycle.py
pip install --quiet pyyaml 2>/dev/null || true
python3 scripts/check_workflow_outputs.py
pip install --quiet jsonschema 2>/dev/null || true
Expand Down Expand Up @@ -405,3 +410,12 @@ secureboot base_name default_tag flavor:
set -euo pipefail
image_name="$(just image_name '{{ base_name }}' '{{ default_tag }}' '{{ flavor }}')"
podman run --rm --entrypoint /bin/sh "localhost/$image_name:{{ default_tag }}" -c 'test -e /usr/lib/modules || test -e /boot'

# Run the bootc upgrade and rollback lifecycle test in QEMU between two
# immutable Utah digests. Asserts initial desktop boot, upgrade to candidate
# digest, upgraded graphical desktop, rollback to initial digest, and produces
# phase diagnostics and screenshots.
lifecycle-e2e base_image="localhost/utah:testing" candidate_image="":
#!/usr/bin/env bash
set -euo pipefail
bash scripts/bootc-lifecycle-e2e.sh "{{ base_image }}" "{{ candidate_image }}"
9 changes: 9 additions & 0 deletions docs/skills/ci-workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,15 @@ Every push to `main` calls `reusable-sync-branches.yml@v1` (same SHA pin)
with `contents: write` and nothing else -- there is no local logic to drift
(workflow, `.github/workflows/sync-main-to-testing.yml`).

## lifecycle-e2e

`.github/workflows/lifecycle-e2e.yml` runs the Dakota-style bootc upgrade and
rollback lifecycle gate. Triggered automatically following `Post-Testing E2E` or
manually via `workflow_dispatch`, it exercises QEMU boot, bootc/uupd update
staging, reboot into candidate digest with graphical validation, rollback, and
restoration of the original baseline digest, publishing structured failure
diagnostics and framebuffer screenshots.

## Verification

```bash
Expand Down
17 changes: 17 additions & 0 deletions docs/skills/local-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,22 @@ QEMU-for-Docker and exposes the noVNC console at the printed URL (comment
above `boot-iso` in `Justfile`), with TPM, UEFI, and `-snapshot` so nothing
persists.

## Bootc Upgrade and Rollback Lifecycle Testing

Validate that atomic updates work end-to-end across deployments:

```bash
just lifecycle-e2e <base_image_or_disk> [candidate_image]
```

The test executes in QEMU:
1. Boots the initial deployment in QEMU, establishes SSH, asserts Utah identity (`ID=utah`, `IMAGE_ID=utah`) and graphical desktop (`graphical.target`, `gdm.service`), and captures the booted digest.
2. Stages an upgrade using bootc/uupd policy (`bootc switch <candidate>` or `bootc upgrade`) and asserts that `.status.staged` exists with the candidate digest.
3. Reboots the VM, asserts that `.status.booted` matches the candidate digest and `.status.rollback` matches the initial digest, and validates desktop smoke checks.
4. Stages a rollback (`bootc rollback`) and asserts that `.status.staged` references the initial digest.
5. Reboots the VM, asserts that `.status.booted` matches the initial digest, and confirms desktop health.
6. Writes structured diagnostics (`lifecycle-summary.json` and `lifecycle-report.md`) detailing active deployments and digests at every phase.

## Verification

```bash
Expand All @@ -141,4 +157,5 @@ just generate-bootable-image testing
just boot-vm # success: GDM appears and GNOME Shell renders in noVNC
just iso testing
just boot-iso # success: live session renders; serial shows UTAH_LIVE_READY
just lifecycle-e2e output/bootable.raw
```
Loading