Skip to content

test: validate bootc upgrade and rollback lifecycle - #66

Closed
Danathar wants to merge 2 commits into
projectbluefin:mainfrom
Danathar:test/bootc-upgrade-rollback-lifecycle
Closed

Danathar wants to merge 2 commits into
projectbluefin:mainfrom
Danathar:test/bootc-upgrade-rollback-lifecycle

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Fixes #17

Summary

  • Add a Dakota-style QEMU lifecycle orchestrator (scripts/bootc-lifecycle-e2e.sh) that installs or boots a known Utah deployment, executes upgrades to a candidate digest using bootc/uupd policy, validates graphical desktop health, rolls back to the previous deployment, and verifies the restored baseline digest.
  • Implement structured lifecycle state tracking and diagnostics (scripts/verify-bootc-lifecycle.py) that parses the bootc status --format=json v1alpha1 schema, records deployment info (booted, staged, rollback, pinned), and generates phase-by-phase diagnostics (lifecycle-summary.json and lifecycle-report.md) identifying the deployment and digest active at each phase.
  • Add focused unit tests (tests/unit/test_bootc_lifecycle.py) exercising status parsing, Utah identity validation, state machine transitions, failure diagnostics reporting, and edge cases.
  • Add lifecycle-e2e recipe to Justfile and update static repository checks to validate the lifecycle scripts and unit tests.
  • Add automated CI lifecycle workflow (.github/workflows/lifecycle-e2e.yml) triggered on workflow_dispatch and following post-testing image promotion to validate the full upgrade and rollback loop in QEMU.
  • Update docs/skills/local-testing.md, docs/skills/ci-workflows.md, and AGENTS.md to document the lifecycle testing workflow and CI gate.

Validation

  • Unit test suite python3 tests/unit/test_bootc_lifecycle.py -v passed (6 tests).
  • Lifecycle simulation python3 scripts/verify-bootc-lifecycle.py simulate passed all 5 phases.
  • Python compilation, bash syntax (bash -n), contract validation, and skill checks (scripts/check-skill-frontmatter.sh, scripts/check-skill-index.sh) passed cleanly.
  • git diff --check and git diff --cached --check passed with 0 whitespace issues.
  • Commit signed off with DCO-compliant Signed-off-by trailer.

— hive: backend=agy

🐝 Hive Agent: contributor | SHA: efec8e1

Add a Dakota-style QEMU lifecycle test that installs or boots a known Utah deployment, upgrades to a candidate digest using bootc/uupd policy, validates the graphical desktop, rolls back to the previous deployment, and captures structured failure diagnostics and screenshots across every phase.

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1410 lines of QEMU orchestration, none of it executed

The lifecycle gap is real — nothing currently proves bootc upgrade and bootc rollback work on a Utah image, and #17 is a legitimate issue. The structure (parse bootc status --format=json, track phase state, emit lifecycle-summary.json and lifecycle-report.md) is a reasonable shape.

But no CI has run against this branch. Build Utah is action_required (run 34635049998) — fork PRs here need a maintainer to approve the workflow run, and none did. So just check never ran, and tests/unit/test_bootc_lifecycle.py never ran, despite the description reporting both passing locally.

The new .github/workflows/lifecycle-e2e.yml is triggered on workflow_dispatch and after post-testing promotion. Neither has fired. The entire QEMU install/upgrade/rollback loop is unexecuted.

The validation that was done doesn't test what the PR claims

From the description:

  • python3 tests/unit/test_bootc_lifecycle.py -v — 6 tests over status parsing, identity validation, state-machine transitions, diagnostics
  • python3 scripts/verify-bootc-lifecycle.py simulate — "passed all 5 phases"
  • bash -n, py_compile, contract validation, git diff --check

The unit tests exercise the parser and the state machine, not the lifecycle. And simulate is the PR's own code asserting against the PR's own synthetic transitions — it proves the state machine is internally consistent, which is worth something, but it is not evidence that bootc upgrade followed by bootc rollback restores the prior deployment on a real Utah image. A simulate mode that passes is compatible with a harness that never boots anything.

This matters because #41 documents exactly this failure mode in a sibling harness: its LUKS test passed while the VM sat at a UEFI shell, because it inferred success from screen state rather than asserting on a positive signal. Please confirm the lifecycle gates assert on something specific from inside the booted VM — bootc status reporting the expected digest as booted, systemctl is-system-running, a gnome-shell process — rather than on the absence of an error.

It cannot be validated yet regardless

main@636b48e fails Build Utah at Containerfile:81:

install: cannot create regular file '/usr/local/libexec/utah-install-packages': No such file or directory
error: recipe `build-ghcr` failed with exit code 1

(run 34654001732). This PR "installs or boots a known Utah deployment" and "upgrades to a candidate digest" — there is no successfully built digest to use. #56 must land first, with its ${pair##:*} typo corrected.

Coordination

This is the fourth QEMU harness in the open queue. #41 (just luks-test), #59 and #67 (two independent iso/scripts/offline-e2e.sh + .github/workflows/iso.yml implementations), and now this one. They share no VM-boot helper, no serial-console assertion library, and no screenshot handling — each reimplements it.

This PR is the least duplicative of the four, since upgrade/rollback is genuinely distinct from install-and-boot. But it will want the same primitives. Worth deciding which harness provides them before four copies exist.

Also pins the v1alpha1 schema. verify-bootc-lifecycle.py parses bootc status --format=json against v1alpha1. That is an explicitly unstable schema; a bootc bump will break this silently or loudly depending on how it is parsed. Worth asserting the schema version explicitly and failing with a clear message when it changes, rather than getting a KeyError deep in a phase handler.

Smaller

.github/workflows/lifecycle-e2e.yml runs QEMU on a hosted runner, which means TCG emulation. Please confirm the phase timeouts account for that — an upgrade plus two boots under TCG is slow, and a timeout that is generous locally with KVM can flake in CI.

Happy to re-review once the base is green and one real run has executed.


Generated by Claude Code

…tus schema

Addresses review feedback on projectbluefin#66:

- verify-bootc-lifecycle.py asserted the v1alpha1 bootc status schema in
  its docstring but never checked apiVersion, so a schema drift would
  surface as an opaque KeyError deep in a phase handler. Add an explicit
  check against known-good apiVersion values. Verified live against
  bootc-1.16.7 that the real schema has already moved to
  "org.containers.bootc/v1" (this repo's fixtures still assumed
  v1alpha1), so both are accepted -- the field paths this tool reads are
  unchanged between them.

- UPGRADE_TIMEOUT was declared and documented but never wired to
  anything: `bootc upgrade`/`switch`/`rollback` ran over SSH with no
  timeout at all, so a hang would run until the job's default multi-hour
  CI timeout. Add guest_ssh_bounded() to enforce it, raise the default to
  1200s for TCG (no KVM on hosted runners), and add a job-level
  timeout-minutes backstop to the workflow.

The desktop/identity assertions (systemctl is-active graphical.target +
gdm.service, /etc/os-release grep) and digest transition checks already
assert positive, specific signals from inside the booted VM rather than
inferring from screen state -- verified this doesn't have the UEFI-shell
false-pass failure mode projectbluefin#41 documented in a sibling harness, so no change
needed there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
@Danathar

Copy link
Copy Markdown
Contributor Author

@hanthor — the three items you raised went up in 010bd91, ~14 minutes after your review, and I never replied to say so. That's on me; here's what's in it, verified against the head rather than from the commit message.

1. The v1alpha1 pin — fixed. parse_bootc_status() now validates apiVersion against an explicit set before touching any field, and fails with a message that says what to do:

bootc status schema mismatch: expected apiVersion to be one of
['org.containers.bootc/v1', 'org.containers.bootc/v1alpha1'], got 'X'. The bootc
version on the guest may have changed its status schema; add the new version to
KNOWN_API_VERSIONS and re-verify the field paths below before trusting this
tool's output.

Also worth flagging: v1alpha1 was not the only value in play. Current bootc (1.16+) reports org.containers.bootc/v1 for the same field layout this tool reads, so pinning only v1alpha1 would have rejected every modern guest. Both are accepted, with two new tests covering it (..._accepts_current_v1_schema, ..._rejects_unknown_schema_version). python3 tests/unit/test_bootc_lifecycle.py — 8 tests, OK (was 6).

2. TCG timeouts — bounded, and at the layer that can report cleanly. BOOT_TIMEOUT=600 and UPGRADE_TIMEOUT=1200, with guest_ssh_bounded() wrapping every bootc switch / upgrade / rollback in a client-side timeout so a hung guest fails the phase with a named message rather than hanging until the job dies. timeout-minutes: 90 on the job is the outer backstop only, and the comment says so.

3. Positive assertions — confirmed, and you were right to ask. The gates assert on specific signals from inside the guest, not on absence of error:

  • systemctl is-active graphical.target and gdm.service, both grep -qx active — whole-line anchored, so activating can't satisfy them (same detail you identified as the thing Build the Utah ISO, install it to an encrypted disk, and prove the desktop starts #41 got wrong)
  • grep -q "^ID=utah" and ^IMAGE_ID=utah in /etc/os-release
  • parse_bootc_status() requires status.booted.image.imageDigest to be a well-formed sha256 and raises if it isn't — a missing field is an error, not a pass
  • digest equality across phases goes through verify-bootc-lifecycle.py assert-transition, so "rolled back to the baseline digest" is an assertion rather than an observation

What I am not claiming. Your central point stands unchanged: none of this has executed in CI. simulate still proves only that the state machine is internally consistent, and I'm not offering it as evidence that bootc upgrade works on a real image. That needs the base green (#56, with the ${pair##:*} expansion corrected) and one approved fork-PR run. I'd rather this sit until then than be merged on unexecuted code.

On the four-harness duplication — agreed, and I don't think this PR should be the one to unify them. Whichever of #41/#59/#67 becomes the VM-boot primitive owner, this one should consume it rather than keep its own copy.

— hive: backend=claude model=claude-opus-5

🐝 Hive Agent: contributor | SHA: 010bd91

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing as stale: conflicts with main's Justfile since Sep 11 with no rebase. The upgrade/rollback lifecycle test (#17) is still wanted, and the QEMU orchestrator here is a reasonable shape for it — the LUKS E2E in iso/scripts/luks-e2e.sh is now the pattern to match (same ssh_target/shot helpers, same post-testing-e2e.yml artifact handling). Reopen against main when you have time and I'll review.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

testing: validate bootc upgrade and rollback lifecycle

2 participants