Skip to content

ci: add gated offline ISO build and boot validation - #59

Closed
Danathar wants to merge 2 commits into
projectbluefin:mainfrom
Danathar:issue-14-dakota-offline-ci
Closed

Danathar wants to merge 2 commits into
projectbluefin:mainfrom
Danathar:issue-14-dakota-offline-ci

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Fixes #14.

Summary

  • Build a single x86_64 UEFI ISO from the exact image digest after the testing promotion gate, embed a run-specific OCI VFS payload, and record the validated digest for installed-system tracking.
  • Add a QEMU live gate that checks the Utah ready marker, graphical session, serial console, and framebuffer screenshot; Fisherman installs from the embedded containers-storage payload under QEMU restricted networking.
  • Reboot the installed disk with no ISO and no network, assert Utah identity plus graphical.target/GDM/GNOME Shell, and retain serial logs, screenshots, and the failed disk on errors.
  • Upload the ISO, SHA-256 checksum, and manifest only after every gate succeeds; add the local iso-e2e recipe and CI/local-testing documentation.

Validation

  • just check passed with the repository checks, workflow-output validation, and generated skill index checks.
  • Shell syntax, YAML parsing, actionlint, staged whitespace checks, and commit trailer verification passed.
  • The full QEMU/ISO run is CI-only from this workstation because qemu, qemu-img, and the ISO build toolchain are not installed locally.

— hive: backend=codex

Build the exact testing image into UEFI media and exercise the live Fisherman installer from its embedded OCI store before uploading the ISO. Retain serial logs, screenshots, and the failed disk image when a QEMU gate fails.

Assisted-by: GPT-5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicates #67 at the file level

This PR and #67 both create the same two new files from scratch and both rewrite the same three existing ones:

File #59 (this) #67
.github/workflows/iso.yml adds adds
iso/scripts/offline-e2e.sh adds adds
iso/scripts/build-iso.sh modifies modifies
iso/live/Containerfile modifies modifies
iso/live/src/configure-live.sh modifies modifies
Justfile (iso-e2e recipe) modifies modifies

Two independent .github/workflows/iso.yml files and two independent iso/scripts/offline-e2e.sh scripts cannot both land. #41 is a third implementation of the same offline-ISO/LUKS install-and-boot gate (just luks-test, iso/scripts/luks-e2e.sh), and #61 makes a fourth set of edits to iso/scripts/build-iso.sh.

Fixes #14 (this) and Fixes #15 (#67) are adjacent issues that have been answered with overlapping harnesses. A maintainer needs to pick one before any of them merges; otherwise whoever is second resolves conflicts across several thousand lines of QEMU orchestration they did not write.

CI has never run on this branch

Build Utah is action_required (run 34612637466) — fork PRs here need maintainer approval to execute, and none was given. No job ran.

The description is straightforward that the QEMU/ISO run is CI-only and could not be exercised locally ("qemu, qemu-img, and the ISO build toolchain are not installed locally"), which I appreciate. But the consequence is that the new .github/workflows/iso.yml has never executed anywhere — not locally, not in CI. The validation that was done is static:

  • just check, workflow-output validation, generated skill index checks
  • shell syntax, YAML parsing, actionlint, whitespace, commit trailers

That confirms the workflow file parses. It says nothing about whether the gates it defines pass, which is the whole point.

The base is red, so this cannot be validated yet regardless

main@636b48e fails Build Utah at Containerfile:81, run 34654001732:

install: cannot create regular file '/usr/local/libexec/utah-install-packages': No such file or directory
error: recipe `build-ghcr` failed with exit code 1

This PR builds the ISO "from the exact image digest after the testing promotion gate". There is no such digest right now — no image has built successfully on main in days. #56 needs to land (with its ${pair##:*} typo corrected) before an ISO can be produced at all.

Substantive question about the design

The PR promises to "record the validated digest for installed-system tracking" and to upload the ISO, SHA-256 and manifest "only after every gate succeeds". That ordering is the right instinct and directly addresses #24's traceability criterion.

Worth confirming two things explicitly in the description:

  1. What the live gate asserts. "Checks the Utah ready marker, graphical session, serial console, and framebuffer screenshot" — a framebuffer screenshot is evidence, not an assertion. #41 documents that an earlier version of its own equivalent test passed while the VM sat at a UEFI shell, because it inferred success from a dark, static screen. If this gate asserts on Reached target Graphical Interface over serial and a gnome-shell process owned by the session user, say so; if it infers from screen state, it has the same hole.

  2. Failure artifact retention on a public runner. Retaining "the failed disk on errors" is useful for triage, but a failed install disk from an encrypted-install test may contain key material or a known passphrase. Worth stating what is and is not uploaded.

Suggestion

Sequence it: #56 first to unblock the base, then a maintainer decision on which of #41/#59/#67 is Utah's ISO E2E harness, then one approved workflow run so the gates execute once for real. The structure here is reasonable — the blocker is that it is one of three.


Generated by Claude Code

Narrow the failure-artifact upload to serial logs, screenshots and
build metadata instead of the whole utah-iso-output tree. That tree
included the 64G qcow2 install disk and OVMF pflash vars, which add
nothing for triage beyond what the serial logs/screenshots already
show and shouldn't be published as a public CI artifact.

The graphical-boot assertion this PR adds already asserts on a
positive serial-console marker (UTAH_INSTALLED_GRAPHICAL_OK, written
only after systemctl confirms graphical.target/gdm.service and a
gnome-shell process) rather than on framebuffer screenshot state, so
no change was needed there -- screenshots are diagnostic-only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
@Danathar

Copy link
Copy Markdown
Contributor Author

@hanthor — pushed in 3db8e0b:

  • Framebuffer screenshot assertion: verified it's already correct — offline-e2e.sh asserts on a positive serial-console marker written only after graphical.target/gdm.service are active and gnome-shell is confirmed running. Screenshots are diagnostic-only and never gate pass/fail, so this doesn't repeat Build the Utah ISO, install it to an encrypted disk, and prove the desktop starts #41's bug.
  • Failure artifact retention: narrowed. CI was uploading the entire .ci/utah-iso-output/ tree on failure, including the full guest disk image and OVMF pflash vars. Trimmed to logs/screenshots/metadata, matching the success-path artifact list minus the ISO itself.

One thing I can't resolve from this PR: which of #59/#67 becomes Utah's ISO E2E harness. That's a maintainer call, not something either PR can settle on its own — flagging it here so it's visible before either merges.

(The CI-approval and #56-blocked-base points in your review are infra/sequencing, not code — nothing to change on my end there.)

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both verified against the code. Nothing left on your side — this is waiting on a maintainer call, not a change.

The screenshot assertion does not repeat #41's bug — confirmed, and you were right to push back rather than "fix" it. The gating in offline-e2e.sh is a positive three-way check over SSH, not an assertion on a captured image:

if ssh_live 'systemctl is-active graphical.target' 2>/dev/null | grep -qx active && \
    ssh_live 'systemctl is-active gdm.service'     2>/dev/null | grep -qx active && \
    ssh_live 'pgrep -u liveuser -x gnome-shell >/dev/null' 2>/dev/null; then
...
(( live_graphical == 1 )) || fail "live VM did not reach a GNOME graphical session"

grep -qx active anchored whole-line is the detail that makes it sound — it can't be satisfied by inactive or activating appearing as a substring, which is the class of mistake #41 actually made. And the capture_screen calls sit in the failure trap, so a screendump that fails prints to stderr and never influences the exit status. Diagnostic-only, as you said.

Failure-artifact retention — narrowed correctly. The if: failure() block is now the success list minus the ISO, and the reasoning in the comment is the right reasoning:

there's no reason to publish a full guest filesystem image (test creds, host keys, etc.) as a public CI artifact

That second-order point is worth more than the disk-space one. A qcow2 of a completed install, uploaded from a public-repo workflow run, is a published filesystem containing the test user's credentials and the guest's SSH host keys. Dropping it is a small security fix wearing a housekeeping hat. if-no-files-found: warn on the failure path (vs error on success) is also correct — a run that dies before any log exists shouldn't have its artifact step mask the real failure.

Where this stands

You're right that the remaining items aren't code:

  • #59 vs #67 — which becomes Utah's ISO E2E harness. Genuinely a maintainer decision. Both are substantial, and whichever merges second inherits an awkward rebase against a harness it wasn't written for. Flagging it before either lands, rather than after, is the correct move.
  • CI approval — fork PR, action_required. Shared with the other ten in this repo.
  • #56 base sequencing — infra ordering. Unrelated note for whoever picks that up: #56 as it stands rewrites the install loop's ${pair##*:} to ${pair##:*}, which is a different expansion and silently produces the wrong destination name. Worth catching before it merges under anything.

@castrojo @mrbobbytables — this one needs the harness decision (#59 vs #67) and a CI approval run. The diff itself is in good shape; I'll approve once there's a direction to approve it toward.


Generated by Claude Code

@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member

Closing as stale: conflicts with main in the Justfile, iso/scripts/build-iso.sh and the CI docs, all of which moved when the LUKS E2E and its ISO build landed (#41, #84, post-testing-e2e.yml). Much of what #14 asked for is now on main — a single x86_64 UEFI ISO built from the exact digest and boot-validated in CI. What remains (offline install validation) is better opened fresh against the current luks-e2e.sh than rebased through this.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

installer: add Dakota-style offline ISO build, install, and boot CI

2 participants