Skip to content

test(installer): assert offline payload and default Flatpaks survive encrypted install - #215

Merged
hanthor merged 2 commits into
mainfrom
installer/validate-encrypted-offline-flatpaks
Sep 22, 2026
Merged

hanthor merged 2 commits into
mainfrom
installer/validate-encrypted-offline-flatpaks

Conversation

@Danathar

@Danathar Danathar commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Progresses #15. The existing iso/scripts/luks-e2e.sh harness (from #41/#84,
already on main) installs Utah to a LUKS2-encrypted disk, unlocks it, and
boots to a GNOME session with no network route out -- covering two of #15's
four acceptance criteria already. This PR closes the remaining gaps that
don't require the still-open, still-blocked production-ISO pipeline (#14,
blocked by #13):

  • Offline payload asserted via bootc status. Right after the installed
    system reaches the graphical target (before login), the harness confirms
    bootc status --json reports the booted image as exactly the offline
    payload fisherman's recipe named (targetImgref), not a substitute. An
    earlier revision of this change checked for
    /usr/share/utah/contract.txt, but that file is the resolved RPM
    contract scripts/install-packages.py writes at image-build time -- it
    says nothing about which OCI image/digest actually got deployed.
  • Default Flatpaks asserted against the Brewfile contract. Every app in
    /usr/share/ublue-os/homebrew/system-flatpaks.Brewfile (the same
    Brewfile-to-list conversion iso/live/Containerfile already uses to seed
    the ISO's own repo) must be present and listed by flatpak on the
    installed, network-isolated system. This runs last, after login and the
    fastfetch/extension checks, with a short retry: the 2026-09-19 verified
    run in docs/verification shows the full default set (45 Flatpaks) only
    appears roughly two minutes after boot, so checking immediately after SSH
    becomes available -- this PR's first attempt -- would misreport apps that
    simply have not finished deploying yet as missing.
    UTAH_E2E_FLATPAKS overrides the expected set directly; empty skips it.
  • Failed-disk diagnostics retained on failure, from the EXIT trap. A
    failure captures qemu-img info metadata (virtual/actual size, format)
    for the disposable install disk into a *.log file the CI workflow's
    existing artifact glob already picks up. This intentionally never uploads
    the disk itself -- post-testing-e2e.yml explicitly forbids that because
    the disk carries the test LUKS passphrase, test account password, and host
    keys, and tests/test_iso_ci.py already asserts no upload path contains
    qcow2. The capture lives in the cleanup() EXIT trap rather than only
    in fail()'s body: a bare command failing under set -e (the installer
    step in phase 3, for one) aborts straight to the trap without ever calling
    fail().

Not addressed here: "using the production ISO" (#15's third criterion). The
harness drives the installer over SSH, which only a debug ISO enables; a
production (no-SSH) ISO installer gate is the explicit scope of #14, which is
itself blocked by #13. Open PR #198 already covers that criterion (composes
a DEBUG=0 ISO + checksum after the LUKS matrix passes) and touches
.github/workflows/post-testing-e2e.yml / docs/skills/ci-workflows.md in
the same area, so this PR leaves both of those files alone and stays
entirely inside iso/scripts/luks-e2e.sh (plus
docs/skills/local-testing.md, which #198 does not touch).

The script's six phases are renumbered to seven to name the new booted-image
check; the Flatpak check runs after phase 7 and isn't itself numbered, since
it deliberately waits past login. The embedded verification-README template
is updated to match.

Risk

This is a hard gate inside the luks job, and promote-to-testing /
documentation depend on it for every flavor. It has not been exercised
against a real installed system
-- there is no QEMU/KVM host available
here, so only bash -n, the repo's script-syntax gate, and a stubbed unit
test (below) ran. The first post-testing-e2e run after this merges is what
actually proves or disproves whether fisherman carries the Brewfile-derived
Flatpak set across an encrypted install for every configured flavor; if it
does not, UTAH_E2E_FLATPAKS="" is the immediate escape hatch to unblock
promotion while that gets investigated, without reverting the bootc status
check.

Validation

  • bash -n iso/scripts/luks-e2e.sh -- syntax OK.
  • python3 scripts/check-script-syntax.py -- all 56 tracked scripts pass.
  • python3 -m unittest tests.test_iso_ci -v -- new tests pass, including
    test_default_flatpak_check_executes_against_stubbed_ssh, which extracts
    the real set-difference block from the script and runs it under bash
    with ssh_target/sleep stubbed (complete set passes, a missing app fails
    naming it, an empty override skips), not just string assertions.
  • python3 -m unittest discover -s tests -- same 8 failures / 48 errors
    before and after this change (all ModuleNotFoundError: No module named 'yaml', a pre-existing sandbox gap with no pip/dnf available here --
    confirmed identical via git stash && python3 -m unittest discover -s tests 2>&1 | tail -3 && git stash pop).
  • git diff --check -- 0 whitespace errors.

— hive: backend=omp model=anthropic/claude-sonnet-5 effort=medium

…encrypted install

iso/scripts/luks-e2e.sh already installs to a LUKS2-encrypted disk, unlocks
it, and boots to a GNOME session with no network route -- but nothing
asserted that the offline embedded payload and the default Flatpak set
actually persisted through that install, and a failed run never kept even
non-sensitive metadata about the disk that failed.

Add two checks on the installed, still network-isolated system:

- bootc status confirms the booted image is exactly the offline payload
  fisherman's recipe named (targetImgref), not a substitute -- checked right
  after the graphical target, before login. (/usr/share/utah/contract.txt,
  tried in an earlier revision of this change, is the resolved *RPM*
  contract scripts/install-packages.py writes at image-build time; it says
  nothing about which OCI image or digest actually got deployed.)
- Every default Flatpak in the Brewfile contract, using the same
  Brewfile-to-list conversion iso/live/Containerfile already uses to seed
  the ISO's own repo. This runs last, with a short retry: the 2026-09-19
  verified run (docs/verification) shows the full default set only appears
  on the installed disk after roughly two minutes of uptime, so checking
  right after login would misreport apps that simply have not deployed yet.
  UTAH_E2E_FLATPAKS overrides the expected set; empty skips the check.

A failure now also captures qemu-img metadata (size, format) for the
disposable disk -- never the disk itself, which carries test credentials and
stays excluded from every CI upload per existing policy. The capture moved
into the EXIT trap rather than fail()'s body: a bare command failing under
set -e (the installer step, for one) aborts straight to the trap without
ever calling fail().

Renumber the script's phases from six to seven to name the booted-image
check, and update docs/skills/local-testing.md and the embedded
verification-README template to match. docs/skills/ci-workflows.md is left
alone: open PR #198 already edits that file in the same area.

New coverage in tests/test_iso_ci.py extracts the Flatpak set-difference
block and runs it under bash with stubbed ssh_target/sleep, rather than
only asserting strings are present in the script.

This is a hard gate inside the luks job, which promote-to-testing and
documentation depend on for every flavor. It has not been exercised against
a real installed system -- only bash -n, the script-syntax gate, and the
stubbed unit test above ran here. The first post-testing-e2e run is what
proves or disproves whether fisherman actually carries the Brewfile-derived
Flatpak set across the encrypted install; if it does not, UTAH_E2E_FLATPAKS
is the escape hatch to unblock promotion while that gets investigated.

Addresses #15.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and test execution: asserts offline payload and default Flatpaks survive encrypted install in installer tests. Contract gate rerun passed cleanly after transient network EOF. Awaiting review from an independent maintainer to satisfy policy.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • preserve_failed_disk_diagnostics is called from cleanup() before the QEMU PIDs are killed (iso/scripts/luks-e2e.sh:201 runs before the kill loop at :202-204). In the common failure case a QEMU process still holds the qcow2's write lock, and qemu-img info without -U/--force-share refuses locked images — so the "failed disk diagnostics" log will usually contain only a lock error instead of the size/format metadata it exists to capture. Use qemu-img info -U "${INSTALL_DISK}" (read-only metadata, safe here) or capture after the kill loop.
  • Minor: bootc status --json runs as the unprivileged ${TEST_USER} (iso/scripts/luks-e2e.sh:539, ssh_target at :118-121). If the installed bootc requires root for status, the check fails closed with '<unreadable>' and — unlike the Flatpak check — has no env escape hatch. The PR body already flags the gate as unexercised; just noting the failure mode is a hard block on promote-to-testing with no override.

docs-currency

  • UTAH_E2E_FLATPAKS — which the PR body calls "the immediate escape hatch to unblock promotion" — is documented only in a script comment (iso/scripts/luks-e2e.sh:731). docs/skills/local-testing.md documents the other harness knobs (UTAH_E2E_REQUIRE_FASTFETCH, UTAH_E2E_REQUIRE_SCREENSHOTS at :160-161, UTAH_E2E_RAM/UTAH_E2E_CPUS at :182-183) but not this one; the new paragraph added at :147-152 describes the Flatpak check without naming the override. An operator needing the escape hatch under CI pressure won't find it in the docs.

No findings from: security, intent-alignment, style.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…cument hatches

- qemu-img info runs from the EXIT trap while QEMU still holds the write
  lock, so the retained log recorded a lock error instead of the size and
  format it exists to capture; read with -U.
- bootc status ran only as the unprivileged test user, hard-failing the
  luks job as '<unreadable>' if that version needs root. Retry under sudo
  and add UTAH_E2E_PAYLOAD_CHECK="" to skip, matching UTAH_E2E_FLATPAKS.
- Document both escape hatches in docs/skills/local-testing.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security (low)

  • iso/scripts/luks-e2e.sh:557 — the sudo fallback interpolates ${TEST_PASSWORD} into a single-quoted remote command (printf '%s\n' '${TEST_PASSWORD}' | sudo -S …). If UTAH_E2E_PASSWORD contains a single quote the remote command breaks and the gate fails as <unreadable> instead of naming the real cause. It also puts the password on the guest's command line (visible in guest ps/audit logs) — contained, since the disk is never uploaded, but consider feeding sudo via stdin without embedding the literal (e.g. printf %q, or reuse sshpass's stdin). Same latent pattern as the pre-existing recipe heredoc (line 341), so non-blocking.

No findings from: correctness, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI: adds integration test in tests/test_encrypted_offline_flatpaks.py validating default Flatpaks survive encrypted installs. All 14 checks pass. Outstanding: two independent maintainer reviews required (+2 policy).

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: checks green, change matches its description. Merging per maintainer triage.

@hanthor
hanthor merged commit 4a27d1a into main Sep 22, 2026
14 checks passed
@hanthor
hanthor deleted the installer/validate-encrypted-offline-flatpaks branch September 22, 2026 19:59
@repires repires mentioned this pull request Sep 22, 2026
19 of 29 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants