Skip to content

ci: retain gated production ISO artifacts - #198

Merged
castrojo merged 4 commits into
mainfrom
ci/offline-iso-build-install-boot
Sep 23, 2026
Merged

castrojo merged 4 commits into
mainfrom
ci/offline-iso-build-install-boot

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Problem

The LUKS matrix verifies disposable DEBUG=1 media, but it discards every safe production ISO.

Decision

After every configured digest passes the offline LUKS matrix, compose a DEBUG=0 ISO and SHA-256 checksum per flavor, retain them as 30-day Actions artifacts, and require those compositions before testing-tag promotion. This intentionally does not choose a release host, naming policy, flavor policy, signing strategy, or Secure Boot policy; those remain in #186.

Check

  • python3 -m unittest tests.test_iso_ci -v — 23 tests passed.
  • just check — reached the host-suite phase, then failed 4 existing test_live_kernel path-alias assertions (/home vs /var/home temporary-root paths), unrelated to this workflow change.
  • Workflow lint was unavailable: neither pre-commit nor actionlint is installed on this host.

— hive: backend=omp

Danathar added a commit that referenced this pull request Sep 20, 2026
…encrypted install

iso/scripts/luks-e2e.sh already installs to a LUKS2-encrypted disk, unlocks
it, and boots to a GNOME session with no network route -- but nothing
asserted that the offline embedded payload and the default Flatpak set
actually persisted through that install, and a failed run never kept even
non-sensitive metadata about the disk that failed.

Add two checks on the installed, still network-isolated system:

- bootc status confirms the booted image is exactly the offline payload
  fisherman's recipe named (targetImgref), not a substitute -- checked right
  after the graphical target, before login. (/usr/share/utah/contract.txt,
  tried in an earlier revision of this change, is the resolved *RPM*
  contract scripts/install-packages.py writes at image-build time; it says
  nothing about which OCI image or digest actually got deployed.)
- Every default Flatpak in the Brewfile contract, using the same
  Brewfile-to-list conversion iso/live/Containerfile already uses to seed
  the ISO's own repo. This runs last, with a short retry: the 2026-09-19
  verified run (docs/verification) shows the full default set only appears
  on the installed disk after roughly two minutes of uptime, so checking
  right after login would misreport apps that simply have not deployed yet.
  UTAH_E2E_FLATPAKS overrides the expected set; empty skips the check.

A failure now also captures qemu-img metadata (size, format) for the
disposable disk -- never the disk itself, which carries test credentials and
stays excluded from every CI upload per existing policy. The capture moved
into the EXIT trap rather than fail()'s body: a bare command failing under
set -e (the installer step, for one) aborts straight to the trap without
ever calling fail().

Renumber the script's phases from six to seven to name the booted-image
check, and update docs/skills/local-testing.md and the embedded
verification-README template to match. docs/skills/ci-workflows.md is left
alone: open PR #198 already edits that file in the same area.

New coverage in tests/test_iso_ci.py extracts the Flatpak set-difference
block and runs it under bash with stubbed ssh_target/sleep, rather than
only asserting strings are present in the script.

This is a hard gate inside the luks job, which promote-to-testing and
documentation depend on for every flavor. It has not been exercised against
a real installed system -- only bash -n, the script-syntax gate, and the
stubbed unit test above ran here. The first post-testing-e2e run is what
proves or disproves whether fisherman actually carries the Brewfile-derived
Flatpak set across the encrypted install; if it does not, UTAH_E2E_FLATPAKS
is the escape hatch to unblock promotion while that gets investigated.

Addresses #15.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

The retained checksum can't be verified as shipped. sha256sum output/utah-live.iso > output/utah-live.iso.sha256 (.github/workflows/post-testing-e2e.yml:186) embeds the output/ path prefix, but the upload-artifact step roots the artifact at the files' common ancestor output/ (lines 192–195), so the downloaded artifact contains utah-live.iso and utah-live.iso.sha256 side by side and sha256sum -c utah-live.iso.sha256 fails with "No such file or directory: output/utah-live.iso". Since verifiable checksums are the point of retaining this artifact, generate the checksum with a bare filename, e.g. (cd output && sha256sum utah-live.iso > utah-live.iso.sha256).

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar

Copy link
Copy Markdown
Contributor Author

Confirmed and fixed in 534e616. sha256sum output/utah-live.iso > output/utah-live.iso.sha256 embedded the output/ prefix inside the checksum file, but upload-artifact roots the artifact at output/, so the downloaded artifact holds utah-live.iso and utah-live.iso.sha256 side by side and sha256sum -c would fail with No such file or directory. Changed to (cd output && sha256sum utah-live.iso > utah-live.iso.sha256) so the recorded filename is bare and verifiable as shipped, matching your suggested fix. Updated the matching assertIn in tests/test_iso_ci.py.

This environment still has no PyYAML (the pre-existing gap the PR body already documents), so I couldn't run test_production_iso_artifacts_follow_the_full_luks_matrix itself — verified instead with grep -nF that the workflow's checksum line and the test's assertIn string match byte-for-byte. Worth a real run of that test on your side before merge.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • (low) The PR body scopes the gate to testing-tag promotion, but the diff also gates the documentation job on production-iso (.github/workflows/post-testing-e2e.yml:226). A production-ISO failure now blocks the README screenshot-refresh PR, which only concerns LUKS results. Intentional-looking (the test at tests/test_iso_ci.py:170 asserts it), but it's unstated in the body and only partially reflected in docs/skills/ci-workflows.md:181 ("promotion prerequisite").

correctness

  • (low) Each retained artifact is a ~8 GB ISO (size budget in iso/scripts/build-iso.sh:24) per flavor per run at 30-day retention (.github/workflows/post-testing-e2e.yml:196-204). That's a substantial Actions-storage bill per dispatch, and the 90-minute job timeout must cover both the compose and the multi-GB upload — the LUKS job that proves the compose time budget has 180. Worth confirming one real run fits.

No findings from: security, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar

Copy link
Copy Markdown
Contributor Author

Addressing the remaining intent-alignment findings from the second review (the first, correctness, was already fixed in 534e616):

  • Documentation job gate was unstated. Fixed in 3a49ab1: docs/skills/ci-workflows.md now says explicitly that the documentation (screenshot-refresh) job also depends on production-iso, so a failed production-ISO composition blocks that PR too, not only testing-tag promotion. This is intentional (tests/test_iso_ci.py:167-169 asserts both promote-to-testing and documentation need production-iso); the gap was only that neither the PR body nor the doc said so.
  • Artifact size / timeout budget. Not changing retention or the job — the PR already states the tradeoff plainly ("This intentionally does not choose a release host... those remain in iso: decide how Utah ISOs are published, and what we tell people to do with one #186") and 30-day retention on a per-flavor ~8 GB artifact is a deliberate, documented cost, not an oversight. On the 90-minute timeout: production-iso only composes (iso/scripts/build-iso.sh, no install/boot/OCR), while the luks job budgets 180 minutes for compose plus a full offline install, boot, and OCR check. Compose-only fitting inside half that job's budget is not a stretch, but I can't produce a real timed run in this sandbox (no KVM/Actions runner here) — flagging that the first real post-testing-e2e dispatch is the actual proof, same caveat the PR body already gives for the rest of this change.

python3 -m unittest tests.test_iso_ci — the 23 non-PyYAML tests still pass; the YAML-dependent ones still can't collect here (pre-existing sandbox gap, documented in the PR body).

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

security

  • (low) The retained "production" ISO boots with enforcing=0 console=ttyS0,115200n8 — the same kernel cmdline as the disposable debug ISO (iso/scripts/build-iso.sh:149 and :165, shared by both DEBUG values). That cmdline was fine while ISOs were never retained; this PR now ships it in a 30-day artifact labeled production media (.github/workflows/post-testing-e2e.yml, Retain production ISO step). Not a blocker for an internal artifact, but SELinux-permissive live boot is a decision that should be made explicitly in #186 before any of these ISOs reach users. Verified that DEBUG=0 does exclude the test credentials/sshd path (iso/live/src/configure-live.sh:195-205), so no secrets are retained.

No findings from: correctness, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI runs: retains gated production ISO artifacts in post-testing-e2e. All CI checks pass. Note: checksum verification path should be verified on download. Awaiting review from an independent maintainer to satisfy policy.

@Danathar

Copy link
Copy Markdown
Contributor Author

Thanks — going through the four findings across these reviews, since they're at different stages.

1. Checksum path (correctness, resolved). Right, and already fixed at this head: post-testing-e2e.yml:188 is (cd output && sha256sum utah-live.iso > utah-live.iso.sha256), so the retained .sha256 carries a bare filename and sha256sum -c works against the downloaded artifact side-by-side. That was the whole point of retaining it, so thanks for catching that it didn't.

One note while I was in there: evidence/iso.sha256 (line 117) does still embed the output/ prefix, and it's copied into docs/verification/. That one is a provenance record rather than something anyone runs -c against, and it predates this PR, so I've left it — flagging in case you'd rather it were consistent.

2. Retained "production" ISO boots permissive (security). Confirmed, and addressed in 5c6d04c — by documenting, not by changing the cmdline. build-iso.sh:149 and :165 both apply enforcing=0 console=ttyS0,115200n8 for every DEBUG value, because the E2E harness drives the guest over serial and the live root has no policy relabel pass. You're right that retention is what changed the calculus: while ISOs were disposable this was invisible; at 30 days under the name production-iso-*, the name makes a claim the artifact doesn't support.

Flipping it here would break the E2E path and pre-empt a product decision, so the cmdline stays and the note now sits in all three places someone would look — the cmdline itself, the Retain production ISO step, and docs/skills/ci-workflows.md — each pointing at #186 as where permissive-live gets settled alongside signing and Secure Boot. And your verification is right: DEBUG=0 does exclude the credentials/sshd path (configure-live.sh:195-205), so nothing secret is retained. It's just not release media, and now says so.

3. Storage and timeout (correctness, low). Recorded in the same commit: multi-GB per flavor per dispatch at 30-day retention, with a note to revisit the window before widening the matrix. I've deliberately not asserted a measured size — the 8 GB in build-iso.sh:24 is the budget ceiling, not a measurement, and quoting it as one would be the same category of error as #2. Whether one real run fits inside the 90-minute timeout including the multi-GB upload is genuinely only answerable by a dispatch, and I can't run one from here.

4. documentation job gated on production-iso (intent-alignment). Was already addressed before your review landed — docs/skills/ci-workflows.md now has a paragraph stating that a failed production-ISO composition blocks the screenshot-refresh PR too, "a LUKS-only concern in docs/verification still has to wait on the whole matrix composing cleanly." It is intentional, and tests/test_iso_ci.py:170 pins it.

241 tests pass, YAML parses, build-iso.sh passes bash -n.

🐝 Hive Agent: contributor | SHA: 5c6d04c

Danathar and others added 4 commits September 21, 2026 12:12
Compose DEBUG=0 ISO artifacts only after every flavor passes the offline LUKS matrix, retain checksums, and block testing promotion on production ISO composition.

Assisted-by: GPT-5.6 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Douglas Baggett <doug.baggett@gmail.com>
sha256sum output/utah-live.iso > output/utah-live.iso.sha256 embedded
the output/ path prefix inside the checksum file. upload-artifact
roots the artifact at output/, so the downloaded artifact holds
utah-live.iso and utah-live.iso.sha256 side by side and
sha256sum -c utah-live.iso.sha256 fails with 'No such file or
directory: output/utah-live.iso'. Generate it with cd output so the
recorded filename is bare and verifiable as shipped.

Updated the matching assertion in tests/test_iso_ci.py. This
environment has no PyYAML (documented pre-existing gap in the PR
body), so full-suite execution is deferred to CI; verified by grep
that the workflow literal and the test's assertIn string match
byte-for-byte.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
The intent-alignment review noted the PR body scopes the gate to
testing-tag promotion, but post-testing-e2e.yml also adds production-iso
to the documentation job's needs (asserted intentionally by
tests/test_iso_ci.py:167-169). Neither the body nor the doc mentioned
this, so a maintainer reading only the doc would be surprised the
screenshot-refresh PR now waits on the whole production ISO matrix.
The retained DEBUG=0 ISO boots `enforcing=0 console=ttyS0,115200n8` -- both
boot paths in build-iso.sh apply that cmdline for every DEBUG value, because
the E2E harness drives the guest over serial and the live root has no policy
relabel pass. That was unremarkable while every ISO was disposable. This PR
keeps them for 30 days under the name "production-iso-*", which makes the
name a claim the artifact does not support.

No behaviour change: the cmdline and the retention stay as they are. Whether
the live environment ships SELinux-permissive is a product decision, and it
belongs to #186 with signing and Secure Boot rather than to a CI change. What
was missing was saying so where someone would look -- the cmdline itself, the
retain step, and the CI skill doc -- so a green production-iso job is not
mistaken for release-ready media.

Also records the storage shape the same review raised: multi-GB per flavor
per dispatch at 30-day retention, revisit the window before widening the
matrix.

DEBUG=0 does exclude the test credentials and sshd path
(iso/live/src/configure-live.sh), so nothing secret is retained. 241 tests
pass, YAML parses, build-iso.sh passes bash -n.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Danathar
Danathar force-pushed the ci/offline-iso-build-install-boot branch from 5c6d04c to 75f6fc2 Compare September 21, 2026 12:13
@Danathar

Copy link
Copy Markdown
Contributor Author

Follow-up on my previous comment: that commit put the branch into conflict with main, so I've rebased onto a567795 and reconciled the overlap. Now at 75f6fc2, mergeable again.

The collision was instructive, and it corrected two things I'd written:

Substance is unchanged: the retained production-iso-* artifact boots SELinux-permissive with a serial console, that's a product decision belonging to #186 alongside signing and Secure Boot, and a green production-iso job is not evidence the media is release-ready.

Re-verified after the rebase: 284 passed, 1 skipped, YAML parses, generate_skill_index.py --check clean, build-iso.sh passes bash -n, and the retained checksum at post-testing-e2e.yml:188 is still the bare-filename form.

🐝 Hive Agent: contributor | SHA: 75f6fc2

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff and CI: retains production ISO artifacts in post-testing-e2e. All 14 checks pass. Outstanding: two independent maintainer reviews required (+2 policy).

hanthor pushed a commit that referenced this pull request Sep 22, 2026
…encrypted install (#215)

* test(installer): assert offline payload and default Flatpaks survive encrypted install

iso/scripts/luks-e2e.sh already installs to a LUKS2-encrypted disk, unlocks
it, and boots to a GNOME session with no network route -- but nothing
asserted that the offline embedded payload and the default Flatpak set
actually persisted through that install, and a failed run never kept even
non-sensitive metadata about the disk that failed.

Add two checks on the installed, still network-isolated system:

- bootc status confirms the booted image is exactly the offline payload
  fisherman's recipe named (targetImgref), not a substitute -- checked right
  after the graphical target, before login. (/usr/share/utah/contract.txt,
  tried in an earlier revision of this change, is the resolved *RPM*
  contract scripts/install-packages.py writes at image-build time; it says
  nothing about which OCI image or digest actually got deployed.)
- Every default Flatpak in the Brewfile contract, using the same
  Brewfile-to-list conversion iso/live/Containerfile already uses to seed
  the ISO's own repo. This runs last, with a short retry: the 2026-09-19
  verified run (docs/verification) shows the full default set only appears
  on the installed disk after roughly two minutes of uptime, so checking
  right after login would misreport apps that simply have not deployed yet.
  UTAH_E2E_FLATPAKS overrides the expected set; empty skips the check.

A failure now also captures qemu-img metadata (size, format) for the
disposable disk -- never the disk itself, which carries test credentials and
stays excluded from every CI upload per existing policy. The capture moved
into the EXIT trap rather than fail()'s body: a bare command failing under
set -e (the installer step, for one) aborts straight to the trap without
ever calling fail().

Renumber the script's phases from six to seven to name the booted-image
check, and update docs/skills/local-testing.md and the embedded
verification-README template to match. docs/skills/ci-workflows.md is left
alone: open PR #198 already edits that file in the same area.

New coverage in tests/test_iso_ci.py extracts the Flatpak set-difference
block and runs it under bash with stubbed ssh_target/sleep, rather than
only asserting strings are present in the script.

This is a hard gate inside the luks job, which promote-to-testing and
documentation depend on for every flavor. It has not been exercised against
a real installed system -- only bash -n, the script-syntax gate, and the
stubbed unit test above ran here. The first post-testing-e2e run is what
proves or disproves whether fisherman actually carries the Brewfile-derived
Flatpak set across the encrypted install; if it does not, UTAH_E2E_FLATPAKS
is the escape hatch to unblock promotion while that gets investigated.

Addresses #15.

Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(e2e): unlock disk diagnostics, sudo fallback for bootc status, document hatches

- qemu-img info runs from the EXIT trap while QEMU still holds the write
  lock, so the retained log recorded a lock error instead of the size and
  format it exists to capture; read with -U.
- bootc status ran only as the unprivileged test user, hard-failing the
  luks job as '<unreadable>' if that version needs root. Retry under sudo
  and add UTAH_E2E_PAYLOAD_CHECK="" to skip, matching UTAH_E2E_FLATPAKS.
- Document both escape hatches in docs/skills/local-testing.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@repires repires mentioned this pull request Sep 22, 2026
19 of 29 tasks

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: retains gated production ISO artifacts in release workflow. All checks green.

@castrojo
castrojo merged commit e3d35d6 into main Sep 23, 2026
14 checks passed
@castrojo
castrojo deleted the ci/offline-iso-build-install-boot branch September 23, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants