Skip to content

Build the Utah ISO, install it to an encrypted disk, and prove the desktop starts - #41

Merged
hanthor merged 36 commits into
mainfrom
feat/iso-luks-e2e
Sep 16, 2026
Merged

hanthor merged 36 commits into
mainfrom
feat/iso-luks-e2e

Conversation

@hanthor

@hanthor hanthor commented Sep 6, 2026 •

Copy link
Copy Markdown
Member

Utah did not build. This makes it build, install and boot, and adds a test that
proves it rather than asserting it.

What was broken

Five failures stacked behind each other in the live ISO build, each only
visible once the one before it was fixed: no dbus-daemon (Utah carries
dbus-broker, and install-flatpaks.sh starts a system bus); bwrap refused a
user namespace without sys_admin, surfacing as the unrelated-looking
No remote refs found for flathub; the Brewfile was parsed twice, so flatpak
received an empty list and read the remote name as the thing to install; no
rsync exists in the stack for the build cache; and dracut failed on
tpm2-tss and pcsc, which Hummingbird's config names unconditionally but
whose helpers no repository packages. The cleanup trap then ran outside the
user namespace, left ~13G behind and failed the recipe after the ISO had
been written.

The image itself was missing three things Fedora's base image supplies and
Hummingbird's does not: flatpak (so Utah declared a Flatpak contract it had
no way to install), systemd-resolved (no resolver at all), and cryptsetup
(so the installer could not create an encrypted volume).

/usr/local/libexec does not exist in the bootc base, so install into it
fails on the first file and stops the image build at step 10. That one is
independent of the rest and breaks just build-local on main today.

The test, and a correction

just luks-test boots the live ISO, proves the live session is a real GNOME
desktop, installs onto a LUKS2 volume from the ISO's embedded container store
with no network, boots the installed disk, answers Plymouth, logs in at the
greeter and requires a gnome-shell process owned by that user.

An earlier version of this test passed while the machine was sitting at a
UEFI shell.
It found the passphrase prompt by watching for a dark, static
screen, and an EFI shell is a dark, static screen, so it typed the passphrase
at firmware and read the echo as a successful boot. Two causes: the harness
reset OVMF's variables, discarding the NVRAM entry the installer writes — and
this ESP has no \EFI\BOOT\BOOTX64.EFI fallback, so nothing was left to boot.
It now asserts on Reached target Graphical Interface over a serial console,
fails fast on an emergency shell, and checks the session over SSH.

Screenshots of each step land in docs/verification/ and are overwritten by
each passing run. They already earned their place: they show the installer
button reading "Install Bluefin" on a Utah ISO, and the Fedora logo on
Utah's greeter.

Utah also shipped no terminal emulator at all — Bluefin's own image test
asserts ptyxis, but it is absent from Bluefin's package contract because
Fedora's base carries it, and Hummingbird packages neither ptyxis, vte291 nor
gnome-console. BlueShell from the TunaOS remote fills that gap until the
factory builds one.

Verification

Local only; CI has not run any of it. The image resolves with 0 dependency
problems, the ISO is 7.7G and bootable, and the end-to-end test passes with
the screenshots in docs/verification/ as the record. just check is green.

Needs projectbluefin/utah-packages#51 for the packages it installs.

🤖 Generated with Claude Code

https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9

hanthor and others added 7 commits September 6, 2026 11:07
…ontract

Utah's desktop contract lists twenty-odd Flatpak applications and
verifies them, and the image had no flatpak at all: it is absent from
the Bluefin package list because Fedora's base image carries it, and
Hummingbird's bootable base does not. The live ISO build made it
visible -- 'flatpak: command not found' while baking the Brewfile
applications into the squashfs.

utah-packages builds flatpak now, so this is the entry that installs it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
(cherry picked from commit 5c1adf7)
Five separate failures, each hidden behind the one before it.

The live layer needs dbus-daemon and could not find it: install-flatpaks.sh
starts a system bus so flatpak can install into /var/lib, and Utah carries
dbus-broker rather than the reference daemon.

flatpak's own installs go through bwrap, which has to create a user
namespace inside the build container. Rootless podman refuses that
without sys_admin, and the refusal surfaces as an unrelated-looking
"No remote refs found for flathub". projectbluefin/iso passes the same
two flags to build the same layer.

The Brewfile was parsed twice. The Containerfile converts it to bare
application ids before copying it in; install-flatpaks.sh then ran the
same awk over the converted file, matched nothing, and passed an empty
array to flatpak -- which reads the remote name as the thing to install.
It now reads the plain list, and refuses to continue on an empty one
rather than producing that error again.

rsync is in neither Hummingbird nor the factory, so the flatpak build
cache is saved with cp. Replacing the directory is what --delete was
doing: a stale object would otherwise be seeded into the next build and
never collected.

dracut fails on tpm2-tss and pcsc. Hummingbird's config adds them
unconditionally, and a module named there is an error rather than a skip
when its helpers are missing -- tpm2-tools and the pcsc-lite tools exist
in neither repository. The live initramfs boots an unencrypted squashfs
and has no TPM or smartcard volume to open, so it omits both; the
installed system boots the payload image's own initramfs, untouched.

Finally the cleanup trap: the assembly step runs under `podman unshare`
and leaves a squashfs root owned by subordinate uids, so the plain rm
failed on every file, left ~13G behind, and failed the recipe after the
ISO had been written. It now unshares to remove it.

Output: output/utah-live.iso, 8.2G, ISO 9660 'UTAH_LIVE' (bootable).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
(cherry picked from commit 5295c95)
`just luks-test` boots the live ISO in QEMU, installs Utah onto a LUKS2
volume through the installer, boots the result with no ISO attached,
answers Plymouth's passphrase prompt, and fails unless the system comes
up. An image can install perfectly and still be unbootable if its
initramfs cannot open the root volume, and nothing here tested that.

Three things had to be fixed before it could pass.

cryptsetup was missing. Hummingbird ships cryptsetup-libs but not the
tool, so the installer stopped at "missing required host tool". The
initramfs already carried the crypt and systemd-cryptsetup dracut
modules, so unlocking was never the gap -- creating the volume was.

The embedded payload was in the wrong store. The ISO wrote it to
/var/lib/containers/storage and the live layer set a storage.conf
naming that as the graphroot, which did nothing: Hummingbird ships
/usr/share/containers/storage.conf.d/00-vendor.conf, drop-ins are read
after /etc, and the driver stayed overlay with the vendor image store at
/usr/lib/containers/storage. The installer looked there, found nothing,
and fell back to pulling ghcr.io/projectbluefin/utah:testing -- which
has never been published. The payload now goes where the platform looks,
which needs no override and cannot drift from one. `podman images` in
the live session now lists it, read-only, as it should.

The install recipe names no image. An empty "image" with a
"targetImgref" is what makes the installer resolve from
containers-storage instead of pulling; naming the image directly always
pulls, however complete the offline store is.

luks-unlock.py is vendored from projectbluefin/iso (Apache-2.0, same
org) so this does not need a checkout of that repository. Its two socat
calls become plain UNIX-socket writes: socat is in neither Hummingbird
nor Fedora here, and Homebrew cannot bootstrap on this platform either.

Verified locally, end to end:
  cryptsetup luksFormat --batch-mode --type=luks2 /dev/vda3
  mkfs.btrfs -f -L root /dev/mapper/fisherman-root
  ... install from the ISO's embedded store, no network ...
  Plymouth prompt stable -> passphrase sent -> screen changed
  PASS: Utah installed to an encrypted disk, unlocked, and booted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
(cherry picked from commit 6337514)
The previous version of this test passed while the installed system was
sitting at a UEFI shell. It answered the passphrase prompt by watching
the framebuffer for a dark, static screen, and an EFI shell is a dark,
static screen -- so it typed "testpassphrase" at a firmware prompt, saw
the screen change as the characters echoed, and called that a boot. The
serial log recorded the whole thing:

  Shell> tpassphrase
  'tpassphrase' is not recognized as an internal or external command

Two causes. The harness reset OVMF's variables before booting the
installed disk, discarding the NVRAM entry the installer had just
written -- and this ESP has no \EFI\BOOT\BOOTX64.EFI fallback, so there
was nothing left to boot. Real hardware keeps its NVRAM; now so does the
test. And "the screen changed" was never evidence of anything.

So the test asserts what it claims now. It boots the installed system
verbosely on a serial console (dropping "rhgb quiet" from the boot
entry), waits for "Reached target Graphical Interface" rather than for a
colour change, fails fast on an emergency shell, then logs in over SSH,
checks gdm is running, types the password at the greeter, and requires a
gnome-shell process owned by that user before it will pass. It makes the
same assertions about the live ISO, which was previously not checked at
all: a live session that fell back to a text console would have passed
every earlier version of this test.

The installer now creates a real account, because a greeter with no user
has nothing to log in as.

Screenshots go to docs/verification and are overwritten by each passing
run. A run that prints PASS is a claim; the greeter and desktop it
produced are evidence, reviewable without a QEMU host. They have already
earned their place: they show the installer button reading "Install
Bluefin" on a Utah ISO, and the Fedora logo on Utah's greeter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
(cherry picked from commit 8be8392)
Utah had no terminal emulator at all. Bluefin's own image test asserts
ptyxis, but ptyxis is not in Bluefin's package contract -- Fedora's base
image carries it -- and Hummingbird packages neither ptyxis nor vte291
nor gnome-console, so there was nothing to install. BlueShell from the
TunaOS OCI remote fills that gap until this factory builds a terminal.
It stays out of the Brewfile-derived list on purpose: that list is the
parity contract with Bluefin and is compared byte for byte.

Installing from an OCI remote needs more of a session than flathub does.
It spawns its own message bus, which will not start without a machine id
that a container image does not have, and then tries to autolaunch one
without a display. Both are provided for the build only. /root is a
symlink to a /var/roothome that does not exist during a build, so the
dconf cache goes somewhere writable instead.

Both end-to-end VMs now hold a VNC display open. Screendumps come from
the monitor either way, so watching costs nothing and turns "it hung
somewhere in phase 5" into something that can be looked at.

`just try-installed` boots the disk luks-test produced, through a qcow2
overlay so the verified artifact stays as the test left it, with VNC, a
browser console and SSH. It carries the firmware variables over for the
same reason the test does: the installer's NVRAM entry is the only thing
that boots this ESP.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
(cherry picked from commit 56cc561)
/usr/local is a symlink to ../var/usrlocal in the bootc base and
/usr/local/libexec does not exist, so the first install into it fails:

  install: cannot create regular file '/usr/local/libexec/utah-install-packages':
  No such file or directory

which stops the image build at its tenth step. The directory is created
later, as a side effect of the cp -a that follows in the same RUN, so
this only bites the first file. install -D makes the parent rather than
relying on a copy that has not happened yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
anaconda-live pulls anaconda-webui, which pulls cockpit-bridge,
cockpit-ws, cockpit-networkmanager and cockpit-storaged. Cockpit cannot
be delivered on this platform at all: Hummingbird's glib2 disables GHmac
unconditionally, and GHmac is the authentication primitive cockpit-ws
uses. anaconda-webui also pulls slitherer, whose qt6-qtwebengine nothing
in either repository provides.

Utah installs through the bootc-installer live path and has no use for
Anaconda WebUI, so both go on the documented parity debt rather than
failing every image build:

  Problem 2: package anaconda-live requires anaconda-webui, but none of
  the providers can be installed

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
hanthor and others added 4 commits September 6, 2026 12:30
The screenshots and the record now come from a full pass on top of
current main -- image, ISO and the six-phase test -- rather than from
the branch this work was rebased off.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The fastfetch screenshot was a bare desktop because the terminal never
appeared. Launching a GTK application over SSH cannot work: the session
bus refuses to let it register --

  error registering application: GDBus.Error:...AccessDenied
  error: ApplicationRegisterFailed

-- so the process starts, logs its version, and exits without mapping a
window. The attempt also leaves a half-initialised user flatpak repo
behind, and every later flatpak call for that account then fails on

  While opening repository ~/.local/share/flatpak/repo: opendir(objects)

which looks exactly like a broken image and is not one: a user who never
had a flatpak launched at them over SSH is unaffected.

An autostart entry, written before the greeter login, is how a session
is told to run something. It starts inside the session, with a bus and a
display of its own. flatpak --system because the user installation does
not exist and flatpak aborts rather than falling back to the system one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
tuna-os now publishes com.mitchellh.ghostty on the TunaOS remote
alongside org.tunaos.BlueShell, so Utah can carry plain Ghostty instead
of the fork's branding. Same terminal either way -- BlueShell reports
itself as ghostty 1.3.2-ptyxis-port -- but the upstream application id
is the one to ship.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The install phase powered the live VM down, slept eight seconds, and
force-quit it. The next phase then booted the disk that VM had just
written, and when the flush lost the race the installed system died
waiting for its own /boot:

  Timed out waiting for device /dev/disk/by-uuid/f377b45c-...
  Dependency failed for boot.mount - /boot.

The same disk booted perfectly in isolation, and the payload image was
byte-identical between passing and failing runs -- the corruption was
the harness's, not the image's. Phase 4 now waits for the QEMU process
to actually exit, forces it only after a minute, and refuses to continue
if it will not die, because booting a disk something may still be
writing proves nothing either way.

The fastfetch screenshot leads the verification record now, with the
terminal opened by a session autostart entry -- and it exposed one last
self-inflicted artifact: ssh_target runs a login shell, ~/.bashrc now
runs fastfetch, so the session-type query came back wearing ASCII art.
Asked non-interactively instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Copilot AI and others added 16 commits September 6, 2026 06:43
The previous package image omitted the contract RPMs required by the image resolver, causing CI to fail with No match for argument errors. Pin the published repository digest that contains the complete dependency set.

Assisted-by: Claude Sonnet 5 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: GitHub Copilot <223556219+Copilot@users.noreply.github.com>
…two harness bugs

Four changes to the end-to-end harness:

- Phase 6 now asserts each enabled GNOME extension reaches ACTIVE, not
  just that gnome-shell started. A session can come up while an
  extension throws at enable() -- exactly the GNOME 51 breaks this
  checks for (GSConnect's clipboard final-type, Search Light's dropped
  shader API) -- and the old check would have passed anyway. State is
  read authoritatively via gnome-extensions info rather than by grepping
  the journal, because an extension's own deliberate warnings (the
  GSConnect guard logs one when it degrades the clipboard portal) are
  not failures.

- On a green run, the latest fastfetch screenshot -- taken inside the
  booted, encrypted install -- is embedded at the top of README.md
  between markers, so the repo front page always shows a real, recent
  end-to-end pass. The block is replaced in place each run rather than
  stacked, and only a passing run reaches it.

- luks-unlock.py read the serial log as strict UTF-8 and crashed on the
  first non-UTF-8 byte the boot happened to emit
  (UnicodeDecodeError: 0xe2), failing a run that was otherwise fine. It
  now decodes with errors="replace".

- The fastfetch line the terminal screenshot relies on is now guarded to
  interactive shells, so it stops leaking into every ssh_target command's
  output -- which had been corrupting parsed values (the extension state
  came back wearing ASCII art).

Verified live: on an image carrying both extension hotpatches, GSConnect
and Search Light both reach State ACTIVE on GNOME 51; the README embed is
idempotent (re-running replaces, does not stack).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The 60s budget was too short and the harness force-quit mid-flush. The
live guest unmounts /boot at around t+425s; killing it at 60s left an
ext4 the installed system could not probe, so phase 6 kept finding an
emergency shell:

  Timed out waiting for device /dev/disk/by-uuid/26457a53-...
  Dependency failed for boot.mount - /boot
  Started emergency.service - Emergency Shell

The serial log shows the live VM completing the unmount long after the
force-quit, which is why the same disk always booted fine in isolation.
Ten minutes, with a progress line each minute so a slow shutdown does not
look like a hang. Waiting is cheap; a half-written disk costs a whole run
and reads as an image bug.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
'just try-installed' boots a qcow2 overlay backed by install.qcow2, out
of the same work directory luks-test uses. Leaving it running while
phase 3 reinstalls to that disk changes the backing file underneath the
overlay and corrupts what phase 4 boots -- which surfaces as the
installed system losing its /boot and dropping to an emergency shell,
indistinguishable from a real image bug.

The disk itself is provably fine in that state: every UUID matches, the
BLS entry is correct, and the same image boots to graphical.target when
started on its own. Refuse to start rather than produce a failure that
reads as someone else's.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Phase 6 kept finding an emergency shell on a disk that was provably
intact: every UUID matched the BLS entry, /boot mounted cleanly on the
host, and the same image reached graphical.target when booted while the
host was quieter. The failure is timing, not corruption.

Unlocking this LUKS2 volume costs about 30 seconds of argon2 before
userspace starts. On a loaded host udev is still settling when systemd
gives up on /boot at its 45s default:

  Timed out waiting for device /dev/disk/by-uuid/2d793565-...
  Dependency failed for boot.mount - /boot

so the same BLS patch that adds the serial console now also sets
systemd.default_device_timeout_sec=180. Host load should not decide
whether the image passes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The State check anchored on '^State:' but gnome-extensions info indents
its fields, so the pattern never matched and every extension came back
state=unknown -- failing a run in which both were ACTIVE. Match the
indented field and take the last token.

Verified against a booted image: the exact pipeline now returns ACTIVE
for gsconnect@andyholmes.github.io and search-light@icedman.github.com.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
gnome-extensions asks gnome-shell over D-Bus, and the shell answers only
once it has finished loading extensions -- strictly after the
gnome-shell process appears, which is what phase 6 waits for. Reading
once returned an empty State and failed a run in which both extensions
were ACTIVE. Poll for up to two minutes instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The State probe returns empty in harness context while the identical
command returns ACTIVE when run by hand against the same booted VM, so
the read -- not the extension -- is what is unreliable. Both extensions
are verifiably ACTIVE on this image; failing the whole run on a probe
that could not read anything reports the wrong thing.

An empty read is now reported loudly as UNREADABLE, dumps the raw
gnome-extensions output so the next run diagnoses itself, and continues.
A state that is read and is not ACTIVE still fails, which is the case
this check exists for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Three things this run exposed.

The extension state assertion was asserting nothing. `awk '{print \$NF}'`
is inside single quotes at the local shell, so the backslash reached awk
literally and every read failed with "backslash not last character on
line". Every state came back empty, and because an unreadable state had
been made non-fatal, the run passed while checking nothing. Drop the
escape, and make an unreadable state fatal again: two minutes of polling
a shell that is already up is not an inconclusive probe, and a check that
cannot run is worse than a red run because it reads as green.

Emergency mode was diagnosed from the wrong evidence. The test bailed the
instant it saw the string and dumped `tail -40`, but emergency.service
runs sulogin and this image locks root, so sulogin exits at once and the
boot carries on -- those 40 lines are the udev noise that scrolled past
afterwards, not the cause. Record it and keep watching, then report the
units that actually failed, and keep a per-attempt copy of the serial log:
each run overwrote it, so every failure destroyed the evidence for the
one before it.

The verification record collected failure screenshots. Copying
"${SHOTS}"/*.png swept up installed-emergency and luks-failed left behind
by earlier attempts, so the record of what passed accumulated the ways
previous runs had not. Copy the four shots it actually shows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Output of `just luks-test` against the GNOME 51 hotpatch build: installed
to a LUKS2-encrypted disk, unlocked at the Plymouth prompt, logged in at
the greeter, and reached a GNOME session. The fastfetch shot is taken
inside that booted install and now leads the README, so the badge on the
front page only ever reflects a run that actually passed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
…r happens

The intermittent "dropped to an emergency shell" is now diagnosed. The
installed boot dies here:

  Timed out waiting for device /dev/disk/by-uuid/59eae804-...
  [DEPEND] Dependency failed for boot.mount - /boot.
  [DEPEND] Dependency failed for ostree-boot-complete.service

GRUB reads that partition fine -- it loads the kernel from (hd0,gpt2) with
its own filesystem driver -- but udev never probes the ext4, so the by-uuid
node never appears and boot.mount has nothing to wait for.

It is our own doing. Making the entries verbose mounts /dev/vda2 read-write
from the live VM and edits it, and that is the last thing to touch /boot
before the guest is stopped. The guest then never actually stops: every run
recorded so far, the passing ones included, hit the 600s deadline and was
force-killed. So the flush was always a race -- usually won, sometimes not,
which is exactly the intermittency.

Two fixes rather than a longer wait. Sync and flush the block device in the
guest while a working guest is still there to do it, and fail loudly if the
unmount does not take. Then ask the guest to power itself off over ssh; ACPI
is kept only as a fallback, having never once worked on this live session.

This also retires the earlier theory: userspace on a failing boot took 3m03s
against 13.7s on a clean one, which read as host load, and
systemd.default_device_timeout_sec=180 was aimed at the wrong knob -- it only
moved the deadline the missing device blew through.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
… setup

The stock unit orders itself both After=local-fs.target and
Before=systemd-sysusers.service. Both cannot hold: local-fs already comes
after sysusers, via systemd-tmpfiles-setup-dev.service and
local-fs-pre.target. systemd finds the cycle and breaks it by deleting a job
from it, and the job it picks is systemd-tmpfiles-setup-dev.service -- the
one that creates the static device nodes:

  local-fs-pre.target: Found ordering cycle: systemd-tmpfiles-setup-dev
    after systemd-sysusers after rechunker-group-fix after local-fs.target
    after local-fs-pre.target - after systemd-tmpfiles-setup-dev
  Job systemd-tmpfiles-setup-dev.service/start deleted to break ordering cycle

The installed system then comes up with no device nodes and every .device
unit times out together -- /dev/ttyS0 as readily as the /boot filesystem,
which is what rules out a disk fault -- so boot.mount fails on a partition
that is intact and the boot ends in an emergency shell. This is the "black
screens and systems will NOT boot" the stock unit's own header warns about.

Drop the local-fs ordering and keep the half that matters: the fix has to run
before systemd-sysusers.service, and it needs a writable /etc, which
systemd-remount-fs.service provides well before local-fs.target.

Found by the LUKS end-to-end test, which is what it is for.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Assembly writes an uncompressed squashfs root of well over 13G into
"${TMPDIR:-/tmp}", and /tmp here is a tmpfs sized to a fraction of RAM
(7.7G). The build gets most of the way through and then dies in a heap of
"No space left on device" from cp, which reads as a broken image rather
than a full staging area.

Default to /var/tmp, which is real disk, and honour UTAH_ISO_WORKDIR for
anyone who wants it elsewhere. Deliberately not TMPDIR: it is commonly
already /tmp, which is the case being fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
A non-debug ISO fails in the least informative way available: it boots
perfectly, reaches graphical.target and even starts sshd, but the liveuser
password, PasswordAuthentication and root login are all gated behind DEBUG=1
in configure-live.sh. Every login is refused, and the whole symptom is seven
and a half minutes of silence followed by "no SSH from the live ISO".

If the serial log says the live system reached its graphical target, then it
booted and this is a login failure, so name the likely cause and the flag.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
rechunker-group-fix.service repairs the /usr/lib/{group,gshadow} damage the
legacy rechunker (hhd-dev/rechunk) leaves behind when rebasing to an image
built without it. Utah rechunks with chunkah, via projectbluefin/actions'
reusable-build, and has never been through legacy-rechunk -- so the service
has nothing to repair here.

It is not merely redundant. Its own ordering is impossible to satisfy:
After=local-fs.target together with Before=systemd-sysusers.service, when
local-fs already comes after sysusers through systemd-tmpfiles-setup-dev.service
and local-fs-pre.target. systemd breaks the cycle by deleting a job from it,
and picks systemd-tmpfiles-setup-dev.service -- the unit that creates the
static device nodes:

  local-fs-pre.target: Found ordering cycle: systemd-tmpfiles-setup-dev
    after systemd-sysusers after rechunker-group-fix after local-fs.target
    after local-fs-pre.target - after systemd-tmpfiles-setup-dev
  Job systemd-tmpfiles-setup-dev.service/start deleted to break ordering cycle

The installed system then times out every .device unit together -- /dev/ttyS0
as readily as the /boot filesystem, which is what rules out a disk fault --
boot.mount fails on an intact partition, and the boot ends in an emergency
shell. That is the "black screens and systems will NOT boot" the stock unit's
own header warns about.

This replaces the drop-in added in 4437d8f, which fixed the ordering but kept
a unit Utah has no use for. Disable it explicitly rather than leaving it to
the vendor preset, which enables it by default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Copilot <223556219+Copilot@users.noreply.github.com>
hivecommons-hive[bot]
hivecommons-hive Bot previously approved these changes Sep 7, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive auto-merge approval head changed since approval — re-queue required.

github-actions Bot and others added 3 commits September 8, 2026 23:48
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Use Fedora 44 base and matching Utah package repository digest. The previous Fedora 43 base could not resolve the package repository dependency graph.

Assisted-by: Claude Sonnet 5 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Keep the complete package repository digest while using the compatible base image.

Assisted-by: Claude Sonnet 5 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI root cause for the red image builds: dash-to-dock make needs sassc, which no configured repo provides — main is red the same way. Fix PR: #53 (precompiled stylesheet + drift guard). Once that lands, rebase/merge main to go green. Details: #52

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] Build Utah fails identically on main: the Containerfile script-install step breaks on missing /usr/local/libexec parent dirs — fix is pending review in utah#56. This PR goes green after utah#56 merges and CI reruns.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] build_main/build_kernel failures are not caused by this digest bump: utah-install-packages fails with "No match for argument" for many packages (adw-gtk3-theme, alsa-firmware, containerd, fzf, ...) because the Utah package factory does not ship them yet. Blocked on utah-packages#51 / utah-packages#70 (add the required recipes). Re-run after the factory PRs land.


scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The most honest PR in the queue, and the largest coordination problem

The write-up is the best example of verification reporting in this repository, and the self-correction deserves calling out: documenting that an earlier version of the test passed while the machine sat at a UEFI shell — because it detected the passphrase prompt by watching for a dark, static screen, and an EFI shell is a dark, static screen — is the single most useful paragraph anyone has written across these 28 PRs. Every other QEMU harness in the queue (#59, #66, #67) has that same hazard and none of them mention it. The fix (assert on Reached target Graphical Interface over serial, fail fast on an emergency shell, check the session over SSH) is the right one, and it should be the standard the others are held to.

The five stacked ISO failures, the dbus-daemon/bwrap/double-parsed-Brewfile/rsync/dracut chain, and the finding that Utah shipped no terminal emulator at all are all substantive. So is the /usr/local/libexec diagnosis — confirmed, that is what is reddening main right now (run 34654001732, install: cannot create regular file '/usr/local/libexec/utah-install-packages').

The problem is scope: 19 files, 35 commits, +1567

This is at least five independent changes in one PR:

  1. /usr/local/libexec fix — also in #51 (correct) and #56 (typo'd) and #46
  2. Live-ISO build fixes (dbus, bwrap, Brewfile, rsync, dracut, cleanup trap)
  3. Image content additions — flatpak, systemd-resolved, cryptsetup
  4. A terminal emulator via BlueShell from the TunaOS remote
  5. The just luks-test end-to-end harness plus committed screenshots

And it overlaps three other open PRs at file level:

File #41 #59 #67 #61
iso/scripts/build-iso.sh ✓ ✓ ✓ ✓
iso/live/Containerfile ✓ ✓ ✓
iso/live/src/configure-live.sh ✓ ✓ ✓
iso/scripts/luks-unlock.py ✓ ✓
packages/utah.toml (cryptsetup) ✓ ✓

Two independent luks-unlock.py implementations and four sets of edits to build-iso.sh. Someone has to decide which harness Utah keeps.

Suggested split

Items 1–3 are small, independently defensible, and unblock other work:

  • The install -D fix should go to #56 (fix its typo) or #51, not land a third time here.
  • flatpak, systemd-resolved, cryptsetup are a coherent "the Hummingbird base lacks what Fedora's supplies" PR, reviewable on its own in about ten lines of manifest. Note cryptsetup placement — #67 puts it under [services], which configure-services.sh reads; it is a storage tool, not a service.
  • The live-ISO build fixes are the highest-value piece after #56 and stand alone cleanly.

That leaves the LUKS harness as the thing to reconcile with #59/#67, which is a maintainer decision rather than a review comment.

Two questions on the harness itself

BlueShell from the TunaOS remote. Filling the terminal gap is right — Bluefin's own image test asserts ptyxis, and Hummingbird packages neither ptyxis, vte291 nor gnome-console. But pulling from an external remote adds a runtime dependency on a third-party org, which is exactly what #47/#48 are removing from the installer path. Worth noting how it is pinned and verified, and filing the "until the factory builds one" as a tracked issue so it does not become permanent by default.

Committed screenshots. docs/verification/screenshots/*.png overwritten by each passing run means binary churn in git history on every green run. They earned their place this time (the installer reading "Install Bluefin" on a Utah ISO, and the Fedora logo on Utah's greeter, are both real findings). But as a standing practice it will bloat the repo. Consider uploading them as workflow artifacts instead, with only the notable ones committed deliberately.

Status

Build Utah fails (run 34528069597) — largely inherited from the red base. The cross-repo dependency on projectbluefin/utah-packages#51 is noted and needs to land first.

The work here is good. It is the packaging of it that is blocking.


Generated by Claude Code

Danathar added a commit to Danathar/utah that referenced this pull request Sep 12, 2026
…tus schema

Addresses review feedback on projectbluefin#66:

- verify-bootc-lifecycle.py asserted the v1alpha1 bootc status schema in
  its docstring but never checked apiVersion, so a schema drift would
  surface as an opaque KeyError deep in a phase handler. Add an explicit
  check against known-good apiVersion values. Verified live against
  bootc-1.16.7 that the real schema has already moved to
  "org.containers.bootc/v1" (this repo's fixtures still assumed
  v1alpha1), so both are accepted -- the field paths this tool reads are
  unchanged between them.

- UPGRADE_TIMEOUT was declared and documented but never wired to
  anything: `bootc upgrade`/`switch`/`rollback` ran over SSH with no
  timeout at all, so a hang would run until the job's default multi-hour
  CI timeout. Add guest_ssh_bounded() to enforce it, raise the default to
  1200s for TCG (no KVM on hosted runners), and add a job-level
  timeout-minutes backstop to the workflow.

The desktop/identity assertions (systemctl is-active graphical.target +
gdm.service, /etc/os-release grep) and digest transition checks already
assert positive, specific signals from inside the booted VM rather than
inferring from screen state -- verified this doesn't have the UEFI-shell
false-pass failure mode projectbluefin#41 documented in a sibling harness, so no change
needed there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LNgJ8VMdoqzGequbxsP2Kc
@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] CI root cause: the build failures are systemic, not from this PR. utah-install-packages cannot resolve packages from the utah-packages repo (e.g. nothing provides anaconda-webui / python3-dbus-next needed by anaconda-live, hwdata/i2c-tools/libXrandr/libdrm needed by ddcutil) because the utah-packages build pipeline is currently red: its pinned quay.io/packit/packit digest was pruned upstream and Hummingbird ruby3.3/3.4/4.0 default-gems conflict. Fixes are open at utah-packages#87 (packit repin), utah-packages#83 (ruby exclusion), and utah-packages#57 (anaconda-live exclusion). Once those land and the package repo republishes, re-run here — no change needed on this branch.


scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

@hivecommons-hive

Copy link
Copy Markdown
Contributor

[scanner] The build_main/build_kernel failures are upstream of this PR: the image build's dnf transaction fails on the Hummingbird (utah-packages) repo — 31 problems, e.g. nothing provides anaconda-webui needed by anaconda-live-45.22, nothing provides hwdata needed by ddcutil, nothing provides hicolor-icon-theme needed by distrobox. Root cause chain in utah-packages: the SRPM lane is dead (pinned packit image sha256:149e6e06… deleted upstream — fix open in utah-packages#85/#87), so the published repo has stale/broken closure; utah-packages#57 (anaconda-live exclusion), #81 (multimedia closure) and #89 (atomic publish after closure validation) address the content side. Nothing to change in this PR — re-run once the utah-packages fixes land and the repo republishes.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=kimi-k3 copilot=1.0.78

Preserve the verified repository pin and package availability fixes while resolving main into PR #41. Document local screenshot provenance and remaining CI integration.

Assisted-by: GPT-6 via Codex

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hanthor
hanthor merged commit 281735c into main Sep 16, 2026
1 check passed
@hanthor
hanthor deleted the feat/iso-luks-e2e branch September 16, 2026 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants