Skip to content

fix(e2e): deliver the kickstart through an initramfs overlay - #74

Merged
castrojo merged 3 commits into
mainfrom
fix/e2e-kickstart-initrd-overlay
Sep 29, 2026
Merged

castrojo merged 3 commits into
mainfrom
fix/e2e-kickstart-initrd-overlay

Conversation

@Danathar

@Danathar Danathar commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What you see today

Both Stable main ISO E2E / e2e-test and Stable NVIDIA ISO E2E / e2e-test have failed on every run since 2026-08-07 (first seen on the #70 runs, most recently on #69's run 36079133044). The job log only says:

Installer did not report completion within 1800s

The installer-serial.log in the iso-e2e-proof-* artifact shows what actually happened. The initrd stage boots fine and hands over to anaconda, which starts its DBus modules and then aborts:

[  210.220863] dbus-daemon[1644]: Activating service name='org.fedoraproject.Anaconda.Modules.Subscription' ...
[  270.039468] ...Subscription[1731]: subscription: The required rhsm systemd service is not available. The Subscription module won't be started.
[  810.220537] dbus-daemon[1644]: Failed to activate service 'org.fedoraproject.Anaconda.Modules.Subscription': timed out (service_start_timeout=600000ms)
[  810.315834] ...Boss[1650]: Skip org.fedoraproject.Anaconda.Modules.Subscription. The optional module has failed to start, so it won't be available during the installation.
13:01:17 Kickstart file /run/install/ks.cfg is missing.
[  810.482723] anaconda[1393]: anaconda: stdout: Kickstart file /run/install/ks.cfg is missing.
Pane is dead (status 1, Sat Sep 26 13:01:19 2026)

The August 7 run (iso-e2e-proof-31227505330) fails at the identical line at 802 s.

Cause

The harness boots the ISO's extracted vmlinuz/initramfs.img directly with inst.ks=http://10.0.2.2:18080/kickstart.ks, and serves the kickstart from a local python3 -m http.server. Nothing in the guest ever fetches that URL:

  • In anaconda, inst.ks= is a store_const boot option. Whatever its value, it only sets ksfile=/run/install/ks.cfg (pyanaconda/argument_parsing.py), and stage 2 aborts if that file does not exist (pyanaconda/startup_utils.py, the exact message above).
  • Fetching the URL and placing the file there is the job of anaconda's dracut module: fetch-kickstart-net.sh downloads it and parse_kickstart() in dracut/anaconda-lib.sh ends with cp "$parsed_kickstart" /run/install/ks.cfg.
  • That module is optional (check() returns 255 in dracut/module-setup.sh) and only lands in an initramfs when anaconda-dracut is installed where dracut runs. Titanoboa builds the initramfs with dracut --no-hostonly --add "dmsquash-live dmsquash-live-autooverlay" in the pulled rootfs (Justfile, initramfs recipe at the pinned 840217d), and the initrd part of the serial log has no anaconda hooks at all: no anaconda: lines, no kickstart fetch, no /run/install.

So the URL was never requested, /run/install/ks.cfg was never written, and anaconda gave up. The 1800 s budget then ran out.

Fix

tests/iso/e2e.sh now does the hand-off the dracut module would have done, without touching the ISO:

  1. Renders the kickstart to e2e-output/kickstart.ks (kept in the proof artifact) and builds a small overlay tree: /etc/e2e/ks.cfg plus e2e-kickstart.service, an initrd unit (DefaultDependencies=no, ConditionPathExists=/etc/initrd-release, Before=initrd-cleanup.service initrd-switch-root.target, wanted by initrd.target through a symlink in initrd.target.wants/) whose only job is mkdir -p /run/install && cp /etc/e2e/ks.cfg /run/install/ks.cfg. /run is carried across switch-root by systemd, which is the same property anaconda's own module relies on (and why /run/initramfs/live is still mounted in stage 2 in the same log).
  2. Packs the tree with cpio -H newc -R 0:0, gzips it, and appends it to the ISO's initramfs.img; the combined file goes to -initrd. The kernel unpacks concatenated initramfs segments in turn; a compressed trailer needs no 4-byte alignment against the ISO's zstd stream, which is exactly how virt-install --initrd-inject appends files.
  3. Keeps inst.ks= so anaconda still runs with --ks, but points it at file:/etc/e2e/ks.cfg. That value is ignored by stage 2 today; if a future initramfs does carry the anaconda module, parse-anaconda-kickstart.sh will parse that same embedded file instead of needing network in the initrd.
  4. Drops the HTTP server, the curl readiness loop and E2E_HTTP_PORT, since nothing consumed them.

Alternative considered: remastering the ISO with ks.cfg at its root and inst.kickstart=/run/initramfs/live/ks.cfg (--kickstart takes a path). That works too but means rewriting a 6 GB ISO on the runner per job; the overlay is a few KB and leaves the artifact under test untouched.

Time budget

The serial log also shows anaconda's optional Subscription module never activates in this image: the module decides not to start (rhsm.service is not installed) but dbus-daemon still waits its full service_start_timeout of 600 s before the Boss skips it. That happens before the kickstart is read, so it will still happen with the kickstart in place, and an install can only begin ~13.5 min after boot. E2E_INSTALL_TIMEOUT therefore defaults to 2700 s now (was 1800); the job's 90 min limit is unaffected. Forbidding that module would have to happen in the ISO's anaconda profile (iso_files/configure_iso_anaconda.sh), which is a product change and out of scope here.

docs/qa.md and docs/skills/test-iso/SKILL.md record the delivery mechanism and the budget note. The workflow installs cpio explicitly.

Verification

  • bash -n tests/iso/e2e.sh and shellcheck tests/iso/e2e.sh (clean at all severities, same as before).
  • Ran the overlay block of the script verbatim against a synthetic zstd-compressed cpio initramfs: cpio -itv lists etc/e2e/ks.cfg, etc/systemd/system/e2e-kickstart.service and the initrd.target.wants symlink, all owned by root; the output file is byte-for-byte the original zstd stream followed by a gzip stream that decompresses to a 070701 newc archive; unpacking both segments in order yields the merged tree with the symlink resolving and ks.cfg identical to the rendered kickstart.
  • systemd-analyze verify on the generated unit passes against a host systemd.
  • YAML loads; no trailing whitespace; files end with a newline.

Not verified locally: the boot itself. Booting the ISO in QEMU is verified only by this PR's own Stable main ISO E2E / Stable NVIDIA ISO E2E checks; the thing to look for in their serial logs is e2e-kickstart.service in the initrd stage and anaconda proceeding past kickstart parsing instead of Kickstart file /run/install/ks.cfg is missing.

Progress: kickstart delivery is proven, next failure is the image reference

Run 36276284373 (head 3c99501, job 108503421574, installer-serial.log in iso-e2e-proof-36276284373) shows the overlay working end to end:

[   18.642392] systemd[1]: Starting e2e-kickstart.service - Stage the E2E kickstart at /run/install/ks.cfg for anaconda...
[   19.170710] systemd[1]: Finished e2e-kickstart.service - Stage the E2E kickstart at /run/install/ks.cfg for anaconda.
[  925.135637] ...Payloads[1771]: ...Configuration is set to 'RPMOSTreeContainerConfigurationData(... transport='containers-storage', url='ghcr.io/ublue-os/bluefin:stable')'.
[ 1018.968759] ...Payloads[1771]: INFO:program:Running... ostree container image deploy --sysroot=/mnt/sysimage --image=ghcr.io/ublue-os/bluefin:stable --transport=containers-storage --no-signature-verification
[ 1021.473939] bootc[4670]: Fetching ostree-unverified-image:containers-storage:ghcr.io/ublue-os/bluefin:stable
[ 1022.418621] ...error: Performing deployment: Preparing import: Fetching manifest: failed to invoke method OpenImage: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]ghcr.io/ublue-os/bluefin:stable" does not resolve to an image ID

No Kickstart file /run/install/ks.cfg is missing. any more; anaconda parsed the kickstart, partitioned vda, ran ostree admin init-fs / os-init, and died in the payload step. (The Subscription module still burned its 600 s in that run, at 920 s.) The NVIDIA job hit the 2700 s budget instead — it never got that far under TCG.

What the ISO actually contains

Pulled iso-36276284373-stable-main-amd64 and read the live squashfs (unsquashfs on /LiveOS/squashfs.img):

  • var/lib/containers/storage/overlay-images/images.json holds exactly one image, d1470a5f…, "names": ["ghcr.io/ublue-os/bluefin:stable"] — the name titanoboa's rootfs-include-container recipe produces with podman pull ghcr.io/ublue-os/bluefin:stable (pinned Justfile 840217d, and the build log shows that very pull). /usr/share/anaconda/interactive-defaults.ks in the ISO uses the identical ostreecontainer --url=ghcr.io/ublue-os/bluefin:stable --transport=containers-storage line.
  • usr/lib/containers/storage/ (the additionalimagestores entry from /usr/share/containers/storage.conf) only has empty images.lock/layers.lock; no image lives there.
  • The nvidia-open build pulls ghcr.io/ublue-os/bluefin-nvidia-open:stable, while iso-e2e-test.yml hard-codes IMAGE_REF=ghcr.io/ublue-os/bluefin for both jobs — so the NVIDIA kickstart was asking for an image its ISO does not carry.

Booting that same ISO locally under KVM the way the harness does (-kernel/-initrd + -cdrom, ISO kargs) and running skopeo inspect containers-storage:ghcr.io/ublue-os/bluefin:stable in the live root resolves the image (same parsed reference string as the CI error), and running tests/iso/e2e.sh locally against the artifact goes through ostree container image deploy (layers already present: 0; layers needed: 74 (3.3 GB)) at ~120 s — so the store name is right and the CI lookup failure is not reproducible on KVM. The CI VM ran on TCG (/dev/kvm is root:kvm 0660 on hosted runners, and the harness's accel=kvm:tcg falls back silently).

Second and third commits

  • tests/iso/e2e.sh reads the stored image names straight out of the ISO (xorriso -find … report_lba for the squashfs offset, unsquashfs -o … var/lib/containers/storage/overlay-images/images.json, falling back to extracting the squashfs if its extents are not contiguous) and renders a single __IMAGE_URL__ from them. IMAGE_REF:IMAGE_TAG is used when it matches a stored name; a single-image store is used as-is (with a note); anything else fails with the list of stored names. unsquashfs joins the required tools.
  • The kickstart gains %pre and %pre-install blocks (--log=/dev/ttyS0) that print e2e-pre: / e2e-pre-install: lines with podman images, skopeo inspect of the chosen URL and the storage mounts, so the installer's own view of the store is in the serial log right before the deploy. This matters because c/image's storage transport reports any store error while resolving a name as "does not resolve to an image ID" (storage_reference.go, resolveImage: if image != nil && err == nil), so the CI message alone cannot distinguish a wrong name from a store that failed to open.
  • The harness prints which accelerator the VM gets, and the workflow enables /dev/kvm for the runner user (the udev rule from the GitHub changelog) and installs squashfs-tools.
  • docs/qa.md and docs/skills/test-iso/SKILL.md document the derived reference and the e2e-pre: evidence.

Verification for this commit: shellcheck -S style and bash -n clean; the derivation dry-run against the real artifact prints Using kickstart ostreecontainer --url=ghcr.io/ublue-os/bluefin:stable (from ISO var/lib/containers/storage/overlay-images/images.json); synthetic stores exercise the single-image and ambiguous branches; the full local KVM E2E against the artifact (tests/iso/e2e.sh, QEMU_ACCEL=kvm, 4 GiB) passed: %pre saw ghcr.io/ublue-os/bluefin stable d1470a5f5ffd in podman images and skopeo inspect resolved the same reference string the CI error quotes, the deploy imported 74 layers, reboot: Restarting system at 557 s, e2e-summary.json "result": "passed", 578 s wall clock end to end.

Refs #69

Prepared with an AI agent (Claude) on behalf of @Danathar

The stable E2E jobs have failed on every run since 2026-08-07 with
"Kickstart file /run/install/ks.cfg is missing." The harness boots the
ISO's extracted vmlinuz/initramfs with inst.ks=http://..., but that boot
option only sets --ks=/run/install/ks.cfg inside anaconda (store_const
in pyanaconda/argument_parsing.py); fetching the URL into that path is
the job of anaconda's dracut module, which the titanoboa initramfs does
not carry. Nothing ever wrote the file, anaconda aborted, and the 1800s
budget timed out.

Append a gzip cpio overlay to the initramfs carrying the rendered
kickstart at /etc/e2e/ks.cfg plus an initrd unit (wanted by
initrd.target, ordered before initrd-cleanup.service) that copies it to
/run/install/ks.cfg, which survives switch-root - the same hand-off
anaconda-lib.sh:parse_kickstart() performs. Point inst.ks= at the
embedded file and drop the HTTP server nothing fetched from.

Raise the default install budget to 2700s: anaconda's optional
Subscription module fails to activate in this image and dbus waits its
full 600s service_start_timeout before installation starts, which the
old 1800s budget did not account for.

Refs #69
@Danathar
Danathar requested a review from a team as a code owner September 26, 2026 22:26
Run 36276284373 proved kickstart delivery (e2e-kickstart.service finished
at 19s, anaconda ran the kickstart to the payload step) and then failed in
`ostree container image deploy --transport=containers-storage` with
"reference ... ghcr.io/ublue-os/bluefin:stable does not resolve to an
image ID".

The store name is not guessable from IMAGE_REF/IMAGE_TAG: titanoboa
`podman pull`s whatever the build passed, so the nvidia-open ISO carries
ghcr.io/ublue-os/bluefin-nvidia-open:stable while the workflow asked the
kickstart for bluefin:stable. Read the stored names out of the ISO
(unsquashfs on /LiveOS/squashfs.img at its LBA offset ->
var/lib/containers/storage/overlay-images/images.json) and render the
single --url from that; IMAGE_REF:IMAGE_TAG only disambiguates when an ISO
carries several images, and a store with no match fails loudly.

The kickstart now also prints `e2e-pre:` lines (podman images, skopeo
inspect of the chosen URL) to the serial log so the installer's own view
of the store is in the evidence, and the workflow enables /dev/kvm for the
runner so accel=kvm:tcg stops falling back to a ~45-minute TCG install.
…ad step

c/image's storage transport reports any store error while resolving a name
as 'does not resolve to an image ID', so record podman images / skopeo
inspect / the storage mounts from a %pre-install block as well, which runs
immediately before 'ostree container image deploy'.
@Danathar

Copy link
Copy Markdown
Contributor Author

Run 36281641176 at 53f67a2: both Stable E2E jobs passed (main and NVIDIA), the first green E2E since 2026-08-07. The proof artifact shows the %pre store listing with ghcr.io/ublue-os/bluefin:stable, skopeo resolving it, Deploy OSTree at 29s, and e2e-summary.json with result: passed and installed-system-booted.

One thing I can't explain: the previous run (36276284373) failed on the main job with OpenImage ... does not resolve to an image ID for the same name in the same ISO. Nothing in the new diagnostics shows a difference, so I'm not claiming that failure is understood, only that it did not recur here. The NVIDIA job's earlier failure was a real bug (its kickstart named bluefin:stable while that ISO embeds bluefin-nvidia-open:stable); the harness now reads the name from the ISO's store.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The kickstart initramfs overlay delivery and ISO store extraction cleanly resolve Anaconda's headless installation failures without requiring network/dracut fetching. arm64 LTS build failure in CI is confirmed upstream/infra due to missing linux/arm64 architecture in ghcr.io/ublue-os/bluefin:lts image index.

@castrojo
castrojo merged commit 0e6046e into main Sep 29, 2026
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants