Repository navigation
fix(e2e): deliver the kickstart through an initramfs overlay - #74
Conversation
The stable E2E jobs have failed on every run since 2026-08-07 with "Kickstart file /run/install/ks.cfg is missing." The harness boots the ISO's extracted vmlinuz/initramfs with inst.ks=http://..., but that boot option only sets --ks=/run/install/ks.cfg inside anaconda (store_const in pyanaconda/argument_parsing.py); fetching the URL into that path is the job of anaconda's dracut module, which the titanoboa initramfs does not carry. Nothing ever wrote the file, anaconda aborted, and the 1800s budget timed out. Append a gzip cpio overlay to the initramfs carrying the rendered kickstart at /etc/e2e/ks.cfg plus an initrd unit (wanted by initrd.target, ordered before initrd-cleanup.service) that copies it to /run/install/ks.cfg, which survives switch-root - the same hand-off anaconda-lib.sh:parse_kickstart() performs. Point inst.ks= at the embedded file and drop the HTTP server nothing fetched from. Raise the default install budget to 2700s: anaconda's optional Subscription module fails to activate in this image and dbus waits its full 600s service_start_timeout before installation starts, which the old 1800s budget did not account for. Refs #69
Run 36276284373 proved kickstart delivery (e2e-kickstart.service finished at 19s, anaconda ran the kickstart to the payload step) and then failed in `ostree container image deploy --transport=containers-storage` with "reference ... ghcr.io/ublue-os/bluefin:stable does not resolve to an image ID". The store name is not guessable from IMAGE_REF/IMAGE_TAG: titanoboa `podman pull`s whatever the build passed, so the nvidia-open ISO carries ghcr.io/ublue-os/bluefin-nvidia-open:stable while the workflow asked the kickstart for bluefin:stable. Read the stored names out of the ISO (unsquashfs on /LiveOS/squashfs.img at its LBA offset -> var/lib/containers/storage/overlay-images/images.json) and render the single --url from that; IMAGE_REF:IMAGE_TAG only disambiguates when an ISO carries several images, and a store with no match fails loudly. The kickstart now also prints `e2e-pre:` lines (podman images, skopeo inspect of the chosen URL) to the serial log so the installer's own view of the store is in the evidence, and the workflow enables /dev/kvm for the runner so accel=kvm:tcg stops falling back to a ~45-minute TCG install.
…ad step c/image's storage transport reports any store error while resolving a name as 'does not resolve to an image ID', so record podman images / skopeo inspect / the storage mounts from a %pre-install block as well, which runs immediately before 'ostree container image deploy'.
|
Run 36281641176 at 53f67a2: both Stable E2E jobs passed (main and NVIDIA), the first green E2E since 2026-08-07. The proof artifact shows the One thing I can't explain: the previous run (36276284373) failed on the main job with |
castrojo
left a comment
There was a problem hiding this comment.
LGTM. The kickstart initramfs overlay delivery and ISO store extraction cleanly resolve Anaconda's headless installation failures without requiring network/dracut fetching. arm64 LTS build failure in CI is confirmed upstream/infra due to missing linux/arm64 architecture in ghcr.io/ublue-os/bluefin:lts image index.
What you see today
Both
Stable main ISO E2E / e2e-testandStable NVIDIA ISO E2E / e2e-testhave failed on every run since 2026-08-07 (first seen on the #70 runs, most recently on #69's run 36079133044). The job log only says:The
installer-serial.login theiso-e2e-proof-*artifact shows what actually happened. The initrd stage boots fine and hands over to anaconda, which starts its DBus modules and then aborts:The August 7 run (
iso-e2e-proof-31227505330) fails at the identical line at 802 s.Cause
The harness boots the ISO's extracted
vmlinuz/initramfs.imgdirectly withinst.ks=http://10.0.2.2:18080/kickstart.ks, and serves the kickstart from a localpython3 -m http.server. Nothing in the guest ever fetches that URL:inst.ks=is astore_constboot option. Whatever its value, it only setsksfile=/run/install/ks.cfg(pyanaconda/argument_parsing.py), and stage 2 aborts if that file does not exist (pyanaconda/startup_utils.py, the exact message above).fetch-kickstart-net.shdownloads it andparse_kickstart()indracut/anaconda-lib.shends withcp "$parsed_kickstart" /run/install/ks.cfg.check()returns 255 indracut/module-setup.sh) and only lands in an initramfs whenanaconda-dracutis installed where dracut runs. Titanoboa builds the initramfs withdracut --no-hostonly --add "dmsquash-live dmsquash-live-autooverlay"in the pulled rootfs (Justfile,initramfsrecipe at the pinned840217d), and the initrd part of the serial log has no anaconda hooks at all: noanaconda:lines, no kickstart fetch, no/run/install.So the URL was never requested,
/run/install/ks.cfgwas never written, and anaconda gave up. The 1800 s budget then ran out.Fix
tests/iso/e2e.shnow does the hand-off the dracut module would have done, without touching the ISO:e2e-output/kickstart.ks(kept in the proof artifact) and builds a small overlay tree:/etc/e2e/ks.cfgpluse2e-kickstart.service, an initrd unit (DefaultDependencies=no,ConditionPathExists=/etc/initrd-release,Before=initrd-cleanup.service initrd-switch-root.target, wanted byinitrd.targetthrough a symlink ininitrd.target.wants/) whose only job ismkdir -p /run/install && cp /etc/e2e/ks.cfg /run/install/ks.cfg./runis carried across switch-root by systemd, which is the same property anaconda's own module relies on (and why/run/initramfs/liveis still mounted in stage 2 in the same log).cpio -H newc -R 0:0, gzips it, and appends it to the ISO'sinitramfs.img; the combined file goes to-initrd. The kernel unpacks concatenated initramfs segments in turn; a compressed trailer needs no 4-byte alignment against the ISO's zstd stream, which is exactly howvirt-install --initrd-injectappends files.inst.ks=so anaconda still runs with--ks, but points it atfile:/etc/e2e/ks.cfg. That value is ignored by stage 2 today; if a future initramfs does carry the anaconda module,parse-anaconda-kickstart.shwill parse that same embedded file instead of needing network in the initrd.curlreadiness loop andE2E_HTTP_PORT, since nothing consumed them.Alternative considered: remastering the ISO with
ks.cfgat its root andinst.kickstart=/run/initramfs/live/ks.cfg(--kickstarttakes a path). That works too but means rewriting a 6 GB ISO on the runner per job; the overlay is a few KB and leaves the artifact under test untouched.Time budget
The serial log also shows anaconda's optional Subscription module never activates in this image: the module decides not to start (
rhsm.serviceis not installed) but dbus-daemon still waits its fullservice_start_timeoutof 600 s before the Boss skips it. That happens before the kickstart is read, so it will still happen with the kickstart in place, and an install can only begin ~13.5 min after boot.E2E_INSTALL_TIMEOUTtherefore defaults to 2700 s now (was 1800); the job's 90 min limit is unaffected. Forbidding that module would have to happen in the ISO's anaconda profile (iso_files/configure_iso_anaconda.sh), which is a product change and out of scope here.docs/qa.mdanddocs/skills/test-iso/SKILL.mdrecord the delivery mechanism and the budget note. The workflow installscpioexplicitly.Verification
bash -n tests/iso/e2e.shandshellcheck tests/iso/e2e.sh(clean at all severities, same as before).cpio -itvlistsetc/e2e/ks.cfg,etc/systemd/system/e2e-kickstart.serviceand theinitrd.target.wantssymlink, all owned by root; the output file is byte-for-byte the original zstd stream followed by a gzip stream that decompresses to a070701newc archive; unpacking both segments in order yields the merged tree with the symlink resolving andks.cfgidentical to the rendered kickstart.systemd-analyze verifyon the generated unit passes against a host systemd.Not verified locally: the boot itself. Booting the ISO in QEMU is verified only by this PR's own
Stable main ISO E2E/Stable NVIDIA ISO E2Echecks; the thing to look for in their serial logs ise2e-kickstart.servicein the initrd stage and anaconda proceeding past kickstart parsing instead ofKickstart file /run/install/ks.cfg is missing.Progress: kickstart delivery is proven, next failure is the image reference
Run 36276284373 (head 3c99501, job 108503421574,
installer-serial.loginiso-e2e-proof-36276284373) shows the overlay working end to end:No
Kickstart file /run/install/ks.cfg is missing.any more; anaconda parsed the kickstart, partitionedvda, ranostree admin init-fs/os-init, and died in the payload step. (The Subscription module still burned its 600 s in that run, at 920 s.) The NVIDIA job hit the 2700 s budget instead — it never got that far under TCG.What the ISO actually contains
Pulled
iso-36276284373-stable-main-amd64and read the live squashfs (unsquashfson/LiveOS/squashfs.img):var/lib/containers/storage/overlay-images/images.jsonholds exactly one image,d1470a5f…,"names": ["ghcr.io/ublue-os/bluefin:stable"]— the name titanoboa'srootfs-include-containerrecipe produces withpodman pull ghcr.io/ublue-os/bluefin:stable(pinned Justfile840217d, and the build log shows that very pull)./usr/share/anaconda/interactive-defaults.ksin the ISO uses the identicalostreecontainer --url=ghcr.io/ublue-os/bluefin:stable --transport=containers-storageline.usr/lib/containers/storage/(theadditionalimagestoresentry from/usr/share/containers/storage.conf) only has emptyimages.lock/layers.lock; no image lives there.ghcr.io/ublue-os/bluefin-nvidia-open:stable, whileiso-e2e-test.ymlhard-codesIMAGE_REF=ghcr.io/ublue-os/bluefinfor both jobs — so the NVIDIA kickstart was asking for an image its ISO does not carry.Booting that same ISO locally under KVM the way the harness does (
-kernel/-initrd+-cdrom, ISO kargs) and runningskopeo inspect containers-storage:ghcr.io/ublue-os/bluefin:stablein the live root resolves the image (same parsed reference string as the CI error), and runningtests/iso/e2e.shlocally against the artifact goes throughostree container image deploy(layers already present: 0; layers needed: 74 (3.3 GB)) at ~120 s — so the store name is right and the CI lookup failure is not reproducible on KVM. The CI VM ran on TCG (/dev/kvmisroot:kvm 0660on hosted runners, and the harness'saccel=kvm:tcgfalls back silently).Second and third commits
tests/iso/e2e.shreads the stored image names straight out of the ISO (xorriso -find … report_lbafor the squashfs offset,unsquashfs -o … var/lib/containers/storage/overlay-images/images.json, falling back to extracting the squashfs if its extents are not contiguous) and renders a single__IMAGE_URL__from them.IMAGE_REF:IMAGE_TAGis used when it matches a stored name; a single-image store is used as-is (with a note); anything else fails with the list of stored names.unsquashfsjoins the required tools.%preand%pre-installblocks (--log=/dev/ttyS0) that printe2e-pre:/e2e-pre-install:lines withpodman images,skopeo inspectof the chosen URL and the storage mounts, so the installer's own view of the store is in the serial log right before the deploy. This matters because c/image's storage transport reports any store error while resolving a name as "does not resolve to an image ID" (storage_reference.go,resolveImage:if image != nil && err == nil), so the CI message alone cannot distinguish a wrong name from a store that failed to open./dev/kvmfor the runner user (the udev rule from the GitHub changelog) and installssquashfs-tools.docs/qa.mdanddocs/skills/test-iso/SKILL.mddocument the derived reference and thee2e-pre:evidence.Verification for this commit:
shellcheck -S styleandbash -nclean; the derivation dry-run against the real artifact printsUsing kickstart ostreecontainer --url=ghcr.io/ublue-os/bluefin:stable (from ISO var/lib/containers/storage/overlay-images/images.json); synthetic stores exercise the single-image and ambiguous branches; the full local KVM E2E against the artifact (tests/iso/e2e.sh,QEMU_ACCEL=kvm, 4 GiB) passed:%presawghcr.io/ublue-os/bluefin stable d1470a5f5ffdinpodman imagesandskopeo inspectresolved the same reference string the CI error quotes, the deploy imported 74 layers,reboot: Restarting systemat 557 s,e2e-summary.json"result": "passed", 578 s wall clock end to end.Refs #69
Prepared with an AI agent (Claude) on behalf of @Danathar