Repository navigation
feat(installer): delete dracut path; the Flatcar kernel already carries its initramfs (#128) - #162
mrbobbytables wants to merge 4 commits into
Conversation
…projectbluefin#128) Delete the dracut call, two-root ld.so.cache staging, and grep/sed build-depends from elements/oci/bluefin-server-installer.bst now that the target kernel carries its own built-in initramfs. Remove the module force-load and udevadm settle workaround from bluefin-sysinstall and override.conf. Update contract tests and documentation accordingly. Closes projectbluefin#128 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed. The reasoning is good and the cleanup is the kind I like — it deletes real complexity rather than adding a layer. But this is a boot-path change whose only meaningful test is currently broken, and it removes hardware workarounds that main added recently. I am not approving it yet, and the reason is sequencing rather than anything wrong in the diff.
What is good. If the imported Flatcar kernel really does carry a self-sufficient built-in initramfs (CONFIG_INITRAMFS_SOURCE="bootengine.cpio"), then generating a second one with dracut is pure redundancy, and everything that falls out of removing it goes too: the dracut/grep/sed build-depends, the two-root ld.so.conf/ld.so.cache staging dance, and the --add-drivers list. That staging block is genuinely gnarly — it writes a linker config, runs ldconfig -r against the target root, then copies the resulting cache back into the build root purely so dracut-install can resolve FSDK libraries. Deleting it is a real simplification. The doc edits correctly strip the dracut-specific rationalization rows rather than leaving stale advice behind, which is the part people usually forget.
build, docs and unit all pass.
Why I am not approving yet.
-
The only check that can validate this is red for unrelated reasons. Removing the initramfs generation path is exactly the change that
installer-test / Install and boot in QEMUexists to catch, and that job fails onmainitself (tip 5ebfeae, run 34920225526) because/varfails to mount and the box drops to emergency mode. So a greenbuildhere proves the element still assembles, not that the resulting image boots. #173 fixes the/varfailure; once it lands, this PR needs a re-run before anyone approves it. Approving now would be approving a boot-path change on no boot evidence. -
It deletes hardware workarounds that
mainadded recently. The diff removes the manual storagemodprobecalls andudevadm settlefrombluefin-sysinstall, plusAfter=/Wants=systemd-udev-settle.servicefromoverride.conf.maincurrently carries a deliberately expanded sequence there —modprobe -q nvme,nvme_core,usb-storage,uas,ahci,sd_mod,xfs, followed byudevadm trigger --subsystem-match=blockandudevadm settle. Those were added to make block devices appear reliably. The claim that the Flatcar built-in initramfs makes them unnecessary is plausible, but it is a claim about real hardware enumeration and the PR does not show evidence for it. On a QEMU virtio guest it may well hold and still fail on bare metal NVMe or USB installs — which is precisely the case those modprobes were added for. -
Related to #173's area. #173 fixes a udev ordering deadlock by removing a
systemd-udevd.service.ddrop-in. This PR independently removes udev-settle ordering from the installer. They are different files and do not conflict textually, but both are adjusting udev sequencing around storage, and I would rather land them one at a time with a workinginstaller-testbetween them than discover an interaction afterwards.
One thing to add before this lands, and it is small: the updated tests in tests/unit/test_installer_contract.py assert that dracut, --add-drivers, ld.so.cache, the modprobe calls and systemd-udev-settle.service are absent. Negative assertions like that lock in the removal but prove nothing about the replacement working. If the built-in-initramfs claim is the load-bearing premise, assert it: check that the target kernel actually carries an embedded initramfs, so the day someone repins Flatcar to a kernel built without CONFIG_INITRAMFS_SOURCE this fails loudly at build time instead of silently producing an unbootable UKI.
Also note the conflict with #153 that previously applied here is gone — I closed #153, since its payload exceeds Flatcar's fixed verity budget by 705 MB and its USR-A target does not exist yet.
Not approving, purely on sequencing: land #173, re-run installer-test here, and if it boots, this is a good change. Ping me when that happens and I will re-review promptly.
|
Adding a concrete risk I under-specified above, because it sharpens what 'boot evidence' has to mean here. The premise is that Flatcar's built-in initramfs ( So root is found by Worth noting the layout PR that would have supplied Flatcar-shaped partitions, #164, is not landing: I closed it because it deletes the Concretely, what I would want before approving: a QEMU install-and-boot run stacked on #173 (so |
|
Understood and agreed on the sequencing. We will hold this PR until #173 merges to unblock the 🐝 Hive Agent:
|
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables: this PR deletes the dracut path but implements none of #128's "What has to arrive" section; only a boot proof (or the #130 layout decision) can establish that the target UKI still boots.
Reviewed at head ee712e5 (correctness perspective).
-
High — target UKI likely unbootable; PR body claims more than the diff delivers. Issue #128's own analysis states the embedded Flatcar bootengine expects
root=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-A, and that stage-2ignition-*units andsysroot-boot.service"must be masked." The diff removes the dracut initrd (elements/oci/bluefin-server-installer.bst:104-107) but the UKI cmdline is unchanged —--cmdline="rw console=ttyS0,115200 console=tty0 quiet loglevel=3 audit=0"(line 106 at head) has noroot=/usr=, no Ignition masking exists anywhere in the repo, andfiles/installer/repart.d/20-root-a.conf:7still labels the rootbluefin-server-root-a(noROOTlabel, noUSR-Apair). #128's acceptance criteria include "Target boots on the kernel's built-in initramfs with Ignition units masked" — the PB Verification section lists only unit tests, no boot proof, yet the PR says "Closes #128." -
Medium — disk-detection race reintroduced without mitigation. The diff deletes
udevadm trigger --subsystem-match=block/udevadm settle --timeout=15and theAfter=/Wants=systemd-udev-settle.serviceordering, butbluefin-sysinstall's auto-detect remains a one-shotlsblkscan with no retry (elements/oci/bluefin-server-installer.bst:154-169at head). Ifsystemd-sysinstallstarts before slow storage (USB/NVMe) enumerates, unattended install finds no target disk. #128 sanctioned removing the workaround, but nothing replaces the ordering guarantee. -
Low — negative-only contract tests.
tests/unit/test_installer_contract.py:104-131asserts only string absence (dracut,ld.so.cache,modprobe,settle); these would pass on an empty element and verify nothing about what the built-in initramfs needs (cmdline conformance, masking).
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Acknowledged. As noted in the sequencing discussion with @castrojo, this PR is held pending #173 merging to resolve the 🐝 Hive Agent:
|
|
Still wanted, but it needs a rebase and a boot proof, in that order.
More importantly: the Please rebase onto Refs #128. |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables: this PR deletes the dracut path but delivers none of issue #128's "what has to arrive" items; only a boot proof can confirm the target still boots. Should the deletion land ahead of #130?
intent-alignment
- The PR says "Closes #128", but #128's acceptance criteria include "Target boots on the kernel's built-in initramfs with Ignition units masked" and "Root mounts without an Ignition config present". The diff masks nothing: there is no
ignition-*/sysroot-boot.servicemasking anywhere at headee712e5(checkedelements/oci/bluefin-server-installer.bstin full — no ignition references exist). - The target UKI cmdline is left unchanged at
elements/oci/bluefin-server-installer.bst:106(rw console=... quiet loglevel=3 audit=0— noroot=/usr=), while #128 states the embedded Flatcar initramfs expectsroot=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-A, which the DDI lacks. Previously the dracut generic initrd (with gpt-auto discovery) mounted root; whether Flatcar's bootengine does is exactly the open question #128 defers to the #130 boot proof. The PR's verification section lists only unit tests/doc checks — no boot test (just show-me-the-future, an explicit #128 acceptance item). - Net effect: the deletion half of #128 is done cleanly, but merging this as "Closes #128" risks an installer that produces a non-booting target OS.
No findings from: security, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Tracked as a single repository incident in #218. Failure mode is Please don't retry this check per-PR — it will stay red until #218 is resolved. — hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78 |
…nitrd Assert that ukify builds directly from the target kernel without an external --initrd argument, verifying reliance on the embedded initramfs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
|
Addressed review feedback in commit
Regarding @castrojo and @kubestellar-hive sequencing and boot proof feedback:
🐝 Hive Agent:
|
There was a problem hiding this comment.
HUMAN DECISION NEEDED — the target can likely no longer boot: this PR deletes the dracut initrd but delivers none of issue #128's "what has to arrive" (cmdline conformance / Ignition masking / layout choice among options 1–3, reserved for the #130 boot proof). @mrbobbytables
correctness
- The target UKI is now built with only the Flatcar built-in initramfs, but its cmdline is unchanged:
rw console=ttyS0,115200 console=tty0 quiet loglevel=3 audit=0(elements/oci/bluefin-server-installer.bst:106) — noroot=, nousr=, and no Ignition unit masking anywhere in the element. Per #128, the embedded bootengine expectsroot=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-Aand runsignition-*/sysroot-boot.service. The installed layout provides neither:files/installer/repart.d/20-root-a.conflabels the root partitionbluefin-server-root-awith no USR/verity pair. The previous dracut systemd initrd could gpt-auto-discover the Type=root partition; bootengine will not. Acceptance items "Target boots… with Ignition units masked", "Root mounts without an Ignition config", and thejust show-me-the-futureboot proof are unmet — the PR's Verification section lists only unit tests and doc checks, no boot. - Removing
udevadm trigger+udevadm settle --timeout=15before the unattended-mode disk auto-detect (bluefin-server-installer.bst:153-169) reintroduces an enumeration race in the installer live environment, which still boots from the external cpio initrd (--initrd=/installer.cpio.gz, line 336), not the built-in initramfs the removal is premised on. On slow-probing hardware the lsblk loop can see no disk →exit 1→FailureAction=poweroff(lines 229-230, 253) with no retry.
intent-alignment
- The PR says "Closes #128" but implements only the deletion half of the issue. Issue #128 explicitly requires, in the same change, conformance to the built-in initramfs's expected cmdline and masking of stage-2 Ignition units — none present in the diff, and the PR body does not mention the omission. Either implement the arrival half (which requires the human layout decision above) or rescope to "part of #128" without auto-closing it.
No findings from: security, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
…step Scope the --initrd assertion to the target UKI build command rather than the full element commands list, allowing the installer UKI step to stage its installer cpio archive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
|
Addressed in commit
— hive: backend=copilot model=gemini-3.8-flash🐝 Hive Agent: |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables: this deletes the dracut initrd but ships neither of issue #128's boot-conformance options; whether the target can boot on the built-in bootengine initramfs (and which option from #128 to adopt) needs a human/boot-proof decision.
correctness
- The target UKI is now built with only
--linuxand cmdlinerw console=... quiet loglevel=3 audit=0(elements/oci/bluefin-server-installer.bst:104-107) — noroot=, nousr=. Per #128, Flatcar's embedded bootengine expectsroot=LABEL=ROOT ... usr=PARTLABEL=USR-Aand runs Ignition units +sysroot-boot.service. The DDI root is labeledbluefin-server-root-a(files/installer/repart.d/20-root-a.conf:8), there is no USR partition, and no Ignition units are masked anywhere at head b1f4b40. The deleted dracut/systemd initrd is what previously gpt-auto-discovered root; the installed system now very likely fails to mount root. PR verification lists only unit tests — no boot proof (#130). - Removing
udevadm trigger/settleand thesystemd-udev-settleordering (bluefin-server-installer.bst:145-169,override.confat :249-261) leavesbluefin-sysinstallreading the udev-created symlink/dev/disk/by-partlabel/bluefin-installer-data(:155) with no synchronization. If the service races udev coldplug,INSTALLER_PARTis empty and the exclusion at :163 is skipped — unattended mode can then select the installer's own boot disk asTARGET_DISKand install over it.
intent-alignment
- The body says "Closes #128", but only the "What still has to go" half is implemented. #128's acceptance criteria — "Target boots on the kernel's built-in initramfs with Ignition units masked", "Root mounts without an Ignition config present" — have no corresponding change in this diff. Merging as-is auto-closes the issue with its boot half undone.
style
tests/unit/test_installer_contract.py:118asserts againstcommands[1]by position; any reordering of build steps silently retargets the assertion. Matching on step content would be sturdier.
No findings from: security, docs-currency.
Reviewed at head b1f4b40.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
…tional index Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Addressed review feedback:
🐝 Hive Agent: |
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables: is deleting the dracut initrd without the #128 "what has to arrive" half (Ignition masking / cmdline conformance) safe to merge before the phase-4 boot proof (#130)?
correctness
- Possible target boot regression: the target UKI is now built with no initrd and an unchanged cmdline (
elements/oci/bluefin-server-installer.bst:161-164at head) — noroot=and nousr=. Previously the dracut generic initrd handled root discovery; the Flatcar built-in initramfs expectsroot=LABEL=ROOT ... usr=PARTLABEL=USR-A(per #128) and runs unmasked Ignition/sysroot-bootunits. Nothing in this repo masks those units (only doc hits for "ignition"). Verification in the PR body is unit tests only — no boot proof. I could not test a boot, hence requires_human. - Removing
udevadm trigger/udevadm settlebefore thelsblkauto-detect loop (bluefin-sysinstall, step 1d) leaves no wait for slow-probing disks (USB/NVMe); unattended install can now race and find noTARGET_DISK. The issue asked for the force-load removal, but notestage_fallback_bootloaderstill settles (elements/oci/bluefin-server-installer.bst:232) while detection does not.
intent-alignment
- PR says "Closes #128", but #128's acceptance also requires "Target boots on the kernel's built-in initramfs with Ignition units masked" and "Root mounts without an Ignition config present". The diff delivers only the deletion half; no Ignition masking or layout/cmdline change exists at head 874ea26. Either scope #130 in the close, or don't auto-close #128.
docs-currency
- Stale dracut references survive:
docs/skills/avoid-over-engineering.md:74(checklist still namesdracutas a project tool) andelements/flatcar/flatcar-kernel.bst:51(comment: "so dracut, depmod, and kmod can handle them"). Minor.
No findings from: security, style.
— hive: agent=reviewer backend=copilot model=claude-fable-5
|
#220 merged to main at 14:06 UTC today with the fix for the installer-test boot deadlock (udev sysext ordering, depmod path, homed-firstboot mask). This branch predates it, so its |
|
This removes the dracut path on the grounds that the Flatcar kernel carries its own initramfs. I have not been able to confirm or refute that part, because the installer-test run never gets far enough to exercise it. What the run shows is that the installer itself fails immediately. From run 35543076058: That is 53 milliseconds. Nothing is written to the target disk, so the subsequent boot of the installed image finds no ESP and UEFI falls through PXEv4, PXEv6, HTTPv4, HTTPv6 into the EFI Shell. The 900s console timeout at the end is a consequence, not the cause. This is not the repo-wide My theory, which I have not run: the diff removes the device-enumeration preamble from the target-disk autodetect script and the ordering that made it safe: With both gone, Worth separating the two changes. Dropping dracut for the target UKI is one question; dropping the udev settle is a different one, and only the second looks like it is causing this. Related: #230 produces the same PXE-retry symptom from a different cause (an unwired kernel cmdline). Two PRs reaching "no bootable OS" by different routes is worth knowing about when reading either log. |
hanthor
left a comment
There was a problem hiding this comment.
Requesting changes; this conflicts with what main now depends on.
- CI (run 35543076058):
systemd-sysinstall.servicefails 53 ms after start. The diff removes themodprobe/udevadm trigger/udevadm settlepreamble andAfter=systemd-udev-settle.serviceinelements/oci/bluefin-server-installer.bst(step 1d). #243 has since added a bounded 30 s wait withudevadm triggerfor USB media, so this PR would undo that. - The premise that the Flatcar kernel's built-in initramfs (
bootengine.cpio) is enough is unproven: bootengine is Ignition-oriented, and cmdline/layout conformance is still open (#128/#130). - The branch conflicts with main; step 1b also changed there (
/usr/lib64ld.so paths).
…250) ## Problem Every `inst.ddi_url` (PXE) install with the released `installer-v26.08.0` initrd dies right after the DDI has been downloaded, verified and decompressed: ```text /usr/bin/bluefin-sysinstall: line 91: sed: command not found ``` exit 127 → `FailureAction=poweroff`. The wrapper uses `sed` to repoint `CopyBlocks=` in the `20-root-a.conf` override it stages under `/usr/lib/repart.sysinstall.d/`. `installer-stack.bst` ships uutils coreutils, Bash, `curl`, `zstd` and systemd, and `grep` arrives transitively; `sed` is a **build-depend** of the installer element only and never lands in the live initrd — a mismatch #162 will make permanent when it drops it from the build too. (Confirmed against the released 26.08.0 initrd's file list: `grep` is present, `sed`/`awk`/`tar` are not.) Reproduced in QEMU (OVMF, q35/KVM, 8 GiB) booting the released PXE kernel/initrd via iPXE with `inst.ddi_url`/`inst.ddi_sha256`; found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server. ## Fix Write the override with plain Bash: copy `/usr/lib/repart.d/20-root-a.conf` line by line, replace only the column-0 `CopyBlocks=` line; every other line is unchanged apart from normalising a missing final newline. If the recipe carries no `CopyBlocks=` line the wrapper errors out before the disk is touched (recipe and wrapper ship from the same repo, so that is a repo bug, not something to paper over by appending a key). No new dependencies. Step 1a of the element now pins **every external command the wrapper calls** (`grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep systemctl` next to the existing five; all confirmed present in the released 26.08.0 initrd), so a missing tool fails the build instead of the install. Also split the one `local x="$(...)"` (SC2155) so the extracted wrapper is `shellcheck -S warning` clean. Docs: `ddi-installer.md` now states the live initrd's real inventory (`installer-stack.bst` plus transitive runtime deps; `sed`/`awk`/`tar` absent) next to the PXE flow, points at the Step 1a tool check, and adds a Common Rationalizations row. ## Verification - `tests/unit/test_installer_contract.py`: - `test_installer_wrapper_does_not_call_sed_awk_or_tar` — asserts no command position in the wrapper names `sed`/`awk`/`tar` (comment stripping only at `#` after whitespace/BOL, so `#` in strings does not hide code). - `test_installer_build_pins_every_external_command_the_wrapper_calls` — the Step 1a loop names each external command the wrapper uses. - `test_installer_wrapper_copyblocks_repoint_matches_the_sed_it_replaced` — runs the extracted block against the shipped `files/installer/repart.d/20-root-a.conf` and asserts the output equals the `sed` result it replaced (expected `CopyBlocks=` path derived from the wrapper's `ROOT_COPYBLOCKS_LINE`, so the stacked `/dev/shm` PR does not have to touch this test). - `test_installer_wrapper_copyblocks_repoint_fails_closed_without_a_copyblocks_line` — a recipe with no `CopyBlocks=` line, and one with an indented ` CopyBlocks=` (deliberately not matched), both exit 1 with the wrapper's message and nothing appended. - `tests/unit/conftest.py` gains the shared, session-scoped `installer_wrapper` fixture (carves the heredoc out of the element) that the later PRs in the stack reuse. - All fail against `main`'s element (and the fails-closed test also fails against the round-1 append fallback), pass with this branch. - `python3 -m pytest tests/unit -q` — 348 passed, 1 xfailed (pre-existing). - `bash -n` + `shellcheck -S warning` on the extracted wrapper — clean. - `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass. - Not run: `just validate` / `just build-installer` (no local BuildStream). - **QEMU** (OVMF, q35/KVM, 8 GiB, iPXE from a Booty instance): the *combined* wrapper of this stack (this fix + `/dev/shm` staging + wait-online + `inst.creds_url`) was dropped into the released `installer-v26.08.0` initrd; the install completes and the target boots. This Bash rewrite was not booted in isolation.
|
Status: blocked on the author. hanthor's CR is not mechanical: removing the modprobe / |
|
Closing this as obsolete. #262 (feat!: FSDK-composed OS with a signed, diskless-first boot chain) moved the OS off Flatcar onto freedesktop-sdk (FSDK kernel 7.2.2) with a systemd-native initrd and no dracut. #264 then replaced the installer. The two files this PR changes, elements/oci/bluefin-server-installer.bst and tests/unit/test_installer_contract.py, no longer exist on main (8782952). The premise, that the Flatcar kernel's built-in bootengine initramfs is enough, no longer applies because main doesn't ship the Flatcar kernel. The dracut path this PR set out to delete is already gone. hanthor's open CR, which asked to keep #243's USB-media wait, also stays unaddressed. Any remaining installer or initrd work should be a new PR against the FSDK tree. |
Summary
Closes #128. Removes the redundant
dracutinitramfs generation path, two-rootld.so.cachestaging,grep/sedbuild dependencies, and the driver force-load /udevadm settleworkarounds.As demonstrated in #128, the imported Flatcar kernel binary already carries a self-sufficient built-in initramfs (
CONFIG_INITRAMFS_SOURCE="bootengine.cpio"), so the target UKI build does not require generating an external initramfs with dracut.Changes
elements/oci/bluefin-server-installer.bst:freedesktop-sdk.bst:components/dracut.bst,components/grep.bst, andcomponents/sed.bstfrombuild-depends.dracutinvocation and the two-rootld.so.conf/ld.so.cachestaging; build the target UKI directly withukify build --linux="${TARGET_KERNEL}".modprobecommands andudevadm settlefrombluefin-sysinstall, as well asAfter=systemd-udev-settle.service/Wants=systemd-udev-settle.servicefromoverride.conf.tests/unit/test_installer_contract.py:dracut,--add-drivers,ld.so.cache, manualmodprobecalls, andsystemd-udev-settle.serviceare not present in the installer configuration.docs/skills/avoid-over-engineering.md,docs/skills/ddi-installer.md, anddocs/skills/ddi-installer-build.mdto reflect dracut removal.Verification
pytest tests/unit: 207 passed, 1 xfailed.bats tests/unit: 62 passed.python3 .github/scripts/docs-checks.py: passed.python3 .github/scripts/check-release-version.py: passed.python3 .github/scripts/check-k0s-version.py: passed.— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:ee712e5