Skip to content

fix(boot): mask systemd-homed-firstboot on the target image - #186

Closed
castrojo wants to merge 1 commit into
mainfrom
fix/mask-homed-firstboot
Closed

castrojo wants to merge 1 commit into
mainfrom
fix/mask-homed-firstboot

Conversation

@castrojo

Copy link
Copy Markdown
Contributor

This is a product bug, not a CI bug

Every headless first boot of Bluefin Server hangs indefinitely on an interactive
prompt — PXE, datacenter provisioning, sysupdate-driven reinstall, and the QEMU
smoke test alike. systemd-homed-firstboot.service is:

  • ConditionFirstBoot=yes
  • ExecStart=homectl firstboot --prompt-new-user --prompt-shell=no --prompt-groups=no --mute-console=yes
  • StandardInput=tty / StandardOutput=tty
  • Before=systemd-user-sessions.service first-boot-complete.target

With nobody at the console, the boot transaction never completes and k0s never
starts. The fix belongs in the image, not in the smoke test: masking it only in
the Justfile would make CI green while every real headless install still hangs.

elements/oci/bluefin-server-ddi.bst already masked systemd-firstboot.service
with the comment "Mask systemd-firstboot to prevent interactive prompts on
headless first boot"
. That mask was incomplete, not a deliberate carve-out:
homed ships its own first-boot wizard as a separate unit, pulled in via
WantedBy=systemd-homed.service, and it was never masked.

Evidence

Integration run
35375246792
(tmp/integration-173-181, a throwaway integration of #173 + #181): build
succeeded, the /var emergency-mode failure was gone (zero Dependency failed,
zero emergency mode), but installer-test still timed out at 900s and the
serial log ends at:

ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
 Bluefin Server 26.08.0
🏠 Please create your user account!
🪪 Please enter user name to create (empty to skip):

That prompt string comes from homectl firstboot, confirming the unit ships and
runs in the built image. FSDK 26.08's systemd is built with -Dfirstboot=true
and all of homed's dependencies (openssl, cryptsetup, util-linux-libfdisk,
linux-pam) enabled.

This is the third independent blocker on installer-test, alongside
#173 (the /var fix) and #181 (the readiness gate). All three must land for
installer-test to pass; none of them subsumes the others.

Change

-      # Mask systemd-firstboot to prevent interactive prompts on headless first boot
+      # Mask both first-boot wizards to prevent interactive prompts on headless
+      # first boot: systemd-firstboot.service prompts for locale/keymap/timezone/
+      # root password, and systemd-homed-firstboot.service runs
+      # `homectl firstboot --prompt-new-user`. Both are ConditionFirstBoot=yes,
+      # read StandardInput=tty, and order themselves before
+      # first-boot-complete.target, so either one blocks the boot transaction
+      # forever on a console nobody is attached to.
       mkdir -p /layer/etc/systemd/system
       ln -sf /dev/null /layer/etc/systemd/system/systemd-firstboot.service
+      ln -sf /dev/null /layer/etc/systemd/system/systemd-homed-firstboot.service

systemd-homed-firstboot.service is reached through Wants=, not Requires=,
so masking it cannot fail the boot.

The existing contract test test_ddi_generates_module_indexes_for_runtime_filesystem_drivers
already asserted the systemd-firstboot.service mask; it now asserts both.
docs/skills/ddi-installer.md carried the canonical statement of this fact and
was widened to name both units.

Audit: other units that could block a headless boot on tty input

Every unit in the image's systemd with StandardInput=tty* was checked. Only the
two first-boot wizards are genuine blockers. Judged safe to leave:

Unit(s) Why safe
getty@, serial-getty@, console-getty, container-getty@ Login terminals; nothing orders a boot target after them
breakpoint-pre-{udev,mount,basic,switch-root} Only activated by an explicit systemd.break= cmdline argument
emergency, rescue Only reached on failure or explicit isolate
debug-shell Not enabled by preset; requires explicit systemctl enable
systemd-storagetm Not enabled; rescue-mode only
systemd-sysinstall No ConditionFirstBoot; only started by system-install.target in the live installer, where it is the intended interactive flow

Nothing in files/ defines a unit with StandardInput=, ConditionFirstBoot=,
or a --prompt ExecStart.

Verification

  • just validate — exit 0
  • python3 -m pytest tests/unit/test_installer_contract.py -q — 8 passed
  • python3 .github/scripts/docs-checks.py — passed

Assisted-by: Claude Opus 4.6 via GitHub Copilot

Every headless first boot of Bluefin Server stalls forever on an
interactive prompt. systemd-homed-firstboot.service is ConditionFirstBoot=yes,
runs `homectl firstboot --prompt-new-user` with StandardInput=tty, and orders
itself Before=first-boot-complete.target. With no operator at the console the
boot transaction never completes, so k0s never starts.

The DDI element already masked systemd-firstboot.service "to prevent
interactive prompts on headless first boot"; that mask was simply incomplete.
homed ships its own wizard as a separate unit, pulled in via
WantedBy=systemd-homed.service, and was never masked.

Mask it alongside systemd-firstboot.service and widen the comment and the
ddi-installer skill to name both units and the reason.

Assisted-by: Claude Opus 4.6 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor Author

CI evidence on this branch: the prompt is gone

Run 35382696469:
build success (41m42s), unit pass, docs pass,
installer-test failure.

The installer-test failure is expected and is not this blocker. This branch
is cut from main, so it does not carry #173 or #181. Comparing serial logs:

integration run 35375246792 (#173 + #181, no homed mask) this run 35382696469 (homed mask only)
Please create your user account! / enter user name present, boot stalls there 0 occurrences
Dependency failed / emergency mode 0 18 (the /var blocker #173 fixes)
final failure 900s readiness timeout behind the prompt /var emergency mode

So masking systemd-homed-firstboot.service removes the interactive stall
entirely, and the boot now proceeds far enough to hit the other known blocker.
All three of #173, #181 and this PR are required for installer-test to pass;
none subsumes the others.

@castrojo

Copy link
Copy Markdown
Contributor Author

Author note — I cannot review my own PR, so recording the evidence and the audit.

This is a product bug, not a CI bug. Every headless first boot of Bluefin Server currently stalls forever. systemd-homed-firstboot.service is ConditionFirstBoot=yes, runs homectl firstboot --prompt-new-user, reads StandardInput=tty, and is ordered Before=systemd-user-sessions.service first-boot-complete.target. On any console nobody is attached to — PXE, datacenter provisioning, a sysupdate-driven reinstall, or our own QEMU smoke test with -serial file: — the boot transaction blocks at the prompt indefinitely.

The existing mask next to it covers systemd-firstboot.service only, and its own comment states the intent: "to prevent interactive prompts on headless first boot". So this was an incomplete mask, not a deliberate carve-out.

Premise verified three ways before changing anything:

  1. Nothing anywhere masks or disables the unit — a repo-wide grep for homed returns no matches, and so does a grep for StandardInput/--prompt/ConditionFirstBoot/first-boot-complete.target across files/ and elements/.
  2. It genuinely ships: FSDK 26.08.0 builds systemd with -Dfirstboot=true plus openssl/pam/libcryptsetup enabled, so meson's homed=auto resolves enabled.
  3. Runtime proof: the serial log of integration run 35375246792 ends at "Please create your user account!" / "Please enter user name to create (empty to skip):", and strings /usr/bin/homectl confirms that first string is literally inside the homectl binary.

Confirmed fixed by a subsequent run. With the unit masked, the wizard is gone (occurrences 1 -> 0) and the installed system boots clean through to localhost login:, with the issue banner re-rendering from http://10.0.2.15:8080/ to http://10.244.0.1:8080/ — the CNI bridge address, which means k0s actually came up.

Why in the image and not the smoke test. Masking it only in the Justfile would have turned installer-test green while every real headless install still hung. That is a green test covering a broken product. The bug is in the image; the test merely exposed it.

Audit of other units that could block a headless boot the same way. Enumerated every unit with StandardInput=tty* and checked activation path and boot-target ordering. Genuine blockers: systemd-firstboot.service (already masked) and systemd-homed-firstboot.service (this PR). Judged safe and deliberately left alone: the getty@/serial-getty@/console-getty family (login terminals; no boot target orders after them), the breakpoint-* units (only via an explicit systemd.break= argument), emergency/rescue (failure paths only), debug-shell and systemd-storagetm (not preset-enabled), and systemd-sysinstall.service (no ConditionFirstBoot, and in the live installer the prompt is the intended flow).

The mask is safe because the unit is reached via WantedBy=systemd-homed.service, i.e. a Wants= relationship, so masking cannot fail the boot.

Context: this is the third independent blocker on installer-test, alongside #173 (the /var udev deadlock) and #181 (the readiness gate). All three are needed; none is sufficient alone.

build, unit and docs pass. Needs two independent approvals, neither of which can be mine.

@castrojo

Copy link
Copy Markdown
Contributor Author

Runtime evidence: installer-test can pass, and this PR is one of the three things required.

I built a throwaway integration of #173 + #181 + #186 and ran it through CI. Run 35388029502 is GREEN — installer-test / Install and boot in QEMU conclusion success. That is the first time that job has ever passed in this repository. It was introduced at 50be1ed (#149) and every run on main since has failed: 34905862943, 34910341724, 34914418293, 34920225526.

Decisive serial console lines:

[    9.620696] bash[755]: kiosk-ready: polling https://127.0.0.1:8080
[   38.124609] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=2
[   69.115572] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=7
[   89.296275] bash[755]: KIOSK_CONSOLE_READY

and host-side: ==> KubeStellar Console is healthy: /healthz status ok, / returned HTTP 200.

Three independent blockers, all required; any two still fail:

  1. fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173 — the /var udev ordering deadlock. Without it the box sits in emergency mode and nothing downstream runs.
  2. fix(boot): mask systemd-homed-firstboot on the target image #186 — systemd-homed-firstboot.service masked. Without it the boot transaction blocks forever on homectl firstboot --prompt-new-user at a console nobody can answer. This is a product bug, not just a CI one: every headless first boot stalls.
  3. fix(ci): make the installer-test readiness gate reachable #181 — the readiness gate itself: probe from inside the guest over HTTPS (the proxy binds guest loopback, so no host-side probe can ever reach it), and emit the marker so it survives MaxLevelConsole=warning.

The marker is not a false positive. At t=38s and t=69s the loop was awake with k0s active and 10 pods present, but hz=000 root=000 — curl could not connect at all — and it refused to emit. It fired only at t=89s once both endpoints answered. The ExecStart body was also simulated offline with stubbed curl/jq: /healthz returns ok at round 3, / returns 200 at round 8, and the marker emits at round 8, never at 3. And it cannot be satisfied trivially by nginx alone — files/k0s/kiosk/nginx.conf defines no /healthz location, so both probes fall through location / and proxy_pass to the console service; nginx-up/console-down yields 502, which curl --fail suppresses. It went green because the console genuinely became ready.

The deadline is not a problem. Readiness is reached at t=89s against a 900s budget — roughly 10x headroom. No change to projectbluefin/actions is needed. (Note the 900 lives in that external reusable workflow, not this repo's Justfile, which defaults to 600.)

Caveat, stated plainly: the run exercised the <4> SyslogLevelPrefix variant of the marker on the throwaway branch, not #181's current head 87c381d, which writes >> /dev/console. Both defeat the same clamp, and #181's is if anything more robust since writing to the device bypasses journald unconditionally. Everything downstream of the marker plumbing — boot, k0s, CNI, the full KubeStellar stack, both HTTPS checks — is validated identically for either.

The integration branch was a throwaway: no temporary PR was opened, and the branch and worktree are deleted. Neither #173's nor #181's diff or head was touched by any of it.

@castrojo

Copy link
Copy Markdown
Contributor Author

Thanks for the review. No findings were raised, so no code changes were needed.

@castrojo

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #210, which lands the identical change plus the matching test and docs update.

Both wizards are now masked in elements/oci/bluefin-server-ddi.bst:

ln -sf /dev/null /layer/etc/systemd/system/systemd-firstboot.service
ln -sf /dev/null /layer/etc/systemd/system/systemd-homed-firstboot.service

Same reasoning as your description: both are ConditionFirstBoot=yes, read StandardInput=tty, and order before first-boot-complete.target, so either one left unmasked hangs a headless boot forever.

@castrojo castrojo closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant