Skip to content

fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock - #173

Closed
mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/installer-var-partition
Closed

mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/installer-var-partition

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Fixes #158

Summary

During target boot in QEMU smoke tests (just show-me-the-future), systemd reported:

Timed out waiting for device /dev/disk/by-partlabel/var.
Dependency failed for /var.

as well as Timed out waiting for device /dev/ttyS0.

Root Cause

In elements/flatcar/flatcar-zfs.bst, the Flatcar ZFS system extension image is extracted into the element install root. The upstream Flatcar sysext includes usr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf, which configures:

[Unit]
After=systemd-sysext.service

While Flatcar treats ZFS as an unmerged runtime sysext (and thus defers udev until sysext merge), Bluefin Server extracts ZFS directly into /usr at build time.

Because systemd-sysext.service is ordered After=local-fs.target, and local-fs.target waits for local mounts including var.mount (/dev/disk/by-partlabel/var), this created a circular ordering deadlock during target boot:

  1. systemd-udevd.service waited for systemd-sysext.service
  2. systemd-sysext.service waited for local-fs.target
  3. local-fs.target waited for var.mount
  4. var.mount waited for dev-disk-by\x2dpartlabel-var.device (which can only become active once systemd-udevd processes the block device uevent)

Because systemd-udevd never started, no device units or devlinks (/dev/disk/by-partlabel/var, /dev/ttyS0) were ever marked active, and systemd dropped to emergency mode after the 90-second device timeout.

Solution

  • Remove systemd-udevd.service.d during flatcar-zfs.bst installation so systemd-udevd starts normally during early boot in sysinit.target.
  • Add contract tests in tests/unit/test_installer_contract.py to verify that flatcar-zfs.bst removes the udevd sysext ordering drop-in and that the installer, DDI, var.mount, and Justfile credentials maintain a consistent partition label contract.

Verification

  • pytest tests/unit: 209 passed, 1 xfailed
  • bats tests/unit: 62 passed
  • python3 .github/scripts/docs-checks.py: passed
  • python3 .github/scripts/check-release-version.py: passed
  • python3 .github/scripts/check-k0s-version.py: passed

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 589d5d7

…deadlock

In elements/flatcar/flatcar-zfs.bst, the extracted Flatcar ZFS sysext archive contains usr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf which configures After=systemd-sysext.service.

Because Bluefin Server extracts ZFS directly into /usr at build time, this ordering override is unnecessary. In addition, systemd-sysext.service runs After=local-fs.target, which waits for var.mount. Because var.mount requires the underlying block device unit from systemd-udevd to transition to active, this caused a circular dependency deadlock during target boot where udevd could never start, resulting in systemd timing out waiting for /dev/disk/by-partlabel/var.

Remove systemd-udevd.service.d during flatcar-zfs installation, and add contract assertions in test_installer_contract.py.

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5

castrojo
castrojo previously approved these changes Sep 18, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the PR that unbreaks main, and I want the evidence on the record because its CI is still red and that red is misleading.

What it fixes. main at tip 5ebfeae (run 34920225526) installs and then drops to emergency mode:

[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode.

This branch (run 35182091959) boots the installed system through to full userspace — no /var failure, no Local File Systems failure, no emergency mode, no maintenance prompt. It reaches the first-boot wizard:

Bluefin Server 26.08.0
Please create your user account!

The installer phase powers off cleanly in both runs (reboot: Power down); the divergence is entirely in the second, installed-system boot. Strict improvement, no observed regression.

Why the fix is right and minimal. The change is one line in elements/flatcar/flatcar-zfs.bst removing the systemd-udevd.service.d drop-in that the Flatcar ZFS sysext carries, plus a contract test. That drop-in ordered udevd in a way that deadlocked against var.mount. Note files/os/systemd/system/var.mount is left completely untouched — it still uses What=/dev/disk/by-partlabel/var — which is what proves the failure was ordering rather than device-node naming. That also settles the competing hypotheses: #167 attributed it to ID_PART_ENTRY_NAME being unavailable on virtio and #165 to root-a starving /var of space; both are falsified by this run, since by-partlabel resolves and SizeMaxBytes=16G is unchanged here. I have closed both with that reasoning.

I checked the blast radius rather than trusting the diff size: %{install-root} at that point contains only the unsquashed flatcar-zfs.raw contents plus the preserved /usr/lib/sysexts/zfs.raw, so this rm -rf cannot affect drop-ins owned by any other element.

The residual red check is not this PR's fault. installer-test still fails here at "Timed out after 900s waiting for KubeStellar Console readiness". That gate is unreachable by construction, for two independent pre-existing reasons:

  1. Transport. Justfile:329 uses hostfwd=tcp:127.0.0.1:8080-:8080; libslirp substitutes the guest DHCP address (10.0.2.15) when the guest side is unspecified, so the forward lands on 10.0.2.15:8080. But files/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yaml binds hostIP: 127.0.0.1 — guest loopback only (commit e1ea72a, #75). The forwarded port never reaches the listener.
  2. Protocol. files/k0s/kiosk/nginx.conf is listen 8080 ssl with error_page 497 https://... (commit c752c48), while Justfile:351 polls http://127.0.0.1:8080/healthz with --fail and requires jq -e '.status == "ok"'. Plain HTTP against the TLS listener returns a 497 redirect, never that JSON — so the gate fails even against a perfectly healthy stack.

Both predate the gate, which was added five days later in #149. Confirming that: installer-test has never passed on main once. It was introduced at 50be1ed and every subsequent main run failed (34905862943, 34910341724, 34914418293, 34920225526); the last green main run, 34897477278, does not contain the job at all. There is no green baseline to regress from.

Approving. Under the two-reviewer policy this now needs one more independent approval — not mine, since I have given it — and I would prioritise that review over everything else in the queue, because every other PR in this repo is currently sitting behind a red installer-test that this change is a precondition for fixing.

Two follow-ups that are explicitly not this PR's job and should be separate issues: the loopback/hostfwd mismatch and the http-vs-https probe above. Landing this will not turn installer-test green on its own — #178 addresses a third component of the same gate (argocd never becoming ready) — but it removes the emergency-mode failure that currently blocks everything else from even being evaluated.

@castrojo

castrojo commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Correction to my review above, from runtime evidence obtained since.

What I got wrong. I quoted this branch's serial log reaching

Bluefin Server 26.08.0
🏠 Please create your user account!
🪪 Please enter user name to create (empty to skip):

as evidence the installed system had booted through to a first-boot wizard, and treated it as a success indicator. It is not. That is where boot stalls, permanently.

systemd-homed-firstboot.service runs homectl firstboot --prompt-new-user with ConditionFirstBoot=yes, StandardInput=tty and Before=first-boot-complete.target. The smoke VM is launched with -serial file:, so nothing can ever answer that prompt. The boot transaction blocks there, k0s never starts, and the console readiness poll consequently times out at 900s. The log simply ends mid-prompt.

Note elements/oci/bluefin-server-ddi.bst masks systemd-firstboot.service — a different unit. systemd-homed-firstboot.service is not masked.

What this does not change: your fix is correct and is confirmed by runtime evidence. I built a throwaway integration of this PR plus #181 and ran it through CI (run 35375246792). The build job succeeded and the installed system shows zero Dependency failed lines and zero emergency mode, against main run 34920225526 which shows Timed out waiting for device /dev/disk/by-partlabel/var, Dependency failed for /var and You are in emergency mode. So removing the udevd sysext ordering drop-in genuinely resolves the /var deadlock, and by-partlabel resolves fine once the ordering is gone — which remains the evidence that #167's and #165's competing explanations were wrong.

What it does change: the ordering of what is left. I previously said installer-test needed this PR plus #181's readiness-gate fix. There is a third, independent blocker in front of both — the homed-firstboot stall — and until it is resolved #181's in-guest probe unit never even executes (there is no kiosk-ready: polling line in the log at all). A confirmation run masking systemd-homed-firstboot.service is in flight to verify it is the last one.

That third blocker is not yours and I am not asking you to fix it here. This PR remains a strict improvement over main with no observed regression, and it is still the precondition for everything else — nothing downstream of /var can even be evaluated until it lands.

My approval stands. Still needs one more independent approval, and I would still prioritise it over anything else in the queue.

@castrojo

Copy link
Copy Markdown
Contributor

Runtime evidence: installer-test can pass, and this PR is one of the three things required.

I built a throwaway integration of #173 + #181 + #186 and ran it through CI. Run 35388029502 is GREEN — installer-test / Install and boot in QEMU conclusion success. That is the first time that job has ever passed in this repository. It was introduced at 50be1ed (#149) and every run on main since has failed: 34905862943, 34910341724, 34914418293, 34920225526.

Decisive serial console lines:

[    9.620696] bash[755]: kiosk-ready: polling https://127.0.0.1:8080
[   38.124609] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=2
[   69.115572] bash[755]: kiosk-ready: waiting k0s=active hz=000 root=000 pods=10 running=7
[   89.296275] bash[755]: KIOSK_CONSOLE_READY

and host-side: ==> KubeStellar Console is healthy: /healthz status ok, / returned HTTP 200.

Three independent blockers, all required; any two still fail:

  1. fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173 — the /var udev ordering deadlock. Without it the box sits in emergency mode and nothing downstream runs.
  2. fix(boot): mask systemd-homed-firstboot on the target image #186 — systemd-homed-firstboot.service masked. Without it the boot transaction blocks forever on homectl firstboot --prompt-new-user at a console nobody can answer. This is a product bug, not just a CI one: every headless first boot stalls.
  3. fix(ci): make the installer-test readiness gate reachable #181 — the readiness gate itself: probe from inside the guest over HTTPS (the proxy binds guest loopback, so no host-side probe can ever reach it), and emit the marker so it survives MaxLevelConsole=warning.

The marker is not a false positive. At t=38s and t=69s the loop was awake with k0s active and 10 pods present, but hz=000 root=000 — curl could not connect at all — and it refused to emit. It fired only at t=89s once both endpoints answered. The ExecStart body was also simulated offline with stubbed curl/jq: /healthz returns ok at round 3, / returns 200 at round 8, and the marker emits at round 8, never at 3. And it cannot be satisfied trivially by nginx alone — files/k0s/kiosk/nginx.conf defines no /healthz location, so both probes fall through location / and proxy_pass to the console service; nginx-up/console-down yields 502, which curl --fail suppresses. It went green because the console genuinely became ready.

The deadline is not a problem. Readiness is reached at t=89s against a 900s budget — roughly 10x headroom. No change to projectbluefin/actions is needed. (Note the 900 lives in that external reusable workflow, not this repo's Justfile, which defaults to 600.)

Caveat, stated plainly: the run exercised the <4> SyslogLevelPrefix variant of the marker on the throwaway branch, not #181's current head 87c381d, which writes >> /dev/console. Both defeat the same clamp, and #181's is if anything more robust since writing to the device bypasses journald unconditionally. Everything downstream of the marker plumbing — boot, k0s, CNI, the full KubeStellar stack, both HTTPS checks — is validated identically for either.

The integration branch was a throwaway: no temporary PR was opened, and the branch and worktree are deleted. Neither #173's nor #181's diff or head was touched by any of it.

@castrojo

Copy link
Copy Markdown
Contributor

Final precision note, correcting the evidence I cited for this PR (the conclusion is unchanged and now rests on better proof).

In my original review I wrote that this branch's installer-test run "reaches Reached target Local File Systems and Reached target Basic System". Those strings do appear in the job log, but they come from the installer environment's boot, not the installed system's. The installed system is deliberately quiet: the target UKI cmdline carries quiet loglevel=3 (elements/oci/bluefin-server-installer.bst:140) and the DDI writes ForwardToConsole=no / MaxLevelConsole=warning (elements/oci/bluefin-server-ddi.bst:59). So the absence of those targets on the installed console is by design, and their presence was not evidence about the installed boot. My attribution was wrong.

The fix is still proven, by stronger evidence:

Negative. Before, main run 34920225526 without this PR:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode.

After, integration run 35375246792 with this PR: grep -c "Dependency failed" = 0. Zero "emergency", zero /var errors anywhere in the job log.

Positive. In run 35382279230 the box reaches a getty login prompt, and the issue banner re-renders from http://10.0.2.15:8080/ (the QEMU DHCP address) to http://10.244.0.1:8080/ (the CNI bridge) — which proves k0s and its pod network actually came up, not merely that errors stopped printing. And in the decisive run 35388029502 the in-guest probe starts at t=9.6s and the console reports healthy at t=89s.

So: removing the udevd sysext ordering drop-in fixes the /var deadlock, and by-partlabel resolves correctly once the ordering is gone — which remains the evidence that #167's ID_PART_ENTRY_NAME theory and #165's/#185's disk-starvation theory were both wrong.

Also worth recording for whoever merges this: no temporary pull request was ever opened for any of that verification. build.yml accepts workflow_dispatch, which takes an arbitrary --ref, so the whole chain ran on a throwaway branch with no PR; track-refs and release both self-skip on dispatch. The branch and worktree are deleted and this PR's head is untouched at 589d5d7.

My approval stands, and this is still the one to land first.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style

  • (low) elements/flatcar/flatcar-zfs.bst:38 — rm -rf .../systemd-udevd.service.d deletes the whole drop-in directory rather than the specific 10-zfs.conf named in the root-cause analysis. If a future Flatcar sysext ships an additional drop-in there (e.g., env or reload settings udevd actually needs), it will be silently removed. Targeting 10-zfs.conf (or failing loudly if unexpected files appear) would keep the fix surgical.

Verified the fix and new contract tests against the tree at 589d5d7: the drop-in removal is real, and every assertion in tests/unit/test_installer_contract.py:141-168 matches actual content (files/os/systemd/system/var.mount:7, files/installer/repart.d/30-var.conf:9, elements/oci/bluefin-server-ddi.bst:67, Justfile fstab.extra credential at line 330).

No findings from: security, intent-alignment, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

… sysext ordering drop-in

Address review feedback from @kubestellar-hive: surgically target 10-zfs.conf rather than removing the entire systemd-udevd.service.d directory, preserving any future drop-ins that Flatcar sysexts might supply, and update test_installer_contract assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @kubestellar-hive in commit 84156e3:

Surgically target 10-zfs.conf specifically rather than removing the entire systemd-udevd.service.d directory, and updated the contract assertion in tests/unit/test_installer_contract.py to match.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor

Closing as superseded by #210.

You found the right deadlock — udevd ordered After=systemd-sysext.service → local-fs.target → var.mount → the udev device unit. But removing 10-zfs.conf from %{install-root} is not sufficient on its own: Flatcar also ships /usr/lib/tmpfiles.d/10-zfs.conf, which recreates the drop-in under /etc/systemd/system/systemd-udevd.service.d/ at boot. The build-time removal is silently undone on the first systemd-tmpfiles run.

#210 carries both halves (elements/flatcar/flatcar-zfs.bst):

rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d"
if [ -f "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" ]; then
  sed -i "/systemd-udevd\.service\.d/d" "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf"
fi

Your test_flatcar_zfs_removes_udevd_sysext_ordering_dropin assertion is worth keeping — if you want to re-file it against the rm -rf form plus a tmpfiles assertion, I will take it.

@castrojo castrojo closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(installer): QEMU smoke cannot discover /var partition

2 participants