Repository navigation
fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock - #173
mrbobbytables wants to merge 2 commits into
Conversation
…deadlock In elements/flatcar/flatcar-zfs.bst, the extracted Flatcar ZFS sysext archive contains usr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf which configures After=systemd-sysext.service. Because Bluefin Server extracts ZFS directly into /usr at build time, this ordering override is unnecessary. In addition, systemd-sysext.service runs After=local-fs.target, which waits for var.mount. Because var.mount requires the underlying block device unit from systemd-udevd to transition to active, this caused a circular dependency deadlock during target boot where udevd could never start, resulting in systemd timing out waiting for /dev/disk/by-partlabel/var. Remove systemd-udevd.service.d during flatcar-zfs installation, and add contract assertions in test_installer_contract.py. Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
This is the PR that unbreaks main, and I want the evidence on the record because its CI is still red and that red is misleading.
What it fixes. main at tip 5ebfeae (run 34920225526) installs and then drops to emergency mode:
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Local File Systems.
You are in emergency mode.
This branch (run 35182091959) boots the installed system through to full userspace — no /var failure, no Local File Systems failure, no emergency mode, no maintenance prompt. It reaches the first-boot wizard:
Bluefin Server 26.08.0
Please create your user account!
The installer phase powers off cleanly in both runs (reboot: Power down); the divergence is entirely in the second, installed-system boot. Strict improvement, no observed regression.
Why the fix is right and minimal. The change is one line in elements/flatcar/flatcar-zfs.bst removing the systemd-udevd.service.d drop-in that the Flatcar ZFS sysext carries, plus a contract test. That drop-in ordered udevd in a way that deadlocked against var.mount. Note files/os/systemd/system/var.mount is left completely untouched — it still uses What=/dev/disk/by-partlabel/var — which is what proves the failure was ordering rather than device-node naming. That also settles the competing hypotheses: #167 attributed it to ID_PART_ENTRY_NAME being unavailable on virtio and #165 to root-a starving /var of space; both are falsified by this run, since by-partlabel resolves and SizeMaxBytes=16G is unchanged here. I have closed both with that reasoning.
I checked the blast radius rather than trusting the diff size: %{install-root} at that point contains only the unsquashed flatcar-zfs.raw contents plus the preserved /usr/lib/sysexts/zfs.raw, so this rm -rf cannot affect drop-ins owned by any other element.
The residual red check is not this PR's fault. installer-test still fails here at "Timed out after 900s waiting for KubeStellar Console readiness". That gate is unreachable by construction, for two independent pre-existing reasons:
- Transport.
Justfile:329useshostfwd=tcp:127.0.0.1:8080-:8080; libslirp substitutes the guest DHCP address (10.0.2.15) when the guest side is unspecified, so the forward lands on 10.0.2.15:8080. Butfiles/k0s/manifests/kubestellar/41-kubestellar-kiosk-proxy.yamlbindshostIP: 127.0.0.1— guest loopback only (commit e1ea72a, #75). The forwarded port never reaches the listener. - Protocol.
files/k0s/kiosk/nginx.confislisten 8080 sslwitherror_page 497 https://...(commit c752c48), whileJustfile:351pollshttp://127.0.0.1:8080/healthzwith--failand requiresjq -e '.status == "ok"'. Plain HTTP against the TLS listener returns a 497 redirect, never that JSON — so the gate fails even against a perfectly healthy stack.
Both predate the gate, which was added five days later in #149. Confirming that: installer-test has never passed on main once. It was introduced at 50be1ed and every subsequent main run failed (34905862943, 34910341724, 34914418293, 34920225526); the last green main run, 34897477278, does not contain the job at all. There is no green baseline to regress from.
Approving. Under the two-reviewer policy this now needs one more independent approval — not mine, since I have given it — and I would prioritise that review over everything else in the queue, because every other PR in this repo is currently sitting behind a red installer-test that this change is a precondition for fixing.
Two follow-ups that are explicitly not this PR's job and should be separate issues: the loopback/hostfwd mismatch and the http-vs-https probe above. Landing this will not turn installer-test green on its own — #178 addresses a third component of the same gate (argocd never becoming ready) — but it removes the emergency-mode failure that currently blocks everything else from even being evaluated.
|
Correction to my review above, from runtime evidence obtained since. What I got wrong. I quoted this branch's serial log reaching as evidence the installed system had booted through to a first-boot wizard, and treated it as a success indicator. It is not. That is where boot stalls, permanently.
Note What this does not change: your fix is correct and is confirmed by runtime evidence. I built a throwaway integration of this PR plus #181 and ran it through CI (run 35375246792). The What it does change: the ordering of what is left. I previously said That third blocker is not yours and I am not asking you to fix it here. This PR remains a strict improvement over My approval stands. Still needs one more independent approval, and I would still prioritise it over anything else in the queue. |
|
Runtime evidence: I built a throwaway integration of #173 + #181 + #186 and ran it through CI. Run 35388029502 is GREEN — Decisive serial console lines: and host-side: Three independent blockers, all required; any two still fail:
The marker is not a false positive. At t=38s and t=69s the loop was awake with k0s active and 10 pods present, but The deadline is not a problem. Readiness is reached at t=89s against a 900s budget — roughly 10x headroom. No change to Caveat, stated plainly: the run exercised the The integration branch was a throwaway: no temporary PR was opened, and the branch and worktree are deleted. Neither #173's nor #181's diff or head was touched by any of it. |
|
Final precision note, correcting the evidence I cited for this PR (the conclusion is unchanged and now rests on better proof). In my original review I wrote that this branch's installer-test run "reaches The fix is still proven, by stronger evidence: Negative. Before, main run 34920225526 without this PR: After, integration run 35375246792 with this PR: Positive. In run 35382279230 the box reaches a getty login prompt, and the issue banner re-renders from So: removing the udevd sysext ordering drop-in fixes the Also worth recording for whoever merges this: no temporary pull request was ever opened for any of that verification. My approval stands, and this is still the one to land first. |
There was a problem hiding this comment.
style
- (low)
elements/flatcar/flatcar-zfs.bst:38—rm -rf .../systemd-udevd.service.ddeletes the whole drop-in directory rather than the specific10-zfs.confnamed in the root-cause analysis. If a future Flatcar sysext ships an additional drop-in there (e.g., env or reload settings udevd actually needs), it will be silently removed. Targeting10-zfs.conf(or failing loudly if unexpected files appear) would keep the fix surgical.
Verified the fix and new contract tests against the tree at 589d5d7: the drop-in removal is real, and every assertion in tests/unit/test_installer_contract.py:141-168 matches actual content (files/os/systemd/system/var.mount:7, files/installer/repart.d/30-var.conf:9, elements/oci/bluefin-server-ddi.bst:67, Justfile fstab.extra credential at line 330).
No findings from: security, intent-alignment, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
… sysext ordering drop-in Address review feedback from @kubestellar-hive: surgically target 10-zfs.conf rather than removing the entire systemd-udevd.service.d directory, preserving any future drop-ins that Flatcar sysexts might supply, and update test_installer_contract assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
|
Addressed review feedback from @kubestellar-hive in commit Surgically target
|
|
Closing as superseded by #210. You found the right deadlock — udevd ordered #210 carries both halves ( Your |
Fixes #158
Summary
During target boot in QEMU smoke tests (
just show-me-the-future), systemd reported:as well as
Timed out waiting for device /dev/ttyS0.Root Cause
In
elements/flatcar/flatcar-zfs.bst, the Flatcar ZFS system extension image is extracted into the element install root. The upstream Flatcar sysext includesusr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf, which configures:While Flatcar treats ZFS as an unmerged runtime sysext (and thus defers udev until sysext merge), Bluefin Server extracts ZFS directly into
/usrat build time.Because
systemd-sysext.serviceis orderedAfter=local-fs.target, andlocal-fs.targetwaits for local mounts includingvar.mount(/dev/disk/by-partlabel/var), this created a circular ordering deadlock during target boot:systemd-udevd.servicewaited forsystemd-sysext.servicesystemd-sysext.servicewaited forlocal-fs.targetlocal-fs.targetwaited forvar.mountvar.mountwaited fordev-disk-by\x2dpartlabel-var.device(which can only become active oncesystemd-udevdprocesses the block device uevent)Because
systemd-udevdnever started, no device units or devlinks (/dev/disk/by-partlabel/var,/dev/ttyS0) were ever marked active, and systemd dropped to emergency mode after the 90-second device timeout.Solution
systemd-udevd.service.dduringflatcar-zfs.bstinstallation sosystemd-udevdstarts normally during early boot insysinit.target.tests/unit/test_installer_contract.pyto verify thatflatcar-zfs.bstremoves the udevd sysext ordering drop-in and that the installer, DDI,var.mount, andJustfilecredentials maintain a consistent partition label contract.Verification
pytest tests/unit: 209 passed, 1 xfailedbats tests/unit: 62 passedpython3 .github/scripts/docs-checks.py: passedpython3 .github/scripts/check-release-version.py: passedpython3 .github/scripts/check-k0s-version.py: passed— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:589d5d7