Repository navigation
Description
Incident
installer-test / Install and boot in QEMU is red on main and on every open PR that
runs it. This is one repository incident, not a per-PR defect.
Default-branch evidence — run
35513647226,
job installer-test / Install and boot in QEMU, step Install and boot exported
server image:
[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for TPM SRK Setup.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.Zu4d5E
error: recipe `test-installer-artifact` failed with exit code 1
The installed system never gets a /dev/disk/by-partlabel/var device node, so
var.mount never activates, local-fs.target fails, and the KubeStellar console the
probe waits on never starts. The 900s readiness timeout is the symptom, not the cause.
Affected open PRs
- chore(deps): update taiki-e/install-action digest to 84f5ac3, and gnome-build-meta to 50.4-5 #49 —
renovate/github-actions-(non-major) - feat: CoreOS-compatible login hardening and core operator key provisioning #80 —
feat/coreos-compatible-hardened-login - docs: appliance identity, mDNS, and Bash prompt design #208 —
docs/appliance-identity-design - chore(renovate): disable the homebrew manager for binary-asset formulae #211 —
fix/renovate-homebrew-binary-formulae
All four are 15–24 commits behind main and carry the identical failure.
Root cause, and what already landed
#220 (cdb16830, fix(boot): resolve QEMU installer boot deadlock and console
readiness probe) addresses exactly this deadlock. elements/flatcar/flatcar-zfs.bst
on main now removes Flatcar's usr/lib/systemd/system/systemd-udevd.service.d
sysext ordering drop-in, documenting the cycle in the element itself:
Leaving this drop-in causes systemd-udevd to wait for systemd-sysext.service,
which waits for local-fs.target, which waits for var.mount, which waits for
the udev device unit — creating a boot deadlock.
#220 landed at 2026-09-21T14:06Z. The first main run after it
(35609896435)
was cancelled, so installer-test has no post-fix verdict on the default branch
yet.
What needs to happen
- Re-run
Build DDI artifactsonmainto get a post-fix(boot): resolve QEMU installer boot deadlock and console readiness probe #220 verdict for
installer-test / Install and boot in QEMU. - If green, press Update branch on chore(deps): update taiki-e/install-action digest to 84f5ac3, and gnome-build-meta to 50.4-5 #49, feat: CoreOS-compatible login hardening and core operator key provisioning #80, docs: appliance identity, mDNS, and Bash prompt design #208 and chore(renovate): disable the homebrew manager for binary-asset formulae #211 so they pick up
cdb16830, then re-run theirinstaller-testlegs. - If still red, diagnose further — the partlabel
vardevice is still missing and
fix(boot): resolve QEMU installer boot deadlock and console readiness probe #220 was not sufficient.
An automated agent cannot do the branch updates here: merging main into any of those
four branches also updates .github/workflows/{build,kernel,track-junctions,unit-tests}.yml,
and the push is refused server-side with "refusing to allow a GitHub App to create or
update workflow .github/workflows/build.yml without workflows permission". This
step needs a human or a token carrying the workflows permission.
Filed by scanner agent (ACMM L5 — hold-gated mode)
🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown
— hive: agent=scanner backend=copilot model=claude-opus-5
Activity
mrbobbytables commentedon Sep 22, 2026
This issue is already resolved by merged PR #220 (commit
cdb16830, fix(boot): resolve QEMU installer boot deadlock and console readiness probe).PR #220 eliminated the circular boot deadlock in
elements/flatcar/flatcar-zfs.bstwheresystemd-udevdwaited onsystemd-sysext.service->local-fs.target->var.mount->/dev/disk/by-partlabel/var.Post-#220 verification on branches carrying this fix confirms that
installer-test / Install and boot in QEMUis green across multiple runs:installer-test / Install and boot in QEMUpassed in 3m30sinstaller-test / Install and boot in QEMUpassed in 3m44sinstaller-test / Install and boot in QEMUpassed in 3m33sThe failures observed on older open PRs (#49, #80, #208, #211) occur because those branches predate #220. As noted in the issue description, updating those branches requires a human or token with
workflowspermission. No further code changes are required inprojectbluefin/server.— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:unknownhivecommons-hive commentedon Sep 24, 2026
New failure signature on
mainafter the Flatcar payload wave (2026-09-23)installer-test / Install and boot in QEMUis red onmainagain, with a different signature than thevar.mountdeadlock this issue originally tracked (that one was fixed by #220 and verified green on run 35781624645,dc48bfb7, 2026-09-22T20:38Z — the last green verdict onmain).Current evidence
Run 35917446585 on
main@a0f65026(2026-09-23T20:39Z), serial log:The guest boots to login, networking is up, but
k0scontroller.serviceis never started (k0s=inactivefor the entire 900s window) — so the console the readiness gate polls never exists. Additionally the installed disk has noOEMpartition, sooem.mount/flatcar-setup-environmentdependency-fail.Regression window
Last green:
dc48bfb7(2026-09-22T20:38Z). First red with this signature:a0f65026(2026-09-23T20:39Z). Everything in between is the Flatcar payload wave merged 2026-09-23T11:54–12:08Z:83fdee5refactor: switch OS payload to Flatcar base, drop displaced FSDK components (refactor: switch OS payload to Flatcar base, drop displaced FSDK components #132)874679afeat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base (feat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base #140)498d192refactor(arch): split release-version into installer and Flatcar axes (refactor(arch): split release-version into installer and Flatcar axes #136)(The intermediate run on
e4d9793failed earlier, at thebuildjob, soa0f65026is the first post-wave installer-test verdict.)Suspects
k0s-first-boot.serviceis enabled viafiles/os/systemd/system-preset/zz-enable-k0s-first-boot.preset. If the Flatcar/usrbase (refactor: switch OS payload to Flatcar base, drop displaced FSDK components #132/feat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base #140) changed how presets are applied on first boot, the unit is simply never started — which matches "inactive with no failure output" exactly.OEMpartition: Flatcar'soem.mount/flatcar-setup-environmentexpect aby-label/OEMdevice the installer's partition layout does not create.Impact
Every open PR that runs this job is red through no fault of its diff — e.g. #240 (run 35924346527) fails with the byte-identical signature. Re-runs will not help; this needs a fix on
main.🐝 Hive Agent:
security| Instance:hosted-projectbluefin-knuckle-gjvq| SHA:unknown— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.88
hivecommons-hive commentedon Sep 24, 2026
Root cause found — fix in #241
Suspect 1 from my previous comment confirmed. Flatcar's
/usrships/usr/lib/systemd/system-preset/99-default.preset:(verified by extracting
flatcar-container.tar.gz4593.2.5; sha256 matches theinclude/flatcar.ymlpin.)systemd reads preset files in lexicographic filename order, first match wins. All six Bluefin presets are named
zz-enable-*.preset, which sorts after99-default.preset, so first-bootpreset-allhitdisable *first and ignored every one of them —k0s-first-boot.servicewas never enabled, hencek0s=inactivefor the whole 900s window. This never bit before the Flatcar payload switch because the FSDK base shipped nodisable *catch-all.#241 renames the presets to
20-bluefin-*.preset(sorting before99-) and adds a unit test preventing any staged preset from regressing behind the catch-all. Its owninstaller-testrun is the end-to-end verification.The
oem.mount/flatcar-setup-environmentdependency failures are separate non-blocking noise (noOEMpartition in the installer layout) and did not gate the smoke test.🐝 Hive Agent:
security| Instance:hosted-projectbluefin-knuckle-gjvq| SHA:unknown— hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.88
castrojo commentedon Sep 26, 2026
Closing as completed: resolved by PR #243 (commit
55156689a1e8), fixing the Flatcar preset disable-all issue that prevented Bluefin units from starting in QEMU install tests.