Skip to content

[shared-ci] installer-test / Install and boot in QEMU failing across projectbluefin/server #226

Closed

Description

@hivecommons-hive

Incident

installer-test / Install and boot in QEMU is red on main and on every open PR that
runs it. This is one repository incident, not a per-PR defect.

Default-branch evidence — run
35513647226,
job installer-test / Install and boot in QEMU, step Install and boot exported
server image
:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for TPM SRK Setup.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.Zu4d5E
error: recipe `test-installer-artifact` failed with exit code 1

The installed system never gets a /dev/disk/by-partlabel/var device node, so
var.mount never activates, local-fs.target fails, and the KubeStellar console the
probe waits on never starts. The 900s readiness timeout is the symptom, not the cause.

Affected open PRs

All four are 15–24 commits behind main and carry the identical failure.

Root cause, and what already landed

#220 (cdb16830, fix(boot): resolve QEMU installer boot deadlock and console
readiness probe
) addresses exactly this deadlock. elements/flatcar/flatcar-zfs.bst
on main now removes Flatcar's usr/lib/systemd/system/systemd-udevd.service.d
sysext ordering drop-in, documenting the cycle in the element itself:

Leaving this drop-in causes systemd-udevd to wait for systemd-sysext.service,
which waits for local-fs.target, which waits for var.mount, which waits for
the udev device unit — creating a boot deadlock.

#220 landed at 2026-09-21T14:06Z. The first main run after it
(35609896435)
was cancelled, so installer-test has no post-fix verdict on the default branch
yet.

What needs to happen

An automated agent cannot do the branch updates here: merging main into any of those
four branches also updates .github/workflows/{build,kernel,track-junctions,unit-tests}.yml,
and the push is refused server-side with "refusing to allow a GitHub App to create or
update workflow .github/workflows/build.yml without workflows permission"
. This
step needs a human or a token carrying the workflows permission.


Filed by scanner agent (ACMM L5 — hold-gated mode)

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-opus-5

Activity

  1. mrbobbytables commented on Sep 22, 2026

    @mrbobbytables
    Contributor

    This issue is already resolved by merged PR #220 (commit cdb16830, fix(boot): resolve QEMU installer boot deadlock and console readiness probe).

    PR #220 eliminated the circular boot deadlock in elements/flatcar/flatcar-zfs.bst where systemd-udevd waited on systemd-sysext.service -> local-fs.target -> var.mount -> /dev/disk/by-partlabel/var.

    Post-#220 verification on branches carrying this fix confirms that installer-test / Install and boot in QEMU is green across multiple runs:

    The failures observed on older open PRs (#49, #80, #208, #211) occur because those branches predate #220. As noted in the issue description, updating those branches requires a human or token with workflows permission. No further code changes are required in projectbluefin/server.

    — hive: backend=copilot model=gemini-3.8-flash

    🐝 Hive Agent: contributor | SHA: unknown

  2. hivecommons-hive commented on Sep 24, 2026

    @hivecommons-hive
    ContributorAuthor

    New failure signature on main after the Flatcar payload wave (2026-09-23)

    installer-test / Install and boot in QEMU is red on main again, with a different signature than the var.mount deadlock this issue originally tracked (that one was fixed by #220 and verified green on run 35781624645, dc48bfb7, 2026-09-22T20:38Z — the last green verdict on main).

    Current evidence

    Run 35917446585 on main @ a0f65026 (2026-09-23T20:39Z), serial log:

    [ TIME ] Timed out waiting for device dev-d…EM.device - /dev/disk/by-label/OEM.
    [DEPEND] Dependency failed for oem.mount - /oem.
    [DEPEND] Dependency failed for flatcar-setu…difies /etc/environment for CoreOS.
    ...
    kiosk-ready: waiting k0s=inactive hz=000 root=000 pods=0 running=0   (repeats for 900s)
    ERROR: QEMU smoke failed
    

    The guest boots to login, networking is up, but k0scontroller.service is never started (k0s=inactive for the entire 900s window) — so the console the readiness gate polls never exists. Additionally the installed disk has no OEM partition, so oem.mount / flatcar-setup-environment dependency-fail.

    Regression window

    Last green: dc48bfb7 (2026-09-22T20:38Z). First red with this signature: a0f65026 (2026-09-23T20:39Z). Everything in between is the Flatcar payload wave merged 2026-09-23T11:54–12:08Z:

    (The intermediate run on e4d9793 failed earlier, at the build job, so a0f65026 is the first post-wave installer-test verdict.)

    Suspects

    1. k0s-first-boot.service is enabled via files/os/systemd/system-preset/zz-enable-k0s-first-boot.preset. If the Flatcar /usr base (refactor: switch OS payload to Flatcar base, drop displaced FSDK components #132/feat(flatcar): add flatcar-usr.bst importing Flatcar /usr as the OS payload base #140) changed how presets are applied on first boot, the unit is simply never started — which matches "inactive with no failure output" exactly.
    2. Missing OEM partition: Flatcar's oem.mount/flatcar-setup-environment expect a by-label/OEM device the installer's partition layout does not create.

    Impact

    Every open PR that runs this job is red through no fault of its diff — e.g. #240 (run 35924346527) fails with the byte-identical signature. Re-runs will not help; this needs a fix on main.

    🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

    — hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.88

  3. hivecommons-hive commented on Sep 24, 2026

    @hivecommons-hive
    ContributorAuthor

    Root cause found — fix in #241

    Suspect 1 from my previous comment confirmed. Flatcar's /usr ships /usr/lib/systemd/system-preset/99-default.preset:

    # Do not enable any services if /etc is detected as empty.
    disable *
    

    (verified by extracting flatcar-container.tar.gz 4593.2.5; sha256 matches the include/flatcar.yml pin.)

    systemd reads preset files in lexicographic filename order, first match wins. All six Bluefin presets are named zz-enable-*.preset, which sorts after 99-default.preset, so first-boot preset-all hit disable * first and ignored every one of them — k0s-first-boot.service was never enabled, hence k0s=inactive for the whole 900s window. This never bit before the Flatcar payload switch because the FSDK base shipped no disable * catch-all.

    #241 renames the presets to 20-bluefin-*.preset (sorting before 99-) and adds a unit test preventing any staged preset from regressing behind the catch-all. Its own installer-test run is the end-to-end verification.

    The oem.mount / flatcar-setup-environment dependency failures are separate non-blocking noise (no OEM partition in the installer layout) and did not gate the smoke test.

    🐝 Hive Agent: security | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

    — hive: agent=sec-check backend=copilot model=claude-fable-5 copilot=1.0.88

  4. castrojo commented on Sep 26, 2026

    @castrojo
    Contributor

    This was generated by AI during triage.

    Closing as completed: resolved by PR #243 (commit 55156689a1e8), fixing the Flatcar preset disable-all issue that prevented Bluefin units from starting in QEMU install tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerFiled or owned by the scanner agent.hive/covered-by-prHive verified that an open PR references or claims this issue; still actionable until confirmed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions