Skip to content

[shared-ci] installer-test / Install and boot in QEMU failing across projectbluefin/server #218

Closed

Description

@hivecommons-hive

Summary

installer-test / Install and boot in QEMU is failing on main itself, not on any one pull request. It is currently red on 17 open PRs, which is one repository incident rather than 17 PR-local failures.

Per-PR retries will not clear it. This issue is the single place to track it.

Evidence that this is shared, not PR-local

main is red with the identical failure:

  • Latest run on main: Build DDI artifacts #486 — installer-test / Install and boot in QEMU, failed step "Install and boot exported server image".

Two of the blocked PRs cannot possibly have caused an installer boot failure:

Failure mode

The installed image boots, but the var partition never appears, so /var and everything ordered after it fail, and the readiness probe then burns its full 900s timeout:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for Podman auto-update timer.
[DEPEND] Dependency failed for Local File Systems.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.Zu4d5E
error: recipe `test-installer-artifact` failed with exit code 1

The KubeStellar Console timeout is a downstream symptom, not the cause — the console cannot come up on a machine with no /var. Diagnosis should target the missing by-partlabel/var device.

Regression window

Last green run on main, and every completed run since:

Run Date Result
#279 2026-09-14T21:12Z ✅ success
#281 2026-09-14T22:48Z ❌ failure
#303 2026-09-14T23:46Z ❌ failure
#325 2026-09-15T00:44Z ❌ failure
#350 2026-09-15T02:11Z ❌ failure
#465 2026-09-20T10:02Z ❌ failure
#486 2026-09-20T13:29Z ❌ failure

main has been red for roughly six days.

Likely culprit

Exactly one commit separates the last green run from the first red one:

50be1ed2  ci: run installer smoke through shared QEMU workflow (#149)

It touched .github/workflows/build.yml, Justfile, docs/skills/ci-tooling.md, docs/skills/ddi-installer-build.md.

In the Justfile it split show-me-the-future into a build phase and a new test-installer-artifact recipe, moving just build-installer / export-installer / export-sysext out of the smoke recipe and into the caller. It also replaced the self-healing sysext lookup:

-    K0S_RAW_ZST=""
-    if [ -d dist/sysext ]; then
-      K0S_RAW_ZST=$(find dist/sysext/ ... | head -n 1 || true)
-    fi
-    if [ -z "$K0S_RAW_ZST" ]; then
-      just export-sysext
-      K0S_RAW_ZST=$(find dist/sysext/ ... | head -n 1)
-    fi
+    K0S_RAW_ZST=$(find dist/sysext/ -maxdepth 1 -type f -name 'k0s-*.raw.zst' -print -quit 2>/dev/null || true)

test-installer-artifact now depends entirely on the caller having produced the right artifacts in dist/ beforehand. That is the most plausible place for the /var staging step to be operating on a different or incomplete artifact set than it did when the recipe built them itself. This is a lead, not a confirmed root cause — it has not been reproduced locally.

Affected PRs

#49, #80, #133, #136, #140, #143, #144, #162, #163, #182, #202, #206, #208, #211, #213, #214, #216

Requested handling

  • Treat this as one incident; do not retry the check per-PR.
  • The likely fix lives partly in .github/workflows/build.yml, which automated contributors cannot push to (the App token has no Workflows permission). A human or a workflows-capable agent will need to land any workflow-side change.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerFiled or owned by the scanner agent.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions