Repository navigation
Description
Summary
installer-test / Install and boot in QEMU is failing on main itself, not on any one pull request. It is currently red on 17 open PRs, which is one repository incident rather than 17 PR-local failures.
Per-PR retries will not clear it. This issue is the single place to track it.
Evidence that this is shared, not PR-local
main is red with the identical failure:
- Latest run on
main: Build DDI artifacts #486 —installer-test / Install and boot in QEMU, failed step "Install and boot exported server image".
Two of the blocked PRs cannot possibly have caused an installer boot failure:
- docs: appliance identity, mDNS, and Bash prompt design #208 —
docs: appliance identity, mDNS, and Bash prompt design— documentation only. - chore(deps): update taiki-e/install-action digest to 84f5ac3, and gnome-build-meta to 50.4-5 #49 — a Renovate
github-actionsdigest bump.
Failure mode
The installed image boots, but the var partition never appears, so /var and everything ordered after it fail, and the readiness probe then burns its full 900s timeout:
[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for Podman auto-update timer.
[DEPEND] Dependency failed for Local File Systems.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.Zu4d5E
error: recipe `test-installer-artifact` failed with exit code 1
The KubeStellar Console timeout is a downstream symptom, not the cause — the console cannot come up on a machine with no /var. Diagnosis should target the missing by-partlabel/var device.
Regression window
Last green run on main, and every completed run since:
| Run | Date | Result |
|---|---|---|
| #279 | 2026-09-14T21:12Z | ✅ success |
| #281 | 2026-09-14T22:48Z | ❌ failure |
| #303 | 2026-09-14T23:46Z | ❌ failure |
| #325 | 2026-09-15T00:44Z | ❌ failure |
| #350 | 2026-09-15T02:11Z | ❌ failure |
| #465 | 2026-09-20T10:02Z | ❌ failure |
| #486 | 2026-09-20T13:29Z | ❌ failure |
main has been red for roughly six days.
Likely culprit
Exactly one commit separates the last green run from the first red one:
50be1ed2 ci: run installer smoke through shared QEMU workflow (#149)
It touched .github/workflows/build.yml, Justfile, docs/skills/ci-tooling.md, docs/skills/ddi-installer-build.md.
In the Justfile it split show-me-the-future into a build phase and a new test-installer-artifact recipe, moving just build-installer / export-installer / export-sysext out of the smoke recipe and into the caller. It also replaced the self-healing sysext lookup:
- K0S_RAW_ZST=""
- if [ -d dist/sysext ]; then
- K0S_RAW_ZST=$(find dist/sysext/ ... | head -n 1 || true)
- fi
- if [ -z "$K0S_RAW_ZST" ]; then
- just export-sysext
- K0S_RAW_ZST=$(find dist/sysext/ ... | head -n 1)
- fi
+ K0S_RAW_ZST=$(find dist/sysext/ -maxdepth 1 -type f -name 'k0s-*.raw.zst' -print -quit 2>/dev/null || true)test-installer-artifact now depends entirely on the caller having produced the right artifacts in dist/ beforehand. That is the most plausible place for the /var staging step to be operating on a different or incomplete artifact set than it did when the recipe built them itself. This is a lead, not a confirmed root cause — it has not been reproduced locally.
Affected PRs
#49, #80, #133, #136, #140, #143, #144, #162, #163, #182, #202, #206, #208, #211, #213, #214, #216
Requested handling
- Treat this as one incident; do not retry the check per-PR.
- The likely fix lives partly in
.github/workflows/build.yml, which automated contributors cannot push to (the App token has no Workflows permission). A human or a workflows-capable agent will need to land any workflow-side change.
— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78
Activity
fix(justfile): probe install-vm kiosk readiness from inside the guest
fix(boot): resolve QEMU installer boot deadlock and console readiness…
fix(justfile): probe install-vm kiosk readiness from inside the guest…
refactor: switch OS payload to Flatcar base, drop displaced FSDK comp…