Repository navigation
Description
One repository incident, not one failure per PR.
Evidence
installer-test / Install and boot in QEMU is red on the default branch main:
- run: https://github.com/projectbluefin/server/actions/runs/34920225526/job/104235482419
- completed:
2026-09-15T03:14:57Z - conclusion:
failure
All other main checks in the same run are green (docs, unit, build, build-kernel all success), so this is isolated to the QEMU installer smoke.
Log excerpt, in failure order:
[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Podman auto-update timer.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.l0iCnY
error: recipe `test-installer-artifact` failed with exit code 1
Root cause
The installed system never gets a /var. The boot blocks on /dev/disk/by-partlabel/var, which never appears, so local-fs.target fails and everything ordered after it — including the podman units the KubeStellar Console depends on — is skipped. The console readiness probe then burns its full 900s budget and the smoke recipe exits 1. The console timeout is the symptom; the missing var partition label is the cause.
Because the failure is on main, every PR branched from main inherits it. This check is currently red on 17 open PRs whose diffs cannot plausibly affect disk partitioning.
Already in flight — do not open a competing PR
Three open PRs target this exact failure path and are waiting on human review:
- fix(installer): mount /var via filesystem label and provide /bin/sh symlinks #167 — mount
/varvia filesystem label and provide/bin/sh - fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173 — remove the udevd sysext ordering drop-in to prevent var mount delay
- fix(installer): cap root-a SizeMaxBytes to 8G to guarantee /var space #165 — cap
root-aSizeMaxBytesto 8G to guarantee/varspace
These are three different theories of the same symptom (wrong match key, ordering race, no space left to allocate the partition). Whoever reviews them should pick the one that matches the partitioning evidence in the retained qemu-evidence artifact rather than merging all three.
Handling for affected PRs
Do not re-run or "repair" installer-test / Install and boot in QEMU on the individual PRs. Defer them until this incident closes or main is green.
🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown
— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78
Activity
kylerankin commentedon Sep 18, 2026
Triage note (evidence from run 34920225526 serial log)
I read the retained serial log for the failing
mainrun. It disambiguates the three theories:ttyS0serial console logs the entire ~900s boot, and systemd reaches the console-readiness check rather than dropping to emergency at the ~90s udevd stall. Ifsystemd-udevdnever started (the proposed deadlock),ttyS0would not emit console output either. udev is clearly functional (its varlink socket comes up and other devlinks populate).ENOSPC/ "No space left to allocate" / repart failure in the log;Repartition Root Diskfinishes. A 16Groot-aon a 16G disk would fail to satisfy30-var.confSizeMinBytes=4Gand repart would error — it does not.dev-disk-by\x2dpartlabel-var.device; the boot otherwise completes. On virtio_blk the GPT partition-name attribute is not exposed to block-device probing, so60-persistent-storage.rulescreates/dev/disk/by-label/var(XFS label) but not/dev/disk/by-partlabel/var. Mounting/varby label is the robust fix.Caveat: this is from the serial log only, not a full
lsblk/partition-table inspection of theqemu-evidenceartifact. If the artifact shows thevarpartition was never created at all, #165 regains priority. Otherwise #167 is the pick.🐝 Hive Agent:
contributor| SHA:3772407castrojo commentedon Sep 18, 2026
Update from Live First-Boot Testing Session (Flatcar Base Layout & KubeStellar Verification)
During live end-to-end VM testing of the flatcar-like layout with the exported server installer and target disk artifacts on branch
fix/boot-zfs-udev-and-depmod:/dev/vda3)./varcleanly without timing out:sed.bstbuild-dependency toelements/flatcar/flatcar-zfs.bst(fixed exit code 127 in install script).systemd-homed-firstboot.serviceinelements/oci/bluefin-server-ddi.bst(prevented interactive homectl TUI prompts on serial consolettyS0).files/k0s/k0s.yamlconfiguringmetricsPort: 8088forkube-routerand seeded it viafiles/k0s/sysext/k0s-manifests.confinto/etc/k0s/k0s.yamlto avoid port collision withkubestellar-kiosk-proxyon 8080.hostIP: 127.0.0.1onkubestellar-kiosk-proxyto prevent unauthenticated console exposure on external interfaces.ssh.authorized_keys.rootcredential injection andsystemd.wants=sshd.serviceto probe the console loopback port safely, strictly assertingROOT_CODE == 200and/healthz status ok.argocd-serverinfiles/k0s/manifests/argocd/install.yamlremains0/1because the manifest only deploys the single API server without supporting dependencies (Redis, repo-server, application-controller, RBAC) needed to pass its readiness probe.fix(installer): bound root-a partition to guarantee room for var (pro…
refactor: switch OS payload to Flatcar base, drop displaced FSDK comp…