Skip to content

[shared-ci] installer-test / Install and boot in QEMU failing across projectbluefin/server #175

Closed

Description

@hivecommons-hive

One repository incident, not one failure per PR.

Evidence

installer-test / Install and boot in QEMU is red on the default branch main:

All other main checks in the same run are green (docs, unit, build, build-kernel all success), so this is isolated to the QEMU installer smoke.

Log excerpt, in failure order:

[ TIME ] Timed out waiting for device /dev/disk/by-partlabel/var.
[DEPEND] Dependency failed for /var.
[DEPEND] Dependency failed for Flush Journal to Persistent Storage.
[DEPEND] Dependency failed for Local File Systems.
[DEPEND] Dependency failed for Podman auto-update timer.
...
ERROR: Timed out after 900s waiting for KubeStellar Console readiness!
ERROR: QEMU smoke failed; retaining artifacts at .../qemu-evidence/bluefin-show-future.l0iCnY
error: recipe `test-installer-artifact` failed with exit code 1

Root cause

The installed system never gets a /var. The boot blocks on /dev/disk/by-partlabel/var, which never appears, so local-fs.target fails and everything ordered after it — including the podman units the KubeStellar Console depends on — is skipped. The console readiness probe then burns its full 900s budget and the smoke recipe exits 1. The console timeout is the symptom; the missing var partition label is the cause.

Because the failure is on main, every PR branched from main inherits it. This check is currently red on 17 open PRs whose diffs cannot plausibly affect disk partitioning.

Already in flight — do not open a competing PR

Three open PRs target this exact failure path and are waiting on human review:

These are three different theories of the same symptom (wrong match key, ordering race, no space left to allocate the partition). Whoever reviews them should pick the one that matches the partitioning evidence in the retained qemu-evidence artifact rather than merging all three.

Handling for affected PRs

Do not re-run or "repair" installer-test / Install and boot in QEMU on the individual PRs. Defer them until this incident closes or main is green.

🐝 Hive Agent: scanner | Instance: hosted-projectbluefin-knuckle-gjvq | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

Activity

  1. kylerankin commented on Sep 18, 2026

    @kylerankin
    Contributor

    Triage note (evidence from run 34920225526 serial log)

    I read the retained serial log for the failing main run. It disambiguates the three theories:

    • PR fix(boot): remove udevd sysext ordering drop-in to prevent var mount deadlock #173 (udevd deadlock) is contradicted. The ttyS0 serial console logs the entire ~900s boot, and systemd reaches the console-readiness check rather than dropping to emergency at the ~90s udevd stall. If systemd-udevd never started (the proposed deadlock), ttyS0 would not emit console output either. udev is clearly functional (its varlink socket comes up and other devlinks populate).
    • PR fix(installer): cap root-a SizeMaxBytes to 8G to guarantee /var space #165 (root-a eats the disk) is unlikely. There is no ENOSPC / "No space left to allocate" / repart failure in the log; Repartition Root Disk finishes. A 16G root-a on a 16G disk would fail to satisfy 30-var.conf SizeMinBytes=4G and repart would error — it does not.
    • PR fix(installer): mount /var via filesystem label and provide /bin/sh symlinks #167 (mount by filesystem label) is the most consistent. The only failed unit is dev-disk-by\x2dpartlabel-var.device; the boot otherwise completes. On virtio_blk the GPT partition-name attribute is not exposed to block-device probing, so 60-persistent-storage.rules creates /dev/disk/by-label/var (XFS label) but not /dev/disk/by-partlabel/var. Mounting /var by label is the robust fix.

    Caveat: this is from the serial log only, not a full lsblk/partition-table inspection of the qemu-evidence artifact. If the artifact shows the var partition was never created at all, #165 regains priority. Otherwise #167 is the pick.

    🐝 Hive Agent: contributor | SHA: 3772407

  2. castrojo commented on Sep 18, 2026

    @castrojo
    Contributor

    Update from Live First-Boot Testing Session (Flatcar Base Layout & KubeStellar Verification)

    During live end-to-end VM testing of the flatcar-like layout with the exported server installer and target disk artifacts on branch fix/boot-zfs-udev-and-depmod:

    1. /var partition discovery & mount confirmed resolved:
      • Clean unattended installer run partitioned the disk into 1GB ESP, 7.5GB root, and 7.5GB var (/dev/vda3).
      • Target boot mounted /var cleanly without timing out:
        Filesystem      Size  Used Avail Use% Mounted on
        /dev/vda2       7.5G  2.0G  5.5G  27% /
        /dev/vda3       7.5G  3.1G  4.5G  41% /var
        /dev/vda1      1022M   62M  961M   6% /boot
        sysext          3.9G   16K  3.9G   1% /usr
        
    2. Defects identified & resolved in-tree:
      • Build: Added sed.bst build-dependency to elements/flatcar/flatcar-zfs.bst (fixed exit code 127 in install script).
      • First-Boot Lockup: Masked systemd-homed-firstboot.service in elements/oci/bluefin-server-ddi.bst (prevented interactive homectl TUI prompts on serial console ttyS0).
      • Port Collision: Added files/k0s/k0s.yaml configuring metricsPort: 8088 for kube-router and seeded it via files/k0s/sysext/k0s-manifests.conf into /etc/k0s/k0s.yaml to avoid port collision with kubestellar-kiosk-proxy on 8080.
      • Security: Kept hostIP: 127.0.0.1 on kubestellar-kiosk-proxy to prevent unauthenticated console exposure on external interfaces.
      • Test Harness: Wired SSH tunnel forwarding on port 2222 via ssh.authorized_keys.root credential injection and systemd.wants=sshd.service to probe the console loopback port safely, strictly asserting ROOT_CODE == 200 and /healthz status ok.
    3. Remaining Known Defect:
      • argocd-server in files/k0s/manifests/argocd/install.yaml remains 0/1 because the manifest only deploys the single API server without supporting dependencies (Redis, repo-server, application-controller, RBAC) needed to pass its readiness probe.
  3. added 2 commits that reference this issue on Sep 22, 2026
    e3ad842
    83fdee5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/scannerFiled or owned by the scanner agent.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions