Skip to content

fix(installer): bare-metal USB install consolidation, preset ordering, and OVMF boot test - #243

Merged
castrojo merged 22 commits into
mainfrom
fix/bare-metal-usb-consolidation
Sep 25, 2026
Merged

castrojo merged 22 commits into
mainfrom
fix/bare-metal-usb-consolidation

Conversation

@castrojo

Copy link
Copy Markdown
Contributor

Consolidates the active remote work branches targeting reliable bare-metal USB installation on the new Flatcar base:

Tested: full unit suite green (344 passed, 1 xfailed), docs-checks.py, actionlint, just validate. Ponytail review applied.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

castrojo and others added 22 commits September 22, 2026 00:28
…B OVMF testing

Attach installer media via emulated xHCI USB drive in QEMU and boot via
OVMF firmware. Verify target GPT partition table using sfdisk --json and
jq for bluefin-server-root-a and var partitions. Add contract unit tests.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tindex=2 and enforce timeout

Enforce INSTALLER_TIMEOUT via timeout utility in test-installer-artifact
stage 1. Define target disk using -drive if=none and attach device
virtio-blk-pci with bootindex=2 so boot priorities are valid for QEMU.
Update contract tests.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…service output to serial console, and add edk2-i386-vars pairing

- In bluefin-sysinstall, wait up to 30s using an absolute SECONDS deadline with periodic udevadm trigger and settle for bluefin-installer-data to be enumerated on slower USB controllers before failing.
- Mirror systemd-sysinstall.service output to journal+console so installer execution progress and error diagnostics are captured on serial console.
- Deduplicate test-installer-boot-usb by having it delegate directly to test-installer-artifact.
- Add Homebrew and system edk2-i386-vars.fd to OVMF vars candidates for x86_64 firmware code pairing with fatal fallback.
- Strictly scope test-installer-boot-usb unit contract assertions and assert exact dynamic settle timeout.

Assisted-by: github-copilot/gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three review findings from #228:

- test-installer-artifact still copied dist/bluefin-server-pxe-vmlinuz-*
  and -initrd-*.cpio.gz into $WORKDIR, but the recipe no longer passes
  -kernel/-initrd, so nothing consumed them. Under set -euo pipefail the
  copies made the test fail on hosts that have the installer image but no
  PXE artifacts. export-installer already asserts the PXE pair exists, so
  that coverage is not lost.

- Replacing the truncate-based blank variable store with exit 1 meant
  hosts shipping only OVMF_CODE could no longer run the test or install-vm.
  Restore the fallback in both recipes and log a warning, keeping the
  edk2-i386-vars.fd candidates this PR added.

- docs/skills/ddi-installer-build.md did not list test-installer-boot-usb
  and still implied the old direct-kernel behaviour of
  test-installer-artifact. Document the OVMF/xHCI USB boot path, the alias,
  and the OVMF_VARS fallback.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…heck

Four review findings from #228:

- StandardOutput/StandardError=journal+console routes installer output to
  /dev/console, which follows the last console= argument (ttyS0), so an
  interactive bare-metal install printed to serial instead of the attached
  display. TTYPath= is only honoured for Standard*=tty, so it did not
  counteract this. The wrapper now redirects its own output to /dev/tty0
  when 'unattended' is absent from the kernel command line and a local VT
  is writable, which restores the display for interactive installs while
  unattended/automated runs keep the serial capture that the drop-in was
  added for. Headless serial-only interactive installs fall through to
  journal+console unchanged.

- The target-disk partition check piped sfdisk into jq with 2>/dev/null and
  || true, so a missing or broken tool was reported as "Target disk did not
  receive expected partition layout". sfdisk and jq are now checked up front
  and their failures reported separately from a genuinely unpartitioned disk.
  jq was an undeclared host dependency of this recipe; it is now named in the
  error path and in docs/skills/ddi-installer-build.md.

- The comment added in the previous commit credited export-installer with
  asserting the PXE pair was exported; those assertions are in export-pxe.

- Restore the blank line before
  test_interactive_installer_uses_local_virtual_console (PEP8 E302) and
  separate the test-installer-boot-usb recipe from the install-vm comment
  block.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…guard

test-installer-artifact now takes an INSTALLER_BOOT_MODE of usb (default,
firmware boots the raw image off an emulated xHCI USB drive) or pxe (QEMU boots
the exported vmlinuz/initrd pair with -kernel/-initrd/-append). Everything after
the install is shared. test-installer-boot-usb and test-installer-boot-pxe are
the named entry points, so the direct-kernel path the previous revision dropped
is exercised again rather than merely existence-checked by export-pxe.

Document next to the -smbios flag that systemd-stub ignores
io.systemd.stub.kernel-cmdline-extra under Secure Boot, so a secure-boot OVMF
build boots interactive and hangs until the install timeout, and point at pxe
mode as the way to test unattended installs on such firmware.

Restrict the interactive /dev/tty0 redirect to boots where the kernel actually
registered tty0 as a console. A writable VT device exists on serial-only bare
metal too, so the -w test alone sent the whole interactive install to a display
nobody was watching.

Correct the ci-tooling sentence that still claimed the uploaded PXE artifacts
feed test-installer-artifact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The kubestellar-console-github-oauth Secret committed a usable jwt-secret
(default-jwt-secret-123456789012345678901234). JWT_SECRET signs console
session tokens, so a value published in git lets anyone forge valid
sessions on every default install.

Mirror the kubeflex-postgres pattern (#98): the committed 01- manifest now
creates only the namespace, and a first-boot generator writes
02-kubestellar-console-secret.yaml with a random 256-bit jwt-secret and
empty OAuth credentials (consumed with optional: true). Idempotent, 0600,
runs from k0s-first-boot.service before k0scontroller starts.

Existing installs rotate on upgrade: tmpfiles C+ replaces the stale 01-
copy (dropping the published Secret) and the generator creates the 02-
file fresh, invalidating any tokens forged with the public value.

Closes #72

Assisted-by: Claude Fable 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
… 'disable *'

Flatcar's /usr ships /usr/lib/systemd/system-preset/99-default.preset
containing 'disable *'. systemd reads preset files in lexicographic
filename order and the first matching line wins, so every Bluefin
preset named zz-enable-*.preset sorted after it and was ignored at
first-boot preset-all. k0s-first-boot.service was therefore never
enabled: the QEMU smoke test sat at 'kiosk-ready: waiting k0s=inactive'
for the full 900s window on every run since the Flatcar payload switch.

Rename the presets to 20-bluefin-*.preset so they sort before
99-default.preset, and add a unit test asserting no staged preset can
regress behind the Flatcar catch-all again.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
…pod states in smoke probe

- Update the two unit tests that still referenced the removed
  zz-enable-*.preset filenames (the rename commit shipped without them).
- Add test_presets_sort_before_flatcar_disable_all so no future preset
  can sort behind Flatcar's 99-default.preset 'disable *' catch-all.
- Teach the installer smoke readiness unit to print the pod table and
  the last Warning events alongside the existing counters, so a run that
  reaches k0s=active but never gets a pod to Running names the blocker
  in the serial log instead of only counting it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>
The DDI seeds a well-known root password ('bluefin', hash public in this
repo) into every shipped image with shadow last-change 19700, so the
credential never expires and is identical fleet-wide. PR #169 already
restored the sshd drop-in hardening (PermitRootLogin prohibit-password,
PasswordAuthentication no), which closes the remote root path; the
remaining gap from #168 is that console bring-up never forces rotation.

Set the shadow last-change field to 0 (the `chage -d 0 root` equivalent):
every login as root — console included — is forced onto a per-install
password before any shell is granted. Update the console banner to
disclose the forced change, and add a unit contract locking in both the
rotation and the SSH no-password-auth policy as a regression guard.

The long-term removal of the baked password (core operator account,
systemd-credentials key provisioning) is tracked by #80; this is the
minimal interim hardening requested by the issue.

Fixes: #168

Hive-Run: #168
Hive-Plan: Security remediation — interim hardening of seeded root credential (issue #168, recommendation item 2)
Hive-Spec: docs/superpowers/specs/2026-09-09-core-login-design.md#interim-shadow-expiry
Assisted-by: GLM (z-ai/glm-5.3-flash) via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mendezr <mendezr@users.noreply.github.com>
…osk waiter

The preset rename in this PR gets k0scontroller active, but the smoke gate
still times out: every pod sits Pending/ContainerCreating for the full 900s
window (kube-proxy ContainerCreating, kube-router Init:0/2, node tainted
not-ready). The current 30s debug line only shows Warning events, and the
FailedScheduling flood from ten Pending pods pushes the informative events
(image pulls, sandbox creation, CNI) out of the tail -n 5 window.

Extend the in-guest waiter's periodic dump so the serial log carries enough
to diagnose the next layer:
- node status line (kiosk-node) — shows Ready/NotReady and the CRI version
- all event kinds minus the FailedScheduling flood (kiosk-events)
- error/fail/taint lines from the k0scontroller journal (kiosk-k0s), where
  the embedded kubelet and containerd log

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>
…n create pod sandboxes

The preset rename got k0s and kubelet running, but every pod sandbox then
failed with 'open /etc/resolv.conf: no such file or directory' and the
console readiness gate timed out (run 36014129412). Flatcar ships the
/etc/resolv.conf symlink inside the image /etc and strips /etc-populating
lines from /usr/lib/tmpfiles.d at image build; this DDI imports only
Flatcar's /usr and first-boots with an empty /etc, so the symlink never
exists.

Recreate Flatcar's stock symlink (/etc/resolv.conf ->
../run/systemd/resolve/resolv.conf, the full resolver list, matching
Flatcar >= 3510.3.0) with a tmpfiles.d rule installed from a new
os-resolv-conf.bst element, and guard it with a unit test.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check <sec-check@hive.kubestellar.io>
…eflight hardening

- Set GrowFileSystem=no on bluefin-server-root-a in 20-root-a.conf to prevent
  interrupted first-boot metadata corruption on the immutable root partition.
- Disable experimental features (reflink, rmapbt, parent, exchange) and widen
  headroom in elements/oci/bluefin-server-ddi.bst mkfs.xfs to prevent ENOSPC
  metadata reservation failure during log replay.
- Remove Flatcar oem.mount, dev-disk-by-label-OEM.device, and fsck drop-ins in
  elements/flatcar/flatcar-usr.bst to prevent 90s device timeout on first boot.
- Default INSTALLER_PART="" before preflight in elements/oci/bluefin-server-installer.bst
  so PXE boots under `set -u` do not fail on unbound variable.
- Update contract tests in test_flatcar_usr.py and test_repart_layout.py.

Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…red selection

- When exactly one non-installer target disk is present, pass it directly
  to systemd-sysinstall so the user is not prompted for the device path.
- When multiple target disks are present, print a hint that systemd-sysinstall's
  native interactive prompt accepts single-digit menu numbers (1, 2, ...).

Assisted-by: claude-3-5-sonnet via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid copying BuildBarn CAS-hardlinked certificate files onto themselves when composing the OS DDI. Align installer documentation with the existing no-grow root recipe.

Assisted-by: github-copilot/gpt-6-sol via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants