Skip to content

ci: boot-test the offline USB installer end to end - #277

Merged
jeefy merged 5 commits into
projectbluefin:mainfrom
jeefy:ci/installer-boot-test
Sep 29, 2026
Merged

jeefy merged 5 commits into
projectbluefin:mainfrom
jeefy:ci/installer-boot-test

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #268. Only the last 3 commits are new; the first one (fix(installer): boot offline, allow credential overrides, …) is #268.

Closes #265

What

  • scripts/dogfood-installer.sh (new, follows dogfood-diskless.sh conventions: OVMF Secure Boot, serial logs, probe on ttyS1, PASS:/FAIL: lines, env overrides):
    1. Boots bluefin-server-installer_<ver>.raw once so systemd-boot enrolls the dev keys.
    2. Boots it again with no network (-nic none) next to a blank 16G disk (virtio serial bluefin-target). A systemd.unit-dropin.systemd-sysinstall.service SMBIOS credential (installed as 50-credential.conf, so it comes after fix(installer): boot offline, allow credential overrides, quiet repart warnings #268's 10-bluefin-installer.conf) reuses the image drop-in's ExecStart= (read from files/os/systemd/system/systemd-sysinstall.service.d/10-bluefin-installer.conf) and appends --erase=yes --confirm=no --variables=yes /dev/disk/by-id/virtio-bluefin-target. firstboot.locale/keymap/timezone and passwd.hashed-password.root credentials answer systemd-firstboot. -no-reboot turns the installer's final reboot into a QEMU exit.
    3. Boots the target alone, with a user NIC and no bootindex, so the firmware entry from --variables=yes is what boots it. Checks: Secure Boot on, /dev/mapper/usr backed only by bluefin_usr_<ver> / bluefin_usr_verity_<ver> on the target's serial, / is xfs on the target, two _empty slot-B partitions that were absent right after install (so the first boot created them), systemctl --failed empty, no [FAILED] on the console.
    4. Boots the target again with the installer attached and runs the same checks, with boot count 2 on the persistent root. This covers the partlabel vs derived-UUID separation.
    • Overrides: DOGFOOD_STATE, DOGFOOD_TARGET_DISK, DOGFOOD_TARGET_DEV, DOGFOOD_SYSINSTALL_ARGS, DOGFOOD_SYSINSTALL_CRED=0, DOGFOOD_MEM, DOGFOOD_TIMEOUT, OVMF_CODE/OVMF_VARS. There is no HTTP server, so there is no port to override.
  • just dogfood-installer recipe.
  • CI: one step appended at the end of boot-test. The image artifact already includes the installer raw because the build job uploads dist/diskless/ whole.
  • Docs: one line in ddi-installer-build.md (recipe) and one in the ci-tooling.md job table (CI coverage).

Existing dogfood scripts are unchanged. The build.yml change is a single appended step, so #270/#275 should merge cleanly.

Findings the test had to handle (not changed here, may need follow-ups)

  • On the installer, systemd-firstboot prompts first on tty0 (locale, keymap, timezone, root password) before sysinstall runs. /dev/console is a real VT there, so it is not headless. A user sees these prompts and sysinstall copies the answers to the target. The test answers them with the standard firstboot.* credentials.
  • After installing, sysinstall waits for "Press any key to proceed" before --reboot=yes takes effect (any_key_to_proceed()). The test's drop-in sets StandardInput=null, so EOF answers that prompt. The same setting makes any new prompt added by a regression fail at once instead of hanging CI. It also sets StandardError=journal+console so sysinstall's errors reach the journal mirror on ttyS2.

Verification (local, 26.09.2 built with flock bst.lock just export-image)

Positive, flock qemu.lock bash scripts/dogfood-installer.sh dist/diskless (~2 min):

PROBE sysinstall=success 0
PROBE installed-slot-b=0
PROBE-LOG installed ├─vdb2 bluefin_usr_26.09.2        erofs            2G
PROBE-LOG installed └─vdb3 bluefin_usr_verity_26.09.2 DM_verity_hash 128M
==> 3/4 first boot of the target (creates slot B and the xfs root)
PROBE secureboot=enabled (user)
PROBE usr-backing=bluefin_usr_26.09.2@bluefin-target
PROBE usr-backing=bluefin_usr_verity_26.09.2@bluefin-target
PROBE root=xfs@bluefin-target
PROBE slot-b=2
PROBE boots=1
PROBE failed=0
==> 4/4 boot the target with the installer still attached
PROBE usr-backing=bluefin_usr_26.09.2@bluefin-target
PROBE usr-backing=bluefin_usr_verity_26.09.2@bluefin-target
PROBE root=xfs@bluefin-target
PROBE-LOG ├─vdb1  bluefin-installer-usr        erofs          268.6M
PROBE-LOG vdb serial=bluefin-installer
PROBE boots=2
PROBE failed=0
PASS: offline installer installed 26.09.2 onto a blank disk; the target booted twice (with and without the installer attached) from its own bluefin_usr_26.09.2 slot with slot B and the xfs root created on first boot and no failed units

Negative 1: credential points at a missing disk (DOGFOOD_TARGET_DEV=/dev/disk/by-id/virtio-missing-disk). Fails in about a minute:

PROBE sysinstall=exit-code 1
[    6.496708] localhost systemd-sysinstall[381]: Failed to issue io.systemd.Repart.Run() varlink call: No such file or directory
FAIL: systemd-sysinstall did not succeed (logs: …/dist/dogfood-installer-neg)
exit=1

Negative 2: no unattended credential (DOGFOOD_SYSINSTALL_CRED=0 DOGFOOD_TIMEOUT=120). The interactive installer hangs, and the test fails:

FAIL: 2-install: no result within 120s (logs: …/dist/dogfood-installer-neg)

Gates: pytest tests/unit 371 passed (clean detached worktree), bats ok, docs-checks.py passed, just validate rc=0, actionlint clean apart from the existing ubuntu-26.04 label warning, shellcheck clean apart from the SC2054 comma-in-array style warnings that dogfood-diskless.sh also has.

Assisted-by: Claude Opus 5.5 via GitHub Copilot

castrojo and others added 5 commits September 28, 2026 22:14
…t warnings

Found by booting the CI-built installer in QEMU:

- With no network the initrd waited forever on network-online
  (systemd-networkd-wait-online, no limit). Mask it in the installer
  UKI's initrd, as the disk UKI already does.
- Rename the sysinstall drop-in to 10-bluefin-installer.conf so
  systemd.unit-dropin.* credential drop-ins (50-*) can override ExecStart
  for unattended installs.
- Mount the installer ESP with fmask=0133,dmask=0022; vfat otherwise
  reports every file executable and systemd-repart warns per definition.

Assisted-by: claude-opus-5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
scripts/dogfood-installer.sh boots bluefin-server-installer_<ver>.raw with
Secure Boot and no network next to a blank disk, makes the shipped
systemd-sysinstall.service unattended with a
systemd.unit-dropin.systemd-sysinstall.service SMBIOS credential (the image
drop-in's ExecStart= plus the target and --erase=yes --confirm=no
--variables=yes, StandardInput=null so no prompt can hang), answers
systemd-firstboot with firstboot.* credentials, and waits for the reboot.
It then boots the target alone and again with the installer attached and
checks that /usr is backed by the target's bluefin_usr_<ver> slot (never
bluefin-installer-usr), that slot B and the xfs root appear on first boot,
and that no unit failed.

Refs projectbluefin#265

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs scripts/dogfood-installer.sh against dist/diskless, next to the
other dogfood recipes.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Append one boot-test step running scripts/dogfood-installer.sh on the
image artifact, which already carries bluefin-server-installer_<ver>.raw
(the build job uploads dist/diskless/ whole). A regression in the
installer UKI command line, the sysinstall drop-in or
run-bluefin-installer.mount now fails CI.

Closes projectbluefin#265

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs the offline installer test after projectbluefin#270's update and rollback step, and
lists it in ci-tooling.md's boot-test scenarios instead of the old table
row. projectbluefin#268, which this was stacked on, is now in main.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

#268 has merged, so this PR's diff is now only its own commits. Merged current main (0fe4e62): the installer test runs after #270's update/rollback step, and ci-tooling.md lists it in the boot-test scenarios.

@jeefy
jeefy merged commit 6d49f9b into projectbluefin:main Sep 29, 2026
6 checks passed
jeefy added a commit that referenced this pull request Sep 29, 2026
* docs(ci-tooling): stop pinning a literal install-action SHA in the examples

The documented taiki-e/install-action SHA went stale the next time the
workflows repinned. Show the pin format with a placeholder and point at
build.yml for the current value, which the prose already told readers to
check.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(factory-integration): drop the podman-as-sysext claim

No podman sysext exists or is planned; the only container runtime we ship
is containerd, inside the opt-in kubeadm sysext. Say that instead of
implying podman (and other runtimes) ship as optional sysexts.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: one canonical source for the signing keys and the release publish

Hard rule 6. The sysupdate-signing.asc / import-pubring.pgp pair, the
gen-dev-keys behaviour, and the CI secrets were restated in
ddi-installer-build.md, and the release-publishes-as-is fact was restated
in both build and architecture skills. Keep the key inventory and
gen-dev-keys in secure-boot-keys.md, the release-signing key in
systemd-sysupdate-verification.md, and the publish contract in the
ci-tooling.md release job; the build skill keeps only the
rotation-needs-a-new-image-version rationale and points at the rest.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(gap-analysis-distros): trim the sources list to docs roots

Section 2 already links every specific claim inline; section 6 restated
the same URLs. Keep one link per distro documentation root.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: extract the offline USB installer into its own skill

The installer section from #264/#268/#277 lived inside ddi-installer.md.
Move it to docs/skills/usb-installer.md covering the image layout and
boot flow, unattended installs through the
systemd.unit-dropin.systemd-sysinstall.service credential, the firstboot
root-password prompt drop-in, and provisioning the installed disk from
its ESP; ddi-installer.md keeps a one-line pointer and a short artifact
table row. ddi-installer.md's key and release paragraphs also collapse
to pointers (the dedupe from the previous commit).

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: route booty-integration, secure-boot-keys and usb-installer

All three skill files existed (or were just added) without a row in the
lazy-load routing table. List them in docs/skills/index.md and in the
AGENTS.md skill routing table.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(readme): add a Download a release section

Point evaluators at the published artifacts before the build-oriented
quick start: the GitHub Release and OCI artifact, which file covers a
diskless USB boot versus an offline install versus Booty network boot,
and where the verification steps live.

Deferred from #276.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: boot test for the offline USB installer (bluefin-server-installer_<ver>.raw)

2 participants