Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@
persist-credentials: false

- name: Set up just
uses: taiki-e/install-action@3a3daafd3b7f23b8f65d9f5ba2db203fb59b79e2 # v2

Check warning on line 142 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / zizmor

ref-version-mismatch

build.yml:142: action's hash pin has mismatched or missing version comment: tag points to commit 75a7c0f87bd6
with:
tool: just@1.59.0

Expand Down Expand Up @@ -179,7 +179,7 @@
persist-credentials: false

- name: Set up just
uses: taiki-e/install-action@3a3daafd3b7f23b8f65d9f5ba2db203fb59b79e2 # v2

Check warning on line 182 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / zizmor

ref-version-mismatch

build.yml:182: action's hash pin has mismatched or missing version comment: tag points to commit 75a7c0f87bd6
with:
tool: just@1.59.0

Expand Down Expand Up @@ -219,7 +219,7 @@
persist-credentials: false

- name: Set up just
uses: taiki-e/install-action@3a3daafd3b7f23b8f65d9f5ba2db203fb59b79e2 # v2

Check warning on line 222 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / zizmor

ref-version-mismatch

build.yml:222: action's hash pin has mismatched or missing version comment: tag points to commit 75a7c0f87bd6
with:
tool: just@1.59.0

Expand Down Expand Up @@ -294,7 +294,7 @@
persist-credentials: false

- name: Set up just
uses: taiki-e/install-action@3a3daafd3b7f23b8f65d9f5ba2db203fb59b79e2 # v2

Check warning on line 297 in .github/workflows/build.yml

View workflow job for this annotation

GitHub Actions / zizmor

ref-version-mismatch

build.yml:297: action's hash pin has mismatched or missing version comment: tag points to commit 75a7c0f87bd6
with:
tool: just@1.59.0

Expand Down Expand Up @@ -553,7 +553,8 @@
# What each bucket's scenarios read, so it downloads only that:
# `image` the image-set parts for dist/diskless (dogfood-diskless.sh
# and dogfood-install.sh boot netboot, dogfood-installer.sh the
# installer); outside releases `next` the image-next parts for
# installer, or netboot with DOGFOOD_INSTALL=diskless-console);
# outside releases `next` the image-next parts for
# dist/diskless-next, `broken` dist/diskless-broken and `sigs`
# dist/ignition-sigs.
include:
Expand All @@ -568,7 +569,7 @@
broken: true
sigs: false
- scenario: installer
image: image-set-{core,installer}
image: image-set-{core,netboot,installer}
next: image-next-*
broken: false
sigs: false
Expand Down
40 changes: 20 additions & 20 deletions docs/skills/ci-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -300,27 +300,24 @@ those of the org config `projectbluefin/.project/prow.yaml`; the hive
Fedora's `edk2-ovmf` is pinned by Koji URL + SHA-256 in `scripts/ci/install-qemu-ovmf.sh`:
Ubuntu 26.04's OVMF 2025.11 rejects PK enrollment and boots with Secure Boot off
([details](ddi-installer-build.md)). Bump by hand: Renovate cannot update the SHA-256.
- diskless netboot, no failed units; `tests/fixtures/nfs/netdb.probe`
must resolve `tcp` and `sunrpc`, reach the local portmapper (`rpcinfo`)
and get an NFSv3 mount past the protocol lookup (`DOGFOOD_EXPECT`);
- `DOGFOOD_TAMPER=raw` and `=sums`: the pull must refuse a corrupted DDI
(checksum) and that DDI with `SHA256SUMS` re-hashed to match it
(signature), and nothing may boot;
- Ignition from the `ignition.config` credential and from
`bluefin-node.ign` over UEFI HTTP boot, both with
`tests/fixtures/ignition/apply-marker.ign` (`DOGFOOD_EXPECT`); outside
releases also signed with the Ignition config key (applied), with the
image signing key or over other bytes (refused), and unsigned on the
`1.<run>.1` set, built with `ignition_allow_unsigned` off (refused;
see booty-integration.md);
- diskless netboot, no failed units; `tests/fixtures/nfs/netdb.probe` must resolve `tcp` and
`sunrpc`, reach the local portmapper (`rpcinfo`) and get an NFSv3 mount past the protocol
lookup (`DOGFOOD_EXPECT`);
- `DOGFOOD_TAMPER=raw` and `=sums`: the pull must refuse a corrupted DDI (checksum) and that
DDI with `SHA256SUMS` re-hashed to match it (signature), and nothing may boot;
- `DOGFOOD_FIRMWARE=foreign-keys`: firmware with vendor keys, not the Bluefin db
(`OVMF_VARS_FOREIGN`), refuses the netboot ESP's loader before systemd-boot runs (#450);
- Ignition from the `ignition.config` credential and from `bluefin-node.ign` over UEFI HTTP
boot, both with `tests/fixtures/ignition/apply-marker.ign` (`DOGFOOD_EXPECT`); outside
releases also signed with the Ignition config key (applied), with the image signing key or
over other bytes (refused), and unsigned on the `1.<run>.1` set, built with
`ignition_allow_unsigned` off (refused; see booty-integration.md);
- releases: diskless boot, `systemd-sysinstall` to disk, boot the disk;
- every other build (the nightly build of `main`, `full-build` and FSDK
pull requests, dispatches): the same, then `systemd-sysupdate` A->B to
`1.<run>.1` and a boot-counted rollback from a corrupted `1.<run>.2`,
with the ZFS and NVIDIA sysexts merged (`DOGFOOD_SYSEXT=zfs,nvidia`;
versions: [ddi-installer-build.md](ddi-installer-build.md)). Releases
skip it: their extra sets would be release-signed versions nobody
publishes.
- every other build (the nightly build of `main`, `full-build` and FSDK pull requests,
dispatches): the same, then `systemd-sysupdate` A->B to `1.<run>.1` and a boot-counted
rollback from a corrupted `1.<run>.2`, with the ZFS and NVIDIA sysexts merged
(`DOGFOOD_SYSEXT=zfs,nvidia`; versions: [ddi-installer-build.md](ddi-installer-build.md)).
Releases skip it: their extra sets would be release-signed versions nobody publishes.
- `scripts/dogfood-installer.sh`: offline unattended install onto a blank disk, then boot
with and without the stick. Outside releases, update to `1.<run>.1`: unreachable and
foreign-signed sources must fail `systemd-sysupdate.service` and appear on the login banner;
Expand All @@ -335,6 +332,9 @@ those of the org config `projectbluefin/.project/prow.yaml`; the hive
`boot-test-logs-installer` retains serial journals, screen snapshots and console proof, plus
`installer-artifact-identity.txt`: the source commit and pre-boot SHA-256 of the installer,
manifest and signature. Partial logs survive regressions; disks and credentials stay excluded.
`DOGFOOD_INSTALL=diskless-console` (#450) logs in at a diskless boot's tty1 with a per-run root
password (hashed, masked), types the "From a diskless node" commands of [ddi-installer.md](ddi-installer.md),
answers `systemd-sysinstall`'s disk, keep/erase, firmware and `yes` questions, boots the disk.
5. **Version Derivation:** The release version is set per build with
`just set-version`: `YY.MM.<run>` for releases, `0.<run>` for every other
build so it can never sort above a release.
Expand Down
14 changes: 12 additions & 2 deletions docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,9 @@ path:
`BOOTX64.EFI` alone.
- **Microsoft-only firmware.** Until shim is Microsoft-signed it boots only
where the Bluefin db is enrolled, and the USB installer's Secure Boot
check still refuses firmware that lacks the Bluefin db.
check still refuses firmware that lacks the Bluefin db. The boot test
shows firmware that holds only other keys refusing the netboot ESP's
loader before anything of Bluefin runs (`DOGFOOD_FIRMWARE=foreign-keys`).
- **UEFI HTTP boot** loads the netboot UKI directly, without shim. shim
fetches `DEFAULT_LOADER` relative to its own URL when HTTP-booted; serving
a shim chain that way is Booty-side work.
Expand Down Expand Up @@ -193,9 +195,17 @@ in-RAM DDI partitions (`/dev/disk/by-loop-ref/rootdisk.raw-part1` and `part2`):
systemctl start run-bluefin-boot.mount
kernel="$(ls /run/bluefin/boot/EFI/Linux/bluefin-server-[0-9]*.efi)"
systemd-sysinstall --kernel="${kernel}" \
--definitions=/run/bluefin/boot/bluefin/repart.d /dev/sdX
--definitions=/run/bluefin/boot/bluefin/repart.d
```

`systemd-sysinstall` lists the disks and asks for one (its number or name),
whether to keep what is on it or erase it, whether to register the install in
the firmware's boot entries, and, after a summary, for `yes`. Name the disk at
the end of the command (e.g. `/dev/sdX`) to skip the first question. The boot
test logs in at a diskless node's console, types this block as written and
answers the questions (`DOGFOOD_INSTALL=diskless-console` of
`scripts/dogfood-installer.sh`).

On older image builds whose `repart.d` used `CopyBlocks=auto`, active system
extensions (`systemd-sysext`, e.g. ZFS or k0s) turn `/usr` into an overlayfs,
causing `systemd-repart` to fail with "Failed to resolve automatic CopyBlocks=
Expand Down
10 changes: 9 additions & 1 deletion scripts/ci/boot-test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ bucket_diskless() {
-- scripts/dogfood-diskless.sh dist/diskless --check
scenario "Refuse an unsigned manifest" DOGFOOD_TAMPER=sums \
-- scripts/dogfood-diskless.sh dist/diskless --check
# Firmware holding only someone else's Secure Boot keys refuses the
# boot loader before Bluefin runs (#450).
scenario "Refuse to boot on firmware with foreign Secure Boot keys" DOGFOOD_FIRMWARE=foreign-keys \
-- scripts/dogfood-diskless.sh dist/diskless --check
scenario "Ignition config from a credential" \
DOGFOOD_IGNITION=tests/fixtures/ignition/apply-marker.ign "${ignition_probe[@]}" \
-- scripts/dogfood-diskless.sh dist/diskless --check
Expand Down Expand Up @@ -100,7 +104,8 @@ bucket_install() {
# The offline USB installer onto a blank disk: outside releases also the
# out-of-the-box update contract of a stick install (#310); the supervised
# console path under concurrent noise, including a real prewrite failure
# (#454); and the refusal without Secure Boot (#309).
# (#454); the refusal without Secure Boot (#309); and the documented
# install from a diskless node, typed at its console (#450).
bucket_installer() {
if [ "${release}" = true ]; then
scenario "Install from the offline USB installer and boot it" \
Expand All @@ -119,6 +124,9 @@ bucket_installer() {
scenario "Refuse to install from the USB installer without Secure Boot" \
DOGFOOD_SECURE_BOOT=off DOGFOOD_STATE=dist/dogfood-installer-secure-boot-off \
-- scripts/dogfood-installer.sh dist/diskless
scenario "Install from a diskless boot at the console" \
DOGFOOD_INSTALL=diskless-console DOGFOOD_STATE=dist/dogfood-installer-diskless-console \
-- scripts/dogfood-installer.sh dist/diskless
}

# The offline USB installer onto disks that are not empty (#359): over a
Expand Down
3 changes: 2 additions & 1 deletion scripts/ci/dogfood-preflight.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ host_tools=(
sfdisk blkid mkswap # (fdisk, util-linux) the foreign-gpt target disk
mkfs.vfat mkfs.ext4 # (dosfstools, e2fsprogs) the same
gpg # (gnupg) the foreign-signed update source
openssl # (openssl) hashes the diskless console's root password
sha256sum base64 dd od truncate realpath mktemp awk
)
# Matched, but never run on the runner.
Expand Down Expand Up @@ -68,7 +69,7 @@ for tool in "${host_tools[@]}"; do
command -v "${tool}" >/dev/null || error "${tool}: command not found (add its package to scripts/ci/install-qemu-ovmf.sh)"
done

for var in OVMF_CODE OVMF_CODE_INSECURE OVMF_VARS; do
for var in OVMF_CODE OVMF_CODE_INSECURE OVMF_VARS OVMF_VARS_FOREIGN; do
[ -s "${!var:-}" ] || error "${var} is not set to a firmware file (scripts/ci/install-qemu-ovmf.sh sets it)"
done

Expand Down
8 changes: 5 additions & 3 deletions scripts/ci/install-qemu-ovmf.sh
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
#!/usr/bin/env bash
# Install what the boot test runs on an Ubuntu runner: QEMU, the disk tools
# the dogfood scripts call, and Fedora's Secure Boot OVMF, then export
# OVMF_CODE, OVMF_CODE_INSECURE and OVMF_VARS to the job's later steps.
# OVMF_CODE, OVMF_CODE_INSECURE, OVMF_VARS and OVMF_VARS_FOREIGN (Fedora's
# store with its vendor keys enrolled) to the job's later steps.
# boot-test and boot-test-preflight both run this, so they cannot drift.
#
# bash scripts/ci/install-qemu-ovmf.sh
Expand All @@ -18,7 +19,7 @@ OVMF_RPM_SHA256=8b157ab2aa0cccccf021f43011308349147c76ae59550f52239ea50678dd0a5f

sudo apt-get update
sudo apt-get install -y --no-install-recommends qemu-system-x86 libarchive-tools \
fdisk dosfstools e2fsprogs mtools
fdisk dosfstools e2fsprogs mtools openssl
sudo chmod 666 /dev/kvm

rpm="${RUNNER_TEMP:?}/edk2-ovmf.rpm"
Expand All @@ -28,9 +29,10 @@ echo "${OVMF_RPM_SHA256} ${rpm}" | sha256sum -c -
mkdir -p "${RUNNER_TEMP}/ovmf"
bsdtar -xf "${rpm}" -C "${RUNNER_TEMP}/ovmf" \
./usr/share/edk2/ovmf/OVMF_CODE.secboot.fd ./usr/share/edk2/ovmf/OVMF_VARS.fd \
./usr/share/edk2/ovmf/OVMF_CODE.fd
./usr/share/edk2/ovmf/OVMF_CODE.fd ./usr/share/edk2/ovmf/OVMF_VARS.secboot.fd
{
echo "OVMF_CODE=${ovmf}/OVMF_CODE.secboot.fd"
echo "OVMF_CODE_INSECURE=${ovmf}/OVMF_CODE.fd"
echo "OVMF_VARS=${ovmf}/OVMF_VARS.fd"
echo "OVMF_VARS_FOREIGN=${ovmf}/OVMF_VARS.secboot.fd"
} >> "${GITHUB_ENV:?}"
31 changes: 30 additions & 1 deletion scripts/dogfood-diskless.sh
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@
# SHA256SUMS re-hashed to match it but no longer matching
# SHA256SUMS.gpg (sums); --check then passes only if the
# initrd's pull refuses it for that reason
# DOGFOOD_FIRMWARE=foreign-keys (--check, netboot, no DOGFOOD_VARS) start from a
# variable store already in user mode with someone else's
# Secure Boot keys (the distribution's pre-enrolled vendor
# keys, OVMF_VARS_FOREIGN) instead of a blank one: --check
# passes only if the firmware itself refuses the netboot
# ESP's boot loader, so neither systemd-boot nor the kernel
# runs
set -euo pipefail

dir="$(realpath "${1:?usage: $0 <artifact dir> [--check]}")"
Expand All @@ -63,6 +70,18 @@ image="bluefin-server_${ver}.raw"
. "$(dirname "${BASH_SOURCE[0]}")/dogfood-firmware.sh"
code="$(ovmf_code)" || exit 1
vars_tmpl="$(ovmf_vars)" || exit 1
firmware="${DOGFOOD_FIRMWARE:-}"
case "${firmware}" in
'') ;;
foreign-keys)
if [ "${mode}" != --check ] || [ "${DOGFOOD_BOOT:-netboot}" != netboot ] || [ -n "${DOGFOOD_VARS:-}" ] \
|| [ -n "${DOGFOOD_REFUSAL:-}${DOGFOOD_TAMPER:-}" ]; then
echo "ERROR: DOGFOOD_FIRMWARE=foreign-keys is a --check of a netboot boot, without DOGFOOD_VARS, DOGFOOD_REFUSAL or DOGFOOD_TAMPER" >&2
exit 1
fi
vars_tmpl="$(ovmf_vars_foreign)" || exit 1 ;;
*) echo "ERROR: DOGFOOD_FIRMWARE must be foreign-keys" >&2; exit 1 ;;
esac

work="$(mktemp -d /tmp/bluefin-dogfood.XXXXXX)"
probe_log="${DOGFOOD_PROBE_LOG:-${work}/probe.log}"
Expand All @@ -89,6 +108,11 @@ for f in "${dir}"/*; do ln -s "${f}" "${srv}/"; done
if [ -n "${DOGFOOD_SERVE_EXTRA:-}" ]; then for f in "${DOGFOOD_SERVE_EXTRA}"/*; do ln -sf "$(realpath "${f}")" "${srv}/"; done; fi
image_re="${image//./\\.}"
refusal_re="${DOGFOOD_REFUSAL:-}"
# OVMF reports each boot option it cannot load on the firmware console; an
# image Secure Boot rejects fails to load with Access Denied (or, under a
# deferring policy, Security Violation).
[ "${firmware}" != foreign-keys ] \
|| refusal_re='BdsDxe: failed to load Boot[0-9A-F]{4} .*: (Access Denied|Security Violation)'
case "${DOGFOOD_TAMPER:-}" in
raw|sums)
rm "${srv}/${image}"; cp "${dir}/${image}" "${srv}/${image}"
Expand Down Expand Up @@ -180,6 +204,7 @@ if [ -n "${DOGFOOD_IGNITION:-}" ]; then
fi
tamper="${DOGFOOD_TAMPER:-}"
refusal="${tamper:+tamper=${tamper}}"
[ -z "${firmware}" ] || refusal="firmware=${firmware}"
[ -n "${refusal_re}" ] && refusal="${refusal:-refusal}"
if [ -n "${tamper}" ]; then
# The initrd's console shows only that the download unit failed; copy the
Expand Down Expand Up @@ -317,7 +342,11 @@ if [ -n "${refusal}" ]; then
elif [ -n "${DOGFOOD_NODE_IGN:-}" ] && { ! served bluefin-node.ign || ! served bluefin-node.ign.gpg '[0-9]+'; }; then
echo "FAIL: ${refusal}: refused before fetching bluefin-node.ign and requesting its .gpg" >&2
status=1
elif ! grep -aq 'Secure boot enabled' "${dir}/dogfood-serial.log"; then
elif [ -n "${firmware}" ] && grep -aqE 'successfully enrolled|Secure boot enabled' "${dir}/dogfood-serial.log"; then
# systemd-boot enrolls the Bluefin keys only into a store in setup mode.
echo "FAIL: ${refusal}: refused, but systemd-boot or the kernel ran first" >&2
status=1
elif [ -z "${firmware}" ] && ! grep -aq 'Secure boot enabled' "${dir}/dogfood-serial.log"; then
echo "FAIL: ${refusal}: refused, but the kernel did not run with Secure Boot enabled" >&2
status=1
else
Expand Down
11 changes: 11 additions & 0 deletions scripts/dogfood-firmware.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
# ovmf_code Secure Boot firmware code (OVMF_CODE)
# ovmf_code_insecure firmware without Secure Boot (OVMF_CODE_INSECURE)
# ovmf_vars blank variable store (OVMF_VARS)
# ovmf_vars_foreign variable store in user mode with the distribution's
# vendor Secure Boot keys, not ours (OVMF_VARS_FOREIGN)

first_existing() { for f in "$@"; do [ -f "$f" ] && { echo "$f"; return 0; }; done; return 1; }

Expand Down Expand Up @@ -40,3 +42,12 @@ ovmf_vars() {
/usr/share/edk2/x64/OVMF_VARS.4m.fd \
|| { echo "ERROR: no blank OVMF_VARS found (set OVMF_VARS)" >&2; return 1; }
}

ovmf_vars_foreign() {
[ -n "${OVMF_VARS_FOREIGN:-}" ] && { echo "${OVMF_VARS_FOREIGN}"; return 0; }
first_existing \
/usr/share/edk2/ovmf/OVMF_VARS.secboot.fd \
/usr/share/OVMF/OVMF_VARS_4M.ms.fd \
/usr/share/OVMF/OVMF_VARS.ms.fd \
|| { echo "ERROR: no OVMF_VARS with enrolled vendor keys found (set OVMF_VARS_FOREIGN)" >&2; return 1; }
}
Loading
Loading