From 6a9c914ced6761803516224d7a652923ec878f5a Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:00 -0500 Subject: [PATCH 01/17] refactor(initrd): share the network boot origin lookup as bluefin-boot-origin Move origin_url() out of bluefin-ignition-credentials into /usr/libexec/bluefin-boot-origin, so the booted OS can find its boot server with the same logic instead of a copy. Order: the StubDeviceURL EFI variable (UEFI HTTP boot), the explicit http(s) rd.systemd.pull= source, then the import.pull credential when the calling unit imports it. Ignition does not import import.pull, so its behaviour is unchanged. Tests drive the helper against fake efivar, cmdline and credential files. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- elements/bluefin-server/boot-origin.bst | 10 +++ .../initrd/initrd-ignition-stack.bst | 2 + .../usr/libexec/bluefin-boot-origin | 61 ++++++++++++++ .../usr/libexec/bluefin-ignition-credentials | 38 ++------- tests/unit/test_boot_origin.py | 82 +++++++++++++++++++ tests/unit/test_signed_lockstep.py | 6 +- 6 files changed, 168 insertions(+), 31 deletions(-) create mode 100644 elements/bluefin-server/boot-origin.bst create mode 100755 files/boot-origin/usr/libexec/bluefin-boot-origin create mode 100644 tests/unit/test_boot_origin.py diff --git a/elements/bluefin-server/boot-origin.bst b/elements/bluefin-server/boot-origin.bst new file mode 100644 index 00000000..1d90987e --- /dev/null +++ b/elements/bluefin-server/boot-origin.bst @@ -0,0 +1,10 @@ +kind: import +description: | + /usr/libexec/bluefin-boot-origin: prints the URL a network-booted node was + booted from (UEFI HTTP boot URL, the explicit rd.systemd.pull= source, or + the import.pull credential), for bluefin-ignition-credentials in the initrd + (initrd-ignition-stack.bst). + +sources: +- kind: local + path: files/boot-origin diff --git a/elements/bluefin-server/initrd/initrd-ignition-stack.bst b/elements/bluefin-server/initrd/initrd-ignition-stack.bst index f1d6cc91..36c12ae6 100644 --- a/elements/bluefin-server/initrd/initrd-ignition-stack.bst +++ b/elements/bluefin-server/initrd/initrd-ignition-stack.bst @@ -16,6 +16,8 @@ depends: - ignition/ignition.bst - ignition/gptfdisk.bst - bluefin-server/initrd/initrd-ignition.bst +# bluefin-boot-origin: the HTTP boot origin, for bluefin-node.ign next to the UKI +- bluefin-server/boot-origin.bst # bash, coreutils - freedesktop-sdk.bst:public-stacks/runtime-gnu.bst diff --git a/files/boot-origin/usr/libexec/bluefin-boot-origin b/files/boot-origin/usr/libexec/bluefin-boot-origin new file mode 100755 index 00000000..7215b3a7 --- /dev/null +++ b/files/boot-origin/usr/libexec/bluefin-boot-origin @@ -0,0 +1,61 @@ +#!/usr/bin/bash +# bluefin-boot-origin — print the URL a network-booted node was booted from. +# +# Sources, first match wins: +# 1. the UEFI HTTP boot URL that systemd-stub records in the StubDeviceURL +# EFI variable (what the netboot UKI's `rd.systemd.pull=...bootorigin:` +# resolves against); +# 2. the explicit http(s) source of `rd.systemd.pull=` on the kernel command +# line (a node chainloaded by iPXE, e.g. from Booty); +# 3. the http(s) source of the `import.pull` system credential, when the +# calling unit imports it (ImportCredential=import.pull): a netboot ESP +# with /loader/credentials, or the dogfood VM's SMBIOS credential. +# +# The URL names the file that was booted or pulled; callers use its directory. +# Exits 1, printing nothing, when the node was not network booted. +# +# Shared by bluefin-ignition-credentials (initrd) and +# bluefin-diskless-update-check (booted OS). Bash builtins and coreutils only. +# BLUEFIN_STUB_URL_VAR and BLUEFIN_CMDLINE override the input paths for tests. + +set -euo pipefail +export LC_ALL=C + +stub_var="${BLUEFIN_STUB_URL_VAR:-/sys/firmware/efi/efivars/StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f}" +cmdline="${BLUEFIN_CMDLINE:-/proc/cmdline}" + +# rd.systemd.pull= and import.pull share one syntax: ::. +pull_url() { + case "$1" in + *:http://*|*:https://*) + printf '%s\n' "http${1#*:http}" + return 0 ;; + esac + return 1 +} + +if [ -r "${stub_var}" ]; then + # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. + url="$(tail -c +5 "${stub_var}" | tr -d '\000')" + if [ -n "${url}" ]; then + printf '%s\n' "${url}" + exit 0 + fi +fi + +if [ -r "${cmdline}" ]; then + read -r -a args < "${cmdline}" || true + for arg in "${args[@]}"; do + case "${arg}" in + rd.systemd.pull=*) pull_url "${arg#rd.systemd.pull=}" && exit 0 ;; + esac + done +fi + +cred="${CREDENTIALS_DIRECTORY:-/nonexistent}/import.pull" +if [ -r "${cred}" ]; then + read -r line < "${cred}" || true + pull_url "${line:-}" && exit 0 +fi + +exit 1 diff --git a/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials b/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials index 9ddaecb3..42cbb6ba 100755 --- a/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials +++ b/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials @@ -16,14 +16,13 @@ # # If both are set, ignition.config wins and ignition.config.url is ignored. # -# Without either credential, a node that was UEFI HTTP booted (systemd-stub -# records the boot URL in the StubDeviceURL EFI variable) looks for -# bluefin-node.ign next to its UKI, the same way the initrd derives the /usr -# image URL. A node chainloaded by iPXE has no StubDeviceURL; its boot server -# (Booty) passes rd.systemd.pull with an explicit http(s) URL instead, and the -# config is looked up in that URL's directory. If the boot server has one -# (HTTP HEAD succeeds), it is used through config.replace; otherwise Ignition -# is skipped. +# Without either credential, a network-booted node looks for bluefin-node.ign +# in the directory it booted from, as reported by /usr/libexec/bluefin-boot-origin: +# the UEFI HTTP boot URL (StubDeviceURL EFI variable), the same way the initrd +# derives the /usr image URL, or, for a node chainloaded by iPXE, the explicit +# http(s) URL its boot server (Booty) passes in rd.systemd.pull. If the boot +# server has one (HTTP HEAD succeeds), it is used through config.replace; +# otherwise Ignition is skipped. # # Output: /run/ignition/user.ign (absent when there is nothing to apply), the highest-priority user config path that # Ignition's "system" provider reads before the platform provider @@ -48,31 +47,12 @@ log() { printf '%s: %s\n' "${PROG}" "$*" >&2; } die() { log "$*"; exit 1; } NODE_CONFIG=bluefin-node.ign -STUB_URL_VAR=/sys/firmware/efi/efivars/StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f +BOOT_ORIGIN="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}" creds="${CREDENTIALS_DIRECTORY:-/nonexistent}" config="${creds}/ignition.config" url_cred="${creds}/ignition.config.url" -# The URL the node booted from: the UEFI HTTP boot URL recorded by -# systemd-stub, else the explicit http(s) source of rd.systemd.pull=. -origin_url() { - if [ -r "${STUB_URL_VAR}" ]; then - # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. - tail -c +5 "${STUB_URL_VAR}" | tr -d '\000' - return 0 - fi - local arg - for arg in $( None: + result = origin( + tmp_path, + stub="http://booty/bluefin/52-54-00-aa-00-02/bluefin-server-netboot.efi", + cmdline="rd.systemd.pull=" + PULL.format(url="http://other/x.raw"), + ) + assert result.returncode == 0 + assert result.stdout == "http://booty/bluefin/52-54-00-aa-00-02/bluefin-server-netboot.efi\n" + + +def test_origin_uses_the_explicit_pull_url_on_the_cmdline(tmp_path: Path) -> None: + result = origin(tmp_path, cmdline="root=tmpfs rd.systemd.pull=" + PULL.format(url="https://srv/r/bluefin-server_1.raw") + " quiet") + assert (result.returncode, result.stdout) == (0, "https://srv/r/bluefin-server_1.raw\n") + + +def test_origin_falls_back_to_the_import_pull_credential(tmp_path: Path) -> None: + result = origin( + tmp_path, + cmdline="rd.systemd.pull=raw,machine,verify=signature,blockdev,bootorigin:rootdisk:bluefin-server_1.raw", + cred=PULL.format(url="http://10.0.2.2:8765/bluefin-server_1.raw"), + ) + assert (result.returncode, result.stdout) == (0, "http://10.0.2.2:8765/bluefin-server_1.raw\n") + + +def test_origin_fails_quietly_when_not_network_booted(tmp_path: Path) -> None: + result = origin(tmp_path, cmdline="rd.systemd.pull=raw,machine,blockdev,bootorigin:rootdisk:x.raw") + assert (result.returncode, result.stdout) == (1, "") + + +def test_ignition_uses_the_shared_origin_helper() -> None: + helper = (ROOT / "files" / "initrd-ignition" / "usr" / "libexec" / "bluefin-ignition-credentials").read_text(encoding="utf-8") + assert 'BOOT_ORIGIN="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}"' in helper + assert "efivars" not in helper and "/proc/cmdline" not in helper, "one copy of the origin logic" + + +def test_helper_ships_in_the_initrd_and_is_executable_bash() -> None: + elements = ROOT / "elements" / "bluefin-server" + doc = yaml.safe_load((elements / "boot-origin.bst").read_text(encoding="utf-8")) + assert doc["kind"] == "import" + assert doc["sources"] == [{"kind": "local", "path": "files/boot-origin"}] + assert "target" not in (doc.get("config") or {}), "the tree mirrors /usr" + ign = yaml.safe_load((elements / "initrd" / "initrd-ignition-stack.bst").read_text(encoding="utf-8")) + assert "bluefin-server/boot-origin.bst" in ign["depends"] + assert ORIGIN.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert ORIGIN.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(ORIGIN)], check=True) diff --git a/tests/unit/test_signed_lockstep.py b/tests/unit/test_signed_lockstep.py index 30eb515f..fa80fae4 100644 --- a/tests/unit/test_signed_lockstep.py +++ b/tests/unit/test_signed_lockstep.py @@ -63,8 +63,10 @@ def test_sysext_units_are_started_after_a_boot_time_merge() -> None: def test_http_booted_nodes_look_for_ignition_next_to_the_uki() -> None: helper = (IGN / "libexec" / "bluefin-ignition-credentials").read_text(encoding="utf-8") - assert "StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" in helper - assert "rd.systemd.pull=*:http://*" in helper, "iPXE-chainloaded nodes use the explicit pull URL" + origin = (ROOT / "files" / "boot-origin" / "usr" / "libexec" / "bluefin-boot-origin").read_text(encoding="utf-8") + assert "/usr/libexec/bluefin-boot-origin" in helper + assert "StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" in origin + assert "rd.systemd.pull=*)" in origin and "*:http://*|*:https://*)" in origin, "iPXE-chainloaded nodes use the explicit pull URL" assert "NODE_CONFIG=bluefin-node.ign" in helper assert "curl -sfI" in helper for stage in ("disks", "fetch", "fetch-offline", "files", "mount"): From c63c7fd1dedb470da0d7473679932afd31a1b4b2 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 02/17] feat(update): update installed nodes automatically, with a Kubernetes reboot interlock FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and systemd-sysupdate-reboot.timer, so installed nodes never updated on their own and non-Kubernetes hosts never moved to the new slot. 80-bluefin-updates.preset enables both. The timer runs the stock `systemd-sysupdate update`, so enabled features (zfs, kubestellar, kubeadm) keep following the OS in lock-step. Kubernetes nodes keep rebooting through kured. A drop-in adds an ExecCondition= to systemd-sysupdate-reboot.service that skips the local reboot while kubelet, k0scontroller or k0sworker is active, activating, deactivating, reloading or refreshing. This is the interlock from #182 by @mrbobbytables, ported to the FSDK tree. The kured hook also fired on every run: systemd-sysupdate.service is Type=simple, so its ExecStartPost touched /run/reboot-required as soon as an update *started*, even with nothing to download. It is now an ExecStopPost that flags only when `systemd-sysupdate pending` reports an installed update. It moves next to 10-diskless.conf under files/os/systemd/system, and os-kured-hook.bst goes away. The reboot service and both timers get the same 10-diskless.conf conditions as systemd-sysupdate.service, so diskless and USB installer boots stay inert. Refs #139, supersedes #182. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/gap-analysis-distros.md | 5 +- elements/bluefin-server/os-k0s-first-boot.bst | 4 +- elements/bluefin-server/os-kured-hook.bst | 9 - elements/bluefin-server/os-stack.bst | 3 - .../system-preset/80-bluefin-updates.preset | 6 + .../10-diskless.conf | 5 + .../20-kubernetes.conf | 8 + .../10-diskless.conf | 5 + .../systemd-sysupdate.service.d/20-kured.conf | 7 + .../10-diskless.conf | 5 + .../kured-hook.conf | 2 - tests/unit/test_automatic_updates.py | 159 ++++++++++++++++++ 12 files changed, 200 insertions(+), 18 deletions(-) delete mode 100644 elements/bluefin-server/os-kured-hook.bst create mode 100644 files/os/systemd/system-preset/80-bluefin-updates.preset create mode 100644 files/os/systemd/system/systemd-sysupdate-reboot.service.d/10-diskless.conf create mode 100644 files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf create mode 100644 files/os/systemd/system/systemd-sysupdate-reboot.timer.d/10-diskless.conf create mode 100644 files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf create mode 100644 files/os/systemd/system/systemd-sysupdate.timer.d/10-diskless.conf delete mode 100644 files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf create mode 100644 tests/unit/test_automatic_updates.py diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 037a14dc..66a78e4a 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -101,7 +101,7 @@ kernel). No other distribution's binaries ship in the image. | **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless). Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | | **Provisioning** | Stock `systemd-sysinstall` copies `/usr` onto a target disk, started either from the offline USB installer (`bluefin-server-installer_.raw`) or on a diskless-booted node (see [ddi-installer.md](ddi-installer.md)). Per-node configuration is opt-in via Ignition, delivered as `ignition.config` / `ignition.config.url` system credentials (the cmdline is locked inside the signed UKI); Ignition runs on every boot, so configs must be idempotent. First-boot systemd credentials also cover root password, `tmpfiles.extra`, `network.*`, and `firstboot.*`. Sources: [bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst), [initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md). | | **Customization** | Adds software through opt-in `systemd-sysext` images (overlay `/usr`): k0s (Kubernetes) and OpenZFS are built in-tree. The base OS `os-release` identifies as `ID=bluefin-server`; the ZFS sysext pins `VERSION_ID` to the image version because its kernel modules are built against the exact FSDK kernel. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release.bst](../../elements/bluefin-server/os-release.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStopPost` that touches `/run/reboot-required` once an update is pending. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [Kured project](https://github.com/weaveworks/kured). | ## 4. Factual Gaps @@ -193,12 +193,11 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [elements/oci/bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst) - [elements/bluefin-server/os-release.bst](../../elements/bluefin-server/os-release.bst) - [elements/bluefin-server/os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst) -- [elements/bluefin-server/os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst) - [elements/bluefin-server/initrd/initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst) - [files/os/repart.d/50-root.conf](../../files/os/repart.d/50-root.conf) - [files/os/sysupdate.d/10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer) - [files/os/sysupdate.d/20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer) - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) -- [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) +- [files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf) - [files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) diff --git a/elements/bluefin-server/os-k0s-first-boot.bst b/elements/bluefin-server/os-k0s-first-boot.bst index 66985341..5a426d3f 100644 --- a/elements/bluefin-server/os-k0s-first-boot.bst +++ b/elements/bluefin-server/os-k0s-first-boot.bst @@ -5,7 +5,9 @@ description: | (ssh-host-keygen.service plus its sshd.service drop-in), the installer source ESP mounts (run-bluefin-boot.mount for netboot, run-bluefin-installer.mount for the USB installer), and the - systemd-sysinstall.service drop-in the USB installer boots into. + systemd-sysinstall.service drop-in the USB installer boots into, and the + systemd-sysupdate(-reboot) drop-ins (diskless conditions, the kured flag, + the Kubernetes reboot interlock). sources: - kind: local diff --git a/elements/bluefin-server/os-kured-hook.bst b/elements/bluefin-server/os-kured-hook.bst deleted file mode 100644 index 1b91b892..00000000 --- a/elements/bluefin-server/os-kured-hook.bst +++ /dev/null @@ -1,9 +0,0 @@ -kind: import -description: Install systemd-sysupdate post-update hook for Kured coordination. - -sources: - - kind: local - path: files/os/systemd/systemd-sysupdate.service.d - -config: - target: /usr/lib/systemd/system/systemd-sysupdate.service.d diff --git a/elements/bluefin-server/os-stack.bst b/elements/bluefin-server/os-stack.bst index 1a9e54c8..542b08db 100644 --- a/elements/bluefin-server/os-stack.bst +++ b/elements/bluefin-server/os-stack.bst @@ -39,9 +39,6 @@ depends: # First-Boot Credentials Provisioning - bluefin-server/os-creds-prov.bst - # Reboot Coordination for Kured - - bluefin-server/os-kured-hook.bst - # System Management Justfile - bluefin-server/os-justfile.bst diff --git a/files/os/systemd/system-preset/80-bluefin-updates.preset b/files/os/systemd/system-preset/80-bluefin-updates.preset new file mode 100644 index 00000000..44837114 --- /dev/null +++ b/files/os/systemd/system-preset/80-bluefin-updates.preset @@ -0,0 +1,6 @@ +# Automatic updates (docs/skills/ddi-installer.md, "Updates"). +# Installed nodes download a new release into the inactive slot and reboot +# into it in the nightly window; Kubernetes nodes leave the reboot to kured. +# Drop-ins condition both timers off on diskless and USB installer boots. +enable systemd-sysupdate.timer +enable systemd-sysupdate-reboot.timer diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/10-diskless.conf b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/10-diskless.conf new file mode 100644 index 00000000..ebba6ddb --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/10-diskless.conf @@ -0,0 +1,5 @@ +[Unit] +# Same conditions as systemd-sysupdate.service.d/10-diskless.conf: diskless +# nodes and the USB installer run from a tmpfs root and have no slots. +ConditionPathExists=!/run/machines/rootdisk.raw +ConditionKernelCommandLine=!root=tmpfs diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf new file mode 100644 index 00000000..9ab630c5 --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf @@ -0,0 +1,8 @@ +[Service] +# Kubernetes nodes reboot through kured (cordon, drain, one node at a time), +# which systemd-sysupdate.service.d/20-kured.conf triggers. While kubelet +# (kubeadm sysext) or k0s runs, the reboot timer stands down instead of +# rebooting the node under its workloads; a unit that is starting, stopping or +# restarting counts as running. A failed ExecCondition skips the unit rather +# than failing it. Interlock from projectbluefin/server#182. +ExecCondition=/usr/bin/sh -c 'for u in kubelet.service k0scontroller.service k0sworker.service; do s=$$(systemctl is-active "$$u"); case "$$s" in active|activating|deactivating|reloading|refreshing) echo "$$u is $$s: leaving the reboot to kured"; exit 1 ;; esac; done' diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.timer.d/10-diskless.conf b/files/os/systemd/system/systemd-sysupdate-reboot.timer.d/10-diskless.conf new file mode 100644 index 00000000..ebba6ddb --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate-reboot.timer.d/10-diskless.conf @@ -0,0 +1,5 @@ +[Unit] +# Same conditions as systemd-sysupdate.service.d/10-diskless.conf: diskless +# nodes and the USB installer run from a tmpfs root and have no slots. +ConditionPathExists=!/run/machines/rootdisk.raw +ConditionKernelCommandLine=!root=tmpfs diff --git a/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf b/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf new file mode 100644 index 00000000..2a790e8a --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf @@ -0,0 +1,7 @@ +[Service] +# Kured reboots Kubernetes nodes one at a time (cordon, drain, reboot) once +# /run/reboot-required exists. Flag it only when a newer version than the +# booted one is installed: `systemd-sysupdate update` also succeeds when there +# was nothing to download. ExecStopPost, not ExecStartPost: the service is +# Type=simple, so ExecStartPost would run as soon as the update started. +ExecStopPost=/usr/bin/sh -c 'if /usr/bin/systemd-sysupdate pending; then touch /run/reboot-required; fi' diff --git a/files/os/systemd/system/systemd-sysupdate.timer.d/10-diskless.conf b/files/os/systemd/system/systemd-sysupdate.timer.d/10-diskless.conf new file mode 100644 index 00000000..ebba6ddb --- /dev/null +++ b/files/os/systemd/system/systemd-sysupdate.timer.d/10-diskless.conf @@ -0,0 +1,5 @@ +[Unit] +# Same conditions as systemd-sysupdate.service.d/10-diskless.conf: diskless +# nodes and the USB installer run from a tmpfs root and have no slots. +ConditionPathExists=!/run/machines/rootdisk.raw +ConditionKernelCommandLine=!root=tmpfs diff --git a/files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf b/files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf deleted file mode 100644 index d92c4dee..00000000 --- a/files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf +++ /dev/null @@ -1,2 +0,0 @@ -[Service] -ExecStartPost=/usr/bin/sh -c 'touch /run/reboot-required' diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py new file mode 100644 index 00000000..c80278c1 --- /dev/null +++ b/tests/unit/test_automatic_updates.py @@ -0,0 +1,159 @@ +"""Contracts for automatic updates on installed nodes and the boot health gate. + +Installed nodes run systemd-sysupdate.timer and systemd-sysupdate-reboot.timer +by vendor preset; Kubernetes nodes leave the reboot to kured; diskless and USB +installer boots keep every update unit inert; boot-complete.target requires +that no unit failed before systemd-bless-boot marks a counted UKI good. +""" + +from __future__ import annotations + +import configparser +import os +import re +import shlex +import stat +import subprocess +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +PRESETS = ROOT / "files" / "os" / "systemd" / "system-preset" +PRESET = PRESETS / "80-bluefin-updates.preset" +ELEMENTS = ROOT / "elements" / "bluefin-server" +DISKLESS = UNITS / "systemd-sysupdate.service.d" / "10-diskless.conf" +KURED = UNITS / "systemd-sysupdate.service.d" / "20-kured.conf" +INTERLOCK = UNITS / "systemd-sysupdate-reboot.service.d" / "20-kubernetes.conf" +DISKLESS_ONLY = "/run/machines/rootdisk.raw" + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def conditions(path: Path) -> list[str]: + return [ + line.strip() + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith("Condition") + ] + + +def preset_lines() -> list[list[str]]: + return [ + line.split() + for line in PRESET.read_text(encoding="utf-8").splitlines() + if line.strip() and not line.startswith("#") + ] + + +def test_preset_enables_the_update_timers() -> None: + assert preset_lines() == [ + ["enable", "systemd-sysupdate.timer"], + ["enable", "systemd-sysupdate-reboot.timer"], + ] + + +def test_preset_overrides_fsdk_and_yields_to_ignition() -> None: + # FSDK's 90-sysupdate.preset disables both timers; the first match wins. + assert "20-ignition.preset" < PRESET.name < "90-sysupdate.preset" + assert PRESET.name < "90-systemd.preset" + + +def test_no_base_image_preset_enables_kubernetes_or_zfs() -> None: + for preset in PRESETS.glob("*.preset"): + for line in preset.read_text(encoding="utf-8").splitlines(): + if line.startswith("enable"): + assert not re.search(r"kubelet|containerd|k0s|zfs", line), (preset, line) + + +@pytest.mark.parametrize( + "dropin", + [ + "systemd-sysupdate-reboot.service.d/10-diskless.conf", + "systemd-sysupdate.timer.d/10-diskless.conf", + "systemd-sysupdate-reboot.timer.d/10-diskless.conf", + ], +) +def test_update_units_are_inert_on_diskless_and_installer_boots(dropin: str) -> None: + expected = conditions(DISKLESS) + assert expected == [ + f"ConditionPathExists=!{DISKLESS_ONLY}", + "ConditionKernelCommandLine=!root=tmpfs", + ] + assert conditions(UNITS / dropin) == expected + + +def test_kured_flag_is_set_only_when_an_update_is_pending() -> None: + service = ini(KURED)["Service"] + assert "ExecStartPost" not in service, "Type=simple: ExecStartPost runs before the update" + cmd = service["ExecStopPost"] + assert "/usr/bin/systemd-sysupdate pending" in cmd + assert "touch /run/reboot-required" in cmd + for dropin in (UNITS / "systemd-sysupdate.service.d").iterdir(): + keys = {key for section in ini(dropin).values() for key in section} + assert "ExecStartPost" not in keys, dropin + # The timer runs FSDK's `systemd-sysupdate update`, which also fetches + # every enabled feature (zfs, kubestellar, kubeadm) in lock-step. + assert "ExecStart" not in keys, dropin + + +def _interlock_script() -> str: + cmd = ini(INTERLOCK)["Service"]["ExecCondition"] + argv = shlex.split(cmd.replace("$$", "$")) + assert argv[:2] == ["/usr/bin/sh", "-c"] and len(argv) == 3, argv + return argv[2] + + +def _run_interlock(tmp_path: Path, states: dict[str, str]) -> subprocess.CompletedProcess[str]: + bindir = tmp_path / "bin" + bindir.mkdir(exist_ok=True) + fake = bindir / "systemctl" + lines = ["#!/bin/sh", '[ "$1" = is-active ] || exit 99', 'case "$2" in'] + for unit, state in states.items(): + lines.append(f' {unit}) echo {state}; [ {state} = active ]; exit $? ;;') + lines += ["esac", "echo inactive", "exit 3"] + fake.write_text("\n".join(lines) + "\n", encoding="utf-8") + fake.chmod(fake.stat().st_mode | stat.S_IXUSR) + env = dict(os.environ, PATH=f"{bindir}:{os.environ['PATH']}") + return subprocess.run(["sh", "-c", _interlock_script()], capture_output=True, text=True, env=env) + + +def test_reboot_proceeds_without_kubernetes(tmp_path: Path) -> None: + result = _run_interlock(tmp_path, {}) + assert result.returncode == 0, result.stdout + result.stderr + + +@pytest.mark.parametrize( + "unit,state", + [ + ("kubelet.service", "active"), + ("k0scontroller.service", "active"), + ("k0sworker.service", "active"), + ("kubelet.service", "activating"), + ("k0sworker.service", "deactivating"), + ], +) +def test_reboot_stands_down_while_kubernetes_runs(tmp_path: Path, unit: str, state: str) -> None: + result = _run_interlock(tmp_path, {unit: state}) + # 1..254 makes systemd skip the unit instead of failing it. + assert result.returncode == 1, result.stdout + result.stderr + assert "leaving the reboot to kured" in result.stdout + + +def test_reboot_proceeds_when_kubernetes_is_stopped_or_failed(tmp_path: Path) -> None: + result = _run_interlock(tmp_path, {"kubelet.service": "failed", "k0sworker.service": "inactive"}) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_kured_hook_moved_into_the_unit_directory() -> None: + stack = yaml.safe_load((ELEMENTS / "os-stack.bst").read_text(encoding="utf-8"))["depends"] + assert "bluefin-server/os-k0s-first-boot.bst" in stack, "installs files/os/systemd/system" + assert "bluefin-server/os-kured-hook.bst" not in stack + assert not (ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d").exists() From b910aec05871592fc6f5072363526cb112e83b0b Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 03/17] feat(update): bless a boot-counted UKI only when no unit failed systemd-bless-boot.service requires boot-complete.target, which by itself only needs sysinit.target, so a new UKI was marked good long before the system was up. Enable FSDK's systemd-boot-check-no-failures.service (RequiredBy=boot-complete.target, after multi-user.target), which FSDK's presets disable. A good boot now reaches multi-user.target with no failed unit; any other boot uses up one of the TriesLeft=3 and systemd-boot falls back after the third. Only sysupdate-installed UKIs are counted, so diskless boots never pull in boot-complete.target. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- files/os/systemd/system-preset/80-bluefin-updates.preset | 3 +++ tests/unit/test_automatic_updates.py | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/files/os/systemd/system-preset/80-bluefin-updates.preset b/files/os/systemd/system-preset/80-bluefin-updates.preset index 44837114..e10ed4ac 100644 --- a/files/os/systemd/system-preset/80-bluefin-updates.preset +++ b/files/os/systemd/system-preset/80-bluefin-updates.preset @@ -4,3 +4,6 @@ # Drop-ins condition both timers off on diskless and USB installer boots. enable systemd-sysupdate.timer enable systemd-sysupdate-reboot.timer +# Boot health gate: boot-complete.target, and with it systemd-bless-boot +# marking a boot-counted UKI good, requires that no unit has failed. +enable systemd-boot-check-no-failures.service diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py index c80278c1..85e939c8 100644 --- a/tests/unit/test_automatic_updates.py +++ b/tests/unit/test_automatic_updates.py @@ -53,10 +53,11 @@ def preset_lines() -> list[list[str]]: ] -def test_preset_enables_the_update_timers() -> None: +def test_preset_enables_the_update_timers_and_the_health_gate() -> None: assert preset_lines() == [ ["enable", "systemd-sysupdate.timer"], ["enable", "systemd-sysupdate-reboot.timer"], + ["enable", "systemd-boot-check-no-failures.service"], ] From 104f87b4f7cfdcab9c84429016249313217d5c8f Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 04/17] feat(update): tell diskless nodes when their boot server has a newer release Diskless nodes update by rebooting, but had no way to know a newer release was being served. bluefin-diskless-update-check.timer (5 min after boot, then hourly) runs only when /run/machines/rootdisk.raw exists, the same test 10-diskless.conf uses. It finds the boot directory with bluefin-boot-origin, fetches SHA256SUMS and SHA256SUMS.gpg, verifies them with gpgv against the image keyring, and compares the bluefin-server_.raw entry with IMAGE_VERSION using `systemd-analyze compare-versions`. When the release is newer it touches /run/reboot-required (kured's sentinel) and logs the new version. Network and signature errors are logged and never set the flag or fail the unit, and the check never reboots. It needs no new runtime dependencies (bash, coreutils, curl, gpgv, systemd-analyze). Behavioural tests run the helper against a local http.server with a throwaway GnuPG key: newer (26.09.10 vs 26.09.2) sets the flag, over the cmdline and the import.pull credential alike. Same or older, a foreign signature, a manifest changed after signing, a missing .gpg, an unreachable server, and a node that was not network booted all leave it unset. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- elements/bluefin-server/boot-origin.bst | 5 +- elements/bluefin-server/os-k0s-first-boot.bst | 4 +- elements/bluefin-server/os-stack.bst | 5 + elements/bluefin-server/os-update-check.bst | 14 ++ .../system-preset/80-bluefin-updates.preset | 3 + .../bluefin-diskless-update-check.service | 22 ++ .../bluefin-diskless-update-check.timer | 12 + .../usr/libexec/bluefin-diskless-update-check | 106 +++++++++ tests/unit/test_automatic_updates.py | 1 + tests/unit/test_diskless_update_check.py | 215 ++++++++++++++++++ 10 files changed, 383 insertions(+), 4 deletions(-) create mode 100644 elements/bluefin-server/os-update-check.bst create mode 100644 files/os/systemd/system/bluefin-diskless-update-check.service create mode 100644 files/os/systemd/system/bluefin-diskless-update-check.timer create mode 100755 files/os/update-check/usr/libexec/bluefin-diskless-update-check create mode 100644 tests/unit/test_diskless_update_check.py diff --git a/elements/bluefin-server/boot-origin.bst b/elements/bluefin-server/boot-origin.bst index 1d90987e..c8e2c0ec 100644 --- a/elements/bluefin-server/boot-origin.bst +++ b/elements/bluefin-server/boot-origin.bst @@ -2,8 +2,9 @@ kind: import description: | /usr/libexec/bluefin-boot-origin: prints the URL a network-booted node was booted from (UEFI HTTP boot URL, the explicit rd.systemd.pull= source, or - the import.pull credential), for bluefin-ignition-credentials in the initrd - (initrd-ignition-stack.bst). + the import.pull credential). One helper for both consumers: + bluefin-ignition-credentials in the initrd (initrd-ignition-stack.bst) and + bluefin-diskless-update-check in the booted OS (os-update-check.bst). sources: - kind: local diff --git a/elements/bluefin-server/os-k0s-first-boot.bst b/elements/bluefin-server/os-k0s-first-boot.bst index 5a426d3f..a294bd70 100644 --- a/elements/bluefin-server/os-k0s-first-boot.bst +++ b/elements/bluefin-server/os-k0s-first-boot.bst @@ -6,8 +6,8 @@ description: | source ESP mounts (run-bluefin-boot.mount for netboot, run-bluefin-installer.mount for the USB installer), and the systemd-sysinstall.service drop-in the USB installer boots into, and the - systemd-sysupdate(-reboot) drop-ins (diskless conditions, the kured flag, - the Kubernetes reboot interlock). + update units: systemd-sysupdate(-reboot) drop-ins (diskless conditions, the + kured flag, the Kubernetes reboot interlock) and the diskless update check. sources: - kind: local diff --git a/elements/bluefin-server/os-stack.bst b/elements/bluefin-server/os-stack.bst index 542b08db..b538d1ed 100644 --- a/elements/bluefin-server/os-stack.bst +++ b/elements/bluefin-server/os-stack.bst @@ -19,6 +19,11 @@ depends: # A/B System Update Configurations - bluefin-server/os-sysupdate.bst + # Diskless update signal (bluefin-diskless-update-check) and the shared + # boot-origin helper it uses + - bluefin-server/os-update-check.bst + - bluefin-server/boot-origin.bst + # k0s component system update configuration - bluefin-server/os-k0s-sysupdate.bst diff --git a/elements/bluefin-server/os-update-check.bst b/elements/bluefin-server/os-update-check.bst new file mode 100644 index 00000000..73f303ad --- /dev/null +++ b/elements/bluefin-server/os-update-check.bst @@ -0,0 +1,14 @@ +kind: import +description: | + /usr/libexec/bluefin-diskless-update-check, run by + bluefin-diskless-update-check.timer (files/os/systemd/system, enabled by + 80-bluefin-updates.preset) on diskless boots only: fetches the signed + SHA256SUMS from the directory the node booted from, and when it lists a + newer bluefin-server_.raw than the running IMAGE_VERSION, touches + /run/reboot-required for kured or the operator. It never reboots. + Runtime tools: bash, coreutils, curl, gpgv, systemd-analyze (os-base.bst) + and bluefin-boot-origin (boot-origin.bst). + +sources: +- kind: local + path: files/os/update-check diff --git a/files/os/systemd/system-preset/80-bluefin-updates.preset b/files/os/systemd/system-preset/80-bluefin-updates.preset index e10ed4ac..27ea26bf 100644 --- a/files/os/systemd/system-preset/80-bluefin-updates.preset +++ b/files/os/systemd/system-preset/80-bluefin-updates.preset @@ -7,3 +7,6 @@ enable systemd-sysupdate-reboot.timer # Boot health gate: boot-complete.target, and with it systemd-bless-boot # marking a boot-counted UKI good, requires that no unit has failed. enable systemd-boot-check-no-failures.service +# Diskless nodes flag /run/reboot-required when their boot server offers a +# newer release; the timer is conditioned off everywhere else. +enable bluefin-diskless-update-check.timer diff --git a/files/os/systemd/system/bluefin-diskless-update-check.service b/files/os/systemd/system/bluefin-diskless-update-check.service new file mode 100644 index 00000000..30b5337f --- /dev/null +++ b/files/os/systemd/system/bluefin-diskless-update-check.service @@ -0,0 +1,22 @@ +[Unit] +Description=Check the Boot Server for a Newer Bluefin Server Release +Documentation=file:/usr/libexec/bluefin-diskless-update-check +# Diskless boots only: the OS DDI pulled at boot is attached from here. This is +# the inverse of systemd-sysupdate.service.d/10-diskless.conf; installed nodes +# update with systemd-sysupdate and the USB installer has no rootdisk.raw. +ConditionPathExists=/run/machines/rootdisk.raw +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +# The pull source of a node booted with the import.pull credential (netboot +# ESP with /loader/credentials, dogfood VM), read by bluefin-boot-origin. +ImportCredential=import.pull +ExecStart=/usr/libexec/bluefin-diskless-update-check +ProtectSystem=strict +ReadWritePaths=/run +ProtectHome=yes +PrivateTmp=yes +NoNewPrivileges=yes +RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 diff --git a/files/os/systemd/system/bluefin-diskless-update-check.timer b/files/os/systemd/system/bluefin-diskless-update-check.timer new file mode 100644 index 00000000..5a231e8f --- /dev/null +++ b/files/os/systemd/system/bluefin-diskless-update-check.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Periodic Check for a Newer Bluefin Server Release (diskless) +Documentation=file:/usr/libexec/bluefin-diskless-update-check +ConditionPathExists=/run/machines/rootdisk.raw + +[Timer] +OnBootSec=5min +OnUnitActiveSec=1h +RandomizedDelaySec=10min + +[Install] +WantedBy=timers.target diff --git a/files/os/update-check/usr/libexec/bluefin-diskless-update-check b/files/os/update-check/usr/libexec/bluefin-diskless-update-check new file mode 100755 index 00000000..ce734182 --- /dev/null +++ b/files/os/update-check/usr/libexec/bluefin-diskless-update-check @@ -0,0 +1,106 @@ +#!/usr/bin/bash +# bluefin-diskless-update-check — flag a diskless node whose boot server +# offers a newer Bluefin Server release. +# +# A diskless node runs the OS DDI it pulled into RAM at boot, and it updates +# by rebooting into whatever its boot server serves at that point. This check +# tells the node when a reboot would bring a newer image: +# +# 1. find the directory the node booted from (bluefin-boot-origin); +# 2. fetch SHA256SUMS and SHA256SUMS.gpg from it and verify the signature +# with gpgv against the image keyring. This is the same trust root as the +# initrd's `rd.systemd.pull ... verify=signature`; +# 3. read the release version from the manifest's bluefin-server_.raw +# entry and compare it with the running IMAGE_VERSION using +# `systemd-analyze compare-versions`, the ordering systemd-sysupdate uses; +# 4. if the release is newer, touch /run/reboot-required (the sentinel kured +# watches) and log the new version. +# +# It never reboots. Network and signature errors are logged and leave the +# flag untouched, and the check always exits 0, so a boot server that cannot +# be reached never fails a unit. +# +# Journal priorities come from SyslogLevelPrefix: <4> warning, <5> notice, +# <6> info. +# +# Test overrides: BLUEFIN_BOOT_ORIGIN (the origin helper), +# BLUEFIN_IMPORT_KEYRING, BLUEFIN_REBOOT_SENTINEL, BLUEFIN_OS_RELEASE, and +# BLUEFIN_CMDLINE / BLUEFIN_STUB_URL_VAR, which bluefin-boot-origin reads. + +set -euo pipefail +export LC_ALL=C + +boot_origin="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}" +sentinel="${BLUEFIN_REBOOT_SENTINEL:-/run/reboot-required}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +# systemd reads /etc/systemd/import-pubring.pgp before the vendor keyring. +keyring="${BLUEFIN_IMPORT_KEYRING:-}" +if [ -z "${keyring}" ]; then + keyring=/usr/lib/systemd/import-pubring.pgp + [ -e /etc/systemd/import-pubring.pgp ] && keyring=/etc/systemd/import-pubring.pgp +fi + +info() { printf '<6>%s\n' "$*"; } +notice() { printf '<5>%s\n' "$*"; } +warn() { printf '<4>%s\n' "$*"; } + +running="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + running="${value}" + done < "${os_release}" +fi +if [ -z "${running}" ]; then + warn "no IMAGE_VERSION in ${os_release}; not checking for updates" + exit 0 +fi + +if ! origin="$("${boot_origin}")"; then + info "not network booted (no boot origin); nothing to check" + exit 0 +fi +base="${origin%/*}" + +tmp="$(mktemp -d)" +trap 'rm -rf "${tmp}"' EXIT + +fetch() { + curl -fsS --proto '=http,https' --retry 2 --connect-timeout 10 --max-time 60 \ + --max-filesize 1048576 -o "${tmp}/$1" "${base}/$1" +} +if ! fetch SHA256SUMS || ! fetch SHA256SUMS.gpg; then + warn "cannot fetch SHA256SUMS and SHA256SUMS.gpg from ${base}/; not flagging an update" + exit 0 +fi + +mkdir -m 0700 "${tmp}/gnupg" +if ! GNUPGHOME="${tmp}/gnupg" gpgv --keyring "${keyring}" \ + "${tmp}/SHA256SUMS.gpg" "${tmp}/SHA256SUMS"; then + warn "SHA256SUMS from ${base}/ does not verify against ${keyring}; not flagging an update" + exit 0 +fi + +offered="" +while read -r _ name; do + name="${name#\*}" + [[ "${name}" =~ ^bluefin-server_([0-9A-Za-z.]+)\.raw$ ]] || continue + version="${BASH_REMATCH[1]}" + if [ -z "${offered}" ] || + systemd-analyze compare-versions "${version}" gt "${offered}" >/dev/null; then + offered="${version}" + fi +done < "${tmp}/SHA256SUMS" +if [ -z "${offered}" ]; then + warn "SHA256SUMS from ${base}/ lists no bluefin-server_.raw; not flagging an update" + exit 0 +fi + +if systemd-analyze compare-versions "${offered}" gt "${running}" >/dev/null; then + mkdir -p "${sentinel%/*}" + touch "${sentinel}" + notice "Bluefin Server ${offered} is available from ${base}/ (running ${running}); flagged ${sentinel}, reboot to update" +else + info "up to date: ${base}/ offers ${offered}, running ${running}" +fi diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py index 85e939c8..26735f8f 100644 --- a/tests/unit/test_automatic_updates.py +++ b/tests/unit/test_automatic_updates.py @@ -58,6 +58,7 @@ def test_preset_enables_the_update_timers_and_the_health_gate() -> None: ["enable", "systemd-sysupdate.timer"], ["enable", "systemd-sysupdate-reboot.timer"], ["enable", "systemd-boot-check-no-failures.service"], + ["enable", "bluefin-diskless-update-check.timer"], ] diff --git a/tests/unit/test_diskless_update_check.py b/tests/unit/test_diskless_update_check.py new file mode 100644 index 00000000..24e90764 --- /dev/null +++ b/tests/unit/test_diskless_update_check.py @@ -0,0 +1,215 @@ +"""Executed coverage for the diskless update signal. + +bluefin-boot-origin finds the URL a node booted from; bluefin-diskless-update-check +fetches the signed SHA256SUMS from that directory and flags /run/reboot-required +when it offers a newer release. Both run here against a local http.server and a +throwaway GnuPG key; no files/boot-keys are needed. +""" + +from __future__ import annotations + +import configparser +import functools +import os +import shutil +import stat +import subprocess +import threading +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +ORIGIN = ROOT / "files" / "boot-origin" / "usr" / "libexec" / "bluefin-boot-origin" +CHECK = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" / "bluefin-diskless-update-check" +RUNNING = "26.09.2" +PULL = "raw,machine,verify=signature,blockdev:rootdisk:{url}" + +UNITS = ROOT / "files" / "os" / "systemd" / "system" +ELEMENTS = ROOT / "elements" / "bluefin-server" +needs_tools = pytest.mark.skipif( + not all(shutil.which(t) for t in ("curl", "gpg", "gpgv", "systemd-analyze")), + reason="needs curl, gpg, gpgv and systemd-analyze", +) + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def test_check_runs_only_on_diskless_boots() -> None: + service = ini(UNITS / "bluefin-diskless-update-check.service") + timer = ini(UNITS / "bluefin-diskless-update-check.timer") + for unit in (service, timer): + assert unit["Unit"]["ConditionPathExists"] == "/run/machines/rootdisk.raw" + assert service["Service"]["Type"] == "oneshot" + assert service["Service"]["ImportCredential"] == "import.pull" + assert service["Service"]["ExecStart"] == "/usr/libexec/bluefin-diskless-update-check" + assert service["Unit"]["After"] == "network-online.target" + assert timer["Install"]["WantedBy"] == "timers.target" + + +def test_check_ships_in_the_os_and_is_executable_bash() -> None: + stack = yaml.safe_load((ELEMENTS / "os-stack.bst").read_text(encoding="utf-8"))["depends"] + assert "bluefin-server/os-update-check.bst" in stack + assert "bluefin-server/boot-origin.bst" in stack + doc = yaml.safe_load((ELEMENTS / "os-update-check.bst").read_text(encoding="utf-8")) + assert doc["kind"] == "import" + assert doc["sources"] == [{"kind": "local", "path": "files/os/update-check"}] + assert "target" not in (doc.get("config") or {}), "the tree mirrors /usr" + assert CHECK.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert CHECK.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(CHECK)], check=True) + + +@pytest.fixture(scope="module") +def keys(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]: + base = tmp_path_factory.mktemp("gpg") + out = {} + for name in ("release", "stranger"): + home = base / name + home.mkdir(mode=0o700) + env = dict(os.environ, GNUPGHOME=str(home)) + subprocess.run( + ["gpg", "--batch", "--quiet", "--passphrase", "", "--quick-gen-key", f"{name} <{name}@example.invalid>", "ed25519", "sign", "never"], + check=True, env=env, capture_output=True, + ) + ring = base / f"{name}.pgp" + subprocess.run(["gpg", "--batch", "--export", "--output", str(ring)], check=True, env=env) + out[name] = home + out[f"{name}-ring"] = ring + return out + + +class QuietHandler(SimpleHTTPRequestHandler): + def log_message(self, *args) -> None: + pass + + +@pytest.fixture() +def server(tmp_path: Path): + root = tmp_path / "srv" + (root / "rel").mkdir(parents=True) + handler = functools.partial(QuietHandler, directory=str(root)) + httpd = ThreadingHTTPServer(("127.0.0.1", 0), handler) + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + yield root / "rel", f"http://127.0.0.1:{httpd.server_address[1]}/rel" + httpd.shutdown() + httpd.server_close() + + +def publish(directory: Path, version: str, gnupghome: Path, *, tamper: bool = False, sign: bool = True) -> None: + names = [ + f"bluefin-server_{version}.raw", + f"bluefin-server_{version}_d3107d37-a9da-32cf-3c19-49fa3b0cb1df.usr.raw", + f"bluefin-server-{version}.efi", + f"bluefin-server-netboot_{version}.efi", + f"bluefin-server-installer_{version}.raw", + f"zfs_{version}.raw.zst", + ] + sums = "".join(f"{'%064x' % i} {name}\n" for i, name in enumerate(names)) + (directory / "SHA256SUMS").write_text(sums, encoding="utf-8") + if sign: + subprocess.run( + ["gpg", "--batch", "--yes", "--detach-sign", "--output", str(directory / "SHA256SUMS.gpg"), str(directory / "SHA256SUMS")], + check=True, env=dict(os.environ, GNUPGHOME=str(gnupghome)), capture_output=True, + ) + if tamper: + (directory / "SHA256SUMS").write_text(sums.replace(version, "99.99.99"), encoding="utf-8") + + +def check(tmp_path: Path, url: str | None, keyring: Path, *, via_cred: bool = False): + os_release = tmp_path / "os-release" + os_release.write_text(f'ID=bluefin-server\nIMAGE_ID=bluefin-server\nIMAGE_VERSION="{RUNNING}"\n', encoding="utf-8") + sentinel = tmp_path / "run" / "reboot-required" + pull = PULL.format(url=f"{url}/bluefin-server_{RUNNING}.raw") if url else "" + env = dict( + os.environ, + BLUEFIN_BOOT_ORIGIN=str(ORIGIN), + BLUEFIN_IMPORT_KEYRING=str(keyring), + BLUEFIN_REBOOT_SENTINEL=str(sentinel), + BLUEFIN_OS_RELEASE=str(os_release), + BLUEFIN_STUB_URL_VAR=str(tmp_path / "no-efivar"), + BLUEFIN_CMDLINE=str(tmp_path / "cmdline"), + ) + env.pop("CREDENTIALS_DIRECTORY", None) + if via_cred: + (tmp_path / "cmdline").write_text("root=tmpfs\n", encoding="utf-8") + (tmp_path / "creds").mkdir() + (tmp_path / "creds" / "import.pull").write_text(pull + "\n", encoding="utf-8") + env["CREDENTIALS_DIRECTORY"] = str(tmp_path / "creds") + else: + (tmp_path / "cmdline").write_text(("rd.systemd.pull=" + pull if pull else "root=tmpfs") + "\n", encoding="utf-8") + result = subprocess.run([str(CHECK)], capture_output=True, text=True, env=env, timeout=120) + assert result.returncode == 0, result.stdout + result.stderr + return sentinel.exists(), result.stdout + result.stderr + + +@needs_tools +@pytest.mark.parametrize("via_cred", [False, True]) +def test_newer_release_sets_the_flag(tmp_path: Path, server, keys, via_cred: bool) -> None: + directory, url = server + # 26.09.10 sorts after 26.09.2 as a version, not as a string. + publish(directory, "26.09.10", keys["release"]) + flagged, log = check(tmp_path, url, keys["release-ring"], via_cred=via_cred) + assert flagged, log + assert f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING})" in log + + +@needs_tools +@pytest.mark.parametrize("version", [RUNNING, "26.09.1", "26.08.30"]) +def test_same_or_older_release_sets_no_flag(tmp_path: Path, server, keys, version: str) -> None: + directory, url = server + publish(directory, version, keys["release"]) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert f"up to date: {url}/ offers {version}, running {RUNNING}" in log + + +@needs_tools +def test_signature_from_another_key_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["stranger"]) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "does not verify" in log + + +@needs_tools +def test_manifest_changed_after_signing_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["release"], tamper=True) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "does not verify" in log + + +@needs_tools +def test_unsigned_manifest_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["release"], sign=False) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "cannot fetch" in log + + +@needs_tools +def test_unreachable_boot_server_sets_no_flag(tmp_path: Path, keys) -> None: + with ThreadingHTTPServer(("127.0.0.1", 0), SimpleHTTPRequestHandler) as probe: + port = probe.server_address[1] + flagged, log = check(tmp_path, f"http://127.0.0.1:{port}/rel", keys["release-ring"]) + assert not flagged, log + assert "cannot fetch" in log + + +@needs_tools +def test_not_network_booted_is_a_no_op(tmp_path: Path, keys) -> None: + flagged, log = check(tmp_path, None, keys["release-ring"]) + assert not flagged, log + assert "not network booted" in log From 615054ae2aace55c3365bd5a2be545a9e6b1d47d Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 05/17] test(dogfood): prove automatic updates, the interlock and the health gate in QEMU dogfood-diskless.sh: every boot reports the update timers, and a diskless boot runs bluefin-diskless-update-check once and reports its result and the flag (the CI boot-test job covers the same-version path on every PR). dogfood-install.sh: the installed disk must have both timers enabled and active. The A->B update now runs through systemd-sysupdate.service, the unit the timer starts, and must set /run/reboot-required. With a stand-in kubelet.service running, systemd-sysupdate-reboot.service must be skipped (Result=exec-condition) instead of rebooting. The updated boot must pass systemd-boot-check-no-failures, reach boot-complete.target and be blessed (`systemd-bless-boot status` = good). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/dogfood-diskless.sh | 6 ++++++ scripts/dogfood-install.sh | 28 +++++++++++++++++++++++----- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/scripts/dogfood-diskless.sh b/scripts/dogfood-diskless.sh index e12cd66f..fbaa92f3 100644 --- a/scripts/dogfood-diskless.sh +++ b/scripts/dogfood-diskless.sh @@ -137,6 +137,12 @@ echo "PROBE var=$(findmnt -no SOURCE,FSTYPE /var)" boots=$(( $(cat /var/lib/dogfood-boots 2>/dev/null || echo 0) + 1 )) echo "${boots}" > /var/lib/dogfood-boots echo "PROBE boots=${boots}" +echo "PROBE update-timers=$(systemctl is-active systemd-sysupdate.timer systemd-sysupdate-reboot.timer bluefin-diskless-update-check.timer | tr '\n' ' ')" +if [ -e /run/machines/rootdisk.raw ]; then + systemctl start bluefin-diskless-update-check.service || true + echo "PROBE update-check=$(systemctl show -P Result bluefin-diskless-update-check.service) flag=$(test -e /run/reboot-required && echo set || echo none)" + journalctl -b -o cat --no-pager -u bluefin-diskless-update-check.service | grep -v '^gpgv:' | tail -n 3 | sed 's/^/PROBE-LOG /' +fi PROBE [ -n "${DOGFOOD_EXTRA_PROBE:-}" ] && cat "${DOGFOOD_EXTRA_PROBE}" >> "${work}/probe.sh" echo 'sync' >> "${work}/probe.sh" diff --git a/scripts/dogfood-install.sh b/scripts/dogfood-install.sh index 91944222..1e9897bd 100644 --- a/scripts/dogfood-install.sh +++ b/scripts/dogfood-install.sh @@ -6,9 +6,14 @@ # and confirm the node runs it from slot B with a boot-counted UKI # 4. with : update to it, corrupt its slot, and confirm boot # counting rolls the node back to on its own -# The updates verify the signed SHA256SUMS and have the optional "zfs" -# sysupdate feature enabled, so the ZFS sysext follows the OS in lock-step -# and must still be active after the rollback. +# The updates run through systemd-sysupdate.service, the unit the +# preset-enabled timer starts, verify the signed SHA256SUMS and have the +# optional "zfs" sysupdate feature enabled, so the ZFS sysext follows the OS +# in lock-step and must still be active after the rollback. Once an update is +# staged the kured flag must be set, and the reboot unit must stand down while +# a stand-in kubelet.service runs. Every disk boot has both update timers +# enabled; a boot-counted UKI is blessed only after boot-complete.target, +# which requires that no unit failed. # Usage: dogfood-install.sh [ []] set -euo pipefail @@ -44,6 +49,8 @@ echo "PROBE boot-entry=$(bootctl status 2>/dev/null | sed -n 's/^ *Current Entry bootctl list --no-pager 2>/dev/null | sed -n 's/^ *\(title\|id\): */PROBE-LOG \1 /p' systemctl start boot-complete.target 2>/dev/null || true echo "PROBE ukis=$(ls /boot/EFI/Linux 2>/dev/null | tr '\n' ' ')" +echo "PROBE timers-enabled=$(systemctl is-enabled systemd-sysupdate.timer systemd-sysupdate-reboot.timer | tr '\n' ' ')" +echo "PROBE health=$(systemctl is-active systemd-boot-check-no-failures.service boot-complete.target | tr '\n' ' ')bless=$(/usr/lib/systemd/systemd-bless-boot status 2>/dev/null)" EOF cat > "${state}/update.probe" <<'EOF' @@ -53,11 +60,17 @@ for f in /usr/lib/sysupdate.d/*.transfer; do sed -e 's|^Path=https://.*|Path=http://10.0.2.2:8765/|' \ "${f}" > "/etc/sysupdate.d/${f##*/}" done +rm -f /run/reboot-required rc=0 -systemd-sysupdate update > /run/sysupdate.log 2>&1 || rc=$? +systemctl start --wait systemd-sysupdate.service || rc=$? echo "PROBE update=${rc}" -tail -n 15 /run/sysupdate.log | sed 's/^/PROBE-LOG /' +journalctl -b -o cat --no-pager -u systemd-sysupdate.service | tail -n 15 | sed 's/^/PROBE-LOG /' timeout 60 systemd-sysupdate list --no-pager 2>&1 | sed 's/^/PROBE-LOG /' +echo "PROBE kured-flag=$(test -e /run/reboot-required && echo set || echo none)" +systemd-run --quiet --unit=kubelet.service sleep 600 +systemctl start systemd-sysupdate-reboot.service +echo "PROBE interlock=$(systemctl show -P Result systemd-sysupdate-reboot.service)" +systemctl stop kubelet.service EOF echo "==> 1/4 diskless boot + systemd-sysinstall" @@ -67,18 +80,23 @@ grep -q 'PROBE install=0' "${state}/1-install.log" echo "==> 2/4 boot the installed disk" DOGFOOD_BOOT=disk run "${dir}" "${state}/disk.probe" | tee "${state}/2-disk.log" grep -q 'PROBE root=xfs' "${state}/2-disk.log" +grep -q 'PROBE timers-enabled=enabled enabled' "${state}/2-disk.log" +grep -q 'PROBE update-timers=active active inactive' "${state}/2-disk.log" [ -n "${next}" ] || { echo "PASS: installed and booted from disk"; exit 0; } echo "==> 3/4 systemd-sysupdate to $(basename "${next}")" DOGFOOD_TIMEOUT="${DOGFOOD_UPDATE_TIMEOUT:-900}" DOGFOOD_BOOT=disk run "${next}" "${state}/update.probe" | tee "${state}/3-update.log" grep -q 'PROBE update=0' "${state}/3-update.log" +grep -q 'PROBE kured-flag=set' "${state}/3-update.log" +grep -q 'PROBE interlock=exec-condition' "${state}/3-update.log" echo "==> 4/4 boot the updated disk" DOGFOOD_BOOT=disk run "${next}" "${state}/disk.probe" | tee "${state}/4-updated.log" new_ver="$(ls "${next}"/bluefin-server-[0-9]*.efi | sed -n 's|.*/bluefin-server-\(.*\)\.efi$|\1|p')" grep -q "PROBE os=bluefin-server ${new_ver}" "${state}/4-updated.log" grep -q "PROBE zfs=active" "${state}/4-updated.log" +grep -q "PROBE health=active active bless=good" "${state}/4-updated.log" [ -n "${broken}" ] || { echo "PASS: installed, updated A->B and booted ${new_ver}"; exit 0; } bad_ver="$(ls "${broken}"/bluefin-server-[0-9]*.efi | sed -n 's|.*/bluefin-server-\(.*\)\.efi$|\1|p')" From a06015dff31efd47d508855ae8116056d937b425 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 06/17] docs(skills): document automatic updates, the kured interlock and the diskless update signal ddi-installer.md "Updates" is the canonical description: timers and preset, the boot health gate, the kured flag and Kubernetes interlock, opting out, the one-time `systemctl preset` for nodes installed before the preset existed, and how kured, operators and Booty act on the diskless flag. The verification skill records that the diskless check trusts the same keyring, and ddi-installer-build.md lists the new dogfood assertions. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/ddi-installer-build.md | 14 +++- docs/skills/ddi-installer.md | 78 +++++++++++++++---- docs/skills/systemd-sysupdate-verification.md | 10 ++- 3 files changed, 83 insertions(+), 19 deletions(-) diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md index 0d8e1067..f29979cb 100644 --- a/docs/skills/ddi-installer-build.md +++ b/docs/skills/ddi-installer-build.md @@ -4,7 +4,7 @@ description: Build, export, and dogfood the Bluefin Server image set (OS DDI, si metadata: type: how-to status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd - /apache/buildstream @@ -102,11 +102,17 @@ Useful environment variables: `SHA256SUMS`; the boot must fail, proving the signature check. - `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe. +Every diskless `--check` boot also runs `bluefin-diskless-update-check` once +and reports `PROBE update-check= flag=`. + `scripts/dogfood-install.sh [ []]` is the full end-to-end check: install from a diskless boot, boot the installed disk, -`systemd-sysupdate` A->B to `` with the default `Verify=yes` against -the signed manifest (with the `zfs` feature enabled, so the ZFS sysext follows -the OS in lock-step), and with `` corrupt the updated slot and +`systemd-sysupdate` A->B to `` through `systemd-sysupdate.service` +(the unit the timer starts) with the default `Verify=yes` against the signed +manifest (with the `zfs` feature enabled, so the ZFS sysext follows the OS in +lock-step), then asserts the kured flag, the Kubernetes reboot interlock, both +timers enabled and the new UKI blessed after `boot-complete.target`, and with +`` corrupt the updated slot and confirm boot counting rolls the node back to `` on its own, with the matching ZFS sysext still merged. CI runs the first two stages (`dogfood-diskless.sh --check`, `dogfood-install.sh dist/diskless`) as the diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index 44d4f673..7af0b7c8 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd - /apache/buildstream @@ -190,17 +190,65 @@ Installed nodes update with `systemd-sysupdate` against the transfers in Sources are the release assets on GitHub Releases, verified against the GPG-signed `SHA256SUMS` with `Verify=yes` (see -`systemd-sysupdate-verification.md`). After an update the node reboots into -slot B; if the new image fails its three tries, systemd-boot rolls back to -slot A on its own. A Kured hook -(`files/os/systemd/system/systemd-sysupdate.service.d/kured-hook.conf`) -touches `/run/reboot-required` for cluster-aware reboot coordination. - -Diskless nodes have no slots, so `systemd-sysupdate.service` is disabled when -booted diskless -(`files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf`). -A diskless node updates by rebooting into a newer image; that is the whole -mechanism. +`systemd-sysupdate-verification.md`). + +### Automatic updates on installed nodes + +`files/os/systemd/system-preset/80-bluefin-updates.preset` enables FSDK's +`systemd-sysupdate.timer` (15 min after boot, then every 2 h, randomized) and +`systemd-sysupdate-reboot.timer` (04:10, randomized), which FSDK's +`90-sysupdate.preset` would otherwise disable. The timer runs the same +`systemd-sysupdate update` as a manual update, so enabled features follow the +OS in lock-step. The reboot unit runs `systemd-sysupdate reboot`, which reboots +only when a newer version than the booted one is installed. + +- **Boot health gate.** The preset enables + `systemd-boot-check-no-failures.service` (`RequiredBy=boot-complete.target`, + after `multi-user.target`), and `systemd-bless-boot` marks a boot-counted UKI + good only after `boot-complete.target`. A good boot is one that reaches + `multi-user.target` with no failed unit; any other boot uses up a try, and + systemd-boot falls back to the previous UKI after the third. Only + sysupdate-installed UKIs are counted, so diskless boots never pull in + `boot-complete.target`. +- **Kubernetes interlock.** Kubernetes nodes reboot through + [kured](https://kured.dev/) (cordon, drain, one node at a time). + `systemd-sysupdate.service.d/20-kured.conf` touches `/run/reboot-required` + after each run once `systemd-sysupdate pending` reports an installed update. + `systemd-sysupdate-reboot.service.d/20-kubernetes.conf` skips the local + reboot (`ExecCondition=`) while `kubelet.service`, `k0scontroller.service` or + `k0sworker.service` is running, the interlock proposed in + [#182](https://github.com/projectbluefin/server/pull/182). Without kured, a + Kubernetes node keeps the staged update until someone reboots it. +- **Opting out.** `systemctl disable --now systemd-sysupdate-reboot.timer` + stages updates without rebooting; also disable `systemd-sysupdate.timer` to + stop updating. Presets apply on first boot only, so nodes installed before + this preset need `systemctl preset systemd-sysupdate.timer + systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service` once. + +### Diskless and installer boots + +Diskless nodes and the USB installer have no slots. Their `10-diskless.conf` +drop-ins condition `systemd-sysupdate.service`, +`systemd-sysupdate-reboot.service` and both timers off when +`/run/machines/rootdisk.raw` exists or the command line has `root=tmpfs`. A +diskless node updates by rebooting into whatever its boot server serves. + +`bluefin-diskless-update-check.timer` (5 min after boot, then hourly; only +when `/run/machines/rootdisk.raw` exists) tells the node when that would be a +newer image. `/usr/libexec/bluefin-boot-origin`, which Ignition also uses, +finds the boot directory from the `StubDeviceURL` EFI variable, the explicit +http(s) `rd.systemd.pull=` source, or the `import.pull` credential. +`bluefin-diskless-update-check` then fetches `SHA256SUMS` and `SHA256SUMS.gpg` +from that directory, verifies them with `gpgv` against the image keyring, and +compares the `bluefin-server_.raw` entry with the running `IMAGE_VERSION` +(`systemd-analyze compare-versions`). When the release is newer it touches +`/run/reboot-required` and logs `Bluefin Server is available from `. +Network and signature errors are logged and leave the flag alone; the unit +never fails and never reboots. Kured reboots a diskless Kubernetes node on the +flag. Other nodes wait for an operator or fleet tooling to act on it +(`test -e /run/reboot-required`, +`journalctl -u bluefin-diskless-update-check`). The boot server (Booty) +decides what the next boot pulls. ## Ignition (opt-in) @@ -250,13 +298,15 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. | "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. | | "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. | | "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. | -| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. | +| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless, and `bluefin-diskless-update-check` only flags that a newer image is being served. | +| "The update timer should reboot Kubernetes nodes too." | Kubernetes nodes reboot through kured (drain first); the reboot unit's `ExecCondition=` stands down while kubelet or k0s runs. | ## Verification - [ ] `just validate` resolves the BuildStream graph without errors. - [ ] `just dogfood-check` passes (diskless boot, Secure Boot, no failed units). -- [ ] `just dogfood-install NEXT=` passes (install, disk boot, A/B update). +- [ ] `just dogfood-install NEXT=` passes (install, disk boot with both update timers enabled, A/B update through `systemd-sysupdate.service` with the kured flag and interlock, blessed after `boot-complete.target`). +- [ ] `just dogfood-check` reports `PROBE update-check=success` for the diskless update check. - [ ] UKIs pin `usrhash=` and are signed with DB; `sbverify` passes in `bluefin-server-boot.bst`. - [ ] No hardcoded device paths in any boot configuration. - [ ] No element named `bluefin-server-ddi` or `bluefin-server-installer` exists; the OS DDI comes from `oci/bluefin-server-image.bst`. diff --git a/docs/skills/systemd-sysupdate-verification.md b/docs/skills/systemd-sysupdate-verification.md index db982f08..60ee1e01 100644 --- a/docs/skills/systemd-sysupdate-verification.md +++ b/docs/skills/systemd-sysupdate-verification.md @@ -4,7 +4,7 @@ description: Configure and operate GPG signature verification for Bluefin Server metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd --- @@ -65,6 +65,14 @@ Only the k0s sysext stays a separate component (`files/os/sysupdate.k0s.d/`, `systemd-sysupdate --component=k0s update`) with its own version axis. Diskless nodes update by rebooting into a newer image; `systemd-sysupdate.service` is disabled when booted diskless. +Update scheduling, the kured flag, and the boot health gate are covered in +[ddi-installer.md](ddi-installer.md) under "Updates". + +The diskless update check (`bluefin-diskless-update-check`) uses the same trust +root: it trusts a newer release only after `gpgv` verifies the boot server's +`SHA256SUMS.gpg` against the keyring systemd reads +(`/etc/systemd/import-pubring.pgp`, else the vendor keyring). An unsigned or +foreign-signed manifest never sets `/run/reboot-required`. ## Signing happens inside the image build From 827e4412a1cdfb9b5cebd31cda3f6f95aba0ff02 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 07/17] docs(skills): point Kubernetes sysext operators at kured for OS reboots While k0s or kubelet runs, the base image's reboot timer stands down. Say so where operators of those sysexts look, linking to ddi-installer.md. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/k0s-sysext-ops.md | 6 +++++- docs/skills/kubeadm-sysext.md | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 74c871d0..7c792b27 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s and KubeStellar systemd-sysext extensi metadata: type: how-to status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd --- @@ -43,6 +43,10 @@ The unit copies `/var/lib/k0s/k0s.raw` to `/run/extensions/k0s.raw`, runs `systemd-sysext refresh`, and then enables either `k0scontroller.service` or `k0sworker.service` depending on whether `/etc/k0s/token` exists. +While either k0s unit runs, the base image's automatic reboot stands down; +deploy kured to roll staged OS updates across the cluster (see "Updates" in +[ddi-installer.md](ddi-installer.md)). + ### Worker nodes A node with a join token at `/etc/k0s/token` becomes a worker. The token can be diff --git a/docs/skills/kubeadm-sysext.md b/docs/skills/kubeadm-sysext.md index 4a70f2a4..7ac3b619 100644 --- a/docs/skills/kubeadm-sysext.md +++ b/docs/skills/kubeadm-sysext.md @@ -4,7 +4,7 @@ description: Build, ship and operate the opt-in kubeadm worker systemd-sysext (k metadata: type: how-to status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" --- # kubeadm worker sysext @@ -63,6 +63,9 @@ the cluster CNI (Cilium's `cilium-cni`). containerd searches `--volume-plugin-dir=/var/lib/kubelet/volumeplugins` because `/usr` is read-only. - `/etc/resolv.conf` links to systemd-resolved's `/run/systemd/resolve/resolv.conf` (base image), the path kubelet's `resolvConf` expects. +- While `kubelet.service` runs or restarts, the base image's automatic reboot + stands down and reboots belong to kured; see "Updates" in + [ddi-installer.md](ddi-installer.md). ## Host tools From 930ab2ea619be9119f5954125972c2e56814cb8a Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:01:01 -0500 Subject: [PATCH 08/17] docs: mark reboot coordination done; a fleet-wide reboot lock remains Single hosts now reboot into updates in a nightly window and Kubernetes nodes go through kured, so issue #139's gap is closed. What remains is a FleetLock-style lock for non-Kubernetes fleets. Update the readiness audit, the roadmap, the distro gap analysis and the README. Closes #139 Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 4 ++-- docs/MVP_1_0_READINESS.md | 6 ++++-- docs/skills/architecture-roadmap.md | 5 +++-- docs/skills/gap-analysis-distros.md | 16 ++++++++++------ 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 6d593ab5..85276144 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ Bluefin Server is currently in **Alpha**: ## What it is -- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image. -- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot, and UKI boot counting rolls back a failed update automatically. +- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image, and flags `/run/reboot-required` when its boot server offers one. +- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot on a timer and reboots into it nightly (Kubernetes nodes leave the reboot to kured), and UKI boot counting rolls back an update that does not boot cleanly (any failed unit counts). - **Secure Boot on** — signed systemd-boot, signed UKIs, signed kernel modules, `lockdown=integrity`. UEFI HTTP boot is supported; [Booty](https://github.com/jeefy/booty) serves the UKI, the OS DDI, the signed manifest, and a per-node `bluefin-node.ign`. - **Opt-in per-node state via Ignition** — pass an `ignition.config` / `ignition.config.url` system credential (or, on UEFI HTTP boot, a `bluefin-node.ign` next to the UKI) and Ignition runs in the initrd on every boot; configs must be idempotent. - **Opt-in sysexts** — k0s (Kubernetes), KubeStellar, and OpenZFS ship as separate `systemd-sysext` images, never in the base `/usr`. ZFS and KubeStellar are version-locked to the image and follow OS updates through optional sysupdate features. diff --git a/docs/MVP_1_0_READINESS.md b/docs/MVP_1_0_READINESS.md index ea4d131a..f2c71a95 100644 --- a/docs/MVP_1_0_READINESS.md +++ b/docs/MVP_1_0_READINESS.md @@ -22,6 +22,7 @@ This audit tracks the gap between the current tree and a first public/usable MVP | A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots; proven by `scripts/dogfood-install.sh ` | | Read-only /usr | ✅ | erofs + dm-verity pinned by `usrhash=` in the signed UKIs (`elements/oci/bluefin-server-usr.bst`, `elements/oci/bluefin-server-boot.bst`) | | Signed diskless pull | ✅ | Netboot UKI pulls with `verify=signature`; the initrd ships gnupg and the keyring (`/etc/systemd/import-pubring.pgp` from `os-sysupdate-keys.bst`) | +| Automatic updates | ✅ | Installed nodes: `80-bluefin-updates.preset` enables `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`; Kubernetes nodes set `/run/reboot-required` and leave the reboot to kured (`ExecCondition=` interlock); `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so only a boot with no failed unit is blessed. Diskless nodes: `bluefin-diskless-update-check` flags a newer signed release on the boot server. See [ddi-installer.md](skills/ddi-installer.md) "Updates" | | Sysext lock-step updates | ✅ | `zfs` / `kubestellar` sysupdate features download the version-locked sysext with each OS update; rollback keeps the matching sysext (verified in the 6-phase `dogfood-install` run) | | First-boot SSH keys | ✅ | Ignition (`ignition.config` / `ignition.config.url` credentials, or `bluefin-node.ign` next to the UKI on UEFI HTTP boot; `tests/fixtures/ignition/var-on-disk.ign`) and `tmpfiles.extra` / sysusers credentials (`elements/bluefin-server/os-creds-prov.bst`) | @@ -29,7 +30,7 @@ Competitor context: [gap-analysis-distros.md](skills/gap-analysis-distros.md) ## Verdict -**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Remaining work is hardening: TPM2-sealed state, reboot coordination outside Kubernetes, merging the Booty branch, and real-hardware boot proofs. +**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Installed nodes update and reboot on their own, with a Kubernetes interlock and a boot health gate. Remaining work is hardening: TPM2-sealed state, a cluster-wide reboot lock for non-Kubernetes fleets, merging the Booty branch, and real-hardware boot proofs. ## Roadmap @@ -60,7 +61,8 @@ Priority order. Each item depends on the ones above it. ### Phase D: release discipline - [ ] TPM2-sealed /var and credential decryption proof on real hardware. -- [ ] Reboot coordination for non-Kubernetes hosts. +- [x] Reboot coordination for non-Kubernetes hosts: nightly `systemd-sysupdate-reboot.timer`, a kured interlock on Kubernetes nodes, and a boot health gate (#139). +- [ ] Cluster-wide reboot lock (FleetLock-style) for multi-node fleets without Kubernetes. - [ ] Merge the Booty `feat/bluefin-http-boot` branch; shim-signed Secure Boot path for enrollment-free first boots. - [ ] Tag `v1.0.0-MVP` once Phase D items land. - [ ] Publish release notes: verified boot path, trust model, known gaps. diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index 66ac9124..c72c21a0 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -4,7 +4,7 @@ description: Roadmap for future Bluefin Server architecture work. Use when plann metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd --- @@ -30,6 +30,7 @@ For reference, so future planning does not redo them: - OCI artifact output (`ghcr.io//bluefin-server:,latest`) alongside the raw release files, via ORAS in CI and `just publish-oci` locally. - Booty HTTP boot: per-MAC serving of the UKI, DDI, `SHA256SUMS(.gpg)`, and per-host `bluefin-node.ign`, verified end to end in QEMU with Secure Boot; the Booty branch is not yet merged upstream. - ZFS and KubeStellar sysexts version-locked to the image, delivered in lock-step with OS updates through sysupdate features; rollback keeps the matching sysext. +- Automatic updates on installed nodes (preset-enabled `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`), a kured interlock for Kubernetes nodes, a boot health gate (`systemd-boot-check-no-failures.service`), and a signed update signal for diskless nodes (`bluefin-diskless-update-check`); see [ddi-installer.md](ddi-installer.md) "Updates". ## Planned work @@ -41,7 +42,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md). | 2 | aarch64 build axis | `project.conf` and `include/arch.yml` already model it; no CI coverage yet. | | 3 | Booty merge and its Secure Boot shim story | The Bluefin HTTP-boot support works from [Booty](https://github.com/jeefy/booty) `feat/bluefin-http-boot`; it still needs to merge, and enrollment-free first boots need a shim-signed path. | | 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. | -| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. | +| 5 | Cluster-wide reboot lock for non-Kubernetes fleets | Single hosts reboot in the nightly window and Kubernetes nodes use kured; several non-Kubernetes hosts sharing a service still have no FleetLock/locksmith-style lock, so they may reboot in the same window. | ## Status notes diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 66a78e4a..7a949719 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -6,7 +6,7 @@ description: | metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" --- # Gap Analysis: Bluefin Server versus Comparable Server OSes @@ -98,10 +98,10 @@ kernel). No other distribution's binaries ship in the image. |------|---------------------------------| | **Philosophy** | Systemd-native, minimal, image-based server OS; diskless-first (a node boots from a network-pulled image in RAM and updates by rebooting), with an optional disk install. Base /usr includes bash for login and bring-up while heavy developer/debug tools live in sysexts or system containers; container workloads and Kubernetes run via opt-in sysexts. Sources: [AGENTS.md](../../AGENTS.md), [factory-integration.md](factory-integration.md). | | **State model** | `/usr` is a read-only erofs filesystem verified by dm-verity, pinned by `usrhash=` on the locked UKI command line. `/etc` is empty on every boot and populated from `/usr/share/factory/etc` by systemd-tmpfiles. Diskless nodes run from tmpfs. Installed nodes get a persistent xfs root via systemd gpt-auto discovery. Sources: [bluefin-server-usr.bst](../../elements/oci/bluefin-server-usr.bst), [bluefin-server-boot.bst](../../elements/oci/bluefin-server-boot.bst), [50-root.conf](../../files/os/repart.d/50-root.conf). | -| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless). Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | +| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. `systemd-sysupdate.timer` is enabled by preset, and `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so an update is blessed only when no unit failed. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless); `bluefin-diskless-update-check` flags `/run/reboot-required` when the boot server offers a newer signed release. Sources: [ddi-installer.md](ddi-installer.md), [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | | **Provisioning** | Stock `systemd-sysinstall` copies `/usr` onto a target disk, started either from the offline USB installer (`bluefin-server-installer_.raw`) or on a diskless-booted node (see [ddi-installer.md](ddi-installer.md)). Per-node configuration is opt-in via Ignition, delivered as `ignition.config` / `ignition.config.url` system credentials (the cmdline is locked inside the signed UKI); Ignition runs on every boot, so configs must be idempotent. First-boot systemd credentials also cover root password, `tmpfiles.extra`, `network.*`, and `firstboot.*`. Sources: [bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst), [initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md). | | **Customization** | Adds software through opt-in `systemd-sysext` images (overlay `/usr`): k0s (Kubernetes) and OpenZFS are built in-tree. The base OS `os-release` identifies as `ID=bluefin-server`; the ZFS sysext pins `VERSION_ID` to the image version because its kernel modules are built against the exact FSDK kernel. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release.bst](../../elements/bluefin-server/os-release.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStopPost` that touches `/run/reboot-required` once an update is pending. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-kubernetes.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf), [Kured project](https://github.com/kubereboot/kured). | ## 4. Factual Gaps @@ -121,7 +121,7 @@ kernel). No other distribution's binaries ship in the image. ### Update delivery -- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook. Diskless nodes need no update service at all (reboot is the update), so this only matters for installed nodes. +- **Resolved:** installed nodes update and reboot on their own (`80-bluefin-updates.preset`), a boot is blessed only when no unit failed, and diskless nodes get a signed update signal (`bluefin-diskless-update-check`). - **Resolved:** the diskless pull of the OS DDI runs with `verify=signature`; the initrd ships gnupg and the image keyring, so the download is checked against the signed `SHA256SUMS` before the pinned `usrhash=` / dm-verity checks even begin. ### Customization @@ -130,11 +130,12 @@ kernel). No other distribution's binaries ship in the image. ### Reboot coordination -- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. +- **Resolved:** single-node and non-Kubernetes hosts reboot in the nightly `systemd-sysupdate-reboot.timer` window; Kubernetes nodes stand down for Kured. +- **Gap:** there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`, so non-Kubernetes hosts that serve together may reboot in the same window. ## 5. Summary of Biggest Gaps -1. **Reboot coordination outside Kubernetes is missing.** No FleetLock/locksmith equivalent for single-node or non-Kubernetes hosts. +1. **No cluster-wide reboot lock outside Kubernetes.** Hosts reboot in a nightly window; there is no FleetLock/locksmith equivalent for non-Kubernetes fleets. 2. **Credential provisioning hardware proof is incomplete.** SSH keys, Ignition configs, network files, and firstboot settings are wired through systemd credentials; TPM2-sealed decryption still needs a hardware boot proof. The earlier gap on the unverified diskless DDI download is closed: the pull @@ -199,5 +200,8 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/sysupdate.d/20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer) - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) +- [files/os/systemd/system-preset/80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset) - [files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf) +- [files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf) +- [files/os/update-check/usr/libexec/bluefin-diskless-update-check](../../files/os/update-check/usr/libexec/bluefin-diskless-update-check) - [files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) From 6d2d3a6b1ad976b0b3750f144eeed2c1525c42b8 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:07 -0500 Subject: [PATCH 09/17] feat(update): hold the nightly reboot with operator lock files Fold #182's operator inhibition into this PR's reboot drop-in so the reboot service keeps one interlock instead of two from different directories: systemd-sysupdate-reboot.service is skipped while /run/reboot-lock (until the next boot) or /etc/reboot-lock (until removed) exists. The drop-in is renamed 20-interlock.conf because it now carries more than the Kubernetes check. Lock files from projectbluefin/server#182 by Bob Killen. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/ddi-installer.md | 2 +- docs/skills/gap-analysis-distros.md | 4 +-- .../{20-kubernetes.conf => 20-interlock.conf} | 9 ++++++ tests/unit/test_automatic_updates.py | 28 ++++++++++++++++--- 4 files changed, 36 insertions(+), 7 deletions(-) rename files/os/systemd/system/systemd-sysupdate-reboot.service.d/{20-kubernetes.conf => 20-interlock.conf} (64%) diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index 7af0b7c8..d13b3ef9 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -214,7 +214,7 @@ only when a newer version than the booted one is installed. [kured](https://kured.dev/) (cordon, drain, one node at a time). `systemd-sysupdate.service.d/20-kured.conf` touches `/run/reboot-required` after each run once `systemd-sysupdate pending` reports an installed update. - `systemd-sysupdate-reboot.service.d/20-kubernetes.conf` skips the local + `systemd-sysupdate-reboot.service.d/20-interlock.conf` skips the local reboot (`ExecCondition=`) while `kubelet.service`, `k0scontroller.service` or `k0sworker.service` is running, the interlock proposed in [#182](https://github.com/projectbluefin/server/pull/182). Without kured, a diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 7a949719..8254056b 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -101,7 +101,7 @@ kernel). No other distribution's binaries ship in the image. | **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. `systemd-sysupdate.timer` is enabled by preset, and `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so an update is blessed only when no unit failed. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless); `bluefin-diskless-update-check` flags `/run/reboot-required` when the boot server offers a newer signed release. Sources: [ddi-installer.md](ddi-installer.md), [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | | **Provisioning** | Stock `systemd-sysinstall` copies `/usr` onto a target disk, started either from the offline USB installer (`bluefin-server-installer_.raw`) or on a diskless-booted node (see [ddi-installer.md](ddi-installer.md)). Per-node configuration is opt-in via Ignition, delivered as `ignition.config` / `ignition.config.url` system credentials (the cmdline is locked inside the signed UKI); Ignition runs on every boot, so configs must be idempotent. First-boot systemd credentials also cover root password, `tmpfiles.extra`, `network.*`, and `firstboot.*`. Sources: [bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst), [initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md). | | **Customization** | Adds software through opt-in `systemd-sysext` images (overlay `/usr`): k0s (Kubernetes) and OpenZFS are built in-tree. The base OS `os-release` identifies as `ID=bluefin-server`; the ZFS sysext pins `VERSION_ID` to the image version because its kernel modules are built against the exact FSDK kernel. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release.bst](../../elements/bluefin-server/os-release.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-kubernetes.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf), [Kured project](https://github.com/kubereboot/kured). | +| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf), [Kured project](https://github.com/kubereboot/kured). | ## 4. Factual Gaps @@ -202,6 +202,6 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) - [files/os/systemd/system-preset/80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset) - [files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf) -- [files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf) +- [files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf) - [files/os/update-check/usr/libexec/bluefin-diskless-update-check](../../files/os/update-check/usr/libexec/bluefin-diskless-update-check) - [files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf similarity index 64% rename from files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf rename to files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf index 9ab630c5..b27f20cf 100644 --- a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-kubernetes.conf +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf @@ -1,3 +1,12 @@ +[Unit] +# Operator hold: while either file exists the nightly reboot is skipped +# (a failed condition skips the unit, it does not fail it). /run/reboot-lock +# lasts until the next boot, /etc/reboot-lock until it is removed. Kured has +# its own lock and does not read them. From projectbluefin/server#182 by Bob +# Killen. +ConditionPathExists=!/run/reboot-lock +ConditionPathExists=!/etc/reboot-lock + [Service] # Kubernetes nodes reboot through kured (cordon, drain, one node at a time), # which systemd-sysupdate.service.d/20-kured.conf triggers. While kubelet diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py index 26735f8f..55733a9a 100644 --- a/tests/unit/test_automatic_updates.py +++ b/tests/unit/test_automatic_updates.py @@ -2,8 +2,9 @@ Installed nodes run systemd-sysupdate.timer and systemd-sysupdate-reboot.timer by vendor preset; Kubernetes nodes leave the reboot to kured; diskless and USB -installer boots keep every update unit inert; boot-complete.target requires -that no unit failed before systemd-bless-boot marks a counted UKI good. +installer boots keep every update unit inert; operator lock files hold the +local reboot; boot-complete.target requires that no unit failed before +systemd-bless-boot marks a counted UKI good. """ from __future__ import annotations @@ -26,7 +27,7 @@ ELEMENTS = ROOT / "elements" / "bluefin-server" DISKLESS = UNITS / "systemd-sysupdate.service.d" / "10-diskless.conf" KURED = UNITS / "systemd-sysupdate.service.d" / "20-kured.conf" -INTERLOCK = UNITS / "systemd-sysupdate-reboot.service.d" / "20-kubernetes.conf" +INTERLOCK = UNITS / "systemd-sysupdate-reboot.service.d" / "20-interlock.conf" DISKLESS_ONLY = "/run/machines/rootdisk.raw" @@ -37,6 +38,15 @@ def ini(path: Path) -> configparser.ConfigParser: return parser +def values(path: Path, key: str) -> list[str]: + """Every value of in order; configparser keeps only the last one.""" + return [ + line.split("=", 1)[1] + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith(f"{key}=") + ] + + def conditions(path: Path) -> list[str]: return [ line.strip() @@ -106,8 +116,18 @@ def test_kured_flag_is_set_only_when_an_update_is_pending() -> None: assert "ExecStart" not in keys, dropin +def test_reboot_is_held_by_the_operator_lock_files() -> None: + # From projectbluefin/server#182: a failed condition skips the unit. + assert conditions(INTERLOCK) == [ + "ConditionPathExists=!/run/reboot-lock", + "ConditionPathExists=!/etc/reboot-lock", + ] + # One interlock drop-in: #182's reboot-coordination.conf must not come back. + assert sorted(p.name for p in INTERLOCK.parent.iterdir()) == ["10-diskless.conf", "20-interlock.conf"] + + def _interlock_script() -> str: - cmd = ini(INTERLOCK)["Service"]["ExecCondition"] + cmd = values(INTERLOCK, "ExecCondition")[-1] argv = shlex.split(cmd.replace("$$", "$")) assert argv[:2] == ["/usr/bin/sh", "-c"] and len(argv) == 3, argv return argv[2] From 901e6b34548434e8270d7d9e25f33dbaf2eda6b3 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:27 -0500 Subject: [PATCH 10/17] fix(update): reboot a boot-counted boot that misses boot-complete.target The health gate stopped the bless but nothing rebooted: systemd-boot only moves on at the next boot, and systemd-sysupdate-reboot does nothing once the newest version is the booted one. A node that came up with a failed unit stayed on the unblessed image. bluefin-boot-deadline.timer (preset-enabled) runs only on boot-counted boots: LoaderBootCountPath exists, not diskless, not root=tmpfs. 15 minutes after boot (OnBootSec=, a drop-in changes it), if systemd-bless-boot still says indeterminate or bad and boot-complete.target is not active, bluefin-boot-deadline logs the failed units and reboots, so systemd-boot uses up the next try and falls back after the last. The last try boots "dirty" and reboots only when another UKI can still be booted, so there is no loop without a fallback. The service has DefaultDependencies=no and no ordering on multi-user or boot-complete, so a hung boot is caught too. Kubernetes: while kubelet or k0s is starting, running or stopping, the node may carry workloads, so it touches /run/reboot-required and kured drains and reboots it in its lock, as for an update. A stopped or failed Kubernetes unit leaves the node NotReady with nothing to drain and no kured, so it reboots directly. /run/reboot-lock and /etc/reboot-lock hold it like the nightly reboot. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../system-preset/80-bluefin-updates.preset | 5 +- .../system/bluefin-boot-deadline.service | 17 ++ .../system/bluefin-boot-deadline.timer | 21 ++ .../20-interlock.conf | 6 +- .../usr/libexec/bluefin-boot-deadline | 119 +++++++++++ tests/unit/test_automatic_updates.py | 1 + tests/unit/test_boot_deadline.py | 199 ++++++++++++++++++ 7 files changed, 364 insertions(+), 4 deletions(-) create mode 100644 files/os/systemd/system/bluefin-boot-deadline.service create mode 100644 files/os/systemd/system/bluefin-boot-deadline.timer create mode 100755 files/os/update-check/usr/libexec/bluefin-boot-deadline create mode 100644 tests/unit/test_boot_deadline.py diff --git a/files/os/systemd/system-preset/80-bluefin-updates.preset b/files/os/systemd/system-preset/80-bluefin-updates.preset index 27ea26bf..1bcf503c 100644 --- a/files/os/systemd/system-preset/80-bluefin-updates.preset +++ b/files/os/systemd/system-preset/80-bluefin-updates.preset @@ -5,8 +5,11 @@ enable systemd-sysupdate.timer enable systemd-sysupdate-reboot.timer # Boot health gate: boot-complete.target, and with it systemd-bless-boot -# marking a boot-counted UKI good, requires that no unit has failed. +# marking a boot-counted UKI good, requires that no unit has failed. A counted +# boot that has not got there by the deadline reboots (or, on a Kubernetes +# node, flags kured), so systemd-boot falls back after the last try. enable systemd-boot-check-no-failures.service +enable bluefin-boot-deadline.timer # Diskless nodes flag /run/reboot-required when their boot server offers a # newer release; the timer is conditioned off everywhere else. enable bluefin-diskless-update-check.timer diff --git a/files/os/systemd/system/bluefin-boot-deadline.service b/files/os/systemd/system/bluefin-boot-deadline.service new file mode 100644 index 00000000..dfba629e --- /dev/null +++ b/files/os/systemd/system/bluefin-boot-deadline.service @@ -0,0 +1,17 @@ +[Unit] +Description=Reboot a Boot-Counted Boot That Missed boot-complete.target +Documentation=file:/usr/libexec/bluefin-boot-deadline man:systemd-bless-boot.service(8) +ConditionPathExists=/sys/firmware/efi/efivars/LoaderBootCountPath-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f +ConditionPathExists=!/run/machines/rootdisk.raw +ConditionKernelCommandLine=!root=tmpfs +# Not ordered after basic.target, multi-user.target or boot-complete.target: +# the deadline must still fire when the boot hangs before any of them. +DefaultDependencies=no +Requires=sysinit.target +After=sysinit.target +Conflicts=shutdown.target +Before=shutdown.target + +[Service] +Type=oneshot +ExecStart=/usr/libexec/bluefin-boot-deadline diff --git a/files/os/systemd/system/bluefin-boot-deadline.timer b/files/os/systemd/system/bluefin-boot-deadline.timer new file mode 100644 index 00000000..cbd3a0b2 --- /dev/null +++ b/files/os/systemd/system/bluefin-boot-deadline.timer @@ -0,0 +1,21 @@ +[Unit] +Description=Deadline for a Boot-Counted Boot to Reach boot-complete.target +Documentation=file:/usr/libexec/bluefin-boot-deadline man:systemd-bless-boot.service(8) +# Boot-counted boots only: systemd-boot sets LoaderBootCountPath when the entry +# it booted carries a tries counter, i.e. a UKI systemd-sysupdate installed and +# systemd-bless-boot has not marked good yet. Never on diskless or USB +# installer boots. +ConditionPathExists=/sys/firmware/efi/efivars/LoaderBootCountPath-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f +ConditionPathExists=!/run/machines/rootdisk.raw +ConditionKernelCommandLine=!root=tmpfs + +[Timer] +# Change it with a drop-in; clear the list first, e.g. +# [Timer] +# OnBootSec= +# OnBootSec=30min +OnBootSec=15min +AccuracySec=10s + +[Install] +WantedBy=timers.target diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf index b27f20cf..f9c57672 100644 --- a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf @@ -1,9 +1,9 @@ [Unit] # Operator hold: while either file exists the nightly reboot is skipped # (a failed condition skips the unit, it does not fail it). /run/reboot-lock -# lasts until the next boot, /etc/reboot-lock until it is removed. Kured has -# its own lock and does not read them. From projectbluefin/server#182 by Bob -# Killen. +# lasts until the next boot, /etc/reboot-lock until it is removed. The same +# files hold bluefin-boot-deadline.service. Kured has its own lock and does +# not read them. From projectbluefin/server#182 by Bob Killen. ConditionPathExists=!/run/reboot-lock ConditionPathExists=!/etc/reboot-lock diff --git a/files/os/update-check/usr/libexec/bluefin-boot-deadline b/files/os/update-check/usr/libexec/bluefin-boot-deadline new file mode 100755 index 00000000..794e669c --- /dev/null +++ b/files/os/update-check/usr/libexec/bluefin-boot-deadline @@ -0,0 +1,119 @@ +#!/usr/bin/bash +# bluefin-boot-deadline — reboot a boot-counted boot that was not blessed in time. +# +# systemd-sysupdate installs a new UKI with a tries counter +# (bluefin-server-+3-0.efi). systemd-boot uses up one try each time it +# boots the entry, and systemd-bless-boot drops the counter once +# boot-complete.target is reached, which systemd-boot-check-no-failures.service +# allows only when no unit failed. A boot that never gets there keeps its +# counter, but systemd-boot only moves on at the next boot, and nothing else +# reboots a node that is already running the newest installed version. +# +# bluefin-boot-deadline.timer runs this once the deadline passes (OnBootSec=, +# 15 min unless a drop-in changes it), on boot-counted boots only. If the boot +# is still not blessed: +# +# - /run/reboot-lock or /etc/reboot-lock: the operator holds reboots (the +# same files stop systemd-sysupdate-reboot.service); log and stay up; +# - kubelet or k0s running: the node may carry workloads, so do what the +# update path does and touch /run/reboot-required for kured to cordon, +# drain and reboot it within its cluster lock; +# - otherwise reboot, so systemd-boot uses up the next try and, after the +# last one, falls back to the previous UKI. A Kubernetes unit that is +# stopped or failed leaves the node NotReady with nothing to drain, and +# kured cannot run there either, so that node reboots directly. +# +# The last try boots as "dirty" (no tries left). It reboots only when another +# UKI still has tries or no counter, so a node with nothing to fall back to +# does not reboot in a loop. +# +# Journal priorities come from SyslogLevelPrefix: <3> err, <4> warning, +# <6> info. +# +# Test overrides: BLUEFIN_BLESS_BOOT, BLUEFIN_BOOT_PATH, BLUEFIN_OS_RELEASE, +# BLUEFIN_REBOOT_LOCKS (space-separated), BLUEFIN_REBOOT_SENTINEL. + +set -euo pipefail +export LC_ALL=C + +bless="${BLUEFIN_BLESS_BOOT:-/usr/lib/systemd/systemd-bless-boot}" +sentinel="${BLUEFIN_REBOOT_SENTINEL:-/run/reboot-required}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +read -r -a locks <<< "${BLUEFIN_REBOOT_LOCKS:-/run/reboot-lock /etc/reboot-lock}" + +info() { printf '<6>%s\n' "$*"; } +warn() { printf '<4>%s\n' "$*"; } +err() { printf '<3>%s\n' "$*"; } + +# Another UKI systemd-boot would pick over this one: no counter, or tries left. +fallback() { + local boot uki + boot="${BLUEFIN_BOOT_PATH:-$(bootctl --print-boot-path 2>/dev/null || echo /boot)}" + for uki in "${boot}"/EFI/Linux/*.efi; do + [ -e "${uki}" ] || continue + case "${uki##*/}" in + *+0.efi|*+0-*.efi) ;; + *) return 0 ;; + esac + done + return 1 +} + +status="$("${bless}" status 2>/dev/null)" || status="" +case "${status}" in + indeterminate|bad) ;; + dirty) + if ! fallback; then + warn "boot entry has no tries left and no other UKI to fall back to; not rebooting" + exit 0 + fi ;; + *) + info "boot is ${status:-not boot-counted}; nothing to do" + exit 0 ;; +esac + +if systemctl is-active --quiet boot-complete.target; then + info "boot-complete.target reached; systemd-bless-boot handles the entry" + exit 0 +fi + +version="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + version="${value}" + done < "${os_release}" +fi + +failed=() +while read -r unit _; do + [ -n "${unit}" ] && failed+=("${unit}") +done < <(systemctl list-units --state=failed --plain --no-legend --no-pager 2>/dev/null || true) +if [ "${#failed[@]}" -gt 0 ]; then + why="failed: ${failed[*]}" +else + why="no unit failed, but the boot never got there" +fi +msg="Bluefin Server ${version:-(unknown version)} did not reach boot-complete.target before the boot deadline (${why}); its boot entry is ${status}" + +for lock in "${locks[@]}"; do + if [ -e "${lock}" ]; then + warn "${msg}; ${lock} holds reboots, not rebooting" + exit 0 + fi +done + +for u in kubelet.service k0scontroller.service k0sworker.service; do + s="$(systemctl is-active "${u}" 2>/dev/null)" || true + case "${s}" in + active|activating|deactivating|reloading|refreshing) + mkdir -p "${sentinel%/*}" + touch "${sentinel}" + warn "${msg}; ${u} is ${s}: flagged ${sentinel} for kured to drain and reboot" + exit 0 ;; + esac +done + +err "${msg}; rebooting so systemd-boot falls back after the last try" +exec systemctl --no-block reboot diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py index 55733a9a..1ea4cb95 100644 --- a/tests/unit/test_automatic_updates.py +++ b/tests/unit/test_automatic_updates.py @@ -68,6 +68,7 @@ def test_preset_enables_the_update_timers_and_the_health_gate() -> None: ["enable", "systemd-sysupdate.timer"], ["enable", "systemd-sysupdate-reboot.timer"], ["enable", "systemd-boot-check-no-failures.service"], + ["enable", "bluefin-boot-deadline.timer"], ["enable", "bluefin-diskless-update-check.timer"], ] diff --git a/tests/unit/test_boot_deadline.py b/tests/unit/test_boot_deadline.py new file mode 100644 index 00000000..381e4306 --- /dev/null +++ b/tests/unit/test_boot_deadline.py @@ -0,0 +1,199 @@ +"""Executed coverage for the rollback half of the boot health gate. + +bluefin-boot-deadline reboots a boot-counted boot that has not reached +boot-complete.target by its deadline, so systemd-boot uses up a try and falls +back. It runs here against fake systemd tools. +""" + +from __future__ import annotations + +import configparser +import os +import stat +import subprocess +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +LIBEXEC = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" +DEADLINE = LIBEXEC / "bluefin-boot-deadline" +COUNTED = "/sys/firmware/efi/efivars/LoaderBootCountPath-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" +BOOTED = "26.09.3" + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def conditions(path: Path) -> list[str]: + return [line for line in path.read_text(encoding="utf-8").splitlines() if line.startswith("Condition")] + + +def executable(path: Path, body: str) -> None: + path.write_text("#!/bin/sh\n" + body, encoding="utf-8") + path.chmod(path.stat().st_mode | stat.S_IXUSR) + + +@pytest.mark.parametrize("unit", ["bluefin-boot-deadline.timer", "bluefin-boot-deadline.service"]) +def test_deadline_runs_only_on_boot_counted_disk_boots(unit: str) -> None: + assert conditions(UNITS / unit) == [ + f"ConditionPathExists={COUNTED}", + "ConditionPathExists=!/run/machines/rootdisk.raw", + "ConditionKernelCommandLine=!root=tmpfs", + ] + + +def test_deadline_timer_defaults_to_fifteen_minutes() -> None: + timer = ini(UNITS / "bluefin-boot-deadline.timer") + assert timer["Timer"]["OnBootSec"] == "15min" + assert timer["Install"]["WantedBy"] == "timers.target" + + +def test_deadline_fires_even_when_the_boot_hangs() -> None: + unit = ini(UNITS / "bluefin-boot-deadline.service")["Unit"] + assert unit["DefaultDependencies"] == "no" + assert unit["After"] == "sysinit.target" + for target in ("basic.target", "multi-user.target", "boot-complete.target"): + assert target not in unit.get("After", ""), target + assert target not in unit.get("Requires", ""), target + service = ini(UNITS / "bluefin-boot-deadline.service")["Service"] + assert service["ExecStart"] == "/usr/libexec/bluefin-boot-deadline" + + +def test_helpers_ship_in_the_os_and_are_executable_bash() -> None: + stack = yaml.safe_load((ROOT / "elements" / "bluefin-server" / "os-stack.bst").read_text(encoding="utf-8")) + assert "bluefin-server/os-update-check.bst" in stack["depends"] + for script in (DEADLINE,): + assert script.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert script.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(script)], check=True) + + +def deadline( + tmp_path: Path, + status: str, + *, + complete: bool = False, + failed: tuple[str, ...] = ("dogfood-broken.service",), + units: dict[str, str] | None = None, + ukis: tuple[str, ...] = (), + lock: bool = False, +): + bindir = tmp_path / "bin" + bindir.mkdir() + calls = tmp_path / "systemctl.calls" + units = units or {} + unit_cases = "".join(f" {u}) echo {s}; [ {s} = active ]; exit $? ;;\n" for u, s in units.items()) + failed_lines = "".join(f"echo '{u} loaded failed failed Test'\n" for u in failed) + executable( + bindir / "systemctl", + f'echo "$*" >> {calls}\n' + 'case "$1 $2" in\n' + f' "is-active --quiet") exit {0 if complete else 3} ;;\n' + ' "list-units --state=failed")\n' + f"{failed_lines}" + " exit 0 ;;\n" + ' "--no-block reboot") exit 0 ;;\n' + "esac\n" + 'if [ "$1" = is-active ]; then\n' + ' case "$2" in\n' + f"{unit_cases}" + " esac\n" + " echo inactive; exit 3\n" + "fi\n" + "exit 99\n", + ) + bless = tmp_path / "bless" + executable(bless, f'[ "$1" = status ] && echo {status}\n') + boot = tmp_path / "boot" + (boot / "EFI" / "Linux").mkdir(parents=True) + for uki in ukis: + (boot / "EFI" / "Linux" / uki).touch() + os_release = tmp_path / "os-release" + os_release.write_text(f'IMAGE_ID=bluefin-server\nIMAGE_VERSION="{BOOTED}"\n', encoding="utf-8") + lockfile = tmp_path / "etc-reboot-lock" + if lock: + lockfile.touch() + sentinel = tmp_path / "run" / "reboot-required" + env = dict( + os.environ, + PATH=f"{bindir}:{os.environ['PATH']}", + BLUEFIN_BLESS_BOOT=str(bless), + BLUEFIN_BOOT_PATH=str(boot), + BLUEFIN_OS_RELEASE=str(os_release), + BLUEFIN_REBOOT_LOCKS=f"{tmp_path / 'run-reboot-lock'} {lockfile}", + BLUEFIN_REBOOT_SENTINEL=str(sentinel), + ) + result = subprocess.run([str(DEADLINE)], capture_output=True, text=True, env=env, timeout=30) + assert result.returncode == 0, result.stdout + result.stderr + rebooted = "--no-block reboot" in (calls.read_text(encoding="utf-8") if calls.exists() else "") + return rebooted, sentinel.exists(), result.stdout + result.stderr + + +@pytest.mark.parametrize("status", ["indeterminate", "bad"]) +def test_unblessed_counted_boot_reboots(tmp_path: Path, status: str) -> None: + rebooted, flagged, log = deadline(tmp_path, status) + assert rebooted and not flagged, log + assert f"<3>Bluefin Server {BOOTED} did not reach boot-complete.target before the boot deadline" in log + assert "failed: dogfood-broken.service" in log + + +def test_hung_boot_with_no_failed_unit_reboots(tmp_path: Path) -> None: + rebooted, _, log = deadline(tmp_path, "indeterminate", failed=()) + assert rebooted, log + assert "no unit failed, but the boot never got there" in log + + +@pytest.mark.parametrize("status", ["clean", "good", ""]) +def test_blessed_or_uncounted_boot_never_reboots(tmp_path: Path, status: str) -> None: + rebooted, flagged, log = deadline(tmp_path, status) + assert not rebooted and not flagged, log + assert "nothing to do" in log + + +def test_boot_that_reached_boot_complete_never_reboots(tmp_path: Path) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", complete=True, failed=()) + assert not rebooted and not flagged, log + + +def test_operator_lock_holds_the_reboot(tmp_path: Path) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", lock=True) + assert not rebooted and not flagged, log + assert "holds reboots, not rebooting" in log + + +@pytest.mark.parametrize( + "unit,state", + [("kubelet.service", "active"), ("k0scontroller.service", "active"), ("k0sworker.service", "activating")], +) +def test_kubernetes_node_is_left_to_kured(tmp_path: Path, unit: str, state: str) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", units={unit: state}) + assert not rebooted and flagged, log + assert f"{unit} is {state}: flagged" in log + + +def test_node_whose_kubelet_failed_reboots_directly(tmp_path: Path) -> None: + # NotReady: nothing to drain, and kured's pod cannot run there. + rebooted, flagged, log = deadline( + tmp_path, "indeterminate", failed=("kubelet.service",), units={"kubelet.service": "failed"} + ) + assert rebooted and not flagged, log + + +def test_last_try_reboots_when_there_is_a_fallback(tmp_path: Path) -> None: + rebooted, _, log = deadline( + tmp_path, "dirty", ukis=("bluefin-server-26.09.4+0-3.efi", "bluefin-server-26.09.3.efi") + ) + assert rebooted, log + + +def test_last_try_without_a_fallback_does_not_loop(tmp_path: Path) -> None: + rebooted, _, log = deadline(tmp_path, "dirty", ukis=("bluefin-server-26.09.4+0-3.efi",)) + assert not rebooted, log + assert "no other UKI to fall back to" in log From 276daebdbfb1b84340864c05b52dde5236219984 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:45 -0500 Subject: [PATCH 11/17] fix(update): do not reboot back into an update that failed its boot tries After a rollback the failed version is still installed, and `systemd-sysupdate pending` still reports it: it does not look at the UKI's tries counter. The kured flag would be set after every update run and the nightly reboot would fire every night, each time landing on the previous version again. bluefin-update-pending wraps `systemd-sysupdate pending` and says no while a newer installed version's UKI has no tries left (bluefin-server-+0-.efi). It gates the kured flag (20-kured.conf) and the nightly reboot (an ExecCondition in 20-interlock.conf). The next newer release takes the failed version's slot and UKI (InstancesMax=2) and clears the block. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../20-interlock.conf | 3 + .../systemd-sysupdate.service.d/20-kured.conf | 5 +- .../usr/libexec/bluefin-update-pending | 50 ++++++++++ tests/unit/test_automatic_updates.py | 11 ++- tests/unit/test_update_pending.py | 93 +++++++++++++++++++ 5 files changed, 158 insertions(+), 4 deletions(-) create mode 100755 files/os/update-check/usr/libexec/bluefin-update-pending create mode 100644 tests/unit/test_update_pending.py diff --git a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf index f9c57672..15c4b653 100644 --- a/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf +++ b/files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf @@ -8,6 +8,9 @@ ConditionPathExists=!/run/reboot-lock ConditionPathExists=!/etc/reboot-lock [Service] +# Only reboot into an update that has not already failed its boot tries; +# after a rollback systemd-sysupdate still reports that version as pending. +ExecCondition=/usr/libexec/bluefin-update-pending # Kubernetes nodes reboot through kured (cordon, drain, one node at a time), # which systemd-sysupdate.service.d/20-kured.conf triggers. While kubelet # (kubeadm sysext) or k0s runs, the reboot timer stands down instead of diff --git a/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf b/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf index 2a790e8a..35af1299 100644 --- a/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf +++ b/files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf @@ -1,7 +1,8 @@ [Service] # Kured reboots Kubernetes nodes one at a time (cordon, drain, reboot) once # /run/reboot-required exists. Flag it only when a newer version than the -# booted one is installed: `systemd-sysupdate update` also succeeds when there +# booted one is installed and has not already failed its boot tries +# (bluefin-update-pending): `systemd-sysupdate update` also succeeds when there # was nothing to download. ExecStopPost, not ExecStartPost: the service is # Type=simple, so ExecStartPost would run as soon as the update started. -ExecStopPost=/usr/bin/sh -c 'if /usr/bin/systemd-sysupdate pending; then touch /run/reboot-required; fi' +ExecStopPost=/usr/bin/sh -c 'if /usr/libexec/bluefin-update-pending; then touch /run/reboot-required; fi' diff --git a/files/os/update-check/usr/libexec/bluefin-update-pending b/files/os/update-check/usr/libexec/bluefin-update-pending new file mode 100755 index 00000000..086493f8 --- /dev/null +++ b/files/os/update-check/usr/libexec/bluefin-update-pending @@ -0,0 +1,50 @@ +#!/usr/bin/bash +# bluefin-update-pending — is there a staged update worth rebooting into? +# +# Exit 0 when `systemd-sysupdate pending` reports a newer installed version +# than the booted one AND that version has not already failed its boot tries. +# Exit 1 otherwise. +# +# A version that failed its tries keeps its UKI as +# bluefin-server-+0-.efi: systemd-boot sorts it last and boots the +# previous one, so a reboot lands on the booted version again. systemd-sysupdate +# does not look at the counter and still reports it as pending, so without this +# check a node that rolled back would flag kured after every update run, or +# reboot in every nightly window, and never leave the version it is on. The +# next newer release takes the failed version's slot and UKI (InstancesMax=2) +# and clears the block. +# +# Used by systemd-sysupdate.service.d/20-kured.conf (the kured flag) and +# systemd-sysupdate-reboot.service.d/20-interlock.conf (the local reboot). +# +# Test overrides: BLUEFIN_SYSUPDATE, BLUEFIN_BOOT_PATH, BLUEFIN_OS_RELEASE. + +set -euo pipefail +export LC_ALL=C + +sysupdate="${BLUEFIN_SYSUPDATE:-/usr/bin/systemd-sysupdate}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +boot="${BLUEFIN_BOOT_PATH:-$(bootctl --print-boot-path 2>/dev/null || echo /boot)}" + +"${sysupdate}" pending || exit 1 + +booted="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + booted="${value}" + done < "${os_release}" +fi +[ -n "${booted}" ] || exit 0 + +for uki in "${boot}"/EFI/Linux/bluefin-server-*+0-*.efi "${boot}"/EFI/Linux/bluefin-server-*+0.efi; do + [ -e "${uki}" ] || continue + name="${uki##*/bluefin-server-}" + version="${name%%+*}" + if systemd-analyze compare-versions "${version}" gt "${booted}" >/dev/null; then + echo "Bluefin Server ${version} is installed but failed its boot tries (${uki##*/}); staying on ${booted} until a newer release replaces it" + exit 1 + fi +done +exit 0 diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py index 1ea4cb95..20fe7684 100644 --- a/tests/unit/test_automatic_updates.py +++ b/tests/unit/test_automatic_updates.py @@ -107,8 +107,9 @@ def test_kured_flag_is_set_only_when_an_update_is_pending() -> None: service = ini(KURED)["Service"] assert "ExecStartPost" not in service, "Type=simple: ExecStartPost runs before the update" cmd = service["ExecStopPost"] - assert "/usr/bin/systemd-sysupdate pending" in cmd - assert "touch /run/reboot-required" in cmd + # bluefin-update-pending wraps `systemd-sysupdate pending` and says no + # when that version already failed its boot tries (test_update_pending.py). + assert cmd == "/usr/bin/sh -c 'if /usr/libexec/bluefin-update-pending; then touch /run/reboot-required; fi'" for dropin in (UNITS / "systemd-sysupdate.service.d").iterdir(): keys = {key for section in ini(dropin).values() for key in section} assert "ExecStartPost" not in keys, dropin @@ -127,6 +128,12 @@ def test_reboot_is_held_by_the_operator_lock_files() -> None: assert sorted(p.name for p in INTERLOCK.parent.iterdir()) == ["10-diskless.conf", "20-interlock.conf"] +def test_reboot_skips_an_update_that_already_failed_its_tries() -> None: + conds = values(INTERLOCK, "ExecCondition") + assert len(conds) == 2 + assert conds[0] == "/usr/libexec/bluefin-update-pending" + + def _interlock_script() -> str: cmd = values(INTERLOCK, "ExecCondition")[-1] argv = shlex.split(cmd.replace("$$", "$")) diff --git a/tests/unit/test_update_pending.py b/tests/unit/test_update_pending.py new file mode 100644 index 00000000..060e2ccb --- /dev/null +++ b/tests/unit/test_update_pending.py @@ -0,0 +1,93 @@ +"""Executed coverage for bluefin-update-pending. + +After a rollback systemd-sysupdate still reports the failed version as +pending; bluefin-update-pending says no while that version's UKI has no tries +left, so the kured flag and the nightly reboot do not bring the node back to +it. It runs here against a fake systemd-sysupdate. +""" + +from __future__ import annotations + +import os +import shutil +import stat +import subprocess +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +PENDING = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" / "bluefin-update-pending" +BOOTED = "26.09.3" + + +def executable(path: Path, body: str) -> None: + path.write_text("#!/bin/sh\n" + body, encoding="utf-8") + path.chmod(path.stat().st_mode | stat.S_IXUSR) + + +def test_pending_is_executable_bash() -> None: + assert PENDING.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert PENDING.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(PENDING)], check=True) + + +def test_kured_flag_and_nightly_reboot_both_use_it() -> None: + kured = (UNITS / "systemd-sysupdate.service.d" / "20-kured.conf").read_text(encoding="utf-8") + interlock = (UNITS / "systemd-sysupdate-reboot.service.d" / "20-interlock.conf").read_text(encoding="utf-8") + assert "if /usr/libexec/bluefin-update-pending; then touch /run/reboot-required; fi" in kured + assert "systemd-sysupdate pending" not in kured + assert "ExecCondition=/usr/libexec/bluefin-update-pending\n" in interlock + + +needs_analyze = pytest.mark.skipif(not shutil.which("systemd-analyze"), reason="needs systemd-analyze") + + +def pending(tmp_path: Path, *, sysupdate_pending: bool, ukis: tuple[str, ...]) -> subprocess.CompletedProcess[str]: + sysupdate = tmp_path / "systemd-sysupdate" + executable(sysupdate, f'[ "$1" = pending ] && exit {0 if sysupdate_pending else 1}\nexit 99\n') + boot = tmp_path / "boot" + (boot / "EFI" / "Linux").mkdir(parents=True) + for uki in ukis: + (boot / "EFI" / "Linux" / uki).touch() + os_release = tmp_path / "os-release" + os_release.write_text(f'IMAGE_VERSION="{BOOTED}"\n', encoding="utf-8") + env = dict( + os.environ, + BLUEFIN_SYSUPDATE=str(sysupdate), + BLUEFIN_BOOT_PATH=str(boot), + BLUEFIN_OS_RELEASE=str(os_release), + ) + return subprocess.run([str(PENDING)], capture_output=True, text=True, env=env, timeout=30) + + +@needs_analyze +def test_pending_update_with_tries_left(tmp_path: Path) -> None: + result = pending(tmp_path, sysupdate_pending=True, ukis=("bluefin-server-26.09.4+2-1.efi", "bluefin-server-26.09.3.efi")) + assert result.returncode == 0, result.stdout + result.stderr + + +@needs_analyze +def test_nothing_pending(tmp_path: Path) -> None: + result = pending(tmp_path, sysupdate_pending=False, ukis=("bluefin-server-26.09.3.efi",)) + assert result.returncode == 1, result.stdout + result.stderr + + +@needs_analyze +@pytest.mark.parametrize("uki", ["bluefin-server-26.09.4+0-3.efi", "bluefin-server-26.09.10+0.efi"]) +def test_update_that_failed_its_tries_is_not_pending(tmp_path: Path, uki: str) -> None: + # After the rollback systemd-sysupdate still reports it as pending. + result = pending(tmp_path, sysupdate_pending=True, ukis=(uki, "bluefin-server-26.09.3.efi")) + assert result.returncode == 1, result.stdout + result.stderr + assert f"is installed but failed its boot tries ({uki}); staying on {BOOTED}" in result.stdout + + +@needs_analyze +def test_older_exhausted_uki_does_not_block_a_newer_update(tmp_path: Path) -> None: + result = pending( + tmp_path, + sysupdate_pending=True, + ukis=("bluefin-server-26.09.2+0-3.efi", "bluefin-server-26.09.3.efi", "bluefin-server-26.09.5+3-0.efi"), + ) + assert result.returncode == 0, result.stdout + result.stderr From ccfd0d5fad1a39226405050f13ff3690e8cc042d Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:03:47 -0500 Subject: [PATCH 12/17] fix(update): do not flag a diskless node pinned to a versioned image When the boot origin names a versioned file, bluefin-server_.raw from an explicit rd.systemd.pull= URL or the import.pull credential (or a versioned bluefin-server-netboot_.efi boot URL), the node pulls the same version again after a reboot. Flagging /run/reboot-required there made kured drain and reboot the node on every check while the server directory held a newer release. Flag only when the next boot resolves the version itself: a netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi via StubDeviceURL) or another unversioned name. A pinned node logs "Bluefin Server is available from / (running ), but this node is pinned to ; not flagging". The behavioural tests cover both flagging origins and the three pinned ones, and the signature and network-error cases now run against a flaggable origin so they still prove something. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../usr/libexec/bluefin-diskless-update-check | 26 ++++-- tests/unit/test_diskless_update_check.py | 87 +++++++++++++++---- 2 files changed, 90 insertions(+), 23 deletions(-) diff --git a/files/os/update-check/usr/libexec/bluefin-diskless-update-check b/files/os/update-check/usr/libexec/bluefin-diskless-update-check index ce734182..e70e36bd 100755 --- a/files/os/update-check/usr/libexec/bluefin-diskless-update-check +++ b/files/os/update-check/usr/libexec/bluefin-diskless-update-check @@ -13,8 +13,19 @@ # 3. read the release version from the manifest's bluefin-server_.raw # entry and compare it with the running IMAGE_VERSION using # `systemd-analyze compare-versions`, the ordering systemd-sysupdate uses; -# 4. if the release is newer, touch /run/reboot-required (the sentinel kured -# watches) and log the new version. +# 4. if the release is newer and the next boot would pick it up, touch +# /run/reboot-required (the sentinel kured watches) and log the new +# version. +# +# Whether the next boot picks it up depends on what the origin names. A +# netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi, +# found through StubDeviceURL) resolves the image at boot: the server swaps the +# UKI and the next boot pulls the newer image next to it. An origin that names +# a versioned file, bluefin-server_.raw (an explicit rd.systemd.pull= URL +# or the import.pull credential) or bluefin-server-netboot_.efi, is +# pinned: the node pulls the same version again after a reboot. Flagging it +# would make kured drain and reboot the node on every check, so a pinned node +# only logs that a newer release is available. # # It never reboots. Network and signature errors are logged and leave the # flag untouched, and the check always exits 0, so a boot server that cannot @@ -62,6 +73,9 @@ if ! origin="$("${boot_origin}")"; then exit 0 fi base="${origin%/*}" +file="${origin##*/}" +pinned="" +[[ "${file}" =~ ^bluefin-server(-netboot)?_[0-9A-Za-z.]+\.(raw|efi)$ ]] && pinned="${file}" tmp="$(mktemp -d)" trap 'rm -rf "${tmp}"' EXIT @@ -97,10 +111,12 @@ if [ -z "${offered}" ]; then exit 0 fi -if systemd-analyze compare-versions "${offered}" gt "${running}" >/dev/null; then +if ! systemd-analyze compare-versions "${offered}" gt "${running}" >/dev/null; then + info "up to date: ${base}/ offers ${offered}, running ${running}" +elif [ -n "${pinned}" ]; then + notice "Bluefin Server ${offered} is available from ${base}/ (running ${running}), but this node is pinned to ${pinned}; not flagging ${sentinel}, point its boot origin at the new release to update" +else mkdir -p "${sentinel%/*}" touch "${sentinel}" notice "Bluefin Server ${offered} is available from ${base}/ (running ${running}); flagged ${sentinel}, reboot to update" -else - info "up to date: ${base}/ offers ${offered}, running ${running}" fi diff --git a/tests/unit/test_diskless_update_check.py b/tests/unit/test_diskless_update_check.py index 24e90764..7cd264b8 100644 --- a/tests/unit/test_diskless_update_check.py +++ b/tests/unit/test_diskless_update_check.py @@ -2,7 +2,8 @@ bluefin-boot-origin finds the URL a node booted from; bluefin-diskless-update-check fetches the signed SHA256SUMS from that directory and flags /run/reboot-required -when it offers a newer release. Both run here against a local http.server and a +when it offers a newer release that the next boot would pull, i.e. unless the +origin names a versioned file (a pinned node only logs it). Both run here against a local http.server and a throwaway GnuPG key; no files/boot-keys are needed. """ @@ -104,14 +105,20 @@ def server(tmp_path: Path): httpd.server_close() -def publish(directory: Path, version: str, gnupghome: Path, *, tamper: bool = False, sign: bool = True) -> None: +def publish( + directory: Path, version: str, gnupghome: Path, *, tamper: bool = False, sign: bool = True, also: tuple[str, ...] = () +) -> None: names = [ - f"bluefin-server_{version}.raw", - f"bluefin-server_{version}_d3107d37-a9da-32cf-3c19-49fa3b0cb1df.usr.raw", - f"bluefin-server-{version}.efi", - f"bluefin-server-netboot_{version}.efi", - f"bluefin-server-installer_{version}.raw", - f"zfs_{version}.raw.zst", + name + for v in [*also, version] + for name in ( + f"bluefin-server_{v}.raw", + f"bluefin-server_{v}_d3107d37-a9da-32cf-3c19-49fa3b0cb1df.usr.raw", + f"bluefin-server-{v}.efi", + f"bluefin-server-netboot_{v}.efi", + f"bluefin-server-installer_{v}.raw", + f"zfs_{v}.raw.zst", + ) ] sums = "".join(f"{'%064x' % i} {name}\n" for i, name in enumerate(names)) (directory / "SHA256SUMS").write_text(sums, encoding="utf-8") @@ -124,11 +131,18 @@ def publish(directory: Path, version: str, gnupghome: Path, *, tamper: bool = Fa (directory / "SHA256SUMS").write_text(sums.replace(version, "99.99.99"), encoding="utf-8") -def check(tmp_path: Path, url: str | None, keyring: Path, *, via_cred: bool = False): +PINNED_RAW = f"bluefin-server_{RUNNING}.raw" +NETBOOT_UKI = "bluefin-server-netboot.efi" + + +def check(tmp_path: Path, url: str | None, keyring: Path, *, via: str = "stub", file: str = NETBOOT_UKI): + """Run the check with the boot origin / handed over : + "cmdline" (rd.systemd.pull=), "cred" (the import.pull credential) or + "stub" (the StubDeviceURL EFI variable of a UEFI HTTP boot).""" os_release = tmp_path / "os-release" os_release.write_text(f'ID=bluefin-server\nIMAGE_ID=bluefin-server\nIMAGE_VERSION="{RUNNING}"\n', encoding="utf-8") sentinel = tmp_path / "run" / "reboot-required" - pull = PULL.format(url=f"{url}/bluefin-server_{RUNNING}.raw") if url else "" + pull = PULL.format(url=f"{url}/{file}") if url else "" env = dict( os.environ, BLUEFIN_BOOT_ORIGIN=str(ORIGIN), @@ -139,27 +153,64 @@ def check(tmp_path: Path, url: str | None, keyring: Path, *, via_cred: bool = Fa BLUEFIN_CMDLINE=str(tmp_path / "cmdline"), ) env.pop("CREDENTIALS_DIRECTORY", None) - if via_cred: - (tmp_path / "cmdline").write_text("root=tmpfs\n", encoding="utf-8") + cmdline = "root=tmpfs" + if url and via == "cred": (tmp_path / "creds").mkdir() (tmp_path / "creds" / "import.pull").write_text(pull + "\n", encoding="utf-8") env["CREDENTIALS_DIRECTORY"] = str(tmp_path / "creds") - else: - (tmp_path / "cmdline").write_text(("rd.systemd.pull=" + pull if pull else "root=tmpfs") + "\n", encoding="utf-8") + elif url and via == "stub": + # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. + stub = tmp_path / "StubDeviceURL" + stub.write_bytes(b"\x06\x00\x00\x00" + f"{url}/{file}\0".encode("utf-16-le")) + env["BLUEFIN_STUB_URL_VAR"] = str(stub) + elif url: + cmdline = "rd.systemd.pull=" + pull + (tmp_path / "cmdline").write_text(cmdline + "\n", encoding="utf-8") result = subprocess.run([str(CHECK)], capture_output=True, text=True, env=env, timeout=120) assert result.returncode == 0, result.stdout + result.stderr return sentinel.exists(), result.stdout + result.stderr @needs_tools -@pytest.mark.parametrize("via_cred", [False, True]) -def test_newer_release_sets_the_flag(tmp_path: Path, server, keys, via_cred: bool) -> None: +@pytest.mark.parametrize( + "via,file", + [ + # Booty's HTTP boot URL: the served UKI changes, and the next boot + # pulls the image next to it. + ("stub", NETBOOT_UKI), + # An unversioned name the boot server repoints at each release. + ("cmdline", "bluefin-server.raw"), + ], +) +def test_newer_release_sets_the_flag(tmp_path: Path, server, keys, via: str, file: str) -> None: directory, url = server # 26.09.10 sorts after 26.09.2 as a version, not as a string. publish(directory, "26.09.10", keys["release"]) - flagged, log = check(tmp_path, url, keys["release-ring"], via_cred=via_cred) + flagged, log = check(tmp_path, url, keys["release-ring"], via=via, file=file) assert flagged, log - assert f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING})" in log + assert f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING}); flagged" in log + + +@needs_tools +@pytest.mark.parametrize( + "via,file", + [ + ("cmdline", PINNED_RAW), + ("cred", PINNED_RAW), + ("stub", f"bluefin-server-netboot_{RUNNING}.efi"), + ], +) +def test_newer_release_on_a_pinned_origin_sets_no_flag(tmp_path: Path, server, keys, via: str, file: str) -> None: + # The directory holds both releases; the origin names the running one, so + # a reboot would pull it again and kured would drain the node every check. + directory, url = server + publish(directory, "26.09.10", keys["release"], also=(RUNNING,)) + flagged, log = check(tmp_path, url, keys["release-ring"], via=via, file=file) + assert not flagged, log + assert ( + f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING}), " + f"but this node is pinned to {file}; not flagging" + ) in log @needs_tools From bee0d96219d36dc253b14cbc8a97310940a4ff93 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:03:58 -0500 Subject: [PATCH 13/17] chore(elements): describe the boot deadline and update-pending helpers os-update-check.bst now installs bluefin-boot-deadline and bluefin-update-pending next to the diskless update check, and os-k0s-first-boot.bst the deadline units and the reboot interlock. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- elements/bluefin-server/os-k0s-first-boot.bst | 3 ++- elements/bluefin-server/os-update-check.bst | 25 +++++++++++++------ 2 files changed, 19 insertions(+), 9 deletions(-) diff --git a/elements/bluefin-server/os-k0s-first-boot.bst b/elements/bluefin-server/os-k0s-first-boot.bst index a294bd70..4f361938 100644 --- a/elements/bluefin-server/os-k0s-first-boot.bst +++ b/elements/bluefin-server/os-k0s-first-boot.bst @@ -7,7 +7,8 @@ description: | run-bluefin-installer.mount for the USB installer), and the systemd-sysinstall.service drop-in the USB installer boots into, and the update units: systemd-sysupdate(-reboot) drop-ins (diskless conditions, the - kured flag, the Kubernetes reboot interlock) and the diskless update check. + kured flag, the reboot interlock: operator lock files, failed-update guard, + Kubernetes), the boot deadline and the diskless update check. sources: - kind: local diff --git a/elements/bluefin-server/os-update-check.bst b/elements/bluefin-server/os-update-check.bst index 73f303ad..73c1386f 100644 --- a/elements/bluefin-server/os-update-check.bst +++ b/elements/bluefin-server/os-update-check.bst @@ -1,13 +1,22 @@ kind: import description: | - /usr/libexec/bluefin-diskless-update-check, run by - bluefin-diskless-update-check.timer (files/os/systemd/system, enabled by - 80-bluefin-updates.preset) on diskless boots only: fetches the signed - SHA256SUMS from the directory the node booted from, and when it lists a - newer bluefin-server_.raw than the running IMAGE_VERSION, touches - /run/reboot-required for kured or the operator. It never reboots. - Runtime tools: bash, coreutils, curl, gpgv, systemd-analyze (os-base.bst) - and bluefin-boot-origin (boot-origin.bst). + Update helpers in /usr/libexec (units in files/os/systemd/system, enabled by + 80-bluefin-updates.preset): + - bluefin-diskless-update-check, run by bluefin-diskless-update-check.timer + on diskless boots only: fetches the signed SHA256SUMS from the directory + the node booted from, and when it lists a newer bluefin-server_.raw + than the running IMAGE_VERSION and the boot origin is not pinned to a + versioned file, touches /run/reboot-required for kured or the operator. + It never reboots. + - bluefin-boot-deadline, run by bluefin-boot-deadline.timer on boot-counted + boots only: reboots (or flags kured on a Kubernetes node) when the boot has + not reached boot-complete.target by the deadline, so systemd-boot falls + back to the previous UKI. + - bluefin-update-pending: `systemd-sysupdate pending`, minus a version whose + UKI already used up its boot tries; gates the kured flag and the nightly + reboot so a rolled-back node does not reboot into it again. + Runtime tools: bash, coreutils, curl, gpgv, bootctl, systemd-analyze + (os-base.bst) and bluefin-boot-origin (boot-origin.bst). sources: - kind: local From d78b17637980c26b24429a25657cd6356a80c08b Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:03:58 -0500 Subject: [PATCH 14/17] test(dogfood): prove the boot-deadline rollback of a failed unit in QEMU DOGFOOD_BROKEN=unit breaks with a unit that fails on that version only (ConditionOSRelease=IMAGE_VERSION=) instead of corrupting its slot, and shortens the boot deadline (DOGFOOD_DEADLINE, default 2min). Each counted boot reaches multi-user.target, misses boot-complete.target and waits for bluefin-boot-deadline to reboot it. The run asserts three such boots, the fallback to with the deadline timer inactive and the boot clean, and then no kured flag and a skipped systemd-sysupdate-reboot.service for the failed version. Every disk boot also reports the deadline timer's state. The default slot mode is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- scripts/dogfood-install.sh | 79 ++++++++++++++++++++++++++++++++++---- 1 file changed, 71 insertions(+), 8 deletions(-) diff --git a/scripts/dogfood-install.sh b/scripts/dogfood-install.sh index 1e9897bd..9f00b3a1 100644 --- a/scripts/dogfood-install.sh +++ b/scripts/dogfood-install.sh @@ -4,8 +4,15 @@ # 2. boot the installed disk (usr slot A, persistent root) # 3. with : systemd-sysupdate to that version over HTTP, reboot, # and confirm the node runs it from slot B with a boot-counted UKI -# 4. with : update to it, corrupt its slot, and confirm boot -# counting rolls the node back to on its own +# 4. with : update to it, break it, and confirm boot counting +# rolls the node back to on its own. DOGFOOD_BROKEN=slot +# (default) corrupts its usr slot, so the initrd fails; DOGFOOD_BROKEN=unit +# adds a unit that fails on that version only, so the boot reaches +# multi-user.target but never boot-complete.target, and +# bluefin-boot-deadline.timer (shortened to DOGFOOD_DEADLINE, default 2min) +# must reboot it until systemd-boot falls back. After the fallback the +# failed version must not count as pending: no kured flag, and the +# nightly reboot unit is skipped. # The updates run through systemd-sysupdate.service, the unit the # preset-enabled timer starts, verify the signed SHA256SUMS and have the # optional "zfs" sysupdate feature enabled, so the ZFS sysext follows the OS @@ -14,7 +21,7 @@ # a stand-in kubelet.service runs. Every disk boot has both update timers # enabled; a boot-counted UKI is blessed only after boot-complete.target, # which requires that no unit failed. -# Usage: dogfood-install.sh [ []] +# Usage: [DOGFOOD_BROKEN=slot|unit] dogfood-install.sh [ []] set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" @@ -51,6 +58,7 @@ systemctl start boot-complete.target 2>/dev/null || true echo "PROBE ukis=$(ls /boot/EFI/Linux 2>/dev/null | tr '\n' ' ')" echo "PROBE timers-enabled=$(systemctl is-enabled systemd-sysupdate.timer systemd-sysupdate-reboot.timer | tr '\n' ' ')" echo "PROBE health=$(systemctl is-active systemd-boot-check-no-failures.service boot-complete.target | tr '\n' ' ')bless=$(/usr/lib/systemd/systemd-bless-boot status 2>/dev/null)" +echo "PROBE deadline=$(systemctl is-active bluefin-boot-deadline.timer) $(systemctl show -P ConditionResult bluefin-boot-deadline.timer)" EOF cat > "${state}/update.probe" <<'EOF' @@ -100,16 +108,71 @@ grep -q "PROBE health=active active bless=good" "${state}/4-updated.log" [ -n "${broken}" ] || { echo "PASS: installed, updated A->B and booted ${new_ver}"; exit 0; } bad_ver="$(ls "${broken}"/bluefin-server-[0-9]*.efi | sed -n 's|.*/bluefin-server-\(.*\)\.efi$|\1|p')" -sed "s|^echo \"PROBE update=|dd if=/dev/urandom of=/dev/disk/by-partlabel/bluefin_usr_${bad_ver} bs=1M count=64 conv=fsync 2>/dev/null \&\& echo PROBE corrupted=${bad_ver}\necho \"PROBE update=|" \ - "${state}/update.probe" > "${state}/break.probe" +mode="${DOGFOOD_BROKEN:-slot}" +case "${mode}" in + slot) + sed "s|^echo \"PROBE update=|dd if=/dev/urandom of=/dev/disk/by-partlabel/bluefin_usr_${bad_ver} bs=1M count=64 conv=fsync 2>/dev/null \&\& echo PROBE corrupted=${bad_ver}\necho \"PROBE update=|" \ + "${state}/update.probe" > "${state}/break.probe" + cp "${state}/disk.probe" "${state}/rollback.probe" ;; + unit) + # A unit that fails on ${bad_ver} only, and a deadline short enough + # for three tries in one run. + { cat "${state}/update.probe"; cat < "${state}/break.probe" +cat > /etc/systemd/system/dogfood-broken.service <<'UNIT' +[Unit] +Description=Dogfood: fail on ${bad_ver} only +ConditionOSRelease=IMAGE_VERSION=${bad_ver} +[Service] +Type=oneshot +ExecStart=/usr/bin/false +[Install] +WantedBy=multi-user.target +UNIT +systemctl enable dogfood-broken.service 2>/dev/null +mkdir -p /etc/systemd/system/bluefin-boot-deadline.timer.d +printf '[Timer]\nOnBootSec=\nOnBootSec=${DOGFOOD_DEADLINE:-2min}\n' > /etc/systemd/system/bluefin-boot-deadline.timer.d/dogfood.conf +echo "PROBE corrupted=${bad_ver} (dogfood-broken.service)" +BREAK + # A broken boot reports the gate and waits for the deadline to reboot + # it; only the fallback boot finishes the probe, and reads the + # deadline's log of every try from the persistent journal. + { cat < "${state}/rollback.probe" +if [ "\$(. /usr/lib/os-release; echo "\${IMAGE_VERSION}")" = ${bad_ver} ]; then + echo "PROBE counted-boot=${bad_ver} bless=\$(/usr/lib/systemd/systemd-bless-boot status) failed=\$(systemctl --failed --no-legend --plain | cut -d' ' -f1 | tr '\n' ' ')deadline=\$(systemctl is-active bluefin-boot-deadline.timer)" + exec sleep infinity +fi +GATE +journalctl -o cat --no-pager -u bluefin-boot-deadline.service | grep 'boot deadline' | sed 's/^/PROBE-LOG /' +rm -f /run/reboot-required +systemctl start --wait systemd-sysupdate.service || true +journalctl -b -o cat --no-pager -u systemd-sysupdate.service | grep 'failed its boot tries' | sed 's/^/PROBE-LOG /' +echo "PROBE after-rollback kured-flag=$(test -e /run/reboot-required && echo set || echo none)" +systemctl start systemd-sysupdate-reboot.service +echo "PROBE after-rollback reboot-unit=$(systemctl show -P Result systemd-sysupdate-reboot.service)" +AFTER + ;; + *) echo "ERROR: DOGFOOD_BROKEN must be slot or unit" >&2; exit 1 ;; +esac -echo "==> 5/6 update to ${bad_ver} and corrupt its slot" +echo "==> 5/6 update to ${bad_ver} and break it (${mode})" DOGFOOD_TIMEOUT="${DOGFOOD_UPDATE_TIMEOUT:-900}" DOGFOOD_BOOT=disk run "${broken}" "${state}/break.probe" | tee "${state}/5-break.log" grep -q "PROBE corrupted=${bad_ver}" "${state}/5-break.log" echo "==> 6/6 boot: ${bad_ver} must fail its tries and fall back to ${new_ver}" -DOGFOOD_TIMEOUT="${DOGFOOD_ROLLBACK_TIMEOUT:-1500}" DOGFOOD_BOOT=disk run "${broken}" "${state}/disk.probe" | tee "${state}/6-rollback.log" +# In unit mode the broken boots log [FAILED] for the deliberately failed unit, +# so the fallback boot is judged by its own probe below. +DOGFOOD_TIMEOUT="${DOGFOOD_ROLLBACK_TIMEOUT:-1800}" DOGFOOD_BOOT=disk run "${broken}" "${state}/rollback.probe" \ + | tee "${state}/6-rollback.log" || [ "${mode}" = unit ] grep -q "PROBE os=bluefin-server ${new_ver}" "${state}/6-rollback.log" grep -q "bluefin-server-${bad_ver}+0-3.efi" "${state}/6-rollback.log" grep -q "PROBE zfs=active" "${state}/6-rollback.log" -echo "PASS: installed, updated A->B, and rolled back from a broken ${bad_ver} to ${new_ver}" +if [ "${mode}" = unit ]; then + [ "$(grep -Ec "PROBE counted-boot=${bad_ver} bless=(indeterminate|dirty) failed=dogfood-broken.service .*deadline=active" "${state}/6-rollback.log")" = 3 ] + [ "$(grep -c "PROBE-LOG Bluefin Server ${bad_ver} did not reach boot-complete.target .*rebooting so systemd-boot falls back" "${state}/6-rollback.log")" = 3 ] + grep -q "PROBE deadline=inactive no" "${state}/6-rollback.log" + grep -q "PROBE health=active active bless=clean" "${state}/6-rollback.log" + grep -q "PROBE after-rollback kured-flag=none" "${state}/6-rollback.log" + grep -q "PROBE after-rollback reboot-unit=exec-condition" "${state}/6-rollback.log" + grep -q "PROBE failed=0" "${state}/6-rollback.log" +fi +echo "PASS: installed, updated A->B, and rolled back from a broken (${mode}) ${bad_ver} to ${new_ver}" From 3927747f298a03077dbc5858472d329d07137d9d Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:04:24 -0500 Subject: [PATCH 15/17] docs(skills): document the boot deadline, the reboot interlock and pinned diskless origins ddi-installer.md "Updates": boot counting alone only rolls back on the next boot, so describe bluefin-boot-deadline (when it runs, why a Kubernetes node is flagged for kured instead of rebooted, the dirty last try, the drop-in to change the deadline), bluefin-update-pending after a rollback, the lock files in 20-interlock.conf (credited to #182), and which diskless origins are pinned. ddi-installer-build.md covers DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/ddi-installer-build.md | 22 +++++--- docs/skills/ddi-installer.md | 81 +++++++++++++++++++++++------- 2 files changed, 80 insertions(+), 23 deletions(-) diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md index f29979cb..f41f057e 100644 --- a/docs/skills/ddi-installer-build.md +++ b/docs/skills/ddi-installer-build.md @@ -4,7 +4,7 @@ description: Build, export, and dogfood the Bluefin Server image set (OS DDI, si metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd - /apache/buildstream @@ -103,7 +103,11 @@ Useful environment variables: - `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe. Every diskless `--check` boot also runs `bluefin-diskless-update-check` once -and reports `PROBE update-check= flag=`. +and reports `PROBE update-check= flag=`. Its origin is the +versioned `import.pull` file, so a newer release in the served directory only +logs that the node is pinned; an HTTP boot through a fixed-name UKI +(`DOGFOOD_BOOT=http`, `DOGFOOD_BOOT_URL=.../bluefin-server-netboot.efi`) +sets the flag. `scripts/dogfood-install.sh [ []]` is the full end-to-end check: install from a diskless boot, boot the installed disk, @@ -112,9 +116,15 @@ end-to-end check: install from a diskless boot, boot the installed disk, manifest (with the `zfs` feature enabled, so the ZFS sysext follows the OS in lock-step), then asserts the kured flag, the Kubernetes reboot interlock, both timers enabled and the new UKI blessed after `boot-complete.target`, and with -`` corrupt the updated slot and -confirm boot counting rolls the node back to `` on its own, with the -matching ZFS sysext still merged. CI runs the first two stages +`` break the update and confirm boot counting rolls the node back +to `` on its own, with the matching ZFS sysext still merged. +`DOGFOOD_BROKEN=slot` (default) corrupts the updated usr slot, so the initrd +fails. `DOGFOOD_BROKEN=unit` adds a unit that fails on ``'s version +only and shortens the boot deadline (`DOGFOOD_DEADLINE`, default `2min`): each +counted boot reaches `multi-user.target`, misses `boot-complete.target`, and +`bluefin-boot-deadline` reboots it. The run asserts three such boots, the +fallback boot with the deadline timer inactive, and then no kured flag and a +skipped `systemd-sysupdate-reboot.service` for the failed version. CI runs the first two stages (`dogfood-diskless.sh --check`, `dogfood-install.sh dist/diskless`) as the `boot-test` job in `.github/workflows/build.yml` on every pull request and push to main. @@ -151,7 +161,7 @@ nodes may already trust. | "Skip `gen-dev-keys`, the build has defaults." | Signing needs real key material in `files/boot-keys/`; the recipe generates throwaway keys so local builds boot under Secure Boot. | | "Test the UKI with `-kernel`/`-initrd`." | That bypasses the signed boot chain. The dogfood scripts boot the netboot ESP or installed disk through OVMF the way firmware does. | | "Reboot loops mean the boot hung." | With Secure Boot in setup mode the first boot enrolls keys and reboots; that is expected once per fresh variable store. | -| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image. `dogfood-install.sh ` proves it. | +| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image; `bluefin-boot-deadline` reboots a boot that comes up with a failed unit. `dogfood-install.sh ` proves it (`DOGFOOD_BROKEN=unit` for the failed-unit case). | | "`chmod 4755` in install-commands makes a file setuid in the image." | No. BuildStream artifacts keep one executable bit per file, so every staged file is 0644/0755. FSDK components declare their special modes as initial scripts, and `oci/bluefin-server-usr.bst` runs them (`os-initial-scripts.bst`) while assembling /sysroot; special modes must be set in the `script` element that writes the image. | ## Red flags diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index d13b3ef9..a80cb172 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d metadata: type: reference status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd - /apache/buildstream @@ -206,24 +206,59 @@ only when a newer version than the booted one is installed. `systemd-boot-check-no-failures.service` (`RequiredBy=boot-complete.target`, after `multi-user.target`), and `systemd-bless-boot` marks a boot-counted UKI good only after `boot-complete.target`. A good boot is one that reaches - `multi-user.target` with no failed unit; any other boot uses up a try, and - systemd-boot falls back to the previous UKI after the third. Only - sysupdate-installed UKIs are counted, so diskless boots never pull in - `boot-complete.target`. -- **Kubernetes interlock.** Kubernetes nodes reboot through - [kured](https://kured.dev/) (cordon, drain, one node at a time). - `systemd-sysupdate.service.d/20-kured.conf` touches `/run/reboot-required` - after each run once `systemd-sysupdate pending` reports an installed update. - `systemd-sysupdate-reboot.service.d/20-interlock.conf` skips the local - reboot (`ExecCondition=`) while `kubelet.service`, `k0scontroller.service` or - `k0sworker.service` is running, the interlock proposed in - [#182](https://github.com/projectbluefin/server/pull/182). Without kured, a - Kubernetes node keeps the staged update until someone reboots it. + `multi-user.target` with no failed unit. Only sysupdate-installed UKIs are + counted, so diskless boots never pull in `boot-complete.target`. +- **Rollback.** systemd-boot only moves on at the *next* boot, so the preset + also enables `bluefin-boot-deadline.timer`. It runs on boot-counted boots + only (the `LoaderBootCountPath` EFI variable exists; never on diskless, + installer, blessed or uncounted boots) and fires 15 minutes after boot. If + `systemd-bless-boot status` is still `indeterminate` (or `bad`) and + `boot-complete.target` is not active, `/usr/libexec/bluefin-boot-deadline` + logs the failed units and reboots, so systemd-boot uses up the next try and, + after the third, boots the previous UKI. The last try boots as `dirty` and + reboots only when another UKI can still be booted, so a node with nothing to + fall back to does not loop. The service is not ordered after + `multi-user.target`, so a boot that hangs gets the same treatment. Change the + deadline with a drop-in (`[Timer]`, `OnBootSec=`, then `OnBootSec=30min`). + - *On Kubernetes nodes* (kubelet or k0s starting, running or stopping) it + does not reboot: a node that has been up for 15 minutes may carry + workloads, so it touches `/run/reboot-required` and kured cordons, drains + and reboots it within its cluster lock, as for an update. Each try then + costs a drain, and without kured the node stays on the unblessed image + until someone reboots it. When the Kubernetes unit itself is stopped or + failed the node is `NotReady`, has nothing to drain and cannot run kured, + so it reboots directly. + - *Operator hold*: `/run/reboot-lock` or `/etc/reboot-lock` stops the + reboot and the kured flag, like the nightly reboot below. + - After a rollback, `systemd-sysupdate pending` still reports the failed + version, which would bring the node back to it every night (or through + kured every two hours). `/usr/libexec/bluefin-update-pending` wraps it and + says no while that version's UKI has no tries left + (`bluefin-server-+0-.efi`); it gates the kured flag and the + nightly reboot. The next newer release takes the failed version's slot and + clears the block. A boot that fails in the initrd reboots on its own + (`emergency.service.d/10-reboot.conf`). +- **Reboot interlock.** `systemd-sysupdate-reboot.service.d/20-interlock.conf` + skips the nightly reboot when: + - `/run/reboot-lock` (until the next boot) or `/etc/reboot-lock` (until + removed) exists, the operator hold from + [#182](https://github.com/projectbluefin/server/pull/182) by Bob Killen. + Kured has its own lock and does not read these files; + - the newest installed version already failed its boot tries + (`bluefin-update-pending`, above); + - `kubelet.service`, `k0scontroller.service` or `k0sworker.service` is + running, the Kubernetes interlock also proposed in #182. Kubernetes nodes + reboot through [kured](https://kured.dev/) (cordon, drain, one node at a + time): `systemd-sysupdate.service.d/20-kured.conf` touches + `/run/reboot-required` after each run once `bluefin-update-pending` reports + an update. Without kured, a Kubernetes node keeps the staged update until + someone reboots it. - **Opting out.** `systemctl disable --now systemd-sysupdate-reboot.timer` stages updates without rebooting; also disable `systemd-sysupdate.timer` to stop updating. Presets apply on first boot only, so nodes installed before this preset need `systemctl preset systemd-sysupdate.timer - systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service` once. + systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service + bluefin-boot-deadline.timer` once. ### Diskless and installer boots @@ -241,8 +276,17 @@ http(s) `rd.systemd.pull=` source, or the `import.pull` credential. `bluefin-diskless-update-check` then fetches `SHA256SUMS` and `SHA256SUMS.gpg` from that directory, verifies them with `gpgv` against the image keyring, and compares the `bluefin-server_.raw` entry with the running `IMAGE_VERSION` -(`systemd-analyze compare-versions`). When the release is newer it touches -`/run/reboot-required` and logs `Bluefin Server is available from `. +(`systemd-analyze compare-versions`). When the release is newer and the next +boot would pull it, it touches `/run/reboot-required` and logs `Bluefin Server + is available from `. That holds when the origin resolves the +version at boot: a netboot UKI served under a fixed name (Booty's +`bluefin-server-netboot.efi`, via `StubDeviceURL`) that the server swaps for +the new release. An origin that names a versioned file, +`bluefin-server_.raw` (an explicit `rd.systemd.pull=` URL or the +`import.pull` credential) or `bluefin-server-netboot_.efi`, is pinned: +the node would pull the same version again, and kured would drain and reboot +it on every check. A pinned node only logs `... but this node is pinned to +; not flagging`; repoint its boot origin to update it. Network and signature errors are logged and leave the flag alone; the unit never fails and never reboots. Kured reboots a diskless Kubernetes node on the flag. Other nodes wait for an operator or fleet tooling to act on it @@ -300,6 +344,8 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. | "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. | | "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless, and `bluefin-diskless-update-check` only flags that a newer image is being served. | | "The update timer should reboot Kubernetes nodes too." | Kubernetes nodes reboot through kured (drain first); the reboot unit's `ExecCondition=` stands down while kubelet or k0s runs. | +| "Boot counting alone rolls back a bad update." | Only on the next boot. `bluefin-boot-deadline.timer` supplies that reboot when a counted boot misses `boot-complete.target`, and `bluefin-update-pending` keeps the node from rebooting into the failed version again. | +| "Flag every diskless node when a newer release is served." | Only when the next boot would pull it; a pinned origin (`bluefin-server_.raw`) would pull the same version and loop kured. | ## Verification @@ -307,6 +353,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. - [ ] `just dogfood-check` passes (diskless boot, Secure Boot, no failed units). - [ ] `just dogfood-install NEXT=` passes (install, disk boot with both update timers enabled, A/B update through `systemd-sysupdate.service` with the kured flag and interlock, blessed after `boot-complete.target`). - [ ] `just dogfood-check` reports `PROBE update-check=success` for the diskless update check. +- [ ] `DOGFOOD_BROKEN=unit scripts/dogfood-install.sh ` passes (a unit failing on ``: three deadline reboots, fallback to ``, then no kured flag and no nightly reboot). - [ ] UKIs pin `usrhash=` and are signed with DB; `sbverify` passes in `bluefin-server-boot.bst`. - [ ] No hardcoded device paths in any boot configuration. - [ ] No element named `bluefin-server-ddi` or `bluefin-server-installer` exists; the OS DDI comes from `oci/bluefin-server-image.bst`. From d9fe1f349f341ff71b71ddc68e9a90997b1344db Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:04:24 -0500 Subject: [PATCH 16/17] docs(skills): tell Kubernetes sysext operators the rollback reboot goes through kured While kubelet or k0s runs, bluefin-boot-deadline flags /run/reboot-required instead of rebooting, so kured also rolls back an unhealthy update. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/k0s-sysext-ops.md | 8 +++++--- docs/skills/kubeadm-sysext.md | 5 +++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 7c792b27..2233ff7a 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s and KubeStellar systemd-sysext extensi metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd --- @@ -43,8 +43,10 @@ The unit copies `/var/lib/k0s/k0s.raw` to `/run/extensions/k0s.raw`, runs `systemd-sysext refresh`, and then enables either `k0scontroller.service` or `k0sworker.service` depending on whether `/etc/k0s/token` exists. -While either k0s unit runs, the base image's automatic reboot stands down; -deploy kured to roll staged OS updates across the cluster (see "Updates" in +While either k0s unit runs, the base image's automatic reboot stands down, +and so does the boot-deadline rollback reboot, which flags +`/run/reboot-required` instead; deploy kured to roll staged OS updates (and +rollbacks) across the cluster (see "Updates" in [ddi-installer.md](ddi-installer.md)). ### Worker nodes diff --git a/docs/skills/kubeadm-sysext.md b/docs/skills/kubeadm-sysext.md index 7ac3b619..74f15802 100644 --- a/docs/skills/kubeadm-sysext.md +++ b/docs/skills/kubeadm-sysext.md @@ -4,7 +4,7 @@ description: Build, ship and operate the opt-in kubeadm worker systemd-sysext (k metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" --- # kubeadm worker sysext @@ -64,7 +64,8 @@ the cluster CNI (Cilium's `cilium-cni`). containerd searches - `/etc/resolv.conf` links to systemd-resolved's `/run/systemd/resolve/resolv.conf` (base image), the path kubelet's `resolvConf` expects. - While `kubelet.service` runs or restarts, the base image's automatic reboot - stands down and reboots belong to kured; see "Updates" in + and the boot-deadline rollback reboot stand down and reboots belong to kured + (the deadline flags `/run/reboot-required`); see "Updates" in [ddi-installer.md](ddi-installer.md). ## Host tools From 2e50d002a7bd047d44e01230c2cb2231b34b38b8 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:04:24 -0500 Subject: [PATCH 17/17] docs: make the rollback and reboot coordination claims precise The README, readiness audit, roadmap and distro gap analysis said boot counting rolls back an update that does not boot cleanly; that now holds because the boot deadline supplies the reboot. Say so, and how long it takes. Reconcile the "Reboot coordination" rows with #182: the lock files and the Kubernetes interlock both live in 20-interlock.conf, credited to #182. The diskless signal is described as skipping pinned nodes. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- README.md | 4 ++-- docs/MVP_1_0_READINESS.md | 6 +++--- docs/skills/architecture-roadmap.md | 4 ++-- docs/skills/gap-analysis-distros.md | 12 +++++++----- 4 files changed, 14 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 85276144..e791ac90 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ Bluefin Server is currently in **Alpha**: ## What it is -- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image, and flags `/run/reboot-required` when its boot server offers one. -- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot on a timer and reboots into it nightly (Kubernetes nodes leave the reboot to kured), and UKI boot counting rolls back an update that does not boot cleanly (any failed unit counts). +- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image, and flags `/run/reboot-required` when its boot server offers one that its next boot would pull (not when the node is pinned to a versioned image). +- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot on a timer and reboots into it nightly (Kubernetes nodes leave the reboot to kured), and UKI boot counting rolls back an update that does not boot cleanly: a boot-counted boot that has not reached `boot-complete.target` (any failed unit counts) within 15 minutes reboots, or on Kubernetes nodes is flagged for kured, until systemd-boot falls back to the previous image, which then stays put until a newer release. `/run/reboot-lock` or `/etc/reboot-lock` holds these reboots. - **Secure Boot on** — signed systemd-boot, signed UKIs, signed kernel modules, `lockdown=integrity`. UEFI HTTP boot is supported; [Booty](https://github.com/jeefy/booty) serves the UKI, the OS DDI, the signed manifest, and a per-node `bluefin-node.ign`. - **Opt-in per-node state via Ignition** — pass an `ignition.config` / `ignition.config.url` system credential (or, on UEFI HTTP boot, a `bluefin-node.ign` next to the UKI) and Ignition runs in the initrd on every boot; configs must be idempotent. - **Opt-in sysexts** — k0s (Kubernetes), KubeStellar, and OpenZFS ship as separate `systemd-sysext` images, never in the base `/usr`. ZFS and KubeStellar are version-locked to the image and follow OS updates through optional sysupdate features. diff --git a/docs/MVP_1_0_READINESS.md b/docs/MVP_1_0_READINESS.md index f2c71a95..1fb19b98 100644 --- a/docs/MVP_1_0_READINESS.md +++ b/docs/MVP_1_0_READINESS.md @@ -19,10 +19,10 @@ This audit tracks the gap between the current tree and a first public/usable MVP | Release workflow lint | ✅ | `actionlint .github/workflows/build.yml` clean | | Release path exists | ✅ | `.github/workflows/build.yml` signs `SHA256SUMS` in-element (`oci/bluefin-server-image.bst`, with a `gpgv` proof against the shipped keyring) and publishes immutable `v` GitHub Releases plus the OCI artifact `ghcr.io//bluefin-server:,latest` | | Automated boot test | ✅ | `boot-test` job in `build.yml` runs `scripts/dogfood-diskless.sh --check` and `scripts/dogfood-install.sh` on every PR and push to main; locally `just dogfood-check` / `just dogfood-install`. UEFI HTTP boot verified with Secure Boot (`DOGFOOD_BOOT=http`), and tampered DDI / unsigned manifest both refused (`DOGFOOD_TAMPER=raw\|sums`) | -| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots; proven by `scripts/dogfood-install.sh ` | +| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots, and `bluefin-boot-deadline.timer` reboots a counted boot that misses `boot-complete.target`; proven by `scripts/dogfood-install.sh ` (`DOGFOOD_BROKEN=slot` or `unit`) | | Read-only /usr | ✅ | erofs + dm-verity pinned by `usrhash=` in the signed UKIs (`elements/oci/bluefin-server-usr.bst`, `elements/oci/bluefin-server-boot.bst`) | | Signed diskless pull | ✅ | Netboot UKI pulls with `verify=signature`; the initrd ships gnupg and the keyring (`/etc/systemd/import-pubring.pgp` from `os-sysupdate-keys.bst`) | -| Automatic updates | ✅ | Installed nodes: `80-bluefin-updates.preset` enables `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`; Kubernetes nodes set `/run/reboot-required` and leave the reboot to kured (`ExecCondition=` interlock); `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so only a boot with no failed unit is blessed. Diskless nodes: `bluefin-diskless-update-check` flags a newer signed release on the boot server. See [ddi-installer.md](skills/ddi-installer.md) "Updates" | +| Automatic updates | ✅ | Installed nodes: `80-bluefin-updates.preset` enables `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`; Kubernetes nodes set `/run/reboot-required` and leave the reboot to kured (`ExecCondition=` interlock); `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so only a boot with no failed unit is blessed, and `bluefin-boot-deadline.timer` reboots (or flags kured for) a counted boot that is not blessed in 15 minutes; `/run/reboot-lock` and `/etc/reboot-lock` hold local reboots. Diskless nodes: `bluefin-diskless-update-check` flags a newer signed release on the boot server unless the node is pinned to a versioned image. See [ddi-installer.md](skills/ddi-installer.md) "Updates" | | Sysext lock-step updates | ✅ | `zfs` / `kubestellar` sysupdate features download the version-locked sysext with each OS update; rollback keeps the matching sysext (verified in the 6-phase `dogfood-install` run) | | First-boot SSH keys | ✅ | Ignition (`ignition.config` / `ignition.config.url` credentials, or `bluefin-node.ign` next to the UKI on UEFI HTTP boot; `tests/fixtures/ignition/var-on-disk.ign`) and `tmpfiles.extra` / sysusers credentials (`elements/bluefin-server/os-creds-prov.bst`) | @@ -61,7 +61,7 @@ Priority order. Each item depends on the ones above it. ### Phase D: release discipline - [ ] TPM2-sealed /var and credential decryption proof on real hardware. -- [x] Reboot coordination for non-Kubernetes hosts: nightly `systemd-sysupdate-reboot.timer`, a kured interlock on Kubernetes nodes, and a boot health gate (#139). +- [x] Reboot coordination for non-Kubernetes hosts: nightly `systemd-sysupdate-reboot.timer`, operator lock files and a kured interlock (both from #182), and a boot health gate that reboots an unhealthy counted boot (#139). - [ ] Cluster-wide reboot lock (FleetLock-style) for multi-node fleets without Kubernetes. - [ ] Merge the Booty `feat/bluefin-http-boot` branch; shim-signed Secure Boot path for enrollment-free first boots. - [ ] Tag `v1.0.0-MVP` once Phase D items land. diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index c72c21a0..4a50f8aa 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -4,7 +4,7 @@ description: Roadmap for future Bluefin Server architecture work. Use when plann metadata: type: reference status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd --- @@ -30,7 +30,7 @@ For reference, so future planning does not redo them: - OCI artifact output (`ghcr.io//bluefin-server:,latest`) alongside the raw release files, via ORAS in CI and `just publish-oci` locally. - Booty HTTP boot: per-MAC serving of the UKI, DDI, `SHA256SUMS(.gpg)`, and per-host `bluefin-node.ign`, verified end to end in QEMU with Secure Boot; the Booty branch is not yet merged upstream. - ZFS and KubeStellar sysexts version-locked to the image, delivered in lock-step with OS updates through sysupdate features; rollback keeps the matching sysext. -- Automatic updates on installed nodes (preset-enabled `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`), a kured interlock for Kubernetes nodes, a boot health gate (`systemd-boot-check-no-failures.service`), and a signed update signal for diskless nodes (`bluefin-diskless-update-check`); see [ddi-installer.md](ddi-installer.md) "Updates". +- Automatic updates on installed nodes (preset-enabled `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`), operator reboot lock files and a kured interlock for Kubernetes nodes (from #182), a boot health gate (`systemd-boot-check-no-failures.service`) whose deadline reboots an unhealthy counted boot back to the previous image (`bluefin-boot-deadline.timer`), and a signed update signal for diskless nodes that are not pinned to a versioned image (`bluefin-diskless-update-check`); see [ddi-installer.md](ddi-installer.md) "Updates". ## Planned work diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 8254056b..ce1b0b00 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -6,7 +6,7 @@ description: | metadata: type: reference status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" --- # Gap Analysis: Bluefin Server versus Comparable Server OSes @@ -98,10 +98,10 @@ kernel). No other distribution's binaries ship in the image. |------|---------------------------------| | **Philosophy** | Systemd-native, minimal, image-based server OS; diskless-first (a node boots from a network-pulled image in RAM and updates by rebooting), with an optional disk install. Base /usr includes bash for login and bring-up while heavy developer/debug tools live in sysexts or system containers; container workloads and Kubernetes run via opt-in sysexts. Sources: [AGENTS.md](../../AGENTS.md), [factory-integration.md](factory-integration.md). | | **State model** | `/usr` is a read-only erofs filesystem verified by dm-verity, pinned by `usrhash=` on the locked UKI command line. `/etc` is empty on every boot and populated from `/usr/share/factory/etc` by systemd-tmpfiles. Diskless nodes run from tmpfs. Installed nodes get a persistent xfs root via systemd gpt-auto discovery. Sources: [bluefin-server-usr.bst](../../elements/oci/bluefin-server-usr.bst), [bluefin-server-boot.bst](../../elements/oci/bluefin-server-boot.bst), [50-root.conf](../../files/os/repart.d/50-root.conf). | -| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. `systemd-sysupdate.timer` is enabled by preset, and `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so an update is blessed only when no unit failed. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless); `bluefin-diskless-update-check` flags `/run/reboot-required` when the boot server offers a newer signed release. Sources: [ddi-installer.md](ddi-installer.md), [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | +| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. `systemd-sysupdate.timer` is enabled by preset, and `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so an update is blessed only when no unit failed; `bluefin-boot-deadline.timer` reboots a counted boot that is not blessed in 15 minutes so systemd-boot falls back, and `bluefin-update-pending` keeps a rolled-back node off the failed version. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless); `bluefin-diskless-update-check` flags `/run/reboot-required` when the boot server offers a newer signed release that the next boot would pull (not for a node pinned to a versioned image). Sources: [ddi-installer.md](ddi-installer.md), [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | | **Provisioning** | Stock `systemd-sysinstall` copies `/usr` onto a target disk, started either from the offline USB installer (`bluefin-server-installer_.raw`) or on a diskless-booted node (see [ddi-installer.md](ddi-installer.md)). Per-node configuration is opt-in via Ignition, delivered as `ignition.config` / `ignition.config.url` system credentials (the cmdline is locked inside the signed UKI); Ignition runs on every boot, so configs must be idempotent. First-boot systemd credentials also cover root password, `tmpfiles.extra`, `network.*`, and `firstboot.*`. Sources: [bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst), [initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md). | | **Customization** | Adds software through opt-in `systemd-sysext` images (overlay `/usr`): k0s (Kubernetes) and OpenZFS are built in-tree. The base OS `os-release` identifies as `ID=bluefin-server`; the ZFS sysext pins `VERSION_ID` to the image version because its kernel modules are built against the exact FSDK kernel. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release.bst](../../elements/bluefin-server/os-release.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf), [Kured project](https://github.com/kubereboot/kured). | +| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. `/run/reboot-lock` (until the next boot) or `/etc/reboot-lock` holds the reboot. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. The lock files and the Kubernetes interlock come from [#182](https://github.com/projectbluefin/server/pull/182). Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf), [Kured project](https://github.com/kubereboot/kured). | ## 4. Factual Gaps @@ -121,7 +121,7 @@ kernel). No other distribution's binaries ship in the image. ### Update delivery -- **Resolved:** installed nodes update and reboot on their own (`80-bluefin-updates.preset`), a boot is blessed only when no unit failed, and diskless nodes get a signed update signal (`bluefin-diskless-update-check`). +- **Resolved:** installed nodes update and reboot on their own (`80-bluefin-updates.preset`), a boot is blessed only when no unit failed and is rebooted back to the previous image when it is not blessed in time (`bluefin-boot-deadline.timer`), and diskless nodes get a signed update signal (`bluefin-diskless-update-check`). - **Resolved:** the diskless pull of the OS DDI runs with `verify=signature`; the initrd ships gnupg and the image keyring, so the download is checked against the signed `SHA256SUMS` before the pinned `usrhash=` / dm-verity checks even begin. ### Customization @@ -130,7 +130,7 @@ kernel). No other distribution's binaries ship in the image. ### Reboot coordination -- **Resolved:** single-node and non-Kubernetes hosts reboot in the nightly `systemd-sysupdate-reboot.timer` window; Kubernetes nodes stand down for Kured. +- **Resolved:** single-node and non-Kubernetes hosts reboot in the nightly `systemd-sysupdate-reboot.timer` window, held by `/run/reboot-lock` or `/etc/reboot-lock`; Kubernetes nodes stand down for Kured (lock files and interlock from #182). - **Gap:** there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`, so non-Kubernetes hosts that serve together may reboot in the same window. ## 5. Summary of Biggest Gaps @@ -203,5 +203,7 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/systemd/system-preset/80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset) - [files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf) - [files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf) +- [files/os/systemd/system/bluefin-boot-deadline.timer](../../files/os/systemd/system/bluefin-boot-deadline.timer) +- [files/os/update-check/usr/libexec/bluefin-boot-deadline](../../files/os/update-check/usr/libexec/bluefin-boot-deadline) - [files/os/update-check/usr/libexec/bluefin-diskless-update-check](../../files/os/update-check/usr/libexec/bluefin-diskless-update-check) - [files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf)