diff --git a/README.md b/README.md index 6d593ab5..e791ac90 100644 --- a/README.md +++ b/README.md @@ -25,8 +25,8 @@ Bluefin Server is currently in **Alpha**: ## What it is -- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image. -- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot, and UKI boot counting rolls back a failed update automatically. +- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image, and flags `/run/reboot-required` when its boot server offers one that its next boot would pull (not when the node is pinned to a versioned image). +- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot on a timer and reboots into it nightly (Kubernetes nodes leave the reboot to kured), and UKI boot counting rolls back an update that does not boot cleanly: a boot-counted boot that has not reached `boot-complete.target` (any failed unit counts) within 15 minutes reboots, or on Kubernetes nodes is flagged for kured, until systemd-boot falls back to the previous image, which then stays put until a newer release. `/run/reboot-lock` or `/etc/reboot-lock` holds these reboots. - **Secure Boot on** — signed systemd-boot, signed UKIs, signed kernel modules, `lockdown=integrity`. UEFI HTTP boot is supported; [Booty](https://github.com/jeefy/booty) serves the UKI, the OS DDI, the signed manifest, and a per-node `bluefin-node.ign`. - **Opt-in per-node state via Ignition** — pass an `ignition.config` / `ignition.config.url` system credential (or, on UEFI HTTP boot, a `bluefin-node.ign` next to the UKI) and Ignition runs in the initrd on every boot; configs must be idempotent. - **Opt-in sysexts** — k0s (Kubernetes), KubeStellar, and OpenZFS ship as separate `systemd-sysext` images, never in the base `/usr`. ZFS and KubeStellar are version-locked to the image and follow OS updates through optional sysupdate features. diff --git a/docs/MVP_1_0_READINESS.md b/docs/MVP_1_0_READINESS.md index ea4d131a..1fb19b98 100644 --- a/docs/MVP_1_0_READINESS.md +++ b/docs/MVP_1_0_READINESS.md @@ -19,9 +19,10 @@ This audit tracks the gap between the current tree and a first public/usable MVP | Release workflow lint | ✅ | `actionlint .github/workflows/build.yml` clean | | Release path exists | ✅ | `.github/workflows/build.yml` signs `SHA256SUMS` in-element (`oci/bluefin-server-image.bst`, with a `gpgv` proof against the shipped keyring) and publishes immutable `v` GitHub Releases plus the OCI artifact `ghcr.io//bluefin-server:,latest` | | Automated boot test | ✅ | `boot-test` job in `build.yml` runs `scripts/dogfood-diskless.sh --check` and `scripts/dogfood-install.sh` on every PR and push to main; locally `just dogfood-check` / `just dogfood-install`. UEFI HTTP boot verified with Secure Boot (`DOGFOOD_BOOT=http`), and tampered DDI / unsigned manifest both refused (`DOGFOOD_TAMPER=raw\|sums`) | -| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots; proven by `scripts/dogfood-install.sh ` | +| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots, and `bluefin-boot-deadline.timer` reboots a counted boot that misses `boot-complete.target`; proven by `scripts/dogfood-install.sh ` (`DOGFOOD_BROKEN=slot` or `unit`) | | Read-only /usr | ✅ | erofs + dm-verity pinned by `usrhash=` in the signed UKIs (`elements/oci/bluefin-server-usr.bst`, `elements/oci/bluefin-server-boot.bst`) | | Signed diskless pull | ✅ | Netboot UKI pulls with `verify=signature`; the initrd ships gnupg and the keyring (`/etc/systemd/import-pubring.pgp` from `os-sysupdate-keys.bst`) | +| Automatic updates | ✅ | Installed nodes: `80-bluefin-updates.preset` enables `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`; Kubernetes nodes set `/run/reboot-required` and leave the reboot to kured (`ExecCondition=` interlock); `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so only a boot with no failed unit is blessed, and `bluefin-boot-deadline.timer` reboots (or flags kured for) a counted boot that is not blessed in 15 minutes; `/run/reboot-lock` and `/etc/reboot-lock` hold local reboots. Diskless nodes: `bluefin-diskless-update-check` flags a newer signed release on the boot server unless the node is pinned to a versioned image. See [ddi-installer.md](skills/ddi-installer.md) "Updates" | | Sysext lock-step updates | ✅ | `zfs` / `kubestellar` sysupdate features download the version-locked sysext with each OS update; rollback keeps the matching sysext (verified in the 6-phase `dogfood-install` run) | | First-boot SSH keys | ✅ | Ignition (`ignition.config` / `ignition.config.url` credentials, or `bluefin-node.ign` next to the UKI on UEFI HTTP boot; `tests/fixtures/ignition/var-on-disk.ign`) and `tmpfiles.extra` / sysusers credentials (`elements/bluefin-server/os-creds-prov.bst`) | @@ -29,7 +30,7 @@ Competitor context: [gap-analysis-distros.md](skills/gap-analysis-distros.md) ## Verdict -**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Remaining work is hardening: TPM2-sealed state, reboot coordination outside Kubernetes, merging the Booty branch, and real-hardware boot proofs. +**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Installed nodes update and reboot on their own, with a Kubernetes interlock and a boot health gate. Remaining work is hardening: TPM2-sealed state, a cluster-wide reboot lock for non-Kubernetes fleets, merging the Booty branch, and real-hardware boot proofs. ## Roadmap @@ -60,7 +61,8 @@ Priority order. Each item depends on the ones above it. ### Phase D: release discipline - [ ] TPM2-sealed /var and credential decryption proof on real hardware. -- [ ] Reboot coordination for non-Kubernetes hosts. +- [x] Reboot coordination for non-Kubernetes hosts: nightly `systemd-sysupdate-reboot.timer`, operator lock files and a kured interlock (both from #182), and a boot health gate that reboots an unhealthy counted boot (#139). +- [ ] Cluster-wide reboot lock (FleetLock-style) for multi-node fleets without Kubernetes. - [ ] Merge the Booty `feat/bluefin-http-boot` branch; shim-signed Secure Boot path for enrollment-free first boots. - [ ] Tag `v1.0.0-MVP` once Phase D items land. - [ ] Publish release notes: verified boot path, trust model, known gaps. diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index 66ac9124..4a50f8aa 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -4,7 +4,7 @@ description: Roadmap for future Bluefin Server architecture work. Use when plann metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd --- @@ -30,6 +30,7 @@ For reference, so future planning does not redo them: - OCI artifact output (`ghcr.io//bluefin-server:,latest`) alongside the raw release files, via ORAS in CI and `just publish-oci` locally. - Booty HTTP boot: per-MAC serving of the UKI, DDI, `SHA256SUMS(.gpg)`, and per-host `bluefin-node.ign`, verified end to end in QEMU with Secure Boot; the Booty branch is not yet merged upstream. - ZFS and KubeStellar sysexts version-locked to the image, delivered in lock-step with OS updates through sysupdate features; rollback keeps the matching sysext. +- Automatic updates on installed nodes (preset-enabled `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`), operator reboot lock files and a kured interlock for Kubernetes nodes (from #182), a boot health gate (`systemd-boot-check-no-failures.service`) whose deadline reboots an unhealthy counted boot back to the previous image (`bluefin-boot-deadline.timer`), and a signed update signal for diskless nodes that are not pinned to a versioned image (`bluefin-diskless-update-check`); see [ddi-installer.md](ddi-installer.md) "Updates". ## Planned work @@ -41,7 +42,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md). | 2 | aarch64 build axis | `project.conf` and `include/arch.yml` already model it; no CI coverage yet. | | 3 | Booty merge and its Secure Boot shim story | The Bluefin HTTP-boot support works from [Booty](https://github.com/jeefy/booty) `feat/bluefin-http-boot`; it still needs to merge, and enrollment-free first boots need a shim-signed path. | | 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. | -| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. | +| 5 | Cluster-wide reboot lock for non-Kubernetes fleets | Single hosts reboot in the nightly window and Kubernetes nodes use kured; several non-Kubernetes hosts sharing a service still have no FleetLock/locksmith-style lock, so they may reboot in the same window. | ## Status notes diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md index 2c2a1f2a..35ce263c 100644 --- a/docs/skills/ddi-installer-build.md +++ b/docs/skills/ddi-installer-build.md @@ -4,7 +4,7 @@ description: Build, export, and dogfood the Bluefin Server image set (OS DDI, si metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd - /apache/buildstream @@ -106,13 +106,29 @@ Useful environment variables: the boot must fail with the RAM message (see `diskless-troubleshooting.md`). - `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe. +Every diskless `--check` boot also runs `bluefin-diskless-update-check` once +and reports `PROBE update-check= flag=`. Its origin is the +versioned `import.pull` file, so a newer release in the served directory only +logs that the node is pinned; an HTTP boot through a fixed-name UKI +(`DOGFOOD_BOOT=http`, `DOGFOOD_BOOT_URL=.../bluefin-server-netboot.efi`) +sets the flag. + `scripts/dogfood-install.sh [ []]` is the full end-to-end check: install from a diskless boot, boot the installed disk, -`systemd-sysupdate` A->B to `` with the default `Verify=yes` against -the signed manifest (with the `zfs` feature enabled, so the ZFS sysext follows -the OS in lock-step), and with `` corrupt the updated slot and -confirm boot counting rolls the node back to `` on its own, with the -matching ZFS sysext still merged. CI runs the first two stages +`systemd-sysupdate` A->B to `` through `systemd-sysupdate.service` +(the unit the timer starts) with the default `Verify=yes` against the signed +manifest (with the `zfs` feature enabled, so the ZFS sysext follows the OS in +lock-step), then asserts the kured flag, the Kubernetes reboot interlock, both +timers enabled and the new UKI blessed after `boot-complete.target`, and with +`` break the update and confirm boot counting rolls the node back +to `` on its own, with the matching ZFS sysext still merged. +`DOGFOOD_BROKEN=slot` (default) corrupts the updated usr slot, so the initrd +fails. `DOGFOOD_BROKEN=unit` adds a unit that fails on ``'s version +only and shortens the boot deadline (`DOGFOOD_DEADLINE`, default `2min`): each +counted boot reaches `multi-user.target`, misses `boot-complete.target`, and +`bluefin-boot-deadline` reboots it. The run asserts three such boots, the +fallback boot with the deadline timer inactive, and then no kured flag and a +skipped `systemd-sysupdate-reboot.service` for the failed version. CI runs the first two stages (`dogfood-diskless.sh --check`, `dogfood-install.sh dist/diskless`) as the `boot-test` job in `.github/workflows/build.yml` on every pull request and push to main. @@ -149,7 +165,7 @@ nodes may already trust. | "Skip `gen-dev-keys`, the build has defaults." | Signing needs real key material in `files/boot-keys/`; the recipe generates throwaway keys so local builds boot under Secure Boot. | | "Test the UKI with `-kernel`/`-initrd`." | That bypasses the signed boot chain. The dogfood scripts boot the netboot ESP or installed disk through OVMF the way firmware does. | | "Reboot loops mean the boot hung." | With Secure Boot in setup mode the first boot enrolls keys and reboots; that is expected once per fresh variable store. | -| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image. `dogfood-install.sh ` proves it. | +| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image; `bluefin-boot-deadline` reboots a boot that comes up with a failed unit. `dogfood-install.sh ` proves it (`DOGFOOD_BROKEN=unit` for the failed-unit case). | | "`chmod 4755` in install-commands makes a file setuid in the image." | No. BuildStream artifacts keep one executable bit per file, so every staged file is 0644/0755. FSDK components declare their special modes as initial scripts, and `oci/bluefin-server-usr.bst` runs them (`os-initial-scripts.bst`) while assembling /sysroot; special modes must be set in the `script` element that writes the image. | ## Red flags diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index de749811..7a8daaf2 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d metadata: type: reference status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd - /apache/buildstream @@ -192,17 +192,109 @@ Installed nodes update with `systemd-sysupdate` against the transfers in Sources are the release assets on GitHub Releases, verified against the GPG-signed `SHA256SUMS` with `Verify=yes` (see -`systemd-sysupdate-verification.md`). After an update the node reboots into -slot B; if the new image fails its three tries, systemd-boot rolls back to -slot A on its own. A Kured hook -(`files/os/systemd/system/systemd-sysupdate.service.d/kured-hook.conf`) -touches `/run/reboot-required` for cluster-aware reboot coordination. - -Diskless nodes have no slots, so `systemd-sysupdate.service` is disabled when -booted diskless -(`files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf`). -A diskless node updates by rebooting into a newer image; that is the whole -mechanism. +`systemd-sysupdate-verification.md`). + +### Automatic updates on installed nodes + +`files/os/systemd/system-preset/80-bluefin-updates.preset` enables FSDK's +`systemd-sysupdate.timer` (15 min after boot, then every 2 h, randomized) and +`systemd-sysupdate-reboot.timer` (04:10, randomized), which FSDK's +`90-sysupdate.preset` would otherwise disable. The timer runs the same +`systemd-sysupdate update` as a manual update, so enabled features follow the +OS in lock-step. The reboot unit runs `systemd-sysupdate reboot`, which reboots +only when a newer version than the booted one is installed. + +- **Boot health gate.** The preset enables + `systemd-boot-check-no-failures.service` (`RequiredBy=boot-complete.target`, + after `multi-user.target`), and `systemd-bless-boot` marks a boot-counted UKI + good only after `boot-complete.target`. A good boot is one that reaches + `multi-user.target` with no failed unit. Only sysupdate-installed UKIs are + counted, so diskless boots never pull in `boot-complete.target`. +- **Rollback.** systemd-boot only moves on at the *next* boot, so the preset + also enables `bluefin-boot-deadline.timer`. It runs on boot-counted boots + only (the `LoaderBootCountPath` EFI variable exists; never on diskless, + installer, blessed or uncounted boots) and fires 15 minutes after boot. If + `systemd-bless-boot status` is still `indeterminate` (or `bad`) and + `boot-complete.target` is not active, `/usr/libexec/bluefin-boot-deadline` + logs the failed units and reboots, so systemd-boot uses up the next try and, + after the third, boots the previous UKI. The last try boots as `dirty` and + reboots only when another UKI can still be booted, so a node with nothing to + fall back to does not loop. The service is not ordered after + `multi-user.target`, so a boot that hangs gets the same treatment. Change the + deadline with a drop-in (`[Timer]`, `OnBootSec=`, then `OnBootSec=30min`). + - *On Kubernetes nodes* (kubelet or k0s starting, running or stopping) it + does not reboot: a node that has been up for 15 minutes may carry + workloads, so it touches `/run/reboot-required` and kured cordons, drains + and reboots it within its cluster lock, as for an update. Each try then + costs a drain, and without kured the node stays on the unblessed image + until someone reboots it. When the Kubernetes unit itself is stopped or + failed the node is `NotReady`, has nothing to drain and cannot run kured, + so it reboots directly. + - *Operator hold*: `/run/reboot-lock` or `/etc/reboot-lock` stops the + reboot and the kured flag, like the nightly reboot below. + - After a rollback, `systemd-sysupdate pending` still reports the failed + version, which would bring the node back to it every night (or through + kured every two hours). `/usr/libexec/bluefin-update-pending` wraps it and + says no while that version's UKI has no tries left + (`bluefin-server-+0-.efi`); it gates the kured flag and the + nightly reboot. The next newer release takes the failed version's slot and + clears the block. A boot that fails in the initrd reboots on its own + (`emergency.service.d/10-reboot.conf`). +- **Reboot interlock.** `systemd-sysupdate-reboot.service.d/20-interlock.conf` + skips the nightly reboot when: + - `/run/reboot-lock` (until the next boot) or `/etc/reboot-lock` (until + removed) exists, the operator hold from + [#182](https://github.com/projectbluefin/server/pull/182) by Bob Killen. + Kured has its own lock and does not read these files; + - the newest installed version already failed its boot tries + (`bluefin-update-pending`, above); + - `kubelet.service`, `k0scontroller.service` or `k0sworker.service` is + running, the Kubernetes interlock also proposed in #182. Kubernetes nodes + reboot through [kured](https://kured.dev/) (cordon, drain, one node at a + time): `systemd-sysupdate.service.d/20-kured.conf` touches + `/run/reboot-required` after each run once `bluefin-update-pending` reports + an update. Without kured, a Kubernetes node keeps the staged update until + someone reboots it. +- **Opting out.** `systemctl disable --now systemd-sysupdate-reboot.timer` + stages updates without rebooting; also disable `systemd-sysupdate.timer` to + stop updating. Presets apply on first boot only, so nodes installed before + this preset need `systemctl preset systemd-sysupdate.timer + systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service + bluefin-boot-deadline.timer` once. + +### Diskless and installer boots + +Diskless nodes and the USB installer have no slots. Their `10-diskless.conf` +drop-ins condition `systemd-sysupdate.service`, +`systemd-sysupdate-reboot.service` and both timers off when +`/run/machines/rootdisk.raw` exists or the command line has `root=tmpfs`. A +diskless node updates by rebooting into whatever its boot server serves. + +`bluefin-diskless-update-check.timer` (5 min after boot, then hourly; only +when `/run/machines/rootdisk.raw` exists) tells the node when that would be a +newer image. `/usr/libexec/bluefin-boot-origin`, which Ignition also uses, +finds the boot directory from the `StubDeviceURL` EFI variable, the explicit +http(s) `rd.systemd.pull=` source, or the `import.pull` credential. +`bluefin-diskless-update-check` then fetches `SHA256SUMS` and `SHA256SUMS.gpg` +from that directory, verifies them with `gpgv` against the image keyring, and +compares the `bluefin-server_.raw` entry with the running `IMAGE_VERSION` +(`systemd-analyze compare-versions`). When the release is newer and the next +boot would pull it, it touches `/run/reboot-required` and logs `Bluefin Server + is available from `. That holds when the origin resolves the +version at boot: a netboot UKI served under a fixed name (Booty's +`bluefin-server-netboot.efi`, via `StubDeviceURL`) that the server swaps for +the new release. An origin that names a versioned file, +`bluefin-server_.raw` (an explicit `rd.systemd.pull=` URL or the +`import.pull` credential) or `bluefin-server-netboot_.efi`, is pinned: +the node would pull the same version again, and kured would drain and reboot +it on every check. A pinned node only logs `... but this node is pinned to +; not flagging`; repoint its boot origin to update it. +Network and signature errors are logged and leave the flag alone; the unit +never fails and never reboots. Kured reboots a diskless Kubernetes node on the +flag. Other nodes wait for an operator or fleet tooling to act on it +(`test -e /run/reboot-required`, +`journalctl -u bluefin-diskless-update-check`). The boot server (Booty) +decides what the next boot pulls. ## Ignition (opt-in) @@ -255,13 +347,18 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. | "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. | | "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. | | "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot; `ignition.*` kargs are ignored with a console warning. | -| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. | +| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless, and `bluefin-diskless-update-check` only flags that a newer image is being served. | +| "The update timer should reboot Kubernetes nodes too." | Kubernetes nodes reboot through kured (drain first); the reboot unit's `ExecCondition=` stands down while kubelet or k0s runs. | +| "Boot counting alone rolls back a bad update." | Only on the next boot. `bluefin-boot-deadline.timer` supplies that reboot when a counted boot misses `boot-complete.target`, and `bluefin-update-pending` keeps the node from rebooting into the failed version again. | +| "Flag every diskless node when a newer release is served." | Only when the next boot would pull it; a pinned origin (`bluefin-server_.raw`) would pull the same version and loop kured. | ## Verification - [ ] `just validate` resolves the BuildStream graph without errors. - [ ] `just dogfood-check` passes (diskless boot, Secure Boot, no failed units). -- [ ] `just dogfood-install NEXT=` passes (install, disk boot, A/B update). +- [ ] `just dogfood-install NEXT=` passes (install, disk boot with both update timers enabled, A/B update through `systemd-sysupdate.service` with the kured flag and interlock, blessed after `boot-complete.target`). +- [ ] `just dogfood-check` reports `PROBE update-check=success` for the diskless update check. +- [ ] `DOGFOOD_BROKEN=unit scripts/dogfood-install.sh ` passes (a unit failing on ``: three deadline reboots, fallback to ``, then no kured flag and no nightly reboot). - [ ] UKIs pin `usrhash=` and are signed with DB; `sbverify` passes in `bluefin-server-boot.bst`. - [ ] No hardcoded device paths in any boot configuration. - [ ] No element named `bluefin-server-ddi` or `bluefin-server-installer` exists; the OS DDI comes from `oci/bluefin-server-image.bst`. diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 037a14dc..ce1b0b00 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -6,7 +6,7 @@ description: | metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-29" --- # Gap Analysis: Bluefin Server versus Comparable Server OSes @@ -98,10 +98,10 @@ kernel). No other distribution's binaries ship in the image. |------|---------------------------------| | **Philosophy** | Systemd-native, minimal, image-based server OS; diskless-first (a node boots from a network-pulled image in RAM and updates by rebooting), with an optional disk install. Base /usr includes bash for login and bring-up while heavy developer/debug tools live in sysexts or system containers; container workloads and Kubernetes run via opt-in sysexts. Sources: [AGENTS.md](../../AGENTS.md), [factory-integration.md](factory-integration.md). | | **State model** | `/usr` is a read-only erofs filesystem verified by dm-verity, pinned by `usrhash=` on the locked UKI command line. `/etc` is empty on every boot and populated from `/usr/share/factory/etc` by systemd-tmpfiles. Diskless nodes run from tmpfs. Installed nodes get a persistent xfs root via systemd gpt-auto discovery. Sources: [bluefin-server-usr.bst](../../elements/oci/bluefin-server-usr.bst), [bluefin-server-boot.bst](../../elements/oci/bluefin-server-boot.bst), [50-root.conf](../../files/os/repart.d/50-root.conf). | -| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless). Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | +| **Updates** | Installed nodes: `systemd-sysupdate` fills the inactive usr / usr-verity slot (`files/os/sysupdate.d/10-usr.transfer`, `11-usr-verity.transfer`) and installs the new disk UKI with boot counting (`20-uki.transfer`), so a failed update rolls back automatically. Assets are published to GitHub Releases and as an OCI artifact; the combined `SHA256SUMS` manifest covering the whole set is signed inside the image build (`oci/bluefin-server-image.bst`) and `Verify=yes` is the default. `systemd-sysupdate.timer` is enabled by preset, and `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so an update is blessed only when no unit failed; `bluefin-boot-deadline.timer` reboots a counted boot that is not blessed in 15 minutes so systemd-boot falls back, and `bluefin-update-pending` keeps a rolled-back node off the failed version. Diskless nodes update by rebooting into a newer image (sysupdate is disabled when booted diskless); `bluefin-diskless-update-check` flags `/run/reboot-required` when the boot server offers a newer signed release that the next boot would pull (not for a node pinned to a versioned image). Sources: [ddi-installer.md](ddi-installer.md), [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer), [20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer), [80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset), [10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf), also `systemd-sysupdate(8)`. | | **Provisioning** | Stock `systemd-sysinstall` copies `/usr` onto a target disk, started either from the offline USB installer (`bluefin-server-installer_.raw`) or on a diskless-booted node (see [ddi-installer.md](ddi-installer.md)). Per-node configuration is opt-in via Ignition, delivered as `ignition.config` / `ignition.config.url` system credentials (the cmdline is locked inside the signed UKI); Ignition runs on every boot, so configs must be idempotent. First-boot systemd credentials also cover root password, `tmpfiles.extra`, `network.*`, and `firstboot.*`. Sources: [bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst), [initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md). | | **Customization** | Adds software through opt-in `systemd-sysext` images (overlay `/usr`): k0s (Kubernetes) and OpenZFS are built in-tree. The base OS `os-release` identifies as `ID=bluefin-server`; the ZFS sysext pins `VERSION_ID` to the image version because its kernel modules are built against the exact FSDK kernel. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release.bst](../../elements/bluefin-server/os-release.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate-reboot.timer` reboots installed nodes into a staged update in a nightly window. `/run/reboot-lock` (until the next boot) or `/etc/reboot-lock` holds the reboot. On Kubernetes nodes an `ExecCondition=` stands the local reboot down while kubelet or k0s runs, and `systemd-sysupdate.service` touches `/run/reboot-required` once an update is pending so Kured drains and reboots nodes one at a time. The lock files and the Kubernetes interlock come from [#182](https://github.com/projectbluefin/server/pull/182). Sources: [20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf), [20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf), [Kured project](https://github.com/kubereboot/kured). | ## 4. Factual Gaps @@ -121,7 +121,7 @@ kernel). No other distribution's binaries ship in the image. ### Update delivery -- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook. Diskless nodes need no update service at all (reboot is the update), so this only matters for installed nodes. +- **Resolved:** installed nodes update and reboot on their own (`80-bluefin-updates.preset`), a boot is blessed only when no unit failed and is rebooted back to the previous image when it is not blessed in time (`bluefin-boot-deadline.timer`), and diskless nodes get a signed update signal (`bluefin-diskless-update-check`). - **Resolved:** the diskless pull of the OS DDI runs with `verify=signature`; the initrd ships gnupg and the image keyring, so the download is checked against the signed `SHA256SUMS` before the pinned `usrhash=` / dm-verity checks even begin. ### Customization @@ -130,11 +130,12 @@ kernel). No other distribution's binaries ship in the image. ### Reboot coordination -- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. +- **Resolved:** single-node and non-Kubernetes hosts reboot in the nightly `systemd-sysupdate-reboot.timer` window, held by `/run/reboot-lock` or `/etc/reboot-lock`; Kubernetes nodes stand down for Kured (lock files and interlock from #182). +- **Gap:** there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`, so non-Kubernetes hosts that serve together may reboot in the same window. ## 5. Summary of Biggest Gaps -1. **Reboot coordination outside Kubernetes is missing.** No FleetLock/locksmith equivalent for single-node or non-Kubernetes hosts. +1. **No cluster-wide reboot lock outside Kubernetes.** Hosts reboot in a nightly window; there is no FleetLock/locksmith equivalent for non-Kubernetes fleets. 2. **Credential provisioning hardware proof is incomplete.** SSH keys, Ignition configs, network files, and firstboot settings are wired through systemd credentials; TPM2-sealed decryption still needs a hardware boot proof. The earlier gap on the unverified diskless DDI download is closed: the pull @@ -193,12 +194,16 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [elements/oci/bluefin-server-image.bst](../../elements/oci/bluefin-server-image.bst) - [elements/bluefin-server/os-release.bst](../../elements/bluefin-server/os-release.bst) - [elements/bluefin-server/os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst) -- [elements/bluefin-server/os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst) - [elements/bluefin-server/initrd/initrd-ignition.bst](../../elements/bluefin-server/initrd/initrd-ignition.bst) - [files/os/repart.d/50-root.conf](../../files/os/repart.d/50-root.conf) - [files/os/sysupdate.d/10-usr.transfer](../../files/os/sysupdate.d/10-usr.transfer) - [files/os/sysupdate.d/20-uki.transfer](../../files/os/sysupdate.d/20-uki.transfer) - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) -- [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) +- [files/os/systemd/system-preset/80-bluefin-updates.preset](../../files/os/systemd/system-preset/80-bluefin-updates.preset) +- [files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/20-kured.conf) +- [files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf](../../files/os/systemd/system/systemd-sysupdate-reboot.service.d/20-interlock.conf) +- [files/os/systemd/system/bluefin-boot-deadline.timer](../../files/os/systemd/system/bluefin-boot-deadline.timer) +- [files/os/update-check/usr/libexec/bluefin-boot-deadline](../../files/os/update-check/usr/libexec/bluefin-boot-deadline) +- [files/os/update-check/usr/libexec/bluefin-diskless-update-check](../../files/os/update-check/usr/libexec/bluefin-diskless-update-check) - [files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf](../../files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 74c871d0..2233ff7a 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s and KubeStellar systemd-sysext extensi metadata: type: how-to status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd --- @@ -43,6 +43,12 @@ The unit copies `/var/lib/k0s/k0s.raw` to `/run/extensions/k0s.raw`, runs `systemd-sysext refresh`, and then enables either `k0scontroller.service` or `k0sworker.service` depending on whether `/etc/k0s/token` exists. +While either k0s unit runs, the base image's automatic reboot stands down, +and so does the boot-deadline rollback reboot, which flags +`/run/reboot-required` instead; deploy kured to roll staged OS updates (and +rollbacks) across the cluster (see "Updates" in +[ddi-installer.md](ddi-installer.md)). + ### Worker nodes A node with a join token at `/etc/k0s/token` becomes a worker. The token can be diff --git a/docs/skills/kubeadm-sysext.md b/docs/skills/kubeadm-sysext.md index af9615f3..bbfe2b13 100644 --- a/docs/skills/kubeadm-sysext.md +++ b/docs/skills/kubeadm-sysext.md @@ -4,7 +4,7 @@ description: Build, ship and operate the opt-in kubeadm worker systemd-sysext (k metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" --- # kubeadm worker sysext @@ -89,6 +89,10 @@ but never newer. Patch releases change neither constraint. `--volume-plugin-dir=/var/lib/kubelet/volumeplugins` because `/usr` is read-only. - `/etc/resolv.conf` links to systemd-resolved's `/run/systemd/resolve/resolv.conf` (base image), the path kubelet's `resolvConf` expects. +- While `kubelet.service` runs or restarts, the base image's automatic reboot + and the boot-deadline rollback reboot stand down and reboots belong to kured + (the deadline flags `/run/reboot-required`); see "Updates" in + [ddi-installer.md](ddi-installer.md). ## Host tools diff --git a/docs/skills/systemd-sysupdate-verification.md b/docs/skills/systemd-sysupdate-verification.md index db982f08..60ee1e01 100644 --- a/docs/skills/systemd-sysupdate-verification.md +++ b/docs/skills/systemd-sysupdate-verification.md @@ -4,7 +4,7 @@ description: Configure and operate GPG signature verification for Bluefin Server metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd --- @@ -65,6 +65,14 @@ Only the k0s sysext stays a separate component (`files/os/sysupdate.k0s.d/`, `systemd-sysupdate --component=k0s update`) with its own version axis. Diskless nodes update by rebooting into a newer image; `systemd-sysupdate.service` is disabled when booted diskless. +Update scheduling, the kured flag, and the boot health gate are covered in +[ddi-installer.md](ddi-installer.md) under "Updates". + +The diskless update check (`bluefin-diskless-update-check`) uses the same trust +root: it trusts a newer release only after `gpgv` verifies the boot server's +`SHA256SUMS.gpg` against the keyring systemd reads +(`/etc/systemd/import-pubring.pgp`, else the vendor keyring). An unsigned or +foreign-signed manifest never sets `/run/reboot-required`. ## Signing happens inside the image build diff --git a/elements/bluefin-server/boot-origin.bst b/elements/bluefin-server/boot-origin.bst new file mode 100644 index 00000000..c8e2c0ec --- /dev/null +++ b/elements/bluefin-server/boot-origin.bst @@ -0,0 +1,11 @@ +kind: import +description: | + /usr/libexec/bluefin-boot-origin: prints the URL a network-booted node was + booted from (UEFI HTTP boot URL, the explicit rd.systemd.pull= source, or + the import.pull credential). One helper for both consumers: + bluefin-ignition-credentials in the initrd (initrd-ignition-stack.bst) and + bluefin-diskless-update-check in the booted OS (os-update-check.bst). + +sources: +- kind: local + path: files/boot-origin diff --git a/elements/bluefin-server/initrd/initrd-ignition-stack.bst b/elements/bluefin-server/initrd/initrd-ignition-stack.bst index f1d6cc91..36c12ae6 100644 --- a/elements/bluefin-server/initrd/initrd-ignition-stack.bst +++ b/elements/bluefin-server/initrd/initrd-ignition-stack.bst @@ -16,6 +16,8 @@ depends: - ignition/ignition.bst - ignition/gptfdisk.bst - bluefin-server/initrd/initrd-ignition.bst +# bluefin-boot-origin: the HTTP boot origin, for bluefin-node.ign next to the UKI +- bluefin-server/boot-origin.bst # bash, coreutils - freedesktop-sdk.bst:public-stacks/runtime-gnu.bst diff --git a/elements/bluefin-server/os-k0s-first-boot.bst b/elements/bluefin-server/os-k0s-first-boot.bst index e45d742a..4b496204 100644 --- a/elements/bluefin-server/os-k0s-first-boot.bst +++ b/elements/bluefin-server/os-k0s-first-boot.bst @@ -6,7 +6,10 @@ description: | sshd-generated@.service drop-ins), the installer source ESP mounts (run-bluefin-boot.mount for netboot, run-bluefin-installer.mount for the USB installer), and the - systemd-sysinstall.service drop-in the USB installer boots into. + systemd-sysinstall.service drop-in the USB installer boots into, and the + update units: systemd-sysupdate(-reboot) drop-ins (diskless conditions, the + kured flag, the reboot interlock: operator lock files, failed-update guard, + Kubernetes), the boot deadline and the diskless update check. sources: - kind: local diff --git a/elements/bluefin-server/os-kured-hook.bst b/elements/bluefin-server/os-kured-hook.bst deleted file mode 100644 index 1b91b892..00000000 --- a/elements/bluefin-server/os-kured-hook.bst +++ /dev/null @@ -1,9 +0,0 @@ -kind: import -description: Install systemd-sysupdate post-update hook for Kured coordination. - -sources: - - kind: local - path: files/os/systemd/systemd-sysupdate.service.d - -config: - target: /usr/lib/systemd/system/systemd-sysupdate.service.d diff --git a/elements/bluefin-server/os-stack.bst b/elements/bluefin-server/os-stack.bst index e48b0b68..8a9a8ad7 100644 --- a/elements/bluefin-server/os-stack.bst +++ b/elements/bluefin-server/os-stack.bst @@ -19,6 +19,11 @@ depends: # A/B System Update Configurations - bluefin-server/os-sysupdate.bst + # Diskless update signal (bluefin-diskless-update-check) and the shared + # boot-origin helper it uses + - bluefin-server/os-update-check.bst + - bluefin-server/boot-origin.bst + # k0s component system update configuration - bluefin-server/os-k0s-sysupdate.bst @@ -39,9 +44,6 @@ depends: # First-Boot Credentials Provisioning - bluefin-server/os-creds-prov.bst - # Reboot Coordination for Kured - - bluefin-server/os-kured-hook.bst - # System Management Justfile - bluefin-server/os-justfile.bst diff --git a/elements/bluefin-server/os-update-check.bst b/elements/bluefin-server/os-update-check.bst new file mode 100644 index 00000000..73c1386f --- /dev/null +++ b/elements/bluefin-server/os-update-check.bst @@ -0,0 +1,23 @@ +kind: import +description: | + Update helpers in /usr/libexec (units in files/os/systemd/system, enabled by + 80-bluefin-updates.preset): + - bluefin-diskless-update-check, run by bluefin-diskless-update-check.timer + on diskless boots only: fetches the signed SHA256SUMS from the directory + the node booted from, and when it lists a newer bluefin-server_.raw + than the running IMAGE_VERSION and the boot origin is not pinned to a + versioned file, touches /run/reboot-required for kured or the operator. + It never reboots. + - bluefin-boot-deadline, run by bluefin-boot-deadline.timer on boot-counted + boots only: reboots (or flags kured on a Kubernetes node) when the boot has + not reached boot-complete.target by the deadline, so systemd-boot falls + back to the previous UKI. + - bluefin-update-pending: `systemd-sysupdate pending`, minus a version whose + UKI already used up its boot tries; gates the kured flag and the nightly + reboot so a rolled-back node does not reboot into it again. + Runtime tools: bash, coreutils, curl, gpgv, bootctl, systemd-analyze + (os-base.bst) and bluefin-boot-origin (boot-origin.bst). + +sources: +- kind: local + path: files/os/update-check diff --git a/files/boot-origin/usr/libexec/bluefin-boot-origin b/files/boot-origin/usr/libexec/bluefin-boot-origin new file mode 100755 index 00000000..7215b3a7 --- /dev/null +++ b/files/boot-origin/usr/libexec/bluefin-boot-origin @@ -0,0 +1,61 @@ +#!/usr/bin/bash +# bluefin-boot-origin — print the URL a network-booted node was booted from. +# +# Sources, first match wins: +# 1. the UEFI HTTP boot URL that systemd-stub records in the StubDeviceURL +# EFI variable (what the netboot UKI's `rd.systemd.pull=...bootorigin:` +# resolves against); +# 2. the explicit http(s) source of `rd.systemd.pull=` on the kernel command +# line (a node chainloaded by iPXE, e.g. from Booty); +# 3. the http(s) source of the `import.pull` system credential, when the +# calling unit imports it (ImportCredential=import.pull): a netboot ESP +# with /loader/credentials, or the dogfood VM's SMBIOS credential. +# +# The URL names the file that was booted or pulled; callers use its directory. +# Exits 1, printing nothing, when the node was not network booted. +# +# Shared by bluefin-ignition-credentials (initrd) and +# bluefin-diskless-update-check (booted OS). Bash builtins and coreutils only. +# BLUEFIN_STUB_URL_VAR and BLUEFIN_CMDLINE override the input paths for tests. + +set -euo pipefail +export LC_ALL=C + +stub_var="${BLUEFIN_STUB_URL_VAR:-/sys/firmware/efi/efivars/StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f}" +cmdline="${BLUEFIN_CMDLINE:-/proc/cmdline}" + +# rd.systemd.pull= and import.pull share one syntax: ::. +pull_url() { + case "$1" in + *:http://*|*:https://*) + printf '%s\n' "http${1#*:http}" + return 0 ;; + esac + return 1 +} + +if [ -r "${stub_var}" ]; then + # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. + url="$(tail -c +5 "${stub_var}" | tr -d '\000')" + if [ -n "${url}" ]; then + printf '%s\n' "${url}" + exit 0 + fi +fi + +if [ -r "${cmdline}" ]; then + read -r -a args < "${cmdline}" || true + for arg in "${args[@]}"; do + case "${arg}" in + rd.systemd.pull=*) pull_url "${arg#rd.systemd.pull=}" && exit 0 ;; + esac + done +fi + +cred="${CREDENTIALS_DIRECTORY:-/nonexistent}/import.pull" +if [ -r "${cred}" ]; then + read -r line < "${cred}" || true + pull_url "${line:-}" && exit 0 +fi + +exit 1 diff --git a/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials b/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials index 9ddaecb3..42cbb6ba 100755 --- a/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials +++ b/files/initrd-ignition/usr/libexec/bluefin-ignition-credentials @@ -16,14 +16,13 @@ # # If both are set, ignition.config wins and ignition.config.url is ignored. # -# Without either credential, a node that was UEFI HTTP booted (systemd-stub -# records the boot URL in the StubDeviceURL EFI variable) looks for -# bluefin-node.ign next to its UKI, the same way the initrd derives the /usr -# image URL. A node chainloaded by iPXE has no StubDeviceURL; its boot server -# (Booty) passes rd.systemd.pull with an explicit http(s) URL instead, and the -# config is looked up in that URL's directory. If the boot server has one -# (HTTP HEAD succeeds), it is used through config.replace; otherwise Ignition -# is skipped. +# Without either credential, a network-booted node looks for bluefin-node.ign +# in the directory it booted from, as reported by /usr/libexec/bluefin-boot-origin: +# the UEFI HTTP boot URL (StubDeviceURL EFI variable), the same way the initrd +# derives the /usr image URL, or, for a node chainloaded by iPXE, the explicit +# http(s) URL its boot server (Booty) passes in rd.systemd.pull. If the boot +# server has one (HTTP HEAD succeeds), it is used through config.replace; +# otherwise Ignition is skipped. # # Output: /run/ignition/user.ign (absent when there is nothing to apply), the highest-priority user config path that # Ignition's "system" provider reads before the platform provider @@ -48,31 +47,12 @@ log() { printf '%s: %s\n' "${PROG}" "$*" >&2; } die() { log "$*"; exit 1; } NODE_CONFIG=bluefin-node.ign -STUB_URL_VAR=/sys/firmware/efi/efivars/StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f +BOOT_ORIGIN="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}" creds="${CREDENTIALS_DIRECTORY:-/nonexistent}" config="${creds}/ignition.config" url_cred="${creds}/ignition.config.url" -# The URL the node booted from: the UEFI HTTP boot URL recorded by -# systemd-stub, else the explicit http(s) source of rd.systemd.pull=. -origin_url() { - if [ -r "${STUB_URL_VAR}" ]; then - # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. - tail -c +5 "${STUB_URL_VAR}" | tr -d '\000' - return 0 - fi - local arg - for arg in $(+3-0.efi). systemd-boot uses up one try each time it +# boots the entry, and systemd-bless-boot drops the counter once +# boot-complete.target is reached, which systemd-boot-check-no-failures.service +# allows only when no unit failed. A boot that never gets there keeps its +# counter, but systemd-boot only moves on at the next boot, and nothing else +# reboots a node that is already running the newest installed version. +# +# bluefin-boot-deadline.timer runs this once the deadline passes (OnBootSec=, +# 15 min unless a drop-in changes it), on boot-counted boots only. If the boot +# is still not blessed: +# +# - /run/reboot-lock or /etc/reboot-lock: the operator holds reboots (the +# same files stop systemd-sysupdate-reboot.service); log and stay up; +# - kubelet or k0s running: the node may carry workloads, so do what the +# update path does and touch /run/reboot-required for kured to cordon, +# drain and reboot it within its cluster lock; +# - otherwise reboot, so systemd-boot uses up the next try and, after the +# last one, falls back to the previous UKI. A Kubernetes unit that is +# stopped or failed leaves the node NotReady with nothing to drain, and +# kured cannot run there either, so that node reboots directly. +# +# The last try boots as "dirty" (no tries left). It reboots only when another +# UKI still has tries or no counter, so a node with nothing to fall back to +# does not reboot in a loop. +# +# Journal priorities come from SyslogLevelPrefix: <3> err, <4> warning, +# <6> info. +# +# Test overrides: BLUEFIN_BLESS_BOOT, BLUEFIN_BOOT_PATH, BLUEFIN_OS_RELEASE, +# BLUEFIN_REBOOT_LOCKS (space-separated), BLUEFIN_REBOOT_SENTINEL. + +set -euo pipefail +export LC_ALL=C + +bless="${BLUEFIN_BLESS_BOOT:-/usr/lib/systemd/systemd-bless-boot}" +sentinel="${BLUEFIN_REBOOT_SENTINEL:-/run/reboot-required}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +read -r -a locks <<< "${BLUEFIN_REBOOT_LOCKS:-/run/reboot-lock /etc/reboot-lock}" + +info() { printf '<6>%s\n' "$*"; } +warn() { printf '<4>%s\n' "$*"; } +err() { printf '<3>%s\n' "$*"; } + +# Another UKI systemd-boot would pick over this one: no counter, or tries left. +fallback() { + local boot uki + boot="${BLUEFIN_BOOT_PATH:-$(bootctl --print-boot-path 2>/dev/null || echo /boot)}" + for uki in "${boot}"/EFI/Linux/*.efi; do + [ -e "${uki}" ] || continue + case "${uki##*/}" in + *+0.efi|*+0-*.efi) ;; + *) return 0 ;; + esac + done + return 1 +} + +status="$("${bless}" status 2>/dev/null)" || status="" +case "${status}" in + indeterminate|bad) ;; + dirty) + if ! fallback; then + warn "boot entry has no tries left and no other UKI to fall back to; not rebooting" + exit 0 + fi ;; + *) + info "boot is ${status:-not boot-counted}; nothing to do" + exit 0 ;; +esac + +if systemctl is-active --quiet boot-complete.target; then + info "boot-complete.target reached; systemd-bless-boot handles the entry" + exit 0 +fi + +version="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + version="${value}" + done < "${os_release}" +fi + +failed=() +while read -r unit _; do + [ -n "${unit}" ] && failed+=("${unit}") +done < <(systemctl list-units --state=failed --plain --no-legend --no-pager 2>/dev/null || true) +if [ "${#failed[@]}" -gt 0 ]; then + why="failed: ${failed[*]}" +else + why="no unit failed, but the boot never got there" +fi +msg="Bluefin Server ${version:-(unknown version)} did not reach boot-complete.target before the boot deadline (${why}); its boot entry is ${status}" + +for lock in "${locks[@]}"; do + if [ -e "${lock}" ]; then + warn "${msg}; ${lock} holds reboots, not rebooting" + exit 0 + fi +done + +for u in kubelet.service k0scontroller.service k0sworker.service; do + s="$(systemctl is-active "${u}" 2>/dev/null)" || true + case "${s}" in + active|activating|deactivating|reloading|refreshing) + mkdir -p "${sentinel%/*}" + touch "${sentinel}" + warn "${msg}; ${u} is ${s}: flagged ${sentinel} for kured to drain and reboot" + exit 0 ;; + esac +done + +err "${msg}; rebooting so systemd-boot falls back after the last try" +exec systemctl --no-block reboot diff --git a/files/os/update-check/usr/libexec/bluefin-diskless-update-check b/files/os/update-check/usr/libexec/bluefin-diskless-update-check new file mode 100755 index 00000000..e70e36bd --- /dev/null +++ b/files/os/update-check/usr/libexec/bluefin-diskless-update-check @@ -0,0 +1,122 @@ +#!/usr/bin/bash +# bluefin-diskless-update-check — flag a diskless node whose boot server +# offers a newer Bluefin Server release. +# +# A diskless node runs the OS DDI it pulled into RAM at boot, and it updates +# by rebooting into whatever its boot server serves at that point. This check +# tells the node when a reboot would bring a newer image: +# +# 1. find the directory the node booted from (bluefin-boot-origin); +# 2. fetch SHA256SUMS and SHA256SUMS.gpg from it and verify the signature +# with gpgv against the image keyring. This is the same trust root as the +# initrd's `rd.systemd.pull ... verify=signature`; +# 3. read the release version from the manifest's bluefin-server_.raw +# entry and compare it with the running IMAGE_VERSION using +# `systemd-analyze compare-versions`, the ordering systemd-sysupdate uses; +# 4. if the release is newer and the next boot would pick it up, touch +# /run/reboot-required (the sentinel kured watches) and log the new +# version. +# +# Whether the next boot picks it up depends on what the origin names. A +# netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi, +# found through StubDeviceURL) resolves the image at boot: the server swaps the +# UKI and the next boot pulls the newer image next to it. An origin that names +# a versioned file, bluefin-server_.raw (an explicit rd.systemd.pull= URL +# or the import.pull credential) or bluefin-server-netboot_.efi, is +# pinned: the node pulls the same version again after a reboot. Flagging it +# would make kured drain and reboot the node on every check, so a pinned node +# only logs that a newer release is available. +# +# It never reboots. Network and signature errors are logged and leave the +# flag untouched, and the check always exits 0, so a boot server that cannot +# be reached never fails a unit. +# +# Journal priorities come from SyslogLevelPrefix: <4> warning, <5> notice, +# <6> info. +# +# Test overrides: BLUEFIN_BOOT_ORIGIN (the origin helper), +# BLUEFIN_IMPORT_KEYRING, BLUEFIN_REBOOT_SENTINEL, BLUEFIN_OS_RELEASE, and +# BLUEFIN_CMDLINE / BLUEFIN_STUB_URL_VAR, which bluefin-boot-origin reads. + +set -euo pipefail +export LC_ALL=C + +boot_origin="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}" +sentinel="${BLUEFIN_REBOOT_SENTINEL:-/run/reboot-required}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +# systemd reads /etc/systemd/import-pubring.pgp before the vendor keyring. +keyring="${BLUEFIN_IMPORT_KEYRING:-}" +if [ -z "${keyring}" ]; then + keyring=/usr/lib/systemd/import-pubring.pgp + [ -e /etc/systemd/import-pubring.pgp ] && keyring=/etc/systemd/import-pubring.pgp +fi + +info() { printf '<6>%s\n' "$*"; } +notice() { printf '<5>%s\n' "$*"; } +warn() { printf '<4>%s\n' "$*"; } + +running="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + running="${value}" + done < "${os_release}" +fi +if [ -z "${running}" ]; then + warn "no IMAGE_VERSION in ${os_release}; not checking for updates" + exit 0 +fi + +if ! origin="$("${boot_origin}")"; then + info "not network booted (no boot origin); nothing to check" + exit 0 +fi +base="${origin%/*}" +file="${origin##*/}" +pinned="" +[[ "${file}" =~ ^bluefin-server(-netboot)?_[0-9A-Za-z.]+\.(raw|efi)$ ]] && pinned="${file}" + +tmp="$(mktemp -d)" +trap 'rm -rf "${tmp}"' EXIT + +fetch() { + curl -fsS --proto '=http,https' --retry 2 --connect-timeout 10 --max-time 60 \ + --max-filesize 1048576 -o "${tmp}/$1" "${base}/$1" +} +if ! fetch SHA256SUMS || ! fetch SHA256SUMS.gpg; then + warn "cannot fetch SHA256SUMS and SHA256SUMS.gpg from ${base}/; not flagging an update" + exit 0 +fi + +mkdir -m 0700 "${tmp}/gnupg" +if ! GNUPGHOME="${tmp}/gnupg" gpgv --keyring "${keyring}" \ + "${tmp}/SHA256SUMS.gpg" "${tmp}/SHA256SUMS"; then + warn "SHA256SUMS from ${base}/ does not verify against ${keyring}; not flagging an update" + exit 0 +fi + +offered="" +while read -r _ name; do + name="${name#\*}" + [[ "${name}" =~ ^bluefin-server_([0-9A-Za-z.]+)\.raw$ ]] || continue + version="${BASH_REMATCH[1]}" + if [ -z "${offered}" ] || + systemd-analyze compare-versions "${version}" gt "${offered}" >/dev/null; then + offered="${version}" + fi +done < "${tmp}/SHA256SUMS" +if [ -z "${offered}" ]; then + warn "SHA256SUMS from ${base}/ lists no bluefin-server_.raw; not flagging an update" + exit 0 +fi + +if ! systemd-analyze compare-versions "${offered}" gt "${running}" >/dev/null; then + info "up to date: ${base}/ offers ${offered}, running ${running}" +elif [ -n "${pinned}" ]; then + notice "Bluefin Server ${offered} is available from ${base}/ (running ${running}), but this node is pinned to ${pinned}; not flagging ${sentinel}, point its boot origin at the new release to update" +else + mkdir -p "${sentinel%/*}" + touch "${sentinel}" + notice "Bluefin Server ${offered} is available from ${base}/ (running ${running}); flagged ${sentinel}, reboot to update" +fi diff --git a/files/os/update-check/usr/libexec/bluefin-update-pending b/files/os/update-check/usr/libexec/bluefin-update-pending new file mode 100755 index 00000000..086493f8 --- /dev/null +++ b/files/os/update-check/usr/libexec/bluefin-update-pending @@ -0,0 +1,50 @@ +#!/usr/bin/bash +# bluefin-update-pending — is there a staged update worth rebooting into? +# +# Exit 0 when `systemd-sysupdate pending` reports a newer installed version +# than the booted one AND that version has not already failed its boot tries. +# Exit 1 otherwise. +# +# A version that failed its tries keeps its UKI as +# bluefin-server-+0-.efi: systemd-boot sorts it last and boots the +# previous one, so a reboot lands on the booted version again. systemd-sysupdate +# does not look at the counter and still reports it as pending, so without this +# check a node that rolled back would flag kured after every update run, or +# reboot in every nightly window, and never leave the version it is on. The +# next newer release takes the failed version's slot and UKI (InstancesMax=2) +# and clears the block. +# +# Used by systemd-sysupdate.service.d/20-kured.conf (the kured flag) and +# systemd-sysupdate-reboot.service.d/20-interlock.conf (the local reboot). +# +# Test overrides: BLUEFIN_SYSUPDATE, BLUEFIN_BOOT_PATH, BLUEFIN_OS_RELEASE. + +set -euo pipefail +export LC_ALL=C + +sysupdate="${BLUEFIN_SYSUPDATE:-/usr/bin/systemd-sysupdate}" +os_release="${BLUEFIN_OS_RELEASE:-/usr/lib/os-release}" +boot="${BLUEFIN_BOOT_PATH:-$(bootctl --print-boot-path 2>/dev/null || echo /boot)}" + +"${sysupdate}" pending || exit 1 + +booted="" +if [ -r "${os_release}" ]; then + while IFS='=' read -r key value; do + [ "${key}" = IMAGE_VERSION ] || continue + value="${value%\"}"; value="${value#\"}" + booted="${value}" + done < "${os_release}" +fi +[ -n "${booted}" ] || exit 0 + +for uki in "${boot}"/EFI/Linux/bluefin-server-*+0-*.efi "${boot}"/EFI/Linux/bluefin-server-*+0.efi; do + [ -e "${uki}" ] || continue + name="${uki##*/bluefin-server-}" + version="${name%%+*}" + if systemd-analyze compare-versions "${version}" gt "${booted}" >/dev/null; then + echo "Bluefin Server ${version} is installed but failed its boot tries (${uki##*/}); staying on ${booted} until a newer release replaces it" + exit 1 + fi +done +exit 0 diff --git a/scripts/dogfood-diskless.sh b/scripts/dogfood-diskless.sh index 0802bf93..8e46de17 100644 --- a/scripts/dogfood-diskless.sh +++ b/scripts/dogfood-diskless.sh @@ -144,6 +144,12 @@ echo "PROBE root-passwd=$(passwd -S root 2>&1 | cut -d' ' -f2)" boots=$(( $(cat /var/lib/dogfood-boots 2>/dev/null || echo 0) + 1 )) echo "${boots}" > /var/lib/dogfood-boots echo "PROBE boots=${boots}" +echo "PROBE update-timers=$(systemctl is-active systemd-sysupdate.timer systemd-sysupdate-reboot.timer bluefin-diskless-update-check.timer | tr '\n' ' ')" +if [ -e /run/machines/rootdisk.raw ]; then + systemctl start bluefin-diskless-update-check.service || true + echo "PROBE update-check=$(systemctl show -P Result bluefin-diskless-update-check.service) flag=$(test -e /run/reboot-required && echo set || echo none)" + journalctl -b -o cat --no-pager -u bluefin-diskless-update-check.service | grep -v '^gpgv:' | tail -n 3 | sed 's/^/PROBE-LOG /' +fi PROBE [ -n "${DOGFOOD_EXTRA_PROBE:-}" ] && cat "${DOGFOOD_EXTRA_PROBE}" >> "${work}/probe.sh" echo 'sync' >> "${work}/probe.sh" diff --git a/scripts/dogfood-install.sh b/scripts/dogfood-install.sh index 92cd1e7f..009813a1 100644 --- a/scripts/dogfood-install.sh +++ b/scripts/dogfood-install.sh @@ -4,12 +4,24 @@ # 2. boot the installed disk (usr slot A, persistent root) # 3. with : systemd-sysupdate to that version over HTTP, reboot, # and confirm the node runs it from slot B with a boot-counted UKI -# 4. with : update to it, corrupt its slot, and confirm boot -# counting rolls the node back to on its own -# The updates verify the signed SHA256SUMS and have the optional "zfs" -# sysupdate feature enabled, so the ZFS sysext follows the OS in lock-step -# and must still be active after the rollback. -# Usage: dogfood-install.sh [ []] +# 4. with : update to it, break it, and confirm boot counting +# rolls the node back to on its own. DOGFOOD_BROKEN=slot +# (default) corrupts its usr slot, so the initrd fails; DOGFOOD_BROKEN=unit +# adds a unit that fails on that version only, so the boot reaches +# multi-user.target but never boot-complete.target, and +# bluefin-boot-deadline.timer (shortened to DOGFOOD_DEADLINE, default 2min) +# must reboot it until systemd-boot falls back. After the fallback the +# failed version must not count as pending: no kured flag, and the +# nightly reboot unit is skipped. +# The updates run through systemd-sysupdate.service, the unit the +# preset-enabled timer starts, verify the signed SHA256SUMS and have the +# optional "zfs" sysupdate feature enabled, so the ZFS sysext follows the OS +# in lock-step and must still be active after the rollback. Once an update is +# staged the kured flag must be set, and the reboot unit must stand down while +# a stand-in kubelet.service runs. Every disk boot has both update timers +# enabled; a boot-counted UKI is blessed only after boot-complete.target, +# which requires that no unit failed. +# Usage: [DOGFOOD_BROKEN=slot|unit] dogfood-install.sh [ []] set -euo pipefail here="$(cd "$(dirname "$0")" && pwd)" @@ -45,6 +57,9 @@ echo "PROBE firstboot-ran=$(systemctl show -P ConditionResult systemd-firstboot. bootctl list --no-pager 2>/dev/null | sed -n 's/^ *\(title\|id\): */PROBE-LOG \1 /p' systemctl start boot-complete.target 2>/dev/null || true echo "PROBE ukis=$(ls /boot/EFI/Linux 2>/dev/null | tr '\n' ' ')" +echo "PROBE timers-enabled=$(systemctl is-enabled systemd-sysupdate.timer systemd-sysupdate-reboot.timer | tr '\n' ' ')" +echo "PROBE health=$(systemctl is-active systemd-boot-check-no-failures.service boot-complete.target | tr '\n' ' ')bless=$(/usr/lib/systemd/systemd-bless-boot status 2>/dev/null)" +echo "PROBE deadline=$(systemctl is-active bluefin-boot-deadline.timer) $(systemctl show -P ConditionResult bluefin-boot-deadline.timer)" EOF cat > "${state}/update.probe" <<'EOF' @@ -54,11 +69,17 @@ for f in /usr/lib/sysupdate.d/*.transfer; do sed -e 's|^Path=https://.*|Path=http://10.0.2.2:8765/|' \ "${f}" > "/etc/sysupdate.d/${f##*/}" done +rm -f /run/reboot-required rc=0 -systemd-sysupdate update > /run/sysupdate.log 2>&1 || rc=$? +systemctl start --wait systemd-sysupdate.service || rc=$? echo "PROBE update=${rc}" -tail -n 15 /run/sysupdate.log | sed 's/^/PROBE-LOG /' +journalctl -b -o cat --no-pager -u systemd-sysupdate.service | tail -n 15 | sed 's/^/PROBE-LOG /' timeout 60 systemd-sysupdate list --no-pager 2>&1 | sed 's/^/PROBE-LOG /' +echo "PROBE kured-flag=$(test -e /run/reboot-required && echo set || echo none)" +systemd-run --quiet --unit=kubelet.service sleep 600 +systemctl start systemd-sysupdate-reboot.service +echo "PROBE interlock=$(systemctl show -P Result systemd-sysupdate-reboot.service)" +systemctl stop kubelet.service EOF echo "==> 1/4 diskless boot + systemd-sysinstall" @@ -68,31 +89,91 @@ grep -q 'PROBE install=0' "${state}/1-install.log" echo "==> 2/4 boot the installed disk" DOGFOOD_BOOT=disk run "${dir}" "${state}/disk.probe" | tee "${state}/2-disk.log" grep -q 'PROBE root=xfs' "${state}/2-disk.log" +grep -q 'PROBE timers-enabled=enabled enabled' "${state}/2-disk.log" +grep -q 'PROBE update-timers=active active inactive' "${state}/2-disk.log" [ -n "${next}" ] || { echo "PASS: installed and booted from disk"; exit 0; } echo "==> 3/4 systemd-sysupdate to $(basename "${next}")" DOGFOOD_TIMEOUT="${DOGFOOD_UPDATE_TIMEOUT:-900}" DOGFOOD_BOOT=disk run "${next}" "${state}/update.probe" | tee "${state}/3-update.log" grep -q 'PROBE update=0' "${state}/3-update.log" +grep -q 'PROBE kured-flag=set' "${state}/3-update.log" +grep -q 'PROBE interlock=exec-condition' "${state}/3-update.log" echo "==> 4/4 boot the updated disk" DOGFOOD_BOOT=disk run "${next}" "${state}/disk.probe" | tee "${state}/4-updated.log" new_ver="$(ls "${next}"/bluefin-server-[0-9]*.efi | sed -n 's|.*/bluefin-server-\(.*\)\.efi$|\1|p')" grep -q "PROBE os=bluefin-server ${new_ver}" "${state}/4-updated.log" grep -q "PROBE zfs=active" "${state}/4-updated.log" +grep -q "PROBE health=active active bless=good" "${state}/4-updated.log" [ -n "${broken}" ] || { echo "PASS: installed, updated A->B and booted ${new_ver}"; exit 0; } bad_ver="$(ls "${broken}"/bluefin-server-[0-9]*.efi | sed -n 's|.*/bluefin-server-\(.*\)\.efi$|\1|p')" -sed "s|^echo \"PROBE update=|dd if=/dev/urandom of=/dev/disk/by-partlabel/bluefin_usr_${bad_ver} bs=1M count=64 conv=fsync 2>/dev/null \&\& echo PROBE corrupted=${bad_ver}\necho \"PROBE update=|" \ - "${state}/update.probe" > "${state}/break.probe" +mode="${DOGFOOD_BROKEN:-slot}" +case "${mode}" in + slot) + sed "s|^echo \"PROBE update=|dd if=/dev/urandom of=/dev/disk/by-partlabel/bluefin_usr_${bad_ver} bs=1M count=64 conv=fsync 2>/dev/null \&\& echo PROBE corrupted=${bad_ver}\necho \"PROBE update=|" \ + "${state}/update.probe" > "${state}/break.probe" + cp "${state}/disk.probe" "${state}/rollback.probe" ;; + unit) + # A unit that fails on ${bad_ver} only, and a deadline short enough + # for three tries in one run. + { cat "${state}/update.probe"; cat < "${state}/break.probe" +cat > /etc/systemd/system/dogfood-broken.service <<'UNIT' +[Unit] +Description=Dogfood: fail on ${bad_ver} only +ConditionOSRelease=IMAGE_VERSION=${bad_ver} +[Service] +Type=oneshot +ExecStart=/usr/bin/false +[Install] +WantedBy=multi-user.target +UNIT +systemctl enable dogfood-broken.service 2>/dev/null +mkdir -p /etc/systemd/system/bluefin-boot-deadline.timer.d +printf '[Timer]\nOnBootSec=\nOnBootSec=${DOGFOOD_DEADLINE:-2min}\n' > /etc/systemd/system/bluefin-boot-deadline.timer.d/dogfood.conf +echo "PROBE corrupted=${bad_ver} (dogfood-broken.service)" +BREAK + # A broken boot reports the gate and waits for the deadline to reboot + # it; only the fallback boot finishes the probe, and reads the + # deadline's log of every try from the persistent journal. + { cat < "${state}/rollback.probe" +if [ "\$(. /usr/lib/os-release; echo "\${IMAGE_VERSION}")" = ${bad_ver} ]; then + echo "PROBE counted-boot=${bad_ver} bless=\$(/usr/lib/systemd/systemd-bless-boot status) failed=\$(systemctl --failed --no-legend --plain | cut -d' ' -f1 | tr '\n' ' ')deadline=\$(systemctl is-active bluefin-boot-deadline.timer)" + exec sleep infinity +fi +GATE +journalctl -o cat --no-pager -u bluefin-boot-deadline.service | grep 'boot deadline' | sed 's/^/PROBE-LOG /' +rm -f /run/reboot-required +systemctl start --wait systemd-sysupdate.service || true +journalctl -b -o cat --no-pager -u systemd-sysupdate.service | grep 'failed its boot tries' | sed 's/^/PROBE-LOG /' +echo "PROBE after-rollback kured-flag=$(test -e /run/reboot-required && echo set || echo none)" +systemctl start systemd-sysupdate-reboot.service +echo "PROBE after-rollback reboot-unit=$(systemctl show -P Result systemd-sysupdate-reboot.service)" +AFTER + ;; + *) echo "ERROR: DOGFOOD_BROKEN must be slot or unit" >&2; exit 1 ;; +esac -echo "==> 5/6 update to ${bad_ver} and corrupt its slot" +echo "==> 5/6 update to ${bad_ver} and break it (${mode})" DOGFOOD_TIMEOUT="${DOGFOOD_UPDATE_TIMEOUT:-900}" DOGFOOD_BOOT=disk run "${broken}" "${state}/break.probe" | tee "${state}/5-break.log" grep -q "PROBE corrupted=${bad_ver}" "${state}/5-break.log" echo "==> 6/6 boot: ${bad_ver} must fail its tries and fall back to ${new_ver}" -DOGFOOD_TIMEOUT="${DOGFOOD_ROLLBACK_TIMEOUT:-1500}" DOGFOOD_BOOT=disk run "${broken}" "${state}/disk.probe" | tee "${state}/6-rollback.log" +# In unit mode the broken boots log [FAILED] for the deliberately failed unit, +# so the fallback boot is judged by its own probe below. +DOGFOOD_TIMEOUT="${DOGFOOD_ROLLBACK_TIMEOUT:-1800}" DOGFOOD_BOOT=disk run "${broken}" "${state}/rollback.probe" \ + | tee "${state}/6-rollback.log" || [ "${mode}" = unit ] grep -q "PROBE os=bluefin-server ${new_ver}" "${state}/6-rollback.log" grep -q "bluefin-server-${bad_ver}+0-3.efi" "${state}/6-rollback.log" grep -q "PROBE zfs=active" "${state}/6-rollback.log" -echo "PASS: installed, updated A->B, and rolled back from a broken ${bad_ver} to ${new_ver}" +if [ "${mode}" = unit ]; then + [ "$(grep -Ec "PROBE counted-boot=${bad_ver} bless=(indeterminate|dirty) failed=dogfood-broken.service .*deadline=active" "${state}/6-rollback.log")" = 3 ] + [ "$(grep -c "PROBE-LOG Bluefin Server ${bad_ver} did not reach boot-complete.target .*rebooting so systemd-boot falls back" "${state}/6-rollback.log")" = 3 ] + grep -q "PROBE deadline=inactive no" "${state}/6-rollback.log" + grep -q "PROBE health=active active bless=clean" "${state}/6-rollback.log" + grep -q "PROBE after-rollback kured-flag=none" "${state}/6-rollback.log" + grep -q "PROBE after-rollback reboot-unit=exec-condition" "${state}/6-rollback.log" + grep -q "PROBE failed=0" "${state}/6-rollback.log" +fi +echo "PASS: installed, updated A->B, and rolled back from a broken (${mode}) ${bad_ver} to ${new_ver}" diff --git a/tests/unit/test_automatic_updates.py b/tests/unit/test_automatic_updates.py new file mode 100644 index 00000000..20fe7684 --- /dev/null +++ b/tests/unit/test_automatic_updates.py @@ -0,0 +1,189 @@ +"""Contracts for automatic updates on installed nodes and the boot health gate. + +Installed nodes run systemd-sysupdate.timer and systemd-sysupdate-reboot.timer +by vendor preset; Kubernetes nodes leave the reboot to kured; diskless and USB +installer boots keep every update unit inert; operator lock files hold the +local reboot; boot-complete.target requires that no unit failed before +systemd-bless-boot marks a counted UKI good. +""" + +from __future__ import annotations + +import configparser +import os +import re +import shlex +import stat +import subprocess +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +PRESETS = ROOT / "files" / "os" / "systemd" / "system-preset" +PRESET = PRESETS / "80-bluefin-updates.preset" +ELEMENTS = ROOT / "elements" / "bluefin-server" +DISKLESS = UNITS / "systemd-sysupdate.service.d" / "10-diskless.conf" +KURED = UNITS / "systemd-sysupdate.service.d" / "20-kured.conf" +INTERLOCK = UNITS / "systemd-sysupdate-reboot.service.d" / "20-interlock.conf" +DISKLESS_ONLY = "/run/machines/rootdisk.raw" + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def values(path: Path, key: str) -> list[str]: + """Every value of in order; configparser keeps only the last one.""" + return [ + line.split("=", 1)[1] + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith(f"{key}=") + ] + + +def conditions(path: Path) -> list[str]: + return [ + line.strip() + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith("Condition") + ] + + +def preset_lines() -> list[list[str]]: + return [ + line.split() + for line in PRESET.read_text(encoding="utf-8").splitlines() + if line.strip() and not line.startswith("#") + ] + + +def test_preset_enables_the_update_timers_and_the_health_gate() -> None: + assert preset_lines() == [ + ["enable", "systemd-sysupdate.timer"], + ["enable", "systemd-sysupdate-reboot.timer"], + ["enable", "systemd-boot-check-no-failures.service"], + ["enable", "bluefin-boot-deadline.timer"], + ["enable", "bluefin-diskless-update-check.timer"], + ] + + +def test_preset_overrides_fsdk_and_yields_to_ignition() -> None: + # FSDK's 90-sysupdate.preset disables both timers; the first match wins. + assert "20-ignition.preset" < PRESET.name < "90-sysupdate.preset" + assert PRESET.name < "90-systemd.preset" + + +def test_no_base_image_preset_enables_kubernetes_or_zfs() -> None: + for preset in PRESETS.glob("*.preset"): + for line in preset.read_text(encoding="utf-8").splitlines(): + if line.startswith("enable"): + assert not re.search(r"kubelet|containerd|k0s|zfs", line), (preset, line) + + +@pytest.mark.parametrize( + "dropin", + [ + "systemd-sysupdate-reboot.service.d/10-diskless.conf", + "systemd-sysupdate.timer.d/10-diskless.conf", + "systemd-sysupdate-reboot.timer.d/10-diskless.conf", + ], +) +def test_update_units_are_inert_on_diskless_and_installer_boots(dropin: str) -> None: + expected = conditions(DISKLESS) + assert expected == [ + f"ConditionPathExists=!{DISKLESS_ONLY}", + "ConditionKernelCommandLine=!root=tmpfs", + ] + assert conditions(UNITS / dropin) == expected + + +def test_kured_flag_is_set_only_when_an_update_is_pending() -> None: + service = ini(KURED)["Service"] + assert "ExecStartPost" not in service, "Type=simple: ExecStartPost runs before the update" + cmd = service["ExecStopPost"] + # bluefin-update-pending wraps `systemd-sysupdate pending` and says no + # when that version already failed its boot tries (test_update_pending.py). + assert cmd == "/usr/bin/sh -c 'if /usr/libexec/bluefin-update-pending; then touch /run/reboot-required; fi'" + for dropin in (UNITS / "systemd-sysupdate.service.d").iterdir(): + keys = {key for section in ini(dropin).values() for key in section} + assert "ExecStartPost" not in keys, dropin + # The timer runs FSDK's `systemd-sysupdate update`, which also fetches + # every enabled feature (zfs, kubestellar, kubeadm) in lock-step. + assert "ExecStart" not in keys, dropin + + +def test_reboot_is_held_by_the_operator_lock_files() -> None: + # From projectbluefin/server#182: a failed condition skips the unit. + assert conditions(INTERLOCK) == [ + "ConditionPathExists=!/run/reboot-lock", + "ConditionPathExists=!/etc/reboot-lock", + ] + # One interlock drop-in: #182's reboot-coordination.conf must not come back. + assert sorted(p.name for p in INTERLOCK.parent.iterdir()) == ["10-diskless.conf", "20-interlock.conf"] + + +def test_reboot_skips_an_update_that_already_failed_its_tries() -> None: + conds = values(INTERLOCK, "ExecCondition") + assert len(conds) == 2 + assert conds[0] == "/usr/libexec/bluefin-update-pending" + + +def _interlock_script() -> str: + cmd = values(INTERLOCK, "ExecCondition")[-1] + argv = shlex.split(cmd.replace("$$", "$")) + assert argv[:2] == ["/usr/bin/sh", "-c"] and len(argv) == 3, argv + return argv[2] + + +def _run_interlock(tmp_path: Path, states: dict[str, str]) -> subprocess.CompletedProcess[str]: + bindir = tmp_path / "bin" + bindir.mkdir(exist_ok=True) + fake = bindir / "systemctl" + lines = ["#!/bin/sh", '[ "$1" = is-active ] || exit 99', 'case "$2" in'] + for unit, state in states.items(): + lines.append(f' {unit}) echo {state}; [ {state} = active ]; exit $? ;;') + lines += ["esac", "echo inactive", "exit 3"] + fake.write_text("\n".join(lines) + "\n", encoding="utf-8") + fake.chmod(fake.stat().st_mode | stat.S_IXUSR) + env = dict(os.environ, PATH=f"{bindir}:{os.environ['PATH']}") + return subprocess.run(["sh", "-c", _interlock_script()], capture_output=True, text=True, env=env) + + +def test_reboot_proceeds_without_kubernetes(tmp_path: Path) -> None: + result = _run_interlock(tmp_path, {}) + assert result.returncode == 0, result.stdout + result.stderr + + +@pytest.mark.parametrize( + "unit,state", + [ + ("kubelet.service", "active"), + ("k0scontroller.service", "active"), + ("k0sworker.service", "active"), + ("kubelet.service", "activating"), + ("k0sworker.service", "deactivating"), + ], +) +def test_reboot_stands_down_while_kubernetes_runs(tmp_path: Path, unit: str, state: str) -> None: + result = _run_interlock(tmp_path, {unit: state}) + # 1..254 makes systemd skip the unit instead of failing it. + assert result.returncode == 1, result.stdout + result.stderr + assert "leaving the reboot to kured" in result.stdout + + +def test_reboot_proceeds_when_kubernetes_is_stopped_or_failed(tmp_path: Path) -> None: + result = _run_interlock(tmp_path, {"kubelet.service": "failed", "k0sworker.service": "inactive"}) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_kured_hook_moved_into_the_unit_directory() -> None: + stack = yaml.safe_load((ELEMENTS / "os-stack.bst").read_text(encoding="utf-8"))["depends"] + assert "bluefin-server/os-k0s-first-boot.bst" in stack, "installs files/os/systemd/system" + assert "bluefin-server/os-kured-hook.bst" not in stack + assert not (ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d").exists() diff --git a/tests/unit/test_boot_deadline.py b/tests/unit/test_boot_deadline.py new file mode 100644 index 00000000..381e4306 --- /dev/null +++ b/tests/unit/test_boot_deadline.py @@ -0,0 +1,199 @@ +"""Executed coverage for the rollback half of the boot health gate. + +bluefin-boot-deadline reboots a boot-counted boot that has not reached +boot-complete.target by its deadline, so systemd-boot uses up a try and falls +back. It runs here against fake systemd tools. +""" + +from __future__ import annotations + +import configparser +import os +import stat +import subprocess +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +LIBEXEC = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" +DEADLINE = LIBEXEC / "bluefin-boot-deadline" +COUNTED = "/sys/firmware/efi/efivars/LoaderBootCountPath-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" +BOOTED = "26.09.3" + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def conditions(path: Path) -> list[str]: + return [line for line in path.read_text(encoding="utf-8").splitlines() if line.startswith("Condition")] + + +def executable(path: Path, body: str) -> None: + path.write_text("#!/bin/sh\n" + body, encoding="utf-8") + path.chmod(path.stat().st_mode | stat.S_IXUSR) + + +@pytest.mark.parametrize("unit", ["bluefin-boot-deadline.timer", "bluefin-boot-deadline.service"]) +def test_deadline_runs_only_on_boot_counted_disk_boots(unit: str) -> None: + assert conditions(UNITS / unit) == [ + f"ConditionPathExists={COUNTED}", + "ConditionPathExists=!/run/machines/rootdisk.raw", + "ConditionKernelCommandLine=!root=tmpfs", + ] + + +def test_deadline_timer_defaults_to_fifteen_minutes() -> None: + timer = ini(UNITS / "bluefin-boot-deadline.timer") + assert timer["Timer"]["OnBootSec"] == "15min" + assert timer["Install"]["WantedBy"] == "timers.target" + + +def test_deadline_fires_even_when_the_boot_hangs() -> None: + unit = ini(UNITS / "bluefin-boot-deadline.service")["Unit"] + assert unit["DefaultDependencies"] == "no" + assert unit["After"] == "sysinit.target" + for target in ("basic.target", "multi-user.target", "boot-complete.target"): + assert target not in unit.get("After", ""), target + assert target not in unit.get("Requires", ""), target + service = ini(UNITS / "bluefin-boot-deadline.service")["Service"] + assert service["ExecStart"] == "/usr/libexec/bluefin-boot-deadline" + + +def test_helpers_ship_in_the_os_and_are_executable_bash() -> None: + stack = yaml.safe_load((ROOT / "elements" / "bluefin-server" / "os-stack.bst").read_text(encoding="utf-8")) + assert "bluefin-server/os-update-check.bst" in stack["depends"] + for script in (DEADLINE,): + assert script.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert script.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(script)], check=True) + + +def deadline( + tmp_path: Path, + status: str, + *, + complete: bool = False, + failed: tuple[str, ...] = ("dogfood-broken.service",), + units: dict[str, str] | None = None, + ukis: tuple[str, ...] = (), + lock: bool = False, +): + bindir = tmp_path / "bin" + bindir.mkdir() + calls = tmp_path / "systemctl.calls" + units = units or {} + unit_cases = "".join(f" {u}) echo {s}; [ {s} = active ]; exit $? ;;\n" for u, s in units.items()) + failed_lines = "".join(f"echo '{u} loaded failed failed Test'\n" for u in failed) + executable( + bindir / "systemctl", + f'echo "$*" >> {calls}\n' + 'case "$1 $2" in\n' + f' "is-active --quiet") exit {0 if complete else 3} ;;\n' + ' "list-units --state=failed")\n' + f"{failed_lines}" + " exit 0 ;;\n" + ' "--no-block reboot") exit 0 ;;\n' + "esac\n" + 'if [ "$1" = is-active ]; then\n' + ' case "$2" in\n' + f"{unit_cases}" + " esac\n" + " echo inactive; exit 3\n" + "fi\n" + "exit 99\n", + ) + bless = tmp_path / "bless" + executable(bless, f'[ "$1" = status ] && echo {status}\n') + boot = tmp_path / "boot" + (boot / "EFI" / "Linux").mkdir(parents=True) + for uki in ukis: + (boot / "EFI" / "Linux" / uki).touch() + os_release = tmp_path / "os-release" + os_release.write_text(f'IMAGE_ID=bluefin-server\nIMAGE_VERSION="{BOOTED}"\n', encoding="utf-8") + lockfile = tmp_path / "etc-reboot-lock" + if lock: + lockfile.touch() + sentinel = tmp_path / "run" / "reboot-required" + env = dict( + os.environ, + PATH=f"{bindir}:{os.environ['PATH']}", + BLUEFIN_BLESS_BOOT=str(bless), + BLUEFIN_BOOT_PATH=str(boot), + BLUEFIN_OS_RELEASE=str(os_release), + BLUEFIN_REBOOT_LOCKS=f"{tmp_path / 'run-reboot-lock'} {lockfile}", + BLUEFIN_REBOOT_SENTINEL=str(sentinel), + ) + result = subprocess.run([str(DEADLINE)], capture_output=True, text=True, env=env, timeout=30) + assert result.returncode == 0, result.stdout + result.stderr + rebooted = "--no-block reboot" in (calls.read_text(encoding="utf-8") if calls.exists() else "") + return rebooted, sentinel.exists(), result.stdout + result.stderr + + +@pytest.mark.parametrize("status", ["indeterminate", "bad"]) +def test_unblessed_counted_boot_reboots(tmp_path: Path, status: str) -> None: + rebooted, flagged, log = deadline(tmp_path, status) + assert rebooted and not flagged, log + assert f"<3>Bluefin Server {BOOTED} did not reach boot-complete.target before the boot deadline" in log + assert "failed: dogfood-broken.service" in log + + +def test_hung_boot_with_no_failed_unit_reboots(tmp_path: Path) -> None: + rebooted, _, log = deadline(tmp_path, "indeterminate", failed=()) + assert rebooted, log + assert "no unit failed, but the boot never got there" in log + + +@pytest.mark.parametrize("status", ["clean", "good", ""]) +def test_blessed_or_uncounted_boot_never_reboots(tmp_path: Path, status: str) -> None: + rebooted, flagged, log = deadline(tmp_path, status) + assert not rebooted and not flagged, log + assert "nothing to do" in log + + +def test_boot_that_reached_boot_complete_never_reboots(tmp_path: Path) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", complete=True, failed=()) + assert not rebooted and not flagged, log + + +def test_operator_lock_holds_the_reboot(tmp_path: Path) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", lock=True) + assert not rebooted and not flagged, log + assert "holds reboots, not rebooting" in log + + +@pytest.mark.parametrize( + "unit,state", + [("kubelet.service", "active"), ("k0scontroller.service", "active"), ("k0sworker.service", "activating")], +) +def test_kubernetes_node_is_left_to_kured(tmp_path: Path, unit: str, state: str) -> None: + rebooted, flagged, log = deadline(tmp_path, "indeterminate", units={unit: state}) + assert not rebooted and flagged, log + assert f"{unit} is {state}: flagged" in log + + +def test_node_whose_kubelet_failed_reboots_directly(tmp_path: Path) -> None: + # NotReady: nothing to drain, and kured's pod cannot run there. + rebooted, flagged, log = deadline( + tmp_path, "indeterminate", failed=("kubelet.service",), units={"kubelet.service": "failed"} + ) + assert rebooted and not flagged, log + + +def test_last_try_reboots_when_there_is_a_fallback(tmp_path: Path) -> None: + rebooted, _, log = deadline( + tmp_path, "dirty", ukis=("bluefin-server-26.09.4+0-3.efi", "bluefin-server-26.09.3.efi") + ) + assert rebooted, log + + +def test_last_try_without_a_fallback_does_not_loop(tmp_path: Path) -> None: + rebooted, _, log = deadline(tmp_path, "dirty", ukis=("bluefin-server-26.09.4+0-3.efi",)) + assert not rebooted, log + assert "no other UKI to fall back to" in log diff --git a/tests/unit/test_boot_origin.py b/tests/unit/test_boot_origin.py new file mode 100644 index 00000000..be803ffc --- /dev/null +++ b/tests/unit/test_boot_origin.py @@ -0,0 +1,82 @@ +"""Executed coverage for files/boot-origin/usr/libexec/bluefin-boot-origin. + +The helper prints the URL a network-booted node was booted from. Ignition in +the initrd looks for bluefin-node.ign next to it, and the diskless update check +fetches the signed SHA256SUMS from its directory. +""" + +from __future__ import annotations + +import os +import stat +import subprocess +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[2] +ORIGIN = ROOT / "files" / "boot-origin" / "usr" / "libexec" / "bluefin-boot-origin" +PULL = "raw,machine,verify=signature,blockdev:rootdisk:{url}" + + +def origin(tmp_path: Path, *, stub: str | None = None, cmdline: str = "", cred: str | None = None): + stub_var = tmp_path / "StubDeviceURL" + if stub is not None: + stub_var.write_bytes(b"\x06\x00\x00\x00" + stub.encode("utf-16-le") + b"\x00\x00") + (tmp_path / "cmdline").write_text(cmdline + "\n", encoding="utf-8") + env = dict(os.environ, BLUEFIN_STUB_URL_VAR=str(stub_var), BLUEFIN_CMDLINE=str(tmp_path / "cmdline")) + env.pop("CREDENTIALS_DIRECTORY", None) + if cred is not None: + creds = tmp_path / "creds" + creds.mkdir() + (creds / "import.pull").write_text(cred + "\n", encoding="utf-8") + env["CREDENTIALS_DIRECTORY"] = str(creds) + return subprocess.run([str(ORIGIN)], capture_output=True, text=True, env=env) + + +def test_origin_prefers_the_uefi_http_boot_url(tmp_path: Path) -> None: + result = origin( + tmp_path, + stub="http://booty/bluefin/52-54-00-aa-00-02/bluefin-server-netboot.efi", + cmdline="rd.systemd.pull=" + PULL.format(url="http://other/x.raw"), + ) + assert result.returncode == 0 + assert result.stdout == "http://booty/bluefin/52-54-00-aa-00-02/bluefin-server-netboot.efi\n" + + +def test_origin_uses_the_explicit_pull_url_on_the_cmdline(tmp_path: Path) -> None: + result = origin(tmp_path, cmdline="root=tmpfs rd.systemd.pull=" + PULL.format(url="https://srv/r/bluefin-server_1.raw") + " quiet") + assert (result.returncode, result.stdout) == (0, "https://srv/r/bluefin-server_1.raw\n") + + +def test_origin_falls_back_to_the_import_pull_credential(tmp_path: Path) -> None: + result = origin( + tmp_path, + cmdline="rd.systemd.pull=raw,machine,verify=signature,blockdev,bootorigin:rootdisk:bluefin-server_1.raw", + cred=PULL.format(url="http://10.0.2.2:8765/bluefin-server_1.raw"), + ) + assert (result.returncode, result.stdout) == (0, "http://10.0.2.2:8765/bluefin-server_1.raw\n") + + +def test_origin_fails_quietly_when_not_network_booted(tmp_path: Path) -> None: + result = origin(tmp_path, cmdline="rd.systemd.pull=raw,machine,blockdev,bootorigin:rootdisk:x.raw") + assert (result.returncode, result.stdout) == (1, "") + + +def test_ignition_uses_the_shared_origin_helper() -> None: + helper = (ROOT / "files" / "initrd-ignition" / "usr" / "libexec" / "bluefin-ignition-credentials").read_text(encoding="utf-8") + assert 'BOOT_ORIGIN="${BLUEFIN_BOOT_ORIGIN:-/usr/libexec/bluefin-boot-origin}"' in helper + assert "efivars" not in helper and "/proc/cmdline" not in helper, "one copy of the origin logic" + + +def test_helper_ships_in_the_initrd_and_is_executable_bash() -> None: + elements = ROOT / "elements" / "bluefin-server" + doc = yaml.safe_load((elements / "boot-origin.bst").read_text(encoding="utf-8")) + assert doc["kind"] == "import" + assert doc["sources"] == [{"kind": "local", "path": "files/boot-origin"}] + assert "target" not in (doc.get("config") or {}), "the tree mirrors /usr" + ign = yaml.safe_load((elements / "initrd" / "initrd-ignition-stack.bst").read_text(encoding="utf-8")) + assert "bluefin-server/boot-origin.bst" in ign["depends"] + assert ORIGIN.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert ORIGIN.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(ORIGIN)], check=True) diff --git a/tests/unit/test_diskless_update_check.py b/tests/unit/test_diskless_update_check.py new file mode 100644 index 00000000..7cd264b8 --- /dev/null +++ b/tests/unit/test_diskless_update_check.py @@ -0,0 +1,266 @@ +"""Executed coverage for the diskless update signal. + +bluefin-boot-origin finds the URL a node booted from; bluefin-diskless-update-check +fetches the signed SHA256SUMS from that directory and flags /run/reboot-required +when it offers a newer release that the next boot would pull, i.e. unless the +origin names a versioned file (a pinned node only logs it). Both run here against a local http.server and a +throwaway GnuPG key; no files/boot-keys are needed. +""" + +from __future__ import annotations + +import configparser +import functools +import os +import shutil +import stat +import subprocess +import threading +from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +import pytest +import yaml + +ROOT = Path(__file__).resolve().parents[2] +ORIGIN = ROOT / "files" / "boot-origin" / "usr" / "libexec" / "bluefin-boot-origin" +CHECK = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" / "bluefin-diskless-update-check" +RUNNING = "26.09.2" +PULL = "raw,machine,verify=signature,blockdev:rootdisk:{url}" + +UNITS = ROOT / "files" / "os" / "systemd" / "system" +ELEMENTS = ROOT / "elements" / "bluefin-server" +needs_tools = pytest.mark.skipif( + not all(shutil.which(t) for t in ("curl", "gpg", "gpgv", "systemd-analyze")), + reason="needs curl, gpg, gpgv and systemd-analyze", +) + + +def ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False, interpolation=None) + parser.optionxform = str + parser.read_string(path.read_text(encoding="utf-8")) + return parser + + +def test_check_runs_only_on_diskless_boots() -> None: + service = ini(UNITS / "bluefin-diskless-update-check.service") + timer = ini(UNITS / "bluefin-diskless-update-check.timer") + for unit in (service, timer): + assert unit["Unit"]["ConditionPathExists"] == "/run/machines/rootdisk.raw" + assert service["Service"]["Type"] == "oneshot" + assert service["Service"]["ImportCredential"] == "import.pull" + assert service["Service"]["ExecStart"] == "/usr/libexec/bluefin-diskless-update-check" + assert service["Unit"]["After"] == "network-online.target" + assert timer["Install"]["WantedBy"] == "timers.target" + + +def test_check_ships_in_the_os_and_is_executable_bash() -> None: + stack = yaml.safe_load((ELEMENTS / "os-stack.bst").read_text(encoding="utf-8"))["depends"] + assert "bluefin-server/os-update-check.bst" in stack + assert "bluefin-server/boot-origin.bst" in stack + doc = yaml.safe_load((ELEMENTS / "os-update-check.bst").read_text(encoding="utf-8")) + assert doc["kind"] == "import" + assert doc["sources"] == [{"kind": "local", "path": "files/os/update-check"}] + assert "target" not in (doc.get("config") or {}), "the tree mirrors /usr" + assert CHECK.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert CHECK.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(CHECK)], check=True) + + +@pytest.fixture(scope="module") +def keys(tmp_path_factory: pytest.TempPathFactory) -> dict[str, Path]: + base = tmp_path_factory.mktemp("gpg") + out = {} + for name in ("release", "stranger"): + home = base / name + home.mkdir(mode=0o700) + env = dict(os.environ, GNUPGHOME=str(home)) + subprocess.run( + ["gpg", "--batch", "--quiet", "--passphrase", "", "--quick-gen-key", f"{name} <{name}@example.invalid>", "ed25519", "sign", "never"], + check=True, env=env, capture_output=True, + ) + ring = base / f"{name}.pgp" + subprocess.run(["gpg", "--batch", "--export", "--output", str(ring)], check=True, env=env) + out[name] = home + out[f"{name}-ring"] = ring + return out + + +class QuietHandler(SimpleHTTPRequestHandler): + def log_message(self, *args) -> None: + pass + + +@pytest.fixture() +def server(tmp_path: Path): + root = tmp_path / "srv" + (root / "rel").mkdir(parents=True) + handler = functools.partial(QuietHandler, directory=str(root)) + httpd = ThreadingHTTPServer(("127.0.0.1", 0), handler) + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + yield root / "rel", f"http://127.0.0.1:{httpd.server_address[1]}/rel" + httpd.shutdown() + httpd.server_close() + + +def publish( + directory: Path, version: str, gnupghome: Path, *, tamper: bool = False, sign: bool = True, also: tuple[str, ...] = () +) -> None: + names = [ + name + for v in [*also, version] + for name in ( + f"bluefin-server_{v}.raw", + f"bluefin-server_{v}_d3107d37-a9da-32cf-3c19-49fa3b0cb1df.usr.raw", + f"bluefin-server-{v}.efi", + f"bluefin-server-netboot_{v}.efi", + f"bluefin-server-installer_{v}.raw", + f"zfs_{v}.raw.zst", + ) + ] + sums = "".join(f"{'%064x' % i} {name}\n" for i, name in enumerate(names)) + (directory / "SHA256SUMS").write_text(sums, encoding="utf-8") + if sign: + subprocess.run( + ["gpg", "--batch", "--yes", "--detach-sign", "--output", str(directory / "SHA256SUMS.gpg"), str(directory / "SHA256SUMS")], + check=True, env=dict(os.environ, GNUPGHOME=str(gnupghome)), capture_output=True, + ) + if tamper: + (directory / "SHA256SUMS").write_text(sums.replace(version, "99.99.99"), encoding="utf-8") + + +PINNED_RAW = f"bluefin-server_{RUNNING}.raw" +NETBOOT_UKI = "bluefin-server-netboot.efi" + + +def check(tmp_path: Path, url: str | None, keyring: Path, *, via: str = "stub", file: str = NETBOOT_UKI): + """Run the check with the boot origin / handed over : + "cmdline" (rd.systemd.pull=), "cred" (the import.pull credential) or + "stub" (the StubDeviceURL EFI variable of a UEFI HTTP boot).""" + os_release = tmp_path / "os-release" + os_release.write_text(f'ID=bluefin-server\nIMAGE_ID=bluefin-server\nIMAGE_VERSION="{RUNNING}"\n', encoding="utf-8") + sentinel = tmp_path / "run" / "reboot-required" + pull = PULL.format(url=f"{url}/{file}") if url else "" + env = dict( + os.environ, + BLUEFIN_BOOT_ORIGIN=str(ORIGIN), + BLUEFIN_IMPORT_KEYRING=str(keyring), + BLUEFIN_REBOOT_SENTINEL=str(sentinel), + BLUEFIN_OS_RELEASE=str(os_release), + BLUEFIN_STUB_URL_VAR=str(tmp_path / "no-efivar"), + BLUEFIN_CMDLINE=str(tmp_path / "cmdline"), + ) + env.pop("CREDENTIALS_DIRECTORY", None) + cmdline = "root=tmpfs" + if url and via == "cred": + (tmp_path / "creds").mkdir() + (tmp_path / "creds" / "import.pull").write_text(pull + "\n", encoding="utf-8") + env["CREDENTIALS_DIRECTORY"] = str(tmp_path / "creds") + elif url and via == "stub": + # efivarfs: 4 attribute bytes, then a NUL-terminated UTF-16LE string. + stub = tmp_path / "StubDeviceURL" + stub.write_bytes(b"\x06\x00\x00\x00" + f"{url}/{file}\0".encode("utf-16-le")) + env["BLUEFIN_STUB_URL_VAR"] = str(stub) + elif url: + cmdline = "rd.systemd.pull=" + pull + (tmp_path / "cmdline").write_text(cmdline + "\n", encoding="utf-8") + result = subprocess.run([str(CHECK)], capture_output=True, text=True, env=env, timeout=120) + assert result.returncode == 0, result.stdout + result.stderr + return sentinel.exists(), result.stdout + result.stderr + + +@needs_tools +@pytest.mark.parametrize( + "via,file", + [ + # Booty's HTTP boot URL: the served UKI changes, and the next boot + # pulls the image next to it. + ("stub", NETBOOT_UKI), + # An unversioned name the boot server repoints at each release. + ("cmdline", "bluefin-server.raw"), + ], +) +def test_newer_release_sets_the_flag(tmp_path: Path, server, keys, via: str, file: str) -> None: + directory, url = server + # 26.09.10 sorts after 26.09.2 as a version, not as a string. + publish(directory, "26.09.10", keys["release"]) + flagged, log = check(tmp_path, url, keys["release-ring"], via=via, file=file) + assert flagged, log + assert f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING}); flagged" in log + + +@needs_tools +@pytest.mark.parametrize( + "via,file", + [ + ("cmdline", PINNED_RAW), + ("cred", PINNED_RAW), + ("stub", f"bluefin-server-netboot_{RUNNING}.efi"), + ], +) +def test_newer_release_on_a_pinned_origin_sets_no_flag(tmp_path: Path, server, keys, via: str, file: str) -> None: + # The directory holds both releases; the origin names the running one, so + # a reboot would pull it again and kured would drain the node every check. + directory, url = server + publish(directory, "26.09.10", keys["release"], also=(RUNNING,)) + flagged, log = check(tmp_path, url, keys["release-ring"], via=via, file=file) + assert not flagged, log + assert ( + f"<5>Bluefin Server 26.09.10 is available from {url}/ (running {RUNNING}), " + f"but this node is pinned to {file}; not flagging" + ) in log + + +@needs_tools +@pytest.mark.parametrize("version", [RUNNING, "26.09.1", "26.08.30"]) +def test_same_or_older_release_sets_no_flag(tmp_path: Path, server, keys, version: str) -> None: + directory, url = server + publish(directory, version, keys["release"]) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert f"up to date: {url}/ offers {version}, running {RUNNING}" in log + + +@needs_tools +def test_signature_from_another_key_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["stranger"]) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "does not verify" in log + + +@needs_tools +def test_manifest_changed_after_signing_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["release"], tamper=True) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "does not verify" in log + + +@needs_tools +def test_unsigned_manifest_sets_no_flag(tmp_path: Path, server, keys) -> None: + directory, url = server + publish(directory, "26.09.10", keys["release"], sign=False) + flagged, log = check(tmp_path, url, keys["release-ring"]) + assert not flagged, log + assert "cannot fetch" in log + + +@needs_tools +def test_unreachable_boot_server_sets_no_flag(tmp_path: Path, keys) -> None: + with ThreadingHTTPServer(("127.0.0.1", 0), SimpleHTTPRequestHandler) as probe: + port = probe.server_address[1] + flagged, log = check(tmp_path, f"http://127.0.0.1:{port}/rel", keys["release-ring"]) + assert not flagged, log + assert "cannot fetch" in log + + +@needs_tools +def test_not_network_booted_is_a_no_op(tmp_path: Path, keys) -> None: + flagged, log = check(tmp_path, None, keys["release-ring"]) + assert not flagged, log + assert "not network booted" in log diff --git a/tests/unit/test_signed_lockstep.py b/tests/unit/test_signed_lockstep.py index 30eb515f..fa80fae4 100644 --- a/tests/unit/test_signed_lockstep.py +++ b/tests/unit/test_signed_lockstep.py @@ -63,8 +63,10 @@ def test_sysext_units_are_started_after_a_boot_time_merge() -> None: def test_http_booted_nodes_look_for_ignition_next_to_the_uki() -> None: helper = (IGN / "libexec" / "bluefin-ignition-credentials").read_text(encoding="utf-8") - assert "StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" in helper - assert "rd.systemd.pull=*:http://*" in helper, "iPXE-chainloaded nodes use the explicit pull URL" + origin = (ROOT / "files" / "boot-origin" / "usr" / "libexec" / "bluefin-boot-origin").read_text(encoding="utf-8") + assert "/usr/libexec/bluefin-boot-origin" in helper + assert "StubDeviceURL-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f" in origin + assert "rd.systemd.pull=*)" in origin and "*:http://*|*:https://*)" in origin, "iPXE-chainloaded nodes use the explicit pull URL" assert "NODE_CONFIG=bluefin-node.ign" in helper assert "curl -sfI" in helper for stage in ("disks", "fetch", "fetch-offline", "files", "mount"): diff --git a/tests/unit/test_update_pending.py b/tests/unit/test_update_pending.py new file mode 100644 index 00000000..060e2ccb --- /dev/null +++ b/tests/unit/test_update_pending.py @@ -0,0 +1,93 @@ +"""Executed coverage for bluefin-update-pending. + +After a rollback systemd-sysupdate still reports the failed version as +pending; bluefin-update-pending says no while that version's UKI has no tries +left, so the kured flag and the nightly reboot do not bring the node back to +it. It runs here against a fake systemd-sysupdate. +""" + +from __future__ import annotations + +import os +import shutil +import stat +import subprocess +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "os" / "systemd" / "system" +PENDING = ROOT / "files" / "os" / "update-check" / "usr" / "libexec" / "bluefin-update-pending" +BOOTED = "26.09.3" + + +def executable(path: Path, body: str) -> None: + path.write_text("#!/bin/sh\n" + body, encoding="utf-8") + path.chmod(path.stat().st_mode | stat.S_IXUSR) + + +def test_pending_is_executable_bash() -> None: + assert PENDING.read_text(encoding="utf-8").startswith("#!/usr/bin/bash\n") + assert PENDING.stat().st_mode & stat.S_IXUSR + subprocess.run(["bash", "-n", str(PENDING)], check=True) + + +def test_kured_flag_and_nightly_reboot_both_use_it() -> None: + kured = (UNITS / "systemd-sysupdate.service.d" / "20-kured.conf").read_text(encoding="utf-8") + interlock = (UNITS / "systemd-sysupdate-reboot.service.d" / "20-interlock.conf").read_text(encoding="utf-8") + assert "if /usr/libexec/bluefin-update-pending; then touch /run/reboot-required; fi" in kured + assert "systemd-sysupdate pending" not in kured + assert "ExecCondition=/usr/libexec/bluefin-update-pending\n" in interlock + + +needs_analyze = pytest.mark.skipif(not shutil.which("systemd-analyze"), reason="needs systemd-analyze") + + +def pending(tmp_path: Path, *, sysupdate_pending: bool, ukis: tuple[str, ...]) -> subprocess.CompletedProcess[str]: + sysupdate = tmp_path / "systemd-sysupdate" + executable(sysupdate, f'[ "$1" = pending ] && exit {0 if sysupdate_pending else 1}\nexit 99\n') + boot = tmp_path / "boot" + (boot / "EFI" / "Linux").mkdir(parents=True) + for uki in ukis: + (boot / "EFI" / "Linux" / uki).touch() + os_release = tmp_path / "os-release" + os_release.write_text(f'IMAGE_VERSION="{BOOTED}"\n', encoding="utf-8") + env = dict( + os.environ, + BLUEFIN_SYSUPDATE=str(sysupdate), + BLUEFIN_BOOT_PATH=str(boot), + BLUEFIN_OS_RELEASE=str(os_release), + ) + return subprocess.run([str(PENDING)], capture_output=True, text=True, env=env, timeout=30) + + +@needs_analyze +def test_pending_update_with_tries_left(tmp_path: Path) -> None: + result = pending(tmp_path, sysupdate_pending=True, ukis=("bluefin-server-26.09.4+2-1.efi", "bluefin-server-26.09.3.efi")) + assert result.returncode == 0, result.stdout + result.stderr + + +@needs_analyze +def test_nothing_pending(tmp_path: Path) -> None: + result = pending(tmp_path, sysupdate_pending=False, ukis=("bluefin-server-26.09.3.efi",)) + assert result.returncode == 1, result.stdout + result.stderr + + +@needs_analyze +@pytest.mark.parametrize("uki", ["bluefin-server-26.09.4+0-3.efi", "bluefin-server-26.09.10+0.efi"]) +def test_update_that_failed_its_tries_is_not_pending(tmp_path: Path, uki: str) -> None: + # After the rollback systemd-sysupdate still reports it as pending. + result = pending(tmp_path, sysupdate_pending=True, ukis=(uki, "bluefin-server-26.09.3.efi")) + assert result.returncode == 1, result.stdout + result.stderr + assert f"is installed but failed its boot tries ({uki}); staying on {BOOTED}" in result.stdout + + +@needs_analyze +def test_older_exhausted_uki_does_not_block_a_newer_update(tmp_path: Path) -> None: + result = pending( + tmp_path, + sysupdate_pending=True, + ukis=("bluefin-server-26.09.2+0-3.efi", "bluefin-server-26.09.3.efi", "bluefin-server-26.09.5+3-0.efi"), + ) + assert result.returncode == 0, result.stdout + result.stderr