Repository navigation
feat(update): automatic updates with a Kubernetes interlock, a boot health gate and a diskless update signal - #271
Conversation
…t-origin Move origin_url() out of bluefin-ignition-credentials into /usr/libexec/bluefin-boot-origin, so the booted OS can find its boot server with the same logic instead of a copy. Order: the StubDeviceURL EFI variable (UEFI HTTP boot), the explicit http(s) rd.systemd.pull= source, then the import.pull credential when the calling unit imports it. Ignition does not import import.pull, so its behaviour is unchanged. Tests drive the helper against fake efivar, cmdline and credential files. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reboot interlock FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and systemd-sysupdate-reboot.timer, so installed nodes never updated on their own and non-Kubernetes hosts never moved to the new slot. 80-bluefin-updates.preset enables both. The timer runs the stock `systemd-sysupdate update`, so enabled features (zfs, kubestellar, kubeadm) keep following the OS in lock-step. Kubernetes nodes keep rebooting through kured. A drop-in adds an ExecCondition= to systemd-sysupdate-reboot.service that skips the local reboot while kubelet, k0scontroller or k0sworker is active, activating, deactivating, reloading or refreshing. This is the interlock from projectbluefin#182 by @mrbobbytables, ported to the FSDK tree. The kured hook also fired on every run: systemd-sysupdate.service is Type=simple, so its ExecStartPost touched /run/reboot-required as soon as an update *started*, even with nothing to download. It is now an ExecStopPost that flags only when `systemd-sysupdate pending` reports an installed update. It moves next to 10-diskless.conf under files/os/systemd/system, and os-kured-hook.bst goes away. The reboot service and both timers get the same 10-diskless.conf conditions as systemd-sysupdate.service, so diskless and USB installer boots stay inert. Refs projectbluefin#139, supersedes projectbluefin#182. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-bless-boot.service requires boot-complete.target, which by itself only needs sysinit.target, so a new UKI was marked good long before the system was up. Enable FSDK's systemd-boot-check-no-failures.service (RequiredBy=boot-complete.target, after multi-user.target), which FSDK's presets disable. A good boot now reaches multi-user.target with no failed unit; any other boot uses up one of the TriesLeft=3 and systemd-boot falls back after the third. Only sysupdate-installed UKIs are counted, so diskless boots never pull in boot-complete.target. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…release Diskless nodes update by rebooting, but had no way to know a newer release was being served. bluefin-diskless-update-check.timer (5 min after boot, then hourly) runs only when /run/machines/rootdisk.raw exists, the same test 10-diskless.conf uses. It finds the boot directory with bluefin-boot-origin, fetches SHA256SUMS and SHA256SUMS.gpg, verifies them with gpgv against the image keyring, and compares the bluefin-server_<ver>.raw entry with IMAGE_VERSION using `systemd-analyze compare-versions`. When the release is newer it touches /run/reboot-required (kured's sentinel) and logs the new version. Network and signature errors are logged and never set the flag or fail the unit, and the check never reboots. It needs no new runtime dependencies (bash, coreutils, curl, gpgv, systemd-analyze). Behavioural tests run the helper against a local http.server with a throwaway GnuPG key: newer (26.09.10 vs 26.09.2) sets the flag, over the cmdline and the import.pull credential alike. Same or older, a foreign signature, a manifest changed after signing, a missing .gpg, an unreachable server, and a node that was not network booted all leave it unset. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gate in QEMU dogfood-diskless.sh: every boot reports the update timers, and a diskless boot runs bluefin-diskless-update-check once and reports its result and the flag (the CI boot-test job covers the same-version path on every PR). dogfood-install.sh: the installed disk must have both timers enabled and active. The A->B update now runs through systemd-sysupdate.service, the unit the timer starts, and must set /run/reboot-required. With a stand-in kubelet.service running, systemd-sysupdate-reboot.service must be skipped (Result=exec-condition) instead of rebooting. The updated boot must pass systemd-boot-check-no-failures, reach boot-complete.target and be blessed (`systemd-bless-boot status` = good). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… diskless update signal ddi-installer.md "Updates" is the canonical description: timers and preset, the boot health gate, the kured flag and Kubernetes interlock, opting out, the one-time `systemctl preset` for nodes installed before the preset existed, and how kured, operators and Booty act on the diskless flag. The verification skill records that the diskless check trusts the same keyring, and ddi-installer-build.md lists the new dogfood assertions. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
While k0s or kubelet runs, the base image's reboot timer stands down. Say so where operators of those sysexts look, linking to ddi-installer.md. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Single hosts now reboot into updates in a nightly window and Kubernetes nodes go through kured, so issue projectbluefin#139's gap is closed. What remains is a FleetLock-style lock for non-Kubernetes fleets. Update the readiness audit, the roadmap, the distro gap analysis and the README. Closes projectbluefin#139 Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed at 930ab2e. The update plumbing looks right to me. Two behaviour gaps, and one overlap with #182 that needs sorting out.
1. The health gate stops the bless but never reboots, so "rolls back an update that does not boot cleanly" doesn't hold on its own
80-bluefin-updates.preset:7-9 enables systemd-boot-check-no-failures.service. If a unit has failed, boot-complete.target isn't reached and systemd-bless-boot leaves the entry at +N-M. Nothing then reboots the machine: systemd-boot only decrements and falls back on a later boot, and systemd-sysupdate-reboot.service only reboots when a newer version than the booted one is installed, which is no longer the case once you are running it. A node that reaches multi-user with one failed unit therefore stays on the unblessed image until something else reboots it. README.md:29 ("UKI boot counting rolls back an update that does not boot cleanly (any failed unit counts)"), ddi-installer.md and gap-analysis-distros.md all describe automatic rollback. Either add the reboot, e.g. OnFailure= on systemd-boot-check-no-failures.service pointing at a unit with FailureAction=reboot (or boot-complete.target failure handling), or reword the docs to say the retry happens on the next reboot. dogfood-install.sh only checks the healthy path (PROBE health=active active bless=good), so it doesn't cover this.
2. The diskless update signal can loop kured on a version-pinned source
bluefin-diskless-update-check:64 checks the directory of whatever URL bluefin-boot-origin returns. It flags /run/reboot-required when that directory's signed SHA256SUMS lists a bluefin-server_<ver>.raw newer than IMAGE_VERSION (lines 85-103). When the origin comes from an explicit rd.systemd.pull= or the import.pull credential, the URL names a specific versioned file. If the server directory holds both A and B, the node pulls A again after the reboot, the check flags B again, and kured drains and reboots the node every cycle. Please flag only when the next boot would actually pick the newer file (the StubDeviceURL/netboot-UKI path), or skip the flag when the origin names a versioned image.
3. Overlap with #182
systemd-sysupdate-reboot.service.d/20-kubernetes.conf ports #182's Kubernetes interlock and handles activating/deactivating/reloading states that #182's systemctl is-active --quiet form misses. It does not carry #182's ConditionPathExists=!/run/reboot-lock / !/etc/reboot-lock operator inhibition. Both PRs also rewrite the same gap-analysis-distros.md "Reboot coordination" rows. If both land as they are, the reboot service gets two ExecCondition= interlocks from different directories, and the docs conflict. Please either fold the lock-file conditions in here so #182 can close, or say explicitly that they are out of scope.
What I verified
pytest tests/unit: 405 passed.bats tests/unit: pass.shellcheck -S warningonbluefin-boot-origin,bluefin-ignition-credentialsandbluefin-diskless-update-check: clean.20-kured.confusesExecStopPost=withsystemd-sysupdate pending. That is right for the upstream Type=simple unit, and it doesn't flag a no-opupdaterun.- The
10-diskless.confdrop-ins condition both timers and the reboot service off on/run/machines/rootdisk.rawandroot=tmpfs. The update check is their inverse. bluefin-diskless-update-checkverifiesSHA256SUMS.gpgwithgpgvagainst the same keyring the initrd pull uses, caps both fetches at 1 MiB, and exits 0 on every network or signature error.- CI at this head: unit (run 36517083650) and docs (36517083675) pass. build (36517083653) was still running when I checked.
Fold projectbluefin#182's operator inhibition into this PR's reboot drop-in so the reboot service keeps one interlock instead of two from different directories: systemd-sysupdate-reboot.service is skipped while /run/reboot-lock (until the next boot) or /etc/reboot-lock (until removed) exists. The drop-in is renamed 20-interlock.conf because it now carries more than the Kubernetes check. Lock files from projectbluefin#182 by Bob Killen. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The health gate stopped the bless but nothing rebooted: systemd-boot only moves on at the next boot, and systemd-sysupdate-reboot does nothing once the newest version is the booted one. A node that came up with a failed unit stayed on the unblessed image. bluefin-boot-deadline.timer (preset-enabled) runs only on boot-counted boots: LoaderBootCountPath exists, not diskless, not root=tmpfs. 15 minutes after boot (OnBootSec=, a drop-in changes it), if systemd-bless-boot still says indeterminate or bad and boot-complete.target is not active, bluefin-boot-deadline logs the failed units and reboots, so systemd-boot uses up the next try and falls back after the last. The last try boots "dirty" and reboots only when another UKI can still be booted, so there is no loop without a fallback. The service has DefaultDependencies=no and no ordering on multi-user or boot-complete, so a hung boot is caught too. Kubernetes: while kubelet or k0s is starting, running or stopping, the node may carry workloads, so it touches /run/reboot-required and kured drains and reboots it in its lock, as for an update. A stopped or failed Kubernetes unit leaves the node NotReady with nothing to drain and no kured, so it reboots directly. /run/reboot-lock and /etc/reboot-lock hold it like the nightly reboot. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ries After a rollback the failed version is still installed, and `systemd-sysupdate pending` still reports it: it does not look at the UKI's tries counter. The kured flag would be set after every update run and the nightly reboot would fire every night, each time landing on the previous version again. bluefin-update-pending wraps `systemd-sysupdate pending` and says no while a newer installed version's UKI has no tries left (bluefin-server-<ver>+0-<done>.efi). It gates the kured flag (20-kured.conf) and the nightly reboot (an ExecCondition in 20-interlock.conf). The next newer release takes the failed version's slot and UKI (InstancesMax=2) and clears the block. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the boot origin names a versioned file, bluefin-server_<ver>.raw from an explicit rd.systemd.pull= URL or the import.pull credential (or a versioned bluefin-server-netboot_<ver>.efi boot URL), the node pulls the same version again after a reboot. Flagging /run/reboot-required there made kured drain and reboot the node on every check while the server directory held a newer release. Flag only when the next boot resolves the version itself: a netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi via StubDeviceURL) or another unversioned name. A pinned node logs "Bluefin Server <ver> is available from <dir>/ (running <ver>), but this node is pinned to <file>; not flagging". The behavioural tests cover both flagging origins and the three pinned ones, and the signature and network-error cases now run against a flaggable origin so they still prove something. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
os-update-check.bst now installs bluefin-boot-deadline and bluefin-update-pending next to the diskless update check, and os-k0s-first-boot.bst the deadline units and the reboot interlock. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DOGFOOD_BROKEN=unit breaks <broken-dir> with a unit that fails on that version only (ConditionOSRelease=IMAGE_VERSION=) instead of corrupting its slot, and shortens the boot deadline (DOGFOOD_DEADLINE, default 2min). Each counted boot reaches multi-user.target, misses boot-complete.target and waits for bluefin-boot-deadline to reboot it. The run asserts three such boots, the fallback to <next-dir> with the deadline timer inactive and the boot clean, and then no kured flag and a skipped systemd-sysupdate-reboot.service for the failed version. Every disk boot also reports the deadline timer's state. The default slot mode is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nned diskless origins ddi-installer.md "Updates": boot counting alone only rolls back on the next boot, so describe bluefin-boot-deadline (when it runs, why a Kubernetes node is flagged for kured instead of rebooted, the dirty last try, the drop-in to change the deadline), bluefin-update-pending after a rollback, the lock files in 20-interlock.conf (credited to projectbluefin#182), and which diskless origins are pinned. ddi-installer-build.md covers DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es through kured While kubelet or k0s runs, bluefin-boot-deadline flags /run/reboot-required instead of rebooting, so kured also rolls back an unhealthy update. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README, readiness audit, roadmap and distro gap analysis said boot counting rolls back an update that does not boot cleanly; that now holds because the boot deadline supplies the reboot. Say so, and how long it takes. Reconcile the "Reboot coordination" rows with projectbluefin#182: the lock files and the Kubernetes interlock both live in 20-interlock.conf, credited to projectbluefin#182. The diskless signal is described as skipping pinned nodes. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings in the locked-root provisioning (projectbluefin#275) and binary patch tracking. The only conflicts were the last_updated dates of ddi-installer-build.md and kubeadm-sysext.md; both sides' content is kept. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks @castrojo, all three points are addressed at f666799. Each one below maps to the commit that fixes it and the evidence. 1. The health gate now reboots:
|
Resolves the ddi-installer.md Common Rationalizations table against projectbluefin#273: keeps projectbluefin#273's ignored-karg note on the Ignition row and this branch's update/rollback rows. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Combines projectbluefin#271's rollback scenarios (DOGFOOD_BROKEN, boot deadline) with this branch's DOGFOOD_PORT handling, the 1.<ver> version-order note and the tamper/RAM harness docs. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Motivation
Closes #139. Supersedes #182 (credit: @mrbobbytables for the Kubernetes interlock idea).
Three gaps in the update story:
90-sysupdate.presetdisablessystemd-sysupdate.timerandsystemd-sysupdate-reboot.timer, so a node only moved to a new slot when someone ransystemd-sysupdateand rebooted. The one reboot signal, the kured hook, was also broken:systemd-sysupdate.serviceisType=simple, so itsExecStartPost=touch /run/reboot-requiredran as soon as an update started, even when there was nothing to update. With the timer enabled, kured would have rebooted every Kubernetes node every couple of hours.systemd-bless-boot.servicerequiresboot-complete.target, which on its own only needssysinit.target, so a new UKI (TriesLeft=3) was marked good long before the system was up.Change
Installed nodes (
files/os/systemd/system-preset/80-bluefin-updates.preset, which sorts after Ignition's20-and before FSDK's90-presets):enable systemd-sysupdate.timerandsystemd-sysupdate-reboot.timer. The timer runs the stocksystemd-sysupdate update, so enabledzfs/kubestellar/kubeadmfeatures keep updating in lock-step.systemd-sysupdate rebootreboots only when a newer version is installed.systemd-sysupdate-reboot.service.d/20-kubernetes.conf): anExecCondition=skips the local reboot whilekubelet.service,k0scontroller.serviceork0sworker.serviceis active, activating, deactivating, reloading or refreshing. Those nodes stay with kured. This is the feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts #182 idea, ported to the FSDK tree.systemd-sysupdate.service.d/20-kured.conf):ExecStopPost=touches/run/reboot-requiredonly whensystemd-sysupdate pendingreports an installed update. The drop-in moves next to10-diskless.conf, andos-kured-hook.bstgoes away.10-diskless.confconditions assystemd-sysupdate.service, so they stay inert on diskless and USB installer boots.Boot health gate:
enable systemd-boot-check-no-failures.service(FSDK ships it;RequiredBy=boot-complete.target, aftermulti-user.target). A boot-counted UKI is now blessed only when the boot reachesmulti-user.targetwith no failed unit. Diskless boots are not boot-counted, so they never pull inboot-complete.target.Diskless update signal (
bluefin-diskless-update-check.{service,timer}, conditioned on/run/machines/rootdisk.raw, the same test10-diskless.confuses):/usr/libexec/bluefin-boot-originis the oldorigin_url()frombluefin-ignition-credentials, factored into one helper that the initrd's Ignition and the booted OS share. It reads theStubDeviceURLEFI variable, then the explicit http(s)rd.systemd.pull=, then theimport.pullcredential (only when the calling unit imports it, so Ignition's behaviour is unchanged)./usr/libexec/bluefin-diskless-update-checkfetchesSHA256SUMSandSHA256SUMS.gpgfrom the boot directory and verifies them withgpgvagainst/etc/systemd/import-pubring.pgp(else the vendor keyring). It compares thebluefin-server_<ver>.rawentry withIMAGE_VERSIONusingsystemd-analyze compare-versions. When the release is newer it touches/run/reboot-requiredand logsBluefin Server <ver> is available from <url> (running <ver>). Network and signature errors are logged and never set the flag or fail the unit, and the check never reboots. It adds no new runtime dependencies (bash, coreutils, curl, gpgv, systemd-analyze). kured acts on the flag on diskless Kubernetes nodes; elsewhere an operator or fleet tooling reboots when convenient, and Booty decides what the next boot pulls. Seedocs/skills/ddi-installer.md→ Updates.Upgrade note: presets apply only on first boot, so nodes installed before this change need one
systemctl preset systemd-sysupdate.timer systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service. This is documented.Verification
Local gates on every one of the 8 commits, each in a throwaway detached worktree with no
files/boot-keys(mirrors CI):python3 -m pytest tests/unit -qpassed (405 at HEAD) andpython3 .github/scripts/docs-checks.pypassed. Alsobats tests/unit(74 ok) andflock bst.lock just validate. CIunitpassed with 405 tests and none skipped, so the gpg/http.server behavioural tests ran on the Ubuntu runner.New tests:
tests/unit/test_automatic_updates.py: preset contents and ordering; the diskless conditions on the reboot service and both timers match10-diskless.conf; the kured flag isExecStopPost+pending, and no drop-in overridesExecStart. The Kubernetes interlock is executed against a fakesystemctl: it proceeds with no Kubernetes unit or with failed/inactive ones, and skips (exit 1) foractive,activatinganddeactivatingkubelet/k0s units.tests/unit/test_boot_origin.py:StubDeviceURLbeats the cmdline, then the explicitrd.systemd.pull=URL, then theimport.pullcredential; a node that was not network booted gets exit 1. Ignition keeps no copy of the logic.tests/unit/test_diskless_update_check.py: the helper runs against a localhttp.serverwith throwawayGNUPGHOMEkeys. Newer (26.09.10vs26.09.2, which tests version ordering rather than string ordering) sets the flag and logs the version, via both the cmdline and the credential. Same, older, a foreign-key signature, a manifest changed after signing, a missing.gpg, an unreachable server, and a node that was not network booted all leave the flag unset and exit 0.QEMU with Secure Boot and dev keys, using image set A
26.09.2and B26.09.3(a throwawayjust set-version; the version change was restored and is not committed):(a) Diskless, same release served (
dogfood-diskless.sh dist/diskless --check):(b) Diskless, boot directory offers a newer release. The served
SHA256SUMSis A's plus B's entries, re-signed with the dev release key (a mirror directory holding both releases); the initrd pull of A still verifies against it:(c)
just dogfood-install dist/diskless-next, which now asserts every line below:Follow-ups
indeterminateand falls back after three tries) is not automated here; the rollback path itself is covered bydogfood-install.sh <dir> <next> <broken>.docs/skills/ddi-installer.mdis now 319 lines, over the docs-check soft warning at 300 (the hard limit is 500).