Skip to content

feat(update): automatic updates with a Kubernetes interlock, a boot health gate and a diskless update signal - #271

Merged
jeefy merged 19 commits into
projectbluefin:mainfrom
jeefy:feat/automatic-updates
Sep 29, 2026
Merged

jeefy merged 19 commits into
projectbluefin:mainfrom
jeefy:feat/automatic-updates

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Closes #139. Supersedes #182 (credit: @mrbobbytables for the Kubernetes interlock idea).

Three gaps in the update story:

  1. Installed nodes never updated on their own. FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and systemd-sysupdate-reboot.timer, so a node only moved to a new slot when someone ran systemd-sysupdate and rebooted. The one reboot signal, the kured hook, was also broken: systemd-sysupdate.service is Type=simple, so its ExecStartPost=touch /run/reboot-required ran as soon as an update started, even when there was nothing to update. With the timer enabled, kured would have rebooted every Kubernetes node every couple of hours.
  2. Boot counting had no health gate. systemd-bless-boot.service requires boot-complete.target, which on its own only needs sysinit.target, so a new UKI (TriesLeft=3) was marked good long before the system was up.
  3. Diskless nodes had no signal that their boot server was serving a newer release.

Change

Installed nodes (files/os/systemd/system-preset/80-bluefin-updates.preset, which sorts after Ignition's 20- and before FSDK's 90- presets):

  • enable systemd-sysupdate.timer and systemd-sysupdate-reboot.timer. The timer runs the stock systemd-sysupdate update, so enabled zfs / kubestellar / kubeadm features keep updating in lock-step. systemd-sysupdate reboot reboots only when a newer version is installed.
  • Kubernetes interlock (systemd-sysupdate-reboot.service.d/20-kubernetes.conf): an ExecCondition= skips the local reboot while kubelet.service, k0scontroller.service or k0sworker.service is active, activating, deactivating, reloading or refreshing. Those nodes stay with kured. This is the feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts #182 idea, ported to the FSDK tree.
  • kured flag fixed (systemd-sysupdate.service.d/20-kured.conf): ExecStopPost= touches /run/reboot-required only when systemd-sysupdate pending reports an installed update. The drop-in moves next to 10-diskless.conf, and os-kured-hook.bst goes away.
  • The reboot service and both timers get the same 10-diskless.conf conditions as systemd-sysupdate.service, so they stay inert on diskless and USB installer boots.

Boot health gate: enable systemd-boot-check-no-failures.service (FSDK ships it; RequiredBy=boot-complete.target, after multi-user.target). A boot-counted UKI is now blessed only when the boot reaches multi-user.target with no failed unit. Diskless boots are not boot-counted, so they never pull in boot-complete.target.

Diskless update signal (bluefin-diskless-update-check.{service,timer}, conditioned on /run/machines/rootdisk.raw, the same test 10-diskless.conf uses):

  • /usr/libexec/bluefin-boot-origin is the old origin_url() from bluefin-ignition-credentials, factored into one helper that the initrd's Ignition and the booted OS share. It reads the StubDeviceURL EFI variable, then the explicit http(s) rd.systemd.pull=, then the import.pull credential (only when the calling unit imports it, so Ignition's behaviour is unchanged).
  • /usr/libexec/bluefin-diskless-update-check fetches SHA256SUMS and SHA256SUMS.gpg from the boot directory and verifies them with gpgv against /etc/systemd/import-pubring.pgp (else the vendor keyring). It compares the bluefin-server_<ver>.raw entry with IMAGE_VERSION using systemd-analyze compare-versions. When the release is newer it touches /run/reboot-required and logs Bluefin Server <ver> is available from <url> (running <ver>). Network and signature errors are logged and never set the flag or fail the unit, and the check never reboots. It adds no new runtime dependencies (bash, coreutils, curl, gpgv, systemd-analyze). kured acts on the flag on diskless Kubernetes nodes; elsewhere an operator or fleet tooling reboots when convenient, and Booty decides what the next boot pulls. See docs/skills/ddi-installer.md → Updates.

Upgrade note: presets apply only on first boot, so nodes installed before this change need one systemctl preset systemd-sysupdate.timer systemd-sysupdate-reboot.timer systemd-boot-check-no-failures.service. This is documented.

Verification

Local gates on every one of the 8 commits, each in a throwaway detached worktree with no files/boot-keys (mirrors CI): python3 -m pytest tests/unit -q passed (405 at HEAD) and python3 .github/scripts/docs-checks.py passed. Also bats tests/unit (74 ok) and flock bst.lock just validate. CI unit passed with 405 tests and none skipped, so the gpg/http.server behavioural tests ran on the Ubuntu runner.

New tests:

  • tests/unit/test_automatic_updates.py: preset contents and ordering; the diskless conditions on the reboot service and both timers match 10-diskless.conf; the kured flag is ExecStopPost + pending, and no drop-in overrides ExecStart. The Kubernetes interlock is executed against a fake systemctl: it proceeds with no Kubernetes unit or with failed/inactive ones, and skips (exit 1) for active, activating and deactivating kubelet/k0s units.
  • tests/unit/test_boot_origin.py: StubDeviceURL beats the cmdline, then the explicit rd.systemd.pull= URL, then the import.pull credential; a node that was not network booted gets exit 1. Ignition keeps no copy of the logic.
  • tests/unit/test_diskless_update_check.py: the helper runs against a local http.server with throwaway GNUPGHOME keys. Newer (26.09.10 vs 26.09.2, which tests version ordering rather than string ordering) sets the flag and logs the version, via both the cmdline and the credential. Same, older, a foreign-key signature, a manifest changed after signing, a missing .gpg, an unreachable server, and a node that was not network booted all leave the flag unset and exit 0.

QEMU with Secure Boot and dev keys, using image set A 26.09.2 and B 26.09.3 (a throwaway just set-version; the version change was restored and is not committed):

(a) Diskless, same release served (dogfood-diskless.sh dist/diskless --check):

PROBE os=bluefin-server 26.09.2
PROBE update-timers=inactive inactive active        # sysupdate + reboot timers inert, check timer active
PROBE update-check=success flag=none
PROBE-LOG up to date: http://10.0.2.2:8765/ offers 26.09.2, running 26.09.2
PROBE diskless-bless=clean boot-complete=inactive   # no boot counting on diskless
PROBE preset=enabled enabled enabled enabled
PROBE failed=0

(b) Diskless, boot directory offers a newer release. The served SHA256SUMS is A's plus B's entries, re-signed with the dev release key (a mirror directory holding both releases); the initrd pull of A still verifies against it:

PROBE os=bluefin-server 26.09.2
PROBE update-check=success flag=set
gpgv: Good signature from "Bluefin Server dev image signing (...)"
Bluefin Server 26.09.3 is available from http://10.0.2.2:8765/ (running 26.09.2); flagged /run/reboot-required, reboot to update
PROBE failed=0

(c) just dogfood-install dist/diskless-next, which now asserts every line below:

==> 2/4 boot the installed disk
PROBE update-timers=active active inactive
PROBE timers-enabled=enabled enabled
PROBE health=active active bless=clean              # first disk boot: sysinstall entry, not counted
==> 3/4 systemd-sysupdate to diskless-next          # via `systemctl start --wait systemd-sysupdate.service`
Successfully installed '.../bluefin-server-26.09.3.efi' as '/boot/EFI/Linux/bluefin-server-26.09.3+3-0.efi'
Successfully installed '.../zfs_26.09.3.raw.zst' as '/var/lib/extensions/zfs_26.09.3.raw'   # feature in lock-step
Newest installed version '26.09.3' is newer than booted version '26.09.2'. Reboot recommended.   # ExecStopPost `pending`
PROBE update=0
PROBE kured-flag=set
PROBE interlock=exec-condition                       # reboot unit skipped while a stand-in kubelet.service ran
==> 4/4 boot the updated disk
PROBE os=bluefin-server 26.09.3
PROBE zfs=active zfs_26.09.3.raw
PROBE ukis=bluefin-server-26.09.3.efi               # +3-0 counter removed: blessed
PROBE health=active active bless=good               # check-no-failures + boot-complete.target, then blessed
PROBE failed=0
PASS: installed, updated A->B and booted 26.09.3

Follow-ups

  • A FleetLock-style cluster-wide reboot lock for non-Kubernetes fleets (roadmap item 5, gap analysis). Single hosts reboot in the 04:10 window; several non-Kubernetes hosts serving together could reboot in the same window.
  • A QEMU negative proof of the health gate (a boot with a failed unit stays indeterminate and falls back after three tries) is not automated here; the rollback path itself is covered by dogfood-install.sh <dir> <next> <broken>.
  • docs/skills/ddi-installer.md is now 319 lines, over the docs-check soft warning at 300 (the hard limit is 500).

jeefy and others added 8 commits September 28, 2026 22:01
…t-origin

Move origin_url() out of bluefin-ignition-credentials into
/usr/libexec/bluefin-boot-origin, so the booted OS can find its boot
server with the same logic instead of a copy. Order: the StubDeviceURL EFI
variable (UEFI HTTP boot), the explicit http(s) rd.systemd.pull= source,
then the import.pull credential when the calling unit imports it.

Ignition does not import import.pull, so its behaviour is unchanged. Tests
drive the helper against fake efivar, cmdline and credential files.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reboot interlock

FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and
systemd-sysupdate-reboot.timer, so installed nodes never updated on their
own and non-Kubernetes hosts never moved to the new slot.
80-bluefin-updates.preset enables both. The timer runs the stock
`systemd-sysupdate update`, so enabled features (zfs, kubestellar,
kubeadm) keep following the OS in lock-step.

Kubernetes nodes keep rebooting through kured. A drop-in adds an
ExecCondition= to systemd-sysupdate-reboot.service that skips the local
reboot while kubelet, k0scontroller or k0sworker is active, activating,
deactivating, reloading or refreshing. This is the interlock from projectbluefin#182 by
@mrbobbytables, ported to the FSDK tree.

The kured hook also fired on every run: systemd-sysupdate.service is
Type=simple, so its ExecStartPost touched /run/reboot-required as soon as
an update *started*, even with nothing to download. It is now an
ExecStopPost that flags only when `systemd-sysupdate pending` reports an
installed update. It moves next to 10-diskless.conf under
files/os/systemd/system, and os-kured-hook.bst goes away.

The reboot service and both timers get the same 10-diskless.conf
conditions as systemd-sysupdate.service, so diskless and USB installer
boots stay inert.

Refs projectbluefin#139, supersedes projectbluefin#182.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-bless-boot.service requires boot-complete.target, which by
itself only needs sysinit.target, so a new UKI was marked good long
before the system was up. Enable FSDK's
systemd-boot-check-no-failures.service (RequiredBy=boot-complete.target,
after multi-user.target), which FSDK's presets disable. A good boot now
reaches multi-user.target with no failed unit; any other boot uses up one
of the TriesLeft=3 and systemd-boot falls back after the third.

Only sysupdate-installed UKIs are counted, so diskless boots never pull in
boot-complete.target.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…release

Diskless nodes update by rebooting, but had no way to know a newer
release was being served. bluefin-diskless-update-check.timer (5 min
after boot, then hourly) runs only when /run/machines/rootdisk.raw exists,
the same test 10-diskless.conf uses. It finds the boot directory with
bluefin-boot-origin, fetches SHA256SUMS and SHA256SUMS.gpg, verifies them
with gpgv against the image keyring, and compares the
bluefin-server_<ver>.raw entry with IMAGE_VERSION using
`systemd-analyze compare-versions`. When the release is newer it touches
/run/reboot-required (kured's sentinel) and logs the new version.

Network and signature errors are logged and never set the flag or fail
the unit, and the check never reboots. It needs no new runtime
dependencies (bash, coreutils, curl, gpgv, systemd-analyze).

Behavioural tests run the helper against a local http.server with a
throwaway GnuPG key: newer (26.09.10 vs 26.09.2) sets the flag, over the
cmdline and the import.pull credential alike. Same or older, a foreign
signature, a manifest changed after signing, a missing .gpg, an
unreachable server, and a node that was not network booted all leave it
unset.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…gate in QEMU

dogfood-diskless.sh: every boot reports the update timers, and a diskless
boot runs bluefin-diskless-update-check once and reports its result and
the flag (the CI boot-test job covers the same-version path on every PR).

dogfood-install.sh: the installed disk must have both timers enabled and
active. The A->B update now runs through systemd-sysupdate.service, the
unit the timer starts, and must set /run/reboot-required. With a
stand-in kubelet.service running, systemd-sysupdate-reboot.service must
be skipped (Result=exec-condition) instead of rebooting. The updated
boot must pass systemd-boot-check-no-failures, reach boot-complete.target
and be blessed (`systemd-bless-boot status` = good).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… diskless update signal

ddi-installer.md "Updates" is the canonical description: timers and
preset, the boot health gate, the kured flag and Kubernetes interlock,
opting out, the one-time `systemctl preset` for nodes installed before the
preset existed, and how kured, operators and Booty act on the diskless
flag. The verification skill records that the diskless check trusts the
same keyring, and ddi-installer-build.md lists the new dogfood assertions.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
While k0s or kubelet runs, the base image's reboot timer stands down.
Say so where operators of those sysexts look, linking to ddi-installer.md.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Single hosts now reboot into updates in a nightly window and Kubernetes
nodes go through kured, so issue projectbluefin#139's gap is closed. What remains is a
FleetLock-style lock for non-Kubernetes fleets. Update the readiness
audit, the roadmap, the distro gap analysis and the README.

Closes projectbluefin#139

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 930ab2e. The update plumbing looks right to me. Two behaviour gaps, and one overlap with #182 that needs sorting out.

1. The health gate stops the bless but never reboots, so "rolls back an update that does not boot cleanly" doesn't hold on its own

80-bluefin-updates.preset:7-9 enables systemd-boot-check-no-failures.service. If a unit has failed, boot-complete.target isn't reached and systemd-bless-boot leaves the entry at +N-M. Nothing then reboots the machine: systemd-boot only decrements and falls back on a later boot, and systemd-sysupdate-reboot.service only reboots when a newer version than the booted one is installed, which is no longer the case once you are running it. A node that reaches multi-user with one failed unit therefore stays on the unblessed image until something else reboots it. README.md:29 ("UKI boot counting rolls back an update that does not boot cleanly (any failed unit counts)"), ddi-installer.md and gap-analysis-distros.md all describe automatic rollback. Either add the reboot, e.g. OnFailure= on systemd-boot-check-no-failures.service pointing at a unit with FailureAction=reboot (or boot-complete.target failure handling), or reword the docs to say the retry happens on the next reboot. dogfood-install.sh only checks the healthy path (PROBE health=active active bless=good), so it doesn't cover this.

2. The diskless update signal can loop kured on a version-pinned source

bluefin-diskless-update-check:64 checks the directory of whatever URL bluefin-boot-origin returns. It flags /run/reboot-required when that directory's signed SHA256SUMS lists a bluefin-server_<ver>.raw newer than IMAGE_VERSION (lines 85-103). When the origin comes from an explicit rd.systemd.pull= or the import.pull credential, the URL names a specific versioned file. If the server directory holds both A and B, the node pulls A again after the reboot, the check flags B again, and kured drains and reboots the node every cycle. Please flag only when the next boot would actually pick the newer file (the StubDeviceURL/netboot-UKI path), or skip the flag when the origin names a versioned image.

3. Overlap with #182

systemd-sysupdate-reboot.service.d/20-kubernetes.conf ports #182's Kubernetes interlock and handles activating/deactivating/reloading states that #182's systemctl is-active --quiet form misses. It does not carry #182's ConditionPathExists=!/run/reboot-lock / !/etc/reboot-lock operator inhibition. Both PRs also rewrite the same gap-analysis-distros.md "Reboot coordination" rows. If both land as they are, the reboot service gets two ExecCondition= interlocks from different directories, and the docs conflict. Please either fold the lock-file conditions in here so #182 can close, or say explicitly that they are out of scope.

What I verified

  • pytest tests/unit: 405 passed. bats tests/unit: pass. shellcheck -S warning on bluefin-boot-origin, bluefin-ignition-credentials and bluefin-diskless-update-check: clean.
  • 20-kured.conf uses ExecStopPost= with systemd-sysupdate pending. That is right for the upstream Type=simple unit, and it doesn't flag a no-op update run.
  • The 10-diskless.conf drop-ins condition both timers and the reboot service off on /run/machines/rootdisk.raw and root=tmpfs. The update check is their inverse.
  • bluefin-diskless-update-check verifies SHA256SUMS.gpg with gpgv against the same keyring the initrd pull uses, caps both fetches at 1 MiB, and exits 0 on every network or signature error.
  • CI at this head: unit (run 36517083650) and docs (36517083675) pass. build (36517083653) was still running when I checked.

jeefy and others added 10 commits September 29, 2026 02:04
Fold projectbluefin#182's operator inhibition into this PR's reboot drop-in so the
reboot service keeps one interlock instead of two from different
directories: systemd-sysupdate-reboot.service is skipped while
/run/reboot-lock (until the next boot) or /etc/reboot-lock (until
removed) exists. The drop-in is renamed 20-interlock.conf because it
now carries more than the Kubernetes check.

Lock files from projectbluefin#182 by Bob Killen.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The health gate stopped the bless but nothing rebooted: systemd-boot only
moves on at the next boot, and systemd-sysupdate-reboot does nothing once
the newest version is the booted one. A node that came up with a failed
unit stayed on the unblessed image.

bluefin-boot-deadline.timer (preset-enabled) runs only on boot-counted
boots: LoaderBootCountPath exists, not diskless, not root=tmpfs. 15
minutes after boot (OnBootSec=, a drop-in changes it), if
systemd-bless-boot still says indeterminate or bad and
boot-complete.target is not active, bluefin-boot-deadline logs the failed
units and reboots, so systemd-boot uses up the next try and falls back
after the last. The last try boots "dirty" and reboots only when another
UKI can still be booted, so there is no loop without a fallback. The
service has DefaultDependencies=no and no ordering on multi-user or
boot-complete, so a hung boot is caught too.

Kubernetes: while kubelet or k0s is starting, running or stopping, the
node may carry workloads, so it touches /run/reboot-required and kured
drains and reboots it in its lock, as for an update. A stopped or failed
Kubernetes unit leaves the node NotReady with nothing to drain and no
kured, so it reboots directly. /run/reboot-lock and /etc/reboot-lock hold
it like the nightly reboot.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ries

After a rollback the failed version is still installed, and
`systemd-sysupdate pending` still reports it: it does not look at the
UKI's tries counter. The kured flag would be set after every update run
and the nightly reboot would fire every night, each time landing on the
previous version again.

bluefin-update-pending wraps `systemd-sysupdate pending` and says no
while a newer installed version's UKI has no tries left
(bluefin-server-<ver>+0-<done>.efi). It gates the kured flag
(20-kured.conf) and the nightly reboot (an ExecCondition in
20-interlock.conf). The next newer release takes the failed version's
slot and UKI (InstancesMax=2) and clears the block.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
When the boot origin names a versioned file, bluefin-server_<ver>.raw from
an explicit rd.systemd.pull= URL or the import.pull credential (or a
versioned bluefin-server-netboot_<ver>.efi boot URL), the node pulls the
same version again after a reboot. Flagging /run/reboot-required there made
kured drain and reboot the node on every check while the server directory
held a newer release.

Flag only when the next boot resolves the version itself: a netboot UKI
served under a fixed name (Booty's bluefin-server-netboot.efi via
StubDeviceURL) or another unversioned name. A pinned node logs
"Bluefin Server <ver> is available from <dir>/ (running <ver>), but this
node is pinned to <file>; not flagging". The behavioural tests cover both
flagging origins and the three pinned ones, and the signature and
network-error cases now run against a flaggable origin so they still prove
something.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
os-update-check.bst now installs bluefin-boot-deadline and
bluefin-update-pending next to the diskless update check, and
os-k0s-first-boot.bst the deadline units and the reboot interlock.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
DOGFOOD_BROKEN=unit breaks <broken-dir> with a unit that fails on that
version only (ConditionOSRelease=IMAGE_VERSION=) instead of corrupting
its slot, and shortens the boot deadline (DOGFOOD_DEADLINE, default 2min).
Each counted boot reaches multi-user.target, misses
boot-complete.target and waits for bluefin-boot-deadline to reboot it.
The run asserts three such boots, the fallback to <next-dir> with the
deadline timer inactive and the boot clean, and then no kured flag and a
skipped systemd-sysupdate-reboot.service for the failed version. Every
disk boot also reports the deadline timer's state. The default slot mode
is unchanged.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nned diskless origins

ddi-installer.md "Updates": boot counting alone only rolls back on the
next boot, so describe bluefin-boot-deadline (when it runs, why a
Kubernetes node is flagged for kured instead of rebooted, the dirty last
try, the drop-in to change the deadline), bluefin-update-pending after a
rollback, the lock files in 20-interlock.conf (credited to projectbluefin#182), and
which diskless origins are pinned. ddi-installer-build.md covers
DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…es through kured

While kubelet or k0s runs, bluefin-boot-deadline flags
/run/reboot-required instead of rebooting, so kured also rolls back an
unhealthy update.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README, readiness audit, roadmap and distro gap analysis said boot
counting rolls back an update that does not boot cleanly; that now holds
because the boot deadline supplies the reboot. Say so, and how long it
takes. Reconcile the "Reboot coordination" rows with projectbluefin#182: the lock files
and the Kubernetes interlock both live in 20-interlock.conf, credited to
projectbluefin#182. The diskless signal is described as skipping pinned nodes.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Brings in the locked-root provisioning (projectbluefin#275) and binary patch tracking.
The only conflicts were the last_updated dates of ddi-installer-build.md
and kubeadm-sysext.md; both sides' content is kept.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @castrojo, all three points are addressed at f666799. Each one below maps to the commit that fixes it and the evidence.

1. The health gate now reboots: 901e6b3, 276daeb, harness in d78b176

901e6b3 fix(update): reboot a boot-counted boot that misses boot-complete.target

  • bluefin-boot-deadline.timer is preset-enabled and runs only on boot-counted boots:
    • ConditionPathExists= on the LoaderBootCountPath EFI variable
    • !/run/machines/rootdisk.raw and !root=tmpfs, so never on diskless or installer boots, and never on blessed or uncounted boots
  • It fires at OnBootSec=15min; a drop-in changes that (OnBootSec= then OnBootSec=30min).
  • When it fires, /usr/libexec/bluefin-boot-deadline checks two things: systemd-bless-boot status is still indeterminate or bad, and boot-complete.target is not active. If both hold, it logs the failed units and runs systemctl reboot. systemd-boot then uses up the next try, and after the third boots the previous UKI.
  • The last try boots as dirty. It reboots only if another UKI is still bootable, so a node with nothing to fall back to does not loop.
  • The service has DefaultDependencies=no and no ordering on multi-user or boot-complete. A boot that hangs therefore hits the deadline too.
  • On the OnFailure=/FailureAction= alternative: systemd-boot-check-no-failures runs only once multi-user.target is reached. That would miss hangs and anything that fails later, so I went with a deadline.

Kubernetes/kured choice (documented in ddi-installer.md "Updates" → "Rollback"):

  • kubelet or k0s starting, running or stopping: the node does not reboot itself. After 15 minutes up it may have been uncordoned and carry workloads, so it touches /run/reboot-required. kured then cordons, drains and reboots it within its cluster lock, the same path as an update. Each try costs a drain. Without kured the node stays unblessed until someone reboots it, the same contract as a staged update.
  • Kubernetes unit stopped or failed: the node is NotReady, has nothing to drain and can't run the kured pod, so it reboots directly.
  • Operator hold: /run/reboot-lock or /etc/reboot-lock holds both the reboot and the flag.

276daeb fix(update): do not reboot back into an update that failed its boot tries covers a follow-on problem I hit while proving this.

  • After a rollback, systemd-sysupdate pending still reports the failed version: sysupdate ignores the UKI tries counter. The kured flag would be set every 2 h and the nightly reboot would fire every night, each time landing back on the old version.
  • bluefin-update-pending wraps pending and says no while a newer installed version's UKI is +0-*. It gates 20-kured.conf and the nightly reboot.
  • The next newer release takes over the failed version's slot and UKI, which clears the block.

QEMU proof (flock qemu.lock, merged head f666799, three fresh builds 26.09.2 → .3 → .4), run as DOGFOOD_BROKEN=unit scripts/dogfood-install.sh dist/diskless dist/diskless-next dist/diskless-broken. The broken set's next boot has a unit that fails on 26.09.4 only (ConditionOSRelease=IMAGE_VERSION=26.09.4), and the deadline is shortened to 2 min:

==> 2/4 boot the installed disk          PROBE health=active active bless=clean   PROBE deadline=inactive no
==> 4/4 boot the updated disk            PROBE health=active active bless=good    PROBE deadline=active yes
==> 6/6 boot: 26.09.4 must fail its tries and fall back to 26.09.3
PROBE counted-boot=26.09.4 bless=indeterminate failed=dogfood-broken.service systemd-boot-check-no-failures.service deadline=active
PROBE counted-boot=26.09.4 bless=indeterminate failed=dogfood-broken.service systemd-boot-check-no-failures.service deadline=active
PROBE counted-boot=26.09.4 bless=dirty failed=dogfood-broken.service systemd-boot-check-no-failures.service deadline=active
PROBE os=bluefin-server 26.09.3
PROBE ukis=bluefin-server-26.09.3.efi bluefin-server-26.09.4+0-3.efi
PROBE health=active active bless=clean
PROBE deadline=inactive no
PROBE-LOG Bluefin Server 26.09.4 did not reach boot-complete.target before the boot deadline (failed: dogfood-broken.service systemd-boot-check-no-failures.service); its boot entry is indeterminate; rebooting so systemd-boot falls back after the last try   (x2)
PROBE-LOG ... its boot entry is dirty; rebooting so systemd-boot falls back after the last try
PROBE-LOG Bluefin Server 26.09.4 is installed but failed its boot tries (bluefin-server-26.09.4+0-3.efi); staying on 26.09.3 until a newer release replaces it
PROBE after-rollback kured-flag=none
PROBE after-rollback reboot-unit=exec-condition
PROBE failed=0
PASS: installed, updated A->B, and rolled back from a broken (unit) 26.09.4 to 26.09.3

The default DOGFOOD_BROKEN=slot path (corrupt the slot) is unchanged; I did not re-run it here. README, ddi-installer.md, gap-analysis-distros.md, the MVP audit and the roadmap now describe what actually happens (3927747, 2e50d00).

2. Pinned diskless origins no longer flag kured: ccfd0d5

  • bluefin-diskless-update-check flags only when the next boot resolves the version itself: a netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi, via StubDeviceURL) or another unversioned name.
  • An origin whose file is versioned is treated as pinned:
    • bluefin-server_<ver>.raw, from an explicit rd.systemd.pull= or the import.pull credential
    • a versioned bluefin-server-netboot_<ver>.efi boot URL
  • A pinned node logs Bluefin Server X is available from <dir>/ (running Y), but this node is pinned to <file>; not flagging and leaves the flag alone.
  • Tests:
    • The flag is set for two origins: the fixed-name UKI via StubDeviceURL, and an unversioned rd.systemd.pull=.
    • The flag stays clear for three pinned origins: rd.systemd.pull=, import.pull and a versioned StubDeviceURL. In each case the directory holds both the running and the newer release.
    • The signature, tamper and network tests now use a flaggable origin, so they still prove the check.
  • QEMU: one served directory with 26.09.2 plus a signed SHA256SUMS that also lists 26.09.4.
# import.pull names bluefin-server_26.09.2.raw
PROBE update-check=success flag=none
PROBE-LOG Bluefin Server 26.09.4 is available from http://10.0.2.2:8765/ (running 26.09.2), but this node is pinned to bluefin-server_26.09.2.raw; not flagging /run/reboot-required, ...
# UEFI HTTP boot of .../bluefin-server-netboot.efi
PROBE update-check=success flag=set
PROBE-LOG Bluefin Server 26.09.4 is available from http://10.0.2.2:8765/ (running 26.09.2); flagged /run/reboot-required, reboot to update

3. #182 folded in: 6d2d3a6

4. Upstream

f666799 merges current main (#275 and binary tracking). The only conflicts were the two skills' last_updated dates.

Gates at f666799

  • pytest tests/unit: 468 passed. The same count passes in a throwaway detached worktree without files/boot-keys.
  • bats tests/unit: 74 ok.
  • python3 .github/scripts/docs-checks.py: passed.
  • flock bst.lock just validate: exit 0.
  • shellcheck -S warning on bluefin-boot-deadline, bluefin-update-pending, bluefin-diskless-update-check, bluefin-boot-origin and dogfood-install.sh: clean.

Resolves the ddi-installer.md Common Rationalizations table against projectbluefin#273:
keeps projectbluefin#273's ignored-karg note on the Ignition row and this branch's
update/rollback rows.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy merged commit a904748 into projectbluefin:main Sep 29, 2026
5 checks passed
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
Combines projectbluefin#271's rollback scenarios (DOGFOOD_BROKEN, boot deadline) with
this branch's DOGFOOD_PORT handling, the 1.<ver> version-order note and
the tamper/RAM harness docs.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts

2 participants