Repository navigation
feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts - #182
mrbobbytables wants to merge 5 commits into
Conversation
castrojo
left a comment
There was a problem hiding this comment.
Approving. This resolves both blockers I raised when closing #145, and it does so the right way rather than the expedient way.
Blocker 1 — the imaginary sentinel — is properly fixed. #145 gated on ConditionPathExists=!/run/kured-active, a file nothing in this repo or Kured ever creates, so on a Kubernetes node the condition always passed and the timer would reboot out from under Kured with no cordon and no drain. This replaces it with a real liveness check:
ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service'
I verified the semantics empirically rather than trusting my reading of them, because this is the whole safety property and I initially thought it was wrong. systemctl is-active with multiple units exits 0 if any listed unit is active (confirmed against a real active/inactive pair: is-active <active> <inactive> -> 0, is-active <inactive> -> 3). So:
- Kubernetes running (any of the three active) ->
is-active0 ->!-> 1 ->ExecConditionnon-zero -> unit is skipped cleanly, Kured retains control. - No Kubernetes ->
is-active3 ->!-> 0 ->ExecConditionpasses -> local reboot proceeds.
That is exactly the intended behaviour, and using ExecCondition rather than ExecStartPre matters: a non-zero ExecCondition in the 1-254 range skips the unit without marking it failed, so a k0s node does not accumulate failed units every night.
Checking all three of kubelet.service, k0scontroller.service and k0sworker.service is the right net — k0s embeds its own kubelet, so a controller node may have k0scontroller active without kubelet.service existing at all, and the any-active semantics cover controller, worker and generic-kubelet hosts alike.
Blocker 2 — shadowing upstream's units — is properly fixed. #145 shipped its own systemd-sysupdate-reboot.service and .timer into /usr/lib/systemd/system/, the same path components/systemd.bst installs to, silently overriding units systemd itself ships and dropping upstream's Requires=/After= ordering. This instead ships a drop-in: os-sysupdate-reboot.bst is kind: import targeting /usr/lib/systemd/system/systemd-sysupdate-reboot.service.d, with a preset to enable the timer. Upstream owns the unit; we own the configuration. That is the systemd-native shape and it satisfies one-canonical-source.
The docs are honest, which I appreciate. gap-analysis-distros.md does not simply delete the gap — it splits it into a Status line for what now works (single-node, maintenance window, lock-file inhibition) and a remaining Gap for what does not (a cluster-wide FleetLock/locksmithd HTTP lock client). The roadmap row 6 is rewritten to match rather than removed. And k0s-sysext-ops.md gains a Reboot Coordination section — that file is named in docs/skills/index.md as the runbook for exactly this topic, and its omission was one of my complaints about #145.
The lock-file inhibition is now real rather than advertised: ConditionPathExists=!/run/reboot-lock and !/etc/reboot-lock are conditions on a unit that exists, giving operators a runtime and a persistent escape hatch.
CI: docs and unit pass. build is still running at the time of this review; it should be green before merge, and I would not merge on my approval alone regardless. installer-test will remain red for the pre-existing /var reason (#173) that fails on main itself.
One thing for the next reviewer to sanity-check rather than assume, since I could not exercise it: the timer's 04:10 maintenance window with randomized delay is inherited from upstream's .timer plus the preset, so confirm the preset actually enables the timer in the built image — the drop-in only configures the service.
Approving. Needs one more independent approval under the two-reviewer policy.
Thank you for turning #145 around this quickly and for taking the harder path on both points instead of patching the sentinel.
There was a problem hiding this comment.
Reviewed (correctness) — no blockers; one edge case worth noting.
Verified the key mechanics at head e306974:
ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service'(files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf:10) is correct: per systemd'scheck_unit_generic(src/systemctl/systemctl-is-active.c), multi-unitis-activeexits 0 if at least one unit is active, so the negation skips reboot whenever any K8s service runs. The mock-systemctl test (tests/unit/test_reboot_coordination.py:104-160) matches this semantic; all 6 tests pass locally.- The new preset (
files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset:2) is shipped becauseelements/bluefin-server/os-sshd-preset.bst:7-10imports the entirefiles/os/systemd/system-presetdirectory — same path otherzz-*presets rely on. - FSDK 26.08 pin (
elements/freedesktop-sdk.bst) ships systemd ≥257, so upstreamsystemd-sysupdate-reboot.{service,timer}exist and the 04:10/30min claims indocs/skills/k0s-sysext-ops.md:53-55match upstreamunits/systemd-sysupdate-reboot.timer.
Low — single-node k0s hosts never auto-reboot without Kured. The ExecCondition gates on k0s service liveness, not on Kured being deployed (reboot-coordination.conf:10). A single-node host running k0s without Kured installed — squarely within this PR's title scope — will stage updates via sysupdate but never apply them: the local timer path is skipped and no cluster coordinator exists. docs/skills/k0s-sysext-ops.md:52-53 states Kured handles the reboot but nothing in the image ships or verifies Kured. Consider documenting that k0s hosts must deploy Kured (or remove the lock files and reboot manually), or the update loop silently stalls.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Addressed review feedback from @kubestellar-hive in commit Updated
|
|
Needs a rebase — one path conflicts with #210. This PR touches The design also gets simpler after #210: the reboot coordinator no longer has to special-case "k0s is the cluster". With upstream kubeadm/kubelet the node is a normal Kubernetes node, so Refs #139. |
|
Tracked as a single repository incident in #218. Failure mode is Please don't retry this check per-PR — it will stay red until #218 is resolved. — hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78 |
There was a problem hiding this comment.
correctness
- (low) The Kubernetes interlock is point-in-time:
ExecCondition=... '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service'(files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf:10) passes whenever no unit is inactivestate at the 04:10 timer fire.systemctl is-activereports non-zero foractivating/failed, so a Kubernetes node whose kubelet/k0s service is mid-restart or crashed at that moment reboots uncoordinated (no cordon/drain). Low likelihood given the 30-min randomized window, but worth a note or aRequisite/retry consideration if it matters for larger fleets.
No findings from: security, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Addressed review feedback from @kubestellar-hive in commit
Regarding @castrojo's rebase note on #210: acknowledged; keeping an eye on #210 and will rebase onto
|
There was a problem hiding this comment.
correctness
The new preset never takes effect — the reboot timer stays disabled, making the whole feature inert. freedesktop-sdk's systemd ships /usr/lib/systemd/system-preset/90-sysupdate.preset containing disable systemd-sysupdate-reboot.timer (installed by FSDK 26.08 components/_private/systemd-base.bst, line 130 of the pinned freedesktop-sdk-26.08.0 tag). Per systemd.preset(5), preset files are evaluated in lexicographic filename order and the earliest matching line wins — 90-sysupdate.preset sorts before zz-enable-sysupdate-reboot.preset (files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset:2), so on first-boot preset-all the timer remains disabled. The drop-in (files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) is installed but never triggered, and issue #139's acceptance criterion ("a single-node VM defers reboots according to the configured strategy") is not met. Fix: name the preset so it sorts before 90- (e.g. 80-enable-sysupdate-reboot.preset). Note the same first-match rule means tests/unit/test_reboot_coordination.py:114 (test_reboot_preset_enables_timer) passes while the shipped image behavior is wrong — a test asserting the preset filename sorts before 90-sysupdate.preset would catch this.
No findings from: security, intent-alignment, style, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Addressed in commit
— hive: backend=copilot model=gemini-3.8-flash🐝 Hive Agent: |
There was a problem hiding this comment.
intent-alignment
- The PR body says the preset is
files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset, but the diff shipsfiles/os/systemd/system-preset/80-enable-sysupdate-reboot.preset. Behavior is unaffected (the wholesystem-presetdir is imported byelements/bluefin-server/os-sshd-preset.bst:6-10), but the description should match the file actually added.
style
tests/unit/test_reboot_coordination.py:106-112justifies the80-prefix by asserting the name sorts before a freedesktop-sdk90-sysupdate.preset"which disables the timer" — I could not find such a preset (upstream systemd'spresets/90-systemd.presethas nosystemd-sysupdate-rebootline, and nothing matching turned up in FSDK). The assertion is trivially true and harmless, but the comment encodes an unverified claim, and the new80-name breaks the repo's existingzz-*preset convention (zz-enable-sshd.preset,zz-enable-networkd.preset, etc.) without a confirmed reason.
Verified while reviewing (head 96a8d42): upstream systemd v257+ ships systemd-sysupdate-reboot.service/.timer and systemd-sysupdate reboot only reboots when a newer version than the booted one is installed, so enabling the timer fleet-wide is safe; systemctl is-active exits 0 if at least one listed unit is active, matching the intended k8s gate in reboot-coordination.conf:13.
No findings from: correctness, security, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
Updated the PR description to accurately reflect
|
|
#220 merged to main at 14:06 UTC today with the fix for the installer-test boot deadlock (udev sysext ordering, depmod path, homed-firstboot mask). This branch predates it, so its |
96a8d42 to
24441c7
Compare
Danathar
left a comment
There was a problem hiding this comment.
Approving at 24441c7. This PR turns on upstream's systemd-sysupdate-reboot.timer for hosts that don't run Kubernetes, and adds a drop-in so the reboot service stands down when a lock file exists or a Kubernetes unit is active.
What I checked:
- I read the full diff against main (dc48bfb). The branch contains the boot-deadlock fixes (#220 and #227), so the installer check is now a real result.
- CI passed on 24441c7, and I read the runs. In run 35782269073,
buildandinstaller-test / Install and boot in QEMUpassed. Unit tests passed in run 35782267704. - I checked the preset order against the pinned FSDK (26.08.0, systemd v261.2). FSDK's
90-sysupdate.presetdisablessystemd-sysupdate-reboot.timer.80-enable-sysupdate-reboot.presetsorts first, and the first match wins, so the timer ends up enabled.os-sshd-preset.bstalready imports the wholefiles/os/systemd/system-presetdirectory, so the new file ships without a new element. systemctl is-active --quiet a b cexits 0 if any listed unit is active. So theExecConditionskips the reboot whenever kubelet or either k0s unit is running, andkubelet.servicestill covers the kubeadm path from #210.pytest tests/unit/test_reboot_coordination.py: 6 passed. When I dropped the!from theExecCondition, the interlock test failed.
One thing for a follow-up, not this PR: the same FSDK preset also disables systemd-sysupdate.timer, and nothing in this repo turns it back on. The upstream reboot service runs systemd-sysupdate reboot, which only reboots when a newer version is already installed. So on current images this timer reboots only after someone stages an update by hand. It does not complete an automatic update loop.
24441c7 to
b4c3320
Compare
|
Rebased onto |
|
I re-read this after the rebase to b4c3320. The patch itself is the same one I approved at 24441c7: What did change is the base under it. #132 merged in between, so the installed OS now gets systemd from Flatcar's The preset no longer does anything, as far as I can tell. I pulled the pinned Flatcar container (4593.2.5, sha256
Command: So The drop-in matters more now, not less. If I'm reading the listing right, main already enables both CI on this head. Locally at b4c3320: |
hanthor
left a comment
There was a problem hiding this comment.
The files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf drop-in is valuable, more so now that Flatcar 4593.2.5 enables systemd-sysupdate-reboot.timer via timers.target.wants with no Kubernetes gate. Happy to approve after a few updates:
- Please rebase;
buildfailed atflatcar-usr.bst, which was a main-branch failure at e4d9793 and is fixed now. 80-enable-sysupdate-reboot.presetlooks redundant now, and the comments intests/unit/test_reboot_coordination.pyreferencing FSDK's90-sysupdate.presetare out of date. Drop the preset or reword it.- Product gap: the default appliance runs k0s and no Kured ships in the manifests, so with this gate staged updates never reboot on a default install. Please document that (or ship Kured) and consider a max-deferral.
…de hosts Configure systemd-sysupdate-reboot coordination for non-Kubernetes and single-node hosts using upstream systemd units with a drop-in configuration and systemd preset. - Avoid colliding with upstream systemd units by configuring upstream's systemd-sysupdate-reboot.service via a drop-in under files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf instead of overriding the unit files. - Enable systemd-sysupdate-reboot.timer by default via files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset. - Support lock-based reboot inhibition via ConditionPathExists on /run/reboot-lock and /etc/reboot-lock. - Gate reboots via ExecCondition checking active Kubernetes services (k0scontroller.service, kubelet.service, k0sworker.service), ensuring Kubernetes clusters preserve Kured's cordon and drain workflow. - Package drop-in through elements/bluefin-server/os-sysupdate-reboot.bst and wire into os-stack.bst. - Add unit test coverage in tests/unit/test_reboot_coordination.py. - Update docs in architecture-roadmap.md, gap-analysis-distros.md, and k0s-sysext-ops.md. Closes projectbluefin#139 Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ured Clarify that k0s hosts must deploy Kured or an equivalent coordinator, and that uncoordinated local reboots remain inhibited on single-node k0s hosts when Kured is absent, per review feedback from @kubestellar-hive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ock inhibition Document in reboot-coordination.conf and k0s-sysext-ops.md that the ExecCondition evaluates point-in-time liveness at timer fire, and recommend using /run/reboot-lock during planned cluster maintenance or service restarts to deterministically inhibit reboots across transient unit state transitions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Rename zz-enable-sysupdate-reboot.preset to 80-enable-sysupdate-reboot.preset so that under systemd.preset lexicographic ordering it evaluates before freedesktop-sdk's 90-sysupdate.preset (which disables systemd-sysupdate-reboot.timer), ensuring the timer is enabled on first boot. Add unit test assertion for preset ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ault-install gap Flatcar's /usr (projectbluefin#132) already enables systemd-sysupdate-reboot.timer via timers.target.wants and ships no 90-sysupdate.preset, so the 80- preset no longer does anything. Remove it, its test, and the FSDK-era comments. Document that the default k0s appliance ships no Kured, so with the drop-in's Kubernetes gate a staged update waits for a manual reboot. Assisted-by: Claude via omp Signed-off-by: castrojo <castrojo@users.noreply.github.com>
b4c3320 to
fa57e7b
Compare
|
Addressed @hanthor's review (COMMENTED at b4c3320) in one push; head is now fa57e7b.
Local verification at fa57e7b: |
|
This has conflicts with main again, and I tried a rebase that isn't mechanical, so I aborted it. Since fa57e7b, main has taken #262 (feat!: FSDK-composed OS with a signed, diskless-first boot chain) and #264 (offline USB installer). That changes two things this PR depends on:
|
|
Thanks @mrbobbytables. The Kubernetes interlock from this PR (an #271 ports it to the current FSDK tree (this branch predates the FSDK migration and now conflicts). It also counts
It is proven in QEMU, including the interlock skipping the reboot ( |
castrojo
left a comment
There was a problem hiding this comment.
Checked this branch (fa57e7b) against current main (8782952) by merging main into it locally. Merging main in leaves the PR functionally inert, not just conflicted:
- The premise of fa57e7b no longer holds on main. fa57e7b dropped
80-enable-sysupdate-reboot.presetbecause "Flatcar's/usrenablessystemd-sysupdate-reboot.timerthrough its owntimers.target.wantssymlink". #262 replaced the Flatcar/usrwith an FSDK-composed one (junctionfreedesktop-sdk-26.08.0). In that FSDK,90-sysupdate.presetdisablessystemd-sysupdate-reboot.timer, which is what Danathar checked against 26.08.0 earlier in this thread. Rebased as it is, the timer stays disabled and the drop-in never runs. The preset has to come back, or the timer has to be enabled some other way. - Layout. Main now keeps unit drop-ins under
files/os/systemd/system/<unit>.d/(for examplefiles/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) and presets as80-bluefin-*.preset. This branch still addsfiles/os/systemd/systemd-sysupdate-reboot.service.d/behind its ownos-sysupdate-reboot.bst. - Conflicts. Merging main hits content conflicts in
docs/skills/architecture-roadmap.md,docs/skills/gap-analysis-distros.mdanddocs/skills/k0s-sysext-ops.md. Main rewrote the state-model, updates and reboot-coordination rows for the FSDK/diskless design, so this branch's text in those rows describes the Flatcar-era tree (ID=flatcar,rwroot,50-root.transfer). - Overlap with #271. #271 enables
systemd-sysupdate-reboot.timerthrough80-bluefin-updates.presetand addssystemd-sysupdate-reboot.service.d/20-kubernetes.confwith the same kubelet/k0scontroller/k0sworker interlock. Its version also treatsactivating/deactivating/reloadingas running, whichsystemctl is-active --quiethere doesn't. What #182 has that #271 doesn't is the/run/reboot-lockand/etc/reboot-lockinhibition. If #271 lands, those twoConditionPathExists=lines are all that is left for this PR to contribute.
I did not push anything. The rework above goes beyond a mechanical rebase.
Fold projectbluefin#182's operator inhibition into this PR's reboot drop-in so the reboot service keeps one interlock instead of two from different directories: systemd-sysupdate-reboot.service is skipped while /run/reboot-lock (until the next boot) or /etc/reboot-lock (until removed) exists. The drop-in is renamed 20-interlock.conf because it now carries more than the Kubernetes check. Lock files from projectbluefin#182 by Bob Killen. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nned diskless origins ddi-installer.md "Updates": boot counting alone only rolls back on the next boot, so describe bluefin-boot-deadline (when it runs, why a Kubernetes node is flagged for kured instead of rebooted, the dirty last try, the drop-in to change the deadline), bluefin-update-pending after a rollback, the lock files in 20-interlock.conf (credited to projectbluefin#182), and which diskless origins are pinned. ddi-installer-build.md covers DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The README, readiness audit, roadmap and distro gap analysis said boot counting rolls back an update that does not boot cleanly; that now holds because the boot deadline supplies the reboot. Say so, and how long it takes. Reconcile the "Reboot coordination" rows with projectbluefin#182: the lock files and the Kubernetes interlock both live in 20-interlock.conf, credited to projectbluefin#182. The diskless signal is described as skipping pinned nodes. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks @mrbobbytables. #271 now supersedes this PR, with your operator lock files included.
The I'm leaving this PR open for you or the maintainers to close. |
|
Superseded by PR #271, which integrates this PR's Kubernetes interlock and operator lock-file mechanisms into the FSDK A/B sysupdate architecture. |
…ealth gate and a diskless update signal (#271) * refactor(initrd): share the network boot origin lookup as bluefin-boot-origin Move origin_url() out of bluefin-ignition-credentials into /usr/libexec/bluefin-boot-origin, so the booted OS can find its boot server with the same logic instead of a copy. Order: the StubDeviceURL EFI variable (UEFI HTTP boot), the explicit http(s) rd.systemd.pull= source, then the import.pull credential when the calling unit imports it. Ignition does not import import.pull, so its behaviour is unchanged. Tests drive the helper against fake efivar, cmdline and credential files. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(update): update installed nodes automatically, with a Kubernetes reboot interlock FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and systemd-sysupdate-reboot.timer, so installed nodes never updated on their own and non-Kubernetes hosts never moved to the new slot. 80-bluefin-updates.preset enables both. The timer runs the stock `systemd-sysupdate update`, so enabled features (zfs, kubestellar, kubeadm) keep following the OS in lock-step. Kubernetes nodes keep rebooting through kured. A drop-in adds an ExecCondition= to systemd-sysupdate-reboot.service that skips the local reboot while kubelet, k0scontroller or k0sworker is active, activating, deactivating, reloading or refreshing. This is the interlock from #182 by @mrbobbytables, ported to the FSDK tree. The kured hook also fired on every run: systemd-sysupdate.service is Type=simple, so its ExecStartPost touched /run/reboot-required as soon as an update *started*, even with nothing to download. It is now an ExecStopPost that flags only when `systemd-sysupdate pending` reports an installed update. It moves next to 10-diskless.conf under files/os/systemd/system, and os-kured-hook.bst goes away. The reboot service and both timers get the same 10-diskless.conf conditions as systemd-sysupdate.service, so diskless and USB installer boots stay inert. Refs #139, supersedes #182. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(update): bless a boot-counted UKI only when no unit failed systemd-bless-boot.service requires boot-complete.target, which by itself only needs sysinit.target, so a new UKI was marked good long before the system was up. Enable FSDK's systemd-boot-check-no-failures.service (RequiredBy=boot-complete.target, after multi-user.target), which FSDK's presets disable. A good boot now reaches multi-user.target with no failed unit; any other boot uses up one of the TriesLeft=3 and systemd-boot falls back after the third. Only sysupdate-installed UKIs are counted, so diskless boots never pull in boot-complete.target. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(update): tell diskless nodes when their boot server has a newer release Diskless nodes update by rebooting, but had no way to know a newer release was being served. bluefin-diskless-update-check.timer (5 min after boot, then hourly) runs only when /run/machines/rootdisk.raw exists, the same test 10-diskless.conf uses. It finds the boot directory with bluefin-boot-origin, fetches SHA256SUMS and SHA256SUMS.gpg, verifies them with gpgv against the image keyring, and compares the bluefin-server_<ver>.raw entry with IMAGE_VERSION using `systemd-analyze compare-versions`. When the release is newer it touches /run/reboot-required (kured's sentinel) and logs the new version. Network and signature errors are logged and never set the flag or fail the unit, and the check never reboots. It needs no new runtime dependencies (bash, coreutils, curl, gpgv, systemd-analyze). Behavioural tests run the helper against a local http.server with a throwaway GnuPG key: newer (26.09.10 vs 26.09.2) sets the flag, over the cmdline and the import.pull credential alike. Same or older, a foreign signature, a manifest changed after signing, a missing .gpg, an unreachable server, and a node that was not network booted all leave it unset. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(dogfood): prove automatic updates, the interlock and the health gate in QEMU dogfood-diskless.sh: every boot reports the update timers, and a diskless boot runs bluefin-diskless-update-check once and reports its result and the flag (the CI boot-test job covers the same-version path on every PR). dogfood-install.sh: the installed disk must have both timers enabled and active. The A->B update now runs through systemd-sysupdate.service, the unit the timer starts, and must set /run/reboot-required. With a stand-in kubelet.service running, systemd-sysupdate-reboot.service must be skipped (Result=exec-condition) instead of rebooting. The updated boot must pass systemd-boot-check-no-failures, reach boot-complete.target and be blessed (`systemd-bless-boot status` = good). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): document automatic updates, the kured interlock and the diskless update signal ddi-installer.md "Updates" is the canonical description: timers and preset, the boot health gate, the kured flag and Kubernetes interlock, opting out, the one-time `systemctl preset` for nodes installed before the preset existed, and how kured, operators and Booty act on the diskless flag. The verification skill records that the diskless check trusts the same keyring, and ddi-installer-build.md lists the new dogfood assertions. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): point Kubernetes sysext operators at kured for OS reboots While k0s or kubelet runs, the base image's reboot timer stands down. Say so where operators of those sysexts look, linking to ddi-installer.md. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: mark reboot coordination done; a fleet-wide reboot lock remains Single hosts now reboot into updates in a nightly window and Kubernetes nodes go through kured, so issue #139's gap is closed. What remains is a FleetLock-style lock for non-Kubernetes fleets. Update the readiness audit, the roadmap, the distro gap analysis and the README. Closes #139 Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(update): hold the nightly reboot with operator lock files Fold #182's operator inhibition into this PR's reboot drop-in so the reboot service keeps one interlock instead of two from different directories: systemd-sysupdate-reboot.service is skipped while /run/reboot-lock (until the next boot) or /etc/reboot-lock (until removed) exists. The drop-in is renamed 20-interlock.conf because it now carries more than the Kubernetes check. Lock files from #182 by Bob Killen. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(update): reboot a boot-counted boot that misses boot-complete.target The health gate stopped the bless but nothing rebooted: systemd-boot only moves on at the next boot, and systemd-sysupdate-reboot does nothing once the newest version is the booted one. A node that came up with a failed unit stayed on the unblessed image. bluefin-boot-deadline.timer (preset-enabled) runs only on boot-counted boots: LoaderBootCountPath exists, not diskless, not root=tmpfs. 15 minutes after boot (OnBootSec=, a drop-in changes it), if systemd-bless-boot still says indeterminate or bad and boot-complete.target is not active, bluefin-boot-deadline logs the failed units and reboots, so systemd-boot uses up the next try and falls back after the last. The last try boots "dirty" and reboots only when another UKI can still be booted, so there is no loop without a fallback. The service has DefaultDependencies=no and no ordering on multi-user or boot-complete, so a hung boot is caught too. Kubernetes: while kubelet or k0s is starting, running or stopping, the node may carry workloads, so it touches /run/reboot-required and kured drains and reboots it in its lock, as for an update. A stopped or failed Kubernetes unit leaves the node NotReady with nothing to drain and no kured, so it reboots directly. /run/reboot-lock and /etc/reboot-lock hold it like the nightly reboot. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(update): do not reboot back into an update that failed its boot tries After a rollback the failed version is still installed, and `systemd-sysupdate pending` still reports it: it does not look at the UKI's tries counter. The kured flag would be set after every update run and the nightly reboot would fire every night, each time landing on the previous version again. bluefin-update-pending wraps `systemd-sysupdate pending` and says no while a newer installed version's UKI has no tries left (bluefin-server-<ver>+0-<done>.efi). It gates the kured flag (20-kured.conf) and the nightly reboot (an ExecCondition in 20-interlock.conf). The next newer release takes the failed version's slot and UKI (InstancesMax=2) and clears the block. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(update): do not flag a diskless node pinned to a versioned image When the boot origin names a versioned file, bluefin-server_<ver>.raw from an explicit rd.systemd.pull= URL or the import.pull credential (or a versioned bluefin-server-netboot_<ver>.efi boot URL), the node pulls the same version again after a reboot. Flagging /run/reboot-required there made kured drain and reboot the node on every check while the server directory held a newer release. Flag only when the next boot resolves the version itself: a netboot UKI served under a fixed name (Booty's bluefin-server-netboot.efi via StubDeviceURL) or another unversioned name. A pinned node logs "Bluefin Server <ver> is available from <dir>/ (running <ver>), but this node is pinned to <file>; not flagging". The behavioural tests cover both flagging origins and the three pinned ones, and the signature and network-error cases now run against a flaggable origin so they still prove something. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(elements): describe the boot deadline and update-pending helpers os-update-check.bst now installs bluefin-boot-deadline and bluefin-update-pending next to the diskless update check, and os-k0s-first-boot.bst the deadline units and the reboot interlock. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(dogfood): prove the boot-deadline rollback of a failed unit in QEMU DOGFOOD_BROKEN=unit breaks <broken-dir> with a unit that fails on that version only (ConditionOSRelease=IMAGE_VERSION=) instead of corrupting its slot, and shortens the boot deadline (DOGFOOD_DEADLINE, default 2min). Each counted boot reaches multi-user.target, misses boot-complete.target and waits for bluefin-boot-deadline to reboot it. The run asserts three such boots, the fallback to <next-dir> with the deadline timer inactive and the boot clean, and then no kured flag and a skipped systemd-sysupdate-reboot.service for the failed version. Every disk boot also reports the deadline timer's state. The default slot mode is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): document the boot deadline, the reboot interlock and pinned diskless origins ddi-installer.md "Updates": boot counting alone only rolls back on the next boot, so describe bluefin-boot-deadline (when it runs, why a Kubernetes node is flagged for kured instead of rebooted, the dirty last try, the drop-in to change the deadline), bluefin-update-pending after a rollback, the lock files in 20-interlock.conf (credited to #182), and which diskless origins are pinned. ddi-installer-build.md covers DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): tell Kubernetes sysext operators the rollback reboot goes through kured While kubelet or k0s runs, bluefin-boot-deadline flags /run/reboot-required instead of rebooting, so kured also rolls back an unhealthy update. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: make the rollback and reboot coordination claims precise The README, readiness audit, roadmap and distro gap analysis said boot counting rolls back an update that does not boot cleanly; that now holds because the boot deadline supplies the reboot. Say so, and how long it takes. Reconcile the "Reboot coordination" rows with #182: the lock files and the Kubernetes interlock both live in 20-interlock.conf, credited to #182. The diskless signal is described as skipping pinned nodes. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolves #139
Supersedes closed #145, resolving both architectural blockers identified in review by @castrojo:
files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.confusesExecConditionto check whether any Kubernetes service is active (systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service). When Kubernetes is active on the node,systemd-sysupdate-reboot.serviceskips reboot execution so that Kured can coordinate the cluster cordon, drain, and rolling reboot through the Kubernetes API.files/os/systemd/system/that collide with upstreamsystemd.bst, upstream systemd ownssystemd-sysupdate-reboot.serviceandsystemd-sysupdate-reboot.timer. We ship:files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.confviaelements/bluefin-server/os-sysupdate-reboot.bstto configure lock inhibition (/run/reboot-lock,/etc/reboot-lock) and Kubernetes service gating.files/os/systemd/system-preset/80-enable-sysupdate-reboot.presetto enablesystemd-sysupdate-reboot.timerfor automated maintenance-window reboots (04:10 daily with 30min randomized delay), ordered before any potential 90-level presets to ensure priority evaluation under systemd lexicographic rules.docs/skills/architecture-roadmap.md,docs/skills/gap-analysis-distros.md, anddocs/skills/k0s-sysext-ops.mdaccurately with verified file citations.tests/unit/test_reboot_coordination.pyverifying drop-in configuration, upstream unit non-collision, and functional execution testing of the Kubernetes interlock logic under active and inactive states.— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:96a8d42