Skip to content

feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts - #182

Closed
mrbobbytables wants to merge 5 commits into
projectbluefin:mainfrom
mrbobbytables:feat/sysupdate-reboot-coordination-native
Closed

mrbobbytables wants to merge 5 commits into
projectbluefin:mainfrom
mrbobbytables:feat/sysupdate-reboot-coordination-native

Conversation

@mrbobbytables

@mrbobbytables mrbobbytables commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Resolves #139

Supersedes closed #145, resolving both architectural blockers identified in review by @castrojo:

  1. Real Kubernetes Interlock: Rather than relying on an imaginary sentinel, files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf uses ExecCondition to check whether any Kubernetes service is active (systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service). When Kubernetes is active on the node, systemd-sysupdate-reboot.service skips reboot execution so that Kured can coordinate the cluster cordon, drain, and rolling reboot through the Kubernetes API.
  2. Upstream Unit Ownership: Instead of creating duplicate unit files in files/os/systemd/system/ that collide with upstream systemd.bst, upstream systemd owns systemd-sysupdate-reboot.service and systemd-sysupdate-reboot.timer. We ship:
    • files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf via elements/bluefin-server/os-sysupdate-reboot.bst to configure lock inhibition (/run/reboot-lock, /etc/reboot-lock) and Kubernetes service gating.
    • files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset to enable systemd-sysupdate-reboot.timer for automated maintenance-window reboots (04:10 daily with 30min randomized delay), ordered before any potential 90-level presets to ensure priority evaluation under systemd lexicographic rules.
  3. Documentation & Tests:
    • Updates docs/skills/architecture-roadmap.md, docs/skills/gap-analysis-distros.md, and docs/skills/k0s-sysext-ops.md accurately with verified file citations.
    • Comprehensive unit tests in tests/unit/test_reboot_coordination.py verifying drop-in configuration, upstream unit non-collision, and functional execution testing of the Kubernetes interlock logic under active and inactive states.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 96a8d42

castrojo
castrojo previously approved these changes Sep 18, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. This resolves both blockers I raised when closing #145, and it does so the right way rather than the expedient way.

Blocker 1 — the imaginary sentinel — is properly fixed. #145 gated on ConditionPathExists=!/run/kured-active, a file nothing in this repo or Kured ever creates, so on a Kubernetes node the condition always passed and the timer would reboot out from under Kured with no cordon and no drain. This replaces it with a real liveness check:

ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service'

I verified the semantics empirically rather than trusting my reading of them, because this is the whole safety property and I initially thought it was wrong. systemctl is-active with multiple units exits 0 if any listed unit is active (confirmed against a real active/inactive pair: is-active <active> <inactive> -> 0, is-active <inactive> -> 3). So:

  • Kubernetes running (any of the three active) -> is-active 0 -> ! -> 1 -> ExecCondition non-zero -> unit is skipped cleanly, Kured retains control.
  • No Kubernetes -> is-active 3 -> ! -> 0 -> ExecCondition passes -> local reboot proceeds.

That is exactly the intended behaviour, and using ExecCondition rather than ExecStartPre matters: a non-zero ExecCondition in the 1-254 range skips the unit without marking it failed, so a k0s node does not accumulate failed units every night.

Checking all three of kubelet.service, k0scontroller.service and k0sworker.service is the right net — k0s embeds its own kubelet, so a controller node may have k0scontroller active without kubelet.service existing at all, and the any-active semantics cover controller, worker and generic-kubelet hosts alike.

Blocker 2 — shadowing upstream's units — is properly fixed. #145 shipped its own systemd-sysupdate-reboot.service and .timer into /usr/lib/systemd/system/, the same path components/systemd.bst installs to, silently overriding units systemd itself ships and dropping upstream's Requires=/After= ordering. This instead ships a drop-in: os-sysupdate-reboot.bst is kind: import targeting /usr/lib/systemd/system/systemd-sysupdate-reboot.service.d, with a preset to enable the timer. Upstream owns the unit; we own the configuration. That is the systemd-native shape and it satisfies one-canonical-source.

The docs are honest, which I appreciate. gap-analysis-distros.md does not simply delete the gap — it splits it into a Status line for what now works (single-node, maintenance window, lock-file inhibition) and a remaining Gap for what does not (a cluster-wide FleetLock/locksmithd HTTP lock client). The roadmap row 6 is rewritten to match rather than removed. And k0s-sysext-ops.md gains a Reboot Coordination section — that file is named in docs/skills/index.md as the runbook for exactly this topic, and its omission was one of my complaints about #145.

The lock-file inhibition is now real rather than advertised: ConditionPathExists=!/run/reboot-lock and !/etc/reboot-lock are conditions on a unit that exists, giving operators a runtime and a persistent escape hatch.

CI: docs and unit pass. build is still running at the time of this review; it should be green before merge, and I would not merge on my approval alone regardless. installer-test will remain red for the pre-existing /var reason (#173) that fails on main itself.

One thing for the next reviewer to sanity-check rather than assume, since I could not exercise it: the timer's 04:10 maintenance window with randomized delay is inherited from upstream's .timer plus the preset, so confirm the preset actually enables the timer in the built image — the drop-in only configures the service.

Approving. Needs one more independent approval under the two-reviewer policy.

Thank you for turning #145 around this quickly and for taking the harder path on both points instead of patching the sentinel.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed (correctness) — no blockers; one edge case worth noting.

Verified the key mechanics at head e306974:

  • ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service' (files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf:10) is correct: per systemd's check_unit_generic (src/systemctl/systemctl-is-active.c), multi-unit is-active exits 0 if at least one unit is active, so the negation skips reboot whenever any K8s service runs. The mock-systemctl test (tests/unit/test_reboot_coordination.py:104-160) matches this semantic; all 6 tests pass locally.
  • The new preset (files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset:2) is shipped because elements/bluefin-server/os-sshd-preset.bst:7-10 imports the entire files/os/systemd/system-preset directory — same path other zz-* presets rely on.
  • FSDK 26.08 pin (elements/freedesktop-sdk.bst) ships systemd ≥257, so upstream systemd-sysupdate-reboot.{service,timer} exist and the 04:10/30min claims in docs/skills/k0s-sysext-ops.md:53-55 match upstream units/systemd-sysupdate-reboot.timer.

Low — single-node k0s hosts never auto-reboot without Kured. The ExecCondition gates on k0s service liveness, not on Kured being deployed (reboot-coordination.conf:10). A single-node host running k0s without Kured installed — squarely within this PR's title scope — will stage updates via sysupdate but never apply them: the local timer path is skipped and no cluster coordinator exists. docs/skills/k0s-sysext-ops.md:52-53 states Kured handles the reboot but nothing in the image ships or verifies Kured. Consider documenting that k0s hosts must deploy Kured (or remove the lock files and reboot manually), or the update loop silently stalls.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @kubestellar-hive in commit d86399f:

Updated docs/skills/k0s-sysext-ops.md to clarify that k0s hosts must deploy Kured (or an equivalent cluster coordinator), and documented that uncoordinated local reboots remain inhibited on single-node k0s hosts when Kured is not deployed.

🐝 Hive Agent: contributor | SHA: unknown

@castrojo

Copy link
Copy Markdown
Contributor

Needs a rebase — one path conflicts with #210.

This PR touches files/k0s/sysext/k0s-manifests.conf, which is deleted there. The rest (files/os/sysupdate.d/, the reboot-coordination units, the tests) applies cleanly in principle.

The design also gets simpler after #210: the reboot coordinator no longer has to special-case "k0s is the cluster". With upstream kubeadm/kubelet the node is a normal Kubernetes node, so kubectl drain against the local kubeconfig is the whole story for the clustered case, and the single-node case is the bluefin-cluster-bootstrap.service host. Worth re-reading docs/superpowers/specs/2026-09-19-kubernetes-cutover-design.md before rebasing.

Refs #139.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

installer-test / Install and boot in QEMU is failing on this PR because it is failing on main itself, not because of anything in this diff.

Tracked as a single repository incident in #218. main has been red on this check since 2026-09-14; the same failure currently blocks 17 open PRs, including a docs-only PR (#208) and a Renovate digest bump (#49).

Failure mode is /dev/disk/by-partlabel/var never appearing, so /var and everything ordered after it depend-fail and the readiness probe times out.

Please don't retry this check per-PR — it will stay red until #218 is resolved.

— hive: agent=scanner backend=copilot model=claude-opus-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • (low) The Kubernetes interlock is point-in-time: ExecCondition=... '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service' (files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf:10) passes whenever no unit is in active state at the 04:10 timer fire. systemctl is-active reports non-zero for activating/failed, so a Kubernetes node whose kubelet/k0s service is mid-restart or crashed at that moment reboots uncoordinated (no cordon/drain). Low likelihood given the 30-min randomized window, but worth a note or a Requisite/retry consideration if it matters for larger fleets.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @kubestellar-hive in commit 288caa0:

  • Added inline documentation to files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf and updated docs/skills/k0s-sysext-ops.md explaining that the ExecCondition evaluates point-in-time unit state at timer trigger, and noting that operators undergoing planned service restarts or maintenance can create /run/reboot-lock for deterministic inhibition across transient unit states.

Regarding @castrojo's rebase note on #210: acknowledged; keeping an eye on #210 and will rebase onto main once #210 lands to adopt the simplified kubeadm/kubelet coordination design.

🐝 Hive Agent: contributor | SHA: 288caa0

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

The new preset never takes effect — the reboot timer stays disabled, making the whole feature inert. freedesktop-sdk's systemd ships /usr/lib/systemd/system-preset/90-sysupdate.preset containing disable systemd-sysupdate-reboot.timer (installed by FSDK 26.08 components/_private/systemd-base.bst, line 130 of the pinned freedesktop-sdk-26.08.0 tag). Per systemd.preset(5), preset files are evaluated in lexicographic filename order and the earliest matching line wins — 90-sysupdate.preset sorts before zz-enable-sysupdate-reboot.preset (files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset:2), so on first-boot preset-all the timer remains disabled. The drop-in (files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) is installed but never triggered, and issue #139's acceptance criterion ("a single-node VM defers reboots according to the configured strategy") is not met. Fix: name the preset so it sorts before 90- (e.g. 80-enable-sysupdate-reboot.preset). Note the same first-match rule means tests/unit/test_reboot_coordination.py:114 (test_reboot_preset_enables_timer) passes while the shipped image behavior is wrong — a test asserting the preset filename sorts before 90-sysupdate.preset would catch this.

No findings from: security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed in commit 96a8d42:

  • Preset Lexicographic Ordering: Renamed zz-enable-sysupdate-reboot.preset to 80-enable-sysupdate-reboot.preset so it sorts before freedesktop-sdk's 90-sysupdate.preset (which disables the timer).
  • Test Enforcement: Added assertion in test_reboot_preset_enables_timer() confirming the preset filename evaluates ahead of 90-sysupdate.preset.
  • Docs: Updated references in docs/skills/gap-analysis-distros.md.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • The PR body says the preset is files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset, but the diff ships files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset. Behavior is unaffected (the whole system-preset dir is imported by elements/bluefin-server/os-sshd-preset.bst:6-10), but the description should match the file actually added.

style

  • tests/unit/test_reboot_coordination.py:106-112 justifies the 80- prefix by asserting the name sorts before a freedesktop-sdk 90-sysupdate.preset "which disables the timer" — I could not find such a preset (upstream systemd's presets/90-systemd.preset has no systemd-sysupdate-reboot line, and nothing matching turned up in FSDK). The assertion is trivially true and harmless, but the comment encodes an unverified claim, and the new 80- name breaks the repo's existing zz-* preset convention (zz-enable-sshd.preset, zz-enable-networkd.preset, etc.) without a confirmed reason.

Verified while reviewing (head 96a8d42): upstream systemd v257+ ships systemd-sysupdate-reboot.service/.timer and systemd-sysupdate reboot only reboots when a newer version than the booted one is installed, so enabling the timer fleet-wide is safe; systemctl is-active exits 0 if at least one listed unit is active, matching the intended k8s gate in reboot-coordination.conf:13.

No findings from: correctness, security, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Updated the PR description to accurately reflect files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset. The 80- prefix was introduced to ensure lexicographic precedence under systemd preset ordering (/usr/lib/systemd/system-preset/), avoiding any override by 90-series presets. All tests in tests/unit/test_reboot_coordination.py pass.

🐝 Hive Agent: contributor | SHA: unknown

@Danathar

Copy link
Copy Markdown
Contributor

#220 merged to main at 14:06 UTC today with the fix for the installer-test boot deadlock (udev sysext ordering, depmod path, homed-firstboot mask). This branch predates it, so its installer-test / Install and boot in QEMU failure is the old repo-wide one, not this change. A rebase onto current main is all it needs to get a real result from that check; I cannot update the branch from here.

@castrojo
castrojo force-pushed the feat/sysupdate-reboot-coordination-native branch from 96a8d42 to 24441c7 Compare September 22, 2026 20:44

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Danathar
Danathar previously approved these changes Sep 22, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 24441c7. This PR turns on upstream's systemd-sysupdate-reboot.timer for hosts that don't run Kubernetes, and adds a drop-in so the reboot service stands down when a lock file exists or a Kubernetes unit is active.

What I checked:

  • I read the full diff against main (dc48bfb). The branch contains the boot-deadlock fixes (#220 and #227), so the installer check is now a real result.
  • CI passed on 24441c7, and I read the runs. In run 35782269073, build and installer-test / Install and boot in QEMU passed. Unit tests passed in run 35782267704.
  • I checked the preset order against the pinned FSDK (26.08.0, systemd v261.2). FSDK's 90-sysupdate.preset disables systemd-sysupdate-reboot.timer. 80-enable-sysupdate-reboot.preset sorts first, and the first match wins, so the timer ends up enabled. os-sshd-preset.bst already imports the whole files/os/systemd/system-preset directory, so the new file ships without a new element.
  • systemctl is-active --quiet a b c exits 0 if any listed unit is active. So the ExecCondition skips the reboot whenever kubelet or either k0s unit is running, and kubelet.service still covers the kubeadm path from #210.
  • pytest tests/unit/test_reboot_coordination.py: 6 passed. When I dropped the ! from the ExecCondition, the interlock test failed.

One thing for a follow-up, not this PR: the same FSDK preset also disables systemd-sysupdate.timer, and nothing in this repo turns it back on. The upstream reboot service runs systemd-sysupdate reboot, which only reboots when a newer version is already installed. So on current images this timer reboots only after someone stages an update by hand. It does not complete an automatic update loop.

@castrojo
castrojo force-pushed the feat/sysupdate-reboot-coordination-native branch from 24441c7 to b4c3320 Compare September 23, 2026 19:55
@castrojo

Copy link
Copy Markdown
Contributor

Rebased onto main (conflict resolved in docs/skills/architecture-roadmap.md by aligning roadmap item status with current main credentials integration and PR #182 reboot coordination). Verified locally with pytest tests/unit, bats tests/unit, and release/k0s version scripts.

@Danathar

Copy link
Copy Markdown
Contributor

I re-read this after the rebase to b4c3320. The patch itself is the same one I approved at 24441c7: git range-diff shows commits 2 and 3 unchanged, and the only content change in 1 and 4 is the roadmap row moving from item 6 to item 5 to follow main.

What did change is the base under it. #132 merged in between, so the installed OS now gets systemd from Flatcar's /usr instead of from FSDK. That changes the story for the preset file, though not for the drop-in.

The preset no longer does anything, as far as I can tell. I pulled the pinned Flatcar container (4593.2.5, sha256 c533ff57… matches include/flatcar.yml) and listed it:

What In Flatcar 4593.2.5 /usr
systemd-sysupdate-reboot.timer / .service shipped (systemd 257)
lib/systemd/system/timers.target.wants/systemd-sysupdate-reboot.timer present, a symlink, so the timer is already enabled with no preset
lib/systemd/system/timers.target.wants/systemd-sysupdate.timer present too
lib/systemd/system-preset/ only 99-default.preset (disable *) and 90-google-compute-engine-oslogin.preset; no 90-sysupdate.preset

Command: curl -sL https://flatcar.cdn.cncf.io/stable/amd64-usr/4593.2.5/flatcar-container.tar.gz | tar tz | grep -E 'sysupdate|system-preset/'

So 80-enable-sysupdate-reboot.preset is harmless, but its reason is gone. The comments that point at FSDK are now out of date: the one in test_reboot_preset_enables_timer ("freedesktop-sdk ships 90-sysupdate.preset which disables the timer"), the one in test_no_upstream_unit_collision (units "shipped by upstream systemd in freedesktop-sdk.bst:components/systemd.bst"), and the commit title of b4c3320. I'd either drop the preset or say that Flatcar enables the timer itself.

The drop-in matters more now, not less. If I'm reading the listing right, main already enables both systemd-sysupdate.timer and systemd-sysupdate-reboot.timer through Flatcar's own symlinks, with no Kubernetes gate and no lock file check. That would mean a node running kubelet can reboot itself at 04:10 after staging an update, without Kured. I haven't confirmed that on a booted image, so treat it as a theory. reboot-coordination.conf in this PR is what adds the gate. The /usr/bin/sh -c '! systemctl is-active …' line still works on Flatcar: /usr/bin/sh is a symlink to bash there, and /usr/bin/systemctl exists.

CI on this head. build failed in run 35912550557 at flatcar/flatcar-usr.bst (rm .../sysinit.target.wants/ensure-sysext.service exits 2). Main fails the same way at e4d9793, the commit this branch is based on (run 35858625353). Main's build passes again at a0f6502 (run 35917446585), so rebasing onto current main should get a real build result. installer-test on current main still fails for everyone: it times out waiting for /dev/disk/by-label/OEM.

Locally at b4c3320: pytest tests/unit 339 passed, 1 xfailed; bats tests/unit 125 tests, 0 failures; docs-checks.py passed.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf drop-in is valuable, more so now that Flatcar 4593.2.5 enables systemd-sysupdate-reboot.timer via timers.target.wants with no Kubernetes gate. Happy to approve after a few updates:

  • Please rebase; build failed at flatcar-usr.bst, which was a main-branch failure at e4d9793 and is fixed now.
  • 80-enable-sysupdate-reboot.preset looks redundant now, and the comments in tests/unit/test_reboot_coordination.py referencing FSDK's 90-sysupdate.preset are out of date. Drop the preset or reword it.
  • Product gap: the default appliance runs k0s and no Kured ships in the manifests, so with this gate staged updates never reboot on a default install. Please document that (or ship Kured) and consider a max-deferral.

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: "A re-run won't pick up the fix. build.yml checks out pull_request.head.ref, so it builds b4c3320 as-is, and this branch's flatcar-usr.bst still runs tar -xzf without the --no-same-owner that #237 added on main. It needs the rebase hanthor asked for."

mrbobbytables and others added 5 commits September 26, 2026 20:30
…de hosts

Configure systemd-sysupdate-reboot coordination for non-Kubernetes and
single-node hosts using upstream systemd units with a drop-in configuration
and systemd preset.

- Avoid colliding with upstream systemd units by configuring upstream's
  systemd-sysupdate-reboot.service via a drop-in under
  files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf
  instead of overriding the unit files.
- Enable systemd-sysupdate-reboot.timer by default via
  files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset.
- Support lock-based reboot inhibition via ConditionPathExists on
  /run/reboot-lock and /etc/reboot-lock.
- Gate reboots via ExecCondition checking active Kubernetes services
  (k0scontroller.service, kubelet.service, k0sworker.service), ensuring
  Kubernetes clusters preserve Kured's cordon and drain workflow.
- Package drop-in through elements/bluefin-server/os-sysupdate-reboot.bst
  and wire into os-stack.bst.
- Add unit test coverage in tests/unit/test_reboot_coordination.py.
- Update docs in architecture-roadmap.md, gap-analysis-distros.md, and
  k0s-sysext-ops.md.

Closes projectbluefin#139

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ured

Clarify that k0s hosts must deploy Kured or an equivalent coordinator, and that uncoordinated local reboots remain inhibited on single-node k0s hosts when Kured is absent, per review feedback from @kubestellar-hive.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ock inhibition

Document in reboot-coordination.conf and k0s-sysext-ops.md that the
ExecCondition evaluates point-in-time liveness at timer fire, and recommend
using /run/reboot-lock during planned cluster maintenance or service restarts
to deterministically inhibit reboots across transient unit state transitions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Rename zz-enable-sysupdate-reboot.preset to 80-enable-sysupdate-reboot.preset
so that under systemd.preset lexicographic ordering it evaluates before
freedesktop-sdk's 90-sysupdate.preset (which disables systemd-sysupdate-reboot.timer),
ensuring the timer is enabled on first boot. Add unit test assertion for preset ordering.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
…ault-install gap

Flatcar's /usr (projectbluefin#132) already enables systemd-sysupdate-reboot.timer via
timers.target.wants and ships no 90-sysupdate.preset, so the 80- preset
no longer does anything. Remove it, its test, and the FSDK-era comments.

Document that the default k0s appliance ships no Kured, so with the
drop-in's Kubernetes gate a staged update waits for a manual reboot.

Assisted-by: Claude via omp
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@castrojo
castrojo force-pushed the feat/sysupdate-reboot-coordination-native branch from b4c3320 to fa57e7b Compare September 27, 2026 00:31
@castrojo

Copy link
Copy Markdown
Contributor

Addressed @hanthor's review (COMMENTED at b4c3320) in one push; head is now fa57e7b.

  1. Rebase - rebased onto main 452503a (includes chore(ci): pin runners to ubuntu-26.04 #237's --no-same-owner fix for flatcar-usr.bst, which is what failed build at b4c3320). Commits 1-4 unchanged in content.
  2. Redundant preset / stale FSDK comments - fa57e7b removes files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset (Flatcar 4593.2.5 enables the timer via timers.target.wants and ships no 90-sysupdate.preset, per Danathar's listing), drops test_reboot_preset_enables_timer and the preset existence assert, rewrites the test_no_upstream_unit_collision comment to point at Flatcar's /usr, and removes the preset from the gap-analysis sources.
  3. Product gap - fa57e7b adds a "Default appliance install" bullet to docs/skills/k0s-sysext-ops.md: the default install runs k0s with no Kured, so the gate stands down and a staged update waits for a manual reboot or a Kured deployment. Max-deferral / shipping Kured is left out of this PR as a separate design change.

Local verification at fa57e7b: pytest tests/unit 425 passed, 1 xfailed; bats tests/unit all ok; .github/scripts/docs-checks.py passed. castrojo pushed fa57e7b, so this needs an independent approval.

@castrojo
castrojo requested a review from hanthor September 27, 2026 00:31
@castrojo

Copy link
Copy Markdown
Contributor

This has conflicts with main again, and I tried a rebase that isn't mechanical, so I aborted it. Since fa57e7b, main has taken #262 (feat!: FSDK-composed OS with a signed, diskless-first boot chain) and #264 (offline USB installer). That changes two things this PR depends on:

  • fa57e7b dropped the reboot preset because Flatcar's /usr enabled systemd-sysupdate-reboot.timer through timers.target.wants. /usr is FSDK-composed again, so that is no longer true, and the preset question is open again.
  • main now masks systemd-sysupdate on diskless and installer boots (files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf: ConditionPathExists=!/run/machines/rootdisk.raw, ConditionKernelCommandLine=!root=tmpfs). docs/skills/gap-analysis-distros.md on main lists 'systemd-sysupdate-reboot not enabled' as an installed-node-only gap. The reboot unit needs the same diskless guard, and the roadmap and gap-analysis rows need rewriting against the new state model. They conflict in architecture-roadmap.md, gap-analysis-distros.md and k0s-sysext-ops.md.
    main needs 0 approvals, so once this is rebased and CI is green on the new head, it can merge. @mrbobbytables, can you rework it against 8782952?

@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Thanks @mrbobbytables. The Kubernetes interlock from this PR (an ExecCondition= on systemd-sysupdate-reboot.service that skips the local reboot while kubelet.service / k0scontroller.service / k0sworker.service runs, leaving the reboot to kured) is carried forward in #271 and credited there.

#271 ports it to the current FSDK tree (this branch predates the FSDK migration and now conflicts). It also counts activating/deactivating/reloading as running, so a restarting kubelet doesn't open a window for a reboot without a drain. The PR also:

  • enables systemd-sysupdate.timer and systemd-sysupdate-reboot.timer by preset (FSDK's 90-sysupdate.preset disables them);
  • fixes the kured hook, which was an ExecStartPost on a Type=simple unit and so flagged a reboot on every run; it is now an ExecStopPost gated on systemd-sysupdate pending;
  • adds a boot health gate (systemd-boot-check-no-failures.service) and a signed update signal for diskless nodes.

It is proven in QEMU, including the interlock skipping the reboot (Result=exec-condition) with a staged update pending. I've left this PR open for you or the maintainers to close in favour of #271.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked this branch (fa57e7b) against current main (8782952) by merging main into it locally. Merging main in leaves the PR functionally inert, not just conflicted:

  • The premise of fa57e7b no longer holds on main. fa57e7b dropped 80-enable-sysupdate-reboot.preset because "Flatcar's /usr enables systemd-sysupdate-reboot.timer through its own timers.target.wants symlink". #262 replaced the Flatcar /usr with an FSDK-composed one (junction freedesktop-sdk-26.08.0). In that FSDK, 90-sysupdate.preset disables systemd-sysupdate-reboot.timer, which is what Danathar checked against 26.08.0 earlier in this thread. Rebased as it is, the timer stays disabled and the drop-in never runs. The preset has to come back, or the timer has to be enabled some other way.
  • Layout. Main now keeps unit drop-ins under files/os/systemd/system/<unit>.d/ (for example files/os/systemd/system/systemd-sysupdate.service.d/10-diskless.conf) and presets as 80-bluefin-*.preset. This branch still adds files/os/systemd/systemd-sysupdate-reboot.service.d/ behind its own os-sysupdate-reboot.bst.
  • Conflicts. Merging main hits content conflicts in docs/skills/architecture-roadmap.md, docs/skills/gap-analysis-distros.md and docs/skills/k0s-sysext-ops.md. Main rewrote the state-model, updates and reboot-coordination rows for the FSDK/diskless design, so this branch's text in those rows describes the Flatcar-era tree (ID=flatcar, rw root, 50-root.transfer).
  • Overlap with #271. #271 enables systemd-sysupdate-reboot.timer through 80-bluefin-updates.preset and adds systemd-sysupdate-reboot.service.d/20-kubernetes.conf with the same kubelet/k0scontroller/k0sworker interlock. Its version also treats activating/deactivating/reloading as running, which systemctl is-active --quiet here doesn't. What #182 has that #271 doesn't is the /run/reboot-lock and /etc/reboot-lock inhibition. If #271 lands, those two ConditionPathExists= lines are all that is left for this PR to contribute.

I did not push anything. The rework above goes beyond a mechanical rebase.

jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
Fold projectbluefin#182's operator inhibition into this PR's reboot drop-in so the
reboot service keeps one interlock instead of two from different
directories: systemd-sysupdate-reboot.service is skipped while
/run/reboot-lock (until the next boot) or /etc/reboot-lock (until
removed) exists. The drop-in is renamed 20-interlock.conf because it
now carries more than the Kubernetes check.

Lock files from projectbluefin#182 by Bob Killen.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
…nned diskless origins

ddi-installer.md "Updates": boot counting alone only rolls back on the
next boot, so describe bluefin-boot-deadline (when it runs, why a
Kubernetes node is flagged for kured instead of rebooted, the dirty last
try, the drop-in to change the deadline), bluefin-update-pending after a
rollback, the lock files in 20-interlock.conf (credited to projectbluefin#182), and
which diskless origins are pinned. ddi-installer-build.md covers
DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
The README, readiness audit, roadmap and distro gap analysis said boot
counting rolls back an update that does not boot cleanly; that now holds
because the boot deadline supplies the reboot. Say so, and how long it
takes. Reconcile the "Reboot coordination" rows with projectbluefin#182: the lock files
and the Kubernetes interlock both live in 20-interlock.conf, credited to
projectbluefin#182. The diskless signal is described as skipping pinned nodes.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Thanks @mrbobbytables. #271 now supersedes this PR, with your operator lock files included.

systemd-sysupdate-reboot.service.d/20-interlock.conf (commit 6d2d3a6 on #271) carries:

The gap-analysis-distros.md "Reboot coordination" rows are reconciled there too, credited to #182; the FleetLock-style cluster lock stays listed as the remaining gap. The timer enablement is a preset there (80-bluefin-updates.preset), and it runs in QEMU in scripts/dogfood-install.sh.

I'm leaving this PR open for you or the maintainers to close.

@castrojo

Copy link
Copy Markdown
Contributor

Superseded by PR #271, which integrates this PR's Kubernetes interlock and operator lock-file mechanisms into the FSDK A/B sysupdate architecture.

@castrojo castrojo closed this Sep 29, 2026
jeefy added a commit that referenced this pull request Sep 29, 2026
…ealth gate and a diskless update signal (#271)

* refactor(initrd): share the network boot origin lookup as bluefin-boot-origin

Move origin_url() out of bluefin-ignition-credentials into
/usr/libexec/bluefin-boot-origin, so the booted OS can find its boot
server with the same logic instead of a copy. Order: the StubDeviceURL EFI
variable (UEFI HTTP boot), the explicit http(s) rd.systemd.pull= source,
then the import.pull credential when the calling unit imports it.

Ignition does not import import.pull, so its behaviour is unchanged. Tests
drive the helper against fake efivar, cmdline and credential files.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(update): update installed nodes automatically, with a Kubernetes reboot interlock

FSDK's 90-sysupdate.preset disables systemd-sysupdate.timer and
systemd-sysupdate-reboot.timer, so installed nodes never updated on their
own and non-Kubernetes hosts never moved to the new slot.
80-bluefin-updates.preset enables both. The timer runs the stock
`systemd-sysupdate update`, so enabled features (zfs, kubestellar,
kubeadm) keep following the OS in lock-step.

Kubernetes nodes keep rebooting through kured. A drop-in adds an
ExecCondition= to systemd-sysupdate-reboot.service that skips the local
reboot while kubelet, k0scontroller or k0sworker is active, activating,
deactivating, reloading or refreshing. This is the interlock from #182 by
@mrbobbytables, ported to the FSDK tree.

The kured hook also fired on every run: systemd-sysupdate.service is
Type=simple, so its ExecStartPost touched /run/reboot-required as soon as
an update *started*, even with nothing to download. It is now an
ExecStopPost that flags only when `systemd-sysupdate pending` reports an
installed update. It moves next to 10-diskless.conf under
files/os/systemd/system, and os-kured-hook.bst goes away.

The reboot service and both timers get the same 10-diskless.conf
conditions as systemd-sysupdate.service, so diskless and USB installer
boots stay inert.

Refs #139, supersedes #182.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(update): bless a boot-counted UKI only when no unit failed

systemd-bless-boot.service requires boot-complete.target, which by
itself only needs sysinit.target, so a new UKI was marked good long
before the system was up. Enable FSDK's
systemd-boot-check-no-failures.service (RequiredBy=boot-complete.target,
after multi-user.target), which FSDK's presets disable. A good boot now
reaches multi-user.target with no failed unit; any other boot uses up one
of the TriesLeft=3 and systemd-boot falls back after the third.

Only sysupdate-installed UKIs are counted, so diskless boots never pull in
boot-complete.target.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(update): tell diskless nodes when their boot server has a newer release

Diskless nodes update by rebooting, but had no way to know a newer
release was being served. bluefin-diskless-update-check.timer (5 min
after boot, then hourly) runs only when /run/machines/rootdisk.raw exists,
the same test 10-diskless.conf uses. It finds the boot directory with
bluefin-boot-origin, fetches SHA256SUMS and SHA256SUMS.gpg, verifies them
with gpgv against the image keyring, and compares the
bluefin-server_<ver>.raw entry with IMAGE_VERSION using
`systemd-analyze compare-versions`. When the release is newer it touches
/run/reboot-required (kured's sentinel) and logs the new version.

Network and signature errors are logged and never set the flag or fail
the unit, and the check never reboots. It needs no new runtime
dependencies (bash, coreutils, curl, gpgv, systemd-analyze).

Behavioural tests run the helper against a local http.server with a
throwaway GnuPG key: newer (26.09.10 vs 26.09.2) sets the flag, over the
cmdline and the import.pull credential alike. Same or older, a foreign
signature, a manifest changed after signing, a missing .gpg, an
unreachable server, and a node that was not network booted all leave it
unset.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(dogfood): prove automatic updates, the interlock and the health gate in QEMU

dogfood-diskless.sh: every boot reports the update timers, and a diskless
boot runs bluefin-diskless-update-check once and reports its result and
the flag (the CI boot-test job covers the same-version path on every PR).

dogfood-install.sh: the installed disk must have both timers enabled and
active. The A->B update now runs through systemd-sysupdate.service, the
unit the timer starts, and must set /run/reboot-required. With a
stand-in kubelet.service running, systemd-sysupdate-reboot.service must
be skipped (Result=exec-condition) instead of rebooting. The updated
boot must pass systemd-boot-check-no-failures, reach boot-complete.target
and be blessed (`systemd-bless-boot status` = good).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): document automatic updates, the kured interlock and the diskless update signal

ddi-installer.md "Updates" is the canonical description: timers and
preset, the boot health gate, the kured flag and Kubernetes interlock,
opting out, the one-time `systemctl preset` for nodes installed before the
preset existed, and how kured, operators and Booty act on the diskless
flag. The verification skill records that the diskless check trusts the
same keyring, and ddi-installer-build.md lists the new dogfood assertions.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): point Kubernetes sysext operators at kured for OS reboots

While k0s or kubelet runs, the base image's reboot timer stands down.
Say so where operators of those sysexts look, linking to ddi-installer.md.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: mark reboot coordination done; a fleet-wide reboot lock remains

Single hosts now reboot into updates in a nightly window and Kubernetes
nodes go through kured, so issue #139's gap is closed. What remains is a
FleetLock-style lock for non-Kubernetes fleets. Update the readiness
audit, the roadmap, the distro gap analysis and the README.

Closes #139

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(update): hold the nightly reboot with operator lock files

Fold #182's operator inhibition into this PR's reboot drop-in so the
reboot service keeps one interlock instead of two from different
directories: systemd-sysupdate-reboot.service is skipped while
/run/reboot-lock (until the next boot) or /etc/reboot-lock (until
removed) exists. The drop-in is renamed 20-interlock.conf because it
now carries more than the Kubernetes check.

Lock files from #182 by Bob Killen.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): reboot a boot-counted boot that misses boot-complete.target

The health gate stopped the bless but nothing rebooted: systemd-boot only
moves on at the next boot, and systemd-sysupdate-reboot does nothing once
the newest version is the booted one. A node that came up with a failed
unit stayed on the unblessed image.

bluefin-boot-deadline.timer (preset-enabled) runs only on boot-counted
boots: LoaderBootCountPath exists, not diskless, not root=tmpfs. 15
minutes after boot (OnBootSec=, a drop-in changes it), if
systemd-bless-boot still says indeterminate or bad and
boot-complete.target is not active, bluefin-boot-deadline logs the failed
units and reboots, so systemd-boot uses up the next try and falls back
after the last. The last try boots "dirty" and reboots only when another
UKI can still be booted, so there is no loop without a fallback. The
service has DefaultDependencies=no and no ordering on multi-user or
boot-complete, so a hung boot is caught too.

Kubernetes: while kubelet or k0s is starting, running or stopping, the
node may carry workloads, so it touches /run/reboot-required and kured
drains and reboots it in its lock, as for an update. A stopped or failed
Kubernetes unit leaves the node NotReady with nothing to drain and no
kured, so it reboots directly. /run/reboot-lock and /etc/reboot-lock hold
it like the nightly reboot.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): do not reboot back into an update that failed its boot tries

After a rollback the failed version is still installed, and
`systemd-sysupdate pending` still reports it: it does not look at the
UKI's tries counter. The kured flag would be set after every update run
and the nightly reboot would fire every night, each time landing on the
previous version again.

bluefin-update-pending wraps `systemd-sysupdate pending` and says no
while a newer installed version's UKI has no tries left
(bluefin-server-<ver>+0-<done>.efi). It gates the kured flag
(20-kured.conf) and the nightly reboot (an ExecCondition in
20-interlock.conf). The next newer release takes the failed version's
slot and UKI (InstancesMax=2) and clears the block.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(update): do not flag a diskless node pinned to a versioned image

When the boot origin names a versioned file, bluefin-server_<ver>.raw from
an explicit rd.systemd.pull= URL or the import.pull credential (or a
versioned bluefin-server-netboot_<ver>.efi boot URL), the node pulls the
same version again after a reboot. Flagging /run/reboot-required there made
kured drain and reboot the node on every check while the server directory
held a newer release.

Flag only when the next boot resolves the version itself: a netboot UKI
served under a fixed name (Booty's bluefin-server-netboot.efi via
StubDeviceURL) or another unversioned name. A pinned node logs
"Bluefin Server <ver> is available from <dir>/ (running <ver>), but this
node is pinned to <file>; not flagging". The behavioural tests cover both
flagging origins and the three pinned ones, and the signature and
network-error cases now run against a flaggable origin so they still prove
something.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* chore(elements): describe the boot deadline and update-pending helpers

os-update-check.bst now installs bluefin-boot-deadline and
bluefin-update-pending next to the diskless update check, and
os-k0s-first-boot.bst the deadline units and the reboot interlock.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* test(dogfood): prove the boot-deadline rollback of a failed unit in QEMU

DOGFOOD_BROKEN=unit breaks <broken-dir> with a unit that fails on that
version only (ConditionOSRelease=IMAGE_VERSION=) instead of corrupting
its slot, and shortens the boot deadline (DOGFOOD_DEADLINE, default 2min).
Each counted boot reaches multi-user.target, misses
boot-complete.target and waits for bluefin-boot-deadline to reboot it.
The run asserts three such boots, the fallback to <next-dir> with the
deadline timer inactive and the boot clean, and then no kured flag and a
skipped systemd-sysupdate-reboot.service for the failed version. Every
disk boot also reports the deadline timer's state. The default slot mode
is unchanged.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): document the boot deadline, the reboot interlock and pinned diskless origins

ddi-installer.md "Updates": boot counting alone only rolls back on the
next boot, so describe bluefin-boot-deadline (when it runs, why a
Kubernetes node is flagged for kured instead of rebooted, the dirty last
try, the drop-in to change the deadline), bluefin-update-pending after a
rollback, the lock files in 20-interlock.conf (credited to #182), and
which diskless origins are pinned. ddi-installer-build.md covers
DOGFOOD_BROKEN=unit and the pinned vs HTTP-boot update check.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): tell Kubernetes sysext operators the rollback reboot goes through kured

While kubelet or k0s runs, bluefin-boot-deadline flags
/run/reboot-required instead of rebooting, so kured also rolls back an
unhealthy update.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: make the rollback and reboot coordination claims precise

The README, readiness audit, roadmap and distro gap analysis said boot
counting rolls back an update that does not boot cleanly; that now holds
because the boot deadline supplies the reboot. Say so, and how long it
takes. Reconcile the "Reboot coordination" rows with #182: the lock files
and the Kubernetes interlock both live in 20-interlock.conf, credited to
#182. The diskless signal is described as skipping pinned nodes.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts

5 participants