From 81428b5011d76ba6fe8c191c08189f30cf30ee1a Mon Sep 17 00:00:00 2001 From: mrbobbytables Date: Fri, 18 Sep 2026 18:15:39 +0000 Subject: [PATCH 1/5] feat(sysupdate): reboot coordination for non-Kubernetes and single-node hosts Configure systemd-sysupdate-reboot coordination for non-Kubernetes and single-node hosts using upstream systemd units with a drop-in configuration and systemd preset. - Avoid colliding with upstream systemd units by configuring upstream's systemd-sysupdate-reboot.service via a drop-in under files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf instead of overriding the unit files. - Enable systemd-sysupdate-reboot.timer by default via files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset. - Support lock-based reboot inhibition via ConditionPathExists on /run/reboot-lock and /etc/reboot-lock. - Gate reboots via ExecCondition checking active Kubernetes services (k0scontroller.service, kubelet.service, k0sworker.service), ensuring Kubernetes clusters preserve Kured's cordon and drain workflow. - Package drop-in through elements/bluefin-server/os-sysupdate-reboot.bst and wire into os-stack.bst. - Add unit test coverage in tests/unit/test_reboot_coordination.py. - Update docs in architecture-roadmap.md, gap-analysis-distros.md, and k0s-sysext-ops.md. Closes #139 Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables --- docs/skills/architecture-roadmap.md | 2 +- docs/skills/gap-analysis-distros.md | 9 +- docs/skills/k0s-sysext-ops.md | 9 +- elements/bluefin-server/os-stack.bst | 3 +- .../bluefin-server/os-sysupdate-reboot.bst | 9 + .../zz-enable-sysupdate-reboot.preset | 2 + .../reboot-coordination.conf | 10 ++ tests/unit/test_reboot_coordination.py | 168 ++++++++++++++++++ 8 files changed, 206 insertions(+), 6 deletions(-) create mode 100644 elements/bluefin-server/os-sysupdate-reboot.bst create mode 100644 files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset create mode 100644 files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf create mode 100644 tests/unit/test_reboot_coordination.py diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index d7dc5f39..4779b96e 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -26,7 +26,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md). | 2 | Mount `/usr` read-only and enforce the state model | DDI is currently booted `rw`; sysext-first design assumes immutable `/usr`. | | 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. | | 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. | -| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. | +| 5 | Distributed reboot lock coordination for multi-node non-Kubernetes clusters | Single-node / maintenance window / lock-file reboot coordination is present via `systemd-sysupdate-reboot`; cluster-wide FleetLock/locksmith HTTP protocol is not implemented. | | 6 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. | ## Status notes diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 9ef63c08..b31fae7e 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. | | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata: sysusers credentials cover root, `tmpfiles.extra` can write arbitrary first-boot files such as SSH `authorized_keys`, `network.*` credentials seed networkd, and `firstboot.*` credentials configure locale/timezone/hostname non-interactively. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). | | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | ## 4. Factual Gaps @@ -119,7 +119,6 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Update delivery - **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place. -- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook. ### Customization @@ -127,7 +126,8 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Reboot coordination -- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. +- **Status:** Bluefin Server provides `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` with maintenance window scheduling and lock-based inhibition (`/run/reboot-lock`, `/etc/reboot-lock`) for single-node and non-Kubernetes hosts, alongside the Kured hook for Kubernetes clusters. +- **Gap:** While single-node and lock-file inhibition is supported via systemd primitives, there is no built-in cluster-wide HTTP lock manager client similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. ## 5. Summary of Biggest Gaps @@ -194,3 +194,6 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) +- [elements/bluefin-server/os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst) +- [files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) +- [files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 7b806f80..97adaf7b 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s systemd-sysext extension — provision metadata: type: how-to status: stable - last_updated: "2026-09-08" + last_updated: "2026-09-18" context7-sources: - /systemd/systemd --- @@ -46,6 +46,13 @@ k0s automatically applies all `.yaml` files under `/var/lib/k0s/manifests/argocd k0s kubectl get pods -A ``` +## Reboot Coordination + +On Bluefin Server hosts running k0s: + +- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. +- **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`. + ## Troubleshooting - **Extension not merged**: Check `systemd-sysext status`. Verify the persistent image is `/var/lib/k0s/k0s.raw`; the boot activation unit copies it into `/run/extensions/k0s.raw`. diff --git a/elements/bluefin-server/os-stack.bst b/elements/bluefin-server/os-stack.bst index 5aa65158..4f9c9d19 100644 --- a/elements/bluefin-server/os-stack.bst +++ b/elements/bluefin-server/os-stack.bst @@ -56,8 +56,9 @@ depends: # First-Boot Credentials Provisioning - bluefin-server/os-creds-prov.bst - # Reboot Coordination for Kured + # Reboot Coordination - bluefin-server/os-kured-hook.bst + - bluefin-server/os-sysupdate-reboot.bst # System Management Justfile - bluefin-server/os-justfile.bst diff --git a/elements/bluefin-server/os-sysupdate-reboot.bst b/elements/bluefin-server/os-sysupdate-reboot.bst new file mode 100644 index 00000000..42742819 --- /dev/null +++ b/elements/bluefin-server/os-sysupdate-reboot.bst @@ -0,0 +1,9 @@ +kind: import +description: Configure reboot coordination drop-in for systemd-sysupdate-reboot. + +sources: + - kind: local + path: files/os/systemd/systemd-sysupdate-reboot.service.d + +config: + target: /usr/lib/systemd/system/systemd-sysupdate-reboot.service.d diff --git a/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset b/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset new file mode 100644 index 00000000..051bc453 --- /dev/null +++ b/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset @@ -0,0 +1,2 @@ +# Enable automatic reboot timer for system updates on non-Kubernetes and single-node hosts +enable systemd-sysupdate-reboot.timer diff --git a/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf new file mode 100644 index 00000000..7dff93c7 --- /dev/null +++ b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf @@ -0,0 +1,10 @@ +[Unit] +Description=Reboot coordination and lock inhibition for systemd-sysupdate +# Defer reboot if administrative or runtime lock files exist +ConditionPathExists=!/run/reboot-lock +ConditionPathExists=!/etc/reboot-lock + +[Service] +# Gate reboot on Kubernetes liveness: if Kubernetes (k0s or kubelet) is active, +# defer reboot coordination to Kured rather than rebooting out of band. +ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service' diff --git a/tests/unit/test_reboot_coordination.py b/tests/unit/test_reboot_coordination.py new file mode 100644 index 00000000..e3eb7a3b --- /dev/null +++ b/tests/unit/test_reboot_coordination.py @@ -0,0 +1,168 @@ +"""Unit tests for systemd-sysupdate reboot coordination. + +Asserts contracts for systemd-sysupdate-reboot drop-in coordination strategies +(immediate, maintenance window, lock-based) and verifies that Kubernetes hosts +gate reboots via real service liveness rather than non-existent sentinels, +without colliding with upstream systemd units. +""" + +import configparser +import os +from pathlib import Path +import subprocess +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] +SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system" +SYSTEMD_PRESET_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system-preset" +SYSUPDATE_DROPIN_DIR = ( + REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d" +) +REBOOT_DROPIN_DIR = ( + REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate-reboot.service.d" +) +ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server" + +REBOOT_DROPIN = REBOOT_DROPIN_DIR / "reboot-coordination.conf" +REBOOT_PRESET = SYSTEMD_PRESET_DIR / "zz-enable-sysupdate-reboot.preset" +REBOOT_ELEMENT = ELEMENTS_DIR / "os-sysupdate-reboot.bst" +OS_STACK = ELEMENTS_DIR / "os-stack.bst" +KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf" + + +def load_ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False) + parser.optionxform = str + parser.read_string(path.read_text()) + return parser + + +def test_reboot_coordination_files_exist(): + assert REBOOT_DROPIN.is_file(), "reboot-coordination.conf drop-in is missing" + assert REBOOT_PRESET.is_file(), "zz-enable-sysupdate-reboot.preset is missing" + assert REBOOT_ELEMENT.is_file(), "os-sysupdate-reboot.bst element is missing" + + +def test_no_upstream_unit_collision(): + # systemd-sysupdate-reboot.service and .timer are shipped by upstream systemd + # in freedesktop-sdk.bst:components/systemd.bst (/usr/lib/systemd/system/). + # Shipping full unit files in files/os/systemd/system collides with upstream. + service_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service" + timer_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer" + assert not service_file.exists(), ( + "systemd-sysupdate-reboot.service must not exist in files/os/systemd/system " + "to avoid colliding with upstream systemd.bst units" + ) + assert not timer_file.exists(), ( + "systemd-sysupdate-reboot.timer must not exist in files/os/systemd/system " + "to avoid colliding with upstream systemd.bst units" + ) + + +def test_reboot_dropin_contract(): + parser = load_ini(REBOOT_DROPIN) + assert parser.has_section("Unit") + assert parser.has_section("Service") + + content = REBOOT_DROPIN.read_text() + # Verify lock-based coordination sentinels + assert "ConditionPathExists=!/run/reboot-lock" in content + assert "ConditionPathExists=!/etc/reboot-lock" in content + + # Ensure imaginary sentinels are NOT referenced + assert "kured-active" not in content, ( + "Condition must not reference /run/kured-active which nothing creates" + ) + + # Verify Kubernetes gating + service = parser["Service"] + exec_condition = service.get("ExecCondition", "") + assert "systemctl is-active" in exec_condition + assert "k0scontroller.service" in exec_condition + assert "kubelet.service" in exec_condition + + +def test_reboot_preset_enables_timer(): + content = REBOOT_PRESET.read_text() + assert "enable systemd-sysupdate-reboot.timer" in content + + +def test_exec_condition_kubernetes_interlock_logic(tmp_path): + parser = load_ini(REBOOT_DROPIN) + exec_condition = parser["Service"]["ExecCondition"] + # Extract command passed to sh -c or run command directly + assert exec_condition.startswith("/usr/bin/sh -c ") + cmd_str = exec_condition[len("/usr/bin/sh -c ") :].strip("'\"") + + # Create mock systemctl + mock_systemctl = tmp_path / "systemctl" + mock_env = {**os.environ, "PATH": f"{tmp_path}:{os.environ.get('PATH', '')}"} + + # Case 1: k0scontroller is active -> ExecCondition must fail (exit 1) + mock_systemctl.write_text("""#!/bin/sh +for arg in "$@"; do + if [ "$arg" = "k0scontroller.service" ]; then + exit 0 + fi +done +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode != 0, ( + "ExecCondition should fail (exit non-zero) when k0scontroller is active, " + "preventing uncoordinated reboot of Kubernetes node" + ) + + # Case 2: kubelet is active -> ExecCondition must fail (exit 1) + mock_systemctl.write_text("""#!/bin/sh +for arg in "$@"; do + if [ "$arg" = "kubelet.service" ]; then + exit 0 + fi +done +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode != 0, ( + "ExecCondition should fail (exit non-zero) when kubelet is active" + ) + + # Case 3: No Kubernetes service is active -> ExecCondition must succeed (exit 0) + mock_systemctl.write_text("""#!/bin/sh +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode == 0, ( + "ExecCondition should succeed (exit 0) when no Kubernetes service is active, " + "allowing scheduled reboot on single-node / non-Kubernetes hosts" + ) + + +def test_reboot_coordination_packaged_in_elements(): + # Verify os-sysupdate-reboot.bst + element_yaml = yaml.safe_load(REBOOT_ELEMENT.read_text()) + assert element_yaml.get("kind") == "import" + assert element_yaml.get("config", {}).get("target") == ( + "/usr/lib/systemd/system/systemd-sysupdate-reboot.service.d" + ) + sources = element_yaml.get("sources", []) + assert any( + s.get("path") == "files/os/systemd/systemd-sysupdate-reboot.service.d" + for s in sources + ) + + # Verify os-stack.bst includes the element + stack_yaml = yaml.safe_load(OS_STACK.read_text()) + depends = stack_yaml.get("depends", []) + assert "bluefin-server/os-sysupdate-reboot.bst" in depends + assert "bluefin-server/os-kured-hook.bst" in depends + + # Verify kured hook creates /run/reboot-required + assert KURED_HOOK.is_file() + assert "touch /run/reboot-required" in KURED_HOOK.read_text() From 4c2f48519cc78f361a670419595c08e16b84ac97 Mon Sep 17 00:00:00 2001 From: mrbobbytables Date: Sat, 19 Sep 2026 23:00:20 +0000 Subject: [PATCH 2/5] docs(k0s-sysext-ops): document reboot behavior on k0s hosts without Kured Clarify that k0s hosts must deploy Kured or an equivalent coordinator, and that uncoordinated local reboots remain inhibited on single-node k0s hosts when Kured is absent, per review feedback from @kubestellar-hive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables --- docs/skills/k0s-sysext-ops.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 97adaf7b..8274d14a 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -50,7 +50,7 @@ k0s kubectl get pods -A On Bluefin Server hosts running k0s: -- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. +- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. - **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`. ## Troubleshooting From c552dbed9acfb073f2f1d5d3cbbac41ee40f2912 Mon Sep 17 00:00:00 2001 From: mrbobbytables Date: Sun, 20 Sep 2026 18:56:10 +0000 Subject: [PATCH 3/5] docs(sysupdate): document point-in-time ExecCondition semantics and lock inhibition Document in reboot-coordination.conf and k0s-sysext-ops.md that the ExecCondition evaluates point-in-time liveness at timer fire, and recommend using /run/reboot-lock during planned cluster maintenance or service restarts to deterministically inhibit reboots across transient unit state transitions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables --- docs/skills/k0s-sysext-ops.md | 2 +- .../reboot-coordination.conf | 3 +++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 8274d14a..65a9558d 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -50,7 +50,7 @@ k0s kubectl get pods -A On Bluefin Server hosts running k0s: -- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. +- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. Because the `ExecCondition` is evaluated at the instant the maintenance timer fires, nodes undergoing planned cluster maintenance or service restarts should set `/run/reboot-lock` to guarantee inhibition across transient unit state transitions. - **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`. ## Troubleshooting diff --git a/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf index 7dff93c7..c54a17ab 100644 --- a/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf +++ b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf @@ -7,4 +7,7 @@ ConditionPathExists=!/etc/reboot-lock [Service] # Gate reboot on Kubernetes liveness: if Kubernetes (k0s or kubelet) is active, # defer reboot coordination to Kured rather than rebooting out of band. +# Note: This ExecCondition evaluates point-in-time liveness. On Kubernetes hosts +# undergoing service maintenance or restart, creating /run/reboot-lock provides +# deterministic inhibition regardless of temporary unit restart states. ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service' From 1910bacfbe14ec11e33ed022d7f5a1e4b075b5f5 Mon Sep 17 00:00:00 2001 From: mrbobbytables Date: Sun, 20 Sep 2026 20:54:21 +0000 Subject: [PATCH 4/5] fix(sysupdate): order preset before freedesktop-sdk 90-sysupdate.preset Rename zz-enable-sysupdate-reboot.preset to 80-enable-sysupdate-reboot.preset so that under systemd.preset lexicographic ordering it evaluates before freedesktop-sdk's 90-sysupdate.preset (which disables systemd-sysupdate-reboot.timer), ensuring the timer is enabled on first boot. Add unit test assertion for preset ordering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables --- docs/skills/gap-analysis-distros.md | 4 ++-- ...reboot.preset => 80-enable-sysupdate-reboot.preset} | 0 tests/unit/test_reboot_coordination.py | 10 ++++++++-- 3 files changed, 10 insertions(+), 4 deletions(-) rename files/os/systemd/system-preset/{zz-enable-sysupdate-reboot.preset => 80-enable-sysupdate-reboot.preset} (100%) diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index b31fae7e..bfaf4a9a 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. | | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata: sysusers credentials cover root, `tmpfiles.extra` can write arbitrary first-boot files such as SSH `authorized_keys`, `network.*` credentials seed networkd, and `firstboot.*` credentials configure locale/timezone/hostname non-interactively. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). | | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [80-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | ## 4. Factual Gaps @@ -196,4 +196,4 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) - [elements/bluefin-server/os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst) - [files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) -- [files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset) +- [files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset) diff --git a/files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset b/files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset similarity index 100% rename from files/os/systemd/system-preset/zz-enable-sysupdate-reboot.preset rename to files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset diff --git a/tests/unit/test_reboot_coordination.py b/tests/unit/test_reboot_coordination.py index e3eb7a3b..8ba55943 100644 --- a/tests/unit/test_reboot_coordination.py +++ b/tests/unit/test_reboot_coordination.py @@ -24,7 +24,7 @@ ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server" REBOOT_DROPIN = REBOOT_DROPIN_DIR / "reboot-coordination.conf" -REBOOT_PRESET = SYSTEMD_PRESET_DIR / "zz-enable-sysupdate-reboot.preset" +REBOOT_PRESET = SYSTEMD_PRESET_DIR / "80-enable-sysupdate-reboot.preset" REBOOT_ELEMENT = ELEMENTS_DIR / "os-sysupdate-reboot.bst" OS_STACK = ELEMENTS_DIR / "os-stack.bst" KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf" @@ -39,7 +39,7 @@ def load_ini(path: Path) -> configparser.ConfigParser: def test_reboot_coordination_files_exist(): assert REBOOT_DROPIN.is_file(), "reboot-coordination.conf drop-in is missing" - assert REBOOT_PRESET.is_file(), "zz-enable-sysupdate-reboot.preset is missing" + assert REBOOT_PRESET.is_file(), "80-enable-sysupdate-reboot.preset is missing" assert REBOOT_ELEMENT.is_file(), "os-sysupdate-reboot.bst element is missing" @@ -85,6 +85,12 @@ def test_reboot_dropin_contract(): def test_reboot_preset_enables_timer(): content = REBOOT_PRESET.read_text() assert "enable systemd-sysupdate-reboot.timer" in content + # systemd preset files are evaluated in lexicographic filename order across + # directories, earliest match wins. freedesktop-sdk ships 90-sysupdate.preset + # which disables the timer; our preset must sort before 90-sysupdate.preset. + assert REBOOT_PRESET.name < "90-sysupdate.preset", ( + f"{REBOOT_PRESET.name} must sort before 90-sysupdate.preset to override FSDK's disable" + ) def test_exec_condition_kubernetes_interlock_logic(tmp_path): From fa57e7ba0166218fbd5d8e27c0a7b9d679a7ffdc Mon Sep 17 00:00:00 2001 From: castrojo Date: Sat, 26 Sep 2026 20:31:34 -0400 Subject: [PATCH 5/5] fix(sysupdate): drop the redundant reboot preset and document the default-install gap Flatcar's /usr (#132) already enables systemd-sysupdate-reboot.timer via timers.target.wants and ships no 90-sysupdate.preset, so the 80- preset no longer does anything. Remove it, its test, and the FSDK-era comments. Document that the default k0s appliance ships no Kured, so with the drop-in's Kubernetes gate a staged update waits for a manual reboot. Assisted-by: Claude via omp Signed-off-by: castrojo --- docs/skills/gap-analysis-distros.md | 3 +-- docs/skills/k0s-sysext-ops.md | 4 ++- .../80-enable-sysupdate-reboot.preset | 2 -- tests/unit/test_reboot_coordination.py | 25 +++++-------------- 4 files changed, 10 insertions(+), 24 deletions(-) delete mode 100644 files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index bfaf4a9a..9c9521e2 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. | | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata: sysusers credentials cover root, `tmpfiles.extra` can write arbitrary first-boot files such as SSH `authorized_keys`, `network.*` credentials seed networkd, and `firstboot.*` credentials configure locale/timezone/hostname non-interactively. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). | | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [80-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | ## 4. Factual Gaps @@ -196,4 +196,3 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) - [elements/bluefin-server/os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst) - [files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) -- [files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset](../../files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 65a9558d..74b191f1 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s systemd-sysext extension — provision metadata: type: how-to status: stable - last_updated: "2026-09-18" + last_updated: "2026-09-26" context7-sources: - /systemd/systemd --- @@ -52,6 +52,8 @@ On Bluefin Server hosts running k0s: - **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. Because the `ExecCondition` is evaluated at the instant the maintenance timer fires, nodes undergoing planned cluster maintenance or service restarts should set `/run/reboot-lock` to guarantee inhibition across transient unit state transitions. - **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`. +- **Default appliance install**: The appliance runs k0s by default and ships no Kured in its manifests. With k0s active, `systemd-sysupdate-reboot.service` therefore stands down on every timer run, so a staged update is not applied until an operator reboots the host manually or deploys Kured. +- **Timer enablement**: Flatcar's `/usr` enables `systemd-sysupdate-reboot.timer` through its own `timers.target.wants` symlink, so no preset is needed; the `reboot-coordination.conf` drop-in is what adds the Kubernetes and lock-file gate. ## Troubleshooting diff --git a/files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset b/files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset deleted file mode 100644 index 051bc453..00000000 --- a/files/os/systemd/system-preset/80-enable-sysupdate-reboot.preset +++ /dev/null @@ -1,2 +0,0 @@ -# Enable automatic reboot timer for system updates on non-Kubernetes and single-node hosts -enable systemd-sysupdate-reboot.timer diff --git a/tests/unit/test_reboot_coordination.py b/tests/unit/test_reboot_coordination.py index 8ba55943..8e6cbdbb 100644 --- a/tests/unit/test_reboot_coordination.py +++ b/tests/unit/test_reboot_coordination.py @@ -14,7 +14,6 @@ REPO_ROOT = Path(__file__).resolve().parents[2] SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system" -SYSTEMD_PRESET_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system-preset" SYSUPDATE_DROPIN_DIR = ( REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d" ) @@ -24,7 +23,6 @@ ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server" REBOOT_DROPIN = REBOOT_DROPIN_DIR / "reboot-coordination.conf" -REBOOT_PRESET = SYSTEMD_PRESET_DIR / "80-enable-sysupdate-reboot.preset" REBOOT_ELEMENT = ELEMENTS_DIR / "os-sysupdate-reboot.bst" OS_STACK = ELEMENTS_DIR / "os-stack.bst" KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf" @@ -39,23 +37,23 @@ def load_ini(path: Path) -> configparser.ConfigParser: def test_reboot_coordination_files_exist(): assert REBOOT_DROPIN.is_file(), "reboot-coordination.conf drop-in is missing" - assert REBOOT_PRESET.is_file(), "80-enable-sysupdate-reboot.preset is missing" assert REBOOT_ELEMENT.is_file(), "os-sysupdate-reboot.bst element is missing" def test_no_upstream_unit_collision(): - # systemd-sysupdate-reboot.service and .timer are shipped by upstream systemd - # in freedesktop-sdk.bst:components/systemd.bst (/usr/lib/systemd/system/). - # Shipping full unit files in files/os/systemd/system collides with upstream. + # systemd-sysupdate-reboot.service and .timer ship in Flatcar's /usr + # (/usr/lib/systemd/system/), which also enables the timer through its own + # timers.target.wants symlink. Shipping full unit files in + # files/os/systemd/system would collide with those upstream units. service_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service" timer_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer" assert not service_file.exists(), ( "systemd-sysupdate-reboot.service must not exist in files/os/systemd/system " - "to avoid colliding with upstream systemd.bst units" + "to avoid colliding with the unit Flatcar ships in /usr" ) assert not timer_file.exists(), ( "systemd-sysupdate-reboot.timer must not exist in files/os/systemd/system " - "to avoid colliding with upstream systemd.bst units" + "to avoid colliding with the unit Flatcar ships in /usr" ) @@ -82,17 +80,6 @@ def test_reboot_dropin_contract(): assert "kubelet.service" in exec_condition -def test_reboot_preset_enables_timer(): - content = REBOOT_PRESET.read_text() - assert "enable systemd-sysupdate-reboot.timer" in content - # systemd preset files are evaluated in lexicographic filename order across - # directories, earliest match wins. freedesktop-sdk ships 90-sysupdate.preset - # which disables the timer; our preset must sort before 90-sysupdate.preset. - assert REBOOT_PRESET.name < "90-sysupdate.preset", ( - f"{REBOOT_PRESET.name} must sort before 90-sysupdate.preset to override FSDK's disable" - ) - - def test_exec_condition_kubernetes_interlock_logic(tmp_path): parser = load_ini(REBOOT_DROPIN) exec_condition = parser["Service"]["ExecCondition"]