diff --git a/docs/skills/architecture-roadmap.md b/docs/skills/architecture-roadmap.md index d7dc5f39..4779b96e 100644 --- a/docs/skills/architecture-roadmap.md +++ b/docs/skills/architecture-roadmap.md @@ -26,7 +26,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md). | 2 | Mount `/usr` read-only and enforce the state model | DDI is currently booted `rw`; sysext-first design assumes immutable `/usr`. | | 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. | | 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. | -| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. | +| 5 | Distributed reboot lock coordination for multi-node non-Kubernetes clusters | Single-node / maintenance window / lock-file reboot coordination is present via `systemd-sysupdate-reboot`; cluster-wide FleetLock/locksmith HTTP protocol is not implemented. | | 6 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. | ## Status notes diff --git a/docs/skills/gap-analysis-distros.md b/docs/skills/gap-analysis-distros.md index 9ef63c08..9c9521e2 100644 --- a/docs/skills/gap-analysis-distros.md +++ b/docs/skills/gap-analysis-distros.md @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from | **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. | | **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata: sysusers credentials cover root, `tmpfiles.extra` can write arbitrary first-boot files such as SSH `authorized_keys`, `network.*` credentials seed networkd, and `firstboot.*` credentials configure locale/timezone/hostname non-interactively. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). | | **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). | -| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). | +| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. | ## 4. Factual Gaps @@ -119,7 +119,6 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Update delivery - **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place. -- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook. ### Customization @@ -127,7 +126,8 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from ### Reboot coordination -- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. +- **Status:** Bluefin Server provides `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` with maintenance window scheduling and lock-based inhibition (`/run/reboot-lock`, `/etc/reboot-lock`) for single-node and non-Kubernetes hosts, alongside the Kured hook for Kubernetes clusters. +- **Gap:** While single-node and lock-file inhibition is supported via systemd primitives, there is no built-in cluster-wide HTTP lock manager client similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`. ## 5. Summary of Biggest Gaps @@ -194,3 +194,5 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma - [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer) - [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf) - [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf) +- [elements/bluefin-server/os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst) +- [files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf) diff --git a/docs/skills/k0s-sysext-ops.md b/docs/skills/k0s-sysext-ops.md index 7b806f80..74b191f1 100644 --- a/docs/skills/k0s-sysext-ops.md +++ b/docs/skills/k0s-sysext-ops.md @@ -4,7 +4,7 @@ description: Operator runbook for the k0s systemd-sysext extension — provision metadata: type: how-to status: stable - last_updated: "2026-09-08" + last_updated: "2026-09-26" context7-sources: - /systemd/systemd --- @@ -46,6 +46,15 @@ k0s automatically applies all `.yaml` files under `/var/lib/k0s/manifests/argocd k0s kubectl get pods -A ``` +## Reboot Coordination + +On Bluefin Server hosts running k0s: + +- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. Because the `ExecCondition` is evaluated at the instant the maintenance timer fires, nodes undergoing planned cluster maintenance or service restarts should set `/run/reboot-lock` to guarantee inhibition across transient unit state transitions. +- **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`. +- **Default appliance install**: The appliance runs k0s by default and ships no Kured in its manifests. With k0s active, `systemd-sysupdate-reboot.service` therefore stands down on every timer run, so a staged update is not applied until an operator reboots the host manually or deploys Kured. +- **Timer enablement**: Flatcar's `/usr` enables `systemd-sysupdate-reboot.timer` through its own `timers.target.wants` symlink, so no preset is needed; the `reboot-coordination.conf` drop-in is what adds the Kubernetes and lock-file gate. + ## Troubleshooting - **Extension not merged**: Check `systemd-sysext status`. Verify the persistent image is `/var/lib/k0s/k0s.raw`; the boot activation unit copies it into `/run/extensions/k0s.raw`. diff --git a/elements/bluefin-server/os-stack.bst b/elements/bluefin-server/os-stack.bst index 5aa65158..4f9c9d19 100644 --- a/elements/bluefin-server/os-stack.bst +++ b/elements/bluefin-server/os-stack.bst @@ -56,8 +56,9 @@ depends: # First-Boot Credentials Provisioning - bluefin-server/os-creds-prov.bst - # Reboot Coordination for Kured + # Reboot Coordination - bluefin-server/os-kured-hook.bst + - bluefin-server/os-sysupdate-reboot.bst # System Management Justfile - bluefin-server/os-justfile.bst diff --git a/elements/bluefin-server/os-sysupdate-reboot.bst b/elements/bluefin-server/os-sysupdate-reboot.bst new file mode 100644 index 00000000..42742819 --- /dev/null +++ b/elements/bluefin-server/os-sysupdate-reboot.bst @@ -0,0 +1,9 @@ +kind: import +description: Configure reboot coordination drop-in for systemd-sysupdate-reboot. + +sources: + - kind: local + path: files/os/systemd/systemd-sysupdate-reboot.service.d + +config: + target: /usr/lib/systemd/system/systemd-sysupdate-reboot.service.d diff --git a/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf new file mode 100644 index 00000000..c54a17ab --- /dev/null +++ b/files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf @@ -0,0 +1,13 @@ +[Unit] +Description=Reboot coordination and lock inhibition for systemd-sysupdate +# Defer reboot if administrative or runtime lock files exist +ConditionPathExists=!/run/reboot-lock +ConditionPathExists=!/etc/reboot-lock + +[Service] +# Gate reboot on Kubernetes liveness: if Kubernetes (k0s or kubelet) is active, +# defer reboot coordination to Kured rather than rebooting out of band. +# Note: This ExecCondition evaluates point-in-time liveness. On Kubernetes hosts +# undergoing service maintenance or restart, creating /run/reboot-lock provides +# deterministic inhibition regardless of temporary unit restart states. +ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service' diff --git a/tests/unit/test_reboot_coordination.py b/tests/unit/test_reboot_coordination.py new file mode 100644 index 00000000..8e6cbdbb --- /dev/null +++ b/tests/unit/test_reboot_coordination.py @@ -0,0 +1,161 @@ +"""Unit tests for systemd-sysupdate reboot coordination. + +Asserts contracts for systemd-sysupdate-reboot drop-in coordination strategies +(immediate, maintenance window, lock-based) and verifies that Kubernetes hosts +gate reboots via real service liveness rather than non-existent sentinels, +without colliding with upstream systemd units. +""" + +import configparser +import os +from pathlib import Path +import subprocess +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] +SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system" +SYSUPDATE_DROPIN_DIR = ( + REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d" +) +REBOOT_DROPIN_DIR = ( + REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate-reboot.service.d" +) +ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server" + +REBOOT_DROPIN = REBOOT_DROPIN_DIR / "reboot-coordination.conf" +REBOOT_ELEMENT = ELEMENTS_DIR / "os-sysupdate-reboot.bst" +OS_STACK = ELEMENTS_DIR / "os-stack.bst" +KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf" + + +def load_ini(path: Path) -> configparser.ConfigParser: + parser = configparser.ConfigParser(strict=False) + parser.optionxform = str + parser.read_string(path.read_text()) + return parser + + +def test_reboot_coordination_files_exist(): + assert REBOOT_DROPIN.is_file(), "reboot-coordination.conf drop-in is missing" + assert REBOOT_ELEMENT.is_file(), "os-sysupdate-reboot.bst element is missing" + + +def test_no_upstream_unit_collision(): + # systemd-sysupdate-reboot.service and .timer ship in Flatcar's /usr + # (/usr/lib/systemd/system/), which also enables the timer through its own + # timers.target.wants symlink. Shipping full unit files in + # files/os/systemd/system would collide with those upstream units. + service_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service" + timer_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer" + assert not service_file.exists(), ( + "systemd-sysupdate-reboot.service must not exist in files/os/systemd/system " + "to avoid colliding with the unit Flatcar ships in /usr" + ) + assert not timer_file.exists(), ( + "systemd-sysupdate-reboot.timer must not exist in files/os/systemd/system " + "to avoid colliding with the unit Flatcar ships in /usr" + ) + + +def test_reboot_dropin_contract(): + parser = load_ini(REBOOT_DROPIN) + assert parser.has_section("Unit") + assert parser.has_section("Service") + + content = REBOOT_DROPIN.read_text() + # Verify lock-based coordination sentinels + assert "ConditionPathExists=!/run/reboot-lock" in content + assert "ConditionPathExists=!/etc/reboot-lock" in content + + # Ensure imaginary sentinels are NOT referenced + assert "kured-active" not in content, ( + "Condition must not reference /run/kured-active which nothing creates" + ) + + # Verify Kubernetes gating + service = parser["Service"] + exec_condition = service.get("ExecCondition", "") + assert "systemctl is-active" in exec_condition + assert "k0scontroller.service" in exec_condition + assert "kubelet.service" in exec_condition + + +def test_exec_condition_kubernetes_interlock_logic(tmp_path): + parser = load_ini(REBOOT_DROPIN) + exec_condition = parser["Service"]["ExecCondition"] + # Extract command passed to sh -c or run command directly + assert exec_condition.startswith("/usr/bin/sh -c ") + cmd_str = exec_condition[len("/usr/bin/sh -c ") :].strip("'\"") + + # Create mock systemctl + mock_systemctl = tmp_path / "systemctl" + mock_env = {**os.environ, "PATH": f"{tmp_path}:{os.environ.get('PATH', '')}"} + + # Case 1: k0scontroller is active -> ExecCondition must fail (exit 1) + mock_systemctl.write_text("""#!/bin/sh +for arg in "$@"; do + if [ "$arg" = "k0scontroller.service" ]; then + exit 0 + fi +done +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode != 0, ( + "ExecCondition should fail (exit non-zero) when k0scontroller is active, " + "preventing uncoordinated reboot of Kubernetes node" + ) + + # Case 2: kubelet is active -> ExecCondition must fail (exit 1) + mock_systemctl.write_text("""#!/bin/sh +for arg in "$@"; do + if [ "$arg" = "kubelet.service" ]; then + exit 0 + fi +done +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode != 0, ( + "ExecCondition should fail (exit non-zero) when kubelet is active" + ) + + # Case 3: No Kubernetes service is active -> ExecCondition must succeed (exit 0) + mock_systemctl.write_text("""#!/bin/sh +exit 3 +""") + mock_systemctl.chmod(0o755) + + proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env) + assert proc.returncode == 0, ( + "ExecCondition should succeed (exit 0) when no Kubernetes service is active, " + "allowing scheduled reboot on single-node / non-Kubernetes hosts" + ) + + +def test_reboot_coordination_packaged_in_elements(): + # Verify os-sysupdate-reboot.bst + element_yaml = yaml.safe_load(REBOOT_ELEMENT.read_text()) + assert element_yaml.get("kind") == "import" + assert element_yaml.get("config", {}).get("target") == ( + "/usr/lib/systemd/system/systemd-sysupdate-reboot.service.d" + ) + sources = element_yaml.get("sources", []) + assert any( + s.get("path") == "files/os/systemd/systemd-sysupdate-reboot.service.d" + for s in sources + ) + + # Verify os-stack.bst includes the element + stack_yaml = yaml.safe_load(OS_STACK.read_text()) + depends = stack_yaml.get("depends", []) + assert "bluefin-server/os-sysupdate-reboot.bst" in depends + assert "bluefin-server/os-kured-hook.bst" in depends + + # Verify kured hook creates /run/reboot-required + assert KURED_HOOK.is_file() + assert "touch /run/reboot-required" in KURED_HOOK.read_text()