Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/skills/architecture-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md).
| 2 | Mount `/usr` read-only and enforce the state model | DDI is currently booted `rw`; sysext-first design assumes immutable `/usr`. |
| 3 | Boot-time selection / automatic rollback of a failed update | No previous OS version is kept once a root update overwrites the slot. |
| 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. |
| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. |
| 5 | Distributed reboot lock coordination for multi-node non-Kubernetes clusters | Single-node / maintenance window / lock-file reboot coordination is present via `systemd-sysupdate-reboot`; cluster-wide FleetLock/locksmith HTTP protocol is not implemented. |
| 6 | Staged rollout behavior for larger fleets | Future after items 1-3 are implemented. |

## Status notes
Expand Down
8 changes: 5 additions & 3 deletions docs/skills/gap-analysis-distros.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
| **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. |
| **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is delivered via `systemd-creds` through the ESP or hypervisor metadata: sysusers credentials cover root, `tmpfiles.extra` can write arbitrary first-boot files such as SSH `authorized_keys`, `network.*` credentials seed networkd, and `firstboot.*` credentials configure locale/timezone/hostname non-interactively. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [tpm2-credential-sealing.md](tpm2-credential-sealing.md), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). |
| **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). |
| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required`. Rolling reboots across Kubernetes nodes rely on Kured reading that file. Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [Kured project](https://github.com/weaveworks/kured). |
| **Reboot coordination** | `systemd-sysupdate.service` has an `ExecStartPost` that touches `/run/reboot-required` for Kured. For non-Kubernetes and single-node hosts, `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` coordinate automated reboots using systemd primitives, supporting maintenance window timers and lock inhibition (`/run/reboot-lock`, `/etc/reboot-lock`). Sources: [os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst), [kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf), [os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst), [reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf), [Kured project](https://github.com/weaveworks/kured), also `systemd-sysupdate(8)`. |

## 4. Factual Gaps

Expand All @@ -119,15 +119,15 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
### Update delivery

- **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place.
- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook.

### Customization

- **No major gap found relative to the design intent.** `systemd-sysext` and Flatcar Bakery compatibility match the intended extension model. The main operational concern is that, because `/usr` is not mounted read-only, a sysext merge vs. runtime writes to `/usr` have different guarantees than on Flatcar or Fedora CoreOS.

### Reboot coordination

- **Gap:** Kured coordinates Kubernetes node reboots but requires Kubernetes to be running. There is no equivalent for single-node or non-Kubernetes Bluefin hosts, and there is no built-in cluster lock manager similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`.
- **Status:** Bluefin Server provides `systemd-sysupdate-reboot.timer` and `systemd-sysupdate-reboot.service` with maintenance window scheduling and lock-based inhibition (`/run/reboot-lock`, `/etc/reboot-lock`) for single-node and non-Kubernetes hosts, alongside the Kured hook for Kubernetes clusters.
- **Gap:** While single-node and lock-file inhibition is supported via systemd primitives, there is no built-in cluster-wide HTTP lock manager client similar to Zincati's FleetLock or Flatcar's `locksmithd`/`etcd-lock`.

## 5. Summary of Biggest Gaps

Expand Down Expand Up @@ -194,3 +194,5 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma
- [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer)
- [files/os/sysusers.d/10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf)
- [files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf](../../files/os/systemd/systemd-sysupdate.service.d/kured-hook.conf)
- [elements/bluefin-server/os-sysupdate-reboot.bst](../../elements/bluefin-server/os-sysupdate-reboot.bst)
- [files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf](../../files/os/systemd/systemd-sysupdate-reboot.service.d/reboot-coordination.conf)
11 changes: 10 additions & 1 deletion docs/skills/k0s-sysext-ops.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Operator runbook for the k0s systemd-sysext extension — provision
metadata:
type: how-to
status: stable
last_updated: "2026-09-08"
last_updated: "2026-09-26"
context7-sources:
- /systemd/systemd
---
Expand Down Expand Up @@ -46,6 +46,15 @@ k0s automatically applies all `.yaml` files under `/var/lib/k0s/manifests/argocd
k0s kubectl get pods -A
```

## Reboot Coordination

On Bluefin Server hosts running k0s:

- **Kubernetes clusters**: `systemd-sysupdate.service` touches `/run/reboot-required` after staging updates. When k0s (`k0scontroller.service`) is active, `systemd-sysupdate-reboot.service` detects the active service via `ExecCondition` and skips uncoordinated local reboots, allowing Kured to cordon, drain, and reboot nodes safely. Note that k0s hosts must deploy Kured (or an equivalent cluster reboot coordinator); if Kured is not deployed on a single-node k0s host, uncoordinated local reboots remain inhibited while k0s services run, requiring manual reboot or manual coordination. Because the `ExecCondition` is evaluated at the instant the maintenance timer fires, nodes undergoing planned cluster maintenance or service restarts should set `/run/reboot-lock` to guarantee inhibition across transient unit state transitions.
- **Single-node / non-Kubernetes hosts**: When k0s is not running, `systemd-sysupdate-reboot.timer` schedules automatic reboots during the maintenance window (04:10 with randomized delay). Reboots can be temporarily inhibited by creating `/run/reboot-lock` or persistently inhibited with `/etc/reboot-lock`.
- **Default appliance install**: The appliance runs k0s by default and ships no Kured in its manifests. With k0s active, `systemd-sysupdate-reboot.service` therefore stands down on every timer run, so a staged update is not applied until an operator reboots the host manually or deploys Kured.
- **Timer enablement**: Flatcar's `/usr` enables `systemd-sysupdate-reboot.timer` through its own `timers.target.wants` symlink, so no preset is needed; the `reboot-coordination.conf` drop-in is what adds the Kubernetes and lock-file gate.

## Troubleshooting

- **Extension not merged**: Check `systemd-sysext status`. Verify the persistent image is `/var/lib/k0s/k0s.raw`; the boot activation unit copies it into `/run/extensions/k0s.raw`.
Expand Down
3 changes: 2 additions & 1 deletion elements/bluefin-server/os-stack.bst
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,9 @@ depends:
# First-Boot Credentials Provisioning
- bluefin-server/os-creds-prov.bst

# Reboot Coordination for Kured
# Reboot Coordination
- bluefin-server/os-kured-hook.bst
- bluefin-server/os-sysupdate-reboot.bst

# System Management Justfile
- bluefin-server/os-justfile.bst
Expand Down
9 changes: 9 additions & 0 deletions elements/bluefin-server/os-sysupdate-reboot.bst
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
kind: import
description: Configure reboot coordination drop-in for systemd-sysupdate-reboot.

sources:
- kind: local
path: files/os/systemd/systemd-sysupdate-reboot.service.d

config:
target: /usr/lib/systemd/system/systemd-sysupdate-reboot.service.d
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[Unit]
Description=Reboot coordination and lock inhibition for systemd-sysupdate
# Defer reboot if administrative or runtime lock files exist
ConditionPathExists=!/run/reboot-lock
ConditionPathExists=!/etc/reboot-lock

[Service]
# Gate reboot on Kubernetes liveness: if Kubernetes (k0s or kubelet) is active,
# defer reboot coordination to Kured rather than rebooting out of band.
# Note: This ExecCondition evaluates point-in-time liveness. On Kubernetes hosts
# undergoing service maintenance or restart, creating /run/reboot-lock provides
# deterministic inhibition regardless of temporary unit restart states.
ExecCondition=/usr/bin/sh -c '! systemctl is-active --quiet kubelet.service k0scontroller.service k0sworker.service'
161 changes: 161 additions & 0 deletions tests/unit/test_reboot_coordination.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
"""Unit tests for systemd-sysupdate reboot coordination.

Asserts contracts for systemd-sysupdate-reboot drop-in coordination strategies
(immediate, maintenance window, lock-based) and verifies that Kubernetes hosts
gate reboots via real service liveness rather than non-existent sentinels,
without colliding with upstream systemd units.
"""

import configparser
import os
from pathlib import Path
import subprocess
import yaml

REPO_ROOT = Path(__file__).resolve().parents[2]
SYSTEMD_SYSTEM_DIR = REPO_ROOT / "files" / "os" / "systemd" / "system"
SYSUPDATE_DROPIN_DIR = (
REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate.service.d"
)
REBOOT_DROPIN_DIR = (
REPO_ROOT / "files" / "os" / "systemd" / "systemd-sysupdate-reboot.service.d"
)
ELEMENTS_DIR = REPO_ROOT / "elements" / "bluefin-server"

REBOOT_DROPIN = REBOOT_DROPIN_DIR / "reboot-coordination.conf"
REBOOT_ELEMENT = ELEMENTS_DIR / "os-sysupdate-reboot.bst"
OS_STACK = ELEMENTS_DIR / "os-stack.bst"
KURED_HOOK = SYSUPDATE_DROPIN_DIR / "kured-hook.conf"


def load_ini(path: Path) -> configparser.ConfigParser:
parser = configparser.ConfigParser(strict=False)
parser.optionxform = str
parser.read_string(path.read_text())
return parser


def test_reboot_coordination_files_exist():
assert REBOOT_DROPIN.is_file(), "reboot-coordination.conf drop-in is missing"
assert REBOOT_ELEMENT.is_file(), "os-sysupdate-reboot.bst element is missing"


def test_no_upstream_unit_collision():
# systemd-sysupdate-reboot.service and .timer ship in Flatcar's /usr
# (/usr/lib/systemd/system/), which also enables the timer through its own
# timers.target.wants symlink. Shipping full unit files in
# files/os/systemd/system would collide with those upstream units.
service_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.service"
timer_file = SYSTEMD_SYSTEM_DIR / "systemd-sysupdate-reboot.timer"
assert not service_file.exists(), (
"systemd-sysupdate-reboot.service must not exist in files/os/systemd/system "
"to avoid colliding with the unit Flatcar ships in /usr"
)
assert not timer_file.exists(), (
"systemd-sysupdate-reboot.timer must not exist in files/os/systemd/system "
"to avoid colliding with the unit Flatcar ships in /usr"
)


def test_reboot_dropin_contract():
parser = load_ini(REBOOT_DROPIN)
assert parser.has_section("Unit")
assert parser.has_section("Service")

content = REBOOT_DROPIN.read_text()
# Verify lock-based coordination sentinels
assert "ConditionPathExists=!/run/reboot-lock" in content
assert "ConditionPathExists=!/etc/reboot-lock" in content

# Ensure imaginary sentinels are NOT referenced
assert "kured-active" not in content, (
"Condition must not reference /run/kured-active which nothing creates"
)

# Verify Kubernetes gating
service = parser["Service"]
exec_condition = service.get("ExecCondition", "")
assert "systemctl is-active" in exec_condition
assert "k0scontroller.service" in exec_condition
assert "kubelet.service" in exec_condition


def test_exec_condition_kubernetes_interlock_logic(tmp_path):
parser = load_ini(REBOOT_DROPIN)
exec_condition = parser["Service"]["ExecCondition"]
# Extract command passed to sh -c or run command directly
assert exec_condition.startswith("/usr/bin/sh -c ")
cmd_str = exec_condition[len("/usr/bin/sh -c ") :].strip("'\"")

# Create mock systemctl
mock_systemctl = tmp_path / "systemctl"
mock_env = {**os.environ, "PATH": f"{tmp_path}:{os.environ.get('PATH', '')}"}

# Case 1: k0scontroller is active -> ExecCondition must fail (exit 1)
mock_systemctl.write_text("""#!/bin/sh
for arg in "$@"; do
if [ "$arg" = "k0scontroller.service" ]; then
exit 0
fi
done
exit 3
""")
mock_systemctl.chmod(0o755)

proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env)
assert proc.returncode != 0, (
"ExecCondition should fail (exit non-zero) when k0scontroller is active, "
"preventing uncoordinated reboot of Kubernetes node"
)

# Case 2: kubelet is active -> ExecCondition must fail (exit 1)
mock_systemctl.write_text("""#!/bin/sh
for arg in "$@"; do
if [ "$arg" = "kubelet.service" ]; then
exit 0
fi
done
exit 3
""")
mock_systemctl.chmod(0o755)

proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env)
assert proc.returncode != 0, (
"ExecCondition should fail (exit non-zero) when kubelet is active"
)

# Case 3: No Kubernetes service is active -> ExecCondition must succeed (exit 0)
mock_systemctl.write_text("""#!/bin/sh
exit 3
""")
mock_systemctl.chmod(0o755)

proc = subprocess.run(["sh", "-c", cmd_str], env=mock_env)
assert proc.returncode == 0, (
"ExecCondition should succeed (exit 0) when no Kubernetes service is active, "
"allowing scheduled reboot on single-node / non-Kubernetes hosts"
)


def test_reboot_coordination_packaged_in_elements():
# Verify os-sysupdate-reboot.bst
element_yaml = yaml.safe_load(REBOOT_ELEMENT.read_text())
assert element_yaml.get("kind") == "import"
assert element_yaml.get("config", {}).get("target") == (
"/usr/lib/systemd/system/systemd-sysupdate-reboot.service.d"
)
sources = element_yaml.get("sources", [])
assert any(
s.get("path") == "files/os/systemd/systemd-sysupdate-reboot.service.d"
for s in sources
)

# Verify os-stack.bst includes the element
stack_yaml = yaml.safe_load(OS_STACK.read_text())
depends = stack_yaml.get("depends", [])
assert "bluefin-server/os-sysupdate-reboot.bst" in depends
assert "bluefin-server/os-kured-hook.bst" in depends

# Verify kured hook creates /run/reboot-required
assert KURED_HOOK.is_file()
assert "touch /run/reboot-required" in KURED_HOOK.read_text()