Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6a9c914
refactor(initrd): share the network boot origin lookup as bluefin-boo…
jeefy Sep 29, 2026
c63c7fd
feat(update): update installed nodes automatically, with a Kubernetes…
jeefy Sep 29, 2026
b910aec
feat(update): bless a boot-counted UKI only when no unit failed
jeefy Sep 29, 2026
104f87b
feat(update): tell diskless nodes when their boot server has a newer …
jeefy Sep 29, 2026
615054a
test(dogfood): prove automatic updates, the interlock and the health …
jeefy Sep 29, 2026
a06015d
docs(skills): document automatic updates, the kured interlock and the…
jeefy Sep 29, 2026
827e441
docs(skills): point Kubernetes sysext operators at kured for OS reboots
jeefy Sep 29, 2026
930ab2e
docs: mark reboot coordination done; a fleet-wide reboot lock remains
jeefy Sep 29, 2026
6d2d3a6
feat(update): hold the nightly reboot with operator lock files
jeefy Sep 29, 2026
901e6b3
fix(update): reboot a boot-counted boot that misses boot-complete.target
jeefy Sep 29, 2026
276daeb
fix(update): do not reboot back into an update that failed its boot t…
jeefy Sep 29, 2026
ccfd0d5
fix(update): do not flag a diskless node pinned to a versioned image
jeefy Sep 29, 2026
bee0d96
chore(elements): describe the boot deadline and update-pending helpers
jeefy Sep 29, 2026
d78b176
test(dogfood): prove the boot-deadline rollback of a failed unit in QEMU
jeefy Sep 29, 2026
3927747
docs(skills): document the boot deadline, the reboot interlock and pi…
jeefy Sep 29, 2026
d9fe1f3
docs(skills): tell Kubernetes sysext operators the rollback reboot go…
jeefy Sep 29, 2026
2e50d00
docs: make the rollback and reboot coordination claims precise
jeefy Sep 29, 2026
f666799
Merge upstream main into feat/automatic-updates
jeefy Sep 29, 2026
ebbafb5
chore: merge upstream main into feat/automatic-updates
jeefy Sep 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ Bluefin Server is currently in **Alpha**:

## What it is

- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_<ver>.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image.
- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot, and UKI boot counting rolls back a failed update automatically.
- **Diskless-first boot** — the netboot UKI pulls `bluefin-server_<ver>.raw` into RAM with `rd.systemd.pull`, verified against the signed `SHA256SUMS` (`verify=signature`), mounts a dm-verity erofs `/usr`, and runs from tmpfs. A diskless node updates by rebooting into a newer image, and flags `/run/reboot-required` when its boot server offers one that its next boot would pull (not when the node is pinned to a versioned image).
- **Optional disk install with A/B rollback** — a running diskless node *is* the installer: see [ddi-installer.md](docs/skills/ddi-installer.md) (`systemd-sysinstall` copies `/usr` into slot A). The first disk boot creates slot B and a persistent xfs root. `systemd-sysupdate` fills the inactive slot on a timer and reboots into it nightly (Kubernetes nodes leave the reboot to kured), and UKI boot counting rolls back an update that does not boot cleanly: a boot-counted boot that has not reached `boot-complete.target` (any failed unit counts) within 15 minutes reboots, or on Kubernetes nodes is flagged for kured, until systemd-boot falls back to the previous image, which then stays put until a newer release. `/run/reboot-lock` or `/etc/reboot-lock` holds these reboots.
- **Secure Boot on** — signed systemd-boot, signed UKIs, signed kernel modules, `lockdown=integrity`. UEFI HTTP boot is supported; [Booty](https://github.com/jeefy/booty) serves the UKI, the OS DDI, the signed manifest, and a per-node `bluefin-node.ign`.
- **Opt-in per-node state via Ignition** — pass an `ignition.config` / `ignition.config.url` system credential (or, on UEFI HTTP boot, a `bluefin-node.ign` next to the UKI) and Ignition runs in the initrd on every boot; configs must be idempotent.
- **Opt-in sysexts** — k0s (Kubernetes), KubeStellar, and OpenZFS ship as separate `systemd-sysext` images, never in the base `/usr`. ZFS and KubeStellar are version-locked to the image and follow OS updates through optional sysupdate features.
Expand Down
8 changes: 5 additions & 3 deletions docs/MVP_1_0_READINESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,18 @@ This audit tracks the gap between the current tree and a first public/usable MVP
| Release workflow lint | ✅ | `actionlint .github/workflows/build.yml` clean |
| Release path exists | ✅ | `.github/workflows/build.yml` signs `SHA256SUMS` in-element (`oci/bluefin-server-image.bst`, with a `gpgv` proof against the shipped keyring) and publishes immutable `v<image-version>` GitHub Releases plus the OCI artifact `ghcr.io/<owner>/bluefin-server:<ver>,latest` |
| Automated boot test | ✅ | `boot-test` job in `build.yml` runs `scripts/dogfood-diskless.sh --check` and `scripts/dogfood-install.sh` on every PR and push to main; locally `just dogfood-check` / `just dogfood-install`. UEFI HTTP boot verified with Secure Boot (`DOGFOOD_BOOT=http`), and tampered DDI / unsigned manifest both refused (`DOGFOOD_TAMPER=raw\|sums`) |
| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots; proven by `scripts/dogfood-install.sh <dir> <next> <broken>` |
| A/B rollback | ✅ | `files/os/repart.d/` provisions usr slots A+B; `systemd-sysupdate` fills the inactive slot and UKI boot counting (`TriesLeft=3` in `files/os/sysupdate.d/20-uki.transfer`) rolls back failed boots, and `bluefin-boot-deadline.timer` reboots a counted boot that misses `boot-complete.target`; proven by `scripts/dogfood-install.sh <dir> <next> <broken>` (`DOGFOOD_BROKEN=slot` or `unit`) |
| Read-only /usr | ✅ | erofs + dm-verity pinned by `usrhash=` in the signed UKIs (`elements/oci/bluefin-server-usr.bst`, `elements/oci/bluefin-server-boot.bst`) |
| Signed diskless pull | ✅ | Netboot UKI pulls with `verify=signature`; the initrd ships gnupg and the keyring (`/etc/systemd/import-pubring.pgp` from `os-sysupdate-keys.bst`) |
| Automatic updates | ✅ | Installed nodes: `80-bluefin-updates.preset` enables `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`; Kubernetes nodes set `/run/reboot-required` and leave the reboot to kured (`ExecCondition=` interlock); `systemd-boot-check-no-failures.service` gates `boot-complete.target`, so only a boot with no failed unit is blessed, and `bluefin-boot-deadline.timer` reboots (or flags kured for) a counted boot that is not blessed in 15 minutes; `/run/reboot-lock` and `/etc/reboot-lock` hold local reboots. Diskless nodes: `bluefin-diskless-update-check` flags a newer signed release on the boot server unless the node is pinned to a versioned image. See [ddi-installer.md](skills/ddi-installer.md) "Updates" |
| Sysext lock-step updates | ✅ | `zfs` / `kubestellar` sysupdate features download the version-locked sysext with each OS update; rollback keeps the matching sysext (verified in the 6-phase `dogfood-install` run) |
| First-boot SSH keys | ✅ | Ignition (`ignition.config` / `ignition.config.url` credentials, or `bluefin-node.ign` next to the UKI on UEFI HTTP boot; `tests/fixtures/ignition/var-on-disk.ign`) and `tmpfiles.extra` / sysusers credentials (`elements/bluefin-server/os-creds-prov.bst`) |

Competitor context: [gap-analysis-distros.md](skills/gap-analysis-distros.md)

## Verdict

**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Remaining work is hardening: TPM2-sealed state, reboot coordination outside Kubernetes, merging the Booty branch, and real-hardware boot proofs.
**Alpha state — on track for MVP 1.0.** The build path, signed releases (GitHub + OCI), automated boot verification (including UEFI HTTP boot and Booty-provisioned install), A/B rollback, read-only /usr, lock-step sysext updates, and unattended provisioning are all implemented and exercised in CI. Installed nodes update and reboot on their own, with a Kubernetes interlock and a boot health gate. Remaining work is hardening: TPM2-sealed state, a cluster-wide reboot lock for non-Kubernetes fleets, merging the Booty branch, and real-hardware boot proofs.

## Roadmap

Expand Down Expand Up @@ -60,7 +61,8 @@ Priority order. Each item depends on the ones above it.
### Phase D: release discipline

- [ ] TPM2-sealed /var and credential decryption proof on real hardware.
- [ ] Reboot coordination for non-Kubernetes hosts.
- [x] Reboot coordination for non-Kubernetes hosts: nightly `systemd-sysupdate-reboot.timer`, operator lock files and a kured interlock (both from #182), and a boot health gate that reboots an unhealthy counted boot (#139).
- [ ] Cluster-wide reboot lock (FleetLock-style) for multi-node fleets without Kubernetes.
- [ ] Merge the Booty `feat/bluefin-http-boot` branch; shim-signed Secure Boot path for enrollment-free first boots.
- [ ] Tag `v1.0.0-MVP` once Phase D items land.
- [ ] Publish release notes: verified boot path, trust model, known gaps.
Expand Down
5 changes: 3 additions & 2 deletions docs/skills/architecture-roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Roadmap for future Bluefin Server architecture work. Use when plann
metadata:
type: reference
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-29"
context7-sources:
- /systemd/systemd
---
Expand All @@ -30,6 +30,7 @@ For reference, so future planning does not redo them:
- OCI artifact output (`ghcr.io/<owner>/bluefin-server:<ver>,latest`) alongside the raw release files, via ORAS in CI and `just publish-oci` locally.
- Booty HTTP boot: per-MAC serving of the UKI, DDI, `SHA256SUMS(.gpg)`, and per-host `bluefin-node.ign`, verified end to end in QEMU with Secure Boot; the Booty branch is not yet merged upstream.
- ZFS and KubeStellar sysexts version-locked to the image, delivered in lock-step with OS updates through sysupdate features; rollback keeps the matching sysext.
- Automatic updates on installed nodes (preset-enabled `systemd-sysupdate.timer` and `systemd-sysupdate-reboot.timer`), operator reboot lock files and a kured interlock for Kubernetes nodes (from #182), a boot health gate (`systemd-boot-check-no-failures.service`) whose deadline reboots an unhealthy counted boot back to the previous image (`bluefin-boot-deadline.timer`), and a signed update signal for diskless nodes that are not pinned to a versioned image (`bluefin-diskless-update-check`); see [ddi-installer.md](ddi-installer.md) "Updates".

## Planned work

Expand All @@ -41,7 +42,7 @@ Priorities are derived from [gap-analysis-distros.md](gap-analysis-distros.md).
| 2 | aarch64 build axis | `project.conf` and `include/arch.yml` already model it; no CI coverage yet. |
| 3 | Booty merge and its Secure Boot shim story | The Bluefin HTTP-boot support works from [Booty](https://github.com/jeefy/booty) `feat/bluefin-http-boot`; it still needs to merge, and enrollment-free first boots need a shim-signed path. |
| 4 | Credential provisioning smoke tests on real hardware | SSH keys, static network, and firstboot settings are wired through systemd credentials; TPM2-sealed credential decryption still needs hardware proof. |
| 5 | Native reboot coordination for non-Kubernetes and single-node hosts | Kured only covers Kubernetes nodes; no FleetLock/locksmith equivalent. |
| 5 | Cluster-wide reboot lock for non-Kubernetes fleets | Single hosts reboot in the nightly window and Kubernetes nodes use kured; several non-Kubernetes hosts sharing a service still have no FleetLock/locksmith-style lock, so they may reboot in the same window. |

## Status notes

Expand Down
30 changes: 23 additions & 7 deletions docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Build, export, and dogfood the Bluefin Server image set (OS DDI, si
metadata:
type: how-to
status: stable
last_updated: "2026-09-28"
last_updated: "2026-09-29"
context7-sources:
- /systemd/systemd
- /apache/buildstream
Expand Down Expand Up @@ -106,13 +106,29 @@ Useful environment variables:
the boot must fail with the RAM message (see `diskless-troubleshooting.md`).
- `DOGFOOD_EXTRA_PROBE=<file>` — shell snippet appended to the in-guest probe.

Every diskless `--check` boot also runs `bluefin-diskless-update-check` once
and reports `PROBE update-check=<result> flag=<set|none>`. Its origin is the
versioned `import.pull` file, so a newer release in the served directory only
logs that the node is pinned; an HTTP boot through a fixed-name UKI
(`DOGFOOD_BOOT=http`, `DOGFOOD_BOOT_URL=.../bluefin-server-netboot.efi`)
sets the flag.

`scripts/dogfood-install.sh <dir> [<next-dir> [<broken-dir>]]` is the full
end-to-end check: install from a diskless boot, boot the installed disk,
`systemd-sysupdate` A->B to `<next-dir>` with the default `Verify=yes` against
the signed manifest (with the `zfs` feature enabled, so the ZFS sysext follows
the OS in lock-step), and with `<broken-dir>` corrupt the updated slot and
confirm boot counting rolls the node back to `<next-dir>` on its own, with the
matching ZFS sysext still merged. CI runs the first two stages
`systemd-sysupdate` A->B to `<next-dir>` through `systemd-sysupdate.service`
(the unit the timer starts) with the default `Verify=yes` against the signed
manifest (with the `zfs` feature enabled, so the ZFS sysext follows the OS in
lock-step), then asserts the kured flag, the Kubernetes reboot interlock, both
timers enabled and the new UKI blessed after `boot-complete.target`, and with
`<broken-dir>` break the update and confirm boot counting rolls the node back
to `<next-dir>` on its own, with the matching ZFS sysext still merged.
`DOGFOOD_BROKEN=slot` (default) corrupts the updated usr slot, so the initrd
fails. `DOGFOOD_BROKEN=unit` adds a unit that fails on `<broken-dir>`'s version
only and shortens the boot deadline (`DOGFOOD_DEADLINE`, default `2min`): each
counted boot reaches `multi-user.target`, misses `boot-complete.target`, and
`bluefin-boot-deadline` reboots it. The run asserts three such boots, the
fallback boot with the deadline timer inactive, and then no kured flag and a
skipped `systemd-sysupdate-reboot.service` for the failed version. CI runs the first two stages
(`dogfood-diskless.sh --check`, `dogfood-install.sh dist/diskless`) as the
`boot-test` job in `.github/workflows/build.yml` on every pull request and
push to main.
Expand Down Expand Up @@ -149,7 +165,7 @@ nodes may already trust.
| "Skip `gen-dev-keys`, the build has defaults." | Signing needs real key material in `files/boot-keys/`; the recipe generates throwaway keys so local builds boot under Secure Boot. |
| "Test the UKI with `-kernel`/`-initrd`." | That bypasses the signed boot chain. The dogfood scripts boot the netboot ESP or installed disk through OVMF the way firmware does. |
| "Reboot loops mean the boot hung." | With Secure Boot in setup mode the first boot enrolls keys and reboots; that is expected once per fresh variable store. |
| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image. `dogfood-install.sh <dir> <next> <broken>` proves it. |
| "A failed update needs manual recovery." | Boot counting handles it: three failed boots of the new UKI and systemd-boot falls back to the previous image; `bluefin-boot-deadline` reboots a boot that comes up with a failed unit. `dogfood-install.sh <dir> <next> <broken>` proves it (`DOGFOOD_BROKEN=unit` for the failed-unit case). |
| "`chmod 4755` in install-commands makes a file setuid in the image." | No. BuildStream artifacts keep one executable bit per file, so every staged file is 0644/0755. FSDK components declare their special modes as initial scripts, and `oci/bluefin-server-usr.bst` runs them (`os-initial-scripts.bst`) while assembling /sysroot; special modes must be set in the `script` element that writes the image. |

## Red flags
Expand Down
Loading