diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..bec20d8a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,247 @@ +# Changelog + +All notable changes to this project are documented in this file. The format is +based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this +project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +## [1.0.0-rc.1] - 2026-10-04 + +The first release since v0.3. Booty is no longer just a PXE server that hands +out Flatcar and CoreOS: it now boots diskless and installed Bluefin Server +nodes over UEFI HTTP Boot with Secure Boot, bootstraps the cluster itself +(kubeadm or k0s, managed or external), rolls releases across the fleet with a +health-gated autopilot, and exposes power control over WoL/SSH/Kubernetes API +from the same UI. Proven on a five-node homelab of Bluefin Server and Flatcar +hosts. + +This is a release candidate. See +[docs/releases/v1.0.0-rc.1.md](docs/releases/v1.0.0-rc.1.md) for upgrade +notes and known limitations. + +### Breaking changes + +- **OCI / ostree image cache removed.** The embedded registry that mirrored + each host's `ostreeImage` into `data/registry/` is gone, together with the + `/registry` and `/v2/` endpoints and the UI's Cache page. Hosts rebase + straight from their own registry, exactly what `{{ .OSTreeImage }}` renders. + `--ociGC` and `--ociGCEmpty` are accepted as deprecated no-ops so an old + command line still starts, and go away in the next release. On start-up an + empty `data/registry/` is removed silently; one that still holds blobs is + left in place with a warning. (#73) +- **Data directory layout migrates to `data///`.** Flatcar, + CoreOS and Bluefin now keep each release under `data///` + behind relative `current`, `previous` and `lastGood` symlinks. Booty + migrates older layouts automatically on start-up; the pre-migration + top-level names (`data/flatcar_production_pxe.vmlinuz`, + `data/fedora-coreos--live-*`) stay as symlinks into + `/current`, so anything that still references them keeps working. + (#40) +- **Bluefin nodes get a `core` user; `root` is locked by the image.** + Upstream Bluefin Server 26.10 (projectbluefin/server#275) ships root locked + and runs sshd without PAM, so the previous "put keys on root" behaviour is + unusable. Booty now provisions `core` with your `--sshAuthorizedKeys`, + passwordless sudo and `sshd.service` enabled. SSH as `core`, not `root`. + (#70) +- **`/update-check` on Bluefin answers `rebootRequired` from the target + rule.** The rule is the same for every OS now: `rebootRequired` when the + host's effective target (its `targetVersion` when set, else the fleet + target) differs from what it reports running. A diskless Bluefin host + behind its target is told to reboot and re-image. (#40) +- **`--builtin` default gains `health` and `wol`.** The default value is now + `hostname,update,booted,sshkeys,health,wol`. Hosts pick up two new systemd + units on next boot: `booty-health.service` POSTs a health report to + `/health` after `multi-user.target`, and `10-booty-wol.link` arms + Wake-on-LAN on the host's NIC. Pass `--builtin=hostname,update,booted,sshkeys` + (or `none`) to opt out. (#40, #67) +- **`GET /cluster` adds `source` / `connected` / `nodes` / `apiServer`, and + `ready` means "connected" in external mode.** In the default + `--controlPlane=external` there is no bootstrap to wait for, so `ready` + reports whether Booty can talk to the API server; `readyAt` is omitted. In + managed mode the meaning is unchanged (`kubeadm init` finished, + `/readyz` answers). (#64) +- **Actuator order is kured, then SSH, then the Kubernetes API.** Reboots go + through kured when its DaemonSet is present, else `--rebootSSHKey`, else a + privileged reboot Pod from Booty's own image as the last resort. An + explicit SSH key now wins over the API actuator, which matches the operator + intent of handing Booty a key. Operator Power buttons never go through + kured. (#65, #67) +- **`--coreOSChannel=none` and `--flatcarChannel=none` are now real + opt-outs.** Setting a channel to `none` stops version checks and downloads + for that OS, answers `/update-check` and `/info` as if no release were + cached, refuses new registrations of it with `400`, and serves an already + registered host a refusal menu. The directory is left on disk and listed by + `/storage` as `tracked:false` until you delete it by hand. (#73) +- **`GET /autopilot` is bounded by default.** The answer now carries the + newest 10 `good` records whose files were pruned (plus every live record) + and the newest 100 of the 200-event ring; `releaseCount` and `eventCount` + say what was left out. Use `?releases=all` and `?events=all` for the whole + history. (#75) + +### Added + +- **Cluster bootstrap from PXE** (#32, #33, #34, #36). Booty provisions a + whole Kubernetes cluster from nothing: it generates the cluster CA under + `data/cluster/pki/` (or loads your `--clusterCADir`), renders the + control-plane host's Ignition and joins every other host as a worker, on + Flatcar, Fedora CoreOS and Bluefin Server. Two distributions are supported: + kubeadm (default) and k0s (`--clusterDistribution=k0s`), under either + `--controlPlane=managed` (Booty renders the control plane itself) or + `external` (today's join-only behaviour). `GET /cluster` reports the live + connection, bootstrap progress and per-host roles. kubeadm join tokens are + minted per boot through the Kubernetes API with `--kubeadmJoin=auto` + (one-hour TTL, cleaned up on the update tick); k0s worker tokens are + minted the same way. CNI (`--cni=cilium|calico|flannel|none`) is installed + from the control plane. +- **Secure Boot, UEFI HTTP Boot and ProxyDHCP** (#25, #37, #38). `--secureBoot` + answers UEFI HTTP Boot clients over ProxyDHCP with a Microsoft-signed iPXE + shim that loads a signed iPXE, so machines with Secure Boot enabled can + netboot without disabling it. Flatcar kernels are verified against the + Flatcar Secure Boot CA extracted from the release itself once you assert + `--secureBootTrusted=flatcar` (and enrol the CA into the firmware `db`); + Fedora CoreOS boots through Fedora's shim; anything else gets a refusal + menu that explains the missing piece instead of a failed boot. + `--proxyDHCP` (PXE 2.1, off by default) answers PXE clients on UDP + 67 + 4011 alongside your existing DHCP server, so the network needs no + `next-server`/`filename` for Booty-managed machines. +- **Bluefin Server support** (#30, #39, #63). Booty serves Bluefin Server + diskless over UEFI HTTP Boot with a per-host Ignition config + (`/bluefin//bluefin-node.ign`), or installs it to disk with + `doInstall` + `installDisk`. Hosts carry `stateDisk` (persistent `/var`), + `extensions` (opt-in `zfs` / `kubestellar` / `k0s` / NVIDIA driver and + `nvidia-container-toolkit` sysexts) and `mode`. Releases are tracked from + the GitHub releases of `--bluefinRepo` (default `projectbluefin/server`) + or from an ORAS OCI artifact via `--bluefinOCI`, every file verified + against the release's `SHA256SUMS` (plus the `SHA256SUMS.gpg` signature + when `--bluefinKeyring` is set, failing closed). Chainloading from iPXE + with Secure Boot off and a legacy BIOS diskless path cover older firmware. +- **Configuration view and editor** (#31). `GET /config` shows every + effective setting with its source (flag / env / default), secrets redacted, + plus the Butane templates in play and whether each is writable. The UI's + Config page reads it, and a template editor (`GET/PUT + /config/template`, `POST /config/template/validate`) validates with Butane + and writes atomically inside the data directory; read-only mounts (a + Kubernetes ConfigMap, for example) are detected and refused with a clear + reason. +- **Autopilot** (#40, #41, #42, #43, #57, #71, #72, #75). Booty is now a + self-healing upgrade controller. `--autopilot=guard` gates every reboot + into a release the host has not passed on (kernel fetched, OS up, no failed + units, Node Ready with workloads healthy inside `--autopilotHealthWindow`), + retries once, rolls the host back to `lastGood` and holds the fleet there + when the release is at fault, and gives up to `needs-hands` after + `--autopilotRetryAfter`. `--autopilot=full` adds a Bluefin canary-serial + rollout (one host at a time, canaries first), release quarantine and + skip-to-next. Health reports come from a `POST /health` builtin unit. A + report stub is kept per blamed release, rendered to + `data/autopilot/reports/-.md` with redaction (hostnames, + MACs, IPs, UUIDs, tokens), and `--autopilotIssues=/` with + `--githubToken` posts Bluefin quarantines as deduplicated GitHub issues + (prefix titles with `--autopilotIssueTitlePrefix` while testing against a + fork). Endpoints: `GET /autopilot` (bounded, see above), + `GET /autopilot/reports/.md`, + `POST /autopilot/{os}/release/{version}/clear`, + `POST /autopilot/host/{mac}/clear`, all surfaced on the Autopilot page. +- **Power** (#67, #68). Booty tracks each host's power state from the + signals it already sees plus a TCP probe, and exposes + `GET /power` and `POST /power/{mac}/{on,reboot,shutdown,cancel}`. + *Power on* sends Wake-on-LAN magic packets to every broadcast and the + host's own IP; *Reboot* and *Shutdown* go through the same SSH and API + actuators the autopilot uses (never kured), with drain, force and reason + fields. The UI shows a per-host badge with a tooltip, three buttons behind + an inline confirm, a fleet summary on the Overview and the events in the + Autopilot timeline. +- **Storage page and `GET /storage`** (#73, #74). Replaces the OCI Cache + page. Reports the data directory by release with size, mtime, the links + that name it (`current` / `previous` / `lastGood`), the hosts running or + pinned to it, and the retention reason (`current`, `previous`, `lastGood`, + `pinned`, or `-` for "the next sync prunes it"). Pruned autopilot history + is joined in behind a *Show pruned release history* switch. +- **"No agent yet" badge and fleet targets** (#68). Hosts that have never + reported (`running`, `lastCheck`, `booted`, `health` all empty -- pre-P1 + boots, or `--builtin=none`) show a muted *no agent yet* badge instead of + an unknown version, and the Overview counts them. `GET /info.targets` + reports the hold-aware fleet target per OS so the Hosts table can mark + every host on-target or behind. +- **Cluster card for external control planes** (#64). The Overview's Cluster + card reads *connected* / *unreachable* in external mode, with the node + count and API server shown when known. +- **Release retention by reference** (#40). Pruning keeps exactly the + releases that `current`, `previous`, `lastGood` or any host's + `targetVersion` names, and removes every other release directory -- a bad + release can be rolled back to `lastGood` without a download, at the cost + of up to three releases on disk per OS. +- **`booty node-reboot`** (#41). New subcommand run by the reboot Pod the + API actuator schedules: `sync`, then `SIGRTMIN+5` to PID 1 (reboot.target). + `--poweroff` sends `SIGRTMIN+4` instead. + +### Fixed + +- **Stability gate cap.** The L2 "node Ready and stable" requirement is + capped at half of `--autopilotHealthWindow`, so a short window stays + passable. (#72) +- **Serial-rollout hold immediacy.** The Bluefin fleet target is held at + `lastGood` the instant a new release lands, not from the controller's next + tick, so a host whose update timer fires in between still sees the held + target. (#72) +- **Netboot-version-aware `SHA256SUMS`.** `/bluefin//` serves the + checksum files of the release this host's UKI or kernel was fetched from + (`netbootVersion`), so an initrd keeps verifying against its own release + even when the autopilot repoints the host mid-boot. (#72) +- **Blank containerd disk format on Bluefin.** A Bluefin kubeadm worker + formats `--containerdDisk` ext4 once if and only if `blkid -p` finds the + device blank (no filesystem, no partition table), refuses its own + `stateDisk` / `installDisk`, and mounts whatever was already there as-is. + The old "never format" rule used to fail the mount and with it containerd, + the kubelet and the join on a fresh disk. (#69) +- **Health gate ignores Booty's own reboot Pod.** The health gate no longer + counts the privileged `booty-node-reboot-` Pod (which necessarily + ends `Failed` when the node reboots underneath it) as an unhealthy + workload. (#71) +- **L0 only from the host's own address.** A kernel or UKI fetch counts as + a boot (starting the health gate, moving the power tracker) only when it + comes from the host's recorded IP, so a stray `curl` of + `/booty.ipxe?mac=…` from a workstation is served as usual but starts no + gate and changes no power state. (#72) +- **cosign v3 verify.** Provenance verification uses + `cosign verify-attestation --new-bundle-format=false`, which keeps reading + the legacy OCI `.att` tag the SLSA generator still emits (cosign v3 + otherwise auto-detects bundles and misses it). (#59) +- **Flaky controller test.** The controller tracks its goroutines so tests + can wait for them. (#72) +- **Storage page follows the bounded `/autopilot` answer.** The Storage page + fetches `?releases=all` so the pruned-history join sees the whole release + history, not just the default window. (#75) +- **Storage tables align across OSes.** Untracked and empty cells read + consistently between Flatcar, CoreOS and Bluefin. (#74) + +### Security + +- **SLSA provenance on release images.** Release tags are built by GitHub + Actions with SLSA level-3 provenance, signed with cosign (keyless), and + the attestation is verified in CI before the build is considered green. + Verify locally with: + + ``` + COSIGN_EXPERIMENTAL=1 cosign verify-attestation --new-bundle-format=false \ + --certificate-identity-regexp=".*" --certificate-oidc-issuer-regexp=".*" \ + --type slsaprovenance ghcr.io/jeefy/booty@ + ``` + +### Documentation + +- README: Secure Boot / UEFI HTTP Boot section with enrolment instructions + (#37, #38), Bluefin Server section covering releases, boot paths, Ignition, + NVIDIA and install (#30, #39, #60, #63), cluster bootstrap and managed k0s + (#32 – #35), autopilot P1–P5 (#40, #41, #42, #43, #57), Power (#67), + Storage (#73), configuration editor (#31), actuator order (#65), `/cluster` + connection fields and what `ready` means (#64), Bluefin `core` user (#70), + no-agent badge and `/info.targets` (#68), bounded `/autopilot` (#75). +- `examples/k8s.yaml` rewritten as a production-shaped Deployment: the + ServiceAccount, Roles and ClusterRole the autopilot and token minting + need, a ConfigMap with the site-only Butane, a Secret with the SSH keys, + `/healthz` probes, `NET_BIND_SERVICE`+`NET_RAW` only, and the + `POD_NAMESPACE` / `BOOTY_IMAGE` environment variables the API actuator + reads. (#27, #41) + +## [0.3] - 2023-12-21 + +See the [v0.3 release on GitHub](https://github.com/jeefy/booty/releases/tag/v0.3). diff --git a/README.md b/README.md index 66bae2fc..9f82045f 100644 --- a/README.md +++ b/README.md @@ -104,6 +104,22 @@ booty --dataDir ./data # --serverIP is autodetected; pass it explicitly behi `booty init [dir]` (default: `--dataDir`, `BOOTY_DATADIR` or `./data`) never overwrites existing files -- rerunning it prints `exists, skipped` -- and ends with the DHCP settings for your network (`next-server`, `filename undionly.kpxe` / `ipxe.efi`), the run command and the UI URL. The starter template is valid on its own: hostname, SSH keys, the update timer and the booted callback come from the [builtin fragment](#composition), so edit it only for what is specific to your fleet. +### Releases + +Container tags: `ghcr.io/jeefy/booty:vX.Y.Z` tags are built by CI from the +tagged commit with SLSA provenance and a keyless cosign signature. Verify a +release with: + +``` +COSIGN_EXPERIMENTAL=1 cosign verify-attestation --new-bundle-format=false \ + --certificate-identity-regexp=".*" --certificate-oidc-issuer-regexp=".*" \ + --type slsaprovenance ghcr.io/jeefy/booty@ +``` + +The `:main` tag tracks the default branch. Release-candidate tags +(`-rc.N`) do **not** move `:latest`. Notable changes between releases are +listed in [CHANGELOG.md](CHANGELOG.md). + ## Features * iPXE boot (BIOS and x86-64 UEFI) into the latest Flatcar-Linux or CoreOS, and UEFI HTTP Boot of Bluefin Server's signed netboot UKI (diskless, or installed to disk) @@ -941,7 +957,7 @@ Binding UDP 69 needs `CAP_NET_BIND_SERVICE` (as do UDP 67 and 4011 for `--proxyD ``` docker run --rm --network=host \ --cap-drop ALL --cap-add NET_BIND_SERVICE --cap-add NET_RAW \ - -v $PWD/data:/data ghcr.io/jeefy/booty:main --serverIP=192.168.1.10 + -v $PWD/data:/data ghcr.io/jeefy/booty:v1.0.0-rc.1 --serverIP=192.168.1.10 ``` On Kubernetes set `securityContext.capabilities: {drop: [ALL], add: [NET_BIND_SERVICE, NET_RAW]}` and use `/healthz` for the liveness and readiness probes. If you use `--tftpPort` above 1024 and always pass the MAC in the URL, no capabilities are required. @@ -957,7 +973,7 @@ docker run --rm -it \ --network=host \ --cap-drop ALL --cap-add NET_BIND_SERVICE --cap-add NET_RAW \ -v $PWD:/data/ \ - ghcr.io/jeefy/booty:main \ + ghcr.io/jeefy/booty:v1.0.0-rc.1 \ --profile=kubeadm-worker \ --joinString="kubeadm join 192.168.1.10:6443 --token ${TOKEN} --discovery-token-ca-cert-hash sha256:${SHA_HASH}" \ --serverIP=192.168.1.10 \ diff --git a/docs/releases/v1.0.0-rc.1.md b/docs/releases/v1.0.0-rc.1.md new file mode 100644 index 00000000..54eca090 --- /dev/null +++ b/docs/releases/v1.0.0-rc.1.md @@ -0,0 +1,319 @@ +# Booty v1.0.0-rc.1 release notes + +First release since v0.3 (December 2023). Booty started as a small iPXE +server that handed Flatcar and Fedora CoreOS to whatever netbooted; it is now +a PXE / UEFI HTTP Boot server that also **bootstraps the Kubernetes cluster** +(kubeadm or k0s, managed or external) and runs **self-healing OS upgrades** +behind a health gate, with rollback, quarantine and redacted upstream bug +reports. Developed and run against a five-node homelab of Bluefin Server and +Flatcar machines. + +This is a release candidate. Please read the breaking changes and the +known-limitations list before pointing real hardware at it, and file issues +on anything that surprises you. + +## Upgrading from v0.3 + +* **Data directory migrates automatically.** On the first start of this + build Booty moves Flatcar / CoreOS / Bluefin artifacts into + `data///` and creates `current`, `previous` and `lastGood` + symlinks per OS. The pre-migration top-level names + (`data/flatcar_production_pxe.vmlinuz`, + `data/fedora-coreos--live-*`) stay as symlinks into + `/current`, so DHCP entries and scripts that reference them keep + working. The migration is idempotent and logged as + `… release layout migrated`. +* **Flags removed.** The OCI image cache that mirrored hosts' `ostreeImage` + into `data/registry/` is gone, with it the `/registry` and `/v2/` + endpoints and the UI's Cache page. Hosts rebase directly from their own + registry (that is what `{{ .OSTreeImage }}` renders). `--ociGC` and + `--ociGCEmpty` are accepted as deprecated no-ops for one release so an + old command line still starts, and go away in v1.0.0 proper. A leftover + `data/registry/` directory is removed when empty, or kept with a warning + that it is unused and safe to delete. +* **Flag defaults changed.** `--builtin` now defaults to + `hostname,update,booted,sshkeys,health,wol` (added: `health`, `wol`). + Hosts will pick up two new systemd units on next boot + (`booty-health.service` and `10-booty-wol.link`). Pass + `--builtin=hostname,update,booted,sshkeys` to keep the previous set. +* **Kubernetes manifest.** If you deploy Booty into a cluster, replace your + manifest with the rewritten [examples/k8s.yaml](../examples/k8s.yaml): + it carries a new `ClusterRole` the autopilot's API actuator needs (node + patching, pod eviction, reboot Pods) and the `POD_NAMESPACE` / + `BOOTY_IMAGE` environment variables that actuator reads. Keep + `BOOTY_IMAGE` equal to the Deployment's `image:`; the downward API cannot + supply it. +* **kured recommended.** The autopilot's preferred reboot actuator is + [kured](https://github.com/kubereboot/kured); without it Booty falls back + to `--rebootSSHKey`, then to a privileged reboot Pod from its own image. + The operator Power buttons skip kured by design. +* **Bluefin `core` user.** Bluefin nodes now get a `core` user with your + SSH keys and passwordless sudo; the image has shipped root locked since + projectbluefin/server 26.10, so the previous root-key behaviour is gone. + SSH as `core`, not `root`. + +## Known limitations (rc) + +* **WoL is untested live.** Wake-on-LAN magic-packet dispatch is + unit-tested but has not been exercised against real hardware on a real + LAN yet. Expect the *Power on* button to need a firmware tweak on some + NICs (ErP off, "Power on by PCI-E" on, `WakeOnLan=magic` armed by the + `wol` builtin). +* **SSH actuator is unit-tested only.** The kured and Kubernetes API + actuators have run against a real cluster; the SSH path + (`--rebootSSHKey`, `sudo systemctl reboot` as `core`) is covered by + tests against an in-process SSH server but has not yet been the actuator + of choice on a live rollout. +* **No authentication, by design.** Booty has no authn/authz on its HTTP + port. Anyone who can reach it can register hosts, set `targetVersion`, + clear quarantines, reboot or shut down a registered host. Treat the boot + VLAN like your DHCP server; this is the documented threat model. +* **Bluefin GPU hosts need projectbluefin/server ≥ 26.10.862.** Earlier + Bluefin releases ship neither the `nvidia-open-` driver sysext + nor `nvidia-container-toolkit`, so the `extensions` field is a no-op on + them. +* **NFSv3 on Bluefin is blocked upstream.** Bluefin Server does not ship + the NFSv3 userspace helpers; tracked at + [projectbluefin/server#316](https://github.com/projectbluefin/server/issues/316). + NFSv4 works. + +## Version stamping + +The container image's version comes from the tag: the workflow passes +`BOOTY_VERSION=${{ github.ref_name }}` as a build-arg, the Dockerfile turns +it into `-X main.version=${BOOTY_VERSION}`, the binary exposes it as +`/info.booty.version`, and the UI's **About** page renders that field. +Running `:v1.0.0-rc.1` should show `Booty version: v1.0.0-rc.1` there. + +## Full changelog + +### Breaking changes + +- **OCI / ostree image cache removed.** The embedded registry that mirrored + each host's `ostreeImage` into `data/registry/` is gone, together with + the `/registry` and `/v2/` endpoints and the UI's Cache page. Hosts + rebase straight from their own registry, exactly what `{{ .OSTreeImage }}` + renders. `--ociGC` and `--ociGCEmpty` are accepted as deprecated no-ops + so an old command line still starts, and go away in the next release. On + start-up an empty `data/registry/` is removed silently; one that still + holds blobs is left in place with a warning. (#73) +- **Data directory layout migrates to `data///`.** Flatcar, + CoreOS and Bluefin now keep each release under `data///` + behind relative `current`, `previous` and `lastGood` symlinks. Booty + migrates older layouts automatically on start-up; the pre-migration + top-level names (`data/flatcar_production_pxe.vmlinuz`, + `data/fedora-coreos--live-*`) stay as symlinks into + `/current`, so anything that still references them keeps working. + (#40) +- **Bluefin nodes get a `core` user; `root` is locked by the image.** + Upstream Bluefin Server 26.10 (projectbluefin/server#275) ships root + locked and runs sshd without PAM, so the previous "put keys on root" + behaviour is unusable. Booty now provisions `core` with your + `--sshAuthorizedKeys`, passwordless sudo and `sshd.service` enabled. + SSH as `core`, not `root`. (#70) +- **`/update-check` on Bluefin answers `rebootRequired` from the target + rule.** The rule is the same for every OS now: `rebootRequired` when the + host's effective target (its `targetVersion` when set, else the fleet + target) differs from what it reports running. A diskless Bluefin host + behind its target is told to reboot and re-image. (#40) +- **`--builtin` default gains `health` and `wol`.** The default value is + now `hostname,update,booted,sshkeys,health,wol`. Hosts pick up two new + systemd units on next boot: `booty-health.service` POSTs a health report + to `/health` after `multi-user.target`, and `10-booty-wol.link` arms + Wake-on-LAN on the host's NIC. Pass + `--builtin=hostname,update,booted,sshkeys` (or `none`) to opt out. + (#40, #67) +- **`GET /cluster` adds `source` / `connected` / `nodes` / `apiServer`, + and `ready` means "connected" in external mode.** In the default + `--controlPlane=external` there is no bootstrap to wait for, so `ready` + reports whether Booty can talk to the API server; `readyAt` is omitted. + In managed mode the meaning is unchanged (`kubeadm init` finished, + `/readyz` answers). (#64) +- **Actuator order is kured, then SSH, then the Kubernetes API.** Reboots + go through kured when its DaemonSet is present, else `--rebootSSHKey`, + else a privileged reboot Pod from Booty's own image as the last resort. + An explicit SSH key now wins over the API actuator, which matches the + operator intent of handing Booty a key. Operator Power buttons never go + through kured. (#65, #67) +- **`--coreOSChannel=none` and `--flatcarChannel=none` are now real + opt-outs.** Setting a channel to `none` stops version checks and + downloads for that OS, answers `/update-check` and `/info` as if no + release were cached, refuses new registrations of it with `400`, and + serves an already registered host a refusal menu. The directory is left + on disk and listed by `/storage` as `tracked:false` until you delete it + by hand. (#73) +- **`GET /autopilot` is bounded by default.** The answer now carries the + newest 10 `good` records whose files were pruned (plus every live + record) and the newest 100 of the 200-event ring; `releaseCount` and + `eventCount` say what was left out. Use `?releases=all` and + `?events=all` for the whole history. (#75) + +### Added + +- **Cluster bootstrap from PXE** (#32, #33, #34, #36). Booty provisions a + whole Kubernetes cluster from nothing: it generates the cluster CA under + `data/cluster/pki/` (or loads your `--clusterCADir`), renders the + control-plane host's Ignition and joins every other host as a worker, on + Flatcar, Fedora CoreOS and Bluefin Server. Two distributions are + supported: kubeadm (default) and k0s (`--clusterDistribution=k0s`), under + either `--controlPlane=managed` (Booty renders the control plane itself) + or `external` (today's join-only behaviour). `GET /cluster` reports the + live connection, bootstrap progress and per-host roles. kubeadm join + tokens are minted per boot through the Kubernetes API with + `--kubeadmJoin=auto` (one-hour TTL, cleaned up on the update tick); k0s + worker tokens are minted the same way. CNI + (`--cni=cilium|calico|flannel|none`) is installed from the control plane. +- **Secure Boot, UEFI HTTP Boot and ProxyDHCP** (#25, #37, #38). + `--secureBoot` answers UEFI HTTP Boot clients over ProxyDHCP with a + Microsoft-signed iPXE shim that loads a signed iPXE, so machines with + Secure Boot enabled can netboot without disabling it. Flatcar kernels + are verified against the Flatcar Secure Boot CA extracted from the + release itself once you assert `--secureBootTrusted=flatcar` (and enrol + the CA into the firmware `db`); Fedora CoreOS boots through Fedora's + shim; anything else gets a refusal menu that explains the missing piece + instead of a failed boot. `--proxyDHCP` (PXE 2.1, off by default) + answers PXE clients on UDP 67 + 4011 alongside your existing DHCP + server, so the network needs no `next-server`/`filename` for + Booty-managed machines. +- **Bluefin Server support** (#30, #39, #63). Booty serves Bluefin Server + diskless over UEFI HTTP Boot with a per-host Ignition config + (`/bluefin//bluefin-node.ign`), or installs it to disk with + `doInstall` + `installDisk`. Hosts carry `stateDisk` (persistent + `/var`), `extensions` (opt-in `zfs` / `kubestellar` / `k0s` / NVIDIA + driver and `nvidia-container-toolkit` sysexts) and `mode`. Releases are + tracked from the GitHub releases of `--bluefinRepo` (default + `projectbluefin/server`) or from an ORAS OCI artifact via + `--bluefinOCI`, every file verified against the release's `SHA256SUMS` + (plus the `SHA256SUMS.gpg` signature when `--bluefinKeyring` is set, + failing closed). Chainloading from iPXE with Secure Boot off and a + legacy BIOS diskless path cover older firmware. +- **Configuration view and editor** (#31). `GET /config` shows every + effective setting with its source (flag / env / default), secrets + redacted, plus the Butane templates in play and whether each is + writable. The UI's Config page reads it, and a template editor + (`GET/PUT /config/template`, `POST /config/template/validate`) validates + with Butane and writes atomically inside the data directory; read-only + mounts (a Kubernetes ConfigMap, for example) are detected and refused + with a clear reason. +- **Autopilot** (#40, #41, #42, #43, #57, #71, #72, #75). Booty is now a + self-healing upgrade controller. `--autopilot=guard` gates every reboot + into a release the host has not passed on (kernel fetched, OS up, no + failed units, Node Ready with workloads healthy inside + `--autopilotHealthWindow`), retries once, rolls the host back to + `lastGood` and holds the fleet there when the release is at fault, and + gives up to `needs-hands` after `--autopilotRetryAfter`. + `--autopilot=full` adds a Bluefin canary-serial rollout (one host at a + time, canaries first), release quarantine and skip-to-next. Health + reports come from a `POST /health` builtin unit. A report stub is kept + per blamed release, rendered to + `data/autopilot/reports/-.md` with redaction (hostnames, + MACs, IPs, UUIDs, tokens), and `--autopilotIssues=/` with + `--githubToken` posts Bluefin quarantines as deduplicated GitHub issues + (prefix titles with `--autopilotIssueTitlePrefix` while testing against + a fork). Endpoints: `GET /autopilot` (bounded, see above), + `GET /autopilot/reports/.md`, + `POST /autopilot/{os}/release/{version}/clear`, + `POST /autopilot/host/{mac}/clear`, all surfaced on the Autopilot page. +- **Power** (#67, #68). Booty tracks each host's power state from the + signals it already sees plus a TCP probe, and exposes `GET /power` and + `POST /power/{mac}/{on,reboot,shutdown,cancel}`. *Power on* sends + Wake-on-LAN magic packets to every broadcast and the host's own IP; + *Reboot* and *Shutdown* go through the same SSH and API actuators the + autopilot uses (never kured), with drain, force and reason fields. The + UI shows a per-host badge with a tooltip, three buttons behind an inline + confirm, a fleet summary on the Overview and the events in the Autopilot + timeline. +- **Storage page and `GET /storage`** (#73, #74). Replaces the OCI Cache + page. Reports the data directory by release with size, mtime, the links + that name it (`current` / `previous` / `lastGood`), the hosts running + or pinned to it, and the retention reason (`current`, `previous`, + `lastGood`, `pinned`, or `-` for "the next sync prunes it"). Pruned + autopilot history is joined in behind a *Show pruned release history* + switch. +- **"No agent yet" badge and fleet targets** (#68). Hosts that have never + reported (`running`, `lastCheck`, `booted`, `health` all empty -- pre-P1 + boots, or `--builtin=none`) show a muted *no agent yet* badge instead of + an unknown version, and the Overview counts them. `GET /info.targets` + reports the hold-aware fleet target per OS so the Hosts table can mark + every host on-target or behind. +- **Cluster card for external control planes** (#64). The Overview's + Cluster card reads *connected* / *unreachable* in external mode, with + the node count and API server shown when known. +- **Release retention by reference** (#40). Pruning keeps exactly the + releases that `current`, `previous`, `lastGood` or any host's + `targetVersion` names, and removes every other release directory -- a + bad release can be rolled back to `lastGood` without a download, at the + cost of up to three releases on disk per OS. +- **`booty node-reboot`** (#41). New subcommand run by the reboot Pod the + API actuator schedules: `sync`, then `SIGRTMIN+5` to PID 1 + (reboot.target). `--poweroff` sends `SIGRTMIN+4` instead. + +### Fixed + +- **Stability gate cap.** The L2 "node Ready and stable" requirement is + capped at half of `--autopilotHealthWindow`, so a short window stays + passable. (#72) +- **Serial-rollout hold immediacy.** The Bluefin fleet target is held at + `lastGood` the instant a new release lands, not from the controller's + next tick, so a host whose update timer fires in between still sees the + held target. (#72) +- **Netboot-version-aware `SHA256SUMS`.** `/bluefin//` serves the + checksum files of the release this host's UKI or kernel was fetched + from (`netbootVersion`), so an initrd keeps verifying against its own + release even when the autopilot repoints the host mid-boot. (#72) +- **Blank containerd disk format on Bluefin.** A Bluefin kubeadm worker + formats `--containerdDisk` ext4 once if and only if `blkid -p` finds + the device blank (no filesystem, no partition table), refuses its own + `stateDisk` / `installDisk`, and mounts whatever was already there + as-is. The old "never format" rule used to fail the mount and with it + containerd, the kubelet and the join on a fresh disk. (#69) +- **Health gate ignores Booty's own reboot Pod.** The health gate no + longer counts the privileged `booty-node-reboot-` Pod (which + necessarily ends `Failed` when the node reboots underneath it) as an + unhealthy workload. (#71) +- **L0 only from the host's own address.** A kernel or UKI fetch counts + as a boot (starting the health gate, moving the power tracker) only + when it comes from the host's recorded IP, so a stray `curl` of + `/booty.ipxe?mac=…` from a workstation is served as usual but starts no + gate and changes no power state. (#72) +- **cosign v3 verify.** Provenance verification uses `cosign + verify-attestation --new-bundle-format=false`, which keeps reading the + legacy OCI `.att` tag the SLSA generator still emits (cosign v3 + otherwise auto-detects bundles and misses it). (#59) +- **Flaky controller test.** The controller tracks its goroutines so + tests can wait for them. (#72) +- **Storage page follows the bounded `/autopilot` answer.** The Storage + page fetches `?releases=all` so the pruned-history join sees the whole + release history, not just the default window. (#75) +- **Storage tables align across OSes.** Untracked and empty cells read + consistently between Flatcar, CoreOS and Bluefin. (#74) + +### Security + +- **SLSA provenance on release images.** Release tags are built by GitHub + Actions with SLSA level-3 provenance, signed with cosign (keyless), and + the attestation is verified in CI before the build is considered green. + Verify locally with: + + ``` + COSIGN_EXPERIMENTAL=1 cosign verify-attestation --new-bundle-format=false \ + --certificate-identity-regexp=".*" --certificate-oidc-issuer-regexp=".*" \ + --type slsaprovenance ghcr.io/jeefy/booty@ + ``` + +### Documentation + +- README: Secure Boot / UEFI HTTP Boot section with enrolment instructions + (#37, #38), Bluefin Server section covering releases, boot paths, + Ignition, NVIDIA and install (#30, #39, #60, #63), cluster bootstrap and + managed k0s (#32 – #35), autopilot P1–P5 (#40, #41, #42, #43, #57), + Power (#67), Storage (#73), configuration editor (#31), actuator order + (#65), `/cluster` connection fields and what `ready` means (#64), + Bluefin `core` user (#70), no-agent badge and `/info.targets` (#68), + bounded `/autopilot` (#75). +- `examples/k8s.yaml` rewritten as a production-shaped Deployment: the + ServiceAccount, Roles and ClusterRole the autopilot and token minting + need, a ConfigMap with the site-only Butane, a Secret with the SSH + keys, `/healthz` probes, `NET_BIND_SERVICE`+`NET_RAW` only, and the + `POD_NAMESPACE` / `BOOTY_IMAGE` environment variables the API actuator + reads. (#27, #41) diff --git a/examples/k8s.yaml b/examples/k8s.yaml index 4315764a..2009bd1c 100644 --- a/examples/k8s.yaml +++ b/examples/k8s.yaml @@ -184,7 +184,7 @@ spec: effect: NoSchedule containers: - name: booty - image: ghcr.io/jeefy/booty:main + image: ghcr.io/jeefy/booty:v1.0.0-rc.1 imagePullPolicy: Always args: - --dataDir=/data @@ -208,7 +208,7 @@ spec: fieldRef: fieldPath: metadata.namespace - name: BOOTY_IMAGE - value: ghcr.io/jeefy/booty:main + value: ghcr.io/jeefy/booty:v1.0.0-rc.1 ports: - name: tftp containerPort: 69