Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,8 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
| Task | Skill |
|---|---|
| Boot / install / update architecture and local build + dogfood | [`docs/skills/ddi-installer.md`](docs/skills/ddi-installer.md), [`docs/skills/ddi-installer-build.md`](docs/skills/ddi-installer-build.md) |
| Offline USB installer (unattended installs, install-time provisioning) | [`docs/skills/usb-installer.md`](docs/skills/usb-installer.md) |
| Network boot at scale (Booty: HTTP boot, per-node Ignition) | [`docs/skills/booty-integration.md`](docs/skills/booty-integration.md) |
| Diskless boot failures, RAM sizing, node logs, Ignition configs | [`docs/skills/diskless-troubleshooting.md`](docs/skills/diskless-troubleshooting.md) |
| Factory role, k0s sysext rationale, lab integration | [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md) |
| Work with `systemd-sysext` / `systemd-confext` | [`docs/skills/systemd-sysext-extensions.md`](docs/skills/systemd-sysext-extensions.md) |
Expand All @@ -72,6 +74,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
| Update the FSDK pin / versioning | [`docs/skills/bump-fsdk-version.md`](docs/skills/bump-fsdk-version.md) |
| CI workflows, action SHA pinning | [`docs/skills/ci-tooling.md`](docs/skills/ci-tooling.md) |
| Release signing / sysupdate trust | [`docs/skills/systemd-sysupdate-verification.md`](docs/skills/systemd-sysupdate-verification.md) |
| Secure Boot / module / signing key inventory, rotation, CI secrets | [`docs/skills/secure-boot-keys.md`](docs/skills/secure-boot-keys.md) |
| Node access (root / SSH) and credential sealing with TPM2 | [`docs/skills/tpm2-credential-sealing.md`](docs/skills/tpm2-credential-sealing.md) |
| System containers (`machinectl`) | [`docs/skills/system-containers.md`](docs/skills/system-containers.md) |
| Cut bloat / avoid over-engineering | [`docs/skills/avoid-over-engineering.md`](docs/skills/avoid-over-engineering.md) |
Expand Down
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,18 @@ Bluefin Server is currently in **Alpha**:

> **Remote diagnostics:** OpenSSH is installed for on-demand diagnostics, but is disabled by default via systemd presets. It can be started manually with `systemctl start sshd` when remote access is needed. See [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md).

## Download a release

Each push to `main` publishes an immutable [GitHub Release](https://github.com/projectbluefin/server/releases) tagged `v<YY.MM.run>` and the same file set as an OCI artifact at `ghcr.io/projectbluefin/bluefin-server` (tags `<ver>` and `latest`). Which file you want:

| File | Use it for |
|---|---|
| `bluefin-server-netboot_<ver>.esp.raw` | Written to a USB stick, boots a node diskless (pulls the OS into RAM over HTTP). |
| `bluefin-server-installer_<ver>.raw` | Written to a USB stick, boots into `systemd-sysinstall` for an offline install to disk. |
| everything else | [Booty](https://github.com/jeefy/booty) syncs the whole release and serves it for network boot at scale. |

Verify the download against the GPG-signed `SHA256SUMS` before use; the release public keyring is [`files/os/sysupdate-keys/import-pubring.gpg`](files/os/sysupdate-keys/import-pubring.gpg), and the verification and attestation steps are in [`docs/skills/systemd-sysupdate-verification.md`](docs/skills/systemd-sysupdate-verification.md).

## Quick start

You need only `podman` and [`just`](https://github.com/casey/just). BuildStream runs inside the FSDK `bst2` container, so BuildStream is not installed locally.
Expand Down
6 changes: 3 additions & 3 deletions docs/skills/ci-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ metadata:
Every `uses:` line must reference a full commit SHA. Never use `@v2` or `@main`.

```yaml
# correct
- uses: taiki-e/install-action@b6b84cf49ebfe0176417bdce007c624f0db37f20 # v2
# correct (see build.yml for the current pinned SHA)
- uses: taiki-e/install-action@<full-commit-sha> # v2

# wrong — mutable tag, supply-chain risk
- uses: taiki-e/install-action@v2
Expand All @@ -48,7 +48,7 @@ Pin the tool version too, not only the action SHA. Without `@<version>`,
a binary chosen by an upstream release rather than by a commit in this repo.

```yaml
- uses: taiki-e/install-action@b6b84cf49ebfe0176417bdce007c624f0db37f20 # v2
- uses: taiki-e/install-action@<full-commit-sha> # v2 — see build.yml for the current pin
with:
tool: just@1.58.0
```
Expand Down
31 changes: 11 additions & 20 deletions docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,30 +54,22 @@ just publish-oci <registry-host>:30500/bluefin-server dist/diskless 1 # plain H

Every image build signs: the UKIs and systemd-boot with DB, kernel modules
with the module signing certificate, and the release `SHA256SUMS` with the
image signing key. `build-image` (and `validate`) depends on `gen-dev-keys`,
which generates throwaway keys in `files/boot-keys/` (gitignored) on first
run: PK/KEK/DB, the module certificate, and the `sysupdate-signing.asc` /
`import-pubring.pgp` pair. Keys are kept unless `--force` is given, and a
partial set (a file of a pair or of the boot set missing or empty) is an error
rather than something to fill in. CI builds
on `main` unpack the `BOOT_KEYS_TARBALL` secret, write `SYSUPDATE_SIGNING_KEY`
to `files/boot-keys/sysupdate-signing.asc`, and copy the committed release
keyring `files/os/sysupdate-keys/import-pubring.gpg` to
`files/boot-keys/import-pubring.pgp`; pull requests get throwaway keys and
their images are never published.
image signing key. What each key is, where it lives, what `just gen-dev-keys`
generates (throwaway keys in the gitignored `files/boot-keys/`, kept unless
`--force`, a partial set is an error), and how CI supplies the real keys:
[secure-boot-keys.md](secure-boot-keys.md). The release-signing half
(`sysupdate-signing.asc` / `import-pubring.pgp`, the committed
`files/os/sysupdate-keys/import-pubring.gpg`, and rotation):
[systemd-sysupdate-verification.md](systemd-sysupdate-verification.md).

**Rotating any key needs a new `image-version`.** Every key ends up in the
image bits: DB signs the UKIs and systemd-boot, the module certificate is
built into the kernel, and `import-pubring.pgp` ships in `/usr`. An image
version names one immutable set of bits, and `systemd-sysupdate` only
installs a version newer than the one it runs, so rebuilding the same version
with new keys publishes different bits under a released name and never
reaches nodes already on it. Rotate keys (`just gen-dev-keys --force`, or new
CI secrets), then `just set-version` to a version that sorts higher before
building. Nodes also need the new Secure Boot keys enrolled and, for the
image signing key, the new keyring; see
"Rotating the Signing Key" in
[systemd-sysupdate-verification.md](systemd-sysupdate-verification.md).
reaches nodes already on it. Rotate keys, then `just set-version` to a
version that sorts higher before building.

## Reproducible builds

Expand Down Expand Up @@ -204,9 +196,8 @@ projects:

`.github/workflows/build.yml` runs `just validate`, exports the image set
(already carrying its signed `SHA256SUMS(.gpg)`), runs the QEMU boot test, and
on `main` publishes `dist/diskless/` as-is: an immutable GitHub Release tagged
`v<image-version>` plus an ORAS OCI artifact at
`ghcr.io/<owner>/bluefin-server:<ver>,latest`. One version is published
on `main` publishes `dist/diskless/` as-is (see the `release` job in
[ci-tooling.md](ci-tooling.md)). One version is published
exactly once; creating an existing tag fails rather than overwriting assets
nodes may already trust.

Expand Down
44 changes: 8 additions & 36 deletions docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,14 @@ GPT partition label with room to spare):
| `bluefin-server-<ver>.efi` | Disk UKI (installed nodes); also the sysupdate source for `$BOOT`. |
| `bluefin-server-netboot_<ver>.efi` | Netboot UKI (diskless nodes); the UEFI HTTP boot / PXE target. |
| `bluefin-server-netboot_<ver>.esp.raw` | Netboot ESP image: signed systemd-boot, the netboot UKI, and Secure Boot key enrollment payloads. Write it to a USB stick to boot diskless without HTTP boot. |
| `bluefin-server-installer_<ver>.raw` | Offline USB installer: the same usr + verity images (labelled `bluefin-installer-usr` / `bluefin-installer-usr-verity`) plus an ESP with signed systemd-boot, the installer UKI, key enrollment payloads, and the disk UKI + install-time `repart.d` under `bluefin/`. Write it to a USB stick to install without a network. |
| `bluefin-server-installer_<ver>.raw` | Offline USB installer; write to a stick to install without a network. See [usb-installer.md](usb-installer.md). |
| `zfs_<ver>.raw.zst` / `kubestellar_<ver>.raw.zst` / `kubeadm_<ver>.raw.zst` | Opt-in sysext assets locked to this image version; installed nodes fetch them through the `zfs` / `kubestellar` / `kubeadm` sysupdate features. |
| `k0s-<k0s-ver>.raw.zst` | Opt-in k0s sysext asset, on its own version axis. |
| `efi-keys/` | PK/KEK/db enrollment payloads. |
| `SHA256SUMS` / `SHA256SUMS.gpg` | One manifest over every file above, signed in-element with `files/boot-keys/sysupdate-signing.asc`; the image trusts the matching `import-pubring.pgp` (see `systemd-sysupdate-verification.md`). |

A release is this directory published as-is: a GitHub Release `v<ver>` plus an
ORAS OCI artifact `ghcr.io/<owner>/bluefin-server:<ver>,latest` (one layer per
file, artifact type `application/vnd.projectbluefin.server.release.v1`).
A release publishes this directory as-is (GitHub Release `v<ver>` plus an ORAS
OCI artifact); see the `release` job in [ci-tooling.md](ci-tooling.md).
`just publish-oci REF [DIR] [PLAIN_HTTP]` pushes the same artifact locally.

The /usr image itself is built by `oci/bluefin-server-usr.bst` with an offline
Expand All @@ -64,10 +63,9 @@ signed with DB, so Secure Boot locks those command lines. `lockdown=integrity`
is always on. `os-sd-boot-signed.bst` signs systemd-boot with the same DB key
so installed disks and the netboot ESP get a loader firmware accepts.

Dev keys come from `just gen-dev-keys` (throwaway keys in the gitignored
`files/boot-keys/`, including the `sysupdate-signing.asc` /
`import-pubring.pgp` pair that signs and verifies `SHA256SUMS`); CI builds on
main use the `BOOT_KEYS_TARBALL` and `SYSUPDATE_SIGNING_KEY` secrets.
The signing keys (dev vs CI, and the pair that signs and verifies
`SHA256SUMS`): [secure-boot-keys.md](secure-boot-keys.md) and
[systemd-sysupdate-verification.md](systemd-sysupdate-verification.md).

### Diskless (netboot UKI)

Expand Down Expand Up @@ -132,34 +130,8 @@ installed disk is identical whichever path installed it. No shell installer.

### From the USB installer (offline)

```bash
sudo dd if=bluefin-server-installer_<ver>.raw of=/dev/<usb> bs=4M conv=fsync status=progress
```

```text
firmware -> systemd-boot -> bluefin-server-installer_<ver>.efi
-> /usr from the stick's bluefin-installer-usr partition (dm-verity, usrhash=)
-> tmpfs root, systemd.unit=system-install.target
-> systemd-sysinstall.service on the monitor (/dev/console = tty0)
```

The installer UKI finds its /usr by partition label, not by the
usrhash-derived UUIDs. Those UUIDs belong to installed usr slots, so an
existing Bluefin install (including the disk being overwritten) is never opened
as the installer's /usr, and an installed node booted with the stick still
plugged in never opens the stick's usr. `run-bluefin-installer.mount` mounts
the stick's ESP (`bluefin-installer`) at `/run/bluefin/installer`; the
`systemd-sysinstall.service` drop-in passes
`--definitions=/run/bluefin/installer/bluefin/repart.d` and
`--kernel=${BLUEFIN_INSTALL_KERNEL}` (the disk UKI, named by the installer
UKI's `systemd.setenv=`). The disk UKI sits outside `EFI/Linux` on the stick so
systemd-boot never offers it there. sysinstall prompts for the target disk,
erasing it, and confirmation, then reboots; remove the stick when it does.

Secure Boot: the stick's systemd-boot and UKIs are signed with the project DB
key. On bare metal put the firmware into Setup Mode and pick the enrollment
entry in the systemd-boot menu (`secure-boot-enroll if-safe` only
auto-enrolls in VMs), or turn Secure Boot off.
See [usb-installer.md](usb-installer.md) — the offline installer image, its
boot flow, unattended installs, and install-time provisioning.

### From a diskless node

Expand Down
9 changes: 5 additions & 4 deletions docs/skills/factory-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ The factory pattern is broader than a single host: a downstream CI lab or OS fac
│ Core server OS: DDI-first, image-updated │
│ • systemd-sysupdate for atomic A/B updates │
│ • systemd-sysext for optional layers (k0s, extensions) │
│ • container runtimes as systemd-sysexts │
└─────────────────────────────────────────────────────────────┘
```

Expand All @@ -47,8 +46,10 @@ See [k0s-sysext.md](k0s-sysext.md) for details.
## Workloads are containers

The workloads the factory tests and ships live in other repositories or image
pipelines. Container runtimes ship as optional `systemd-sysext`s rather than in
the base /usr image (hard rule 4 forbids container runtimes in the base image).
pipelines. No container runtime ships in the base /usr image (hard rule 4);
the only runtime we ship is containerd, inside the opt-in kubeadm sysext (see
[kubeadm-sysext.md](kubeadm-sysext.md)). Everything else runs in system
containers or in the workload images themselves.

> Bluefin Server is the factory floor; optional workloads and variant images run on that floor.

Expand All @@ -60,7 +61,7 @@ the base /usr image (hard rule 4 forbids container runtimes in the base image).
| Atomic, rollback-capable updates | Image-based A/B updates via `systemd-sysupdate`; diskless nodes update by rebooting |
| Minimal attack surface / lean base OS | Verity-sealed read-only /usr with bash; optional tools as sysexts |
| Kubernetes control plane on every node | k0s delivered as `systemd-sysext` |
| Container workloads | `podman` (and other runtimes) as optional `systemd-sysext`s |
| Container workloads | containerd via the opt-in kubeadm sysext; anything else runs in system containers |
| Signed, verifiable release artifacts | Signed UKIs + `SHA256SUMS` signed in-element, verified against `/etc/systemd/import-pubring.pgp` |

## SSH and Remote Diagnostics
Expand Down
25 changes: 6 additions & 19 deletions docs/skills/gap-analysis-distros.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,31 +147,18 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma

### Upstream distribution documentation

Specific claims carry inline links in section 2. Documentation roots:

- Ubuntu Server / autoinstall / cloud-init
- <https://documentation.ubuntu.com/server/>
- <https://raw.githubusercontent.com/canonical/ubuntu-server-documentation/main/docs/how-to/software/automatic-updates.md>
- <https://raw.githubusercontent.com/canonical/ubuntu-server-documentation/main/docs/how-to/software/upgrade-your-release.md>
- <https://raw.githubusercontent.com/canonical/ubuntu-server-documentation/main/docs/how-to/software/package-management.md>
- <https://canonical-subiquity.readthedocs-hosted.com/en/latest/intro-to-autoinstall.html>
- <https://cloudinit.readthedocs.io/>
- Talos Linux
- <https://docs.siderolabs.com/talos/v1.13/overview/what-is-talos.md>
- <https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/lifecycle-management/upgrading-talos.md>
- <https://docs.siderolabs.com/talos/v1.13/build-and-extend-talos/custom-images-and-development/system-extensions.md>
- <https://docs.siderolabs.com/talos/v1.13/reference/configuration/overview.md>
- Flatcar Container Linux
- <https://raw.githubusercontent.com/flatcar/flatcar-docs/main/docs/setup/releases/update-strategies.md>
- <https://raw.githubusercontent.com/flatcar/flatcar-docs/main/docs/provisioning/ignition/_index.md>
- <https://raw.githubusercontent.com/flatcar/flatcar-docs/main/docs/provisioning/sysext/_index.md>
- Talos Linux — <https://docs.siderolabs.com/talos/v1.13/>
- Flatcar Container Linux — <https://www.flatcar.org/docs/latest/>
- Fedora CoreOS / rpm-ostree / Zincati
- <https://raw.githubusercontent.com/coreos/fedora-coreos-docs/main/modules/ROOT/pages/index.adoc>
- <https://raw.githubusercontent.com/coreos/fedora-coreos-docs/main/modules/ROOT/pages/auto-updates.adoc>
- <https://raw.githubusercontent.com/coreos/fedora-coreos-docs/main/modules/ROOT/pages/producing-ign.adoc>
- <https://raw.githubusercontent.com/coreos/fedora-coreos-docs/main/modules/ROOT/pages/running-containers.adoc>
- <https://raw.githubusercontent.com/coreos/fedora-coreos-docs/main/modules/ROOT/pages/faq.adoc>
- <https://raw.githubusercontent.com/coreos/rpm-ostree/main/README.md>
- <https://docs.fedoraproject.org/en-US/fedora-coreos/>
- <https://coreos.github.io/rpm-ostree/>
- <https://coreos.github.io/zincati/>
- <https://coreos.github.io/zincati/development/fleetlock/>

### systemd reference

Expand Down
3 changes: 3 additions & 0 deletions docs/skills/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w
|---|---|---|
| [`avoid-over-engineering.md`](avoid-over-engineering.md) | Cutting scope, deleting code, resisting bloat | Rules and red flags for keeping solutions small. |
| [`architecture-roadmap.md`](architecture-roadmap.md) | Future architecture direction, long-lead design | Roadmap for systemd-native architecture work. |
| [`booty-integration.md`](booty-integration.md) | HTTP boot, iPXE chainloading, per-node Ignition, kubeadm worker provisioning via Booty | How the Booty network boot server serves releases to nodes. |
| [`bump-fsdk-version.md`](bump-fsdk-version.md) | Pinning or retagging the FSDK junction | Update the pinned FSDK release and derived tags. |
| [`ci-tooling.md`](ci-tooling.md) | GitHub Actions, workflow SHA pinning, CI conventions | CI conventions and release pipeline rules. |
| [`ddi-installer-build.md`](ddi-installer-build.md) | Building and dogfooding the image set | Local build, export, and QEMU dogfood workflow. |
Expand All @@ -34,11 +35,13 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w
| [`kubeadm-sysext.md`](kubeadm-sysext.md) | Joining a kubeadm cluster, kubelet/containerd sysext, Kubernetes kernel options | Build and runtime contract for the kubeadm worker sysext. |
| [`k0s-sysext-ops.md`](k0s-sysext-ops.md) | Operating k0s or the KubeStellar appliance on Bluefin Server | Runtime operation and troubleshooting for the k0s and KubeStellar sysexts. |
| [`k0s-sysext.md`](k0s-sysext.md) | Building the k0s sysext | BuildStream element and publish steps for the k0s sysext. |
| [`secure-boot-keys.md`](secure-boot-keys.md) | Rotating signing keys, setting up CI secrets, debugging signature verification | Key inventory, `gen-dev-keys`, CI secrets, and rotation. |
| [`skill-improvement.md`](skill-improvement.md) | Adding, splitting, or refactoring skills | Meta-skill that owns the documentation loop. |
| [`system-containers.md`](system-containers.md) | Running `systemd-nspawn` toolboxes | System container operation with `machinectl`. |
| [`systemd-sysext-extensions.md`](systemd-sysext-extensions.md) | Optional layers via `systemd-sysext` / `systemd-confext` | Extension identity, loading, and version matching. |
| [`systemd-sysupdate-verification.md`](systemd-sysupdate-verification.md) | Image-based A/B updates and signed manifests | Release signing, `systemd-sysupdate`, and trust model. |
| [`tpm2-credential-sealing.md`](tpm2-credential-sealing.md) | First-boot credentials, root/SSH node access, TPM2 sealing | Provisioning credentials, node access, and sealing with `systemd-creds`. |
| [`usb-installer.md`](usb-installer.md) | The offline USB installer image, unattended installs, install-time provisioning | `bluefin-server-installer_<ver>.raw` boot flow, sysinstall drop-in, ESP credentials. |

## Standing facts

Expand Down
Loading