Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion Justfile
Original file line number Diff line number Diff line change
Expand Up @@ -352,7 +352,6 @@ test-installer-artifact:
-drive if=pflash,format=raw,readonly=on,file="$OVMF_CODE" \
-drive if=pflash,format=raw,file="$WORKDIR/ovmf-vars.fd" \
-nic user,model=virtio-net-pci \
-smbios "type=11,value=io.systemd.credential.binary:fstab.extra=L2Rldi9kaXNrL2J5LXBhcnRsYWJlbC92YXIgL3ZhciB4ZnMgZGVmYXVsdHMgMCAwCg==" \
-smbios "type=11,value=io.systemd.credential.binary:systemd.extra-unit.bluefin-kiosk-ready.service=${READY_UNIT}" \
-smbios "type=11,value=io.systemd.stub.kernel-cmdline-extra=console=tty0 console=ttyS0,,115200 systemd.mask=systemd-firstboot.service systemd.mask=systemd-homed-firstboot.service systemd.wants=bluefin-kiosk-ready.service" \
-nographic \
Expand Down
10 changes: 5 additions & 5 deletions docs/MVP_1_0_READINESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,15 +20,15 @@ This audit tracks the gap between the current tree and a first public/usable MVP
| Release path exists | ✅ | `.github/workflows/build.yml` builds, signs, uploads to GitHub Release |
| Cluster build pipeline | ✅ | Phase A complete; pipeline builds DDI, installer, and k0s sysext with uutils |
| Automated boot test | 🔄 | Phase B in progress for Alpha; `bluefin-server-boot-test` workflow running on lab cluster |
| A/B root rollback | ❌ | `50-root.transfer` names `root-a`/`root-b`, installer only creates `root-a` |
| Root immutability | ❌ | DDI boots read/write (`rw` on cmdline) |
| A/B /usr rollback | ❌ | `50-root.transfer` now targets the single writable ROOT slot; USR-B is a spare /usr slot but is empty and unverity'd until the /usr DDI ticket stages verity into it |
| Root immutability | 🔄 | `/usr` is mounted read-only at runtime (`mount.usr=PARTLABEL=USR-A`, issue #134); the UKI cmdline still boots `rw` and is deferred to Phase C |
| First-boot SSH keys | ❌ | Only root password credential path exists |

Competitor context: [gap-analysis-distros.md](skills/gap-analysis-distros.md)

## Verdict

**Alpha state — in progress for MVP 1.0.** Phase A (build path, uutils, k0s sysext, validate) is complete. Phase B (automated boot test on lab cluster) is actively in progress for the Alpha milestone. Full MVP 1.0 release requires concluding Phase B boot verification and Phase C runtime hardening (automated rollback, read-only `/usr`, credential delivery).
**Alpha state — in progress for MVP 1.0.** Phase A (build path, uutils, k0s sysext, validate) is complete. Phase B (automated boot test on lab cluster) is actively in progress for the Alpha milestone. Full MVP 1.0 release requires concluding Phase B boot verification and Phase C runtime hardening (automated `/usr` rollback and credential delivery).

## Roadmap

Expand All @@ -51,8 +51,8 @@ Priority order. Each item depends on the ones above it.

### Phase C: update/rollback and provisioning

- [ ] Add `root-b` to installer repart recipes and verify `systemd-sysupdate` stages into the inactive slot.
- [ ] Switch UKI cmdline from `rw` to `ro` and rely on `/var` for mutable state.
- [ ] Stage the /usr DDI into USR-A, populate USR-B as a verity-protected spare, and verify `systemd-sysupdate` stages into the inactive /usr slot.
- [ ] Switch UKI cmdline from `rw` to `ro`; mutable state now lives on the writable ROOT slot (`/var`, `/etc`, `/home`).
- [ ] Consume `systemd-creds` for SSH authorized keys and static network config.
- [ ] Add boot menu entry to select the previous slot after a failed update.

Expand Down
2 changes: 1 addition & 1 deletion docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ The release process is driven by `.github/workflows/build.yml`:
- [ ] The live installer does not bake hardcoded SSH keys or pre-hashed root
passwords.
- [ ] The DDI is decompressed after the cpio step.
- [ ] `files/installer/repart.d/20-root-a.conf` has `GrowFileSystem=yes`.
- [ ] `files/installer/repart.d/50-root.conf` has `GrowFileSystem=yes`.

## See also

Expand Down
19 changes: 11 additions & 8 deletions docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ metadata:
- Writing or refining `systemd-repart`, `bootctl`, or `ukify` configurations.
- Packaging or publishing DDI assets to GitHub Releases.
- Managing partition recipes for the target disk layout (`10-esp.conf`,
`20-root-a.conf`, `30-var.conf`).
`20-usr-a.conf`, `30-usr-b.conf`, `40-oem.conf`, `50-root.conf`).

## When NOT to Use

Expand Down Expand Up @@ -74,13 +74,14 @@ container runtime pod sandboxes.
5. `systemd-sysinstall` reads partition recipes from
`/usr/lib/repart.sysinstall.d/` if it is populated; otherwise it falls back
to `/usr/lib/repart.d/`. The target recipes are staged at
`/usr/lib/repart.d/` (`10-esp.conf`, `20-root-a.conf`, `30-var.conf`).
6. `20-root-a.conf` copies the DDI block-for-block from
`/usr/lib/repart.d/` (`10-esp.conf`, `20-usr-a.conf`, `30-usr-b.conf`,
`40-oem.conf`, `50-root.conf`).
6. `20-usr-a.conf` copies the OS /usr DDI block-for-block from
`/dev/disk/by-partlabel/bluefin-installer-data` (the embedded DDI data
partition on the installer media).
7. Target OS volume expansion is handled by `systemd-growfs`; the target OS
stack includes `xfsprogs` so the root and `/var` filesystems can grow to fill
their partitions on first boot.
stack includes `xfsprogs` so the writable root filesystem can grow to fill
its partition on first boot.

## Partition Layout

Expand All @@ -96,8 +97,10 @@ container runtime pod sandboxes.
| Partition | Type | Size | Contents |
|---|---|---|---|
| ESP | vfat | 500 MiB – 1 GiB | `systemd-boot` + target OS UKI (`bluefin-server.efi`) |
| `bluefin-server-root-a` | XFS | 4 GiB – 8 GiB | OS root filesystem (copied from installer data partition) |
| `var` | XFS | ≥ 4 GiB | Writable persistent `/var`; grows to fill remaining disk |
| USR-A | 5dfbf5f4 (Flatcar root-fs) | 2 GiB | Read-only /usr (OS /usr DDI, copied block-for-block; mounted read-only) |
| USR-B | 5dfbf5f4 (Flatcar root-fs) | 2 GiB | Spare read-only /usr slot; A/B staging later |
| OEM | 0fc63daf (Flatcar OEM) | ≥ 128 MiB | ext4; cloud metadata / Ignition (filesystem label `OEM`) |
| ROOT | 3884dd41 (Flatcar) | ≥ 4 GiB | Writable root (/etc, /var, /home); grows to fill disk; seeds k0s sysext |

## Installer Boot Flow

Expand Down Expand Up @@ -189,7 +192,7 @@ offline installation.
- [ ] The interactive installer service sets `TTYPath=/dev/tty0` so the TUI
appears on the attached display even when serial is the primary console.
- [ ] `bluefin-server-installer.bst` decompresses the DDI after the cpio step.
- [ ] `files/installer/repart.d/20-root-a.conf` has `GrowFileSystem=yes`.
- [ ] `files/installer/repart.d/50-root.conf` has `GrowFileSystem=yes`.

## See also

Expand Down
13 changes: 6 additions & 7 deletions docs/skills/gap-analysis-distros.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from
| Axis | Bluefin Server (as-implemented) |
|------|---------------------------------|
| **Philosophy** | Systemd-native, minimal, image-based server OS appliance; base DDI includes bash for login and bring-up while heavy developer/debug tools live in sysexts or system containers; intended to run container workloads and Kubernetes via optional sysexts. Sources: [AGENTS.md](../../AGENTS.md), [factory-integration.md](factory-integration.md). |
| **State model** | Target OS DDI is an XFS filesystem image. A separate persistent `/var` partition is created by the installer. There is no second root slot provisioned today, and the UKI cmdline currently uses `rw`, so the root is not mounted read-only at runtime. Sources: [bluefin-server-ddi.bst](../../elements/oci/bluefin-server-ddi.bst), [20-root-a.conf](../../files/installer/repart.d/20-root-a.conf), [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst). |
| **State model** | Target OS DDI is an XFS filesystem image copied into the read-only USR-A slot (issue #134). `/var` lives on the writable ROOT slot; USR-B is a spare /usr slot. The UKI boots with `mount.usr=PARTLABEL=USR-A`, so `/usr` is mounted read-only at runtime. Sources: [bluefin-server-ddi.bst](../../elements/oci/bluefin-server-ddi.bst), [20-usr-a.conf](../../files/installer/repart.d/20-usr-a.conf), [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst). |
| **Updates** | `systemd-sysupdate` reads root/UKI transfers from `files/os/sysupdate.d/` and the optional k0s transfer from the `k0s` component directory. Assets are published to GitHub Releases, and the combined `SHA256SUMS` manifest is signed in CI with a GPG key. `Verify=yes` is the default. Sources: [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md), [50-root.transfer](../../files/os/sysupdate.d/50-root.transfer), [60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer), [70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer), also `systemd-sysupdate(8)`. |
| **Provisioning** | The installer is an offline `systemd-sysinstall` image that embeds the DDI as a data partition. First-boot configuration is intended to be delivered via `systemd-creds` through the ESP or hypervisor metadata. Today only `passwd.hashed-password.root` is consumed via `systemd-sysusers.d`; the documented `tmpfiles.extra` path for SSH keys and similar files is not implemented. Sources: [bluefin-server-installer.bst](../../elements/oci/bluefin-server-installer.bst), [10-root-creds.conf](../../files/os/sysusers.d/10-root-creds.conf), [os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst), [systemd-creds(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd-creds.html). |
| **Customization** | Adds software through `systemd-sysext` (overlay `/usr`) and `systemd-confext` (overlay `/etc`) images. The base OS `os-release` advertises `ID=flatcar` and a matching `VERSION_ID` so pre-built Flatcar Bakery extensions load. k0s is shipped as a separately built, optionally enabled sysext. Sources: [systemd-sysext-extensions.md](systemd-sysext-extensions.md), [k0s-sysext.md](k0s-sysext.md), [os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst), [systemd-sysext(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html). |
Expand All @@ -105,9 +105,8 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from

### Root filesystem and A/B rollback

- **Gap:** Bluefin's `systemd-sysupdate` root transfer already names two target partitions (`root-a` and `root-b`) in `50-root.transfer`, but the installer only creates one root partition (`20-root-a.conf`).
There is no `root-b` partition yet, so `systemd-sysupdate` cannot stage an update into an inactive slot and the OS has no atomic rollback path comparable to Flatcar/Fedora CoreOS/Talos today.
- **Gap:** The DDI filesystem is created as a writable XFS image and the installed UKI boots it with `rw`. A read-only `/usr` state model, as intended by the sysext-first design, is not enforced at runtime.
- **Gap:** `systemd-sysupdate` root transfer now targets the single writable ROOT slot (`50-root.transfer`, `MatchPattern=ROOT`). Atomic A/B rollback for `/usr` is provided by the USR-A/USR-B pair (issue #134), but USR-B is left empty and unverity'd until the /usr DDI ticket stages verity into it; until then there is no active rollback path comparable to Flatcar/Fedora CoreOS/Talos.
- **Gap:** The read-only `/usr` state model, as intended by the sysext-first design, is now enforced at runtime (`USR-A` mounted read-only via `mount.usr=PARTLABEL=USR-A`, issue #134).
- **Gap:** There is no mechanism to select the previous OS version at boot if an update fails; recovery currently depends on reinstalling from media.

### Provisioning
Expand All @@ -119,12 +118,12 @@ Bluefin Server is a BuildStream 2-based, image-based Linux server OS built from

### Update delivery

- **Gap:** The root transfer uses `Type=partition Path=auto`, which requires `systemd-sysupdate` to discover a matching GPT partition label (`bluefin-server-root-a`/`root-b`). This is correct, but without a `root-b` partition the transfer effectively overwrites the running root in place.
- **Gap:** The root transfer uses `Type=partition Path=auto` with `MatchPattern=ROOT`, which matches the single writable ROOT slot (issue #134). Because ROOT is a single slot, an update overwrites the running root in place; atomic rollback now lives in the /usr USR-A/USR-B pair once the /usr DDI stages verity into it.
- **Gap:** `systemd-sysupdate-reboot.service`/`systemd-sysupdate-reboot.timer` are not enabled or configured; the only reboot signal today is the Kured hook.

### Customization

- **No major gap found relative to the design intent.** `systemd-sysext` and Flatcar Bakery compatibility match the intended extension model. The main operational concern is that, because `/usr` is not mounted read-only, a sysext merge vs. runtime writes to `/usr` have different guarantees than on Flatcar or Fedora CoreOS.
- **No major gap found relative to the design intent.** `systemd-sysext` and Flatcar Bakery compatibility match the intended extension model. `/usr` is now mounted read-only at runtime (`USR-A`, issue #134), so sysext merges and runtime writes to `/usr` have the same guarantees as on Flatcar or Fedora CoreOS.

### Reboot coordination

Expand Down Expand Up @@ -189,7 +188,7 @@ These gaps drive the priorities in [architecture-roadmap.md](architecture-roadma
- [elements/bluefin-server/os-release-flatcar.bst](../../elements/bluefin-server/os-release-flatcar.bst)
- [elements/bluefin-server/os-creds-prov.bst](../../elements/bluefin-server/os-creds-prov.bst)
- [elements/bluefin-server/os-kured-hook.bst](../../elements/bluefin-server/os-kured-hook.bst)
- [files/installer/repart.d/20-root-a.conf](../../files/installer/repart.d/20-root-a.conf)
- [files/installer/repart.d/20-usr-a.conf](../../files/installer/repart.d/20-usr-a.conf)
- [files/os/sysupdate.d/50-root.transfer](../../files/os/sysupdate.d/50-root.transfer)
- [files/os/sysupdate.d/60-uki.transfer](../../files/os/sysupdate.d/60-uki.transfer)
- [files/os/sysupdate.k0s.d/70-k0s.transfer](../../files/os/sysupdate.k0s.d/70-k0s.transfer)
Expand Down
4 changes: 2 additions & 2 deletions elements/flatcar/flatcar-zfs.bst
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@ config:
# a runtime sysext, so udevd was ordered After=systemd-sysext.service.
# In Bluefin Server, ZFS is extracted directly into the OS rootfs at build time.
# Leaving this drop-in causes systemd-udevd to wait for systemd-sysext.service,
# which waits for local-fs.target, which waits for var.mount, which waits for
# the udev device unit — creating a boot deadlock.
# which waits for local-fs.target, which waits for the udev device unit —
# creating a boot deadlock.
rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d"
if [ -f "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" ]; then
sed -i "/systemd-udevd\.service\.d/d" "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf"
Expand Down
6 changes: 4 additions & 2 deletions elements/installer/installer-repart.bst
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,10 @@ description: |

Partition layout:
10-esp.conf EFI System Partition (vfat, 500M–1G)
20-root-a.conf Root slot A — DDI filesystem image copied here
30-var.conf /var — xfs, writable, grows to fill disk
20-usr-a.conf Read-only /usr slot A — OS /usr DDI copied here
30-usr-b.conf Read-only /usr slot B — spare, empty (A/B + verity later)
40-oem.conf OEM — ext4, cloud-metadata / Ignition (label OEM)
50-root.conf Writable root — grows to fill disk, seeds k0s sysext

sources:
- kind: local
Expand Down
8 changes: 5 additions & 3 deletions elements/oci/bluefin-server-ddi.bst
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: |
Output: bluefin-server-ddi-<ver>.raw.zst + SHA256SUMS

The output .raw is a filesystem image consumed by installer repart config:
files/installer/repart.d/20-root-a.conf via CopyBlocks=.
files/installer/repart.d/20-usr-a.conf via CopyBlocks=.

build-depends:
- base/base-stack.bst
Expand Down Expand Up @@ -70,8 +70,10 @@ config:
# Ensure standard /etc/hosts exists for container runtimes (containerd/podman)
printf '127.0.0.1 localhost\n::1 localhost ip6-localhost ip6-loopback\n' > /layer/etc/hosts

# Ensure persistent /var is mounted from its partition label
printf '/dev/disk/by-partlabel/var /var xfs defaults 0 0\n' > /layer/etc/fstab
# The Flatcar layout (issue #134) has no separate /var partition: /var
# lives on the writable ROOT slot, so the base fstab carries no mounts.
# ponytail: empty fstab — root mounts via the kernel, /var is on ROOT.
: > /layer/etc/fstab

# Ensure all standard virtual, serial, and hypervisor consoles allow root login
printf 'tty0\ntty1\ntty2\ntty3\ntty4\ntty5\ntty6\nttyS0\nttyS1\nconsole\nhvc0\n' > /layer/etc/securetty
Expand Down
4 changes: 2 additions & 2 deletions elements/oci/bluefin-server-installer.bst
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ config:
udevadm settle --timeout=10 || true
local target="${TARGET_DISK:-}"
if [ -z "${target}" ]; then
local root_part="$(readlink -f /dev/disk/by-partlabel/bluefin-server-root-a 2>/dev/null || true)"
local root_part="$(readlink -f /dev/disk/by-partlabel/ROOT 2>/dev/null || true)"
if [ -n "${root_part}" ]; then
target="$(lsblk -p -n -d -o PKNAME "${root_part}" 2>/dev/null || true)"
fi
Expand Down Expand Up @@ -410,7 +410,7 @@ config:
# 20-ddi-data.conf: Data (XFS) — bluefin-server-ddi.raw copied here block-for-block
#
# At install time, systemd-sysinstall mounts the data partition to
# locate the DDI at the path expected by repart.d/20-root-a.conf (CopyBlocks=).
# locate the DDI at the path expected by repart.d/20-usr-a.conf (CopyBlocks=).
- |
set -eux
OUT="%{install-root}"
Expand Down
8 changes: 4 additions & 4 deletions files/installer/repart.d/10-esp.conf
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# EFI System Partition
# Created empty by systemd-repart; populated post-install by the installer
# script via: bootctl install --esp-path=<mounted-ESP> --root=<mounted-root>
# which copies systemd-boot and registers the OS UKI from the installed rootfs.
# EFI System Partition — Flatcar/GPT ESP (type GUID c12a7328-…).
# Created empty by systemd-repart; populated post-install by the installer via
# `bootctl install`, which installs systemd-boot and registers the OS UKI.
[Partition]
Type=esp
Format=vfat
Label=EFI-SYSTEM
SizeMinBytes=500M
SizeMaxBytes=1G
11 changes: 0 additions & 11 deletions files/installer/repart.d/20-root-a.conf

This file was deleted.

11 changes: 11 additions & 0 deletions files/installer/repart.d/20-usr-a.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Read-only /usr slot A — Flatcar root-fs type GUID (5dfbf5f4-…).
# systemd-repart copies the OS /usr DDI block-for-block from the embedded
# installer data partition on the boot medium (stamped `bluefin-installer-data`).
# Mounted read-only at runtime via `mount.usr=PARTLABEL=USR-A`.
[Partition]
Type=5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6
Label=USR-A
CopyBlocks=/dev/disk/by-partlabel/bluefin-installer-data
SizeMinBytes=2G
SizeMaxBytes=2G
GrowFileSystem=yes
7 changes: 7 additions & 0 deletions files/installer/repart.d/30-usr-b.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Read-only /usr slot B — spare, empty and sized to match USR-A.
# A/B staging + verity for /usr lands here in the /usr DDI ticket (issue #134).
[Partition]
Type=5dfbf5f4-2848-4bac-aa5e-0d9a20b745a6
Label=USR-B
SizeMinBytes=2G
SizeMaxBytes=2G
13 changes: 0 additions & 13 deletions files/installer/repart.d/30-var.conf

This file was deleted.

9 changes: 9 additions & 0 deletions files/installer/repart.d/40-oem.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# OEM slot — Flatcar OEM type GUID (0fc63daf-…).
# Holds cloud-provider metadata / Ignition config. The boot path matches on the
# *filesystem* label, so FSLabel=OEM is set explicitly (Format=ext4).
[Partition]
Type=0fc63daf-8483-4772-8e79-3d69d8477de4
Label=OEM
FSLabel=OEM
Format=ext4
SizeMinBytes=128M
11 changes: 11 additions & 0 deletions files/installer/repart.d/50-root.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Writable root slot — Flatcar type GUID (3884dd41-…).
# The writable overlay for the read-only /usr: holds /etc, /var, /home and state.
# Grows to fill the remaining disk. Seeds the offline k0s sysext raw so
# k0s-first-boot.service finds it at /var/lib/k0s/k0s.raw (see issue #134).
[Partition]
Type=3884dd41-8582-4404-b9a8-e9b84f2df50e
Label=ROOT
Format=
GrowFileSystem=yes
CopyFiles=/k0s.raw:/var/lib/k0s/k0s.raw
SizeMinBytes=4G
1 change: 0 additions & 1 deletion files/os/systemd/system-preset/zz-enable-var-mount.preset

This file was deleted.

Loading
Loading