Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
| Update the FSDK pin / versioning | [`docs/skills/bump-fsdk-version.md`](docs/skills/bump-fsdk-version.md) |
| CI workflows, action SHA pinning | [`docs/skills/ci-tooling.md`](docs/skills/ci-tooling.md) |
| Release signing / sysupdate trust | [`docs/skills/systemd-sysupdate-verification.md`](docs/skills/systemd-sysupdate-verification.md) |
| Credential sealing with TPM2 | [`docs/skills/tpm2-credential-sealing.md`](docs/skills/tpm2-credential-sealing.md) |
| Node access (root / SSH) and credential sealing with TPM2 | [`docs/skills/tpm2-credential-sealing.md`](docs/skills/tpm2-credential-sealing.md) |
| System containers (`machinectl`) | [`docs/skills/system-containers.md`](docs/skills/system-containers.md) |
| Cut bloat / avoid over-engineering | [`docs/skills/avoid-over-engineering.md`](docs/skills/avoid-over-engineering.md) |
| Add or refactor skills | [`docs/skills/skill-improvement.md`](docs/skills/skill-improvement.md) |
Expand Down
4 changes: 3 additions & 1 deletion docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Build, export, and dogfood the Bluefin Server image set (OS DDI, si
metadata:
type: how-to
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-28"
context7-sources:
- /systemd/systemd
- /apache/buildstream
Expand Down Expand Up @@ -86,6 +86,8 @@ Useful environment variables:

- `DOGFOOD_IGNITION=<file>` — pass an Ignition config as the `ignition.config`
credential (see `tests/fixtures/ignition/var-on-disk.ign`).
- `DOGFOOD_CREDS=<dir>` — pass every file in `<dir>` as a system credential
named after the file (SMBIOS type 11).
- `DOGFOOD_STATE_DISK=<file>` — attach a persistent second disk (`/dev/vdb`).
- `DOGFOOD_VARS=<file>` — persistent UEFI variable store (keeps enrolled keys
across runs).
Expand Down
4 changes: 2 additions & 2 deletions docs/skills/factory-integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Understand Bluefin Server's role as the core OS for an image-based
metadata:
type: reference
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-28"
---
# Factory Integration

Expand Down Expand Up @@ -65,7 +65,7 @@ the base /usr image (hard rule 4 forbids container runtimes in the base image).

## SSH and Remote Diagnostics

> `sshd` is present in the OS image for on-demand diagnostics and bring-up troubleshooting, but is disabled by default via `disable sshd.service` in systemd presets. Operators can start it on-demand with `systemctl start sshd` or enable it when remote access is required. Root login is permitted with password and pubkey.
> `sshd` is present in the OS image for on-demand diagnostics and bring-up troubleshooting, but is disabled by default via `disable sshd.service` in systemd presets. Operators can start it on-demand with `systemctl start sshd` or enable it when remote access is required. Login is key-only (`PermitRootLogin prohibit-password`, `PasswordAuthentication no`), and root ships locked; provision keys per node as described under "Node access" in [tpm2-credential-sealing.md](tpm2-credential-sealing.md).

## When to Use

Expand Down
4 changes: 2 additions & 2 deletions docs/skills/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Lazy-load manifest for Bluefin Server skills. Load this file after
metadata:
type: index
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-28"
---
# docs/skills — Index

Expand Down Expand Up @@ -37,7 +37,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w
| [`system-containers.md`](system-containers.md) | Running `systemd-nspawn` toolboxes | System container operation with `machinectl`. |
| [`systemd-sysext-extensions.md`](systemd-sysext-extensions.md) | Optional layers via `systemd-sysext` / `systemd-confext` | Extension identity, loading, and version matching. |
| [`systemd-sysupdate-verification.md`](systemd-sysupdate-verification.md) | Image-based A/B updates and signed manifests | Release signing, `systemd-sysupdate`, and trust model. |
| [`tpm2-credential-sealing.md`](tpm2-credential-sealing.md) | TPM2-bound first-boot credentials | Credential sealing with `systemd-creds` and TPM2. |
| [`tpm2-credential-sealing.md`](tpm2-credential-sealing.md) | First-boot credentials, root/SSH node access, TPM2 sealing | Provisioning credentials, node access, and sealing with `systemd-creds`. |

## Standing facts

Expand Down
92 changes: 71 additions & 21 deletions docs/skills/tpm2-credential-sealing.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Securing provisioning credentials (such as hashed root passwords or
metadata:
type: how-to
status: stable
last_updated: "2026-09-26"
last_updated: "2026-09-28"
context7-sources:
- /systemd/systemd
---
Expand All @@ -21,8 +21,10 @@ decrypts and routes at boot:

| Credential | Consumer | Purpose |
|---|---|---|
| `passwd.hashed-password.root` | `systemd-sysusers` | Optional root password hash for break-glass provisioning. |
| `tmpfiles.extra` | `systemd-tmpfiles` | Extra tmpfiles rules, such as writing an operator's SSH `authorized_keys`. |
| `passwd.hashed-password.root`, `passwd.plaintext-password.root` | `systemd-firstboot` | Optional console root password; see [Node access](#node-access). |
| `ssh.authorized_keys.root` | `systemd-tmpfiles` (stock `provision.conf`) | Root's SSH `authorized_keys`; see [Node access](#node-access). |
| `ssh.listen` | `systemd-ssh-generator` | Socket-activated sshd on the given address or port; see [Node access](#node-access). |
| `tmpfiles.extra` | `systemd-tmpfiles` | Extra tmpfiles rules, such as files for other accounts. |
| `network.network.*`, `network.netdev.*`, `network.link.*`, `network.conf.*` | `systemd-network-generator` | Static network, routes, virtual devices, and networkd config. |
| `network.dns`, `network.search_domains` | `systemd-resolved` | DNS resolver defaults. |
| `firstboot.locale`, `firstboot.locale-messages`, `firstboot.keymap`, `firstboot.timezone`, `firstboot.hostname` | `bluefin-firstboot-credentials.service` | Non-interactive locale, keymap, timezone, and hostname setup. |
Expand All @@ -31,6 +33,59 @@ The stock DHCP network remains installed in `/usr/lib/systemd/network/20-wired.n
Credential-generated network files are emitted under `/run/systemd/network/` and
can use lower numeric prefixes such as `10-static.network` to override DHCP.

## Node access

The image ships no password. Root's `/etc/shadow` entry is `!unprovisioned`
(`elements/oci/bluefin-server-usr.bst`), systemd's marker for a root account
nobody has configured: `passwd -S root` reports `L`, console login, `su` and
`sulogin` refuse it, and sshd accepts keys only
(`files/os/ssh/sshd_config.d/bluefin-server.conf`). Access is provisioned per
node:

- **SSH key:** Ignition `passwd.users` with `name: root` and
`sshAuthorizedKeys` (the path Booty's `bluefin-node.ign` uses; see
Ignition in [ddi-installer.md](ddi-installer.md)), or the
`ssh.authorized_keys.root` credential, whose content is the
`authorized_keys` file. Both write `/root/.ssh/authorized_keys`.
`sshd.service` is disabled by preset; enable it per node in the same
Ignition config (`systemd.units`: `name: sshd.service`, `enabled: true`),
or pass the `ssh.listen` credential (e.g. `22`), for which
`systemd-ssh-generator` socket-activates a per-connection sshd with the
same key-only policy.
- **Console password:** `passwd.hashed-password.root` (a `crypt(5)` hash with
no trailing newline, e.g. `mkpasswd --method=yescrypt | tr -d '\n'`) or
`passwd.plaintext-password.root`.
`systemd-sysusers` cannot apply these, because it only sets passwords for
accounts it creates and root already exists in the factory `/etc/shadow`.
`systemd-firstboot.service` applies them instead: it treats `!unprovisioned`
as unconfigured. Its interactive root password prompt is removed
(`files/os/creds/systemd/system/systemd-firstboot.service.d/`) so a node
without the credential boots unattended and stays locked.

`systemd-firstboot` runs on first boot only. A diskless node rebuilds `/etc`
from `/usr/share/factory/etc` and is on its first boot every time, so the
credential applies on every boot. An installed node applies it on the first
boot of the installed disk and keeps the result in `/etc/shadow` on the
persistent xfs root; A/B updates replace only the usr, usr-verity and UKI
slots (`files/os/sysupdate.d/`), so the password survives them. A credential
added to an installed node after its first boot is ignored; change the
password with `passwd` instead.

Deliver the credentials like any other: SMBIOS type 11 or QEMU fw_cfg on VMs,
an Ignition-capable boot server, or encrypted `.cred` files in
`/loader/credentials/` on the ESP (see below). The USB installer needs no
login: `systemd-sysinstall` runs on the console. It does not carry the
installer's credentials over, so place them in `/loader/credentials/` on the
installed disk's ESP before its first boot.

With root locked, `emergency.target` and `rescue.target` on the booted system
get no shell: `sulogin` reports the locked account and boot continues. Do not
set `SYSTEMD_SULOGIN_FORCE`, which opens a passwordless root shell on the
console; the kernel command line is sealed in the signed UKI, so it cannot be
added at boot either. For break-glass console access, provision a password
credential and `sulogin` asks for it. The initrd never offers a shell; it
prints the errors and reboots.

## Verify TPM2 device availability

```bash
Expand Down Expand Up @@ -83,28 +138,23 @@ On bare metal, an operator mounts the ESP partition (e.g. filesystem label
`EFI-SYSTEM` or partition type `C12A7328-F81F-11D2-BA4B-00A0C93EC93B`) and
places encrypted credential files into `/loader/credentials/`. During boot,
`systemd-creds` decrypts them and passes them to matching system consumers.
ESP credentials are untrusted, so systemd only accepts them encrypted; a
`--with-key=null` credential is refused on a machine with a TPM2 and Secure
Boot on. PCR 11 changes with every UKI, so a diskless node that boots each new
release needs its `--tpm2-pcrs=7+11` credentials re-sealed per release.

### SSH key provisioning with tmpfiles.extra

To provision SSH keys at first boot on bare metal or virtual machines, deliver a
`tmpfiles.extra` credential (for example `/loader/credentials/tmpfiles.extra.cred`).
`systemd-tmpfiles` consumes this credential on first boot. The payload must create
parent directories before writing the file.

For the `root` account (or the `core` operator account once provisioned via PR #80):

```text
d /root 0700 root root -
d /root/.ssh 0700 root root -
f~ /root/.ssh/authorized_keys 0600 root root - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTEAAAA...
```
### SSH key provisioning

Or for `core` (requires the account to already exist in `/etc/passwd` via sysusers):
Deliver root's `authorized_keys` as the `ssh.authorized_keys.root` credential
(for example `/loader/credentials/ssh.authorized_keys.root.cred`); systemd's
stock `provision.conf` tmpfiles rule writes it to `/root/.ssh/authorized_keys`.
For other accounts, use a `tmpfiles.extra` credential; the payload must create
parent directories before writing the file:

```text
d /var/home/core 0700 core core -
d /var/home/core/.ssh 0700 core core -
f~ /var/home/core/.ssh/authorized_keys 0600 core core - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTEAAAA...
d /var/home/alice 0700 alice alice -
d /var/home/alice/.ssh 0700 alice alice -
f~ /var/home/alice/.ssh/authorized_keys 0600 alice alice - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTEAAAA...
```

Example static network credential name: `network.network.10-static`.
Expand Down
4 changes: 4 additions & 0 deletions elements/bluefin-server/os-creds-prov.bst
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@ description: |
Supports sysusers credentials, systemd-network-generator network.* credentials,
and the firstboot.* credentials (locale, keymap, timezone, hostname).

The systemd-firstboot.service drop-in removes upstream's interactive root
password prompt: root ships locked, so without it every headless boot would
wait on the console. The passwd.*.root credentials still apply.

Flatcar's /usr is built without systemd-firstboot, so the firstboot.*
credentials are applied by /usr/libexec/bluefin-firstboot-credentials, a
bash + coreutils helper that writes the same files systemd-firstboot would
Expand Down
3 changes: 2 additions & 1 deletion elements/bluefin-server/os-k0s-first-boot.bst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@ kind: import
description: |
Install the systemd units from files/os/systemd/system: the first-boot k0s
sysext activation units, first-start SSH host key generation
(ssh-host-keygen.service plus its sshd.service drop-in), the installer
(ssh-host-keygen.service plus its sshd.service and systemd-ssh-generator
sshd-generated@.service drop-ins), the installer
source ESP mounts (run-bluefin-boot.mount for netboot,
run-bluefin-installer.mount for the USB installer), and the
systemd-sysinstall.service drop-in the USB installer boots into.
Expand Down
9 changes: 6 additions & 3 deletions elements/oci/bluefin-server-usr.bst
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,13 @@ config:
set -eu
systemd-sysusers --root=/sysroot
chmod 0600 /sysroot/etc/shadow /sysroot/etc/gshadow
# Console bring-up password: root/bluefin, expired so the first login
# forces a rotation. sshd refuses passwords (os-sshd-config.bst).
# No shipped password: root is locked. "!unprovisioned" is systemd's
# marker for a root account nobody has configured yet; login, su and
# sulogin treat it as locked, and systemd-firstboot still applies a
# passwd.hashed-password.root / passwd.plaintext-password.root
# credential over it (docs/skills/tpm2-credential-sealing.md).
sed -i '/^root:/d' /sysroot/etc/shadow
printf 'root:%s:0:0:99999:7:::\n' '$6$bluefin123$o2pQjPN1n3ZCnYUROlA01xAdnGO7mhVRAVw98x4xc8LAUIJv7b44bBNoER6fM4P.rrj4ePYFfkx9WvZICU8nt/' >> /sysroot/etc/shadow
printf 'root:!unprovisioned:::::::\n' >> /sysroot/etc/shadow
grep -q '^root:' /sysroot/etc/passwd
chmod 0400 /sysroot/etc/shadow /sysroot/etc/gshadow

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Root ships locked as "!unprovisioned" (bluefin-server-usr.bst), which
# systemd-firstboot reads as "root not configured yet". Upstream's
# --prompt-root-password would then block every headless boot, and the USB
# installer's console, on a password prompt. Drop only that prompt: the
# passwd.hashed-password.root / passwd.plaintext-password.root credentials
# still apply, and without one root stays locked.
[Service]
ExecStart=
ExecStart=systemd-firstboot --prompt-locale --prompt-keymap-auto --prompt-timezone --mute-console=yes
2 changes: 1 addition & 1 deletion files/os/issue.d/30-bluefin.issue
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
Bluefin Server (\l)
Default login: root / bluefin (password change forced at first login)
No default login; root is locked. Provision access: https://github.com/projectbluefin/server/blob/main/docs/skills/tpm2-credential-sealing.md#node-access

10 changes: 5 additions & 5 deletions files/os/ssh/sshd_config.d/bluefin-server.conf
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# SSH configuration for Bluefin Server.
# Hardened defaults: no direct root password login, no password auth at all.
# Bring-up diagnostics stay available on the local console (root / bluefin,
# see the issue banner); for remote access inject an authorized_keys entry
# via the installer or systemd credentials. The seeded default password must
# never be usable over the network.
# Hardened defaults: key-only, no password auth at all. Root ships locked;
# provision an authorized key per node through Ignition or the
# ssh.authorized_keys.root credential (docs/skills/tpm2-credential-sealing.md).
# A console password set through a credential must never be usable over the
# network.

PermitRootLogin prohibit-password
PubkeyAuthentication yes
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# systemd-ssh-generator's per-connection sshd (behind the ssh.listen
# credential, AF_VSOCK and the local AF_UNIX socket) needs the same
# per-device host keys as sshd.service; without them it exits with
# "no hostkeys available". See ssh-host-keygen.service.
[Unit]
Wants=ssh-host-keygen.service
After=ssh-host-keygen.service
4 changes: 3 additions & 1 deletion files/os/sysusers.d/10-root-creds.conf
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
# Read root user hashed password from systemd credential storage
# root's account entry. Its password is not set here: the image ships root
# locked ("!unprovisioned", bluefin-server-usr.bst) and systemd-firstboot
# applies the passwd.*.root credentials (docs/skills/tpm2-credential-sealing.md).
u root 0 "root" /root /bin/bash
7 changes: 7 additions & 0 deletions scripts/dogfood-diskless.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
# --check headless: boot, run a probe, exit 0 if no unit failed
# Environment:
# DOGFOOD_IGNITION=<file> pass an Ignition config as the ignition.config credential
# DOGFOOD_CREDS=<dir> pass every file in <dir> as a system credential named after it
# DOGFOOD_STATE_DISK=<file> attach a persistent second disk (/dev/vdb), created if missing
# DOGFOOD_EXTRA_PROBE=<file> shell snippet appended to the in-guest probe
# DOGFOOD_VARS=<file> persistent UEFI variable store (keeps enrolled keys)
Expand Down Expand Up @@ -119,6 +120,11 @@ fi
if [ -n "${DOGFOOD_IGNITION:-}" ]; then
qemu+=(-smbios "$(cred ignition.config "${DOGFOOD_IGNITION}")")
fi
if [ -n "${DOGFOOD_CREDS:-}" ]; then
for f in "${DOGFOOD_CREDS}"/*; do
if [ -f "${f}" ]; then qemu+=(-smbios "$(cred "${f##*/}" "${f}")"); fi
done
fi

echo "Serving ${dir} on :${port}; ${boot} boot of ${ver} (Secure Boot)"
if [ "${mode}" != "--check" ]; then
Expand All @@ -134,6 +140,7 @@ echo "PROBE root=$(findmnt -no FSTYPE /)"
echo "PROBE verity=$(veritysetup status usr | sed -n 's/^ *status: *//p')"
echo "PROBE os=$(. /usr/lib/os-release; echo "${IMAGE_ID} ${IMAGE_VERSION}")"
echo "PROBE var=$(findmnt -no SOURCE,FSTYPE /var)"
echo "PROBE root-passwd=$(passwd -S root 2>&1 | cut -d' ' -f2)"
boots=$(( $(cat /var/lib/dogfood-boots 2>/dev/null || echo 0) + 1 ))
echo "${boots}" > /var/lib/dogfood-boots
echo "PROBE boots=${boots}"
Expand Down
1 change: 1 addition & 0 deletions scripts/dogfood-install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ cat > "${state}/disk.probe" <<'EOF'
echo "PROBE usr-part=$(lsblk -rsno PARTLABEL /dev/mapper/usr | grep bluefin_usr_ | tr '\n' ' ')"
echo "PROBE zfs=$(systemctl is-active zfs.target) $(ls /var/lib/extensions 2>/dev/null | tr '\n' ' ')"
echo "PROBE boot-entry=$(bootctl status 2>/dev/null | sed -n 's/^ *Current Entry: *//p' | head -n1)"
echo "PROBE firstboot-ran=$(systemctl show -P ConditionResult systemd-firstboot.service)"
bootctl list --no-pager 2>/dev/null | sed -n 's/^ *\(title\|id\): */PROBE-LOG \1 /p'
systemctl start boot-complete.target 2>/dev/null || true
echo "PROBE ukis=$(ls /boot/EFI/Linux 2>/dev/null | tr '\n' ' ')"
Expand Down
Loading