Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server DDI live installe
metadata:
type: reference
status: stable
last_updated: "2026-09-09"
last_updated: "2026-09-24"
context7-sources:
- /systemd/systemd
- /apache/buildstream
Expand Down Expand Up @@ -108,6 +108,28 @@ container runtime pod sandboxes.
| `bluefin-server-root-a` | XFS | 4 GiB – 8 GiB | OS root filesystem (copied from installer data partition) |
| `var` | XFS | ≥ 4 GiB | Writable persistent `/var`; grows to fill remaining disk |

## State Across Updates

`systemd-sysupdate` replaces the root partition block-for-block in each A/B slot,
so everything under `/etc` is re-created from the new DDI on every update. Only
the `var` partition, the ESP, and the bootloader state survive.

The root credential is the one piece of `/etc` an operator changes at runtime,
so the image keeps the authoritative root entry of `/etc/shadow` in
`/var/lib/bluefin-server/root-shadow`:

- `bluefin-root-cred.service` restores the persisted entry into `/etc/shadow`
before `multi-user.target` — and therefore before `getty`, `sshd`, and the k0s
console — seeding the persisted copy from the image on the first boot after
install.
- `bluefin-root-cred-persist.path` copies `/etc/shadow` back out whenever it
changes, so a password rotated with `passwd` or `chage` is what the next slot
boots with.

Removing the persisted copy and rebooting falls back to the credential baked
into the image. Removing that baked credential entirely is the operator-login
redesign in [`#80`](https://github.com/projectbluefin/server/issues/80).

## Installer Boot Flow

```text
Expand Down
42 changes: 42 additions & 0 deletions elements/bluefin-server/os-root-cred.bst
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
kind: manual
description: |
Install the root credential persistence helper and enable its units.

/etc/shadow lives inside the OS DDI, which systemd-sysupdate replaces in both
root slots on every update. A root password rotated on the running host would
otherwise roll back to the credential baked into the image as soon as the
other slot boots, so this image keeps the authoritative root entry in the
persistent /var partition and reconciles /etc/shadow with it on every boot.

The units themselves are staged by bluefin-server/os-k0s-first-boot.bst, which
installs files/os/systemd/system into /usr/lib/systemd/system. Enablement is an
explicit multi-user.target.wants symlink created here: systemd presets are not
applied in this image, so a preset would not enable anything.

bluefin-root-cred-persist.service is intentionally left unenabled — it is the
triggered half of bluefin-root-cred-persist.path, and starting it at boot
could persist the image's baked-in credential over the operator's.

build-depends:
- base/base-stack.bst

sources:
- kind: local
path: files/os/libexec
directory: libexec-src

# Disable binary stripping; this element only installs a shell script and the
# runtime-minimal sandbox does not include the stripper toolchain.
variables:
strip-binaries: ""

config:
install-commands:
- install -Dm755 libexec-src/bluefin-root-cred-sync
"%{install-root}%{prefix}/libexec/bluefin-root-cred-sync"
- |
set -euo pipefail
wants="%{install-root}%{prefix}/lib/systemd/system/multi-user.target.wants"
install -d -m 0755 "${wants}"
ln -sf ../bluefin-root-cred.service "${wants}/bluefin-root-cred.service"
ln -sf ../bluefin-root-cred-persist.path "${wants}/bluefin-root-cred-persist.path"
3 changes: 3 additions & 0 deletions elements/bluefin-server/os-stack.bst
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ depends:
# A/B System Update Configurations
- bluefin-server/os-sysupdate.bst

# Root credential persistence across A/B DDI replacements
- bluefin-server/os-root-cred.bst

# k0s component system update configuration
- bluefin-server/os-k0s-sysupdate.bst

Expand Down
137 changes: 137 additions & 0 deletions files/os/libexec/bluefin-root-cred-sync
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
#!/bin/bash
# bluefin-root-cred-sync — keep the root credential outside the replaceable DDI.
#
# /etc/shadow ships inside the OS DDI, and systemd-sysupdate replaces that whole
# filesystem image in both root slots on every update. A root password rotated
# on the running host therefore rolls back to the credential baked into the
# image as soon as the other slot boots. This helper keeps the authoritative
# root entry in the persistent /var partition instead, and reconciles the image
# with it:
#
# boot restore the persisted root entry into /etc/shadow, seeding the
# persisted copy from the image on the first boot after install
# persist copy the current /etc/shadow root entry into the persisted copy
#
# bluefin-root-cred.service runs `boot` before every login path;
# bluefin-root-cred-persist.path runs `persist` whenever /etc/shadow changes.
#
# Only the root entry is kept, never the whole file: every other shadow entry is
# an image-managed system account that must track the image, not the host.
#
# Recovery: remove the persisted copy and reboot to fall back to the credential
# baked into the image (the documented console bring-up credential).
#
# Deliberately pure bash builtins plus coreutils. The DDI runtime is
# runtime-minimal: /bin/sh, sed, awk and grep are not guaranteed to exist, and
# this code runs before the login services it protects.
#
# Usage: bluefin-root-cred-sync {boot|persist} [shadow-file] [state-file]
# The two paths are overridable for tests only; the shipped units pass neither.

set -euo pipefail
umask 077

PROG="${0##*/}"
MODE="${1:-}"
SHADOW="${2:-/etc/shadow}"
STATE="${3:-/var/lib/bluefin-server/root-shadow}"

log() { printf '%s: %s\n' "$PROG" "$*" >&2; }
die() { log "$*"; exit 1; }

# Print the entry for a user out of a colon-separated shadow database.
user_entry() { # $1 = file, $2 = user
local line
while IFS= read -r line || [ -n "$line" ]; do
if [ "${line%%:*}" = "$2" ]; then
printf '%s\n' "$line"
return 0
fi
done < "$1"
return 1
}

# A shadow entry is name:password:lastchange:min:max:warn:inactive:expire:reserved.
# Refuse anything else: a truncated or half-written entry must never replace
# either copy of the credential.
shadow_entry_is_sane() { # $1 = entry
local entry="$1" rest="$1" fields=1 hash
while [ "${rest#*:}" != "$rest" ]; do
fields=$((fields + 1))
rest="${rest#*:}"
done
[ "$fields" -eq 9 ] || return 1
hash="${entry#*:}"
hash="${hash%%:*}"
[ -n "$hash" ] || return 1
return 0
}

# Replace the entry for a user in a shadow database, or append it when absent.
# Writes next to the target and renames, so the database is never truncated,
# and re-asserts 0600 on the result.
set_user_entry() { # $1 = file, $2 = user, $3 = entry
local file="$1" user="$2" entry="$3" tmp line replaced=0
tmp="$(mktemp "${file}.XXXXXX")" ||
die "cannot create a temporary file next to ${file}"
while IFS= read -r line || [ -n "$line" ]; do
if [ "${line%%:*}" = "$user" ]; then
printf '%s\n' "$entry" >>"$tmp"
replaced=1
else
printf '%s\n' "$line" >>"$tmp"
fi
done <"$file"
[ "$replaced" -eq 1 ] || printf '%s\n' "$entry" >>"$tmp"
chmod 0600 "$tmp"
mv -f "$tmp" "$file"
}

# Copy the live root entry into the persistent store.
persist() {
local entry tmp dir
entry="$(user_entry "$SHADOW" root)" ||
die "no root entry in ${SHADOW}; refusing to persist"
shadow_entry_is_sane "$entry" ||
die "root entry in ${SHADOW} is malformed; refusing to persist"
case "$STATE" in
*/*) dir="${STATE%/*}" ;;
*) dir="." ;;
esac
[ -d "$dir" ] || mkdir -p "$dir" || die "cannot create ${dir}"
tmp="$(mktemp "${STATE}.XXXXXX")" ||
die "cannot create a temporary file next to ${STATE}"
printf '%s\n' "$entry" >"$tmp"
chmod 0600 "$tmp"
mv -f "$tmp" "$STATE"
log "persisted the root entry from ${SHADOW} in ${STATE}"
}

# Make the persisted root entry authoritative for this boot.
boot() {
local entry current=""
if [ ! -e "$STATE" ]; then
log "no persisted root credential in ${STATE}; seeding it from ${SHADOW}"
persist
return 0
fi
entry="$(user_entry "$STATE" root)" ||
die "${STATE} holds no root entry; leaving ${SHADOW} untouched"
shadow_entry_is_sane "$entry" ||
die "${STATE} holds a malformed root entry; leaving ${SHADOW} untouched"
current="$(user_entry "$SHADOW" root)" || current=""
if [ "$current" = "$entry" ]; then
log "root entry in ${SHADOW} already matches ${STATE}"
return 0
fi
set_user_entry "$SHADOW" root "$entry"
log "restored the persisted root entry into ${SHADOW}"
}

case "$MODE" in
boot) boot ;;
persist) persist ;;
*)
die "usage: ${PROG} {boot|persist} [shadow-file] [state-file]"
;;
esac
22 changes: 22 additions & 0 deletions files/os/systemd/system/bluefin-root-cred-persist.path
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# Persist root credential changes made on the running host.
#
# Changing the root password (passwd, chage) rewrites /etc/shadow in the root
# slot the host is currently booted from, and systemd-sysupdate replaces that
# slot wholesale on the next update. Watching /etc/shadow keeps the persisted
# copy in the persistent /var partition in step with the host, so the slot that
# boots next gets the operator's credential instead of the image's.
#
# Enablement is the explicit multi-user.target.wants symlink created by
# elements/bluefin-server/os-root-cred.bst; systemd presets are not applied in
# this image.
[Unit]
Description=Persist root credential changes to the /var root credential store
# Arm the watch only once the boot-time restore has settled /etc/shadow.
After=bluefin-root-cred.service

[Path]
PathChanged=/etc/shadow
Unit=bluefin-root-cred-persist.service

[Install]
WantedBy=multi-user.target
16 changes: 16 additions & 0 deletions files/os/systemd/system/bluefin-root-cred-persist.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Copy the live root credential out of /etc/shadow into the persistent store.
#
# Triggered by bluefin-root-cred-persist.path whenever /etc/shadow changes.
# Deliberately not enabled at boot: bluefin-root-cred.service owns the boot-time
# direction, and running both concurrently could persist the image's baked-in
# credential over the operator's.
[Unit]
Description=Persist the /etc/shadow root credential into /var
RequiresMountsFor=/var/lib/bluefin-server
ConditionPathExists=/etc/shadow

[Service]
Type=oneshot
StateDirectory=bluefin-server
StateDirectoryMode=0700
ExecStart=/usr/libexec/bluefin-root-cred-sync persist
31 changes: 31 additions & 0 deletions files/os/systemd/system/bluefin-root-cred.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Restore the authoritative root credential into /etc/shadow.
#
# /etc/shadow ships inside the OS DDI, and systemd-sysupdate replaces that
# filesystem image in both root slots on every update. Without this unit a root
# password rotated on the running host would roll back to the credential baked
# into the image as soon as the other slot boots. bluefin-root-cred-sync keeps
# the authoritative root entry in the persistent /var partition and reconciles
# the image with it here, before any process can authenticate.
#
# Enablement is the explicit multi-user.target.wants symlink created by
# elements/bluefin-server/os-root-cred.bst: systemd presets are not applied in
# this image, so a preset would enable nothing.
[Unit]
Description=Restore the persisted root credential into /etc/shadow
Documentation=man:shadow(5)
# /var must be mounted, and every login path (getty, sshd, k0s console) hangs
# off multi-user.target.
RequiresMountsFor=/var/lib/bluefin-server
After=local-fs.target
Before=multi-user.target
ConditionPathExists=/etc/shadow

[Service]
Type=oneshot
RemainAfterExit=yes
StateDirectory=bluefin-server
StateDirectoryMode=0700
ExecStart=/usr/libexec/bluefin-root-cred-sync boot

[Install]
WantedBy=multi-user.target
Loading
Loading