Skip to content

fix(security): persist the root credential across DDI slot replacements - #246

Closed
sramkrishna wants to merge 1 commit into
projectbluefin:mainfrom
sramkrishna:fix/root-cred-persist-across-updates
Closed

sramkrishna wants to merge 1 commit into
projectbluefin:mainfrom
sramkrishna:fix/root-cred-persist-across-updates

Conversation

@sramkrishna

Copy link
Copy Markdown

Finding

/etc/shadow ships inside the OS DDI (elements/oci/bluefin-server-ddi.bst:86 seeds it), and files/os/sysupdate.d/50-root.transfer copies that filesystem image block-for-block into both bluefin-server-root-a and bluefin-server-root-b on every update. Everything under /etc is therefore rebuilt from the new DDI on each A/B update — only var, the ESP and the bootloader state survive.

A root password rotated on the running host consequently reverted to the public bluefin hash baked into the image as soon as the other slot booted. Filed from the review of #242 as #244.

Fix

Keep the authoritative root entry outside the replaced filesystem — in the persistent var partition — and reconcile /etc/shadow with it:

Piece Role
files/os/libexec/bluefin-root-cred-sync boot restores the stored entry into /etc/shadow; persist copies the live entry into the store
bluefin-root-cred.service runs the boot restore Before=multi-user.target (covers getty, sshd and the k0s console), seeding the store from the image on the first boot after install
bluefin-root-cred-persist.path + .service PathChanged=/etc/shadow triggers persist, so passwd/chage rotations reach the store
elements/bluefin-server/os-root-cred.bst installs the helper and creates the enablement symlinks

Worth a reviewer's attention:

  • The persist service is deliberately not enabled at boot. Only the .path triggers it; running both halves at startup could persist the image's baked-in credential over the operator's.
  • Enablement is an explicit multi-user.target.wants symlink, because presets are not applied in this image — a preset alone would not establish the runtime enablement symlink. Same approach as os-k0s-first-boot.bst.
  • The helper is pure bash plus coreutils. The DDI runtime is runtime-minimal and guarantees neither /bin/sh nor sed/awk/grep, and this runs before login.
  • Fail loudly, never silently. A corrupt or malformed store makes the unit fail and leaves /etc/shadow untouched instead of quietly falling back to the public credential. The store is root-only: 0600 file, 0700 StateDirectory.
  • RequiresMountsFor=/var/lib/bluefin-server on both halves, so the store can never be written to an unmounted (and therefore replaceable) /var.
  • The DDI still bakes the bootstrap credential, and the baked SSH host keys are untouched. Removing those is the operator-login redesign in feat: CoreOS-compatible login hardening and core operator key provisioning #80 — out of scope here, and this change leaves that path working.

Tests

  • tests/unit/test_root_credential_persistence.py — 19 new tests: the helper driven against real shadow files (first-boot seeding, a rotated credential surviving a replaced image slot, root-entry-only replacement, idempotent restore, 0600/0700 modes, corrupt/rootless/truncated store refused, unusable invocation rejected), plus contracts for the restore ordering, the store location, the watcher's scope, and enablement — including that every enabled name actually ships, so a dangling symlink cannot silently disable the restore.
  • Full suite: 352 passed, 1 xfailed.
  • python3 .github/scripts/docs-checks.py — passed.
  • shellcheck -s bash files/os/libexec/bluefin-root-cred-sync — clean.
  • The element's install-commands were run into a scratch install-root to confirm the helper lands at /usr/libexec/bluefin-root-cred-sync mode 0755 and both symlinks resolve.
  • just validate needs the FSDK bst2 container (podman + sudo), which is unavailable here — please let CI cover graph resolution.

docs: docs/skills/ddi-installer.md gains a "State Across Updates" section documenting what survives an update and how to reset the credential.

Fixes #244

🤖 Generated with Claude Code

— hive: backend=litellm model=deepseek/deepseek-v4.1-flash

🐝 Hive Agent: contributor | SHA: 0a4bb5e

/etc/shadow ships inside the OS DDI, and 50-root.transfer replaces that
filesystem image in both bluefin-server-root-* slots on every update. A root
password rotated on the running host therefore rolled back to the credential
baked into the image as soon as the other slot booted — the finding recorded
in projectbluefin#244.

Keep the authoritative root entry in the persistent /var partition and
reconcile /etc/shadow with it on every boot:

- bluefin-root-cred.service restores the persisted entry into /etc/shadow
  before multi-user.target, which covers getty, sshd and the k0s console, and
  seeds the persisted copy from the image on the first boot after install.
- bluefin-root-cred-persist.path copies /etc/shadow back into
  /var/lib/bluefin-server/root-shadow whenever it changes, so a password
  rotated with passwd or chage is what the next slot boots with. Its service
  is deliberately not enabled at boot: starting it there could persist the
  image's credential over the operator's.
- Both halves run against a root-only store (0600 file, 0700 StateDirectory)
  and fail loudly on a corrupt store rather than silently falling back to the
  public password.

The helper is pure bash plus coreutils, because the DDI runtime is
runtime-minimal and guarantees neither /bin/sh nor sed/awk/grep. Enablement is
an explicit multi-user.target.wants symlink: systemd presets are not applied
in this image, so a preset alone would not establish the runtime enablement
symlink.

The DDI keeps baking the bootstrap credential; removing it entirely is the
operator-login redesign in projectbluefin#80.

Tested: pytest tests/unit/test_root_credential_persistence.py (19 tests over
the helper against real shadow files, the restore ordering, and enablement),
the full tests/unit suite (352 passed, 1 xfailed),
python3 .github/scripts/docs-checks.py, shellcheck on the helper, and a
simulated run of the element's install-commands into a scratch install-root.
`just validate` needs the FSDK bst2 container, so it did not run here.

Assisted-by: litellm/deepseek-v4.1-flash
Hive-Run: projectbluefin#244
Hive-Plan: root credential persistence across DDI slot replacement
Hive-Spec: projectbluefin#244
Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: sramkrishna <sramkrishna@users.noreply.github.com>

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The helper itself is carefully done (atomic temp-file + rename, 0600/0700, fails closed on a malformed store, good test coverage). A few things need addressing before it can land:

  1. The public default becomes permanent. On first boot, boot() in files/os/libexec/bluefin-root-cred-sync finds no store and runs persist, so the image's hash is written to /var/lib/bluefin-server/root-shadow. If the operator never rotates, the public bluefin hash (lastchange 0) becomes the "authoritative" credential and gets restored over any future image that locks or removes root. Please only persist after an operator change, e.g. skip seeding when the entry equals the image's entry, or seed a sentinel instead.
  2. Login race. bluefin-root-cred.service only has Before=multi-user.target. getty@.service and sshd.service are pulled in by multi-user.target but aren't ordered after this unit, so a login can happen before the restore runs. Before=systemd-user-sessions.service (getty and sshd are ordered after it) or explicit Before=getty@.service sshd.service would close this.
  3. This and #80 are mutually exclusive. #80 removes the root password entirely in favour of a key-only core user, while this PR makes the root password persist across slot replacements. Both can't land as-is; maintainers need to pick one direction first.
  4. Minor: the note that "presets are not applied in this image" is out of date after #243 (the explicit .wants symlink in os-root-cred.bst is still fine). Several new files lack a trailing newline, and the test fixture uses lastchange 19700 while main now seeds 0.

installer-test is red, but the branch predates #243 (main is green at c78a9a5), so a rebase should sort that out.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 0a4bb5e (diff + bluefin-server-ddi.bst, os-k0s-first-boot.bst, var.mount, sshd.service.d, 50-root.transfer; 19 new tests run locally, pass).

correctness

  • Before=multi-user.target does not order the restore before sshd/getty — the claimed guarantee is not implemented. files/os/systemd/system/bluefin-root-cred.service:20 only orders the unit before the target. sshd.service, getty@.service and the k0s console units are all WantedBy=multi-user.target, so systemd starts them in parallel with bluefin-root-cred.service; the target itself just waits for all of them. Every boot after a rotation therefore has a window where sshd/login accept the public bluefin hash before boot() rewrites /etc/shadow. The PR body, bluefin-root-cred.service:8, bluefin-root-cred-sync:15 and docs/skills/ddi-installer.md:122 all state the opposite, and tests/unit/test_root_credential_persistence.py:168 pins the wrong invariant. Fix: order against the login gates directly — e.g. Before=systemd-user-sessions.service sshd.service getty.target (or DefaultDependencies=no + Before=sysinit.target, keeping RequiresMountsFor=), or use the repo's existing sshd.service.d/ drop-in idiom (Wants=/After=bluefin-root-cred.service) as done for ssh-host-keygen.service. Update the test to assert the real ordering.
  • (low) set_user_entry (bluefin-root-cred-sync:75-87) rewrites /etc/shadow without taking the shadow-utils lock (/etc/.pwd.lock); a concurrent passwd/chage could lose one side's write. Unlikely at boot; noting so it isn't assumed safe if boot is ever re-run at runtime.

docs-currency

  • (low) elements/bluefin-server/os-k0s-first-boot.bst:3-5 description enumerates the units it stages; it now also ships the three bluefin-root-cred* units and should say so, since os-root-cred.bst relies on that element for staging.

No findings from: security, intent-alignment, style.

Confidence: 3/5 (needs attention) — 1 high finding

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

jeefy added a commit that referenced this pull request Sep 26, 2026
…254)

## Problem

On the installed image `bluefin-firstboot-credentials.service` fails on every boot that carries a `firstboot.*` credential:

```text
bluefin-firstboot-credentials.service: Failed at step EXEC spawning systemd-firstboot: No such file or directory
```

so `firstboot.locale`, `firstboot.locale-messages`, `firstboot.keymap`, `firstboot.timezone` and `firstboot.hostname` are silent no-ops — including the ones `systemd-sysinstall` itself propagates from the installer environment (locale/keymap/timezone, see the "Propagates installer environment settings" bullet in `ddi-installer.md`). Observed in QEMU on the installed `installer-v26.08.0` image with credentials staged on the ESP (`Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, …` followed by the EXEC failure). Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which delivers `firstboot.hostname` per host.

## Root cause

The unit runs `ExecStart=systemd-firstboot --force --welcome=no`, written when the OS payload was FSDK's systemd. #132 cut the payload over to Flatcar's `/usr` (`flatcar-usr.bst`), and **Flatcar builds systemd without `systemd-firstboot`**: `flatcar-container.tar.gz` for 4593.2.5 contains neither `/usr/bin/systemd-firstboot` nor `systemd-firstboot.service` (verified by listing the archive; the `ln -sf /dev/null …/systemd-firstboot.service` mask in `bluefin-server-ddi.bst` is therefore a no-op too, harmless). `installer-stack.bst` still has FSDK's systemd with `systemd-firstboot`, which is why the *installer* side works and the *target* side does not.

## Fix

Replace the binary with `/usr/libexec/bluefin-firstboot-credentials`, a bash + coreutils helper (both shipped by Flatcar), staged by `os-creds-prov.bst`. It reads the five credentials from `$CREDENTIALS_DIRECTORY`, validates them at least as strictly as systemd's validators (`hostname_is_valid`-style: labels `[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?`, no leading/trailing hyphen, ≤ 64 chars total; timezone must match `[A-Za-z0-9_+-]+(/…)*` **and** exist under `/usr/share/zoneinfo`; locale/keymap restricted to safe characters and < 128 bytes like `locale_is_valid()`/`keymap_is_valid()`; everything ASCII-only under `export LC_ALL=C`, since PID 1 exports the `LANG` this helper itself writes and `[[:alnum:]]` would otherwise widen with it; a NUL byte in a credential is rejected rather than silently dropped by `$(<file)`), and writes the same files `systemd-firstboot` would, in the same format — plus `/etc/hostname`. On `firstboot.hostname`: systemd ≤ v259 (Flatcar 4593.2.5 ships 257.9) has no such credential; systemd v260 added it as a *static* hostname honoured only when none is set yet, whereas this helper applies it unconditionally and additionally sets the live kernel hostname (systemd's own transient hostname credential remains `system.hostname`, consumed by PID 1):

| credential | file |
|---|---|
| `firstboot.locale` / `firstboot.locale-messages` | `/etc/locale.conf` (`LANG=` / `LC_MESSAGES=`) |
| `firstboot.keymap` | `/etc/vconsole.conf` (`KEYMAP=`) |
| `firstboot.timezone` | `/etc/localtime -> ../usr/share/zoneinfo/<tz>` |
| `firstboot.hostname` | `/etc/hostname` + live kernel hostname (`/proc/sys/kernel/hostname`) so the first DHCP request carries it — new relative to `main` (the old `ExecStartPost` set the kernel hostname but never persisted `/etc/hostname`); semantics differ from systemd v260's (always applied, not only when no static hostname exists) |

Any invalid value fails the run **before** anything is written (unit fails visibly, stamp file not created; control characters in the rejected value are `%q`-quoted in the log). The helper `sync`s before exiting so the `ExecStartPost` stamp cannot outlive the files on power loss. The unit's ordering, conditions, `ImportCredential=` lines and the `/etc/.bluefin-firstboot-credentials` stamp are unchanged; the old `ExecStartPost` hostname one-liner folds into the helper. Shebang is `/usr/bin/bash` so EXEC does not depend on the tmpfiles-created `/bin` symlink. `--root=DIR` exists for tests only.

Known, deliberate differences from `systemd-firstboot` (stated in the helper's header): trailing newlines in a credential are stripped rather than rejected; `LC_MESSAGES=` is written even when it equals `LANG=`; locales/keymaps are shape-checked, not checked against what is installed (Flatcar ships the full zoneinfo, so the timezone existence check is real).

### Why not ship the binary

Pulling FSDK's `systemd-firstboot` into the Flatcar `/usr` was the other option and was rejected: it links `libsystemd-shared-<ver>.so` from a different systemd build, so it would drag a second libsystemd-shared into the image and reintroduce exactly the FSDK/Flatcar userspace split #132 removed. The helper is ~100 lines with no dependencies beyond what the payload already has, and follows the precedent of #246's `files/os/libexec/bluefin-root-cred-sync`.

## Verification

- `tests/unit/test_credentials_provisioning.py` — unit no longer names `systemd-firstboot`, `ExecStart` points at the helper and precedes the stamp; element stages `files/os/libexec` and installs the helper 0755.
- `tests/unit/test_firstboot_credentials_behavior.py` (new, 26 tests) — executes the helper against a scratch `--root`: all five credentials produce the exact files above; a subset writes only its files; `--force`-style overwrite of existing `/etc/hostname` and `/etc/localtime`; interior hyphens/digits in labels accepted; twelve invalid inputs (`bad host`, `-leading`, `trailing-`, `foo-.bar`, `foo..bar`, `hôte`, `host\r`, 65-char hostname, `../../etc/passwd` and unknown timezone, shell metacharacters in locale, `..` in keymap) exit 1 with a named error and write **nothing**; a non-ASCII **keymap/locale** (`[[:alnum:]]` regexes) is rejected with `LC_ALL=C.UTF-8`/`LANG=C.UTF-8` in the environment — this test **fails when `export LC_ALL=C` is removed from the helper** (verified by mutation; the round-1 hostname variant did not, because `[A-Za-z0-9]` does not widen under `C.UTF-8`); 127-byte keymap accepted / 128 rejected; locale-only and messages-only `locale.conf`; invalid `locale-messages`; a NUL byte; the CR case logs `$'host\r'`; empty credential treated as absent; missing and nonexistent `CREDENTIALS_DIRECTORY` are errors; `bash -n`; `shellcheck -S warning` (skipped visibly, not silently passed, when shellcheck is absent — CI's `unit-tests.yml` does not install it).
- `python3 -m pytest tests/unit -q` — 371 passed, 1 xfailed (pre-existing). `test_files_payload_reachability` confirms the new `files/os/libexec/` path is staged.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- Not run: `just validate` / `just build-ddi` (no local BuildStream) and a rebuilt-image boot. The helper has only been exercised by the executed unit tests against a scratch `--root`; a first boot with `firstboot.*` credentials on a rebuilt DDI is the remaining proof and I am happy to run it against a CI artefact.

Docs: `ddi-installer.md` describes what the unit actually runs and writes (same files as `systemd-firstboot` plus the hostname handling, with the ≤ v259 / v260 distinction); `tpm2-credential-sealing.md` drops `systemd-firstboot` as a consumer of `passwd.hashed-password.root`.

No relation to the installer PR stack (#250, #251, #252, #253); no textual or compose overlap with #246 (`bluefin-root-cred-sync` is installed from a separate element).
@castrojo

Copy link
Copy Markdown
Contributor

Status: blocked on a maintainer decision. hanthor's point 3 stands: this PR and #80 (key-only core user, root password login removed) are mutually exclusive, so a maintainer has to pick the direction first. If this direction wins, the other CR items remain for the author: persist only after an operator change (not the image's public hash on first boot), order the restore Before=systemd-user-sessions.service so getty/sshd cannot race it, refresh the #243 preset note, trailing newlines and the lastchange fixture, then rebase onto current main for a real installer-test result.

@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Status note: after #262 the root filesystem on installed nodes is a persistent xfs partition that systemd-sysupdate never replaces (only the /usr and verity slots and the UKI are A/B), so /etc/shadow no longer reverts on update. The baked public hash is being removed in a follow-up PR that ships root locked (it will reference #244). This PR's persistence service may no longer be needed; leaving the call to you.

@castrojo

Copy link
Copy Markdown
Contributor

Closing: architecture superseded by PR #262 (FSDK-composed diskless/UKI OS model). With root filesystem managed via FSDK/UKI rather than sysupdate A/B slots, root credential persistence across slot replacement is no longer applicable.

@castrojo castrojo closed this Sep 29, 2026
@castrojo

Copy link
Copy Markdown
Contributor

Reopening: A/B slot replacement remains part of server architecture. Branch is conflicting and has open change requests regarding credential handling and service ordering.

@castrojo castrojo reopened this Sep 29, 2026
@castrojo

Copy link
Copy Markdown
Contributor

Superseded by PR #262 and PR #275. Under the FSDK architecture, installed nodes use a persistent xfs root partition that systemd-sysupdate does not replace on slot updates, and root credential management is handled via systemd-creds and systemd-firstboot.

@castrojo castrojo closed this Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review backlog (security): Seeded root hash is re-baked into every root slot on sysupdate

4 participants