Repository navigation
fix(security): persist the root credential across DDI slot replacements - #246
sramkrishna wants to merge 1 commit into
Conversation
/etc/shadow ships inside the OS DDI, and 50-root.transfer replaces that filesystem image in both bluefin-server-root-* slots on every update. A root password rotated on the running host therefore rolled back to the credential baked into the image as soon as the other slot booted — the finding recorded in projectbluefin#244. Keep the authoritative root entry in the persistent /var partition and reconcile /etc/shadow with it on every boot: - bluefin-root-cred.service restores the persisted entry into /etc/shadow before multi-user.target, which covers getty, sshd and the k0s console, and seeds the persisted copy from the image on the first boot after install. - bluefin-root-cred-persist.path copies /etc/shadow back into /var/lib/bluefin-server/root-shadow whenever it changes, so a password rotated with passwd or chage is what the next slot boots with. Its service is deliberately not enabled at boot: starting it there could persist the image's credential over the operator's. - Both halves run against a root-only store (0600 file, 0700 StateDirectory) and fail loudly on a corrupt store rather than silently falling back to the public password. The helper is pure bash plus coreutils, because the DDI runtime is runtime-minimal and guarantees neither /bin/sh nor sed/awk/grep. Enablement is an explicit multi-user.target.wants symlink: systemd presets are not applied in this image, so a preset alone would not establish the runtime enablement symlink. The DDI keeps baking the bootstrap credential; removing it entirely is the operator-login redesign in projectbluefin#80. Tested: pytest tests/unit/test_root_credential_persistence.py (19 tests over the helper against real shadow files, the restore ordering, and enablement), the full tests/unit suite (352 passed, 1 xfailed), python3 .github/scripts/docs-checks.py, shellcheck on the helper, and a simulated run of the element's install-commands into a scratch install-root. `just validate` needs the FSDK bst2 container, so it did not run here. Assisted-by: litellm/deepseek-v4.1-flash Hive-Run: projectbluefin#244 Hive-Plan: root credential persistence across DDI slot replacement Hive-Spec: projectbluefin#244 Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: sramkrishna <sramkrishna@users.noreply.github.com>
hanthor
left a comment
There was a problem hiding this comment.
Thanks for this. The helper itself is carefully done (atomic temp-file + rename, 0600/0700, fails closed on a malformed store, good test coverage). A few things need addressing before it can land:
- The public default becomes permanent. On first boot,
boot()infiles/os/libexec/bluefin-root-cred-syncfinds no store and runspersist, so the image's hash is written to/var/lib/bluefin-server/root-shadow. If the operator never rotates, the publicbluefinhash (lastchange 0) becomes the "authoritative" credential and gets restored over any future image that locks or removes root. Please only persist after an operator change, e.g. skip seeding when the entry equals the image's entry, or seed a sentinel instead. - Login race.
bluefin-root-cred.serviceonly hasBefore=multi-user.target.getty@.serviceandsshd.serviceare pulled in by multi-user.target but aren't ordered after this unit, so a login can happen before the restore runs.Before=systemd-user-sessions.service(getty and sshd are ordered after it) or explicitBefore=getty@.service sshd.servicewould close this. - This and #80 are mutually exclusive. #80 removes the root password entirely in favour of a key-only
coreuser, while this PR makes the root password persist across slot replacements. Both can't land as-is; maintainers need to pick one direction first. - Minor: the note that "presets are not applied in this image" is out of date after #243 (the explicit
.wantssymlink inos-root-cred.bstis still fine). Several new files lack a trailing newline, and the test fixture uses lastchange19700while main now seeds0.
installer-test is red, but the branch predates #243 (main is green at c78a9a5), so a rebase should sort that out.
There was a problem hiding this comment.
Reviewed at head 0a4bb5e (diff + bluefin-server-ddi.bst, os-k0s-first-boot.bst, var.mount, sshd.service.d, 50-root.transfer; 19 new tests run locally, pass).
correctness
Before=multi-user.targetdoes not order the restore beforesshd/getty— the claimed guarantee is not implemented.files/os/systemd/system/bluefin-root-cred.service:20only orders the unit before the target.sshd.service,getty@.serviceand the k0s console units are allWantedBy=multi-user.target, so systemd starts them in parallel withbluefin-root-cred.service; the target itself just waits for all of them. Every boot after a rotation therefore has a window wheresshd/loginaccept the publicbluefinhash beforeboot()rewrites/etc/shadow. The PR body,bluefin-root-cred.service:8,bluefin-root-cred-sync:15anddocs/skills/ddi-installer.md:122all state the opposite, andtests/unit/test_root_credential_persistence.py:168pins the wrong invariant. Fix: order against the login gates directly — e.g.Before=systemd-user-sessions.service sshd.service getty.target(orDefaultDependencies=no+Before=sysinit.target, keepingRequiresMountsFor=), or use the repo's existingsshd.service.d/drop-in idiom (Wants=/After=bluefin-root-cred.service) as done forssh-host-keygen.service. Update the test to assert the real ordering.- (low)
set_user_entry(bluefin-root-cred-sync:75-87) rewrites/etc/shadowwithout taking the shadow-utils lock (/etc/.pwd.lock); a concurrentpasswd/chagecould lose one side's write. Unlikely at boot; noting so it isn't assumed safe ifbootis ever re-run at runtime.
docs-currency
- (low)
elements/bluefin-server/os-k0s-first-boot.bst:3-5description enumerates the units it stages; it now also ships the threebluefin-root-cred*units and should say so, sinceos-root-cred.bstrelies on that element for staging.
No findings from: security, intent-alignment, style.
Confidence: 3/5 (needs attention) — 1 high finding
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
…254) ## Problem On the installed image `bluefin-firstboot-credentials.service` fails on every boot that carries a `firstboot.*` credential: ```text bluefin-firstboot-credentials.service: Failed at step EXEC spawning systemd-firstboot: No such file or directory ``` so `firstboot.locale`, `firstboot.locale-messages`, `firstboot.keymap`, `firstboot.timezone` and `firstboot.hostname` are silent no-ops — including the ones `systemd-sysinstall` itself propagates from the installer environment (locale/keymap/timezone, see the "Propagates installer environment settings" bullet in `ddi-installer.md`). Observed in QEMU on the installed `installer-v26.08.0` image with credentials staged on the ESP (`Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, …` followed by the EXEC failure). Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which delivers `firstboot.hostname` per host. ## Root cause The unit runs `ExecStart=systemd-firstboot --force --welcome=no`, written when the OS payload was FSDK's systemd. #132 cut the payload over to Flatcar's `/usr` (`flatcar-usr.bst`), and **Flatcar builds systemd without `systemd-firstboot`**: `flatcar-container.tar.gz` for 4593.2.5 contains neither `/usr/bin/systemd-firstboot` nor `systemd-firstboot.service` (verified by listing the archive; the `ln -sf /dev/null …/systemd-firstboot.service` mask in `bluefin-server-ddi.bst` is therefore a no-op too, harmless). `installer-stack.bst` still has FSDK's systemd with `systemd-firstboot`, which is why the *installer* side works and the *target* side does not. ## Fix Replace the binary with `/usr/libexec/bluefin-firstboot-credentials`, a bash + coreutils helper (both shipped by Flatcar), staged by `os-creds-prov.bst`. It reads the five credentials from `$CREDENTIALS_DIRECTORY`, validates them at least as strictly as systemd's validators (`hostname_is_valid`-style: labels `[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?`, no leading/trailing hyphen, ≤ 64 chars total; timezone must match `[A-Za-z0-9_+-]+(/…)*` **and** exist under `/usr/share/zoneinfo`; locale/keymap restricted to safe characters and < 128 bytes like `locale_is_valid()`/`keymap_is_valid()`; everything ASCII-only under `export LC_ALL=C`, since PID 1 exports the `LANG` this helper itself writes and `[[:alnum:]]` would otherwise widen with it; a NUL byte in a credential is rejected rather than silently dropped by `$(<file)`), and writes the same files `systemd-firstboot` would, in the same format — plus `/etc/hostname`. On `firstboot.hostname`: systemd ≤ v259 (Flatcar 4593.2.5 ships 257.9) has no such credential; systemd v260 added it as a *static* hostname honoured only when none is set yet, whereas this helper applies it unconditionally and additionally sets the live kernel hostname (systemd's own transient hostname credential remains `system.hostname`, consumed by PID 1): | credential | file | |---|---| | `firstboot.locale` / `firstboot.locale-messages` | `/etc/locale.conf` (`LANG=` / `LC_MESSAGES=`) | | `firstboot.keymap` | `/etc/vconsole.conf` (`KEYMAP=`) | | `firstboot.timezone` | `/etc/localtime -> ../usr/share/zoneinfo/<tz>` | | `firstboot.hostname` | `/etc/hostname` + live kernel hostname (`/proc/sys/kernel/hostname`) so the first DHCP request carries it — new relative to `main` (the old `ExecStartPost` set the kernel hostname but never persisted `/etc/hostname`); semantics differ from systemd v260's (always applied, not only when no static hostname exists) | Any invalid value fails the run **before** anything is written (unit fails visibly, stamp file not created; control characters in the rejected value are `%q`-quoted in the log). The helper `sync`s before exiting so the `ExecStartPost` stamp cannot outlive the files on power loss. The unit's ordering, conditions, `ImportCredential=` lines and the `/etc/.bluefin-firstboot-credentials` stamp are unchanged; the old `ExecStartPost` hostname one-liner folds into the helper. Shebang is `/usr/bin/bash` so EXEC does not depend on the tmpfiles-created `/bin` symlink. `--root=DIR` exists for tests only. Known, deliberate differences from `systemd-firstboot` (stated in the helper's header): trailing newlines in a credential are stripped rather than rejected; `LC_MESSAGES=` is written even when it equals `LANG=`; locales/keymaps are shape-checked, not checked against what is installed (Flatcar ships the full zoneinfo, so the timezone existence check is real). ### Why not ship the binary Pulling FSDK's `systemd-firstboot` into the Flatcar `/usr` was the other option and was rejected: it links `libsystemd-shared-<ver>.so` from a different systemd build, so it would drag a second libsystemd-shared into the image and reintroduce exactly the FSDK/Flatcar userspace split #132 removed. The helper is ~100 lines with no dependencies beyond what the payload already has, and follows the precedent of #246's `files/os/libexec/bluefin-root-cred-sync`. ## Verification - `tests/unit/test_credentials_provisioning.py` — unit no longer names `systemd-firstboot`, `ExecStart` points at the helper and precedes the stamp; element stages `files/os/libexec` and installs the helper 0755. - `tests/unit/test_firstboot_credentials_behavior.py` (new, 26 tests) — executes the helper against a scratch `--root`: all five credentials produce the exact files above; a subset writes only its files; `--force`-style overwrite of existing `/etc/hostname` and `/etc/localtime`; interior hyphens/digits in labels accepted; twelve invalid inputs (`bad host`, `-leading`, `trailing-`, `foo-.bar`, `foo..bar`, `hôte`, `host\r`, 65-char hostname, `../../etc/passwd` and unknown timezone, shell metacharacters in locale, `..` in keymap) exit 1 with a named error and write **nothing**; a non-ASCII **keymap/locale** (`[[:alnum:]]` regexes) is rejected with `LC_ALL=C.UTF-8`/`LANG=C.UTF-8` in the environment — this test **fails when `export LC_ALL=C` is removed from the helper** (verified by mutation; the round-1 hostname variant did not, because `[A-Za-z0-9]` does not widen under `C.UTF-8`); 127-byte keymap accepted / 128 rejected; locale-only and messages-only `locale.conf`; invalid `locale-messages`; a NUL byte; the CR case logs `$'host\r'`; empty credential treated as absent; missing and nonexistent `CREDENTIALS_DIRECTORY` are errors; `bash -n`; `shellcheck -S warning` (skipped visibly, not silently passed, when shellcheck is absent — CI's `unit-tests.yml` does not install it). - `python3 -m pytest tests/unit -q` — 371 passed, 1 xfailed (pre-existing). `test_files_payload_reachability` confirms the new `files/os/libexec/` path is staged. - `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass. - Not run: `just validate` / `just build-ddi` (no local BuildStream) and a rebuilt-image boot. The helper has only been exercised by the executed unit tests against a scratch `--root`; a first boot with `firstboot.*` credentials on a rebuilt DDI is the remaining proof and I am happy to run it against a CI artefact. Docs: `ddi-installer.md` describes what the unit actually runs and writes (same files as `systemd-firstboot` plus the hostname handling, with the ≤ v259 / v260 distinction); `tpm2-credential-sealing.md` drops `systemd-firstboot` as a consumer of `passwd.hashed-password.root`. No relation to the installer PR stack (#250, #251, #252, #253); no textual or compose overlap with #246 (`bluefin-root-cred-sync` is installed from a separate element).
|
Status: blocked on a maintainer decision. hanthor's point 3 stands: this PR and #80 (key-only |
|
Status note: after #262 the root filesystem on installed nodes is a persistent xfs partition that systemd-sysupdate never replaces (only the /usr and verity slots and the UKI are A/B), so /etc/shadow no longer reverts on update. The baked public hash is being removed in a follow-up PR that ships root locked (it will reference #244). This PR's persistence service may no longer be needed; leaving the call to you. |
|
Closing: architecture superseded by PR #262 (FSDK-composed diskless/UKI OS model). With root filesystem managed via FSDK/UKI rather than sysupdate A/B slots, root credential persistence across slot replacement is no longer applicable. |
|
Reopening: A/B slot replacement remains part of server architecture. Branch is conflicting and has open change requests regarding credential handling and service ordering. |
Finding
/etc/shadowships inside the OS DDI (elements/oci/bluefin-server-ddi.bst:86seeds it), andfiles/os/sysupdate.d/50-root.transfercopies that filesystem image block-for-block into bothbluefin-server-root-aandbluefin-server-root-bon every update. Everything under/etcis therefore rebuilt from the new DDI on each A/B update — onlyvar, the ESP and the bootloader state survive.A root password rotated on the running host consequently reverted to the public
bluefinhash baked into the image as soon as the other slot booted. Filed from the review of #242 as #244.Fix
Keep the authoritative root entry outside the replaced filesystem — in the persistent
varpartition — and reconcile/etc/shadowwith it:files/os/libexec/bluefin-root-cred-syncbootrestores the stored entry into/etc/shadow;persistcopies the live entry into the storebluefin-root-cred.servicebootrestoreBefore=multi-user.target(coversgetty,sshdand the k0s console), seeding the store from the image on the first boot after installbluefin-root-cred-persist.path+.servicePathChanged=/etc/shadowtriggerspersist, sopasswd/chagerotations reach the storeelements/bluefin-server/os-root-cred.bstWorth a reviewer's attention:
.pathtriggers it; running both halves at startup could persist the image's baked-in credential over the operator's.multi-user.target.wantssymlink, because presets are not applied in this image — a preset alone would not establish the runtime enablement symlink. Same approach asos-k0s-first-boot.bst.runtime-minimaland guarantees neither/bin/shnorsed/awk/grep, and this runs before login./etc/shadowuntouched instead of quietly falling back to the public credential. The store is root-only: 0600 file, 0700StateDirectory.RequiresMountsFor=/var/lib/bluefin-serveron both halves, so the store can never be written to an unmounted (and therefore replaceable)/var.Tests
tests/unit/test_root_credential_persistence.py— 19 new tests: the helper driven against real shadow files (first-boot seeding, a rotated credential surviving a replaced image slot, root-entry-only replacement, idempotent restore, 0600/0700 modes, corrupt/rootless/truncated store refused, unusable invocation rejected), plus contracts for the restore ordering, the store location, the watcher's scope, and enablement — including that every enabled name actually ships, so a dangling symlink cannot silently disable the restore.352 passed, 1 xfailed.python3 .github/scripts/docs-checks.py— passed.shellcheck -s bash files/os/libexec/bluefin-root-cred-sync— clean.install-commandswere run into a scratch install-root to confirm the helper lands at/usr/libexec/bluefin-root-cred-syncmode 0755 and both symlinks resolve.just validateneeds the FSDKbst2container (podman + sudo), which is unavailable here — please let CI cover graph resolution.docs:
docs/skills/ddi-installer.mdgains a "State Across Updates" section documenting what survives an update and how to reset the credential.Fixes #244
🤖 Generated with Claude Code
— hive: backend=litellm model=deepseek/deepseek-v4.1-flash
🐝 Hive Agent:
contributor| SHA:0a4bb5e