Skip to content

fix(security): ship root locked; provision access per node - #275

Merged
jeefy merged 4 commits into
projectbluefin:mainfrom
jeefy:fix/lock-root-account
Sep 29, 2026
Merged

jeefy merged 4 commits into
projectbluefin:mainfrom
jeefy:fix/lock-root-account

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Motivation

oci/bluefin-server-usr.bst baked a SHA-512 hash of the password bluefin (salt bluefin123, in git) into the factory /etc/shadow for root, and 30-bluefin.issue printed Default login: root / bluefin on every console. Anyone with console, IPMI or serial access to any node, or a copy of the public DDI, knew root's password. SSH already refused passwords; the console did not.

Change

Root ships locked as root:!unprovisioned:::::::. I picked !unprovisioned over !/!* because it is the only locked value the stock tools handle correctly end to end (FSDK 26.08 ships systemd v261.2):

  • login, su and sulogin treat any leading ! as locked, and passwd -S root reports L.
  • systemd-sysusers writes the same value for a root account it creates without a credential ("let firstboot set the password later").
  • systemd-firstboot reads !unprovisioned as "root not configured", so it still applies passwd.hashed-password.root / passwd.plaintext-password.root. With !* it would treat root as configured and never apply them.
  • systemd-sysusers cannot apply those credentials here: it only sets passwords for accounts it creates, and root already exists in the factory /etc/shadow that the initrd copies in (bluefin-populate-etc.service). So the one documented console-password mechanism is systemd-firstboot, and it works the same way on diskless and installed nodes.

systemd-firstboot.service drop-in (files/os/creds/.../systemd-firstboot.service.d/10-bluefin-no-root-prompt.conf): upstream runs it with --prompt-root-password. Once root is unprovisioned, that prompt would block every headless boot, and the USB installer's console, until someone typed a password. The drop-in runs upstream's exact command minus that one flag. The credentials still apply, and without one root stays locked.

sshd-generated@.service drop-in: systemd-ssh-generator socket-activates a per-connection sshd for the ssh.listen credential, AF_VSOCK and the local AF_UNIX socket. That sshd started without host keys (no hostkeys available -- exiting), because ssh-host-keygen.service was only ordered before sshd.service. The drop-in pulls the same key generation in first, which makes ssh.listen a working credential-only way to open SSH on a locked node.

Banner: the default-login line is gone. In its place is one line: No default login; root is locked. Provision access: <link to the Node access section>.

Docs: the canonical place is the new "Node access" section in docs/skills/tpm2-credential-sealing.md, which already owns the credential table. It covers:

  • SSH keys through Ignition (Booty's path) or the ssh.authorized_keys.root credential
  • turning sshd on through Ignition or ssh.listen
  • the console password through passwd.*.root
  • when that applies on diskless and installed nodes
  • USB-installed nodes
  • emergency/rescue with a locked root

factory-integration.md no longer claims root password login over SSH is allowed (closes #245). The routing tables send "node access" to that skill. ddi-installer-build.md documents DOGFOOD_CREDS.

Emergency / rescue: with root locked, emergency.target/rescue.target on the booted system get no shell: sulogin prints Cannot open access to console, the root account is locked. (seen in QEMU, below). I deliberately do not set SYSTEMD_SULOGIN_FORCE, which would open a passwordless root shell on the console. It can't be set at boot either, because the kernel command line is sealed in the signed UKI. For break-glass console access, provision a password credential; sulogin then asks for it. The initrd already prints errors and reboots rather than offering a shell.

Harness: DOGFOOD_CREDS=<dir> passes each file as an SMBIOS type-11 credential named after the file. Every probe now reports root-passwd (passwd -S), and the installed-disk probe also reports whether systemd-firstboot ran.

Tests (tests/unit/test_root_credential_contract.py, replacing the old assertions on the hash and the banner). They check that:

  • root is seeded !unprovisioned
  • no crypt(5) hash ($1$ $2?$ $5$ $6$ $7$ $y$ $gy$ $md5$) appears anywhere under elements/, files/ or include/ (the pattern is self-tested)
  • the firstboot drop-in resets ExecStart and carries no root/--prompt flags, and is installed
  • the generated-sshd drop-in pulls host keygen
  • the banner has no password/login/root / text and links a heading that exists in the skill
  • the sshd policy is unchanged

Related

Verification

Gates on the final HEAD: python3 -m pytest tests/unit -q in a clean detached worktree with no boot keys → 376 passed. bats tests/unit → 74 ok. python3 .github/scripts/docs-checks.py → passed. just validate → passed.

QEMU/OVMF with Secure Boot enforcing, using scripts/dogfood-diskless.sh and dogfood-install.sh. Probes report on ttyS1. Images came from just export-image of this branch (26.09.2, plus 26.09.3 as NEXT).

Run Setup Result
A3 diskless, no credentials root-passwd=L, shadow-root=!unprovisioned, firstboot=success argv[]=systemd-firstboot --prompt-locale --prompt-keymap-auto --prompt-timezone --mute-console=yes, banner line as above, failed=0, PASS
A diskless, no credentials, sulogin on ttyS1 Cannot open access to console, the root account is locked. The probe then hung on sulogin's "Press Enter" and timed out; that's a probe artifact, and A3 is the clean rerun.
B2 diskless, DOGFOOD_CREDS: passwd.hashed-password.root (yescrypt), ssh.authorized_keys.root, ssh.listen=22 root-passwd=P, shadow-root-matches-cred=yes ($y$j9T$…), authorized_keys=… 600 root, ssh-key-login=ok-root over the ssh.listen socket, ssh-password-login=root@127.0.0.1: Permission denied (publickey)., failed=0, PASS
B same, before the sshd-generated@ drop-in password and key applied, but the generated sshd logged sshd: no hostkeys available -- exiting (this is why the drop-in exists)
C diskless, Ignition fixture tests/fixtures/ignition/var-on-disk.ign + state disk authorized_keys=ssh-ed25519 AAAA... dogfood 600 root, shadow-root=!unprovisioned, root-passwd=L, var=/dev/vdb1 xfs, failed=0, PASS
E just dogfood-install dist/diskless-next, no credentials install=0 → disk boot (firstboot-ran=yes, root-passwd=L) → update=0 → os=bluefin-server 26.09.3 from bluefin_usr_26.09.3 (firstboot-ran=no, root-passwd=L); PASS: installed, updated A->B and booted 26.09.3
F dogfood-install.sh with DOGFOOD_CREDS (password + key) disk boot firstboot-ran=yes, root-passwd=P → A/B to 26.09.3 slot B: firstboot-ran=no, root-passwd=P; PASS: installed, updated A->B and booted 26.09.3 (the password persists across A/B)
D0 / D2 USB installer bluefin-server-installer_26.09.2.raw, main vs this branch Both stop at the same systemd-firstboot keymap prompt on tty0 (the screendumps show the same screen). This is existing behaviour (--prompt-keymap-auto on a VT, and the factory /etc has no vconsole.conf); this PR doesn't change it.
D3 USB installer, this branch, Enter at the keymap prompt installer-firstboot=active success (no root password prompt), installer-sysinstall=active running with systemd-sysinstall --variables=yes --reboot=yes --mute-console=yes --definitions=/run/bluefin/installer/bluefin/repart.d --kernel=…/bluefin-server-26.09.2.efi, installer-shadow-root=!unprovisioned, installer-failed=0

E and F ran on a build from before the sshd-generated@ drop-in commit. That drop-in only touches the generator's per-connection sshd, not the install or update path. A3, B2 and D3 ran on the final tree.

Follow-ups (not in this PR)

  • On a VT console (the USB installer's tty0), systemd-firstboot asks for a keymap before systemd-sysinstall starts. That's existing behaviour, and a separate UX decision: ship vconsole.conf, or drop --prompt-keymap-auto.
  • USB-installed nodes on TPM2 + Secure Boot machines need credentials sealed to that machine's TPM, because null-key ESP credentials are refused there. Having systemd-sysinstall --load-credential= re-seal installer-side credentials, which fix(installer): boot offline, allow credential overrides, quiet repart warnings #268's drop-in rename makes possible, would remove the manual ESP step.
  • os-creds-prov.bst / bluefin-firstboot-credentials still say the /usr is "Flatcar's … built without systemd-firstboot". FSDK builds it (-Dfirstboot=true), and this PR relies on that. A cleanup could fold the firstboot.* helper into stock systemd-firstboot.
  • AGENTS.md shows just dogfood-install NEXT=<dir>, but just takes the recipe argument positionally (just dogfood-install <dir>).

jeefy and others added 4 commits September 28, 2026 22:00
The /usr image baked a SHA-512 hash of the public password "bluefin" into
the factory /etc/shadow and the console banner advertised it, so anyone
with console, IPMI or serial access to any node, or a copy of the public
DDI, knew root's password.

Seed root as "!unprovisioned" instead: login, su and sulogin treat it as
locked, and systemd-firstboot treats it as "root not configured", so a
passwd.hashed-password.root / passwd.plaintext-password.root credential
still sets it (systemd-sysusers cannot: it only sets passwords for accounts
it creates). Drop upstream's --prompt-root-password from
systemd-firstboot.service so a node without the credential, and the USB
installer's console, boot unattended instead of waiting on a prompt.

Replace the banner's default-login line with a pointer to the node access
docs, and pin the contract in tests: root ships !unprovisioned, no crypt(5)
hash appears under elements/, files/ or include/, firstboot never prompts
for a root password, and the banner advertises no credentials.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-ssh-generator socket-activates a per-connection sshd for the
ssh.listen credential, AF_VSOCK and the local AF_UNIX socket. It ran
without host keys, which ssh-host-keygen.service only generates before
sshd.service, so every connection ended in "no hostkeys available".
Pull the same one-shot key generation in before it, which makes
ssh.listen a working credential-only way to open SSH on a node whose
root is locked.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a "Node access" section to the credentials skill: root ships locked,
SSH keys come from Ignition or the ssh.authorized_keys.root credential,
the console password from passwd.*.root credentials applied by
systemd-firstboot, when that applies on diskless and installed nodes (and
why it survives A/B updates), how USB-installed nodes get credentials, and
what a locked root means for emergency/rescue (no SYSTEMD_SULOGIN_FORCE).

Fix factory-integration.md, which still claimed root password login over
SSH is permitted (projectbluefin#245), and route "node access" to the skill.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tate

DOGFOOD_CREDS=<dir> hands every file in <dir> to the guest as an SMBIOS
type 11 credential named after the file, so provisioning credentials
(passwd.*.root, ssh.authorized_keys.root, ...) can be dogfooded without
editing the harness. Every probe now reports root's `passwd -S` status,
and the installed-disk probe whether systemd-firstboot ran.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy merged commit 8824f9e into projectbluefin:main Sep 29, 2026
5 checks passed
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
Brings in the locked-root provisioning (projectbluefin#275) and binary patch tracking.
The only conflicts were the last_updated dates of ddi-installer-build.md
and kubeadm-sysext.md; both sides' content is kept.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review backlog (docs-currency): factory-integration.md still says root login is permitted with password over SSH

1 participant