Skip to content

fix(ssh): accept a provisioned key while root is still locked - #700

Merged
jeefy merged 1 commit into
projectbluefin:mainfrom
jeefy:fix/sshd-use-pam
Nov 21, 2026
Merged

jeefy merged 1 commit into
projectbluefin:mainfrom
jeefy:fix/sshd-use-pam

Conversation

@jeefy

@jeefy jeefy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary: Root ships locked, and giving root an SSH key through Ignition or the ssh.authorized_keys.root credential leaves it locked. The image's sshd ran without PAM, and sshd without PAM refuses a locked account even when the key is correct. So the documented ssh root@<node> login failed on network-booted, Booty-provisioned and credential-provisioned nodes. USB installs only got around it because the first-boot root password prompt unlocks root. This turns PAM on for sshd. Password logins stay off.


Why

files/os/ssh/sshd_config replaces FSDK's copy. FSDK's own copy has UsePAM yes (its openssh.bst seds it in), but ours has only the Include, so sshd fell back to UsePAM no. Root's shadow entry is !unprovisioned (elements/oci/bluefin-server-usr.bst), and Ignition 2.27 sets a password on an existing user only when passwordHash is given. The homelab templates, Booty's bluefin-node.ign and the credential path all give root a key and nothing else.

The repo already knew about this: the dogfood SSH check ran usermod -p '*' root first because "sshd refuses a locked root even for keys". That hid the problem from CI.

I reproduced it with OpenSSH in a Fedora 44 container, with root set to !unprovisioned and the image's auth settings:

UsePAM no -> root@127.0.0.1: Permission denied (publickey).
User root not allowed because account is locked
UsePAM yes -> LOGIN-OK

What changes

  • files/os/ssh/sshd_config.d/bluefin-server.conf: UsePAM yes. FSDK ships /etc/pam.d/sshd and its password-auth stack (both are in the image's factory /etc). PasswordAuthentication no and KbdInteractiveAuthentication no stay, so PAM is never asked for a password. It only runs the account and session stacks.
  • scripts/dogfood-installer.sh: the step-6 SSH login no longer unlocks root. It records passwd -S root and fails unless root was still locked (L) when the key login worked.
  • Comments and docs that said "sshd (UsePAM no)" now name PrintMotd (the motd behaviour they describe is unchanged), and the usb-installer.md note on * versus !* is corrected.

This is in the sshd drop-in, which is factory /etc, so it reaches new and diskless nodes. Installed nodes that already have /etc/ssh/sshd_config.d/bluefin-server.conf keep their copy. A separate PR moves the vendor sshd, sysctl and journald config out of factory /etc.

Tests

  • tests/unit/test_root_credential_contract.py::test_sshd_accepts_keys_for_a_root_that_has_no_password fails without the change and passes with it.
  • pytest -n auto: 1878 passed, 3 skipped.
  • just validate: passes.
  • python3 .github/scripts/docs-checks.py: passes.
  • Container repro above: done.
  • Not run: the image build and the QEMU dogfood. The dogfood-installer step-6 check needs a full-build run.

🤖 Generated with Claude Code

Root ships locked ("!unprovisioned"). Ignition users entries and the
ssh.authorized_keys.root credential install a key without touching the
password, so root stays locked, and sshd without PAM refuses a locked
account even for keys ("User root not allowed because account is
locked"). The documented ssh root@<node> path was refused on netboot,
Booty and credential-provisioned nodes.

UsePAM yes hands the account check to FSDK's /etc/pam.d/sshd, which
ignores the password field. PasswordAuthentication and
KbdInteractiveAuthentication stay off, so PAM is never asked for a
password.

The dogfood SSH login no longer unlocks root first, and asserts root is
still locked when the key login succeeds.

Assisted-by: Claude via Claude Code
Co-Authored-By: Claude Code <noreply@anthropic.com>
@jeefy
jeefy requested review from bketelsen and hanthor November 20, 2026 03:21
@jeefy

jeefy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves no-dco A commit is missing Signed-off-by — run `git rebase --signoff` role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 20, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: This change lets a server accept a login with an SSH key even while the administrator account (root) still has no password, which is how most nodes are set up. I found no problems that block it. One side effect is worth knowing: locking an account's password no longer stops that account from logging in with a key.


security (info, FYI)

  • files/os/ssh/sshd_config.d/bluefin-server.conf:17: with UsePAM yes, sshd no longer checks the ! lock in shadow. FSDK's /etc/pam.d/sshd uses password-auth, whose account stack is pam_unix.so (expiry only). So passwd -l <user> stops blocking key logins for every account, not only root. That matches Fedora/RHEL and is what the PR wants. Operators who used passwd -l to turn off SSH must now remove authorized_keys or expire the account (usermod -e 1). No password path opens: PasswordAuthentication no and KbdInteractiveAuthentication no (lines 10-11) keep the PAM auth stack out.

What I checked

  • files/os/ssh/sshd_config holds only the Include, so the drop-in's UsePAM yes applies.
  • FSDK's sshd and password-auth stacks have no pam_motd. That means no second banner, so the comments that now name PrintMotd are accurate.
  • scripts/dogfood-installer.sh:880/:1117: step 6 runs only in unattended mode. That mode gives root !* (line 573), so passwd -S reports L. Console mode, which sets a real root password, exits before step 6 (line 1060).

Limits: build, image-check and boot-test were skipped at 845a133. Neither the image build nor the dogfood SSH check ran, so the new step-6 check has not run yet.

No findings from: correctness, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

-- hive: agent=reviewer backend=claude model=claude-opus-5-5 effort=medium claude=2.1.327

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Ready for a human merge: verdict approved 845a133, 18/18 checks green, soak 30m0s elapsed. Blocked only by role/untrusted (no verified hive claim or registry tier).

The hive never approves on your behalf: approve and merge it on GitHub if you accept it.

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 20, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 21, 2026
Merged via the queue into projectbluefin:main with commit 5baa002 Nov 21, 2026
131 checks passed
@hivecommons-hive hivecommons-hive Bot removed role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing labels Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves no-dco A commit is missing Signed-off-by — run `git rebase --signoff`

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant