Skip to content

fix(os): ship the sshd, sysctl and journald drop-ins in /usr - #701

Open
jeefy wants to merge 2 commits into
projectbluefin:mainfrom
jeefy:fix/vendor-config-in-usr
Open

jeefy wants to merge 2 commits into
projectbluefin:mainfrom
jeefy:fix/vendor-config-in-usr

Conversation

@jeefy

@jeefy jeefy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary: Installed servers never receive changes to three of the image's own settings files: the SSH server settings, a kernel console setting and a journal setting. Those files are copied into /etc once, on the first boot, and an installed disk keeps /etc across updates, so the copy stays as it was on that first boot. This moves them into the read-only system image, which every update replaces, and adds a one-time cleanup of the old copies on servers that already have them. A copy an operator has edited is left alone.

Depends on #700. This branch is stacked on it: the first commit here is #700's, so please review and merge that one first.


Why

elements/oci/bluefin-server-usr.bst moves the image's /etc to /usr/share/factory/etc and writes a tmpfiles C line for each entry. C copies only when the target is missing (tmpfiles.d(5)), and a disk install keeps /etc across A/B updates. The element's own comment says what follows: "a copy would stay at the version the node first booted". resolved, system.conf and the bash prompt were moved to /usr for this reason. These three were not:

  • /etc/ssh/sshd_config.d/bluefin-server.conf (os-sshd-config.bst, target: /etc/ssh)
  • /etc/sysctl.d/10-console.conf and /etc/systemd/journald.conf.d/10-console.conf, both written by bluefin-server-usr.bst

The sshd drop-in has already changed twice since the first FSDK release (v26.09.673): in 8824f9e, and to add Subsystem sftp in b9033de. #700 changes it a third time. None of those changes reached a node installed before them.

What changes

  • sysctl, journald: written to /usr/lib/sysctl.d/10-console.conf and /usr/lib/systemd/journald.conf.d/10-console.conf. Both daemons read /usr/lib directly, and an /etc file with the same name still overrides it. The content is unchanged.
  • sshd: sshd has no /usr config directory. The drop-in now lives in /usr/lib/bluefin/sshd_config.d/, and files/os/tmpfiles.d/50-bluefin-sshd.conf links it in with L /etc/ssh/sshd_config.d/bluefin-server.conf. This is the same pattern as 50-bluefin-prompt.conf. L leaves an operator's own file alone. The main sshd_config, which only includes sshd_config.d/*.conf, stays in the factory /etc. os-sshd-config.bst becomes a small manual element that installs the two files to their two places, and the drop-in source moves to files/os/sshd_config.d/.
  • bluefin-migrate-etc-vendor (a new bluefin-host helper, internal/host/etcvendor) cleans up existing installs:
    • It removes each old copy only while its sha256 matches a version a release shipped. I took those hashes from every v2* tag (106 + 74 + 2 tags; three sshd versions, one each for the other two files), plus the version fix(ssh): accept a provisioned key while root is still locked #700 leaves.
    • It skips edited files, symlinks and missing files.
    • Its unit is enabled in /usr (sysinit.target.wants), like the other migrations. That way updated nodes run it too, which a preset would not do.
    • It runs once (stamp in /var/lib/bluefin), on installed nodes only (diskless /etc starts empty every boot), and is ordered before systemd-tmpfiles-setup.service, so the same boot links the new file in.
  • docs/skills/ddi-installer.md records the rule: a vendor setting that has to follow updates ships in /usr.

Tests

  • New tests/unit/test_etc_vendor_dropins.py checks:
    • no vendor drop-in is written to the factory /etc;
    • the link rule and the path it resolves to;
    • the unit's conditions and ordering;
    • the current drop-in hashes are in the migration list;
    • the real bluefin-migrate-etc-vendor binary removes unmodified copies, keeps an edited one, and writes the stamp.
  • etcvendor_test.go covers the same cases plus symlinks and missing files.
  • A Fedora 44 container run with systemd-tmpfiles and OpenSSH 10.2 shows:
    • a fresh /etc gets the link;
    • an old node's kept /etc/ssh (copy removed) gets the link;
    • an operator's file stays;
    • sshd -T through the link reports usepam yes, permitrootlogin prohibit-password and passwordauthentication no.
  • pytest -n auto: 1885 passed, 3 skipped.
  • go vet ./... and go test -race ./... (files/homelab/cluster): pass.
  • ruff, mypy, docs-checks.py: clean.
  • just validate: passes.
  • Not run: the image build and a QEMU update from an older release. This needs a full-build run, and ideally an e2e-opt-in run, to prove it on a real update.

🤖 Generated with Claude Code

jeefy and others added 2 commits November 19, 2026 21:20
Root ships locked ("!unprovisioned"). Ignition users entries and the
ssh.authorized_keys.root credential install a key without touching the
password, so root stays locked, and sshd without PAM refuses a locked
account even for keys ("User root not allowed because account is
locked"). The documented ssh root@<node> path was refused on netboot,
Booty and credential-provisioned nodes.

UsePAM yes hands the account check to FSDK's /etc/pam.d/sshd, which
ignores the password field. PasswordAuthentication and
KbdInteractiveAuthentication stay off, so PAM is never asked for a
password.

The dogfood SSH login no longer unlocks root first, and asserts root is
still locked when the key login succeeds.

Assisted-by: Claude via Claude Code
Co-Authored-By: Claude Code <noreply@anthropic.com>
These three drop-ins lived in the factory /etc. systemd-tmpfiles copies
factory files with C, which never overwrites, and a disk install keeps
/etc across A/B updates, so an installed node kept the copy it first
booted. No change to them, including the UsePAM fix, could reach it.
resolved, system.conf and the prompt already moved to /usr for this.

- sysctl and journald: written to /usr/lib/sysctl.d and
  /usr/lib/systemd/journald.conf.d, which both read directly.
- sshd: installed in /usr/lib/bluefin/sshd_config.d and linked into
  /etc/ssh/sshd_config.d by a tmpfiles L line. sshd has no /usr config
  directory, and L leaves an operator's own file of that name alone.
  The main sshd_config, which only includes sshd_config.d, stays in the
  factory /etc.
- bluefin-migrate-etc-vendor (bluefin-host) runs once on installed
  nodes, before systemd-tmpfiles-setup, and removes each old copy only
  while it is byte-identical to a version a release shipped (hashes
  from every v26.09+ tag). An edited copy stays and keeps winning.

Assisted-by: Claude via Claude Code
Co-Authored-By: Claude Code <noreply@anthropic.com>
@jeefy
jeefy requested review from ahmedadan and hanthor November 20, 2026 04:37
@jeefy

jeefy commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ahmedadan after the PR has been reviewed.
You can assign the PR to them by writing /assign @ahmedadan in a comment when ready.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves no-dco A commit is missing Signed-off-by — run `git rebase --signoff` role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 20, 2026
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 20, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Ready for a human merge: verdict approved 132d56d, 18/18 checks green, soak 30m0s elapsed. Blocked only by role/untrusted (no verified hive claim or registry tier).

The hive never approves on your behalf: approve and merge it on GitHub if you accept it.

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing and removed stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 20, 2026
@jeefy
jeefy added this pull request to the merge queue Nov 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 21, 2026
@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/090-ready-for-human-merge Hive would merge this but trust/role blocks it; a maintainer's /lgtm /approve is all that is missing labels Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves no-dco A commit is missing Signed-off-by — run `git rebase --signoff` role/untrusted Author has no registry profile or verified hive claim (or is revoked); never auto-merged stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant