Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,11 @@ hash partition (`bluefin_usr_verity_<ver>`), and the root hash is recorded in
live in `/usr/share/factory/etc` and are copied in by `systemd-tmpfiles`. A
diskless node keeps `/etc` on tmpfs, so it starts empty on every boot; a disk
install keeps it on the root partition (`files/os/repart.d/50-root.conf`),
where it persists across A/B updates.
where it persists across A/B updates. `systemd-tmpfiles` copies a factory
file only when it is missing, so a vendor setting that must follow updates
ships in `/usr` instead: read from `/usr/lib` (sysctl, journald, resolved,
`system.conf`) or linked into `/etc` with an `L` line (the sshd drop-in).
`bluefin-migrate-etc-vendor` removes the unmodified copies older images left.

## The boot chain

Expand Down
2 changes: 1 addition & 1 deletion docs/skills/systemd-sysupdate-verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ last success, the last error and the run it last recorded in
`/var/lib/bluefin-update-status/state`, so they survive reboots and updates,
and writes `/run/issue.d/40-bluefin-update.issue` (agetty, next to
`/usr/lib/issue.d/30-bluefin.issue`) and `/run/motd`. login (`MOTD_FILE` in
`login.defs`) and sshd (`UsePAM no`) only print `/etc/motd`, so
`login.defs`) and sshd (`PrintMotd`) only print `/etc/motd`, so
`tmpfiles.d/40-bluefin-motd.conf` links it to `/run/motd` unless the operator
has an `/etc/motd` of their own. It only reads local state and contacts
nothing. A failed check also stays a failed unit, `systemd-sysupdate.service`,
Expand Down
2 changes: 1 addition & 1 deletion docs/skills/tpm2-credential-sealing.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ The image ships no password. Root's `/etc/shadow` entry is `!unprovisioned`
(`elements/oci/bluefin-server-usr.bst`), systemd's marker for a root account
nobody has configured: `passwd -S root` reports `L`, console login, `su` and
`sulogin` refuse it, and sshd accepts keys only
(`files/os/ssh/sshd_config.d/bluefin-server.conf`). Access is provisioned per
(`files/os/sshd_config.d/bluefin-server.conf`). Access is provisioned per
node:

- **SSH key:** Ignition `passwd.users` with `name: root` and
Expand Down
4 changes: 2 additions & 2 deletions docs/skills/usb-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,8 +343,8 @@ printf 'd /home/%s 0700 %s %s -\nd /home/%s/.ssh 0700 %s %s -\nf+~ /home/%s/.ssh
# ssh.listen: systemd-ssh-generator takes the binary form only (v261 does
# not unbase64); services take the base64 form systemd-creds writes
printf 22 | enc ssh.listen - | base64 -d > ssh.listen.cred
# "*": no password login, but not locked like sysusers' default "!*", which
# sshd (UsePAM no) refuses even for keys
# "*": no password login; sshd takes the key either way (UsePAM yes), but
# "*" is not reported locked like sysusers' default "!*"
printf '*' | enc "passwd.hashed-password.$user" "passwd.hashed-password.$user.cred"
printf '!*' | enc passwd.hashed-password.root passwd.hashed-password.root.cred
# copy the five .cred files into /loader/credentials/ on the stick's ESP
Expand Down
8 changes: 4 additions & 4 deletions elements/bluefin-server/os-host.bst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ description: |
bluefin-boot-deadline, bluefin-diskless-update-check,
bluefin-sysext-fetch, bluefin-boot-origin;
- first boot: bluefin-hostname, bluefin-firstboot-credentials,
bluefin-repair-etc-modes;
bluefin-repair-etc-modes, bluefin-migrate-etc-vendor;
- bluefin-sysext-modules, which module sysexts' load units run.
Unit drop-ins call `bluefin-host kube-interlock`, `kured-flag`,
`mask-uki-transfer` and `boot-is-good` directly.
Expand Down Expand Up @@ -57,8 +57,8 @@ config:
install -D -m 0755 bluefin-host "%{install-root}/usr/bin/bluefin-host"
mkdir -p "%{install-root}/usr/libexec"
for helper in boot-deadline boot-origin diskless-update-check \
firstboot-credentials hostname reboot-lock repair-etc-modes \
sysext-fetch sysext-modules update-pending update-status \
update-status-migrate; do
firstboot-credentials hostname migrate-etc-vendor reboot-lock \
repair-etc-modes sysext-fetch sysext-modules update-pending \
update-status update-status-migrate; do
ln -s ../bin/bluefin-host "%{install-root}/usr/libexec/bluefin-${helper}"
done
27 changes: 24 additions & 3 deletions elements/bluefin-server/os-sshd-config.bst
Original file line number Diff line number Diff line change
@@ -1,9 +1,30 @@
kind: import
description: Install sshd configuration and drop-ins.
kind: manual
description: |
sshd configuration. The main sshd_config, which only includes
/etc/ssh/sshd_config.d/*.conf, goes to the factory /etc. The vendor
drop-in goes to /usr/lib/bluefin/sshd_config.d, and
files/os/tmpfiles.d/50-bluefin-sshd.conf (os-resolv-conf.bst) links it into
/etc/ssh/sshd_config.d: a disk install keeps /etc across A/B updates, so a
copy would stay at the version the node first booted, while the link
follows /usr. L leaves an operator's own file of that name alone.

build-depends:
- base/base-stack.bst

sources:
- kind: local
path: files/os/ssh
directory: ssh-src
- kind: local
path: files/os/sshd_config.d
directory: dropin-src

variables:
strip-binaries: ""

config:
target: /etc/ssh
install-commands:
- |
set -eu
install -Dm0644 ssh-src/sshd_config "%{install-root}/etc/ssh/sshd_config"
install -Dm0644 dropin-src/*.conf -t "%{install-root}/usr/lib/bluefin/sshd_config.d/"
17 changes: 14 additions & 3 deletions elements/oci/bluefin-server-usr.bst
Original file line number Diff line number Diff line change
Expand Up @@ -72,16 +72,21 @@ config:
grep -q '^root:' /sysroot/etc/passwd
chmod 0400 /sysroot/etc/shadow /sysroot/etc/gshadow

mkdir -p /sysroot/etc/systemd/system /sysroot/etc/systemd/journald.conf.d /sysroot/etc/sysctl.d
mkdir -p /sysroot/etc/systemd/system
# Headless: homed-firstboot would block the console asking for a user.
ln -sf /dev/null /sysroot/etc/systemd/system/systemd-homed-firstboot.service
# The initrd's import.pull credential (local name "rootdisk") stays in
# /run/credentials after switch-root; keep the host from re-running it.
# /etc is the only unit directory that outranks generator output.
ln -sf /dev/null /sysroot/etc/systemd/system/systemd-import@var-lib-machines-rootdisk.raw.service
ln -sf /dev/null /sysroot/etc/systemd/system/systemd-loop@var-lib-machines-rootdisk.raw.service
printf '[Journal]\nForwardToConsole=no\nMaxLevelConsole=warning\n' > /sysroot/etc/systemd/journald.conf.d/10-console.conf
printf 'kernel.printk = 3 4 1 3\n' > /sysroot/etc/sysctl.d/10-console.conf
# Vendor drop-ins go to /usr/lib, not the factory /etc: tmpfiles' C
# never overwrites, so a copy in /etc would stay at the version an
# installed node first booted. An /etc file of the same name overrides.
# bluefin-migrate-etc-vendor removes the copies older images left.
mkdir -p /sysroot/usr/lib/systemd/journald.conf.d /sysroot/usr/lib/sysctl.d
printf '[Journal]\nForwardToConsole=no\nMaxLevelConsole=warning\n' > /sysroot/usr/lib/systemd/journald.conf.d/10-console.conf
printf 'kernel.printk = 3 4 1 3\n' > /sysroot/usr/lib/sysctl.d/10-console.conf
printf 'LANG=C.UTF-8\n' > /sysroot/etc/locale.conf
ln -sf ../usr/share/zoneinfo/UTC /sysroot/etc/localtime
rm -f /sysroot/etc/machine-id
Expand Down Expand Up @@ -160,6 +165,12 @@ config:
[Install]
WantedBy=multi-user.target
EOF
# Enabled in /usr, not by preset, so nodes that updated into it run it.
# Before systemd-tmpfiles-setup, so the same boot links the /usr copy in
# where it removed an unmodified one.
mkdir -p /sysroot/usr/lib/systemd/system/sysinit.target.wants
ln -sf ../bluefin-migrate-etc-vendor.service \
/sysroot/usr/lib/systemd/system/sysinit.target.wants/bluefin-migrate-etc-vendor.service
mkdir -p /sysroot/usr/lib/systemd/system/multi-user.target.wants
ln -sf ../bluefin-repair-etc-modes.service \
/sysroot/usr/lib/systemd/system/multi-user.target.wants/bluefin-repair-etc-modes.service
Expand Down
2 changes: 2 additions & 0 deletions files/homelab/cluster/cmd/bluefin-host/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ import (
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/bootorigin"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/diskless"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/etcmodes"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/etcvendor"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/firstboot"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/fleetlock"
"github.com/projectbluefin/server/files/homelab/cluster/internal/host/hostname"
Expand Down Expand Up @@ -45,6 +46,7 @@ var helpers = map[string]command{
"diskless-update-check": diskless.Check,
"firstboot-credentials": firstboot.Main,
"hostname": hostname.Main,
"migrate-etc-vendor": etcvendor.Main,
"reboot-lock": fleetlock.Main,
"repair-etc-modes": etcmodes.Main,
"sysext-fetch": sysextfetch.Main,
Expand Down
97 changes: 97 additions & 0 deletions files/homelab/cluster/internal/host/etcvendor/etcvendor.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
// Package etcvendor is bluefin-migrate-etc-vendor: one-shot removal of the
// vendor drop-ins older images copied into /etc, now that they ship in /usr.
//
// Images up to this one put the sshd, sysctl and journald drop-ins in the
// factory /etc, which systemd-tmpfiles copies (C) only when missing. A disk
// install keeps /etc across A/B updates, so those copies stayed at the
// version the node first booted. The drop-ins now live in /usr, linked (sshd)
// or read directly (sysctl, journald). This removes a copy only while its
// content is byte-identical to a version an image shipped, so a file an
// operator edited stays, and tmpfiles then links the current one in. It runs
// once; a stamp in /var suppresses later boots.
package etcvendor

import (
"crypto/sha256"
"encoding/hex"
"errors"
"io/fs"
"os"
"slices"

"github.com/projectbluefin/server/files/homelab/cluster/internal/host/sys"
)

const Stamp = "/var/lib/bluefin/etc-vendor-migrate.stamp"

// Shipped maps each /etc path an older image copied in to the sha256 of
// every version of it a release carried (from v26.09.673, the first FSDK
// image, on), and of the sshd drop-in as #700 left it. Only an exact match
// is removed.
var Shipped = map[string][]string{
"ssh/sshd_config.d/bluefin-server.conf": {
"d6db893a293261e9bc1a7ec183378f3199140039258e42ff7f231bcb0f2fc5b2",
"cd0d2c8586c7fb9825f398990ed46ac32454b54704333485e79e31d6fd03051f",
"b1c4085814e5b17a9ae70715a822421ebe57e485fd4b77dcb88b6e07558ef217",
"0e04923abd1f70d8012dd95292ec7f343f17a6e6e41277517072fbe323b19989",
},
"sysctl.d/10-console.conf": {
"815e73d09439ab850feafddda67cb818025d12211ebf6ab28293cacaafa7552e",
},
"systemd/journald.conf.d/10-console.conf": {
"f4ef13fbdd83470bb0a462f525c6fa00d75428ecbafb4346f45bbdbeaee17db3",
},
}

// Migrate removes etc/<path> for every path in shipped that is a regular
// file with one of its listed hashes, then creates stamp. It returns the
// paths it removed.
func Migrate(etc string, shipped map[string][]string, stamp string) ([]string, error) {
var removed []string
for rel, sums := range shipped {
p := etc + "/" + rel
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return removed, err
}
// A link (the new layout) or anything else an operator put there stays.
if !fi.Mode().IsRegular() {
continue
}
data, err := os.ReadFile(p)
if err != nil {
return removed, err
}
sum := sha256.Sum256(data)
if !slices.Contains(sums, hex.EncodeToString(sum[:])) {
continue
}
if err := os.Remove(p); err != nil {
return removed, err
}
removed = append(removed, rel)
}
if err := sys.MkdirP(sys.Dir(stamp)); err != nil {
return removed, err
}
return removed, sys.Truncate(stamp)
}

// Main is bluefin-migrate-etc-vendor.
// Test overrides: BLUEFIN_ETC, BLUEFIN_ETC_VENDOR_STAMP.
func Main(h *sys.Host, _ []string) int {
etc := h.Env("BLUEFIN_ETC", "/etc")
stamp := h.Env("BLUEFIN_ETC_VENDOR_STAMP", Stamp)
removed, err := Migrate(etc, Shipped, stamp)
for _, rel := range removed {
h.Printf("bluefin-migrate-etc-vendor: removed the unmodified %s/%s; /usr now ships it\n", etc, rel)
}
if err != nil {
h.Eprintf("bluefin-migrate-etc-vendor: %v\n", err)
return 1
}
return 0
}
78 changes: 78 additions & 0 deletions files/homelab/cluster/internal/host/etcvendor/etcvendor_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
package etcvendor

import (
"crypto/sha256"
"encoding/hex"
"os"
"path/filepath"
"sort"
"testing"
)

func sum(s string) string {
b := sha256.Sum256([]byte(s))
return hex.EncodeToString(b[:])
}

func write(t *testing.T, p, data string) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte(data), 0o644); err != nil {
t.Fatal(err)
}
}

func TestMigrateRemovesOnlyUnmodifiedShippedCopies(t *testing.T) {
dir := t.TempDir()
etc := filepath.Join(dir, "etc")
shipped := map[string][]string{
"a.conf": {sum("old\n"), sum("older\n")},
"edited.conf": {sum("old\n")},
"linked.conf": {sum("old\n")},
"missing.conf": {sum("old\n")},
"d/b.conf": {sum("b\n")},
}
write(t, filepath.Join(etc, "a.conf"), "older\n")
write(t, filepath.Join(etc, "edited.conf"), "old\n# mine\n")
write(t, filepath.Join(etc, "d", "b.conf"), "b\n")
write(t, filepath.Join(dir, "usr.conf"), "old\n")
if err := os.Symlink(filepath.Join(dir, "usr.conf"), filepath.Join(etc, "linked.conf")); err != nil {
t.Fatal(err)
}
stamp := filepath.Join(dir, "var", "lib", "bluefin", "stamp")

removed, err := Migrate(etc, shipped, stamp)
if err != nil {
t.Fatal(err)
}
sort.Strings(removed)
if len(removed) != 2 || removed[0] != "a.conf" || removed[1] != "d/b.conf" {
t.Fatalf("removed %q; want [a.conf d/b.conf]", removed)
}
for _, kept := range []string{"edited.conf", "linked.conf"} {
if _, err := os.Lstat(filepath.Join(etc, kept)); err != nil {
t.Errorf("%s: %v; want it kept", kept, err)
}
}
if _, err := os.Stat(filepath.Join(dir, "usr.conf")); err != nil {
t.Errorf("the link target was touched: %v", err)
}
if _, err := os.Stat(stamp); err != nil {
t.Errorf("stamp: %v", err)
}
}

func TestShippedHashesAreFullSHA256(t *testing.T) {
for rel, sums := range Shipped {
if len(sums) == 0 {
t.Errorf("%s: no hashes", rel)
}
for _, s := range sums {
if b, err := hex.DecodeString(s); err != nil || len(b) != sha256.Size {
t.Errorf("%s: %q is not a sha256", rel, s)
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@ PermitRootLogin prohibit-password
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
# Without PAM, sshd refuses a locked account ("!" password) even for a key,
# and root stays locked when Ignition or the ssh.authorized_keys.root
# credential gives it only a key. With PAM, FSDK's /etc/pam.d/sshd account
# stack decides instead; it ignores the password field. The two lines above
# keep PAM out of authentication, so this opens no password path.
UsePAM yes

# Keep standard SSH file transfer available without another server process.
Subsystem sftp internal-sftp
23 changes: 23 additions & 0 deletions files/os/systemd/system/bluefin-migrate-etc-vendor.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[Unit]
Description=Remove Vendor Drop-ins Older Images Copied into /etc
Documentation=file:/usr/libexec/bluefin-migrate-etc-vendor
# Older images shipped the sshd, sysctl and journald drop-ins in the factory
# /etc, which tmpfiles copies only when missing, so an installed node kept
# the copy it first booted. They ship in /usr now; this removes each copy
# that is still exactly as an image shipped it, and tmpfiles-setup then
# links the /usr one in. Enabled in /usr (elements/oci/bluefin-server-usr.bst)
# so updated nodes run it; once per node. Installed nodes only: a diskless
# node's /etc starts empty every boot.
ConditionPathExists=!/var/lib/bluefin/etc-vendor-migrate.stamp
ConditionPathExists=!/run/machines/rootdisk.raw
ConditionKernelCommandLine=!root=tmpfs
DefaultDependencies=no
After=local-fs.target
Before=systemd-tmpfiles-setup.service sysinit.target
Conflicts=shutdown.target
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/libexec/bluefin-migrate-etc-vendor
2 changes: 1 addition & 1 deletion files/os/tmpfiles.d/40-bluefin-motd.conf
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# login (MOTD_FILE in login.defs) and sshd (UsePAM no) print /etc/motd only.
# login (MOTD_FILE in login.defs) and sshd (PrintMotd) print /etc/motd only.
# Point it at the runtime message bluefin-update-status writes. Without "+",
# an /etc/motd the operator wrote stays.
L /etc/motd - - - - ../run/motd
6 changes: 6 additions & 0 deletions files/os/tmpfiles.d/50-bluefin-sshd.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# sshd's vendor drop-in (/usr/lib/bluefin/sshd_config.d, os-sshd-config.bst).
# A link, not a copy in the factory /etc, so installed nodes, which keep /etc
# across A/B updates, get its changes too. L leaves an operator's own file
# alone; bluefin-migrate-etc-vendor removes the unmodified copies older
# images left, before this runs.
L /etc/ssh/sshd_config.d/bluefin-server.conf - - - - ../../../usr/lib/bluefin/sshd_config.d/bluefin-server.conf
Loading