From a0f7013e37ecd165e90bf333e5f11d80e8b7559a Mon Sep 17 00:00:00 2001 From: "sec-check[bot]" Date: Sun, 20 Sep 2026 07:02:03 -0400 Subject: [PATCH 1/5] fix: generate SSH host keys on device, not in the published DDI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The DDI build ran ssh-keygen at image build time, embedding SSH host private keys in bluefin-server-ddi-*.raw.zst — a public GitHub Release asset. Anyone could extract the keys and impersonate any deployment of that release over SSH. Replace build-time generation with a first-start oneshot (ssh-host-keygen.service, ssh-keygen -A, guarded by ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key) pulled in via an sshd.service drop-in, so each device mints its own host identity the first time sshd starts. The image now ships only an empty /etc/ssh. Signed-off-by: sec-check[bot] --- elements/bluefin-server/os-k0s-first-boot.bst | 5 ++++- elements/oci/bluefin-server-ddi.bst | 9 ++++----- files/os/systemd/system/ssh-host-keygen.service | 12 ++++++++++++ .../system/sshd.service.d/10-host-keygen.conf | 6 ++++++ 4 files changed, 26 insertions(+), 6 deletions(-) create mode 100644 files/os/systemd/system/ssh-host-keygen.service create mode 100644 files/os/systemd/system/sshd.service.d/10-host-keygen.conf diff --git a/elements/bluefin-server/os-k0s-first-boot.bst b/elements/bluefin-server/os-k0s-first-boot.bst index 038a1d99..49e5894a 100644 --- a/elements/bluefin-server/os-k0s-first-boot.bst +++ b/elements/bluefin-server/os-k0s-first-boot.bst @@ -1,5 +1,8 @@ kind: import -description: Install the first-boot k0s sysext activation unit. +description: | + Install the systemd units from files/os/systemd/system: the first-boot k0s + sysext activation units, var.mount, and first-start SSH host key generation + (ssh-host-keygen.service plus its sshd.service drop-in). sources: - kind: local diff --git a/elements/oci/bluefin-server-ddi.bst b/elements/oci/bluefin-server-ddi.bst index 9f801570..be12df1b 100644 --- a/elements/oci/bluefin-server-ddi.bst +++ b/elements/oci/bluefin-server-ddi.bst @@ -78,12 +78,11 @@ config: printf 'root:x:0:\n' > /layer/etc/group chmod 0644 /layer/etc/group - # Pre-generate SSH host keys for first boot remote diagnostics + # SSH host keys are generated per device on first sshd start + # (ssh-host-keygen.service), never at build time: this DDI is published + # as a public release asset, so baked-in keys would expose every + # deployment's host private keys. mkdir -p /layer/etc/ssh - ssh-keygen -q -N "" -t ed25519 -f /layer/etc/ssh/ssh_host_ed25519_key - ssh-keygen -q -N "" -t rsa -b 3072 -f /layer/etc/ssh/ssh_host_rsa_key - chmod 0600 /layer/etc/ssh/ssh_host_*_key - chmod 0644 /layer/etc/ssh/ssh_host_*.pub # Sum actual file sizes across bind mounts (du -sb reports content bytes, not overlay blocks). ROOT_BYTES=$(du -sb /layer | cut -f1) diff --git a/files/os/systemd/system/ssh-host-keygen.service b/files/os/systemd/system/ssh-host-keygen.service new file mode 100644 index 00000000..8388ff6a --- /dev/null +++ b/files/os/systemd/system/ssh-host-keygen.service @@ -0,0 +1,12 @@ +[Unit] +Description=Generate SSH host keys on first sshd start +Documentation=man:ssh-keygen(1) +# Host keys are deliberately NOT baked into the DDI: the release artifact is +# public, so build-time keys would hand every deployment's host identity to +# anyone who downloads the image. Generate them per device instead. +ConditionPathExists=!/etc/ssh/ssh_host_ed25519_key +Before=sshd.service + +[Service] +Type=oneshot +ExecStart=/usr/bin/ssh-keygen -A diff --git a/files/os/systemd/system/sshd.service.d/10-host-keygen.conf b/files/os/systemd/system/sshd.service.d/10-host-keygen.conf new file mode 100644 index 00000000..f213297d --- /dev/null +++ b/files/os/systemd/system/sshd.service.d/10-host-keygen.conf @@ -0,0 +1,6 @@ +# Ensure per-device host keys exist before sshd starts. See +# ssh-host-keygen.service: keys are generated on the device, never at image +# build time, so the public release DDI carries no private key material. +[Unit] +Wants=ssh-host-keygen.service +After=ssh-host-keygen.service From 643131abb31538da8d1ca46a5aa356fcf212cc9e Mon Sep 17 00:00:00 2001 From: castrojo Date: Thu, 17 Sep 2026 22:53:23 -0400 Subject: [PATCH 2/5] fix(zfs): remove Flatcar sysext udevd drop-in from extracted zfs rootfs The upstream Flatcar zfs sysext includes an override for systemd-udevd: usr/lib/systemd/system/systemd-udevd.service.d/10-zfs.conf (After=systemd-sysext.service). When extracted directly into the base OS rootfs, this drop-in orders udevd after systemd-sysext.service. During boot, udevd was stopped while systemd-sysext.service and /var.mount were pending, deadlocking udev device unit discovery (/dev/disk/by-partlabel/var). Assisted-by: GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo (cherry picked from commit 11f2c50fea79529dded892f2239b9679e56126bc) Signed-off-by: sec-check[bot] --- elements/flatcar/flatcar-zfs.bst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/elements/flatcar/flatcar-zfs.bst b/elements/flatcar/flatcar-zfs.bst index 82832969..b0e6aa18 100644 --- a/elements/flatcar/flatcar-zfs.bst +++ b/elements/flatcar/flatcar-zfs.bst @@ -28,6 +28,11 @@ config: # Extract the SquashFS sysext directly into the element install root unsquashfs -no-xattrs -f -d "%{install-root}" flatcar-zfs.raw + # Drop the Flatcar sysext udevd dropin (After=systemd-sysext.service). + # When extracted directly into the OS rootfs, this drop-in causes systemd-udevd + # to stop at early boot waiting for systemd-sysext.service, deadlocking udev + # device creation before /var and storage partitions are mounted. + rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d" # Also preserve the raw sysext image under /usr/lib/sysexts for runtime sysext usage mkdir -p "%{install-root}/usr/lib/sysexts" install -D -m 0644 flatcar-zfs.raw "%{install-root}/usr/lib/sysexts/zfs.raw" From e156d768b6a1c45762ea9ffc0403c0473dcc31ad Mon Sep 17 00:00:00 2001 From: castrojo Date: Thu, 17 Sep 2026 23:26:34 -0400 Subject: [PATCH 3/5] fix(zfs): strip udev drop-in copy from tmpfiles configuration In addition to removing the extracted /usr/lib/systemd/system/systemd-udevd.service.d/ drop-in directory, also remove the 'C /etc/systemd/system/systemd-udevd.service.d/10-zfs.conf' copy rule in /usr/lib/tmpfiles.d/10-zfs.conf so systemd-tmpfiles does not recreate the udevd sysext ordering override in /etc. Assisted-by: GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo (cherry picked from commit 2d5dd1401c8e317c00cb4fd39e119cf3b6ea0d2d) Signed-off-by: sec-check[bot] --- elements/flatcar/flatcar-zfs.bst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/elements/flatcar/flatcar-zfs.bst b/elements/flatcar/flatcar-zfs.bst index b0e6aa18..08580ee5 100644 --- a/elements/flatcar/flatcar-zfs.bst +++ b/elements/flatcar/flatcar-zfs.bst @@ -28,11 +28,15 @@ config: # Extract the SquashFS sysext directly into the element install root unsquashfs -no-xattrs -f -d "%{install-root}" flatcar-zfs.raw - # Drop the Flatcar sysext udevd dropin (After=systemd-sysext.service). + # Drop the Flatcar sysext udevd dropin (After=systemd-sysext.service) + # and the tmpfiles rule that copies it into /etc/systemd/system/systemd-udevd.service.d/. # When extracted directly into the OS rootfs, this drop-in causes systemd-udevd # to stop at early boot waiting for systemd-sysext.service, deadlocking udev # device creation before /var and storage partitions are mounted. rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d" + if [ -f "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" ]; then + sed -i '/systemd-udevd\.service\.d/d' "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" + fi # Also preserve the raw sysext image under /usr/lib/sysexts for runtime sysext usage mkdir -p "%{install-root}/usr/lib/sysexts" install -D -m 0644 flatcar-zfs.raw "%{install-root}/usr/lib/sysexts/zfs.raw" From cb8e6339724982073df3a77723cfa49423321001 Mon Sep 17 00:00:00 2001 From: "sec-check[bot]" Date: Mon, 21 Sep 2026 10:15:56 -0400 Subject: [PATCH 4/5] fix(zfs): declare sed build dependency for flatcar-zfs sysext cleanup The install-commands call sed -i to strip the systemd-udevd drop-in rule from 10-zfs.conf, but sed is not in the build sandbox. The build failed with 'sed: line 13: sed: command not found' (exit 127). Signed-off-by: sec-check[bot] --- elements/flatcar/flatcar-zfs.bst | 1 + 1 file changed, 1 insertion(+) diff --git a/elements/flatcar/flatcar-zfs.bst b/elements/flatcar/flatcar-zfs.bst index 08580ee5..e41cf70f 100644 --- a/elements/flatcar/flatcar-zfs.bst +++ b/elements/flatcar/flatcar-zfs.bst @@ -10,6 +10,7 @@ description: | build-depends: - base/base-stack.bst + - freedesktop-sdk.bst:components/sed.bst - freedesktop-sdk.bst:components/squashfs-tools.bst variables: From dc7549828a00095f62d16703adc599beb83a57a0 Mon Sep 17 00:00:00 2001 From: "sec-check[bot]" Date: Mon, 21 Sep 2026 10:42:35 -0400 Subject: [PATCH 5/5] fix(zfs): adopt main's flatcar-zfs sysext cleanup verbatim #220 landed the same udevd drop-in removal on main, plus two things this branch lacked: the components/sed.bst build dependency the `sed -i` call needs, and a mksquashfs rebuild of the raw sysext from the cleaned tree so the ordering override cannot reappear if the raw is ever merged. Without sed.bst the build failed with `sed: command not found` (exit 127), taking down both `build` and `build-kernel`. Diverging from main on this file also left the PR unmergeable, which stopped Actions scheduling any run at all. Taking main's version verbatim fixes the build and removes the conflict. Signed-off-by: sec-check[bot] --- elements/flatcar/flatcar-zfs.bst | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/elements/flatcar/flatcar-zfs.bst b/elements/flatcar/flatcar-zfs.bst index e41cf70f..eabdcf8e 100644 --- a/elements/flatcar/flatcar-zfs.bst +++ b/elements/flatcar/flatcar-zfs.bst @@ -29,15 +29,20 @@ config: # Extract the SquashFS sysext directly into the element install root unsquashfs -no-xattrs -f -d "%{install-root}" flatcar-zfs.raw - # Drop the Flatcar sysext udevd dropin (After=systemd-sysext.service) - # and the tmpfiles rule that copies it into /etc/systemd/system/systemd-udevd.service.d/. - # When extracted directly into the OS rootfs, this drop-in causes systemd-udevd - # to stop at early boot waiting for systemd-sysext.service, deadlocking udev - # device creation before /var and storage partitions are mounted. + # Drop Flatcar's udevd sysext ordering override. In Flatcar, ZFS is loaded as + # a runtime sysext, so udevd was ordered After=systemd-sysext.service. + # In Bluefin Server, ZFS is extracted directly into the OS rootfs at build time. + # Leaving this drop-in causes systemd-udevd to wait for systemd-sysext.service, + # which waits for local-fs.target, which waits for var.mount, which waits for + # the udev device unit — creating a boot deadlock. rm -rf "%{install-root}/usr/lib/systemd/system/systemd-udevd.service.d" if [ -f "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" ]; then - sed -i '/systemd-udevd\.service\.d/d' "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" + sed -i "/systemd-udevd\.service\.d/d" "%{install-root}/usr/lib/tmpfiles.d/10-zfs.conf" fi - # Also preserve the raw sysext image under /usr/lib/sysexts for runtime sysext usage + + # Preserve a raw sysext image under /usr/lib/sysexts for runtime sysext usage. + # Rebuild it from the cleaned tree rather than shipping the upstream image, so + # the udevd ordering override above cannot reappear if the raw is ever merged. + mksquashfs "%{install-root}" zfs-cleaned.raw -noappend -no-xattrs -all-root mkdir -p "%{install-root}/usr/lib/sysexts" - install -D -m 0644 flatcar-zfs.raw "%{install-root}/usr/lib/sysexts/zfs.raw" + install -D -m 0644 zfs-cleaned.raw "%{install-root}/usr/lib/sysexts/zfs.raw"