Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/scripts/track-binaries.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@

Stdlib only. GITHUB_TOKEN or GH_TOKEN, if set, is sent to api.github.com only.
"""

from __future__ import annotations

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated to #688: this adds from __future__ import annotations to the binary tracker. If it is needed, a separate PR, or a line in the body saying why, would keep this diff to the sysupdate fix.


import argparse
import hashlib
Expand Down
3 changes: 3 additions & 0 deletions docs/skills/systemd-sysupdate-verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,9 @@ the same names
(`files/os/sysupdate.d/<name>.feature` and `3N-<name>.transfer`), enabled per
node with `updatectl enable zfs` or a drop-in such as
`/etc/sysupdate.d/zfs.feature.d/enable.conf` containing `[Feature] Enabled=true`.
`bluefin-sysupdate-migrate` renames that drop-in directory to
`zfs.feature.d.disabled` when the release stops shipping the feature, and
restores it when a later release ships the feature again.
The k0s and NVIDIA Container Toolkit sysexts stay separate components
(`files/os/sysupdate.k0s.d/`, `files/os/sysupdate.nvidia-container-toolkit.d/`;
`systemd-sysupdate --component=<name> update`) with their own version axes.
Expand Down
5 changes: 4 additions & 1 deletion elements/bluefin-server/os-k0s-first-boot.bst
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,10 @@ description: |
Kubernetes; the boot deadline; the diskless update check;
bluefin-update-status.service for the login banners and
bluefin-update-status-migrate.service, which enables it on nodes that
updated from an image without it).
updated from an image without it), and bluefin-sysupdate-migrate.service
plus the systemd-sysupdate.service drop-in
(05-bluefin-prune-deprecated-features.conf), which archive enabled
sysupdate features the remote release no longer ships.

sources:
- kind: local
Expand Down
3 changes: 3 additions & 0 deletions elements/bluefin-server/os-stack.bst
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ depends:
# A/B System Update Configurations
- bluefin-server/os-sysupdate.bst

# Prune sysupdate features the release no longer ships, before an update
- bluefin-server/os-sysupdate-feature-migrate.bst

# Runtime helpers (bluefin-host): update status, reboot interlocks, boot
# deadline, diskless update check, sysext fetch, first-boot naming
- bluefin-server/os-host.bst
Expand Down
15 changes: 15 additions & 0 deletions elements/bluefin-server/os-sysupdate-feature-migrate.bst
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
kind: import
description: |
Install the systemd-sysupdate deprecated-feature migration helper so a
feature the release no longer ships cannot silently stop the next update.

The executable lands in /usr/bin; the boot-time unit (enabled by a wants
symlink in bluefin-server-usr.bst) and the ExecStartPre drop-in on systemd-sysupdate.service are
shipped next to the other system units.

sources:
- kind: local
path: files/os/sysupdate-migrate

config:
target: /usr/bin
4 changes: 4 additions & 0 deletions elements/oci/bluefin-server-usr.bst
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,10 @@ config:
# does, once per installed node (projectbluefin/server#367).
ln -sf ../bluefin-update-status-migrate.service \
/sysroot/usr/lib/systemd/system/multi-user.target.wants/bluefin-update-status-migrate.service
# Presets apply on first boot only; enable the sysupdate feature prune in
# /usr so nodes updating into this image run it too (#688).
ln -sf ../bluefin-sysupdate-migrate.service \
/sysroot/usr/lib/systemd/system/multi-user.target.wants/bluefin-sysupdate-migrate.service
# Enabled in /usr, not by preset: a node that updated into this image
# must still give its FleetLock slot back once an operator configures a
# server, or its group stops rebooting. Inert without a server.
Expand Down
17 changes: 17 additions & 0 deletions files/os/systemd/system/bluefin-sysupdate-migrate.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[Unit]
Description=Bluefin: prune deprecated systemd-sysupdate features
Documentation=https://github.com/projectbluefin/server/issues/688
# Run ahead of the OTA update path so a feature that the release no longer
# ships cannot silently stop the next update. This ordering covers the boot
# path; the ExecStartPre drop-in on systemd-sysupdate.service covers updates
# triggered later by the timer or by hand.
Before=systemd-sysupdate.timer
Before=systemd-sysupdate.service

[Service]
Type=oneshot
ExecStart=/usr/bin/bluefin-sysupdate-migrate
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[Service]
# Prune retired sysupdate features before every update, so an enabled feature
# that the release no longer ships cannot make systemd-sysupdate report
# "No update needed" instead of updating. The helper is idempotent, so running
# it on every update is safe. The "-" keeps a helper failure from blocking updates.
ExecStartPre=-/usr/bin/bluefin-sysupdate-migrate

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Info: this covers the timer and systemctl start systemd-sysupdate paths. updatectl update goes through systemd-sysupdated.service, which this repo already ships a drop-in for (systemd-sysupdated.service.d/10-diskless.conf). That daemon execs the systemd-sysupdate binary itself, not this unit, so a manual updatectl update does not get the prune. The boot-time bluefin-sysupdate-migrate.service still catches it on the next boot. A matching ExecStartPre=- on systemd-sysupdated.service would cover that path too, if you want it.

179 changes: 179 additions & 0 deletions files/os/sysupdate-migrate/bluefin-sysupdate-migrate
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
#!/usr/bin/env bash
#
# bluefin-sysupdate-migrate -- prune systemd-sysupdate features the current
# release no longer ships, so a retired feature cannot silently stop updates.
#
# systemd-sysupdate only applies an update set when every feature enabled under
# /etc/sysupdate.d/<feature>.feature.d/ is also provided by the release. When a
# feature leaves the release manifest -- for example Homebrew becoming a
# first-boot seed instead of an opt-in extension -- the matching /etc drop-in
# can no longer be satisfied, and systemd-sysupdate quietly reports "No update
# needed" instead of updating.
#
# The node that stalls still ships the retired feature's transfers locally; only
# the remote release dropped them. So a feature is retired when none of the
# local transfers that list it find a file matching their [Source] MatchPattern
# in the remote SHA256SUMS at the newest version. A feature with no local
# transfer at all is also retired. If a manifest cannot be fetched, the feature
# is left alone: an offline node must never lose a working feature.
#
# This one-shot helper runs ahead of the update path, archives each retired
# /etc drop-in, and prints a clear message so an operator is never left
# wondering why an update was skipped.

set -euo pipefail

# Paths are overridable so the logic can be exercised by unit tests against a
# scratch tree without touching the real system directories.
etc_feature_dir="${SYSUPDATE_ETC_DIR:-/etc/sysupdate.d}"
lib_transfer_dir="${SYSUPDATE_LIB_DIR:-/usr/lib/sysupdate.d}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low: only /usr/lib/sysupdate.d is scanned for transfers. systemd-sysupdate also reads *.transfer from /etc/sysupdate.d, /run/sysupdate.d and /usr/local/lib/sysupdate.d. Suppose an operator ships a custom /etc/sysupdate.d/*.transfer with Features=foo and enables foo. Then feature_seen[foo] is never set, so the drop-in is archived as a feature with no local transfer. Because feature_state[foo] can never become yes, the drop-in is never restored. That is a silent opt-out of a working feature. Scanning /etc (and /run) transfers too would close the gap. Alternatively, skip any feature that is defined by a transfer outside /usr/lib.

archive_suffix="${SYSUPDATE_ARCHIVE_SUFFIX:-.disabled}"
fetch_timeout="${SYSUPDATE_FETCH_TIMEOUT:-20}"

# systemd captures ExecStartPre stdout/stderr into the journal, so plain prints
# are enough; the wrapper keeps one message shape for both paths.
info() {
printf 'bluefin-sysupdate-migrate: %s\n' "$*"
}
warn() {
printf 'bluefin-sysupdate-migrate: warning: %s\n' "$*" >&2
}

# Per feature: "yes" once a remote manifest lists a match for one of its
# transfers, "unknown" when a manifest could not be read, else absent.
declare -A feature_state=()
declare -A feature_seen=()
declare -A manifest_cache=()

trim() {
local v="$1"
v="${v#"${v%%[![:space:]]*}"}"
printf '%s' "${v%"${v##*[![:space:]]}"}"
}

# Fetch (once per URL) the SHA256SUMS next to a transfer's Source Path.
# Sets the caller's "manifest" variable (no subshell, so the cache persists);
# returns 1 when it cannot be read.
fetch_manifest() {
local url="${1%/}/SHA256SUMS" body
if [ -z "${manifest_cache[$url]+x}" ]; then
if body="$(curl -fsSL --max-time "$fetch_timeout" -- "$url" 2>/dev/null)"; then
# `curl -f` only fails on HTTP >= 400, so a captive portal or proxy
# error page can come back 200 with an HTML body and be trusted as a
# manifest. Only trust a real SHA256SUMS: at least one "<64 hex>
# <filename>" line. Anything else is treated as unreachable, so a
# non-release response never archives a working feature.
if printf '%s\n' "$body" | grep -qE '^[0-9a-f]{64}([[:space:]]|\*)'; then
manifest_cache[$url]="$body"
else
manifest_cache[$url]="__FETCH_FAILED__"
fi
else
manifest_cache[$url]="__FETCH_FAILED__"
fi
fi
[ "${manifest_cache[$url]}" != "__FETCH_FAILED__" ] || return 1
manifest="${manifest_cache[$url]}"
}

# Turn a MatchPattern into an anchored regex; @v is the version wildcard.
pattern_regex() {
local pat="$1"
pat="$(printf '%s' "$pat" | sed -e 's/[][\.^$*+?(){}|\/]/\\&/g')"
printf '^%s$' "${pat//@v/[^[:space:]]+}"
}

# Record, for every transfer, whether the remote release still ships the files
# its features need.
collect_feature_state() {
local transfer line section key value feature features path pattern manifest regex name found
shopt -s nullglob
for transfer in "$lib_transfer_dir"/*.transfer; do

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness (high): a stalled node still lists the retired feature here.

collect_provided_features reads Features= from the running image's $lib_transfer_dir (/usr/lib/sysupdate.d, line 23). The stall in #688 happens before the update. At that point the running image still ships the retired feature's transfers: per #688, 26.11.2232 shipped 39-homebrew-integration.transfer / 40-homebrew.transfer. Only the newer release on the server dropped them.

So on the stalled node homebrew is in provided_features, nothing is archived, and systemd-sysupdate keeps reporting "No update needed". The helper only prunes after the node has already updated, when the drop-in no longer blocks anything. The same applies to every future retirement: image N, which has this helper and the transfer, cannot see that N+1 dropped the transfer.

The tests miss this because _make_tree(provided=[], enabled=["homebrew"]) models the post-update tree.

Fix: decide whether a feature is retired from the remote release, not the local tree. For each enabled feature, read its transfer's [Source] Path= / MatchPattern=. Fetch that source's SHA256SUMS and archive the drop-in only when the newest published version has no file matching the pattern. Sketch:

+# The local tree cannot see a retirement: the running image still ships the
+# retired transfer until the node updates. Ask the published release instead.
+source_key() {  # $1 transfer file, $2 key; first match inside [Source]
+    awk -v k="$2" '/^\[Source\]/{s=1;next} /^\[/{s=0} s && index($0,k"=")==1 {print substr($0,length(k)+2); exit}' "$1"
+}
+remote_ships() {  # $1 transfer file, $2 newest published version
+    local path pattern
+    path="$(source_key "$1" Path)"; pattern="$(source_key "$1" MatchPattern)"
+    curl -fsSL "${path%/}/SHA256SUMS" | awk '{sub(/^\*/,"",$2); print $2}' \
+        | grep -qxF -- "${pattern//@v/$2}"
+}

Derive the newest version from 10-usr.transfer's pattern against the same SHA256SUMS. When the fetch fails (offline), leave every drop-in in place. Check that curl ships in the image. If it does not, put the check in bluefin-host (Go, already has net/http per os-host.bst), next to bluefin-update-status. That also delivers part 2 of #688.

features="" path="" pattern="" section=""
while IFS= read -r line; do
line="$(trim "$line")"
case "$line" in
"["*"]") section="$line"; continue ;;
""|"#"*|";"*) continue ;;
esac
key="$(trim "${line%%=*}")"
value="$(trim "${line#*=}")"
case "$section:$key" in
"[Transfer]:Features") features="$features ${value//,/ }" ;;
"[Source]:Path") path="$value" ;;
"[Source]:MatchPattern") pattern="${value%% *}" ;;
esac
done < "$transfer"

[ -n "$features" ] || continue
found=unknown
if [ -n "$path" ] && [ -n "$pattern" ] && fetch_manifest "$path"; then
found=no
regex="$(pattern_regex "$pattern")"
while IFS= read -r line; do
name="${line##*[[:space:]]}"
name="${name#\*}"
if [[ "$name" =~ $regex ]]; then found=yes; break; fi
done <<< "$manifest"
fi
set -f
for feature in $features; do
feature_seen["$feature"]=1
case "$found:${feature_state[$feature]:-}" in
yes:*) feature_state["$feature"]=yes ;;
unknown:yes) ;;
unknown:*) feature_state["$feature"]=unknown ;;
no:yes|no:unknown) ;;
no:*) feature_state["$feature"]=no ;;
esac
done
set +f
done
}

main() {
if [ ! -d "$etc_feature_dir" ]; then
info "no ${etc_feature_dir} directory; nothing to do"
return 0
fi

collect_feature_state

local dir name changed
changed=0
shopt -s nullglob
for dir in "$etc_feature_dir"/*.feature.d; do
name="$(basename "$dir")"
name="${name%.feature.d}"
if [ -n "${feature_seen[$name]:-}" ] && [ "${feature_state[$name]:-}" != "no" ]; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness (still open from the earlier review): collect_feature_state scans only $lib_transfer_dir (/usr/lib/sysupdate.d, line 91). A feature that is defined by a transfer in /etc/sysupdate.d/*.transfer (or /run/sysupdate.d) is never put in feature_seen. So this check archives that feature's /etc/sysupdate.d/<name>.feature.d on every run of the helper. That happens before every update, through the ExecStartPre drop-in. systemd-sysupdate reads transfers from /etc and /run as well as /usr/lib, so an operator's own opt-in feature is silently disabled.

Fix: also parse *.transfer in /etc/sysupdate.d and /run/sysupdate.d in collect_feature_state. Then archive a feature with no transfer only when no transfer in any of the three dirs lists it. Add a test with a transfer under SYSUPDATE_ETC_DIR whose feature stays enabled.

continue
fi
if [ -e "${dir}${archive_suffix}" ]; then
warn "feature ${name} already archived as ${name}${archive_suffix}; leaving it in place"
continue
fi
mv -- "$dir" "${dir}${archive_suffix}"
info "archived ${dir} -> ${dir}${archive_suffix}: enabled feature ${name} is not in the release manifest and would otherwise make systemd-sysupdate report 'No update needed'"
changed=$((changed + 1))
done

# A release that ships the feature again restores the operator's opt-in.
for dir in "$etc_feature_dir"/*.feature.d"${archive_suffix}"; do
name="$(basename "$dir")"
name="${name%.feature.d"${archive_suffix}"}"
[ "${feature_state[$name]:-}" = "yes" ] || continue
if [ -e "${dir%"${archive_suffix}"}" ]; then
warn "feature ${name} has both an active and an archived drop-in; leaving the archive in place"
continue
fi
mv -- "$dir" "${dir%"${archive_suffix}"}"
info "restored ${dir%"${archive_suffix}"} from ${dir}: feature ${name} is in the release manifest again"
changed=$((changed + 1))
done

if [ "$changed" -eq 0 ]; then
info "no deprecated sysupdate features found; the next update is unaffected"
fi
}

main "$@"
Loading
Loading