From b595f66faad897338b1bb91023d44e9ca60c49f8 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:23:31 -0500 Subject: [PATCH 1/8] feat(initrd): print a failure summary on every console before rebooting The emergency drop-in printed 'journalctl -p err' to /dev/console (only the last console=, ttyS0 on the netboot UKI) and rebooted after 10 s, so the reason scrolled away and never reached the monitor. emergency.service now runs /usr/libexec/bluefin-boot-diagnostics failure-summary: each failed unit with a one-line hint (network, download, verity, Ignition stage) and the lines its processes logged, the network links when the network or download failed, and the last errors, written to every active console. It stays up for 60 s (bluefin.failure_delay credential, 10 s floor against hot loops) before the unattended reboot. Type=idle keeps the isolate's status lines out of the summary. The helper also carries the pull RAM check and the Ignition karg warning wired up in the next commits. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../system/emergency.service.d/10-reboot.conf | 15 +- .../usr/libexec/bluefin-boot-diagnostics | 292 ++++++++++++++ tests/unit/test_initrd_diagnostics.py | 381 ++++++++++++++++++ 3 files changed, 684 insertions(+), 4 deletions(-) create mode 100755 files/initrd/usr/libexec/bluefin-boot-diagnostics create mode 100644 tests/unit/test_initrd_diagnostics.py diff --git a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf index db23c58b..ff82f25f 100644 --- a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf +++ b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf @@ -1,11 +1,18 @@ [Unit] # An unattended server cannot use an emergency shell (root is locked). -# Print what failed and reboot, so systemd-boot's boot counting can fall -# back to the previous image after the new one exhausts its tries. +# Print what failed and reboot, so a diskless node retries and systemd-boot's +# boot counting can fall back to the previous image after the new one +# exhausts its tries. SuccessAction=reboot-force FailureAction=reboot-force [Service] -Type=oneshot +# idle: start once the isolate's stop jobs are done, so their status lines +# do not interleave with the summary. +Type=idle +# How long the summary stays on screen before the reboot (seconds, default +# 60); the bluefin.failure_delay= kernel argument overrides it where the +# command line is not locked by Secure Boot. +ImportCredential=bluefin.failure_delay ExecStart= -ExecStart=-/usr/bin/bash -c 'echo "Boot failed; errors follow, rebooting in 10s." >/dev/console; journalctl -b -p err --no-pager -o short-monotonic >/dev/console 2>&1; sleep 10' +ExecStart=-/usr/libexec/bluefin-boot-diagnostics failure-summary diff --git a/files/initrd/usr/libexec/bluefin-boot-diagnostics b/files/initrd/usr/libexec/bluefin-boot-diagnostics new file mode 100755 index 00000000..f7c77130 --- /dev/null +++ b/files/initrd/usr/libexec/bluefin-boot-diagnostics @@ -0,0 +1,292 @@ +#!/usr/bin/bash +# bluefin-boot-diagnostics — say on the console why an initrd boot fails. +# +# check-pull UNIT Run before UNIT (a systemd-import@.service instance) +# downloads the OS image into RAM. HEADs the image URL and +# fails with a plain "needs ~X MiB RAM, have Y" message if +# the image cannot fit, or if the server cannot be reached. +# ignition-kargs Warn that ignition.* kernel arguments are ignored. +# failure-summary ExecStart of emergency.service: print the failed units, +# their last log lines and the last errors, then wait +# (bluefin.failure_delay, default 60 s) so the reboot that +# follows does not scroll them away. +# +# Messages go to every active console (/sys/class/tty/console/active, e.g. the +# monitor and the serial port). /dev/console, where systemd prints its status +# lines, is only the last console= on the kernel command line. +# +# bash builtins, coreutils, curl, systemctl and journalctl only: the initrd +# has no sed or awk. + +set -uo pipefail +export LC_ALL=C + +PROG="${0##*/}" +DOCS=https://github.com/projectbluefin/server/blob/main/docs/skills/diskless-troubleshooting.md +MEMINFO="${BLUEFIN_DIAG_MEMINFO:-/proc/meminfo}" +CMDLINE="${BLUEFIN_DIAG_CMDLINE:-/proc/cmdline}" +CONSOLES="${BLUEFIN_DIAG_CONSOLES:-/sys/class/tty/console/active}" +DEVDIR="${BLUEFIN_DIAG_DEVDIR:-/dev}" +NETDIR="${BLUEFIN_DIAG_NETDIR:-/sys/class/net}" +# systemd-importd stores machine images under /run/machines in the initrd: +# the /run tmpfs, which systemd caps at 20% of RAM. +PULL_DIR=/run +# Room /run and RAM must keep after the image has landed. +MARGIN_MIB=64 +DELAY_NAME=bluefin.failure_delay +DEFAULT_DELAY=60 +# The reboot is intentional (unattended retry, boot counting); a floor keeps +# a persistent failure from turning into a hot reboot loop. +MIN_DELAY=10 +MAX_DELAY=86400 +BAR='==============================================================================' + +# log PRIORITY MESSAGE: to the journal (stderr, sd-daemon priority prefix). +log() { printf '<%s>%s: %s\n' "$1" "${PROG}" "$2" >&2; } + +# Copy stdin to every active console. A console that blocks must not hold up +# the boot, so each write gets a timeout. +to_consoles() { + local text c active=console + text="$(cat)" + [ -r "${CONSOLES}" ] && active="$(<"${CONSOLES}")" + for c in ${active}; do + timeout 5 tee -a "${DEVDIR}/${c}" >/dev/null <<<"${text}" || true + done +} + +# shout PRIORITY LINE...: log every line and print them framed on the consoles. +shout() { + local pri="$1" line + shift + for line in "$@"; do log "${pri}" "${line}"; done + { printf '\n%s\n' "${BAR}"; printf '%s\n' "$@"; printf '%s\n' "${BAR}"; } | to_consoles +} + +mib() { printf '%s' "$(( ($1 + 1048575) / 1048576 ))"; } + +meminfo_kib() { + local key value _ + while read -r key value _; do + [ "${key}" = "$1:" ] && { printf '%s' "${value}"; return 0; } + done < "${MEMINFO}" + return 1 +} + +# One line per network link: name, operstate, carrier. sysfs only, so it also +# works after emergency.target has stopped systemd-networkd. +links() { + local dev name state carrier out=() + for dev in "${NETDIR}"/*; do + name="${dev##*/}" + [ -e "${dev}/device" ] || continue + state="$(<"${dev}/operstate")" 2>/dev/null || state=unknown + carrier="$(<"${dev}/carrier")" 2>/dev/null || carrier=0 + [ "${carrier}" = 1 ] && carrier=carrier || carrier="no carrier" + out+=("${name} ${state} (${carrier})") + done + if [ "${#out[@]}" -eq 0 ]; then + printf 'no network interface found (missing NIC driver in the initrd?)\n' + else + printf '%s\n' "${out[@]}" + fi +} + +check_pull() { + local unit="${1:?usage: ${PROG} check-pull }" + local desc url out rc status='' len='' line lower + # systemd-import-generator describes each pull unit as "Download of "; + # take the URL it resolved (rd.systemd.pull, bootorigin, or the import.pull + # credential) rather than deriving it a second time. + desc="$(systemctl show -P Description "${unit}" 2>/dev/null)" || desc= + url="${desc#Download of }" + case "${url}" in + http://*|https://*) ;; + *) log 6 "${unit}: no http(s) source (${desc:-no description}); nothing to check"; return 0 ;; + esac + + out="$(curl -sSIL --retry 3 --retry-delay 2 --connect-timeout 10 --max-time 120 "${url}" 2>&1)" + rc=$? + if [ "${rc}" -ne 0 ]; then + local -a netlines + mapfile -t netlines < <(links) + shout 3 "BLUEFIN: CANNOT REACH THE OS IMAGE SERVER" \ + "URL: ${url}" \ + "${out##*$'\n'}" \ + "Network links:" "${netlines[@]/#/ }" \ + "Check the cable, DHCP and the boot server. ${DOCS}#failure-modes" + return 1 + fi + + # Headers of every response in the redirect chain; keep the last one. + while IFS= read -r line; do + line="${line%$'\r'}" + lower="${line,,}" + case "${lower}" in + http/*) status="${line#* }"; status="${status%% *}"; len= ;; + content-length:*) len="${line#*:}"; len="${len//[[:space:]]/}" ;; + esac + done <<<"${out}" + + case "${status}" in + 2??) ;; + 405|501) + log 4 "${url}: HEAD not supported (HTTP ${status}); cannot check the image size" + return 0 ;; + *) + shout 3 "BLUEFIN: THE OS IMAGE SERVER ANSWERED HTTP ${status:-?}" \ + "URL: ${url}" \ + "The boot server must serve the OS image next to the netboot UKI. ${DOCS}#failure-modes" + return 1 ;; + esac + if ! [[ "${len}" =~ ^[0-9]+$ ]]; then + log 4 "${url}: no Content-Length; cannot check the image size" + return 0 + fi + + local total_kib avail_kib bavail bsize btotal + total_kib="$(meminfo_kib MemTotal)" || { log 4 "no MemTotal in ${MEMINFO}"; return 0; } + avail_kib="$(meminfo_kib MemAvailable)" || avail_kib="${total_kib}" + read -r bavail bsize btotal < <(stat -f -c '%a %S %b' "${PULL_DIR}") + if ! [[ "${bavail:-}${bsize:-}${btotal:-}" =~ ^[0-9]+$ ]] || [ "${btotal}" -eq 0 ] || [ "${total_kib}" -eq 0 ]; then + log 4 "cannot read the size of ${PULL_DIR} or of RAM; skipping the RAM check" + return 0 + fi + + local need=$(( len + MARGIN_MIB * 1048576 )) + local run_free=$(( bavail * bsize )) run_size=$(( btotal * bsize )) + local total=$(( total_kib * 1024 )) avail=$(( avail_kib * 1024 )) + if [ "${need}" -le "${run_free}" ] && [ "${need}" -le "${avail}" ]; then + log 6 "OS image $(mib "${len}") MiB fits: /run has $(mib "${run_free}") MiB free, MemAvailable $(mib "${avail}") MiB" + return 0 + fi + + # /run is sized as a fixed share of RAM, so scale what it lacks up to RAM. + # Work in KiB so the product cannot overflow. + local used_kib=$(( (run_size - run_free) / 1024 )) need_kib=$(( need / 1024 )) + local want_run_kib=$(( (used_kib + need_kib) * total_kib / (run_size / 1024) )) + local want_run=$(( want_run_kib * 1024 )) + local want_mem=$(( total - avail + need )) + local want=$(( want_run > want_mem ? want_run : want_mem )) + # Round up to 64 MiB. + want=$(( (want + 67108863) / 67108864 * 67108864 )) + shout 2 "BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT" \ + "This machine needs ~$(mib "${want}") MiB RAM, have $(mib "${total}") MiB." \ + "The OS image ${url##*/} ($(mib "${len}") MiB) is downloaded into /run, a RAM" \ + "file system capped at $(( run_size * 100 / total ))% of RAM: $(mib "${run_free}") MiB free there, MemAvailable $(mib "${avail}") MiB." \ + "Add RAM or install to disk. ${DOCS}#minimum-ram" + return 1 +} + +# Keys of ignition.* kernel arguments (values may carry secrets). +ignition_karg_keys() { + local -a args + local arg keys=() + read -ra args < "${CMDLINE}" || true + for arg in "${args[@]}"; do + case "${arg}" in + ignition.*) keys+=("${arg%%=*}") ;; + esac + done + [ "${#keys[@]}" -gt 0 ] && printf '%s\n' "${keys[*]}" +} + +ignition_kargs() { + local keys + keys="$(ignition_karg_keys)" || return 0 + shout 4 "BLUEFIN: IGNORED KERNEL ARGUMENTS: ${keys}" \ + "Bluefin Server never reads Ignition settings from the kernel command line." \ + "Pass the config as the ignition.config or ignition.config.url system credential" \ + "(SMBIOS type 11, QEMU fw_cfg, or /loader/credentials/ on the ESP), or serve" \ + "bluefin-node.ign next to the netboot UKI. ${DOCS}#ignition" +} + +# What a failed unit most likely means. +hint() { + local stage + case "$1" in + bluefin-pull-check@*) + echo "the OS image cannot be downloaded into RAM (see the message above)" ;; + systemd-import@*) + echo "downloading or verifying the OS image failed: server reachable? SHA256SUMS signed by a key this image trusts? room in /run?" ;; + systemd-loop@*) + echo "attaching the OS image as a block device failed (also fails when the download did not happen)" ;; + systemd-networkd-wait-online.service) + echo "no network link became routable within 300 s: check the link and that a DHCP server answers" ;; + systemd-veritysetup@*|sysusr-usr.mount|sysroot-usr.mount) + echo "/usr could not be verified or mounted: the OS image does not match the usrhash= of this UKI" ;; + bluefin-ignition-credentials.service) + echo "staging the Ignition config failed (invalid ignition.config.url?). ${DOCS}#ignition" ;; + ignition-*.service) + stage="${1#ignition-}" + stage="${stage%.service}" + echo "Ignition stage '${stage}' failed. Every diskless boot runs every stage, so the config must be idempotent. ${DOCS}#ignition" ;; + esac +} + +failure_delay() { + local value='' arg + local -a args + if [ -r "${CREDENTIALS_DIRECTORY:-/nonexistent}/${DELAY_NAME}" ]; then + value="$(<"${CREDENTIALS_DIRECTORY}/${DELAY_NAME}")" + fi + read -ra args < "${CMDLINE}" || true + for arg in "${args[@]}"; do + case "${arg}" in "${DELAY_NAME}="*) value="${arg#*=}" ;; esac + done + value="${value//[[:space:]]/}" + [[ "${value}" =~ ^[0-9]{1,6}$ ]] || value="${DEFAULT_DELAY}" + value=$(( 10#${value} )) + [ "${value}" -lt "${MIN_DELAY}" ] && value="${MIN_DELAY}" + [ "${value}" -gt "${MAX_DELAY}" ] && value="${MAX_DELAY}" + printf '%s' "${value}" +} + +indent() { local l; while IFS= read -r l; do printf '%s%s\n' "$1" "${l}"; done; } + +failure_summary() { + local delay unit h keys shown=0 + local -a units matches + delay="$(failure_delay)" + mapfile -t units < <(systemctl list-units --state=failed --no-legend --plain --full 2>/dev/null | cut -d' ' -f1) + { + printf '\n%s\n' "${BAR}" + printf 'BLUEFIN: BOOT FAILED\n' + if [ "${#units[@]}" -eq 0 ]; then + printf 'No unit failed: a device or job timed out (see the errors below).\n' + fi + for unit in "${units[@]}"; do + [ -n "${unit}" ] || continue + shown=$(( shown + 1 )) + [ "${shown}" -le 6 ] || { printf '(%s more failed units)\n' "$(( ${#units[@]} - 6 ))"; break; } + printf '\nFAILED: %s\n' "${unit}" + h="$(hint "${unit}")" + [ -z "${h}" ] || printf ' -> %s\n' "${h}" + # What the unit's own processes logged (not systemd's lines about it); + # a download's errors come from systemd-importd. + matches=("_SYSTEMD_UNIT=${unit}") + case "${unit}" in systemd-import@*) matches+=(_SYSTEMD_UNIT=systemd-importd.service) ;; esac + journalctl -b "${matches[@]}" -n 6 -o cat --no-pager 2>/dev/null | indent ' | ' + case "${unit}" in + systemd-networkd-wait-online.service|systemd-import@*|bluefin-pull-check@*) + printf ' network links:\n'; links | indent ' ' ;; + esac + done + if keys="$(ignition_karg_keys)"; then + printf '\nIgnored kernel arguments: %s (use the ignition.config credential instead)\n' "${keys}" + fi + printf '\nLast errors (journalctl -b -p err):\n' + journalctl -b -p err -n 15 -o short-monotonic --no-pager 2>/dev/null | indent ' ' + printf '\nHelp: %s\n' "${DOCS}" + printf 'Rebooting in %s s (set the %s credential to change the delay).\n' "${delay}" "${DELAY_NAME}" + printf '%s\n' "${BAR}" + } | to_consoles + sleep "${delay}" +} + +case "${1:-}" in + check-pull) shift; check_pull "$@" ;; + ignition-kargs) ignition_kargs ;; + failure-summary) failure_summary ;; + *) printf 'usage: %s check-pull UNIT | ignition-kargs | failure-summary\n' "${PROG}" >&2; exit 2 ;; +esac diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py new file mode 100644 index 00000000..b72a08a7 --- /dev/null +++ b/tests/unit/test_initrd_diagnostics.py @@ -0,0 +1,381 @@ +"""Initrd boot-failure diagnostics: units, drop-ins and the libexec helper. + +files/initrd/usr/libexec/bluefin-boot-diagnostics runs in the initrd only, so +these tests drive it under bash on the host with systemctl, journalctl, stat +and sleep stubbed on $PATH, a fake /proc/meminfo, fake consoles, and a local +http.server answering the HEAD request the RAM check makes. +""" + +from __future__ import annotations + +import http.server +import os +import re +import shutil +import socket +import subprocess +import threading +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "initrd" / "usr" / "lib" / "systemd" / "system" +IGN_UNITS = ROOT / "files" / "initrd-ignition" / "usr" / "lib" / "systemd" / "system" +HELPER = ROOT / "files" / "initrd" / "usr" / "libexec" / "bluefin-boot-diagnostics" +BOOT = ROOT / "elements" / "oci" / "bluefin-server-boot.bst" +MIB = 1024 * 1024 +IMPORT_UNIT = "systemd-import@run-machines-rootdisk.raw.service" + + +def unit(path: Path) -> dict[str, dict[str, list[str]]]: + """Parse a unit file keeping every value of repeated keys.""" + sections: dict[str, dict[str, list[str]]] = {} + current = None + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith(("#", ";")): + continue + if line.startswith("["): + current = sections.setdefault(line.strip("[]"), {}) + continue + key, _, value = line.partition("=") + current.setdefault(key, []).append(value) + return sections + + +def test_emergency_prints_the_summary_then_reboots() -> None: + u = unit(UNITS / "emergency.service.d" / "10-reboot.conf") + assert u["Unit"]["SuccessAction"] == ["reboot-force"] + assert u["Unit"]["FailureAction"] == ["reboot-force"] + # Reset the stock sulogin ExecStart, then run the summary (never fatal). + assert u["Service"]["ExecStart"] == ["", "-/usr/libexec/bluefin-boot-diagnostics failure-summary"] + assert u["Service"]["ImportCredential"] == ["bluefin.failure_delay"] + assert u["Service"]["Type"] == ["idle"] + + +@pytest.mark.parametrize("stage", ["fetch-offline", "fetch", "disks", "mount", "files"]) +def test_ignition_failures_reach_the_summary(stage: str) -> None: + u = unit(IGN_UNITS / f"ignition-{stage}.service") + assert u["Unit"]["OnFailure"] == ["emergency.target"] + + +def test_helper_is_executable_bash_without_sed_or_awk() -> None: + text = HELPER.read_text(encoding="utf-8") + assert text.startswith("#!/usr/bin/bash\n") + assert os.access(HELPER, os.X_OK) + subprocess.run(["bash", "-n", str(HELPER)], check=True) + code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#")) + assert not re.search(r"\b(sed|awk|gawk)\b", code), "the initrd has no sed or awk" + + +@pytest.mark.skipif(shutil.which("shellcheck") is None, reason="shellcheck not installed") +def test_helper_is_shellcheck_clean() -> None: + subprocess.run(["shellcheck", "-S", "style", str(HELPER)], check=True) + + +class _Handler(http.server.BaseHTTPRequestHandler): + routes: dict[str, tuple[int, dict[str, str]]] = {} + + def do_HEAD(self) -> None: # noqa: N802 (http.server API) + status, headers = self.routes.get(self.path, (404, {"Content-Length": "0"})) + self.send_response(status) + for key, value in headers.items(): + self.send_header(key, value) + self.end_headers() + + def log_message(self, *args) -> None: + pass + + +@pytest.fixture(scope="module") +def server(): + httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + yield f"http://127.0.0.1:{httpd.server_address[1]}", _Handler.routes + httpd.shutdown() + + +STUBS = { + "systemctl": """#!/bin/bash +case "$1" in + show) printf '%s\\n' "${FAKE_DESCRIPTION}" ;; + list-units) for u in ${FAKE_FAILED:-}; do printf '%s loaded failed failed Fake unit\\n' "$u"; done ;; +esac +""", + "journalctl": """#!/bin/bash +printf 'journal-line args: %s\\n' "$*" +""", + "stat": """#!/bin/bash +printf '%s\\n' "${FAKE_STATFS}" +""", + "sleep": """#!/bin/bash +printf '%s\\n' "$1" >> "${FAKE_SLEEP_LOG}" +""", +} + + +class Env: + def __init__(self, tmp: Path) -> None: + self.tmp = tmp + self.bin = tmp / "bin" + self.bin.mkdir() + for name, body in STUBS.items(): + (self.bin / name).write_text(body, encoding="utf-8") + (self.bin / name).chmod(0o755) + self.dev = tmp / "dev" + self.dev.mkdir() + (tmp / "consoles").write_text("tty0 ttyS0\n", encoding="utf-8") + self.net = tmp / "net" + nic = self.net / "enp1s0" + (nic / "device").mkdir(parents=True) + (nic / "operstate").write_text("down\n", encoding="utf-8") + (nic / "carrier").write_text("0\n", encoding="utf-8") + (self.net / "lo").mkdir() # virtual: no device link, not listed + self.cmdline("root=tmpfs") + self.meminfo(total_kib=4_000_000, avail_kib=3_600_000) + self.statfs(free_mib=800, size_mib=800) + self.creds = tmp / "creds" + self.creds.mkdir() + self.env = dict( + os.environ, + PATH=f"{self.bin}:{os.environ['PATH']}", + BLUEFIN_DIAG_MEMINFO=str(tmp / "meminfo"), + BLUEFIN_DIAG_CMDLINE=str(tmp / "cmdline"), + BLUEFIN_DIAG_CONSOLES=str(tmp / "consoles"), + BLUEFIN_DIAG_DEVDIR=str(self.dev), + BLUEFIN_DIAG_NETDIR=str(self.net), + CREDENTIALS_DIRECTORY=str(self.creds), + FAKE_SLEEP_LOG=str(tmp / "sleep.log"), + FAKE_DESCRIPTION="", + FAKE_FAILED="", + ) + + def cmdline(self, text: str) -> None: + (self.tmp / "cmdline").write_text(text + "\n", encoding="utf-8") + + def meminfo(self, total_kib: int, avail_kib: int) -> None: + self.total_kib = total_kib + self.avail_kib = avail_kib + (self.tmp / "meminfo").write_text( + f"MemTotal: {total_kib} kB\nMemFree: {avail_kib} kB\n" + f"MemAvailable: {avail_kib} kB\nBuffers: 0 kB\n", + encoding="utf-8", + ) + + def statfs(self, free_mib: int, size_mib: int) -> None: + self.run_free = free_mib * MIB + self.run_size = size_mib * MIB + self.fake_statfs = f"{free_mib * 256} 4096 {size_mib * 256}" + + def run(self, *args: str, **extra: str) -> subprocess.CompletedProcess: + env = dict(self.env, FAKE_STATFS=self.fake_statfs, **extra) + return subprocess.run( + ["bash", str(HELPER), *args], capture_output=True, text=True, env=env, timeout=60 + ) + + def console(self, name: str = "ttyS0") -> str: + path = self.dev / name + return path.read_text(encoding="utf-8") if path.exists() else "" + + def sleeps(self) -> list[str]: + log = self.tmp / "sleep.log" + return log.read_text(encoding="utf-8").split() if log.exists() else [] + + +@pytest.fixture +def env(tmp_path: Path) -> Env: + return Env(tmp_path) + + +def check_pull(env: Env, url: str) -> subprocess.CompletedProcess: + return env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION=f"Download of {url}") + + +def test_image_that_fits_passes_quietly(env: Env, server) -> None: + base, routes = server + routes["/fits.raw"] = (200, {"Content-Length": str(540 * MIB)}) + result = check_pull(env, f"{base}/fits.raw") + assert result.returncode == 0, result.stderr + assert "540 MiB fits" in result.stderr + assert env.console() == "" + + +def _expected_mib(env: Env, size: int) -> int: + """Mirror of the helper's estimate, rounded up to 64 MiB.""" + need = size + 64 * MIB + used_kib = (env.run_size - env.run_free) // 1024 + want_run = (used_kib + need // 1024) * env.total_kib // (env.run_size // 1024) * 1024 + want_mem = env.total_kib * 1024 - env.avail_kib * 1024 + need + want = max(want_run, want_mem) + return -(-want // (64 * MIB)) * 64 + + +def test_image_larger_than_run_fails_with_the_ram_it_needs(env: Env, server) -> None: + base, routes = server + size = 540 * MIB + routes["/bluefin-server_1.raw"] = (200, {"Content-Length": str(size)}) + # 2 GB machine: /run is 20% of it. + env.meminfo(total_kib=2_000_000, avail_kib=1_800_000) + env.statfs(free_mib=380, size_mib=390) + result = check_pull(env, f"{base}/bluefin-server_1.raw") + assert result.returncode == 1 + want = _expected_mib(env, size) + assert 2900 < want < 3200 + for tty in ("tty0", "ttyS0"): + text = env.console(tty) + assert "NOT ENOUGH RAM FOR A DISKLESS BOOT" in text + assert f"needs ~{want} MiB RAM, have 1954 MiB" in text + assert "bluefin-server_1.raw (540 MiB)" in text + assert "capped at 19% of RAM: 380 MiB free" in text + assert f"<2>bluefin-boot-diagnostics: This machine needs ~{want} MiB RAM" in result.stderr + + +def test_low_memavailable_fails_even_when_run_has_room(env: Env, server) -> None: + base, routes = server + size = 300 * MIB + routes["/small.raw"] = (200, {"Content-Length": str(size)}) + env.meminfo(total_kib=4_000_000, avail_kib=200_000) + env.statfs(free_mib=780, size_mib=800) + result = check_pull(env, f"{base}/small.raw") + assert result.returncode == 1 + assert f"needs ~{_expected_mib(env, size)} MiB RAM, have 3907 MiB" in env.console() + + +def test_redirects_are_followed_to_the_final_size(env: Env, server) -> None: + base, routes = server + routes["/redirect.raw"] = (302, {"Location": "/huge.raw", "Content-Length": "0"}) + routes["/huge.raw"] = (200, {"Content-Length": str(4000 * MIB)}) + result = check_pull(env, f"{base}/redirect.raw") + assert result.returncode == 1 + assert "redirect.raw (4000 MiB)" in env.console() + + +def test_missing_image_fails_with_the_http_status(env: Env, server) -> None: + base, _ = server + result = check_pull(env, f"{base}/missing.raw") + assert result.returncode == 1 + assert "THE OS IMAGE SERVER ANSWERED HTTP 404" in env.console() + assert f"URL: {base}/missing.raw" in env.console() + + +@pytest.mark.parametrize("status", [405, 501]) +def test_servers_without_head_are_not_blocked(env: Env, server, status: int) -> None: + base, routes = server + routes[f"/nohead{status}.raw"] = (status, {"Content-Length": "0"}) + result = check_pull(env, f"{base}/nohead{status}.raw") + assert result.returncode == 0 + assert "HEAD not supported" in result.stderr + assert env.console() == "" + + +def test_missing_content_length_skips_the_check(env: Env, server) -> None: + base, routes = server + routes["/chunked.raw"] = (200, {"Transfer-Encoding": "chunked"}) + result = check_pull(env, f"{base}/chunked.raw") + assert result.returncode == 0 + assert "no Content-Length" in result.stderr + + +def test_unreachable_server_names_the_url_and_the_links(env: Env) -> None: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + result = check_pull(env, f"http://127.0.0.1:{port}/img.raw") + assert result.returncode == 1 + text = env.console() + assert "CANNOT REACH THE OS IMAGE SERVER" in text + assert f"URL: http://127.0.0.1:{port}/img.raw" in text + assert "curl: (7)" in text + assert " enp1s0 down (no carrier)" in text + assert "lo " not in text + + +def test_non_http_sources_are_not_checked(env: Env) -> None: + result = env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION="Download of file:///run/img.raw") + assert result.returncode == 0 + assert env.console() == "" + + +def test_ignition_kargs_are_named_without_their_values(env: Env) -> None: + env.cmdline("root=tmpfs ignition.config.url=http://10.0.0.1/secret-token.ign ignition.platform.id=metal quiet") + result = env.run("ignition-kargs") + assert result.returncode == 0 + text = env.console("tty0") + assert "IGNORED KERNEL ARGUMENTS: ignition.config.url ignition.platform.id" in text + assert "ignition.config or ignition.config.url system credential" in text + assert "secret-token" not in text + result.stderr + assert "<4>bluefin-boot-diagnostics: BLUEFIN: IGNORED KERNEL ARGUMENTS" in result.stderr + + +def test_no_ignition_kargs_no_warning(env: Env) -> None: + env.cmdline("root=tmpfs rd.systemd.pull=raw,machine:rootdisk:http://x/y.raw") + result = env.run("ignition-kargs") + assert result.returncode == 0 + assert result.stderr == "" + assert env.console() == "" + + +def test_failure_summary_explains_each_failed_unit(env: Env) -> None: + result = env.run( + "failure-summary", + FAKE_FAILED=f"ignition-disks.service {IMPORT_UNIT} systemd-networkd-wait-online.service", + ) + assert result.returncode == 0 + text = env.console("tty0") + assert text == env.console("ttyS0") + assert "BLUEFIN: BOOT FAILED" in text + assert "FAILED: ignition-disks.service" in text + assert "Ignition stage 'disks' failed" in text + assert "must be idempotent" in text + assert f"FAILED: {IMPORT_UNIT}" in text + assert "SHA256SUMS signed by a key this image trusts" in text + assert "no network link became routable within 300 s" in text + assert "enp1s0 down (no carrier)" in text + assert ( + f"| journal-line args: -b _SYSTEMD_UNIT={IMPORT_UNIT} _SYSTEMD_UNIT=systemd-importd.service -n 6 -o cat --no-pager" + in text + ), "a failed download shows systemd-importd's reason" + assert "| journal-line args: -b _SYSTEMD_UNIT=ignition-disks.service -n 6" in text + assert "Last errors (journalctl -b -p err):" in text + assert "docs/skills/diskless-troubleshooting.md\n" in text + assert "Rebooting in 60 s" in text + assert env.sleeps() == ["60"] + + +def test_failure_summary_without_failed_units_points_at_timeouts(env: Env) -> None: + env.cmdline("root=tmpfs ignition.config.url=http://x/y.ign") + env.run("failure-summary") + text = env.console() + assert "No unit failed: a device or job timed out" in text + assert "Ignored kernel arguments: ignition.config.url" in text + + +@pytest.mark.parametrize( + "credential,karg,expected", + [ + (None, None, "60"), + ("120\n", None, "120"), + ("3", None, "10"), # floor: no hot reboot loop + ("0600", None, "600"), # decimal, not octal + ("999999999", None, "60"), # not a sane number + ("soon", None, "60"), + ("120", "30", "30"), # the kernel argument wins where it can be set + (None, "100000", "86400"), + ], +) +def test_failure_delay(env: Env, credential: str | None, karg: str | None, expected: str) -> None: + if credential is not None: + (env.creds / "bluefin.failure_delay").write_text(credential, encoding="utf-8") + if karg is not None: + env.cmdline(f"root=tmpfs bluefin.failure_delay={karg}") + env.run("failure-summary") + assert env.sleeps() == [expected] + assert f"Rebooting in {expected} s" in env.console() + + +def test_unknown_mode_is_a_usage_error(env: Env) -> None: + result = env.run("bogus") + assert result.returncode == 2 + assert "usage:" in result.stderr From 59aff9e98750b79127018ba1fa837e85d608aad3 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:23:41 -0500 Subject: [PATCH 2/8] feat(initrd): check that the OS image fits in RAM before the diskless pull The netboot initrd downloads the whole DDI (540 MiB) into /run, a tmpfs systemd caps at 20% of RAM. A small machine failed deep in the download, and any failed download only surfaced when the /usr devices timed out 300 s later. A drop-in makes every systemd-import@.service require bluefin-pull-check@%i.service, which takes the URL the import generator resolved (rd.systemd.pull, bootorigin, or the import.pull credential), HEADs it and fails with 'needs ~X MiB RAM, have Y' when the image plus 64 MiB does not fit in /run or MemAvailable, or with the URL and link state when the server is unreachable or answers an error. Both the check and the download go straight to the failure summary on failure. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../system/bluefin-pull-check@.service | 20 ++++++++++++++++++ .../10-bluefin-pull-check.conf | 12 +++++++++++ tests/unit/test_initrd_diagnostics.py | 21 +++++++++++++++++++ 3 files changed, 53 insertions(+) create mode 100644 files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service create mode 100644 files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf diff --git a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service new file mode 100644 index 00000000..d31862e4 --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service @@ -0,0 +1,20 @@ +[Unit] +Description=Check that the OS image fits in RAM before downloading it +Documentation=file:/usr/libexec/bluefin-boot-diagnostics +ConditionPathExists=/etc/initrd-release +DefaultDependencies=no +# Pulled in and ordered before systemd-import@%i.service by its drop-in +# systemd-import@.service.d/10-bluefin-pull-check.conf. Needs the network, +# like the download itself. +Wants=network-online.target +After=network-online.target + +# Print why on the console and reboot (emergency.service.d/10-reboot.conf) +# rather than start a download that cannot fit. +OnFailure=emergency.target +OnFailureJobMode=isolate + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service diff --git a/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf new file mode 100644 index 00000000..d7b7417c --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf @@ -0,0 +1,12 @@ +[Unit] +# The diskless initrd downloads the whole OS image into /run (RAM). Check +# first that it fits, so a small machine fails with "needs ~X MiB RAM" +# instead of an out-of-space error deep in the download. +Requires=bluefin-pull-check@%i.service +After=bluefin-pull-check@%i.service + +# Without this a failed download only surfaces when the /usr devices time out +# (systemd.default_device_timeout_sec=300); go straight to the failure +# summary and reboot instead. +OnFailure=emergency.target +OnFailureJobMode=isolate diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py index b72a08a7..5c0456e0 100644 --- a/tests/unit/test_initrd_diagnostics.py +++ b/tests/unit/test_initrd_diagnostics.py @@ -54,6 +54,27 @@ def test_emergency_prints_the_summary_then_reboots() -> None: assert u["Service"]["Type"] == ["idle"] +def test_every_download_is_preceded_by_the_ram_check() -> None: + u = unit(UNITS / "systemd-import@.service.d" / "10-bluefin-pull-check.conf") + assert u["Unit"]["Requires"] == ["bluefin-pull-check@%i.service"] + assert u["Unit"]["After"] == ["bluefin-pull-check@%i.service"] + # A failed download goes straight to the summary, not a 300 s device timeout. + assert u["Unit"]["OnFailure"] == ["emergency.target"] + assert u["Unit"]["OnFailureJobMode"] == ["isolate"] + + +def test_ram_check_unit_checks_its_import_instance() -> None: + u = unit(UNITS / "bluefin-pull-check@.service") + assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"] + assert u["Unit"]["DefaultDependencies"] == ["no"] + assert "network-online.target" in u["Unit"]["After"] + assert u["Unit"]["OnFailure"] == ["emergency.target"] + assert u["Unit"]["OnFailureJobMode"] == ["isolate"] + assert u["Service"]["ExecStart"] == [ + "/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service" + ] + + @pytest.mark.parametrize("stage", ["fetch-offline", "fetch", "disks", "mount", "files"]) def test_ignition_failures_reach_the_summary(stage: str) -> None: u = unit(IGN_UNITS / f"ignition-{stage}.service") From 2223d1fe5c28af3c50d5f4367ba163f51df317ae Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:23:51 -0500 Subject: [PATCH 3/8] feat(initrd): warn about ignored ignition.* kernel arguments Ignition configs come only from the ignition.config / ignition.config.url credentials or bluefin-node.ign; ignition.config.url= and friends on the command line were silently ignored. bluefin-ignition-kargs.service, linked into initrd.target.wants, prints a framed warning naming the ignition.* keys it found (never their values) and how to pass the config instead; the failure summary repeats it. The UKI build now also checks that the diagnostics helper and the tools it runs are in the initrd. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- elements/oci/bluefin-server-boot.bst | 5 +++++ .../systemd/system/bluefin-ignition-kargs.service | 13 +++++++++++++ tests/unit/test_initrd_diagnostics.py | 10 ++++++++++ 3 files changed, 28 insertions(+) create mode 100644 files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service diff --git a/elements/oci/bluefin-server-boot.bst b/elements/oci/bluefin-server-boot.bst index 9437a2c3..49dbd85e 100644 --- a/elements/oci/bluefin-server-boot.bst +++ b/elements/oci/bluefin-server-boot.bst @@ -154,6 +154,7 @@ config: ln -sf ../systemd-resolved.service "${wants}/initrd.target.wants/systemd-resolved.service" ln -sf ../imports.target "${wants}/initrd.target.wants/imports.target" ln -sf ../bluefin-populate-etc.service "${wants}/initrd.target.wants/bluefin-populate-etc.service" + ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service" ln -sf ../systemd-networkd-wait-online.service "${wants}/network-online.target.wants/systemd-networkd-wait-online.service" for tool in systemd-importd systemd-pull; do [ -x "usr/lib/systemd/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } @@ -162,6 +163,10 @@ config: [ -x "usr/bin/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } done [ -x usr/lib/systemd/systemd-veritysetup ] || { echo "ERROR: initrd lacks systemd-veritysetup" >&2; exit 1; } + # bluefin-boot-diagnostics (failure summary, RAM check) and what it runs. + for tool in libexec/bluefin-boot-diagnostics bin/bash bin/curl bin/stat bin/timeout bin/tee; do + [ -x "usr/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } + done rm -rf usr/share/man usr/share/doc usr/share/info usr/include usr/lib/*/pkgconfig find . -name '*.a' -delete du -sh /initrd diff --git a/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service new file mode 100644 index 00000000..175a680d --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service @@ -0,0 +1,13 @@ +[Unit] +Description=Check for ignored Ignition kernel arguments +Documentation=file:/usr/libexec/bluefin-boot-diagnostics +ConditionPathExists=/etc/initrd-release +DefaultDependencies=no +# Linked into initrd.target.wants by oci/bluefin-server-boot.bst. Warn before +# Ignition would have run, so the warning sits next to where it is missing. +Before=bluefin-ignition-credentials.service initrd.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/libexec/bluefin-boot-diagnostics ignition-kargs diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py index 5c0456e0..3ab0bc3a 100644 --- a/tests/unit/test_initrd_diagnostics.py +++ b/tests/unit/test_initrd_diagnostics.py @@ -75,6 +75,16 @@ def test_ram_check_unit_checks_its_import_instance() -> None: ] +def test_ignition_karg_warning_runs_in_every_initrd() -> None: + u = unit(UNITS / "bluefin-ignition-kargs.service") + assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"] + assert "bluefin-ignition-credentials.service" in " ".join(u["Unit"]["Before"]) + assert u["Service"]["ExecStart"] == ["/usr/libexec/bluefin-boot-diagnostics ignition-kargs"] + boot = BOOT.read_text(encoding="utf-8") + assert 'ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service"' in boot + assert "libexec/bluefin-boot-diagnostics" in boot, "the UKI build checks the helper is in the initrd" + + @pytest.mark.parametrize("stage", ["fetch-offline", "fetch", "disks", "mount", "files"]) def test_ignition_failures_reach_the_summary(stage: str) -> None: u = unit(IGN_UNITS / f"ignition-{stage}.service") From 7baee237f9fa76e39a2c552bf4a3a70a242ec984 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:26:25 -0500 Subject: [PATCH 4/8] docs: diskless troubleshooting skill (failure summary, RAM, logs, Ignition subset) New docs/skills/diskless-troubleshooting.md is the one place for what a failing diskless node prints and how to tune the delay, the minimum RAM (DDI x5 + margin, measured in QEMU: -m 3072 refused, -m 3584 boots), the network/download/verity/Ignition failure modes and their timing, how to keep logs (serial console, /var on disk via Ignition, journal-upload), and the supported Ignition subset with its idempotency rules. ddi-installer.md links to it instead of restating, and a test pins every console link to an existing section. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- AGENTS.md | 1 + docs/skills/ddi-installer-build.md | 2 + docs/skills/ddi-installer.md | 18 ++- docs/skills/diskless-troubleshooting.md | 172 ++++++++++++++++++++++++ docs/skills/index.md | 3 +- tests/unit/test_initrd_diagnostics.py | 15 +++ 6 files changed, 204 insertions(+), 7 deletions(-) create mode 100644 docs/skills/diskless-troubleshooting.md diff --git a/AGENTS.md b/AGENTS.md index 4d24050c..e6419e9f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,6 +63,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j | Task | Skill | |---|---| | Boot / install / update architecture and local build + dogfood | [`docs/skills/ddi-installer.md`](docs/skills/ddi-installer.md), [`docs/skills/ddi-installer-build.md`](docs/skills/ddi-installer-build.md) | +| Diskless boot failures, RAM sizing, node logs, Ignition configs | [`docs/skills/diskless-troubleshooting.md`](docs/skills/diskless-troubleshooting.md) | | Factory role, k0s sysext rationale, lab integration | [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md) | | Work with `systemd-sysext` / `systemd-confext` | [`docs/skills/systemd-sysext-extensions.md`](docs/skills/systemd-sysext-extensions.md) | | Build or run the kubeadm worker sysext | [`docs/skills/kubeadm-sysext.md`](docs/skills/kubeadm-sysext.md) | diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md index 0d8e1067..797f2854 100644 --- a/docs/skills/ddi-installer-build.md +++ b/docs/skills/ddi-installer-build.md @@ -100,6 +100,8 @@ Useful environment variables: - `DOGFOOD_SERVE_EXTRA=` — also serve the files in ``. - `DOGFOOD_TAMPER=raw|sums` — serve a corrupted DDI or a re-hashed, unsigned `SHA256SUMS`; the boot must fail, proving the signature check. +- `DOGFOOD_MEM=` — guest RAM (default 4096); below the diskless minimum + the boot must fail with the RAM message (see `diskless-troubleshooting.md`). - `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe. `scripts/dogfood-install.sh [ []]` is the full diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index 44d4f673..de749811 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd - /apache/buildstream @@ -87,9 +87,11 @@ ships gnupg and the image keyring, and importd fetches `SHA256SUMS` and using the manifest. The manifest hash check pins the DDI, the signed UKI pins `usrhash=`, and dm-verity checks every `/usr` block against it; a tampered DDI or a re-hashed unsigned manifest is refused. A failed boot never drops to an -emergency shell: the initrd prints the errors and reboots -(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`), -which is what lets boot counting work unattended. +emergency shell: the initrd prints a failure summary on every console and +reboots (`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`), +which is what lets boot counting work unattended. Before the pull, a check +refuses an image that cannot fit in RAM. What the node prints, minimum RAM, +failure modes and logs: [diskless-troubleshooting.md](diskless-troubleshooting.md). ### UEFI HTTP boot @@ -222,7 +224,10 @@ EFI variable and, when the server offers `bluefin-node.ign` next to the UKI, applies it. Ignition runs on **every** boot (there is no first-boot marker on a tmpfs root), so configs must be idempotent. See `tests/fixtures/ignition/var-on-disk.ign` for a dogfood-tested example -(persistent /var on a second disk plus an SSH key). +(persistent /var on a second disk plus an SSH key). The supported stages and +config sections, the idempotency rules, and the warning for ignored +`ignition.*` kernel arguments are in +the Ignition section of [diskless-troubleshooting.md](diskless-troubleshooting.md). ## PXE / HTTP boot service @@ -249,7 +254,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. | "Kernel image is at `/boot/vmlinuz`." | FSDK installs kernels into `/usr/lib/modules//vmlinuz`; `bluefin-server-boot.bst` picks it up from there for ukify. | | "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. | | "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. | -| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. | +| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot; `ignition.*` kargs are ignored with a console warning. | | "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. | ## Verification @@ -264,6 +269,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. ## See also - [ddi-installer-build.md](ddi-installer-build.md) — local build, export, and dogfood workflow. +- [diskless-troubleshooting.md](diskless-troubleshooting.md) — failure summary, minimum RAM, logs, supported Ignition subset. - [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md) — release signing and transfer verification. - [CONTEXT.md](../../CONTEXT.md) — canonical project domain glossary (OS DDI, Netboot UKI, Disk UKI, Slot). - `systemd-sysinstall(8)`, `systemd-repart(8)`, `systemd-sysupdate(8)`, `bootctl(1)`, `ukify(1)` diff --git a/docs/skills/diskless-troubleshooting.md b/docs/skills/diskless-troubleshooting.md new file mode 100644 index 00000000..bcb4c9de --- /dev/null +++ b/docs/skills/diskless-troubleshooting.md @@ -0,0 +1,172 @@ +--- +name: diskless-troubleshooting +description: Use when a diskless (netboot UKI) node fails to boot, reboot-loops, needs more RAM, or needs its logs kept, and when writing an Ignition config for Bluefin Server; covers the initrd failure summary, minimum RAM, network and download failure modes, getting logs, and the supported Ignition subset. +metadata: + type: how-to + status: stable + last_updated: "2026-09-28" + context7-sources: + - /systemd/systemd +--- +# Diskless Boot Troubleshooting + +## When to Use + +- A diskless node reboots in a loop, or never reaches a login prompt. +- Sizing RAM for diskless nodes. +- Keeping logs from diskless boots. +- Writing or debugging an Ignition config (diskless or installed). + +## When NOT to Use + +- The boot chain, pull and install architecture itself (see + [ddi-installer.md](ddi-installer.md)). +- Building and dogfooding images (see + [ddi-installer-build.md](ddi-installer-build.md)). + +## What a failing node prints + +A failed initrd never drops to a shell (root is locked): it isolates +`emergency.target`, whose `emergency.service` drop-in +(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`) runs +`/usr/libexec/bluefin-boot-diagnostics failure-summary` and then reboots. The +summary lists each failed unit with a one-line hint and its last journal lines, +the network links when the network or the download failed, any ignored +`ignition.*` kernel arguments, and the last `journalctl -b -p err` lines. It is +written to every active console (`/sys/class/tty/console/active`: the monitor +and `ttyS0`), not only to `/dev/console`, which is just the last `console=`. + +The summary stays up for 60 s, then the node reboots and retries. Retrying is +deliberate: a diskless node recovers unattended once the server, network or +config is fixed, and on installed nodes boot counting falls back to the +previous image. Change the delay (10 to 86400 s) with the +`bluefin.failure_delay` system credential (SMBIOS type 11, QEMU fw_cfg, or +`/loader/credentials/bluefin.failure_delay.cred` on the ESP); the +`bluefin.failure_delay=` kernel argument overrides it only where the command +line is not locked by Secure Boot. The 10 s floor keeps a persistent failure +from becoming a hot reboot loop. + +Before the download, `bluefin-pull-check@.service` (pulled in by +`systemd-import@.service.d/10-bluefin-pull-check.conf`) sends an HTTP HEAD to +the image URL systemd-import-generator resolved and fails early, with its own +framed console message, when the server cannot be reached, answers with an +error, or the image cannot fit in RAM. A failed download goes straight to the +summary instead of waiting out the 300 s `/usr` device timeout. + +## Minimum RAM + +The netboot initrd downloads the whole OS DDI (`bluefin-server_.raw`, +540 MiB for `26.09.x`) into `/run/machines`. systemd mounts `/run` as a tmpfs +capped at 20% of RAM, and the DDI stays there for the whole boot because it +backs `/usr`. The pull check requires the DDI plus 64 MiB to fit in both the +free space of `/run` and `MemAvailable`, so usable RAM (`MemTotal`) must be at +least about five times the DDI plus 64 MiB: about 3 GiB for a 540 MiB DDI. +Measured in QEMU with the `26.09.x` DDI, `-m 3072` (2816 MiB `MemTotal`) is +refused and `-m 3584` boots. Plan on 4 GiB plus the workload. + +A machine below the minimum fails before the download (QEMU `-m 2048`): + +```text +BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT +This machine needs ~3072 MiB RAM, have 1876 MiB. +The OS image bluefin-server_26.09.2.raw (540 MiB) is downloaded into /run, a RAM +file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB. +``` + +## Failure modes + +| Symptom in the summary | Cause | Timing | +|---|---|---| +| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the pull check fails and the node reboots | +| "no network interface found" | NIC driver not in the initrd module list (`initrd-modules` in `oci/bluefin-server-boot.bst`) | As above | +| "CANNOT REACH THE OS IMAGE SERVER" | Routable link but no route to, or no answer from, the boot server | Seconds after the network is online | +| "THE OS IMAGE SERVER ANSWERED HTTP 404" | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | Seconds | +| "NOT ENOUGH RAM FOR A DISKLESS BOOT" | See [Minimum RAM](#minimum-ram) | Seconds | +| `systemd-import@...service` failed, with systemd-importd's reason (e.g. `DOWNLOAD INVALID: Checksum ... did not check out`) | Download interrupted, the DDI does not match `SHA256SUMS`, or `SHA256SUMS.gpg` is missing or not signed by a key in the image keyring | Right after the download | +| `systemd-veritysetup@usr.service` or `sysusr-usr.mount` failed | The DDI does not match the `usrhash=` of the UKI that booted (mixed versions on the server) | After the download | +| "No unit failed: a device or job timed out" | A device never appeared within `systemd.default_device_timeout_sec=300` | 300 s | +| `ignition-.service` failed | See [Ignition](#ignition) | During the stage | + +## Getting logs from a diskless node + +Every diskless boot starts with a fresh machine ID and keeps its journal in +RAM (`/run/log/journal` in the initrd, then `/var/log/journal` on the tmpfs +root), so nothing survives a reboot by default. + +- **Initrd failures** never reach a disk: read the console. The netboot UKI + sets `console=tty0 console=ttyS0,115200`; use a serial cable, IPMI + serial-over-LAN or the BMC console log. `netconsole` is built into the + kernel, but its `netconsole=` argument cannot be added to the sealed command + line, so it is not available in the initrd. +- **Persistent logs** after switch-root: provision a `/var` disk with Ignition + (`tests/fixtures/ignition/var-on-disk.ign`, the same state-disk pattern the + boot server's per-host `bluefin-node.ign` uses; `/var/log/journal` is created + on it by `systemd-tmpfiles`, so journald persists without extra config). + Each boot writes under its own machine ID, so read across boots with + `journalctl --merge --list-boots` and `journalctl --merge -b -1`. Do not pin + the ID by writing `/etc/machine-id` with Ignition: a set machine ID ends the + first-boot state that applies the presets on every diskless boot, and + preset-enabled units stop starting. The `system.machine_id` credential (a + 32-hex ID, or `firmware` for the SMBIOS UUID) pins it without that effect. +- **Remote logs**: the image ships `systemd-journal-upload`; an Ignition config + can write `/etc/systemd/journal-upload.conf` (`[Upload]` / `URL=`) and enable + `systemd-journal-upload.service` to stream the journal of the running system + to a `systemd-journal-remote` collector. It starts after switch-root, so it + does not cover initrd failures. + +## Ignition + +Configs arrive as the `ignition.config` / `ignition.config.url` credentials or +as `bluefin-node.ign` next to the netboot UKI (see +the Ignition section of [ddi-installer.md](ddi-installer.md)). Ignition kernel +arguments (`ignition.config.url=`, `ignition.platform.id=`, ...) are never read; +if any are on the command line, `bluefin-ignition-kargs.service` prints a +framed warning naming them (not their values) and the failure summary repeats +it. + +Stages wired into the initrd (`files/initrd-ignition/`), in order: +`fetch-offline`, `fetch` (only when the config needs the network), `disks`, +`mount`, `files`, and `umount` (`ExecStop` of `ignition-mount.service`). The +`kargs` stage is not wired, so `kernelArguments` is silently not applied: the +command line is sealed in the signed UKI. + +| Config section | Supported | +|---|---| +| `ignition.config.merge` / `replace`, `timeouts`, `security.tls` | Yes | +| `storage.disks` (partitions via `sgdisk`) | Yes | +| `storage.filesystems` | `xfs`, `ext4`, `vfat`, `swap`; no `btrfs` | +| `storage.luks` | Key files only; no `clevis` (TPM2 / Tang) | +| `storage.raid` | No (`mdadm` absent) | +| `storage.files`, `directories`, `links` | Yes | +| `systemd.units` (contents, dropins, `enabled`) | Yes; enabled units go through a preset applied by the real root | +| `passwd.users`, `passwd.groups` | Yes | +| `kernelArguments` | No | + +There is no first-boot marker: every diskless boot runs every stage against a +fresh `/etc`, so a config must be idempotent or the node fails the same stage +on every boot: + +- **Disks**: keep `wipeTable: false` and `wipePartitionEntry: false`; an + existing partition must match the config (number, label, size, type) or + `ignition-disks` fails. +- **Filesystems**: `wipeFilesystem: false` reuses a filesystem with the same + format (and label/UUID when set) and fails on any other; + `wipeFilesystem: true` erases the disk on **every** boot. +- **Files**: a file with `contents` fails the `files` stage when something is + already at its path and `overwrite` is false (the default): true from the + second boot for files on a persistent filesystem, and on every boot for + paths shipped in the factory `/etc`. Use `"overwrite": true`. + +## Verification + +- [ ] `just dogfood-check` passes (normal diskless boot). +- [ ] `DOGFOOD_MEM=2048 scripts/dogfood-diskless.sh dist/diskless --check` + shows "NOT ENOUGH RAM FOR A DISKLESS BOOT" in `dist/diskless/dogfood-serial.log`. +- [ ] `DOGFOOD_TAMPER=sums scripts/dogfood-diskless.sh dist/diskless --check` + shows "BLUEFIN: BOOT FAILED" and the failed `systemd-import@` unit. +- [ ] `python3 -m pytest tests/unit/test_initrd_diagnostics.py` passes. + +## See also + +- [ddi-installer.md](ddi-installer.md) — boot chain, diskless pull, Ignition credentials. +- [ddi-installer-build.md](ddi-installer-build.md) — dogfood harness and its `DOGFOOD_*` variables. diff --git a/docs/skills/index.md b/docs/skills/index.md index 084d53d3..92526196 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -4,7 +4,7 @@ description: Lazy-load manifest for Bluefin Server skills. Load this file after metadata: type: index status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" --- # docs/skills — Index @@ -28,6 +28,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w | [`ci-tooling.md`](ci-tooling.md) | GitHub Actions, workflow SHA pinning, CI conventions | CI conventions and release pipeline rules. | | [`ddi-installer-build.md`](ddi-installer-build.md) | Building and dogfooding the image set | Local build, export, and QEMU dogfood workflow. | | [`ddi-installer.md`](ddi-installer.md) | Boot flow, diskless pull, `systemd-sysinstall`, A/B sysupdate | Boot, install, and update architecture. | +| [`diskless-troubleshooting.md`](diskless-troubleshooting.md) | Diskless node reboot-loops, RAM sizing, keeping logs, writing Ignition configs | Failure summary, minimum RAM, failure modes, logs, supported Ignition subset. | | [`factory-integration.md`](factory-integration.md) | Lab integration, boot-test workflow, factory role | How Bluefin Server is consumed by the CI lab. | | [`gap-analysis-distros.md`](gap-analysis-distros.md) | Comparing Bluefin Server to other server OSes | Source-verified comparison to Ubuntu, Talos, Flatcar, FCOS. | | [`kubeadm-sysext.md`](kubeadm-sysext.md) | Joining a kubeadm cluster, kubelet/containerd sysext, Kubernetes kernel options | Build and runtime contract for the kubeadm worker sysext. | diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py index 3ab0bc3a..6e741215 100644 --- a/tests/unit/test_initrd_diagnostics.py +++ b/tests/unit/test_initrd_diagnostics.py @@ -410,3 +410,18 @@ def test_unknown_mode_is_a_usage_error(env: Env) -> None: result = env.run("bogus") assert result.returncode == 2 assert "usage:" in result.stderr + + +def test_console_links_point_at_existing_doc_sections() -> None: + text = HELPER.read_text(encoding="utf-8") + base = re.search(r"^DOCS=https://github\.com/projectbluefin/server/blob/main/(\S+)$", text, re.M) + assert base, "DOCS must link the repository's main branch" + doc = (ROOT / base.group(1)).read_text(encoding="utf-8") + slugs = { + re.sub(r"[^a-z0-9 -]", "", line.lstrip("#").strip().lower()).replace(" ", "-") + for line in doc.splitlines() + if line.startswith("## ") + } + anchors = set(re.findall(r"\$\{DOCS\}#([a-z0-9-]+)", text)) + assert anchors == {"minimum-ram", "failure-modes", "ignition"} + assert anchors <= slugs From c58c77d04fe761030e528519ff7cc2d4e3e470ed Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Mon, 28 Sep 2026 22:50:02 -0500 Subject: [PATCH 5/8] docs(diskless-troubleshooting): read earlier boots with journalctl -D Measured in QEMU with a /var disk: each diskless boot journals under a new machine ID, so plain journalctl lists one boot and --merge --list-boots lists none, while journalctl -D /var/log/journal --list-boots lists both. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/diskless-troubleshooting.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/skills/diskless-troubleshooting.md b/docs/skills/diskless-troubleshooting.md index bcb4c9de..3c22d081 100644 --- a/docs/skills/diskless-troubleshooting.md +++ b/docs/skills/diskless-troubleshooting.md @@ -102,8 +102,10 @@ root), so nothing survives a reboot by default. (`tests/fixtures/ignition/var-on-disk.ign`, the same state-disk pattern the boot server's per-host `bluefin-node.ign` uses; `/var/log/journal` is created on it by `systemd-tmpfiles`, so journald persists without extra config). - Each boot writes under its own machine ID, so read across boots with - `journalctl --merge --list-boots` and `journalctl --merge -b -1`. Do not pin + Each boot writes under its own machine ID and plain `journalctl` shows only + the current one, so read across boots with + `journalctl -D /var/log/journal --list-boots` and + `journalctl -D /var/log/journal -b -1`. Do not pin the ID by writing `/etc/machine-id` with Ignition: a set machine ID ends the first-boot state that applies the presets on every diskless boot, and preset-enabled units stop starting. The `system.machine_id` credential (a From 3a0c43b4bc8260a3c5a636990be45f9ce6636c2c Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:24 -0500 Subject: [PATCH 6/8] fix(initrd): only an image that cannot fit stops the diskless pull The pull check failed, and so stopped the boot, on any final HEAD status other than 2xx, 405 or 501, and when curl could not reach the server. A server that serves the image on GET but answers HEAD with 403 or 404, as signed object-store URLs do because the signature covers the method, never booted, although systemd-import would have downloaded the image. The check now fails only when a 2xx HEAD gives a Content-Length that cannot fit in /run and MemAvailable. Any other outcome (no answer, a 4xx or 5xx, no Content-Length, unreadable RAM or /run sizes) prints one line on every console, "BLUEFIN: cannot check that the OS image fits in RAM (); downloading it anyway", and exits 0. If the download then fails, its own OnFailure= puts systemd-importd's reason in the failure summary. The check runs in a subshell and reports "does not fit" with a status of its own, which the helper turns into exit 1; any other status, such as bash aborting on an unset variable, exits 0, so a bug in the check cannot stop the boot either. Since the check now fails only on RAM, its failure-summary hint says so and no longer lists the network links. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../system/bluefin-pull-check@.service | 6 +- .../usr/libexec/bluefin-boot-diagnostics | 62 +++++++++------- tests/unit/test_initrd_diagnostics.py | 74 ++++++++++++------- 3 files changed, 88 insertions(+), 54 deletions(-) diff --git a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service index d31862e4..dddced00 100644 --- a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service +++ b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service @@ -9,8 +9,10 @@ DefaultDependencies=no Wants=network-online.target After=network-online.target -# Print why on the console and reboot (emergency.service.d/10-reboot.conf) -# rather than start a download that cannot fit. +# The check fails only when the server gives a size that cannot fit in RAM: +# print why on the console and reboot (emergency.service.d/10-reboot.conf) +# rather than start that download. When it cannot check the size, it notes +# that on the console and succeeds, and the download runs. OnFailure=emergency.target OnFailureJobMode=isolate diff --git a/files/initrd/usr/libexec/bluefin-boot-diagnostics b/files/initrd/usr/libexec/bluefin-boot-diagnostics index f7c77130..59ff4382 100755 --- a/files/initrd/usr/libexec/bluefin-boot-diagnostics +++ b/files/initrd/usr/libexec/bluefin-boot-diagnostics @@ -3,8 +3,11 @@ # # check-pull UNIT Run before UNIT (a systemd-import@.service instance) # downloads the OS image into RAM. HEADs the image URL and -# fails with a plain "needs ~X MiB RAM, have Y" message if -# the image cannot fit, or if the server cannot be reached. +# fails with a plain "needs ~X MiB RAM, have Y" message +# only when a 2xx answer gives a size that cannot fit. If +# it cannot check (no answer, an HTTP error, no +# Content-Length), it prints a note and exits 0 so the +# download still runs. # ignition-kargs Warn that ignition.* kernel arguments are ignored. # failure-summary ExecStart of emergency.service: print the failed units, # their last log lines and the last errors, then wait @@ -33,6 +36,10 @@ NETDIR="${BLUEFIN_DIAG_NETDIR:-/sys/class/net}" PULL_DIR=/run # Room /run and RAM must keep after the image has landed. MARGIN_MIB=64 +# check_pull's status when the image cannot fit. It differs from bash's own +# error statuses (1, 2, 126, 127, 128+n), so a bug in the check is never +# taken for "does not fit". +TOO_BIG=3 DELAY_NAME=bluefin.failure_delay DEFAULT_DELAY=60 # The reboot is intentional (unattended retry, boot counting); a floor keeps @@ -92,6 +99,15 @@ links() { fi } +# cannot_check REASON: note on the consoles that the size was not checked. The +# download goes ahead: it may still work (a signed object-store URL can refuse +# HEAD and serve GET), and if it fails the failure summary says why. +cannot_check() { + local msg="BLUEFIN: cannot check that the OS image fits in RAM ($1); downloading it anyway" + log 4 "${msg}" + printf '%s\n' "${msg}" | to_consoles +} + check_pull() { local unit="${1:?usage: ${PROG} check-pull }" local desc url out rc status='' len='' line lower @@ -105,17 +121,13 @@ check_pull() { *) log 6 "${unit}: no http(s) source (${desc:-no description}); nothing to check"; return 0 ;; esac + # Only the size is wanted here; whether the image downloads is for + # systemd-import to find out. out="$(curl -sSIL --retry 3 --retry-delay 2 --connect-timeout 10 --max-time 120 "${url}" 2>&1)" rc=$? if [ "${rc}" -ne 0 ]; then - local -a netlines - mapfile -t netlines < <(links) - shout 3 "BLUEFIN: CANNOT REACH THE OS IMAGE SERVER" \ - "URL: ${url}" \ - "${out##*$'\n'}" \ - "Network links:" "${netlines[@]/#/ }" \ - "Check the cable, DHCP and the boot server. ${DOCS}#failure-modes" - return 1 + cannot_check "HEAD failed: ${out##*$'\n'}" + return 0 fi # Headers of every response in the redirect chain; keep the last one. @@ -130,26 +142,19 @@ check_pull() { case "${status}" in 2??) ;; - 405|501) - log 4 "${url}: HEAD not supported (HTTP ${status}); cannot check the image size" - return 0 ;; - *) - shout 3 "BLUEFIN: THE OS IMAGE SERVER ANSWERED HTTP ${status:-?}" \ - "URL: ${url}" \ - "The boot server must serve the OS image next to the netboot UKI. ${DOCS}#failure-modes" - return 1 ;; + *) cannot_check "HEAD answered HTTP ${status:-?}"; return 0 ;; esac if ! [[ "${len}" =~ ^[0-9]+$ ]]; then - log 4 "${url}: no Content-Length; cannot check the image size" + cannot_check "HEAD answered HTTP ${status} with no Content-Length" return 0 fi local total_kib avail_kib bavail bsize btotal - total_kib="$(meminfo_kib MemTotal)" || { log 4 "no MemTotal in ${MEMINFO}"; return 0; } + total_kib="$(meminfo_kib MemTotal)" || { cannot_check "no MemTotal in ${MEMINFO}"; return 0; } avail_kib="$(meminfo_kib MemAvailable)" || avail_kib="${total_kib}" read -r bavail bsize btotal < <(stat -f -c '%a %S %b' "${PULL_DIR}") if ! [[ "${bavail:-}${bsize:-}${btotal:-}" =~ ^[0-9]+$ ]] || [ "${btotal}" -eq 0 ] || [ "${total_kib}" -eq 0 ]; then - log 4 "cannot read the size of ${PULL_DIR} or of RAM; skipping the RAM check" + cannot_check "cannot read the size of ${PULL_DIR} or of RAM" return 0 fi @@ -175,7 +180,7 @@ check_pull() { "The OS image ${url##*/} ($(mib "${len}") MiB) is downloaded into /run, a RAM" \ "file system capped at $(( run_size * 100 / total ))% of RAM: $(mib "${run_free}") MiB free there, MemAvailable $(mib "${avail}") MiB." \ "Add RAM or install to disk. ${DOCS}#minimum-ram" - return 1 + return "${TOO_BIG}" } # Keys of ignition.* kernel arguments (values may carry secrets). @@ -206,7 +211,7 @@ hint() { local stage case "$1" in bluefin-pull-check@*) - echo "the OS image cannot be downloaded into RAM (see the message above)" ;; + echo "the OS image is too big for the RAM of this machine (see the message above)" ;; systemd-import@*) echo "downloading or verifying the OS image failed: server reachable? SHA256SUMS signed by a key this image trusts? room in /run?" ;; systemd-loop@*) @@ -268,7 +273,7 @@ failure_summary() { case "${unit}" in systemd-import@*) matches+=(_SYSTEMD_UNIT=systemd-importd.service) ;; esac journalctl -b "${matches[@]}" -n 6 -o cat --no-pager 2>/dev/null | indent ' | ' case "${unit}" in - systemd-networkd-wait-online.service|systemd-import@*|bluefin-pull-check@*) + systemd-networkd-wait-online.service|systemd-import@*) printf ' network links:\n'; links | indent ' ' ;; esac done @@ -285,7 +290,14 @@ failure_summary() { } case "${1:-}" in - check-pull) shift; check_pull "$@" ;; + check-pull) + shift + # Only "the image cannot fit" fails the unit, and so stops the boot. The + # check runs in a subshell so that a bug in it (an unset variable, a value + # it did not expect) lets the download go ahead instead. + rc=0 + ( check_pull "$@" ) || rc=$? + [ "${rc}" -ne "${TOO_BIG}" ] || exit 1 ;; ignition-kargs) ignition_kargs ;; failure-summary) failure_summary ;; *) printf 'usage: %s check-pull UNIT | ignition-kargs | failure-summary\n' "${PROG}" >&2; exit 2 ;; diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py index 6e741215..555e7d43 100644 --- a/tests/unit/test_initrd_diagnostics.py +++ b/tests/unit/test_initrd_diagnostics.py @@ -220,8 +220,20 @@ def env(tmp_path: Path) -> Env: return Env(tmp_path) -def check_pull(env: Env, url: str) -> subprocess.CompletedProcess: - return env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION=f"Download of {url}") +def check_pull(env: Env, url: str, **extra: str) -> subprocess.CompletedProcess: + return env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION=f"Download of {url}", **extra) + + +def cannot_check(reason: str) -> str: + return f"BLUEFIN: cannot check that the OS image fits in RAM ({reason}); downloading it anyway" + + +def assert_download_goes_ahead(env: Env, result: subprocess.CompletedProcess, reason: str) -> None: + note = cannot_check(reason) + assert result.returncode == 0, result.stderr + for tty in ("tty0", "ttyS0"): + assert env.console(tty) == f"{note}\n" + assert f"<4>bluefin-boot-diagnostics: {note}\n" in result.stderr def test_image_that_fits_passes_quietly(env: Env, server) -> None: @@ -283,44 +295,45 @@ def test_redirects_are_followed_to_the_final_size(env: Env, server) -> None: assert "redirect.raw (4000 MiB)" in env.console() -def test_missing_image_fails_with_the_http_status(env: Env, server) -> None: - base, _ = server - result = check_pull(env, f"{base}/missing.raw") - assert result.returncode == 1 - assert "THE OS IMAGE SERVER ANSWERED HTTP 404" in env.console() - assert f"URL: {base}/missing.raw" in env.console() - - -@pytest.mark.parametrize("status", [405, 501]) -def test_servers_without_head_are_not_blocked(env: Env, server, status: int) -> None: +@pytest.mark.parametrize("status", [403, 404, 405, 500, 501]) +def test_head_errors_do_not_block_the_download(env: Env, server, status: int) -> None: + # A signed object-store URL covers the method: GET works, HEAD gets 403. base, routes = server - routes[f"/nohead{status}.raw"] = (status, {"Content-Length": "0"}) - result = check_pull(env, f"{base}/nohead{status}.raw") - assert result.returncode == 0 - assert "HEAD not supported" in result.stderr - assert env.console() == "" + routes[f"/head{status}.raw"] = (status, {"Content-Length": "0"}) + result = check_pull(env, f"{base}/head{status}.raw") + assert_download_goes_ahead(env, result, f"HEAD answered HTTP {status}") def test_missing_content_length_skips_the_check(env: Env, server) -> None: base, routes = server routes["/chunked.raw"] = (200, {"Transfer-Encoding": "chunked"}) result = check_pull(env, f"{base}/chunked.raw") - assert result.returncode == 0 - assert "no Content-Length" in result.stderr + assert_download_goes_ahead(env, result, "HEAD answered HTTP 200 with no Content-Length") -def test_unreachable_server_names_the_url_and_the_links(env: Env) -> None: +def test_unreachable_server_does_not_block_the_download(env: Env) -> None: with socket.socket() as s: s.bind(("127.0.0.1", 0)) port = s.getsockname()[1] result = check_pull(env, f"http://127.0.0.1:{port}/img.raw") - assert result.returncode == 1 + assert result.returncode == 0, result.stderr text = env.console() - assert "CANNOT REACH THE OS IMAGE SERVER" in text - assert f"URL: http://127.0.0.1:{port}/img.raw" in text - assert "curl: (7)" in text - assert " enp1s0 down (no carrier)" in text - assert "lo " not in text + assert text.startswith("BLUEFIN: cannot check that the OS image fits in RAM (HEAD failed: curl: (7) ") + assert text.endswith("); downloading it anyway\n") + assert text.count("\n") == 1 + assert env.console("tty0") == text + assert f"<4>bluefin-boot-diagnostics: {text}" in result.stderr + + +def test_a_broken_check_does_not_block_the_download(env: Env, server) -> None: + base, routes = server + routes["/broken.raw"] = (200, {"Content-Length": str(540 * MIB)}) + # A value the check does not expect makes bash abort it (unbound variable). + (env.tmp / "meminfo").write_text("MemTotal: 4000000 kB\nMemAvailable: n/a kB\n", encoding="utf-8") + result = check_pull(env, f"{base}/broken.raw") + assert "unbound variable" in result.stderr + assert result.returncode == 0 + assert env.console() == "" def test_non_http_sources_are_not_checked(env: Env) -> None: @@ -375,6 +388,13 @@ def test_failure_summary_explains_each_failed_unit(env: Env) -> None: assert env.sleeps() == ["60"] +def test_failure_summary_blames_ram_for_a_failed_check(env: Env) -> None: + env.run("failure-summary", FAKE_FAILED="bluefin-pull-check@run-machines-rootdisk.raw.service") + text = env.console() + assert "-> the OS image is too big for the RAM of this machine" in text + assert "network links:" not in text + + def test_failure_summary_without_failed_units_points_at_timeouts(env: Env) -> None: env.cmdline("root=tmpfs ignition.config.url=http://x/y.ign") env.run("failure-summary") @@ -423,5 +443,5 @@ def test_console_links_point_at_existing_doc_sections() -> None: if line.startswith("## ") } anchors = set(re.findall(r"\$\{DOCS\}#([a-z0-9-]+)", text)) - assert anchors == {"minimum-ram", "failure-modes", "ignition"} + assert anchors == {"minimum-ram", "ignition"} assert anchors <= slugs From cfce154f006cdea0a86c21fd7b070639e4a14433 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:44 -0500 Subject: [PATCH 7/8] fix(initrd): want the pull check instead of requiring it systemd-import@.service required bluefin-pull-check@%i.service, so any failure of the check, or a missing check unit, failed the download's start job. The check already stops the boot through its own OnFailure=emergency.target when the image cannot fit, so Wants= with the existing After= keeps it ordered before the download without making the download depend on it. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../usr/lib/systemd/system/bluefin-pull-check@.service | 2 +- .../systemd-import@.service.d/10-bluefin-pull-check.conf | 7 +++++-- tests/unit/test_initrd_diagnostics.py | 4 +++- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service index dddced00..5c233783 100644 --- a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service +++ b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service @@ -3,7 +3,7 @@ Description=Check that the OS image fits in RAM before downloading it Documentation=file:/usr/libexec/bluefin-boot-diagnostics ConditionPathExists=/etc/initrd-release DefaultDependencies=no -# Pulled in and ordered before systemd-import@%i.service by its drop-in +# Wanted by, and ordered before, systemd-import@%i.service through its drop-in # systemd-import@.service.d/10-bluefin-pull-check.conf. Needs the network, # like the download itself. Wants=network-online.target diff --git a/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf index d7b7417c..a5a7b2df 100644 --- a/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf +++ b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf @@ -1,8 +1,11 @@ [Unit] # The diskless initrd downloads the whole OS image into /run (RAM). Check # first that it fits, so a small machine fails with "needs ~X MiB RAM" -# instead of an out-of-space error deep in the download. -Requires=bluefin-pull-check@%i.service +# instead of an out-of-space error deep in the download. Wants=, not +# Requires=: the check stops the boot through its own OnFailure=, which it +# triggers only when it knows the image cannot fit; a check that is missing +# or broken must not hold up the download. +Wants=bluefin-pull-check@%i.service After=bluefin-pull-check@%i.service # Without this a failed download only surfaces when the /usr devices time out diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py index 555e7d43..3d9abafe 100644 --- a/tests/unit/test_initrd_diagnostics.py +++ b/tests/unit/test_initrd_diagnostics.py @@ -56,7 +56,9 @@ def test_emergency_prints_the_summary_then_reboots() -> None: def test_every_download_is_preceded_by_the_ram_check() -> None: u = unit(UNITS / "systemd-import@.service.d" / "10-bluefin-pull-check.conf") - assert u["Unit"]["Requires"] == ["bluefin-pull-check@%i.service"] + # Only the check's own OnFailure= may stop the boot; its failure never fails the download. + assert u["Unit"]["Wants"] == ["bluefin-pull-check@%i.service"] + assert "Requires" not in u["Unit"] assert u["Unit"]["After"] == ["bluefin-pull-check@%i.service"] # A failed download goes straight to the summary, not a 300 s device timeout. assert u["Unit"]["OnFailure"] == ["emergency.target"] From f155a779af0acc15e21f50ae1250201e8a1b4538 Mon Sep 17 00:00:00 2001 From: Jeffrey Sica Date: Tue, 29 Sep 2026 02:02:52 -0500 Subject: [PATCH 8/8] docs(diskless-troubleshooting): pull check stops the boot only on RAM Describe the one-line note the pull check prints when it cannot check the size, and replace the "CANNOT REACH THE OS IMAGE SERVER" and "ANSWERED HTTP 404" rows, which the check no longer prints, with the systemd-importd reasons the failure summary shows for the same causes. Without a network the download, not the check, now fails after the 300 s wait, and a machine below the minimum RAM fails during the download when the server does not give the size. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- docs/skills/diskless-troubleshooting.md | 29 +++++++++++++++---------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/docs/skills/diskless-troubleshooting.md b/docs/skills/diskless-troubleshooting.md index 3c22d081..0ff6a17a 100644 --- a/docs/skills/diskless-troubleshooting.md +++ b/docs/skills/diskless-troubleshooting.md @@ -4,7 +4,7 @@ description: Use when a diskless (netboot UKI) node fails to boot, reboot-loops, metadata: type: how-to status: stable - last_updated: "2026-09-28" + last_updated: "2026-09-29" context7-sources: - /systemd/systemd --- @@ -46,12 +46,17 @@ previous image. Change the delay (10 to 86400 s) with the line is not locked by Secure Boot. The 10 s floor keeps a persistent failure from becoming a hot reboot loop. -Before the download, `bluefin-pull-check@.service` (pulled in by -`systemd-import@.service.d/10-bluefin-pull-check.conf`) sends an HTTP HEAD to -the image URL systemd-import-generator resolved and fails early, with its own -framed console message, when the server cannot be reached, answers with an -error, or the image cannot fit in RAM. A failed download goes straight to the -summary instead of waiting out the 300 s `/usr` device timeout. +Before the download, `bluefin-pull-check@.service` (wanted by +`systemd-import@.service.d/10-bluefin-pull-check.conf` and ordered before the +download) sends an HTTP HEAD to the image URL systemd-import-generator +resolved. It stops the boot, with its own framed console message, only when a +2xx answer gives a `Content-Length` that cannot fit in RAM. When it cannot +check the size (no answer, an HTTP error such as the 403 a signed +object-store URL gives to HEAD, or no `Content-Length`), it prints +`BLUEFIN: cannot check that the OS image fits in RAM (); downloading it anyway` +and the download runs: whether the image can be fetched is for systemd-import +to find out. A failed download goes straight to the summary instead of +waiting out the 300 s `/usr` device timeout. ## Minimum RAM @@ -62,7 +67,9 @@ backs `/usr`. The pull check requires the DDI plus 64 MiB to fit in both the free space of `/run` and `MemAvailable`, so usable RAM (`MemTotal`) must be at least about five times the DDI plus 64 MiB: about 3 GiB for a 540 MiB DDI. Measured in QEMU with the `26.09.x` DDI, `-m 3072` (2816 MiB `MemTotal`) is -refused and `-m 3584` boots. Plan on 4 GiB plus the workload. +refused and `-m 3584` boots. Plan on 4 GiB plus the workload. If the server +does not give the size, the check is skipped and a machine below the minimum +fails during the download instead (`systemd-import@...service` failed). A machine below the minimum fails before the download (QEMU `-m 2048`): @@ -77,10 +84,10 @@ file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB. | Symptom in the summary | Cause | Timing | |---|---|---| -| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the pull check fails and the node reboots | +| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the download fails and the node reboots | | "no network interface found" | NIC driver not in the initrd module list (`initrd-modules` in `oci/bluefin-server-boot.bst`) | As above | -| "CANNOT REACH THE OS IMAGE SERVER" | Routable link but no route to, or no answer from, the boot server | Seconds after the network is online | -| "THE OS IMAGE SERVER ANSWERED HTTP 404" | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | Seconds | +| `systemd-import@...service` failed, systemd-importd says `Transfer failed: ...` | Routable link but no route to, or no answer from, the boot server | When the download starts | +| `systemd-import@...service` failed, systemd-importd says `HTTP request to ... failed with code 404.` | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | When the download starts | | "NOT ENOUGH RAM FOR A DISKLESS BOOT" | See [Minimum RAM](#minimum-ram) | Seconds | | `systemd-import@...service` failed, with systemd-importd's reason (e.g. `DOWNLOAD INVALID: Checksum ... did not check out`) | Download interrupted, the DDI does not match `SHA256SUMS`, or `SHA256SUMS.gpg` is missing or not signed by a key in the image keyring | Right after the download | | `systemd-veritysetup@usr.service` or `sysusr-usr.mount` failed | The DDI does not match the `usrhash=` of the UKI that booted (mixed versions on the server) | After the download |