diff --git a/AGENTS.md b/AGENTS.md
index 4d24050c..e6419e9f 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -63,6 +63,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
| Task | Skill |
|---|---|
| Boot / install / update architecture and local build + dogfood | [`docs/skills/ddi-installer.md`](docs/skills/ddi-installer.md), [`docs/skills/ddi-installer-build.md`](docs/skills/ddi-installer-build.md) |
+| Diskless boot failures, RAM sizing, node logs, Ignition configs | [`docs/skills/diskless-troubleshooting.md`](docs/skills/diskless-troubleshooting.md) |
| Factory role, k0s sysext rationale, lab integration | [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md) |
| Work with `systemd-sysext` / `systemd-confext` | [`docs/skills/systemd-sysext-extensions.md`](docs/skills/systemd-sysext-extensions.md) |
| Build or run the kubeadm worker sysext | [`docs/skills/kubeadm-sysext.md`](docs/skills/kubeadm-sysext.md) |
diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md
index 0d8e1067..797f2854 100644
--- a/docs/skills/ddi-installer-build.md
+++ b/docs/skills/ddi-installer-build.md
@@ -100,6 +100,8 @@ Useful environment variables:
- `DOGFOOD_SERVE_EXTRA=
` — also serve the files in ``.
- `DOGFOOD_TAMPER=raw|sums` — serve a corrupted DDI or a re-hashed, unsigned
`SHA256SUMS`; the boot must fail, proving the signature check.
+- `DOGFOOD_MEM=` — guest RAM (default 4096); below the diskless minimum
+ the boot must fail with the RAM message (see `diskless-troubleshooting.md`).
- `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe.
`scripts/dogfood-install.sh [ []]` is the full
diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md
index 44d4f673..de749811 100644
--- a/docs/skills/ddi-installer.md
+++ b/docs/skills/ddi-installer.md
@@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d
metadata:
type: reference
status: stable
- last_updated: "2026-09-27"
+ last_updated: "2026-09-28"
context7-sources:
- /systemd/systemd
- /apache/buildstream
@@ -87,9 +87,11 @@ ships gnupg and the image keyring, and importd fetches `SHA256SUMS` and
using the manifest. The manifest hash check pins the DDI, the signed UKI pins
`usrhash=`, and dm-verity checks every `/usr` block against it; a tampered DDI
or a re-hashed unsigned manifest is refused. A failed boot never drops to an
-emergency shell: the initrd prints the errors and reboots
-(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`),
-which is what lets boot counting work unattended.
+emergency shell: the initrd prints a failure summary on every console and
+reboots (`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`),
+which is what lets boot counting work unattended. Before the pull, a check
+refuses an image that cannot fit in RAM. What the node prints, minimum RAM,
+failure modes and logs: [diskless-troubleshooting.md](diskless-troubleshooting.md).
### UEFI HTTP boot
@@ -222,7 +224,10 @@ EFI variable and, when the server offers `bluefin-node.ign` next to the UKI,
applies it. Ignition runs on **every** boot (there is no first-boot marker on
a tmpfs root), so configs must be idempotent. See
`tests/fixtures/ignition/var-on-disk.ign` for a dogfood-tested example
-(persistent /var on a second disk plus an SSH key).
+(persistent /var on a second disk plus an SSH key). The supported stages and
+config sections, the idempotency rules, and the warning for ignored
+`ignition.*` kernel arguments are in
+the Ignition section of [diskless-troubleshooting.md](diskless-troubleshooting.md).
## PXE / HTTP boot service
@@ -249,7 +254,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS.
| "Kernel image is at `/boot/vmlinuz`." | FSDK installs kernels into `/usr/lib/modules//vmlinuz`; `bluefin-server-boot.bst` picks it up from there for ukify. |
| "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. |
| "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. |
-| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. |
+| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot; `ignition.*` kargs are ignored with a console warning. |
| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. |
## Verification
@@ -264,6 +269,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS.
## See also
- [ddi-installer-build.md](ddi-installer-build.md) — local build, export, and dogfood workflow.
+- [diskless-troubleshooting.md](diskless-troubleshooting.md) — failure summary, minimum RAM, logs, supported Ignition subset.
- [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md) — release signing and transfer verification.
- [CONTEXT.md](../../CONTEXT.md) — canonical project domain glossary (OS DDI, Netboot UKI, Disk UKI, Slot).
- `systemd-sysinstall(8)`, `systemd-repart(8)`, `systemd-sysupdate(8)`, `bootctl(1)`, `ukify(1)`
diff --git a/docs/skills/diskless-troubleshooting.md b/docs/skills/diskless-troubleshooting.md
new file mode 100644
index 00000000..0ff6a17a
--- /dev/null
+++ b/docs/skills/diskless-troubleshooting.md
@@ -0,0 +1,181 @@
+---
+name: diskless-troubleshooting
+description: Use when a diskless (netboot UKI) node fails to boot, reboot-loops, needs more RAM, or needs its logs kept, and when writing an Ignition config for Bluefin Server; covers the initrd failure summary, minimum RAM, network and download failure modes, getting logs, and the supported Ignition subset.
+metadata:
+ type: how-to
+ status: stable
+ last_updated: "2026-09-29"
+ context7-sources:
+ - /systemd/systemd
+---
+# Diskless Boot Troubleshooting
+
+## When to Use
+
+- A diskless node reboots in a loop, or never reaches a login prompt.
+- Sizing RAM for diskless nodes.
+- Keeping logs from diskless boots.
+- Writing or debugging an Ignition config (diskless or installed).
+
+## When NOT to Use
+
+- The boot chain, pull and install architecture itself (see
+ [ddi-installer.md](ddi-installer.md)).
+- Building and dogfooding images (see
+ [ddi-installer-build.md](ddi-installer-build.md)).
+
+## What a failing node prints
+
+A failed initrd never drops to a shell (root is locked): it isolates
+`emergency.target`, whose `emergency.service` drop-in
+(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`) runs
+`/usr/libexec/bluefin-boot-diagnostics failure-summary` and then reboots. The
+summary lists each failed unit with a one-line hint and its last journal lines,
+the network links when the network or the download failed, any ignored
+`ignition.*` kernel arguments, and the last `journalctl -b -p err` lines. It is
+written to every active console (`/sys/class/tty/console/active`: the monitor
+and `ttyS0`), not only to `/dev/console`, which is just the last `console=`.
+
+The summary stays up for 60 s, then the node reboots and retries. Retrying is
+deliberate: a diskless node recovers unattended once the server, network or
+config is fixed, and on installed nodes boot counting falls back to the
+previous image. Change the delay (10 to 86400 s) with the
+`bluefin.failure_delay` system credential (SMBIOS type 11, QEMU fw_cfg, or
+`/loader/credentials/bluefin.failure_delay.cred` on the ESP); the
+`bluefin.failure_delay=` kernel argument overrides it only where the command
+line is not locked by Secure Boot. The 10 s floor keeps a persistent failure
+from becoming a hot reboot loop.
+
+Before the download, `bluefin-pull-check@.service` (wanted by
+`systemd-import@.service.d/10-bluefin-pull-check.conf` and ordered before the
+download) sends an HTTP HEAD to the image URL systemd-import-generator
+resolved. It stops the boot, with its own framed console message, only when a
+2xx answer gives a `Content-Length` that cannot fit in RAM. When it cannot
+check the size (no answer, an HTTP error such as the 403 a signed
+object-store URL gives to HEAD, or no `Content-Length`), it prints
+`BLUEFIN: cannot check that the OS image fits in RAM (); downloading it anyway`
+and the download runs: whether the image can be fetched is for systemd-import
+to find out. A failed download goes straight to the summary instead of
+waiting out the 300 s `/usr` device timeout.
+
+## Minimum RAM
+
+The netboot initrd downloads the whole OS DDI (`bluefin-server_.raw`,
+540 MiB for `26.09.x`) into `/run/machines`. systemd mounts `/run` as a tmpfs
+capped at 20% of RAM, and the DDI stays there for the whole boot because it
+backs `/usr`. The pull check requires the DDI plus 64 MiB to fit in both the
+free space of `/run` and `MemAvailable`, so usable RAM (`MemTotal`) must be at
+least about five times the DDI plus 64 MiB: about 3 GiB for a 540 MiB DDI.
+Measured in QEMU with the `26.09.x` DDI, `-m 3072` (2816 MiB `MemTotal`) is
+refused and `-m 3584` boots. Plan on 4 GiB plus the workload. If the server
+does not give the size, the check is skipped and a machine below the minimum
+fails during the download instead (`systemd-import@...service` failed).
+
+A machine below the minimum fails before the download (QEMU `-m 2048`):
+
+```text
+BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT
+This machine needs ~3072 MiB RAM, have 1876 MiB.
+The OS image bluefin-server_26.09.2.raw (540 MiB) is downloaded into /run, a RAM
+file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB.
+```
+
+## Failure modes
+
+| Symptom in the summary | Cause | Timing |
+|---|---|---|
+| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the download fails and the node reboots |
+| "no network interface found" | NIC driver not in the initrd module list (`initrd-modules` in `oci/bluefin-server-boot.bst`) | As above |
+| `systemd-import@...service` failed, systemd-importd says `Transfer failed: ...` | Routable link but no route to, or no answer from, the boot server | When the download starts |
+| `systemd-import@...service` failed, systemd-importd says `HTTP request to ... failed with code 404.` | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | When the download starts |
+| "NOT ENOUGH RAM FOR A DISKLESS BOOT" | See [Minimum RAM](#minimum-ram) | Seconds |
+| `systemd-import@...service` failed, with systemd-importd's reason (e.g. `DOWNLOAD INVALID: Checksum ... did not check out`) | Download interrupted, the DDI does not match `SHA256SUMS`, or `SHA256SUMS.gpg` is missing or not signed by a key in the image keyring | Right after the download |
+| `systemd-veritysetup@usr.service` or `sysusr-usr.mount` failed | The DDI does not match the `usrhash=` of the UKI that booted (mixed versions on the server) | After the download |
+| "No unit failed: a device or job timed out" | A device never appeared within `systemd.default_device_timeout_sec=300` | 300 s |
+| `ignition-.service` failed | See [Ignition](#ignition) | During the stage |
+
+## Getting logs from a diskless node
+
+Every diskless boot starts with a fresh machine ID and keeps its journal in
+RAM (`/run/log/journal` in the initrd, then `/var/log/journal` on the tmpfs
+root), so nothing survives a reboot by default.
+
+- **Initrd failures** never reach a disk: read the console. The netboot UKI
+ sets `console=tty0 console=ttyS0,115200`; use a serial cable, IPMI
+ serial-over-LAN or the BMC console log. `netconsole` is built into the
+ kernel, but its `netconsole=` argument cannot be added to the sealed command
+ line, so it is not available in the initrd.
+- **Persistent logs** after switch-root: provision a `/var` disk with Ignition
+ (`tests/fixtures/ignition/var-on-disk.ign`, the same state-disk pattern the
+ boot server's per-host `bluefin-node.ign` uses; `/var/log/journal` is created
+ on it by `systemd-tmpfiles`, so journald persists without extra config).
+ Each boot writes under its own machine ID and plain `journalctl` shows only
+ the current one, so read across boots with
+ `journalctl -D /var/log/journal --list-boots` and
+ `journalctl -D /var/log/journal -b -1`. Do not pin
+ the ID by writing `/etc/machine-id` with Ignition: a set machine ID ends the
+ first-boot state that applies the presets on every diskless boot, and
+ preset-enabled units stop starting. The `system.machine_id` credential (a
+ 32-hex ID, or `firmware` for the SMBIOS UUID) pins it without that effect.
+- **Remote logs**: the image ships `systemd-journal-upload`; an Ignition config
+ can write `/etc/systemd/journal-upload.conf` (`[Upload]` / `URL=`) and enable
+ `systemd-journal-upload.service` to stream the journal of the running system
+ to a `systemd-journal-remote` collector. It starts after switch-root, so it
+ does not cover initrd failures.
+
+## Ignition
+
+Configs arrive as the `ignition.config` / `ignition.config.url` credentials or
+as `bluefin-node.ign` next to the netboot UKI (see
+the Ignition section of [ddi-installer.md](ddi-installer.md)). Ignition kernel
+arguments (`ignition.config.url=`, `ignition.platform.id=`, ...) are never read;
+if any are on the command line, `bluefin-ignition-kargs.service` prints a
+framed warning naming them (not their values) and the failure summary repeats
+it.
+
+Stages wired into the initrd (`files/initrd-ignition/`), in order:
+`fetch-offline`, `fetch` (only when the config needs the network), `disks`,
+`mount`, `files`, and `umount` (`ExecStop` of `ignition-mount.service`). The
+`kargs` stage is not wired, so `kernelArguments` is silently not applied: the
+command line is sealed in the signed UKI.
+
+| Config section | Supported |
+|---|---|
+| `ignition.config.merge` / `replace`, `timeouts`, `security.tls` | Yes |
+| `storage.disks` (partitions via `sgdisk`) | Yes |
+| `storage.filesystems` | `xfs`, `ext4`, `vfat`, `swap`; no `btrfs` |
+| `storage.luks` | Key files only; no `clevis` (TPM2 / Tang) |
+| `storage.raid` | No (`mdadm` absent) |
+| `storage.files`, `directories`, `links` | Yes |
+| `systemd.units` (contents, dropins, `enabled`) | Yes; enabled units go through a preset applied by the real root |
+| `passwd.users`, `passwd.groups` | Yes |
+| `kernelArguments` | No |
+
+There is no first-boot marker: every diskless boot runs every stage against a
+fresh `/etc`, so a config must be idempotent or the node fails the same stage
+on every boot:
+
+- **Disks**: keep `wipeTable: false` and `wipePartitionEntry: false`; an
+ existing partition must match the config (number, label, size, type) or
+ `ignition-disks` fails.
+- **Filesystems**: `wipeFilesystem: false` reuses a filesystem with the same
+ format (and label/UUID when set) and fails on any other;
+ `wipeFilesystem: true` erases the disk on **every** boot.
+- **Files**: a file with `contents` fails the `files` stage when something is
+ already at its path and `overwrite` is false (the default): true from the
+ second boot for files on a persistent filesystem, and on every boot for
+ paths shipped in the factory `/etc`. Use `"overwrite": true`.
+
+## Verification
+
+- [ ] `just dogfood-check` passes (normal diskless boot).
+- [ ] `DOGFOOD_MEM=2048 scripts/dogfood-diskless.sh dist/diskless --check`
+ shows "NOT ENOUGH RAM FOR A DISKLESS BOOT" in `dist/diskless/dogfood-serial.log`.
+- [ ] `DOGFOOD_TAMPER=sums scripts/dogfood-diskless.sh dist/diskless --check`
+ shows "BLUEFIN: BOOT FAILED" and the failed `systemd-import@` unit.
+- [ ] `python3 -m pytest tests/unit/test_initrd_diagnostics.py` passes.
+
+## See also
+
+- [ddi-installer.md](ddi-installer.md) — boot chain, diskless pull, Ignition credentials.
+- [ddi-installer-build.md](ddi-installer-build.md) — dogfood harness and its `DOGFOOD_*` variables.
diff --git a/docs/skills/index.md b/docs/skills/index.md
index 084d53d3..92526196 100644
--- a/docs/skills/index.md
+++ b/docs/skills/index.md
@@ -4,7 +4,7 @@ description: Lazy-load manifest for Bluefin Server skills. Load this file after
metadata:
type: index
status: stable
- last_updated: "2026-09-27"
+ last_updated: "2026-09-28"
---
# docs/skills — Index
@@ -28,6 +28,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w
| [`ci-tooling.md`](ci-tooling.md) | GitHub Actions, workflow SHA pinning, CI conventions | CI conventions and release pipeline rules. |
| [`ddi-installer-build.md`](ddi-installer-build.md) | Building and dogfooding the image set | Local build, export, and QEMU dogfood workflow. |
| [`ddi-installer.md`](ddi-installer.md) | Boot flow, diskless pull, `systemd-sysinstall`, A/B sysupdate | Boot, install, and update architecture. |
+| [`diskless-troubleshooting.md`](diskless-troubleshooting.md) | Diskless node reboot-loops, RAM sizing, keeping logs, writing Ignition configs | Failure summary, minimum RAM, failure modes, logs, supported Ignition subset. |
| [`factory-integration.md`](factory-integration.md) | Lab integration, boot-test workflow, factory role | How Bluefin Server is consumed by the CI lab. |
| [`gap-analysis-distros.md`](gap-analysis-distros.md) | Comparing Bluefin Server to other server OSes | Source-verified comparison to Ubuntu, Talos, Flatcar, FCOS. |
| [`kubeadm-sysext.md`](kubeadm-sysext.md) | Joining a kubeadm cluster, kubelet/containerd sysext, Kubernetes kernel options | Build and runtime contract for the kubeadm worker sysext. |
diff --git a/elements/oci/bluefin-server-boot.bst b/elements/oci/bluefin-server-boot.bst
index 9437a2c3..49dbd85e 100644
--- a/elements/oci/bluefin-server-boot.bst
+++ b/elements/oci/bluefin-server-boot.bst
@@ -154,6 +154,7 @@ config:
ln -sf ../systemd-resolved.service "${wants}/initrd.target.wants/systemd-resolved.service"
ln -sf ../imports.target "${wants}/initrd.target.wants/imports.target"
ln -sf ../bluefin-populate-etc.service "${wants}/initrd.target.wants/bluefin-populate-etc.service"
+ ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service"
ln -sf ../systemd-networkd-wait-online.service "${wants}/network-online.target.wants/systemd-networkd-wait-online.service"
for tool in systemd-importd systemd-pull; do
[ -x "usr/lib/systemd/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
@@ -162,6 +163,10 @@ config:
[ -x "usr/bin/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
done
[ -x usr/lib/systemd/systemd-veritysetup ] || { echo "ERROR: initrd lacks systemd-veritysetup" >&2; exit 1; }
+ # bluefin-boot-diagnostics (failure summary, RAM check) and what it runs.
+ for tool in libexec/bluefin-boot-diagnostics bin/bash bin/curl bin/stat bin/timeout bin/tee; do
+ [ -x "usr/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
+ done
rm -rf usr/share/man usr/share/doc usr/share/info usr/include usr/lib/*/pkgconfig
find . -name '*.a' -delete
du -sh /initrd
diff --git a/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service
new file mode 100644
index 00000000..175a680d
--- /dev/null
+++ b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Check for ignored Ignition kernel arguments
+Documentation=file:/usr/libexec/bluefin-boot-diagnostics
+ConditionPathExists=/etc/initrd-release
+DefaultDependencies=no
+# Linked into initrd.target.wants by oci/bluefin-server-boot.bst. Warn before
+# Ignition would have run, so the warning sits next to where it is missing.
+Before=bluefin-ignition-credentials.service initrd.target
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/libexec/bluefin-boot-diagnostics ignition-kargs
diff --git a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service
new file mode 100644
index 00000000..5c233783
--- /dev/null
+++ b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service
@@ -0,0 +1,22 @@
+[Unit]
+Description=Check that the OS image fits in RAM before downloading it
+Documentation=file:/usr/libexec/bluefin-boot-diagnostics
+ConditionPathExists=/etc/initrd-release
+DefaultDependencies=no
+# Wanted by, and ordered before, systemd-import@%i.service through its drop-in
+# systemd-import@.service.d/10-bluefin-pull-check.conf. Needs the network,
+# like the download itself.
+Wants=network-online.target
+After=network-online.target
+
+# The check fails only when the server gives a size that cannot fit in RAM:
+# print why on the console and reboot (emergency.service.d/10-reboot.conf)
+# rather than start that download. When it cannot check the size, it notes
+# that on the console and succeeds, and the download runs.
+OnFailure=emergency.target
+OnFailureJobMode=isolate
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service
diff --git a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf
index db23c58b..ff82f25f 100644
--- a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf
+++ b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf
@@ -1,11 +1,18 @@
[Unit]
# An unattended server cannot use an emergency shell (root is locked).
-# Print what failed and reboot, so systemd-boot's boot counting can fall
-# back to the previous image after the new one exhausts its tries.
+# Print what failed and reboot, so a diskless node retries and systemd-boot's
+# boot counting can fall back to the previous image after the new one
+# exhausts its tries.
SuccessAction=reboot-force
FailureAction=reboot-force
[Service]
-Type=oneshot
+# idle: start once the isolate's stop jobs are done, so their status lines
+# do not interleave with the summary.
+Type=idle
+# How long the summary stays on screen before the reboot (seconds, default
+# 60); the bluefin.failure_delay= kernel argument overrides it where the
+# command line is not locked by Secure Boot.
+ImportCredential=bluefin.failure_delay
ExecStart=
-ExecStart=-/usr/bin/bash -c 'echo "Boot failed; errors follow, rebooting in 10s." >/dev/console; journalctl -b -p err --no-pager -o short-monotonic >/dev/console 2>&1; sleep 10'
+ExecStart=-/usr/libexec/bluefin-boot-diagnostics failure-summary
diff --git a/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf
new file mode 100644
index 00000000..a5a7b2df
--- /dev/null
+++ b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf
@@ -0,0 +1,15 @@
+[Unit]
+# The diskless initrd downloads the whole OS image into /run (RAM). Check
+# first that it fits, so a small machine fails with "needs ~X MiB RAM"
+# instead of an out-of-space error deep in the download. Wants=, not
+# Requires=: the check stops the boot through its own OnFailure=, which it
+# triggers only when it knows the image cannot fit; a check that is missing
+# or broken must not hold up the download.
+Wants=bluefin-pull-check@%i.service
+After=bluefin-pull-check@%i.service
+
+# Without this a failed download only surfaces when the /usr devices time out
+# (systemd.default_device_timeout_sec=300); go straight to the failure
+# summary and reboot instead.
+OnFailure=emergency.target
+OnFailureJobMode=isolate
diff --git a/files/initrd/usr/libexec/bluefin-boot-diagnostics b/files/initrd/usr/libexec/bluefin-boot-diagnostics
new file mode 100755
index 00000000..59ff4382
--- /dev/null
+++ b/files/initrd/usr/libexec/bluefin-boot-diagnostics
@@ -0,0 +1,304 @@
+#!/usr/bin/bash
+# bluefin-boot-diagnostics — say on the console why an initrd boot fails.
+#
+# check-pull UNIT Run before UNIT (a systemd-import@.service instance)
+# downloads the OS image into RAM. HEADs the image URL and
+# fails with a plain "needs ~X MiB RAM, have Y" message
+# only when a 2xx answer gives a size that cannot fit. If
+# it cannot check (no answer, an HTTP error, no
+# Content-Length), it prints a note and exits 0 so the
+# download still runs.
+# ignition-kargs Warn that ignition.* kernel arguments are ignored.
+# failure-summary ExecStart of emergency.service: print the failed units,
+# their last log lines and the last errors, then wait
+# (bluefin.failure_delay, default 60 s) so the reboot that
+# follows does not scroll them away.
+#
+# Messages go to every active console (/sys/class/tty/console/active, e.g. the
+# monitor and the serial port). /dev/console, where systemd prints its status
+# lines, is only the last console= on the kernel command line.
+#
+# bash builtins, coreutils, curl, systemctl and journalctl only: the initrd
+# has no sed or awk.
+
+set -uo pipefail
+export LC_ALL=C
+
+PROG="${0##*/}"
+DOCS=https://github.com/projectbluefin/server/blob/main/docs/skills/diskless-troubleshooting.md
+MEMINFO="${BLUEFIN_DIAG_MEMINFO:-/proc/meminfo}"
+CMDLINE="${BLUEFIN_DIAG_CMDLINE:-/proc/cmdline}"
+CONSOLES="${BLUEFIN_DIAG_CONSOLES:-/sys/class/tty/console/active}"
+DEVDIR="${BLUEFIN_DIAG_DEVDIR:-/dev}"
+NETDIR="${BLUEFIN_DIAG_NETDIR:-/sys/class/net}"
+# systemd-importd stores machine images under /run/machines in the initrd:
+# the /run tmpfs, which systemd caps at 20% of RAM.
+PULL_DIR=/run
+# Room /run and RAM must keep after the image has landed.
+MARGIN_MIB=64
+# check_pull's status when the image cannot fit. It differs from bash's own
+# error statuses (1, 2, 126, 127, 128+n), so a bug in the check is never
+# taken for "does not fit".
+TOO_BIG=3
+DELAY_NAME=bluefin.failure_delay
+DEFAULT_DELAY=60
+# The reboot is intentional (unattended retry, boot counting); a floor keeps
+# a persistent failure from turning into a hot reboot loop.
+MIN_DELAY=10
+MAX_DELAY=86400
+BAR='=============================================================================='
+
+# log PRIORITY MESSAGE: to the journal (stderr, sd-daemon priority prefix).
+log() { printf '<%s>%s: %s\n' "$1" "${PROG}" "$2" >&2; }
+
+# Copy stdin to every active console. A console that blocks must not hold up
+# the boot, so each write gets a timeout.
+to_consoles() {
+ local text c active=console
+ text="$(cat)"
+ [ -r "${CONSOLES}" ] && active="$(<"${CONSOLES}")"
+ for c in ${active}; do
+ timeout 5 tee -a "${DEVDIR}/${c}" >/dev/null <<<"${text}" || true
+ done
+}
+
+# shout PRIORITY LINE...: log every line and print them framed on the consoles.
+shout() {
+ local pri="$1" line
+ shift
+ for line in "$@"; do log "${pri}" "${line}"; done
+ { printf '\n%s\n' "${BAR}"; printf '%s\n' "$@"; printf '%s\n' "${BAR}"; } | to_consoles
+}
+
+mib() { printf '%s' "$(( ($1 + 1048575) / 1048576 ))"; }
+
+meminfo_kib() {
+ local key value _
+ while read -r key value _; do
+ [ "${key}" = "$1:" ] && { printf '%s' "${value}"; return 0; }
+ done < "${MEMINFO}"
+ return 1
+}
+
+# One line per network link: name, operstate, carrier. sysfs only, so it also
+# works after emergency.target has stopped systemd-networkd.
+links() {
+ local dev name state carrier out=()
+ for dev in "${NETDIR}"/*; do
+ name="${dev##*/}"
+ [ -e "${dev}/device" ] || continue
+ state="$(<"${dev}/operstate")" 2>/dev/null || state=unknown
+ carrier="$(<"${dev}/carrier")" 2>/dev/null || carrier=0
+ [ "${carrier}" = 1 ] && carrier=carrier || carrier="no carrier"
+ out+=("${name} ${state} (${carrier})")
+ done
+ if [ "${#out[@]}" -eq 0 ]; then
+ printf 'no network interface found (missing NIC driver in the initrd?)\n'
+ else
+ printf '%s\n' "${out[@]}"
+ fi
+}
+
+# cannot_check REASON: note on the consoles that the size was not checked. The
+# download goes ahead: it may still work (a signed object-store URL can refuse
+# HEAD and serve GET), and if it fails the failure summary says why.
+cannot_check() {
+ local msg="BLUEFIN: cannot check that the OS image fits in RAM ($1); downloading it anyway"
+ log 4 "${msg}"
+ printf '%s\n' "${msg}" | to_consoles
+}
+
+check_pull() {
+ local unit="${1:?usage: ${PROG} check-pull }"
+ local desc url out rc status='' len='' line lower
+ # systemd-import-generator describes each pull unit as "Download of ";
+ # take the URL it resolved (rd.systemd.pull, bootorigin, or the import.pull
+ # credential) rather than deriving it a second time.
+ desc="$(systemctl show -P Description "${unit}" 2>/dev/null)" || desc=
+ url="${desc#Download of }"
+ case "${url}" in
+ http://*|https://*) ;;
+ *) log 6 "${unit}: no http(s) source (${desc:-no description}); nothing to check"; return 0 ;;
+ esac
+
+ # Only the size is wanted here; whether the image downloads is for
+ # systemd-import to find out.
+ out="$(curl -sSIL --retry 3 --retry-delay 2 --connect-timeout 10 --max-time 120 "${url}" 2>&1)"
+ rc=$?
+ if [ "${rc}" -ne 0 ]; then
+ cannot_check "HEAD failed: ${out##*$'\n'}"
+ return 0
+ fi
+
+ # Headers of every response in the redirect chain; keep the last one.
+ while IFS= read -r line; do
+ line="${line%$'\r'}"
+ lower="${line,,}"
+ case "${lower}" in
+ http/*) status="${line#* }"; status="${status%% *}"; len= ;;
+ content-length:*) len="${line#*:}"; len="${len//[[:space:]]/}" ;;
+ esac
+ done <<<"${out}"
+
+ case "${status}" in
+ 2??) ;;
+ *) cannot_check "HEAD answered HTTP ${status:-?}"; return 0 ;;
+ esac
+ if ! [[ "${len}" =~ ^[0-9]+$ ]]; then
+ cannot_check "HEAD answered HTTP ${status} with no Content-Length"
+ return 0
+ fi
+
+ local total_kib avail_kib bavail bsize btotal
+ total_kib="$(meminfo_kib MemTotal)" || { cannot_check "no MemTotal in ${MEMINFO}"; return 0; }
+ avail_kib="$(meminfo_kib MemAvailable)" || avail_kib="${total_kib}"
+ read -r bavail bsize btotal < <(stat -f -c '%a %S %b' "${PULL_DIR}")
+ if ! [[ "${bavail:-}${bsize:-}${btotal:-}" =~ ^[0-9]+$ ]] || [ "${btotal}" -eq 0 ] || [ "${total_kib}" -eq 0 ]; then
+ cannot_check "cannot read the size of ${PULL_DIR} or of RAM"
+ return 0
+ fi
+
+ local need=$(( len + MARGIN_MIB * 1048576 ))
+ local run_free=$(( bavail * bsize )) run_size=$(( btotal * bsize ))
+ local total=$(( total_kib * 1024 )) avail=$(( avail_kib * 1024 ))
+ if [ "${need}" -le "${run_free}" ] && [ "${need}" -le "${avail}" ]; then
+ log 6 "OS image $(mib "${len}") MiB fits: /run has $(mib "${run_free}") MiB free, MemAvailable $(mib "${avail}") MiB"
+ return 0
+ fi
+
+ # /run is sized as a fixed share of RAM, so scale what it lacks up to RAM.
+ # Work in KiB so the product cannot overflow.
+ local used_kib=$(( (run_size - run_free) / 1024 )) need_kib=$(( need / 1024 ))
+ local want_run_kib=$(( (used_kib + need_kib) * total_kib / (run_size / 1024) ))
+ local want_run=$(( want_run_kib * 1024 ))
+ local want_mem=$(( total - avail + need ))
+ local want=$(( want_run > want_mem ? want_run : want_mem ))
+ # Round up to 64 MiB.
+ want=$(( (want + 67108863) / 67108864 * 67108864 ))
+ shout 2 "BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT" \
+ "This machine needs ~$(mib "${want}") MiB RAM, have $(mib "${total}") MiB." \
+ "The OS image ${url##*/} ($(mib "${len}") MiB) is downloaded into /run, a RAM" \
+ "file system capped at $(( run_size * 100 / total ))% of RAM: $(mib "${run_free}") MiB free there, MemAvailable $(mib "${avail}") MiB." \
+ "Add RAM or install to disk. ${DOCS}#minimum-ram"
+ return "${TOO_BIG}"
+}
+
+# Keys of ignition.* kernel arguments (values may carry secrets).
+ignition_karg_keys() {
+ local -a args
+ local arg keys=()
+ read -ra args < "${CMDLINE}" || true
+ for arg in "${args[@]}"; do
+ case "${arg}" in
+ ignition.*) keys+=("${arg%%=*}") ;;
+ esac
+ done
+ [ "${#keys[@]}" -gt 0 ] && printf '%s\n' "${keys[*]}"
+}
+
+ignition_kargs() {
+ local keys
+ keys="$(ignition_karg_keys)" || return 0
+ shout 4 "BLUEFIN: IGNORED KERNEL ARGUMENTS: ${keys}" \
+ "Bluefin Server never reads Ignition settings from the kernel command line." \
+ "Pass the config as the ignition.config or ignition.config.url system credential" \
+ "(SMBIOS type 11, QEMU fw_cfg, or /loader/credentials/ on the ESP), or serve" \
+ "bluefin-node.ign next to the netboot UKI. ${DOCS}#ignition"
+}
+
+# What a failed unit most likely means.
+hint() {
+ local stage
+ case "$1" in
+ bluefin-pull-check@*)
+ echo "the OS image is too big for the RAM of this machine (see the message above)" ;;
+ systemd-import@*)
+ echo "downloading or verifying the OS image failed: server reachable? SHA256SUMS signed by a key this image trusts? room in /run?" ;;
+ systemd-loop@*)
+ echo "attaching the OS image as a block device failed (also fails when the download did not happen)" ;;
+ systemd-networkd-wait-online.service)
+ echo "no network link became routable within 300 s: check the link and that a DHCP server answers" ;;
+ systemd-veritysetup@*|sysusr-usr.mount|sysroot-usr.mount)
+ echo "/usr could not be verified or mounted: the OS image does not match the usrhash= of this UKI" ;;
+ bluefin-ignition-credentials.service)
+ echo "staging the Ignition config failed (invalid ignition.config.url?). ${DOCS}#ignition" ;;
+ ignition-*.service)
+ stage="${1#ignition-}"
+ stage="${stage%.service}"
+ echo "Ignition stage '${stage}' failed. Every diskless boot runs every stage, so the config must be idempotent. ${DOCS}#ignition" ;;
+ esac
+}
+
+failure_delay() {
+ local value='' arg
+ local -a args
+ if [ -r "${CREDENTIALS_DIRECTORY:-/nonexistent}/${DELAY_NAME}" ]; then
+ value="$(<"${CREDENTIALS_DIRECTORY}/${DELAY_NAME}")"
+ fi
+ read -ra args < "${CMDLINE}" || true
+ for arg in "${args[@]}"; do
+ case "${arg}" in "${DELAY_NAME}="*) value="${arg#*=}" ;; esac
+ done
+ value="${value//[[:space:]]/}"
+ [[ "${value}" =~ ^[0-9]{1,6}$ ]] || value="${DEFAULT_DELAY}"
+ value=$(( 10#${value} ))
+ [ "${value}" -lt "${MIN_DELAY}" ] && value="${MIN_DELAY}"
+ [ "${value}" -gt "${MAX_DELAY}" ] && value="${MAX_DELAY}"
+ printf '%s' "${value}"
+}
+
+indent() { local l; while IFS= read -r l; do printf '%s%s\n' "$1" "${l}"; done; }
+
+failure_summary() {
+ local delay unit h keys shown=0
+ local -a units matches
+ delay="$(failure_delay)"
+ mapfile -t units < <(systemctl list-units --state=failed --no-legend --plain --full 2>/dev/null | cut -d' ' -f1)
+ {
+ printf '\n%s\n' "${BAR}"
+ printf 'BLUEFIN: BOOT FAILED\n'
+ if [ "${#units[@]}" -eq 0 ]; then
+ printf 'No unit failed: a device or job timed out (see the errors below).\n'
+ fi
+ for unit in "${units[@]}"; do
+ [ -n "${unit}" ] || continue
+ shown=$(( shown + 1 ))
+ [ "${shown}" -le 6 ] || { printf '(%s more failed units)\n' "$(( ${#units[@]} - 6 ))"; break; }
+ printf '\nFAILED: %s\n' "${unit}"
+ h="$(hint "${unit}")"
+ [ -z "${h}" ] || printf ' -> %s\n' "${h}"
+ # What the unit's own processes logged (not systemd's lines about it);
+ # a download's errors come from systemd-importd.
+ matches=("_SYSTEMD_UNIT=${unit}")
+ case "${unit}" in systemd-import@*) matches+=(_SYSTEMD_UNIT=systemd-importd.service) ;; esac
+ journalctl -b "${matches[@]}" -n 6 -o cat --no-pager 2>/dev/null | indent ' | '
+ case "${unit}" in
+ systemd-networkd-wait-online.service|systemd-import@*)
+ printf ' network links:\n'; links | indent ' ' ;;
+ esac
+ done
+ if keys="$(ignition_karg_keys)"; then
+ printf '\nIgnored kernel arguments: %s (use the ignition.config credential instead)\n' "${keys}"
+ fi
+ printf '\nLast errors (journalctl -b -p err):\n'
+ journalctl -b -p err -n 15 -o short-monotonic --no-pager 2>/dev/null | indent ' '
+ printf '\nHelp: %s\n' "${DOCS}"
+ printf 'Rebooting in %s s (set the %s credential to change the delay).\n' "${delay}" "${DELAY_NAME}"
+ printf '%s\n' "${BAR}"
+ } | to_consoles
+ sleep "${delay}"
+}
+
+case "${1:-}" in
+ check-pull)
+ shift
+ # Only "the image cannot fit" fails the unit, and so stops the boot. The
+ # check runs in a subshell so that a bug in it (an unset variable, a value
+ # it did not expect) lets the download go ahead instead.
+ rc=0
+ ( check_pull "$@" ) || rc=$?
+ [ "${rc}" -ne "${TOO_BIG}" ] || exit 1 ;;
+ ignition-kargs) ignition_kargs ;;
+ failure-summary) failure_summary ;;
+ *) printf 'usage: %s check-pull UNIT | ignition-kargs | failure-summary\n' "${PROG}" >&2; exit 2 ;;
+esac
diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py
new file mode 100644
index 00000000..3d9abafe
--- /dev/null
+++ b/tests/unit/test_initrd_diagnostics.py
@@ -0,0 +1,449 @@
+"""Initrd boot-failure diagnostics: units, drop-ins and the libexec helper.
+
+files/initrd/usr/libexec/bluefin-boot-diagnostics runs in the initrd only, so
+these tests drive it under bash on the host with systemctl, journalctl, stat
+and sleep stubbed on $PATH, a fake /proc/meminfo, fake consoles, and a local
+http.server answering the HEAD request the RAM check makes.
+"""
+
+from __future__ import annotations
+
+import http.server
+import os
+import re
+import shutil
+import socket
+import subprocess
+import threading
+from pathlib import Path
+
+import pytest
+
+ROOT = Path(__file__).resolve().parents[2]
+UNITS = ROOT / "files" / "initrd" / "usr" / "lib" / "systemd" / "system"
+IGN_UNITS = ROOT / "files" / "initrd-ignition" / "usr" / "lib" / "systemd" / "system"
+HELPER = ROOT / "files" / "initrd" / "usr" / "libexec" / "bluefin-boot-diagnostics"
+BOOT = ROOT / "elements" / "oci" / "bluefin-server-boot.bst"
+MIB = 1024 * 1024
+IMPORT_UNIT = "systemd-import@run-machines-rootdisk.raw.service"
+
+
+def unit(path: Path) -> dict[str, dict[str, list[str]]]:
+ """Parse a unit file keeping every value of repeated keys."""
+ sections: dict[str, dict[str, list[str]]] = {}
+ current = None
+ for raw in path.read_text(encoding="utf-8").splitlines():
+ line = raw.strip()
+ if not line or line.startswith(("#", ";")):
+ continue
+ if line.startswith("["):
+ current = sections.setdefault(line.strip("[]"), {})
+ continue
+ key, _, value = line.partition("=")
+ current.setdefault(key, []).append(value)
+ return sections
+
+
+def test_emergency_prints_the_summary_then_reboots() -> None:
+ u = unit(UNITS / "emergency.service.d" / "10-reboot.conf")
+ assert u["Unit"]["SuccessAction"] == ["reboot-force"]
+ assert u["Unit"]["FailureAction"] == ["reboot-force"]
+ # Reset the stock sulogin ExecStart, then run the summary (never fatal).
+ assert u["Service"]["ExecStart"] == ["", "-/usr/libexec/bluefin-boot-diagnostics failure-summary"]
+ assert u["Service"]["ImportCredential"] == ["bluefin.failure_delay"]
+ assert u["Service"]["Type"] == ["idle"]
+
+
+def test_every_download_is_preceded_by_the_ram_check() -> None:
+ u = unit(UNITS / "systemd-import@.service.d" / "10-bluefin-pull-check.conf")
+ # Only the check's own OnFailure= may stop the boot; its failure never fails the download.
+ assert u["Unit"]["Wants"] == ["bluefin-pull-check@%i.service"]
+ assert "Requires" not in u["Unit"]
+ assert u["Unit"]["After"] == ["bluefin-pull-check@%i.service"]
+ # A failed download goes straight to the summary, not a 300 s device timeout.
+ assert u["Unit"]["OnFailure"] == ["emergency.target"]
+ assert u["Unit"]["OnFailureJobMode"] == ["isolate"]
+
+
+def test_ram_check_unit_checks_its_import_instance() -> None:
+ u = unit(UNITS / "bluefin-pull-check@.service")
+ assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"]
+ assert u["Unit"]["DefaultDependencies"] == ["no"]
+ assert "network-online.target" in u["Unit"]["After"]
+ assert u["Unit"]["OnFailure"] == ["emergency.target"]
+ assert u["Unit"]["OnFailureJobMode"] == ["isolate"]
+ assert u["Service"]["ExecStart"] == [
+ "/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service"
+ ]
+
+
+def test_ignition_karg_warning_runs_in_every_initrd() -> None:
+ u = unit(UNITS / "bluefin-ignition-kargs.service")
+ assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"]
+ assert "bluefin-ignition-credentials.service" in " ".join(u["Unit"]["Before"])
+ assert u["Service"]["ExecStart"] == ["/usr/libexec/bluefin-boot-diagnostics ignition-kargs"]
+ boot = BOOT.read_text(encoding="utf-8")
+ assert 'ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service"' in boot
+ assert "libexec/bluefin-boot-diagnostics" in boot, "the UKI build checks the helper is in the initrd"
+
+
+@pytest.mark.parametrize("stage", ["fetch-offline", "fetch", "disks", "mount", "files"])
+def test_ignition_failures_reach_the_summary(stage: str) -> None:
+ u = unit(IGN_UNITS / f"ignition-{stage}.service")
+ assert u["Unit"]["OnFailure"] == ["emergency.target"]
+
+
+def test_helper_is_executable_bash_without_sed_or_awk() -> None:
+ text = HELPER.read_text(encoding="utf-8")
+ assert text.startswith("#!/usr/bin/bash\n")
+ assert os.access(HELPER, os.X_OK)
+ subprocess.run(["bash", "-n", str(HELPER)], check=True)
+ code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#"))
+ assert not re.search(r"\b(sed|awk|gawk)\b", code), "the initrd has no sed or awk"
+
+
+@pytest.mark.skipif(shutil.which("shellcheck") is None, reason="shellcheck not installed")
+def test_helper_is_shellcheck_clean() -> None:
+ subprocess.run(["shellcheck", "-S", "style", str(HELPER)], check=True)
+
+
+class _Handler(http.server.BaseHTTPRequestHandler):
+ routes: dict[str, tuple[int, dict[str, str]]] = {}
+
+ def do_HEAD(self) -> None: # noqa: N802 (http.server API)
+ status, headers = self.routes.get(self.path, (404, {"Content-Length": "0"}))
+ self.send_response(status)
+ for key, value in headers.items():
+ self.send_header(key, value)
+ self.end_headers()
+
+ def log_message(self, *args) -> None:
+ pass
+
+
+@pytest.fixture(scope="module")
+def server():
+ httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler)
+ thread = threading.Thread(target=httpd.serve_forever, daemon=True)
+ thread.start()
+ yield f"http://127.0.0.1:{httpd.server_address[1]}", _Handler.routes
+ httpd.shutdown()
+
+
+STUBS = {
+ "systemctl": """#!/bin/bash
+case "$1" in
+ show) printf '%s\\n' "${FAKE_DESCRIPTION}" ;;
+ list-units) for u in ${FAKE_FAILED:-}; do printf '%s loaded failed failed Fake unit\\n' "$u"; done ;;
+esac
+""",
+ "journalctl": """#!/bin/bash
+printf 'journal-line args: %s\\n' "$*"
+""",
+ "stat": """#!/bin/bash
+printf '%s\\n' "${FAKE_STATFS}"
+""",
+ "sleep": """#!/bin/bash
+printf '%s\\n' "$1" >> "${FAKE_SLEEP_LOG}"
+""",
+}
+
+
+class Env:
+ def __init__(self, tmp: Path) -> None:
+ self.tmp = tmp
+ self.bin = tmp / "bin"
+ self.bin.mkdir()
+ for name, body in STUBS.items():
+ (self.bin / name).write_text(body, encoding="utf-8")
+ (self.bin / name).chmod(0o755)
+ self.dev = tmp / "dev"
+ self.dev.mkdir()
+ (tmp / "consoles").write_text("tty0 ttyS0\n", encoding="utf-8")
+ self.net = tmp / "net"
+ nic = self.net / "enp1s0"
+ (nic / "device").mkdir(parents=True)
+ (nic / "operstate").write_text("down\n", encoding="utf-8")
+ (nic / "carrier").write_text("0\n", encoding="utf-8")
+ (self.net / "lo").mkdir() # virtual: no device link, not listed
+ self.cmdline("root=tmpfs")
+ self.meminfo(total_kib=4_000_000, avail_kib=3_600_000)
+ self.statfs(free_mib=800, size_mib=800)
+ self.creds = tmp / "creds"
+ self.creds.mkdir()
+ self.env = dict(
+ os.environ,
+ PATH=f"{self.bin}:{os.environ['PATH']}",
+ BLUEFIN_DIAG_MEMINFO=str(tmp / "meminfo"),
+ BLUEFIN_DIAG_CMDLINE=str(tmp / "cmdline"),
+ BLUEFIN_DIAG_CONSOLES=str(tmp / "consoles"),
+ BLUEFIN_DIAG_DEVDIR=str(self.dev),
+ BLUEFIN_DIAG_NETDIR=str(self.net),
+ CREDENTIALS_DIRECTORY=str(self.creds),
+ FAKE_SLEEP_LOG=str(tmp / "sleep.log"),
+ FAKE_DESCRIPTION="",
+ FAKE_FAILED="",
+ )
+
+ def cmdline(self, text: str) -> None:
+ (self.tmp / "cmdline").write_text(text + "\n", encoding="utf-8")
+
+ def meminfo(self, total_kib: int, avail_kib: int) -> None:
+ self.total_kib = total_kib
+ self.avail_kib = avail_kib
+ (self.tmp / "meminfo").write_text(
+ f"MemTotal: {total_kib} kB\nMemFree: {avail_kib} kB\n"
+ f"MemAvailable: {avail_kib} kB\nBuffers: 0 kB\n",
+ encoding="utf-8",
+ )
+
+ def statfs(self, free_mib: int, size_mib: int) -> None:
+ self.run_free = free_mib * MIB
+ self.run_size = size_mib * MIB
+ self.fake_statfs = f"{free_mib * 256} 4096 {size_mib * 256}"
+
+ def run(self, *args: str, **extra: str) -> subprocess.CompletedProcess:
+ env = dict(self.env, FAKE_STATFS=self.fake_statfs, **extra)
+ return subprocess.run(
+ ["bash", str(HELPER), *args], capture_output=True, text=True, env=env, timeout=60
+ )
+
+ def console(self, name: str = "ttyS0") -> str:
+ path = self.dev / name
+ return path.read_text(encoding="utf-8") if path.exists() else ""
+
+ def sleeps(self) -> list[str]:
+ log = self.tmp / "sleep.log"
+ return log.read_text(encoding="utf-8").split() if log.exists() else []
+
+
+@pytest.fixture
+def env(tmp_path: Path) -> Env:
+ return Env(tmp_path)
+
+
+def check_pull(env: Env, url: str, **extra: str) -> subprocess.CompletedProcess:
+ return env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION=f"Download of {url}", **extra)
+
+
+def cannot_check(reason: str) -> str:
+ return f"BLUEFIN: cannot check that the OS image fits in RAM ({reason}); downloading it anyway"
+
+
+def assert_download_goes_ahead(env: Env, result: subprocess.CompletedProcess, reason: str) -> None:
+ note = cannot_check(reason)
+ assert result.returncode == 0, result.stderr
+ for tty in ("tty0", "ttyS0"):
+ assert env.console(tty) == f"{note}\n"
+ assert f"<4>bluefin-boot-diagnostics: {note}\n" in result.stderr
+
+
+def test_image_that_fits_passes_quietly(env: Env, server) -> None:
+ base, routes = server
+ routes["/fits.raw"] = (200, {"Content-Length": str(540 * MIB)})
+ result = check_pull(env, f"{base}/fits.raw")
+ assert result.returncode == 0, result.stderr
+ assert "540 MiB fits" in result.stderr
+ assert env.console() == ""
+
+
+def _expected_mib(env: Env, size: int) -> int:
+ """Mirror of the helper's estimate, rounded up to 64 MiB."""
+ need = size + 64 * MIB
+ used_kib = (env.run_size - env.run_free) // 1024
+ want_run = (used_kib + need // 1024) * env.total_kib // (env.run_size // 1024) * 1024
+ want_mem = env.total_kib * 1024 - env.avail_kib * 1024 + need
+ want = max(want_run, want_mem)
+ return -(-want // (64 * MIB)) * 64
+
+
+def test_image_larger_than_run_fails_with_the_ram_it_needs(env: Env, server) -> None:
+ base, routes = server
+ size = 540 * MIB
+ routes["/bluefin-server_1.raw"] = (200, {"Content-Length": str(size)})
+ # 2 GB machine: /run is 20% of it.
+ env.meminfo(total_kib=2_000_000, avail_kib=1_800_000)
+ env.statfs(free_mib=380, size_mib=390)
+ result = check_pull(env, f"{base}/bluefin-server_1.raw")
+ assert result.returncode == 1
+ want = _expected_mib(env, size)
+ assert 2900 < want < 3200
+ for tty in ("tty0", "ttyS0"):
+ text = env.console(tty)
+ assert "NOT ENOUGH RAM FOR A DISKLESS BOOT" in text
+ assert f"needs ~{want} MiB RAM, have 1954 MiB" in text
+ assert "bluefin-server_1.raw (540 MiB)" in text
+ assert "capped at 19% of RAM: 380 MiB free" in text
+ assert f"<2>bluefin-boot-diagnostics: This machine needs ~{want} MiB RAM" in result.stderr
+
+
+def test_low_memavailable_fails_even_when_run_has_room(env: Env, server) -> None:
+ base, routes = server
+ size = 300 * MIB
+ routes["/small.raw"] = (200, {"Content-Length": str(size)})
+ env.meminfo(total_kib=4_000_000, avail_kib=200_000)
+ env.statfs(free_mib=780, size_mib=800)
+ result = check_pull(env, f"{base}/small.raw")
+ assert result.returncode == 1
+ assert f"needs ~{_expected_mib(env, size)} MiB RAM, have 3907 MiB" in env.console()
+
+
+def test_redirects_are_followed_to_the_final_size(env: Env, server) -> None:
+ base, routes = server
+ routes["/redirect.raw"] = (302, {"Location": "/huge.raw", "Content-Length": "0"})
+ routes["/huge.raw"] = (200, {"Content-Length": str(4000 * MIB)})
+ result = check_pull(env, f"{base}/redirect.raw")
+ assert result.returncode == 1
+ assert "redirect.raw (4000 MiB)" in env.console()
+
+
+@pytest.mark.parametrize("status", [403, 404, 405, 500, 501])
+def test_head_errors_do_not_block_the_download(env: Env, server, status: int) -> None:
+ # A signed object-store URL covers the method: GET works, HEAD gets 403.
+ base, routes = server
+ routes[f"/head{status}.raw"] = (status, {"Content-Length": "0"})
+ result = check_pull(env, f"{base}/head{status}.raw")
+ assert_download_goes_ahead(env, result, f"HEAD answered HTTP {status}")
+
+
+def test_missing_content_length_skips_the_check(env: Env, server) -> None:
+ base, routes = server
+ routes["/chunked.raw"] = (200, {"Transfer-Encoding": "chunked"})
+ result = check_pull(env, f"{base}/chunked.raw")
+ assert_download_goes_ahead(env, result, "HEAD answered HTTP 200 with no Content-Length")
+
+
+def test_unreachable_server_does_not_block_the_download(env: Env) -> None:
+ with socket.socket() as s:
+ s.bind(("127.0.0.1", 0))
+ port = s.getsockname()[1]
+ result = check_pull(env, f"http://127.0.0.1:{port}/img.raw")
+ assert result.returncode == 0, result.stderr
+ text = env.console()
+ assert text.startswith("BLUEFIN: cannot check that the OS image fits in RAM (HEAD failed: curl: (7) ")
+ assert text.endswith("); downloading it anyway\n")
+ assert text.count("\n") == 1
+ assert env.console("tty0") == text
+ assert f"<4>bluefin-boot-diagnostics: {text}" in result.stderr
+
+
+def test_a_broken_check_does_not_block_the_download(env: Env, server) -> None:
+ base, routes = server
+ routes["/broken.raw"] = (200, {"Content-Length": str(540 * MIB)})
+ # A value the check does not expect makes bash abort it (unbound variable).
+ (env.tmp / "meminfo").write_text("MemTotal: 4000000 kB\nMemAvailable: n/a kB\n", encoding="utf-8")
+ result = check_pull(env, f"{base}/broken.raw")
+ assert "unbound variable" in result.stderr
+ assert result.returncode == 0
+ assert env.console() == ""
+
+
+def test_non_http_sources_are_not_checked(env: Env) -> None:
+ result = env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION="Download of file:///run/img.raw")
+ assert result.returncode == 0
+ assert env.console() == ""
+
+
+def test_ignition_kargs_are_named_without_their_values(env: Env) -> None:
+ env.cmdline("root=tmpfs ignition.config.url=http://10.0.0.1/secret-token.ign ignition.platform.id=metal quiet")
+ result = env.run("ignition-kargs")
+ assert result.returncode == 0
+ text = env.console("tty0")
+ assert "IGNORED KERNEL ARGUMENTS: ignition.config.url ignition.platform.id" in text
+ assert "ignition.config or ignition.config.url system credential" in text
+ assert "secret-token" not in text + result.stderr
+ assert "<4>bluefin-boot-diagnostics: BLUEFIN: IGNORED KERNEL ARGUMENTS" in result.stderr
+
+
+def test_no_ignition_kargs_no_warning(env: Env) -> None:
+ env.cmdline("root=tmpfs rd.systemd.pull=raw,machine:rootdisk:http://x/y.raw")
+ result = env.run("ignition-kargs")
+ assert result.returncode == 0
+ assert result.stderr == ""
+ assert env.console() == ""
+
+
+def test_failure_summary_explains_each_failed_unit(env: Env) -> None:
+ result = env.run(
+ "failure-summary",
+ FAKE_FAILED=f"ignition-disks.service {IMPORT_UNIT} systemd-networkd-wait-online.service",
+ )
+ assert result.returncode == 0
+ text = env.console("tty0")
+ assert text == env.console("ttyS0")
+ assert "BLUEFIN: BOOT FAILED" in text
+ assert "FAILED: ignition-disks.service" in text
+ assert "Ignition stage 'disks' failed" in text
+ assert "must be idempotent" in text
+ assert f"FAILED: {IMPORT_UNIT}" in text
+ assert "SHA256SUMS signed by a key this image trusts" in text
+ assert "no network link became routable within 300 s" in text
+ assert "enp1s0 down (no carrier)" in text
+ assert (
+ f"| journal-line args: -b _SYSTEMD_UNIT={IMPORT_UNIT} _SYSTEMD_UNIT=systemd-importd.service -n 6 -o cat --no-pager"
+ in text
+ ), "a failed download shows systemd-importd's reason"
+ assert "| journal-line args: -b _SYSTEMD_UNIT=ignition-disks.service -n 6" in text
+ assert "Last errors (journalctl -b -p err):" in text
+ assert "docs/skills/diskless-troubleshooting.md\n" in text
+ assert "Rebooting in 60 s" in text
+ assert env.sleeps() == ["60"]
+
+
+def test_failure_summary_blames_ram_for_a_failed_check(env: Env) -> None:
+ env.run("failure-summary", FAKE_FAILED="bluefin-pull-check@run-machines-rootdisk.raw.service")
+ text = env.console()
+ assert "-> the OS image is too big for the RAM of this machine" in text
+ assert "network links:" not in text
+
+
+def test_failure_summary_without_failed_units_points_at_timeouts(env: Env) -> None:
+ env.cmdline("root=tmpfs ignition.config.url=http://x/y.ign")
+ env.run("failure-summary")
+ text = env.console()
+ assert "No unit failed: a device or job timed out" in text
+ assert "Ignored kernel arguments: ignition.config.url" in text
+
+
+@pytest.mark.parametrize(
+ "credential,karg,expected",
+ [
+ (None, None, "60"),
+ ("120\n", None, "120"),
+ ("3", None, "10"), # floor: no hot reboot loop
+ ("0600", None, "600"), # decimal, not octal
+ ("999999999", None, "60"), # not a sane number
+ ("soon", None, "60"),
+ ("120", "30", "30"), # the kernel argument wins where it can be set
+ (None, "100000", "86400"),
+ ],
+)
+def test_failure_delay(env: Env, credential: str | None, karg: str | None, expected: str) -> None:
+ if credential is not None:
+ (env.creds / "bluefin.failure_delay").write_text(credential, encoding="utf-8")
+ if karg is not None:
+ env.cmdline(f"root=tmpfs bluefin.failure_delay={karg}")
+ env.run("failure-summary")
+ assert env.sleeps() == [expected]
+ assert f"Rebooting in {expected} s" in env.console()
+
+
+def test_unknown_mode_is_a_usage_error(env: Env) -> None:
+ result = env.run("bogus")
+ assert result.returncode == 2
+ assert "usage:" in result.stderr
+
+
+def test_console_links_point_at_existing_doc_sections() -> None:
+ text = HELPER.read_text(encoding="utf-8")
+ base = re.search(r"^DOCS=https://github\.com/projectbluefin/server/blob/main/(\S+)$", text, re.M)
+ assert base, "DOCS must link the repository's main branch"
+ doc = (ROOT / base.group(1)).read_text(encoding="utf-8")
+ slugs = {
+ re.sub(r"[^a-z0-9 -]", "", line.lstrip("#").strip().lower()).replace(" ", "-")
+ for line in doc.splitlines()
+ if line.startswith("## ")
+ }
+ anchors = set(re.findall(r"\$\{DOCS\}#([a-z0-9-]+)", text))
+ assert anchors == {"minimum-ram", "ignition"}
+ assert anchors <= slugs