diff --git a/AGENTS.md b/AGENTS.md index 4d24050c..e6419e9f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -63,6 +63,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j | Task | Skill | |---|---| | Boot / install / update architecture and local build + dogfood | [`docs/skills/ddi-installer.md`](docs/skills/ddi-installer.md), [`docs/skills/ddi-installer-build.md`](docs/skills/ddi-installer-build.md) | +| Diskless boot failures, RAM sizing, node logs, Ignition configs | [`docs/skills/diskless-troubleshooting.md`](docs/skills/diskless-troubleshooting.md) | | Factory role, k0s sysext rationale, lab integration | [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md) | | Work with `systemd-sysext` / `systemd-confext` | [`docs/skills/systemd-sysext-extensions.md`](docs/skills/systemd-sysext-extensions.md) | | Build or run the kubeadm worker sysext | [`docs/skills/kubeadm-sysext.md`](docs/skills/kubeadm-sysext.md) | diff --git a/docs/skills/ddi-installer-build.md b/docs/skills/ddi-installer-build.md index 0d8e1067..797f2854 100644 --- a/docs/skills/ddi-installer-build.md +++ b/docs/skills/ddi-installer-build.md @@ -100,6 +100,8 @@ Useful environment variables: - `DOGFOOD_SERVE_EXTRA=` — also serve the files in ``. - `DOGFOOD_TAMPER=raw|sums` — serve a corrupted DDI or a re-hashed, unsigned `SHA256SUMS`; the boot must fail, proving the signature check. +- `DOGFOOD_MEM=` — guest RAM (default 4096); below the diskless minimum + the boot must fail with the RAM message (see `diskless-troubleshooting.md`). - `DOGFOOD_EXTRA_PROBE=` — shell snippet appended to the in-guest probe. `scripts/dogfood-install.sh [ []]` is the full diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index 44d4f673..de749811 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d metadata: type: reference status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" context7-sources: - /systemd/systemd - /apache/buildstream @@ -87,9 +87,11 @@ ships gnupg and the image keyring, and importd fetches `SHA256SUMS` and using the manifest. The manifest hash check pins the DDI, the signed UKI pins `usrhash=`, and dm-verity checks every `/usr` block against it; a tampered DDI or a re-hashed unsigned manifest is refused. A failed boot never drops to an -emergency shell: the initrd prints the errors and reboots -(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`), -which is what lets boot counting work unattended. +emergency shell: the initrd prints a failure summary on every console and +reboots (`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`), +which is what lets boot counting work unattended. Before the pull, a check +refuses an image that cannot fit in RAM. What the node prints, minimum RAM, +failure modes and logs: [diskless-troubleshooting.md](diskless-troubleshooting.md). ### UEFI HTTP boot @@ -222,7 +224,10 @@ EFI variable and, when the server offers `bluefin-node.ign` next to the UKI, applies it. Ignition runs on **every** boot (there is no first-boot marker on a tmpfs root), so configs must be idempotent. See `tests/fixtures/ignition/var-on-disk.ign` for a dogfood-tested example -(persistent /var on a second disk plus an SSH key). +(persistent /var on a second disk plus an SSH key). The supported stages and +config sections, the idempotency rules, and the warning for ignored +`ignition.*` kernel arguments are in +the Ignition section of [diskless-troubleshooting.md](diskless-troubleshooting.md). ## PXE / HTTP boot service @@ -249,7 +254,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. | "Kernel image is at `/boot/vmlinuz`." | FSDK installs kernels into `/usr/lib/modules//vmlinuz`; `bluefin-server-boot.bst` picks it up from there for ukify. | | "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. | | "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. | -| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. | +| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot; `ignition.*` kargs are ignored with a console warning. | | "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. | ## Verification @@ -264,6 +269,7 @@ specifying the URL to pull `bluefin-server_.raw` and its signed SHA256SUMS. ## See also - [ddi-installer-build.md](ddi-installer-build.md) — local build, export, and dogfood workflow. +- [diskless-troubleshooting.md](diskless-troubleshooting.md) — failure summary, minimum RAM, logs, supported Ignition subset. - [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md) — release signing and transfer verification. - [CONTEXT.md](../../CONTEXT.md) — canonical project domain glossary (OS DDI, Netboot UKI, Disk UKI, Slot). - `systemd-sysinstall(8)`, `systemd-repart(8)`, `systemd-sysupdate(8)`, `bootctl(1)`, `ukify(1)` diff --git a/docs/skills/diskless-troubleshooting.md b/docs/skills/diskless-troubleshooting.md new file mode 100644 index 00000000..0ff6a17a --- /dev/null +++ b/docs/skills/diskless-troubleshooting.md @@ -0,0 +1,181 @@ +--- +name: diskless-troubleshooting +description: Use when a diskless (netboot UKI) node fails to boot, reboot-loops, needs more RAM, or needs its logs kept, and when writing an Ignition config for Bluefin Server; covers the initrd failure summary, minimum RAM, network and download failure modes, getting logs, and the supported Ignition subset. +metadata: + type: how-to + status: stable + last_updated: "2026-09-29" + context7-sources: + - /systemd/systemd +--- +# Diskless Boot Troubleshooting + +## When to Use + +- A diskless node reboots in a loop, or never reaches a login prompt. +- Sizing RAM for diskless nodes. +- Keeping logs from diskless boots. +- Writing or debugging an Ignition config (diskless or installed). + +## When NOT to Use + +- The boot chain, pull and install architecture itself (see + [ddi-installer.md](ddi-installer.md)). +- Building and dogfooding images (see + [ddi-installer-build.md](ddi-installer-build.md)). + +## What a failing node prints + +A failed initrd never drops to a shell (root is locked): it isolates +`emergency.target`, whose `emergency.service` drop-in +(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`) runs +`/usr/libexec/bluefin-boot-diagnostics failure-summary` and then reboots. The +summary lists each failed unit with a one-line hint and its last journal lines, +the network links when the network or the download failed, any ignored +`ignition.*` kernel arguments, and the last `journalctl -b -p err` lines. It is +written to every active console (`/sys/class/tty/console/active`: the monitor +and `ttyS0`), not only to `/dev/console`, which is just the last `console=`. + +The summary stays up for 60 s, then the node reboots and retries. Retrying is +deliberate: a diskless node recovers unattended once the server, network or +config is fixed, and on installed nodes boot counting falls back to the +previous image. Change the delay (10 to 86400 s) with the +`bluefin.failure_delay` system credential (SMBIOS type 11, QEMU fw_cfg, or +`/loader/credentials/bluefin.failure_delay.cred` on the ESP); the +`bluefin.failure_delay=` kernel argument overrides it only where the command +line is not locked by Secure Boot. The 10 s floor keeps a persistent failure +from becoming a hot reboot loop. + +Before the download, `bluefin-pull-check@.service` (wanted by +`systemd-import@.service.d/10-bluefin-pull-check.conf` and ordered before the +download) sends an HTTP HEAD to the image URL systemd-import-generator +resolved. It stops the boot, with its own framed console message, only when a +2xx answer gives a `Content-Length` that cannot fit in RAM. When it cannot +check the size (no answer, an HTTP error such as the 403 a signed +object-store URL gives to HEAD, or no `Content-Length`), it prints +`BLUEFIN: cannot check that the OS image fits in RAM (); downloading it anyway` +and the download runs: whether the image can be fetched is for systemd-import +to find out. A failed download goes straight to the summary instead of +waiting out the 300 s `/usr` device timeout. + +## Minimum RAM + +The netboot initrd downloads the whole OS DDI (`bluefin-server_.raw`, +540 MiB for `26.09.x`) into `/run/machines`. systemd mounts `/run` as a tmpfs +capped at 20% of RAM, and the DDI stays there for the whole boot because it +backs `/usr`. The pull check requires the DDI plus 64 MiB to fit in both the +free space of `/run` and `MemAvailable`, so usable RAM (`MemTotal`) must be at +least about five times the DDI plus 64 MiB: about 3 GiB for a 540 MiB DDI. +Measured in QEMU with the `26.09.x` DDI, `-m 3072` (2816 MiB `MemTotal`) is +refused and `-m 3584` boots. Plan on 4 GiB plus the workload. If the server +does not give the size, the check is skipped and a machine below the minimum +fails during the download instead (`systemd-import@...service` failed). + +A machine below the minimum fails before the download (QEMU `-m 2048`): + +```text +BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT +This machine needs ~3072 MiB RAM, have 1876 MiB. +The OS image bluefin-server_26.09.2.raw (540 MiB) is downloaded into /run, a RAM +file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB. +``` + +## Failure modes + +| Symptom in the summary | Cause | Timing | +|---|---|---| +| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the download fails and the node reboots | +| "no network interface found" | NIC driver not in the initrd module list (`initrd-modules` in `oci/bluefin-server-boot.bst`) | As above | +| `systemd-import@...service` failed, systemd-importd says `Transfer failed: ...` | Routable link but no route to, or no answer from, the boot server | When the download starts | +| `systemd-import@...service` failed, systemd-importd says `HTTP request to ... failed with code 404.` | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | When the download starts | +| "NOT ENOUGH RAM FOR A DISKLESS BOOT" | See [Minimum RAM](#minimum-ram) | Seconds | +| `systemd-import@...service` failed, with systemd-importd's reason (e.g. `DOWNLOAD INVALID: Checksum ... did not check out`) | Download interrupted, the DDI does not match `SHA256SUMS`, or `SHA256SUMS.gpg` is missing or not signed by a key in the image keyring | Right after the download | +| `systemd-veritysetup@usr.service` or `sysusr-usr.mount` failed | The DDI does not match the `usrhash=` of the UKI that booted (mixed versions on the server) | After the download | +| "No unit failed: a device or job timed out" | A device never appeared within `systemd.default_device_timeout_sec=300` | 300 s | +| `ignition-.service` failed | See [Ignition](#ignition) | During the stage | + +## Getting logs from a diskless node + +Every diskless boot starts with a fresh machine ID and keeps its journal in +RAM (`/run/log/journal` in the initrd, then `/var/log/journal` on the tmpfs +root), so nothing survives a reboot by default. + +- **Initrd failures** never reach a disk: read the console. The netboot UKI + sets `console=tty0 console=ttyS0,115200`; use a serial cable, IPMI + serial-over-LAN or the BMC console log. `netconsole` is built into the + kernel, but its `netconsole=` argument cannot be added to the sealed command + line, so it is not available in the initrd. +- **Persistent logs** after switch-root: provision a `/var` disk with Ignition + (`tests/fixtures/ignition/var-on-disk.ign`, the same state-disk pattern the + boot server's per-host `bluefin-node.ign` uses; `/var/log/journal` is created + on it by `systemd-tmpfiles`, so journald persists without extra config). + Each boot writes under its own machine ID and plain `journalctl` shows only + the current one, so read across boots with + `journalctl -D /var/log/journal --list-boots` and + `journalctl -D /var/log/journal -b -1`. Do not pin + the ID by writing `/etc/machine-id` with Ignition: a set machine ID ends the + first-boot state that applies the presets on every diskless boot, and + preset-enabled units stop starting. The `system.machine_id` credential (a + 32-hex ID, or `firmware` for the SMBIOS UUID) pins it without that effect. +- **Remote logs**: the image ships `systemd-journal-upload`; an Ignition config + can write `/etc/systemd/journal-upload.conf` (`[Upload]` / `URL=`) and enable + `systemd-journal-upload.service` to stream the journal of the running system + to a `systemd-journal-remote` collector. It starts after switch-root, so it + does not cover initrd failures. + +## Ignition + +Configs arrive as the `ignition.config` / `ignition.config.url` credentials or +as `bluefin-node.ign` next to the netboot UKI (see +the Ignition section of [ddi-installer.md](ddi-installer.md)). Ignition kernel +arguments (`ignition.config.url=`, `ignition.platform.id=`, ...) are never read; +if any are on the command line, `bluefin-ignition-kargs.service` prints a +framed warning naming them (not their values) and the failure summary repeats +it. + +Stages wired into the initrd (`files/initrd-ignition/`), in order: +`fetch-offline`, `fetch` (only when the config needs the network), `disks`, +`mount`, `files`, and `umount` (`ExecStop` of `ignition-mount.service`). The +`kargs` stage is not wired, so `kernelArguments` is silently not applied: the +command line is sealed in the signed UKI. + +| Config section | Supported | +|---|---| +| `ignition.config.merge` / `replace`, `timeouts`, `security.tls` | Yes | +| `storage.disks` (partitions via `sgdisk`) | Yes | +| `storage.filesystems` | `xfs`, `ext4`, `vfat`, `swap`; no `btrfs` | +| `storage.luks` | Key files only; no `clevis` (TPM2 / Tang) | +| `storage.raid` | No (`mdadm` absent) | +| `storage.files`, `directories`, `links` | Yes | +| `systemd.units` (contents, dropins, `enabled`) | Yes; enabled units go through a preset applied by the real root | +| `passwd.users`, `passwd.groups` | Yes | +| `kernelArguments` | No | + +There is no first-boot marker: every diskless boot runs every stage against a +fresh `/etc`, so a config must be idempotent or the node fails the same stage +on every boot: + +- **Disks**: keep `wipeTable: false` and `wipePartitionEntry: false`; an + existing partition must match the config (number, label, size, type) or + `ignition-disks` fails. +- **Filesystems**: `wipeFilesystem: false` reuses a filesystem with the same + format (and label/UUID when set) and fails on any other; + `wipeFilesystem: true` erases the disk on **every** boot. +- **Files**: a file with `contents` fails the `files` stage when something is + already at its path and `overwrite` is false (the default): true from the + second boot for files on a persistent filesystem, and on every boot for + paths shipped in the factory `/etc`. Use `"overwrite": true`. + +## Verification + +- [ ] `just dogfood-check` passes (normal diskless boot). +- [ ] `DOGFOOD_MEM=2048 scripts/dogfood-diskless.sh dist/diskless --check` + shows "NOT ENOUGH RAM FOR A DISKLESS BOOT" in `dist/diskless/dogfood-serial.log`. +- [ ] `DOGFOOD_TAMPER=sums scripts/dogfood-diskless.sh dist/diskless --check` + shows "BLUEFIN: BOOT FAILED" and the failed `systemd-import@` unit. +- [ ] `python3 -m pytest tests/unit/test_initrd_diagnostics.py` passes. + +## See also + +- [ddi-installer.md](ddi-installer.md) — boot chain, diskless pull, Ignition credentials. +- [ddi-installer-build.md](ddi-installer-build.md) — dogfood harness and its `DOGFOOD_*` variables. diff --git a/docs/skills/index.md b/docs/skills/index.md index 084d53d3..92526196 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -4,7 +4,7 @@ description: Lazy-load manifest for Bluefin Server skills. Load this file after metadata: type: index status: stable - last_updated: "2026-09-27" + last_updated: "2026-09-28" --- # docs/skills — Index @@ -28,6 +28,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w | [`ci-tooling.md`](ci-tooling.md) | GitHub Actions, workflow SHA pinning, CI conventions | CI conventions and release pipeline rules. | | [`ddi-installer-build.md`](ddi-installer-build.md) | Building and dogfooding the image set | Local build, export, and QEMU dogfood workflow. | | [`ddi-installer.md`](ddi-installer.md) | Boot flow, diskless pull, `systemd-sysinstall`, A/B sysupdate | Boot, install, and update architecture. | +| [`diskless-troubleshooting.md`](diskless-troubleshooting.md) | Diskless node reboot-loops, RAM sizing, keeping logs, writing Ignition configs | Failure summary, minimum RAM, failure modes, logs, supported Ignition subset. | | [`factory-integration.md`](factory-integration.md) | Lab integration, boot-test workflow, factory role | How Bluefin Server is consumed by the CI lab. | | [`gap-analysis-distros.md`](gap-analysis-distros.md) | Comparing Bluefin Server to other server OSes | Source-verified comparison to Ubuntu, Talos, Flatcar, FCOS. | | [`kubeadm-sysext.md`](kubeadm-sysext.md) | Joining a kubeadm cluster, kubelet/containerd sysext, Kubernetes kernel options | Build and runtime contract for the kubeadm worker sysext. | diff --git a/elements/oci/bluefin-server-boot.bst b/elements/oci/bluefin-server-boot.bst index 9437a2c3..49dbd85e 100644 --- a/elements/oci/bluefin-server-boot.bst +++ b/elements/oci/bluefin-server-boot.bst @@ -154,6 +154,7 @@ config: ln -sf ../systemd-resolved.service "${wants}/initrd.target.wants/systemd-resolved.service" ln -sf ../imports.target "${wants}/initrd.target.wants/imports.target" ln -sf ../bluefin-populate-etc.service "${wants}/initrd.target.wants/bluefin-populate-etc.service" + ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service" ln -sf ../systemd-networkd-wait-online.service "${wants}/network-online.target.wants/systemd-networkd-wait-online.service" for tool in systemd-importd systemd-pull; do [ -x "usr/lib/systemd/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } @@ -162,6 +163,10 @@ config: [ -x "usr/bin/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } done [ -x usr/lib/systemd/systemd-veritysetup ] || { echo "ERROR: initrd lacks systemd-veritysetup" >&2; exit 1; } + # bluefin-boot-diagnostics (failure summary, RAM check) and what it runs. + for tool in libexec/bluefin-boot-diagnostics bin/bash bin/curl bin/stat bin/timeout bin/tee; do + [ -x "usr/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; } + done rm -rf usr/share/man usr/share/doc usr/share/info usr/include usr/lib/*/pkgconfig find . -name '*.a' -delete du -sh /initrd diff --git a/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service new file mode 100644 index 00000000..175a680d --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service @@ -0,0 +1,13 @@ +[Unit] +Description=Check for ignored Ignition kernel arguments +Documentation=file:/usr/libexec/bluefin-boot-diagnostics +ConditionPathExists=/etc/initrd-release +DefaultDependencies=no +# Linked into initrd.target.wants by oci/bluefin-server-boot.bst. Warn before +# Ignition would have run, so the warning sits next to where it is missing. +Before=bluefin-ignition-credentials.service initrd.target + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/libexec/bluefin-boot-diagnostics ignition-kargs diff --git a/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service new file mode 100644 index 00000000..5c233783 --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service @@ -0,0 +1,22 @@ +[Unit] +Description=Check that the OS image fits in RAM before downloading it +Documentation=file:/usr/libexec/bluefin-boot-diagnostics +ConditionPathExists=/etc/initrd-release +DefaultDependencies=no +# Wanted by, and ordered before, systemd-import@%i.service through its drop-in +# systemd-import@.service.d/10-bluefin-pull-check.conf. Needs the network, +# like the download itself. +Wants=network-online.target +After=network-online.target + +# The check fails only when the server gives a size that cannot fit in RAM: +# print why on the console and reboot (emergency.service.d/10-reboot.conf) +# rather than start that download. When it cannot check the size, it notes +# that on the console and succeeds, and the download runs. +OnFailure=emergency.target +OnFailureJobMode=isolate + +[Service] +Type=oneshot +RemainAfterExit=yes +ExecStart=/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service diff --git a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf index db23c58b..ff82f25f 100644 --- a/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf +++ b/files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf @@ -1,11 +1,18 @@ [Unit] # An unattended server cannot use an emergency shell (root is locked). -# Print what failed and reboot, so systemd-boot's boot counting can fall -# back to the previous image after the new one exhausts its tries. +# Print what failed and reboot, so a diskless node retries and systemd-boot's +# boot counting can fall back to the previous image after the new one +# exhausts its tries. SuccessAction=reboot-force FailureAction=reboot-force [Service] -Type=oneshot +# idle: start once the isolate's stop jobs are done, so their status lines +# do not interleave with the summary. +Type=idle +# How long the summary stays on screen before the reboot (seconds, default +# 60); the bluefin.failure_delay= kernel argument overrides it where the +# command line is not locked by Secure Boot. +ImportCredential=bluefin.failure_delay ExecStart= -ExecStart=-/usr/bin/bash -c 'echo "Boot failed; errors follow, rebooting in 10s." >/dev/console; journalctl -b -p err --no-pager -o short-monotonic >/dev/console 2>&1; sleep 10' +ExecStart=-/usr/libexec/bluefin-boot-diagnostics failure-summary diff --git a/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf new file mode 100644 index 00000000..a5a7b2df --- /dev/null +++ b/files/initrd/usr/lib/systemd/system/systemd-import@.service.d/10-bluefin-pull-check.conf @@ -0,0 +1,15 @@ +[Unit] +# The diskless initrd downloads the whole OS image into /run (RAM). Check +# first that it fits, so a small machine fails with "needs ~X MiB RAM" +# instead of an out-of-space error deep in the download. Wants=, not +# Requires=: the check stops the boot through its own OnFailure=, which it +# triggers only when it knows the image cannot fit; a check that is missing +# or broken must not hold up the download. +Wants=bluefin-pull-check@%i.service +After=bluefin-pull-check@%i.service + +# Without this a failed download only surfaces when the /usr devices time out +# (systemd.default_device_timeout_sec=300); go straight to the failure +# summary and reboot instead. +OnFailure=emergency.target +OnFailureJobMode=isolate diff --git a/files/initrd/usr/libexec/bluefin-boot-diagnostics b/files/initrd/usr/libexec/bluefin-boot-diagnostics new file mode 100755 index 00000000..59ff4382 --- /dev/null +++ b/files/initrd/usr/libexec/bluefin-boot-diagnostics @@ -0,0 +1,304 @@ +#!/usr/bin/bash +# bluefin-boot-diagnostics — say on the console why an initrd boot fails. +# +# check-pull UNIT Run before UNIT (a systemd-import@.service instance) +# downloads the OS image into RAM. HEADs the image URL and +# fails with a plain "needs ~X MiB RAM, have Y" message +# only when a 2xx answer gives a size that cannot fit. If +# it cannot check (no answer, an HTTP error, no +# Content-Length), it prints a note and exits 0 so the +# download still runs. +# ignition-kargs Warn that ignition.* kernel arguments are ignored. +# failure-summary ExecStart of emergency.service: print the failed units, +# their last log lines and the last errors, then wait +# (bluefin.failure_delay, default 60 s) so the reboot that +# follows does not scroll them away. +# +# Messages go to every active console (/sys/class/tty/console/active, e.g. the +# monitor and the serial port). /dev/console, where systemd prints its status +# lines, is only the last console= on the kernel command line. +# +# bash builtins, coreutils, curl, systemctl and journalctl only: the initrd +# has no sed or awk. + +set -uo pipefail +export LC_ALL=C + +PROG="${0##*/}" +DOCS=https://github.com/projectbluefin/server/blob/main/docs/skills/diskless-troubleshooting.md +MEMINFO="${BLUEFIN_DIAG_MEMINFO:-/proc/meminfo}" +CMDLINE="${BLUEFIN_DIAG_CMDLINE:-/proc/cmdline}" +CONSOLES="${BLUEFIN_DIAG_CONSOLES:-/sys/class/tty/console/active}" +DEVDIR="${BLUEFIN_DIAG_DEVDIR:-/dev}" +NETDIR="${BLUEFIN_DIAG_NETDIR:-/sys/class/net}" +# systemd-importd stores machine images under /run/machines in the initrd: +# the /run tmpfs, which systemd caps at 20% of RAM. +PULL_DIR=/run +# Room /run and RAM must keep after the image has landed. +MARGIN_MIB=64 +# check_pull's status when the image cannot fit. It differs from bash's own +# error statuses (1, 2, 126, 127, 128+n), so a bug in the check is never +# taken for "does not fit". +TOO_BIG=3 +DELAY_NAME=bluefin.failure_delay +DEFAULT_DELAY=60 +# The reboot is intentional (unattended retry, boot counting); a floor keeps +# a persistent failure from turning into a hot reboot loop. +MIN_DELAY=10 +MAX_DELAY=86400 +BAR='==============================================================================' + +# log PRIORITY MESSAGE: to the journal (stderr, sd-daemon priority prefix). +log() { printf '<%s>%s: %s\n' "$1" "${PROG}" "$2" >&2; } + +# Copy stdin to every active console. A console that blocks must not hold up +# the boot, so each write gets a timeout. +to_consoles() { + local text c active=console + text="$(cat)" + [ -r "${CONSOLES}" ] && active="$(<"${CONSOLES}")" + for c in ${active}; do + timeout 5 tee -a "${DEVDIR}/${c}" >/dev/null <<<"${text}" || true + done +} + +# shout PRIORITY LINE...: log every line and print them framed on the consoles. +shout() { + local pri="$1" line + shift + for line in "$@"; do log "${pri}" "${line}"; done + { printf '\n%s\n' "${BAR}"; printf '%s\n' "$@"; printf '%s\n' "${BAR}"; } | to_consoles +} + +mib() { printf '%s' "$(( ($1 + 1048575) / 1048576 ))"; } + +meminfo_kib() { + local key value _ + while read -r key value _; do + [ "${key}" = "$1:" ] && { printf '%s' "${value}"; return 0; } + done < "${MEMINFO}" + return 1 +} + +# One line per network link: name, operstate, carrier. sysfs only, so it also +# works after emergency.target has stopped systemd-networkd. +links() { + local dev name state carrier out=() + for dev in "${NETDIR}"/*; do + name="${dev##*/}" + [ -e "${dev}/device" ] || continue + state="$(<"${dev}/operstate")" 2>/dev/null || state=unknown + carrier="$(<"${dev}/carrier")" 2>/dev/null || carrier=0 + [ "${carrier}" = 1 ] && carrier=carrier || carrier="no carrier" + out+=("${name} ${state} (${carrier})") + done + if [ "${#out[@]}" -eq 0 ]; then + printf 'no network interface found (missing NIC driver in the initrd?)\n' + else + printf '%s\n' "${out[@]}" + fi +} + +# cannot_check REASON: note on the consoles that the size was not checked. The +# download goes ahead: it may still work (a signed object-store URL can refuse +# HEAD and serve GET), and if it fails the failure summary says why. +cannot_check() { + local msg="BLUEFIN: cannot check that the OS image fits in RAM ($1); downloading it anyway" + log 4 "${msg}" + printf '%s\n' "${msg}" | to_consoles +} + +check_pull() { + local unit="${1:?usage: ${PROG} check-pull }" + local desc url out rc status='' len='' line lower + # systemd-import-generator describes each pull unit as "Download of "; + # take the URL it resolved (rd.systemd.pull, bootorigin, or the import.pull + # credential) rather than deriving it a second time. + desc="$(systemctl show -P Description "${unit}" 2>/dev/null)" || desc= + url="${desc#Download of }" + case "${url}" in + http://*|https://*) ;; + *) log 6 "${unit}: no http(s) source (${desc:-no description}); nothing to check"; return 0 ;; + esac + + # Only the size is wanted here; whether the image downloads is for + # systemd-import to find out. + out="$(curl -sSIL --retry 3 --retry-delay 2 --connect-timeout 10 --max-time 120 "${url}" 2>&1)" + rc=$? + if [ "${rc}" -ne 0 ]; then + cannot_check "HEAD failed: ${out##*$'\n'}" + return 0 + fi + + # Headers of every response in the redirect chain; keep the last one. + while IFS= read -r line; do + line="${line%$'\r'}" + lower="${line,,}" + case "${lower}" in + http/*) status="${line#* }"; status="${status%% *}"; len= ;; + content-length:*) len="${line#*:}"; len="${len//[[:space:]]/}" ;; + esac + done <<<"${out}" + + case "${status}" in + 2??) ;; + *) cannot_check "HEAD answered HTTP ${status:-?}"; return 0 ;; + esac + if ! [[ "${len}" =~ ^[0-9]+$ ]]; then + cannot_check "HEAD answered HTTP ${status} with no Content-Length" + return 0 + fi + + local total_kib avail_kib bavail bsize btotal + total_kib="$(meminfo_kib MemTotal)" || { cannot_check "no MemTotal in ${MEMINFO}"; return 0; } + avail_kib="$(meminfo_kib MemAvailable)" || avail_kib="${total_kib}" + read -r bavail bsize btotal < <(stat -f -c '%a %S %b' "${PULL_DIR}") + if ! [[ "${bavail:-}${bsize:-}${btotal:-}" =~ ^[0-9]+$ ]] || [ "${btotal}" -eq 0 ] || [ "${total_kib}" -eq 0 ]; then + cannot_check "cannot read the size of ${PULL_DIR} or of RAM" + return 0 + fi + + local need=$(( len + MARGIN_MIB * 1048576 )) + local run_free=$(( bavail * bsize )) run_size=$(( btotal * bsize )) + local total=$(( total_kib * 1024 )) avail=$(( avail_kib * 1024 )) + if [ "${need}" -le "${run_free}" ] && [ "${need}" -le "${avail}" ]; then + log 6 "OS image $(mib "${len}") MiB fits: /run has $(mib "${run_free}") MiB free, MemAvailable $(mib "${avail}") MiB" + return 0 + fi + + # /run is sized as a fixed share of RAM, so scale what it lacks up to RAM. + # Work in KiB so the product cannot overflow. + local used_kib=$(( (run_size - run_free) / 1024 )) need_kib=$(( need / 1024 )) + local want_run_kib=$(( (used_kib + need_kib) * total_kib / (run_size / 1024) )) + local want_run=$(( want_run_kib * 1024 )) + local want_mem=$(( total - avail + need )) + local want=$(( want_run > want_mem ? want_run : want_mem )) + # Round up to 64 MiB. + want=$(( (want + 67108863) / 67108864 * 67108864 )) + shout 2 "BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT" \ + "This machine needs ~$(mib "${want}") MiB RAM, have $(mib "${total}") MiB." \ + "The OS image ${url##*/} ($(mib "${len}") MiB) is downloaded into /run, a RAM" \ + "file system capped at $(( run_size * 100 / total ))% of RAM: $(mib "${run_free}") MiB free there, MemAvailable $(mib "${avail}") MiB." \ + "Add RAM or install to disk. ${DOCS}#minimum-ram" + return "${TOO_BIG}" +} + +# Keys of ignition.* kernel arguments (values may carry secrets). +ignition_karg_keys() { + local -a args + local arg keys=() + read -ra args < "${CMDLINE}" || true + for arg in "${args[@]}"; do + case "${arg}" in + ignition.*) keys+=("${arg%%=*}") ;; + esac + done + [ "${#keys[@]}" -gt 0 ] && printf '%s\n' "${keys[*]}" +} + +ignition_kargs() { + local keys + keys="$(ignition_karg_keys)" || return 0 + shout 4 "BLUEFIN: IGNORED KERNEL ARGUMENTS: ${keys}" \ + "Bluefin Server never reads Ignition settings from the kernel command line." \ + "Pass the config as the ignition.config or ignition.config.url system credential" \ + "(SMBIOS type 11, QEMU fw_cfg, or /loader/credentials/ on the ESP), or serve" \ + "bluefin-node.ign next to the netboot UKI. ${DOCS}#ignition" +} + +# What a failed unit most likely means. +hint() { + local stage + case "$1" in + bluefin-pull-check@*) + echo "the OS image is too big for the RAM of this machine (see the message above)" ;; + systemd-import@*) + echo "downloading or verifying the OS image failed: server reachable? SHA256SUMS signed by a key this image trusts? room in /run?" ;; + systemd-loop@*) + echo "attaching the OS image as a block device failed (also fails when the download did not happen)" ;; + systemd-networkd-wait-online.service) + echo "no network link became routable within 300 s: check the link and that a DHCP server answers" ;; + systemd-veritysetup@*|sysusr-usr.mount|sysroot-usr.mount) + echo "/usr could not be verified or mounted: the OS image does not match the usrhash= of this UKI" ;; + bluefin-ignition-credentials.service) + echo "staging the Ignition config failed (invalid ignition.config.url?). ${DOCS}#ignition" ;; + ignition-*.service) + stage="${1#ignition-}" + stage="${stage%.service}" + echo "Ignition stage '${stage}' failed. Every diskless boot runs every stage, so the config must be idempotent. ${DOCS}#ignition" ;; + esac +} + +failure_delay() { + local value='' arg + local -a args + if [ -r "${CREDENTIALS_DIRECTORY:-/nonexistent}/${DELAY_NAME}" ]; then + value="$(<"${CREDENTIALS_DIRECTORY}/${DELAY_NAME}")" + fi + read -ra args < "${CMDLINE}" || true + for arg in "${args[@]}"; do + case "${arg}" in "${DELAY_NAME}="*) value="${arg#*=}" ;; esac + done + value="${value//[[:space:]]/}" + [[ "${value}" =~ ^[0-9]{1,6}$ ]] || value="${DEFAULT_DELAY}" + value=$(( 10#${value} )) + [ "${value}" -lt "${MIN_DELAY}" ] && value="${MIN_DELAY}" + [ "${value}" -gt "${MAX_DELAY}" ] && value="${MAX_DELAY}" + printf '%s' "${value}" +} + +indent() { local l; while IFS= read -r l; do printf '%s%s\n' "$1" "${l}"; done; } + +failure_summary() { + local delay unit h keys shown=0 + local -a units matches + delay="$(failure_delay)" + mapfile -t units < <(systemctl list-units --state=failed --no-legend --plain --full 2>/dev/null | cut -d' ' -f1) + { + printf '\n%s\n' "${BAR}" + printf 'BLUEFIN: BOOT FAILED\n' + if [ "${#units[@]}" -eq 0 ]; then + printf 'No unit failed: a device or job timed out (see the errors below).\n' + fi + for unit in "${units[@]}"; do + [ -n "${unit}" ] || continue + shown=$(( shown + 1 )) + [ "${shown}" -le 6 ] || { printf '(%s more failed units)\n' "$(( ${#units[@]} - 6 ))"; break; } + printf '\nFAILED: %s\n' "${unit}" + h="$(hint "${unit}")" + [ -z "${h}" ] || printf ' -> %s\n' "${h}" + # What the unit's own processes logged (not systemd's lines about it); + # a download's errors come from systemd-importd. + matches=("_SYSTEMD_UNIT=${unit}") + case "${unit}" in systemd-import@*) matches+=(_SYSTEMD_UNIT=systemd-importd.service) ;; esac + journalctl -b "${matches[@]}" -n 6 -o cat --no-pager 2>/dev/null | indent ' | ' + case "${unit}" in + systemd-networkd-wait-online.service|systemd-import@*) + printf ' network links:\n'; links | indent ' ' ;; + esac + done + if keys="$(ignition_karg_keys)"; then + printf '\nIgnored kernel arguments: %s (use the ignition.config credential instead)\n' "${keys}" + fi + printf '\nLast errors (journalctl -b -p err):\n' + journalctl -b -p err -n 15 -o short-monotonic --no-pager 2>/dev/null | indent ' ' + printf '\nHelp: %s\n' "${DOCS}" + printf 'Rebooting in %s s (set the %s credential to change the delay).\n' "${delay}" "${DELAY_NAME}" + printf '%s\n' "${BAR}" + } | to_consoles + sleep "${delay}" +} + +case "${1:-}" in + check-pull) + shift + # Only "the image cannot fit" fails the unit, and so stops the boot. The + # check runs in a subshell so that a bug in it (an unset variable, a value + # it did not expect) lets the download go ahead instead. + rc=0 + ( check_pull "$@" ) || rc=$? + [ "${rc}" -ne "${TOO_BIG}" ] || exit 1 ;; + ignition-kargs) ignition_kargs ;; + failure-summary) failure_summary ;; + *) printf 'usage: %s check-pull UNIT | ignition-kargs | failure-summary\n' "${PROG}" >&2; exit 2 ;; +esac diff --git a/tests/unit/test_initrd_diagnostics.py b/tests/unit/test_initrd_diagnostics.py new file mode 100644 index 00000000..3d9abafe --- /dev/null +++ b/tests/unit/test_initrd_diagnostics.py @@ -0,0 +1,449 @@ +"""Initrd boot-failure diagnostics: units, drop-ins and the libexec helper. + +files/initrd/usr/libexec/bluefin-boot-diagnostics runs in the initrd only, so +these tests drive it under bash on the host with systemctl, journalctl, stat +and sleep stubbed on $PATH, a fake /proc/meminfo, fake consoles, and a local +http.server answering the HEAD request the RAM check makes. +""" + +from __future__ import annotations + +import http.server +import os +import re +import shutil +import socket +import subprocess +import threading +from pathlib import Path + +import pytest + +ROOT = Path(__file__).resolve().parents[2] +UNITS = ROOT / "files" / "initrd" / "usr" / "lib" / "systemd" / "system" +IGN_UNITS = ROOT / "files" / "initrd-ignition" / "usr" / "lib" / "systemd" / "system" +HELPER = ROOT / "files" / "initrd" / "usr" / "libexec" / "bluefin-boot-diagnostics" +BOOT = ROOT / "elements" / "oci" / "bluefin-server-boot.bst" +MIB = 1024 * 1024 +IMPORT_UNIT = "systemd-import@run-machines-rootdisk.raw.service" + + +def unit(path: Path) -> dict[str, dict[str, list[str]]]: + """Parse a unit file keeping every value of repeated keys.""" + sections: dict[str, dict[str, list[str]]] = {} + current = None + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith(("#", ";")): + continue + if line.startswith("["): + current = sections.setdefault(line.strip("[]"), {}) + continue + key, _, value = line.partition("=") + current.setdefault(key, []).append(value) + return sections + + +def test_emergency_prints_the_summary_then_reboots() -> None: + u = unit(UNITS / "emergency.service.d" / "10-reboot.conf") + assert u["Unit"]["SuccessAction"] == ["reboot-force"] + assert u["Unit"]["FailureAction"] == ["reboot-force"] + # Reset the stock sulogin ExecStart, then run the summary (never fatal). + assert u["Service"]["ExecStart"] == ["", "-/usr/libexec/bluefin-boot-diagnostics failure-summary"] + assert u["Service"]["ImportCredential"] == ["bluefin.failure_delay"] + assert u["Service"]["Type"] == ["idle"] + + +def test_every_download_is_preceded_by_the_ram_check() -> None: + u = unit(UNITS / "systemd-import@.service.d" / "10-bluefin-pull-check.conf") + # Only the check's own OnFailure= may stop the boot; its failure never fails the download. + assert u["Unit"]["Wants"] == ["bluefin-pull-check@%i.service"] + assert "Requires" not in u["Unit"] + assert u["Unit"]["After"] == ["bluefin-pull-check@%i.service"] + # A failed download goes straight to the summary, not a 300 s device timeout. + assert u["Unit"]["OnFailure"] == ["emergency.target"] + assert u["Unit"]["OnFailureJobMode"] == ["isolate"] + + +def test_ram_check_unit_checks_its_import_instance() -> None: + u = unit(UNITS / "bluefin-pull-check@.service") + assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"] + assert u["Unit"]["DefaultDependencies"] == ["no"] + assert "network-online.target" in u["Unit"]["After"] + assert u["Unit"]["OnFailure"] == ["emergency.target"] + assert u["Unit"]["OnFailureJobMode"] == ["isolate"] + assert u["Service"]["ExecStart"] == [ + "/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service" + ] + + +def test_ignition_karg_warning_runs_in_every_initrd() -> None: + u = unit(UNITS / "bluefin-ignition-kargs.service") + assert u["Unit"]["ConditionPathExists"] == ["/etc/initrd-release"] + assert "bluefin-ignition-credentials.service" in " ".join(u["Unit"]["Before"]) + assert u["Service"]["ExecStart"] == ["/usr/libexec/bluefin-boot-diagnostics ignition-kargs"] + boot = BOOT.read_text(encoding="utf-8") + assert 'ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service"' in boot + assert "libexec/bluefin-boot-diagnostics" in boot, "the UKI build checks the helper is in the initrd" + + +@pytest.mark.parametrize("stage", ["fetch-offline", "fetch", "disks", "mount", "files"]) +def test_ignition_failures_reach_the_summary(stage: str) -> None: + u = unit(IGN_UNITS / f"ignition-{stage}.service") + assert u["Unit"]["OnFailure"] == ["emergency.target"] + + +def test_helper_is_executable_bash_without_sed_or_awk() -> None: + text = HELPER.read_text(encoding="utf-8") + assert text.startswith("#!/usr/bin/bash\n") + assert os.access(HELPER, os.X_OK) + subprocess.run(["bash", "-n", str(HELPER)], check=True) + code = "\n".join(line for line in text.splitlines() if not line.lstrip().startswith("#")) + assert not re.search(r"\b(sed|awk|gawk)\b", code), "the initrd has no sed or awk" + + +@pytest.mark.skipif(shutil.which("shellcheck") is None, reason="shellcheck not installed") +def test_helper_is_shellcheck_clean() -> None: + subprocess.run(["shellcheck", "-S", "style", str(HELPER)], check=True) + + +class _Handler(http.server.BaseHTTPRequestHandler): + routes: dict[str, tuple[int, dict[str, str]]] = {} + + def do_HEAD(self) -> None: # noqa: N802 (http.server API) + status, headers = self.routes.get(self.path, (404, {"Content-Length": "0"})) + self.send_response(status) + for key, value in headers.items(): + self.send_header(key, value) + self.end_headers() + + def log_message(self, *args) -> None: + pass + + +@pytest.fixture(scope="module") +def server(): + httpd = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + yield f"http://127.0.0.1:{httpd.server_address[1]}", _Handler.routes + httpd.shutdown() + + +STUBS = { + "systemctl": """#!/bin/bash +case "$1" in + show) printf '%s\\n' "${FAKE_DESCRIPTION}" ;; + list-units) for u in ${FAKE_FAILED:-}; do printf '%s loaded failed failed Fake unit\\n' "$u"; done ;; +esac +""", + "journalctl": """#!/bin/bash +printf 'journal-line args: %s\\n' "$*" +""", + "stat": """#!/bin/bash +printf '%s\\n' "${FAKE_STATFS}" +""", + "sleep": """#!/bin/bash +printf '%s\\n' "$1" >> "${FAKE_SLEEP_LOG}" +""", +} + + +class Env: + def __init__(self, tmp: Path) -> None: + self.tmp = tmp + self.bin = tmp / "bin" + self.bin.mkdir() + for name, body in STUBS.items(): + (self.bin / name).write_text(body, encoding="utf-8") + (self.bin / name).chmod(0o755) + self.dev = tmp / "dev" + self.dev.mkdir() + (tmp / "consoles").write_text("tty0 ttyS0\n", encoding="utf-8") + self.net = tmp / "net" + nic = self.net / "enp1s0" + (nic / "device").mkdir(parents=True) + (nic / "operstate").write_text("down\n", encoding="utf-8") + (nic / "carrier").write_text("0\n", encoding="utf-8") + (self.net / "lo").mkdir() # virtual: no device link, not listed + self.cmdline("root=tmpfs") + self.meminfo(total_kib=4_000_000, avail_kib=3_600_000) + self.statfs(free_mib=800, size_mib=800) + self.creds = tmp / "creds" + self.creds.mkdir() + self.env = dict( + os.environ, + PATH=f"{self.bin}:{os.environ['PATH']}", + BLUEFIN_DIAG_MEMINFO=str(tmp / "meminfo"), + BLUEFIN_DIAG_CMDLINE=str(tmp / "cmdline"), + BLUEFIN_DIAG_CONSOLES=str(tmp / "consoles"), + BLUEFIN_DIAG_DEVDIR=str(self.dev), + BLUEFIN_DIAG_NETDIR=str(self.net), + CREDENTIALS_DIRECTORY=str(self.creds), + FAKE_SLEEP_LOG=str(tmp / "sleep.log"), + FAKE_DESCRIPTION="", + FAKE_FAILED="", + ) + + def cmdline(self, text: str) -> None: + (self.tmp / "cmdline").write_text(text + "\n", encoding="utf-8") + + def meminfo(self, total_kib: int, avail_kib: int) -> None: + self.total_kib = total_kib + self.avail_kib = avail_kib + (self.tmp / "meminfo").write_text( + f"MemTotal: {total_kib} kB\nMemFree: {avail_kib} kB\n" + f"MemAvailable: {avail_kib} kB\nBuffers: 0 kB\n", + encoding="utf-8", + ) + + def statfs(self, free_mib: int, size_mib: int) -> None: + self.run_free = free_mib * MIB + self.run_size = size_mib * MIB + self.fake_statfs = f"{free_mib * 256} 4096 {size_mib * 256}" + + def run(self, *args: str, **extra: str) -> subprocess.CompletedProcess: + env = dict(self.env, FAKE_STATFS=self.fake_statfs, **extra) + return subprocess.run( + ["bash", str(HELPER), *args], capture_output=True, text=True, env=env, timeout=60 + ) + + def console(self, name: str = "ttyS0") -> str: + path = self.dev / name + return path.read_text(encoding="utf-8") if path.exists() else "" + + def sleeps(self) -> list[str]: + log = self.tmp / "sleep.log" + return log.read_text(encoding="utf-8").split() if log.exists() else [] + + +@pytest.fixture +def env(tmp_path: Path) -> Env: + return Env(tmp_path) + + +def check_pull(env: Env, url: str, **extra: str) -> subprocess.CompletedProcess: + return env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION=f"Download of {url}", **extra) + + +def cannot_check(reason: str) -> str: + return f"BLUEFIN: cannot check that the OS image fits in RAM ({reason}); downloading it anyway" + + +def assert_download_goes_ahead(env: Env, result: subprocess.CompletedProcess, reason: str) -> None: + note = cannot_check(reason) + assert result.returncode == 0, result.stderr + for tty in ("tty0", "ttyS0"): + assert env.console(tty) == f"{note}\n" + assert f"<4>bluefin-boot-diagnostics: {note}\n" in result.stderr + + +def test_image_that_fits_passes_quietly(env: Env, server) -> None: + base, routes = server + routes["/fits.raw"] = (200, {"Content-Length": str(540 * MIB)}) + result = check_pull(env, f"{base}/fits.raw") + assert result.returncode == 0, result.stderr + assert "540 MiB fits" in result.stderr + assert env.console() == "" + + +def _expected_mib(env: Env, size: int) -> int: + """Mirror of the helper's estimate, rounded up to 64 MiB.""" + need = size + 64 * MIB + used_kib = (env.run_size - env.run_free) // 1024 + want_run = (used_kib + need // 1024) * env.total_kib // (env.run_size // 1024) * 1024 + want_mem = env.total_kib * 1024 - env.avail_kib * 1024 + need + want = max(want_run, want_mem) + return -(-want // (64 * MIB)) * 64 + + +def test_image_larger_than_run_fails_with_the_ram_it_needs(env: Env, server) -> None: + base, routes = server + size = 540 * MIB + routes["/bluefin-server_1.raw"] = (200, {"Content-Length": str(size)}) + # 2 GB machine: /run is 20% of it. + env.meminfo(total_kib=2_000_000, avail_kib=1_800_000) + env.statfs(free_mib=380, size_mib=390) + result = check_pull(env, f"{base}/bluefin-server_1.raw") + assert result.returncode == 1 + want = _expected_mib(env, size) + assert 2900 < want < 3200 + for tty in ("tty0", "ttyS0"): + text = env.console(tty) + assert "NOT ENOUGH RAM FOR A DISKLESS BOOT" in text + assert f"needs ~{want} MiB RAM, have 1954 MiB" in text + assert "bluefin-server_1.raw (540 MiB)" in text + assert "capped at 19% of RAM: 380 MiB free" in text + assert f"<2>bluefin-boot-diagnostics: This machine needs ~{want} MiB RAM" in result.stderr + + +def test_low_memavailable_fails_even_when_run_has_room(env: Env, server) -> None: + base, routes = server + size = 300 * MIB + routes["/small.raw"] = (200, {"Content-Length": str(size)}) + env.meminfo(total_kib=4_000_000, avail_kib=200_000) + env.statfs(free_mib=780, size_mib=800) + result = check_pull(env, f"{base}/small.raw") + assert result.returncode == 1 + assert f"needs ~{_expected_mib(env, size)} MiB RAM, have 3907 MiB" in env.console() + + +def test_redirects_are_followed_to_the_final_size(env: Env, server) -> None: + base, routes = server + routes["/redirect.raw"] = (302, {"Location": "/huge.raw", "Content-Length": "0"}) + routes["/huge.raw"] = (200, {"Content-Length": str(4000 * MIB)}) + result = check_pull(env, f"{base}/redirect.raw") + assert result.returncode == 1 + assert "redirect.raw (4000 MiB)" in env.console() + + +@pytest.mark.parametrize("status", [403, 404, 405, 500, 501]) +def test_head_errors_do_not_block_the_download(env: Env, server, status: int) -> None: + # A signed object-store URL covers the method: GET works, HEAD gets 403. + base, routes = server + routes[f"/head{status}.raw"] = (status, {"Content-Length": "0"}) + result = check_pull(env, f"{base}/head{status}.raw") + assert_download_goes_ahead(env, result, f"HEAD answered HTTP {status}") + + +def test_missing_content_length_skips_the_check(env: Env, server) -> None: + base, routes = server + routes["/chunked.raw"] = (200, {"Transfer-Encoding": "chunked"}) + result = check_pull(env, f"{base}/chunked.raw") + assert_download_goes_ahead(env, result, "HEAD answered HTTP 200 with no Content-Length") + + +def test_unreachable_server_does_not_block_the_download(env: Env) -> None: + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + port = s.getsockname()[1] + result = check_pull(env, f"http://127.0.0.1:{port}/img.raw") + assert result.returncode == 0, result.stderr + text = env.console() + assert text.startswith("BLUEFIN: cannot check that the OS image fits in RAM (HEAD failed: curl: (7) ") + assert text.endswith("); downloading it anyway\n") + assert text.count("\n") == 1 + assert env.console("tty0") == text + assert f"<4>bluefin-boot-diagnostics: {text}" in result.stderr + + +def test_a_broken_check_does_not_block_the_download(env: Env, server) -> None: + base, routes = server + routes["/broken.raw"] = (200, {"Content-Length": str(540 * MIB)}) + # A value the check does not expect makes bash abort it (unbound variable). + (env.tmp / "meminfo").write_text("MemTotal: 4000000 kB\nMemAvailable: n/a kB\n", encoding="utf-8") + result = check_pull(env, f"{base}/broken.raw") + assert "unbound variable" in result.stderr + assert result.returncode == 0 + assert env.console() == "" + + +def test_non_http_sources_are_not_checked(env: Env) -> None: + result = env.run("check-pull", IMPORT_UNIT, FAKE_DESCRIPTION="Download of file:///run/img.raw") + assert result.returncode == 0 + assert env.console() == "" + + +def test_ignition_kargs_are_named_without_their_values(env: Env) -> None: + env.cmdline("root=tmpfs ignition.config.url=http://10.0.0.1/secret-token.ign ignition.platform.id=metal quiet") + result = env.run("ignition-kargs") + assert result.returncode == 0 + text = env.console("tty0") + assert "IGNORED KERNEL ARGUMENTS: ignition.config.url ignition.platform.id" in text + assert "ignition.config or ignition.config.url system credential" in text + assert "secret-token" not in text + result.stderr + assert "<4>bluefin-boot-diagnostics: BLUEFIN: IGNORED KERNEL ARGUMENTS" in result.stderr + + +def test_no_ignition_kargs_no_warning(env: Env) -> None: + env.cmdline("root=tmpfs rd.systemd.pull=raw,machine:rootdisk:http://x/y.raw") + result = env.run("ignition-kargs") + assert result.returncode == 0 + assert result.stderr == "" + assert env.console() == "" + + +def test_failure_summary_explains_each_failed_unit(env: Env) -> None: + result = env.run( + "failure-summary", + FAKE_FAILED=f"ignition-disks.service {IMPORT_UNIT} systemd-networkd-wait-online.service", + ) + assert result.returncode == 0 + text = env.console("tty0") + assert text == env.console("ttyS0") + assert "BLUEFIN: BOOT FAILED" in text + assert "FAILED: ignition-disks.service" in text + assert "Ignition stage 'disks' failed" in text + assert "must be idempotent" in text + assert f"FAILED: {IMPORT_UNIT}" in text + assert "SHA256SUMS signed by a key this image trusts" in text + assert "no network link became routable within 300 s" in text + assert "enp1s0 down (no carrier)" in text + assert ( + f"| journal-line args: -b _SYSTEMD_UNIT={IMPORT_UNIT} _SYSTEMD_UNIT=systemd-importd.service -n 6 -o cat --no-pager" + in text + ), "a failed download shows systemd-importd's reason" + assert "| journal-line args: -b _SYSTEMD_UNIT=ignition-disks.service -n 6" in text + assert "Last errors (journalctl -b -p err):" in text + assert "docs/skills/diskless-troubleshooting.md\n" in text + assert "Rebooting in 60 s" in text + assert env.sleeps() == ["60"] + + +def test_failure_summary_blames_ram_for_a_failed_check(env: Env) -> None: + env.run("failure-summary", FAKE_FAILED="bluefin-pull-check@run-machines-rootdisk.raw.service") + text = env.console() + assert "-> the OS image is too big for the RAM of this machine" in text + assert "network links:" not in text + + +def test_failure_summary_without_failed_units_points_at_timeouts(env: Env) -> None: + env.cmdline("root=tmpfs ignition.config.url=http://x/y.ign") + env.run("failure-summary") + text = env.console() + assert "No unit failed: a device or job timed out" in text + assert "Ignored kernel arguments: ignition.config.url" in text + + +@pytest.mark.parametrize( + "credential,karg,expected", + [ + (None, None, "60"), + ("120\n", None, "120"), + ("3", None, "10"), # floor: no hot reboot loop + ("0600", None, "600"), # decimal, not octal + ("999999999", None, "60"), # not a sane number + ("soon", None, "60"), + ("120", "30", "30"), # the kernel argument wins where it can be set + (None, "100000", "86400"), + ], +) +def test_failure_delay(env: Env, credential: str | None, karg: str | None, expected: str) -> None: + if credential is not None: + (env.creds / "bluefin.failure_delay").write_text(credential, encoding="utf-8") + if karg is not None: + env.cmdline(f"root=tmpfs bluefin.failure_delay={karg}") + env.run("failure-summary") + assert env.sleeps() == [expected] + assert f"Rebooting in {expected} s" in env.console() + + +def test_unknown_mode_is_a_usage_error(env: Env) -> None: + result = env.run("bogus") + assert result.returncode == 2 + assert "usage:" in result.stderr + + +def test_console_links_point_at_existing_doc_sections() -> None: + text = HELPER.read_text(encoding="utf-8") + base = re.search(r"^DOCS=https://github\.com/projectbluefin/server/blob/main/(\S+)$", text, re.M) + assert base, "DOCS must link the repository's main branch" + doc = (ROOT / base.group(1)).read_text(encoding="utf-8") + slugs = { + re.sub(r"[^a-z0-9 -]", "", line.lstrip("#").strip().lower()).replace(" ", "-") + for line in doc.splitlines() + if line.startswith("## ") + } + anchors = set(re.findall(r"\$\{DOCS\}#([a-z0-9-]+)", text)) + assert anchors == {"minimum-ram", "ignition"} + assert anchors <= slugs