Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
| Task | Skill |
|---|---|
| Boot / install / update architecture and local build + dogfood | [`docs/skills/ddi-installer.md`](docs/skills/ddi-installer.md), [`docs/skills/ddi-installer-build.md`](docs/skills/ddi-installer-build.md) |
| Diskless boot failures, RAM sizing, node logs, Ignition configs | [`docs/skills/diskless-troubleshooting.md`](docs/skills/diskless-troubleshooting.md) |
| Factory role, k0s sysext rationale, lab integration | [`docs/skills/factory-integration.md`](docs/skills/factory-integration.md) |
| Work with `systemd-sysext` / `systemd-confext` | [`docs/skills/systemd-sysext-extensions.md`](docs/skills/systemd-sysext-extensions.md) |
| Build or run the kubeadm worker sysext | [`docs/skills/kubeadm-sysext.md`](docs/skills/kubeadm-sysext.md) |
Expand Down
2 changes: 2 additions & 0 deletions docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ Useful environment variables:
- `DOGFOOD_SERVE_EXTRA=<dir>` — also serve the files in `<dir>`.
- `DOGFOOD_TAMPER=raw|sums` — serve a corrupted DDI or a re-hashed, unsigned
`SHA256SUMS`; the boot must fail, proving the signature check.
- `DOGFOOD_MEM=<MiB>` — guest RAM (default 4096); below the diskless minimum
the boot must fail with the RAM message (see `diskless-troubleshooting.md`).
- `DOGFOOD_EXTRA_PROBE=<file>` — shell snippet appended to the in-guest probe.

`scripts/dogfood-install.sh <dir> [<next-dir> [<broken-dir>]]` is the full
Expand Down
18 changes: 12 additions & 6 deletions docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Use when building or debugging the Bluefin Server boot chain, the d
metadata:
type: reference
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-28"
context7-sources:
- /systemd/systemd
- /apache/buildstream
Expand Down Expand Up @@ -87,9 +87,11 @@ ships gnupg and the image keyring, and importd fetches `SHA256SUMS` and
using the manifest. The manifest hash check pins the DDI, the signed UKI pins
`usrhash=`, and dm-verity checks every `/usr` block against it; a tampered DDI
or a re-hashed unsigned manifest is refused. A failed boot never drops to an
emergency shell: the initrd prints the errors and reboots
(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`),
which is what lets boot counting work unattended.
emergency shell: the initrd prints a failure summary on every console and
reboots (`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`),
which is what lets boot counting work unattended. Before the pull, a check
refuses an image that cannot fit in RAM. What the node prints, minimum RAM,
failure modes and logs: [diskless-troubleshooting.md](diskless-troubleshooting.md).

### UEFI HTTP boot

Expand Down Expand Up @@ -222,7 +224,10 @@ EFI variable and, when the server offers `bluefin-node.ign` next to the UKI,
applies it. Ignition runs on **every** boot (there is no first-boot marker on
a tmpfs root), so configs must be idempotent. See
`tests/fixtures/ignition/var-on-disk.ign` for a dogfood-tested example
(persistent /var on a second disk plus an SSH key).
(persistent /var on a second disk plus an SSH key). The supported stages and
config sections, the idempotency rules, and the warning for ignored
`ignition.*` kernel arguments are in
the Ignition section of [diskless-troubleshooting.md](diskless-troubleshooting.md).

## PXE / HTTP boot service

Expand All @@ -249,7 +254,7 @@ specifying the URL to pull `bluefin-server_<ver>.raw` and its signed SHA256SUMS.
| "Kernel image is at `/boot/vmlinuz`." | FSDK installs kernels into `/usr/lib/modules/<kver>/vmlinuz`; `bluefin-server-boot.bst` picks it up from there for ukify. |
| "The initrd needs dracut." | The initrd is a hand-assembled systemd userspace (`initrd-stack.bst`) packed as newc cpio + zstd. No dracut anywhere in the tree. |
| "The diskless pull is unverified." | The initrd pulls with `verify=signature` against the keyring it ships; the signed UKI also pins `usrhash=`, and dm-verity checks every `/usr` block read. |
| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot. |
| "Ignition needs a karg." | The cmdline is sealed in the signed UKI. Ignition configs arrive as `ignition.config` / `ignition.config.url` system credentials, or as `bluefin-node.ign` next to the UKI on a UEFI HTTP boot; `ignition.*` kargs are ignored with a console warning. |
| "Diskless nodes need sysupdate." | Diskless nodes update by rebooting into a newer image; sysupdate is disabled when booted diskless. |

## Verification
Expand All @@ -264,6 +269,7 @@ specifying the URL to pull `bluefin-server_<ver>.raw` and its signed SHA256SUMS.
## See also

- [ddi-installer-build.md](ddi-installer-build.md) — local build, export, and dogfood workflow.
- [diskless-troubleshooting.md](diskless-troubleshooting.md) — failure summary, minimum RAM, logs, supported Ignition subset.
- [systemd-sysupdate-verification.md](systemd-sysupdate-verification.md) — release signing and transfer verification.
- [CONTEXT.md](../../CONTEXT.md) — canonical project domain glossary (OS DDI, Netboot UKI, Disk UKI, Slot).
- `systemd-sysinstall(8)`, `systemd-repart(8)`, `systemd-sysupdate(8)`, `bootctl(1)`, `ukify(1)`
181 changes: 181 additions & 0 deletions docs/skills/diskless-troubleshooting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
---
name: diskless-troubleshooting
description: Use when a diskless (netboot UKI) node fails to boot, reboot-loops, needs more RAM, or needs its logs kept, and when writing an Ignition config for Bluefin Server; covers the initrd failure summary, minimum RAM, network and download failure modes, getting logs, and the supported Ignition subset.
metadata:
type: how-to
status: stable
last_updated: "2026-09-29"
context7-sources:
- /systemd/systemd
---
# Diskless Boot Troubleshooting

## When to Use

- A diskless node reboots in a loop, or never reaches a login prompt.
- Sizing RAM for diskless nodes.
- Keeping logs from diskless boots.
- Writing or debugging an Ignition config (diskless or installed).

## When NOT to Use

- The boot chain, pull and install architecture itself (see
[ddi-installer.md](ddi-installer.md)).
- Building and dogfooding images (see
[ddi-installer-build.md](ddi-installer-build.md)).

## What a failing node prints

A failed initrd never drops to a shell (root is locked): it isolates
`emergency.target`, whose `emergency.service` drop-in
(`files/initrd/usr/lib/systemd/system/emergency.service.d/10-reboot.conf`) runs
`/usr/libexec/bluefin-boot-diagnostics failure-summary` and then reboots. The
summary lists each failed unit with a one-line hint and its last journal lines,
the network links when the network or the download failed, any ignored
`ignition.*` kernel arguments, and the last `journalctl -b -p err` lines. It is
written to every active console (`/sys/class/tty/console/active`: the monitor
and `ttyS0`), not only to `/dev/console`, which is just the last `console=`.

The summary stays up for 60 s, then the node reboots and retries. Retrying is
deliberate: a diskless node recovers unattended once the server, network or
config is fixed, and on installed nodes boot counting falls back to the
previous image. Change the delay (10 to 86400 s) with the
`bluefin.failure_delay` system credential (SMBIOS type 11, QEMU fw_cfg, or
`/loader/credentials/bluefin.failure_delay.cred` on the ESP); the
`bluefin.failure_delay=` kernel argument overrides it only where the command
line is not locked by Secure Boot. The 10 s floor keeps a persistent failure
from becoming a hot reboot loop.

Before the download, `bluefin-pull-check@.service` (wanted by
`systemd-import@.service.d/10-bluefin-pull-check.conf` and ordered before the
download) sends an HTTP HEAD to the image URL systemd-import-generator
resolved. It stops the boot, with its own framed console message, only when a
2xx answer gives a `Content-Length` that cannot fit in RAM. When it cannot
check the size (no answer, an HTTP error such as the 403 a signed
object-store URL gives to HEAD, or no `Content-Length`), it prints
`BLUEFIN: cannot check that the OS image fits in RAM (<reason>); downloading it anyway`
and the download runs: whether the image can be fetched is for systemd-import
to find out. A failed download goes straight to the summary instead of
waiting out the 300 s `/usr` device timeout.

## Minimum RAM

The netboot initrd downloads the whole OS DDI (`bluefin-server_<ver>.raw`,
540 MiB for `26.09.x`) into `/run/machines`. systemd mounts `/run` as a tmpfs
capped at 20% of RAM, and the DDI stays there for the whole boot because it
backs `/usr`. The pull check requires the DDI plus 64 MiB to fit in both the
free space of `/run` and `MemAvailable`, so usable RAM (`MemTotal`) must be at
least about five times the DDI plus 64 MiB: about 3 GiB for a 540 MiB DDI.
Measured in QEMU with the `26.09.x` DDI, `-m 3072` (2816 MiB `MemTotal`) is
refused and `-m 3584` boots. Plan on 4 GiB plus the workload. If the server
does not give the size, the check is skipped and a machine below the minimum
fails during the download instead (`systemd-import@...service` failed).

A machine below the minimum fails before the download (QEMU `-m 2048`):

```text
BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT
This machine needs ~3072 MiB RAM, have 1876 MiB.
The OS image bluefin-server_26.09.2.raw (540 MiB) is downloaded into /run, a RAM
file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB.
```

## Failure modes

| Symptom in the summary | Cause | Timing |
|---|---|---|
| `systemd-networkd-wait-online.service` failed, "no network link became routable" | No DHCP answer, no carrier | `systemd-networkd-wait-online --any --timeout=300`: 300 s, then the download fails and the node reboots |
| "no network interface found" | NIC driver not in the initrd module list (`initrd-modules` in `oci/bluefin-server-boot.bst`) | As above |
| `systemd-import@...service` failed, systemd-importd says `Transfer failed: ...` | Routable link but no route to, or no answer from, the boot server | When the download starts |
| `systemd-import@...service` failed, systemd-importd says `HTTP request to ... failed with code 404.` | The DDI is not next to the netboot UKI (or at the `import.pull` URL) | When the download starts |
| "NOT ENOUGH RAM FOR A DISKLESS BOOT" | See [Minimum RAM](#minimum-ram) | Seconds |
| `systemd-import@...service` failed, with systemd-importd's reason (e.g. `DOWNLOAD INVALID: Checksum ... did not check out`) | Download interrupted, the DDI does not match `SHA256SUMS`, or `SHA256SUMS.gpg` is missing or not signed by a key in the image keyring | Right after the download |
| `systemd-veritysetup@usr.service` or `sysusr-usr.mount` failed | The DDI does not match the `usrhash=` of the UKI that booted (mixed versions on the server) | After the download |
| "No unit failed: a device or job timed out" | A device never appeared within `systemd.default_device_timeout_sec=300` | 300 s |
| `ignition-<stage>.service` failed | See [Ignition](#ignition) | During the stage |

## Getting logs from a diskless node

Every diskless boot starts with a fresh machine ID and keeps its journal in
RAM (`/run/log/journal` in the initrd, then `/var/log/journal` on the tmpfs
root), so nothing survives a reboot by default.

- **Initrd failures** never reach a disk: read the console. The netboot UKI
sets `console=tty0 console=ttyS0,115200`; use a serial cable, IPMI
serial-over-LAN or the BMC console log. `netconsole` is built into the
kernel, but its `netconsole=` argument cannot be added to the sealed command
line, so it is not available in the initrd.
- **Persistent logs** after switch-root: provision a `/var` disk with Ignition
(`tests/fixtures/ignition/var-on-disk.ign`, the same state-disk pattern the
boot server's per-host `bluefin-node.ign` uses; `/var/log/journal` is created
on it by `systemd-tmpfiles`, so journald persists without extra config).
Each boot writes under its own machine ID and plain `journalctl` shows only
the current one, so read across boots with
`journalctl -D /var/log/journal --list-boots` and
`journalctl -D /var/log/journal -b -1`. Do not pin
the ID by writing `/etc/machine-id` with Ignition: a set machine ID ends the
first-boot state that applies the presets on every diskless boot, and
preset-enabled units stop starting. The `system.machine_id` credential (a
32-hex ID, or `firmware` for the SMBIOS UUID) pins it without that effect.
- **Remote logs**: the image ships `systemd-journal-upload`; an Ignition config
can write `/etc/systemd/journal-upload.conf` (`[Upload]` / `URL=`) and enable
`systemd-journal-upload.service` to stream the journal of the running system
to a `systemd-journal-remote` collector. It starts after switch-root, so it
does not cover initrd failures.

## Ignition

Configs arrive as the `ignition.config` / `ignition.config.url` credentials or
as `bluefin-node.ign` next to the netboot UKI (see
the Ignition section of [ddi-installer.md](ddi-installer.md)). Ignition kernel
arguments (`ignition.config.url=`, `ignition.platform.id=`, ...) are never read;
if any are on the command line, `bluefin-ignition-kargs.service` prints a
framed warning naming them (not their values) and the failure summary repeats
it.

Stages wired into the initrd (`files/initrd-ignition/`), in order:
`fetch-offline`, `fetch` (only when the config needs the network), `disks`,
`mount`, `files`, and `umount` (`ExecStop` of `ignition-mount.service`). The
`kargs` stage is not wired, so `kernelArguments` is silently not applied: the
command line is sealed in the signed UKI.

| Config section | Supported |
|---|---|
| `ignition.config.merge` / `replace`, `timeouts`, `security.tls` | Yes |
| `storage.disks` (partitions via `sgdisk`) | Yes |
| `storage.filesystems` | `xfs`, `ext4`, `vfat`, `swap`; no `btrfs` |
| `storage.luks` | Key files only; no `clevis` (TPM2 / Tang) |
| `storage.raid` | No (`mdadm` absent) |
| `storage.files`, `directories`, `links` | Yes |
| `systemd.units` (contents, dropins, `enabled`) | Yes; enabled units go through a preset applied by the real root |
| `passwd.users`, `passwd.groups` | Yes |
| `kernelArguments` | No |

There is no first-boot marker: every diskless boot runs every stage against a
fresh `/etc`, so a config must be idempotent or the node fails the same stage
on every boot:

- **Disks**: keep `wipeTable: false` and `wipePartitionEntry: false`; an
existing partition must match the config (number, label, size, type) or
`ignition-disks` fails.
- **Filesystems**: `wipeFilesystem: false` reuses a filesystem with the same
format (and label/UUID when set) and fails on any other;
`wipeFilesystem: true` erases the disk on **every** boot.
- **Files**: a file with `contents` fails the `files` stage when something is
already at its path and `overwrite` is false (the default): true from the
second boot for files on a persistent filesystem, and on every boot for
paths shipped in the factory `/etc`. Use `"overwrite": true`.

## Verification

- [ ] `just dogfood-check` passes (normal diskless boot).
- [ ] `DOGFOOD_MEM=2048 scripts/dogfood-diskless.sh dist/diskless --check`
shows "NOT ENOUGH RAM FOR A DISKLESS BOOT" in `dist/diskless/dogfood-serial.log`.
- [ ] `DOGFOOD_TAMPER=sums scripts/dogfood-diskless.sh dist/diskless --check`
shows "BLUEFIN: BOOT FAILED" and the failed `systemd-import@` unit.
- [ ] `python3 -m pytest tests/unit/test_initrd_diagnostics.py` passes.

## See also

- [ddi-installer.md](ddi-installer.md) — boot chain, diskless pull, Ignition credentials.
- [ddi-installer-build.md](ddi-installer-build.md) — dogfood harness and its `DOGFOOD_*` variables.
3 changes: 2 additions & 1 deletion docs/skills/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Lazy-load manifest for Bluefin Server skills. Load this file after
metadata:
type: index
status: stable
last_updated: "2026-09-27"
last_updated: "2026-09-28"
---
# docs/skills — Index

Expand All @@ -28,6 +28,7 @@ This is the lazy-load routing table for agent skills. Keep this file in memory w
| [`ci-tooling.md`](ci-tooling.md) | GitHub Actions, workflow SHA pinning, CI conventions | CI conventions and release pipeline rules. |
| [`ddi-installer-build.md`](ddi-installer-build.md) | Building and dogfooding the image set | Local build, export, and QEMU dogfood workflow. |
| [`ddi-installer.md`](ddi-installer.md) | Boot flow, diskless pull, `systemd-sysinstall`, A/B sysupdate | Boot, install, and update architecture. |
| [`diskless-troubleshooting.md`](diskless-troubleshooting.md) | Diskless node reboot-loops, RAM sizing, keeping logs, writing Ignition configs | Failure summary, minimum RAM, failure modes, logs, supported Ignition subset. |
| [`factory-integration.md`](factory-integration.md) | Lab integration, boot-test workflow, factory role | How Bluefin Server is consumed by the CI lab. |
| [`gap-analysis-distros.md`](gap-analysis-distros.md) | Comparing Bluefin Server to other server OSes | Source-verified comparison to Ubuntu, Talos, Flatcar, FCOS. |
| [`kubeadm-sysext.md`](kubeadm-sysext.md) | Joining a kubeadm cluster, kubelet/containerd sysext, Kubernetes kernel options | Build and runtime contract for the kubeadm worker sysext. |
Expand Down
5 changes: 5 additions & 0 deletions elements/oci/bluefin-server-boot.bst
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ config:
ln -sf ../systemd-resolved.service "${wants}/initrd.target.wants/systemd-resolved.service"
ln -sf ../imports.target "${wants}/initrd.target.wants/imports.target"
ln -sf ../bluefin-populate-etc.service "${wants}/initrd.target.wants/bluefin-populate-etc.service"
ln -sf ../bluefin-ignition-kargs.service "${wants}/initrd.target.wants/bluefin-ignition-kargs.service"
ln -sf ../systemd-networkd-wait-online.service "${wants}/network-online.target.wants/systemd-networkd-wait-online.service"
for tool in systemd-importd systemd-pull; do
[ -x "usr/lib/systemd/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
Expand All @@ -162,6 +163,10 @@ config:
[ -x "usr/bin/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
done
[ -x usr/lib/systemd/systemd-veritysetup ] || { echo "ERROR: initrd lacks systemd-veritysetup" >&2; exit 1; }
# bluefin-boot-diagnostics (failure summary, RAM check) and what it runs.
for tool in libexec/bluefin-boot-diagnostics bin/bash bin/curl bin/stat bin/timeout bin/tee; do
[ -x "usr/${tool}" ] || { echo "ERROR: initrd lacks ${tool}" >&2; exit 1; }
done
rm -rf usr/share/man usr/share/doc usr/share/info usr/include usr/lib/*/pkgconfig
find . -name '*.a' -delete
du -sh /initrd
Expand Down
13 changes: 13 additions & 0 deletions files/initrd/usr/lib/systemd/system/bluefin-ignition-kargs.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[Unit]
Description=Check for ignored Ignition kernel arguments
Documentation=file:/usr/libexec/bluefin-boot-diagnostics
ConditionPathExists=/etc/initrd-release
DefaultDependencies=no
# Linked into initrd.target.wants by oci/bluefin-server-boot.bst. Warn before
# Ignition would have run, so the warning sits next to where it is missing.
Before=bluefin-ignition-credentials.service initrd.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/libexec/bluefin-boot-diagnostics ignition-kargs
22 changes: 22 additions & 0 deletions files/initrd/usr/lib/systemd/system/bluefin-pull-check@.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
[Unit]
Description=Check that the OS image fits in RAM before downloading it
Documentation=file:/usr/libexec/bluefin-boot-diagnostics
ConditionPathExists=/etc/initrd-release
DefaultDependencies=no
# Wanted by, and ordered before, systemd-import@%i.service through its drop-in
# systemd-import@.service.d/10-bluefin-pull-check.conf. Needs the network,
# like the download itself.
Wants=network-online.target
After=network-online.target

# The check fails only when the server gives a size that cannot fit in RAM:
# print why on the console and reboot (emergency.service.d/10-reboot.conf)
# rather than start that download. When it cannot check the size, it notes
# that on the console and succeeds, and the download runs.
OnFailure=emergency.target
OnFailureJobMode=isolate

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/libexec/bluefin-boot-diagnostics check-pull systemd-import@%i.service
Loading