Skip to content

feat(initrd): explain diskless boot failures and ignored Ignition kargs - #273

Merged
jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:feat/diskless-diagnostics
Sep 29, 2026
Merged

jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:feat/diskless-diagnostics

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Diskless boot failures were hard to diagnose and some looped with no clear message:

  • The failure path printed journalctl -p err to /dev/console and force-rebooted after 10 s. /dev/console is only the last console= on the netboot command line (ttyS0), so the monitor never showed the reason, and it scrolled away within seconds.
  • The netboot initrd pulls the whole DDI (540 MiB) into /run/machines, on a tmpfs that systemd caps at 20% of RAM. A small machine failed deep in the download.
  • A failed download (bad signature, HTTP error, out of space) only came up when the /usr devices timed out, systemd.default_device_timeout_sec=300 later.
  • ignition.config.url= and the other ignition.* kernel arguments were silently ignored.
  • Nothing documented the minimum RAM, how to keep logs, or which parts of Ignition are wired up.

Change

Everything is systemd units and drop-ins plus one libexec helper, files/initrd/usr/libexec/bluefin-boot-diagnostics. The helper uses only bash builtins, coreutils, curl, systemctl and journalctl, because the initrd has no sed or awk.

  1. Failure summary (emergency.service.d/10-reboot.conf). failure-summary lists each failed unit with a one-line hint and what its own processes logged. For a failed download, that includes systemd-importd's reason. It also shows network link state when the network or download failed, any ignored ignition.* args, and the last errors. It writes to every active console (/sys/class/tty/console/active), then waits 60 s before the unattended reboot. The bluefin.failure_delay credential changes the delay (the kernel argument works only where the command line isn't locked). There is a 10 s floor so a failure can't turn into a hot loop. Type=idle keeps the isolate's status lines out of the summary. The retry-by-reboot behavior is unchanged.

  2. RAM / reachability check before the pull. A drop-in makes every systemd-import@.service require bluefin-pull-check@%i.service. That unit takes the URL the import generator already resolved (from the unit's Download of <URL> description, so it covers rd.systemd.pull, bootorigin and the import.pull credential without deriving the URL a second time) and sends it an HTTP HEAD request. It fails with needs ~X MiB RAM, have Y if the image plus 64 MiB won't fit in both /run and MemAvailable. It also fails with the URL and link state if the server can't be reached, or with the HTTP status on an error response. 405/501 and a missing Content-Length skip the check rather than block the boot. The same drop-in sends a failed download straight to the summary instead of waiting out the 300 s device timeout.

  3. Ignition guard rails. bluefin-ignition-kargs.service prints a framed warning that names any ignition.* keys on /proc/cmdline (never their values) and says how to pass the config instead. A failed ignition-<stage>.service gets a hint that names the stage and points at the idempotency rules. The every-boot semantics are unchanged; there is no first-boot marker.

  4. Docs. New docs/skills/diskless-troubleshooting.md is the single source for the summary, minimum RAM, failure modes and their timing, getting logs, and the supported Ignition subset with its idempotency rules. ddi-installer.md links to it instead of repeating it. It is also listed in index.md and AGENTS.md, and DOGFOOD_MEM is documented.

  5. Tests. tests/unit/test_initrd_diagnostics.py (35 tests):

    • unit and drop-in contracts;
    • helper behavior against a fake /proc/meminfo, fake consoles, a fake sysfs net dir, and a local http.server answering HEAD (fits / too big for /run / low MemAvailable / redirect / 404 / 405 / 501 / no Content-Length / connection refused);
    • karg warning;
    • failure summary and delay parsing (floor, octal-looking values, karg override);
    • every console link points at an existing doc section.

    The UKI build also checks that the helper and the tools it runs are in the initrd.

Verification

  • python3 -m pytest tests/unit -q: 406 passed, run in a throwaway detached worktree with no boot keys.
  • bats tests/unit: 74 ok.
  • python3 .github/scripts/docs-checks.py: passed.
  • flock … just validate: EXIT=0.
  • shellcheck -S style: clean.
  • The image was built with flock /tmp/opencode/bst.lock just export-image. All QEMU runs used flock /tmp/opencode/qemu.lock with Secure Boot OVMF, except (c).

(a) Normal diskless boot: just dogfood-check passes

PROBE secureboot=enabled (user)
PROBE verity=verified
PROBE diag=bluefin-boot-diagnostics: OS image 540 MiB fits: /run has 757 MiB free, MemAvailable 3301 MiB|
PROBE upload=/usr/lib/systemd/systemd-journal-remote /usr/lib/systemd/systemd-journal-upload
PROBE failed=0
PASS: netboot boot with no failed units

(b) Low RAM (DOGFOOD_MEM=2048, DDI 540 MiB)

The check fails about 6 s into the boot, the summary follows, and the node reboots at 72 s:

==============================================================================
BLUEFIN: NOT ENOUGH RAM FOR A DISKLESS BOOT
This machine needs ~3072 MiB RAM, have 1876 MiB.
The OS image bluefin-server_26.09.2.raw (540 MiB) is downloaded into /run, a RAM
file system capped at 20% of RAM: 371 MiB free there, MemAvailable 1374 MiB.
Add RAM or install to disk. https://github.com/projectbluefin/server/blob/main/docs/skills/diskless-troubleshooting.md#minimum-ram
==============================================================================
...
BLUEFIN: BOOT FAILED

FAILED: bluefin-pull-check@run-machines-rootdisk.raw.service
  -> the OS image cannot be downloaded into RAM (see the message above)
  | bluefin-boot-diagnostics: This machine needs ~3072 MiB RAM, have 1876 MiB.
...
Rebooting in 60 s (set the bluefin.failure_delay credential to change the delay).

The RAM boundary was checked on the first build of this branch, which had the same check code. DOGFOOD_MEM=3072 was refused (needs ~3072 MiB RAM, have 2816 MiB) and DOGFOOD_MEM=3584 booted with PROBE failed=0.

(c) ignition.* kernel arguments

The dogfood harness cannot add kernel arguments under Secure Boot, because the signed UKI seals the command line. So this ran through a non-Secure-Boot path: the same kernel, initrd and command line extracted from the netboot UKI, booted with -kernel/-initrd/-append on plain OVMF, with ignition.config.url=http://…/secret-token.ign ignition.platform.id=metal appended. The unit tests also cover this. The boot reached multi-user.target, and the console showed:

==============================================================================
BLUEFIN: IGNORED KERNEL ARGUMENTS: ignition.config.url ignition.platform.id
Bluefin Server never reads Ignition settings from the kernel command line.
Pass the config as the ignition.config or ignition.config.url system credential
(SMBIOS type 11, QEMU fw_cfg, or /loader/credentials/ on the ESP), or serve
bluefin-node.ign next to the netboot UKI. https://github.com/projectbluefin/server/blob/main/docs/skills/diskless-troubleshooting.md#ignition
==============================================================================

The token appears only in the kernel's own Command line: echo, not in the warning.

(d) Broken pull (DOGFOOD_TAMPER=sums)

The summary appears at 9 s, instead of after the 300 s device timeout:

BLUEFIN: BOOT FAILED

FAILED: systemd-import@run-machines-rootdisk.raw.service
  -> downloading or verifying the OS image failed: server reachable? SHA256SUMS signed by a key this image trusts? room in /run?
  | (transfer1) DOWNLOAD INVALID: Checksum of bluefin-server_26.09.2.raw file did not check out, file has been tampered with.
  | Transfer process failed with exit code 1.
  | Method call failed: io.systemd.Import.TransferFailed
  network links:
    enp0s2 up (carrier)
...
Rebooting in 60 s (set the bluefin.failure_delay credential to change the delay).

Extra runs

  • Non-idempotent Ignition config (a file with contents at a path shipped in the factory /etc, run on the first build):

    FAILED: ignition-files.service
      -> Ignition stage 'files' failed. Every diskless boot runs every stage, so the config must be idempotent. …#ignition
      | Ignition failed: … error creating file "/sysroot/etc/os-release": A non regular file exists there already and overwrite is false
    
  • Logs across boots (var-on-disk.ign plus a state disk, booted twice): /var is on /dev/vdb1, and there is one journal directory per boot (a fresh machine ID each time). Plain journalctl --list-boots shows 1 boot, --merge --list-boots shows 0, and journalctl -D /var/log/journal --list-boots shows 2. The docs recommend the last one.

  • Pinning the machine ID through Ignition breaks things. An earlier attempt wrote /etc/machine-id with Ignition. That ended first-boot mode, so preset-enabled units (including the dogfood probe) never started. The docs now warn against it.

Not done / notes

  • No-DHCP path not run in QEMU. It needs the 300 s wait-online timeout. The pull check's connection-failure branch and the wait-online hint are covered by the unit tests (connection refused, no carrier).
  • system.machine_id credential is untested. The docs name it as the way to pin the machine ID without losing first-boot mode. That comes from reading systemd v261 source (main.c decides first boot from /etc/machine-id before machine_id_setup() reads the credential); it was not run in QEMU.
  • Possible follow-up: mounting the initrd /run with a larger size= would cut the diskless minimum from about 3 GiB to roughly DDI + working set. This PR only diagnoses the limit.

jeefy and others added 5 commits September 28, 2026 22:23
The emergency drop-in printed 'journalctl -p err' to /dev/console (only the
last console=, ttyS0 on the netboot UKI) and rebooted after 10 s, so the
reason scrolled away and never reached the monitor.

emergency.service now runs /usr/libexec/bluefin-boot-diagnostics
failure-summary: each failed unit with a one-line hint (network, download,
verity, Ignition stage) and the lines its processes logged, the network
links when the network or download failed, and the last errors, written to
every active console. It stays up for 60 s (bluefin.failure_delay credential,
10 s floor against hot loops) before the unattended reboot. Type=idle keeps
the isolate's status lines out of the summary. The helper also carries the
pull RAM check and the Ignition karg warning wired up in the next commits.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… pull

The netboot initrd downloads the whole DDI (540 MiB) into /run, a tmpfs
systemd caps at 20% of RAM. A small machine failed deep in the download, and
any failed download only surfaced when the /usr devices timed out 300 s later.

A drop-in makes every systemd-import@.service require
bluefin-pull-check@%i.service, which takes the URL the import generator
resolved (rd.systemd.pull, bootorigin, or the import.pull credential), HEADs
it and fails with 'needs ~X MiB RAM, have Y' when the image plus 64 MiB does
not fit in /run or MemAvailable, or with the URL and link state when the
server is unreachable or answers an error. Both the check and the download
go straight to the failure summary on failure.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ignition configs come only from the ignition.config / ignition.config.url
credentials or bluefin-node.ign; ignition.config.url= and friends on the
command line were silently ignored.

bluefin-ignition-kargs.service, linked into initrd.target.wants, prints a
framed warning naming the ignition.* keys it found (never their values) and
how to pass the config instead; the failure summary repeats it. The UKI
build now also checks that the diagnostics helper and the tools it runs are
in the initrd.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ition subset)

New docs/skills/diskless-troubleshooting.md is the one place for what a
failing diskless node prints and how to tune the delay, the minimum RAM
(DDI x5 + margin, measured in QEMU: -m 3072 refused, -m 3584 boots), the
network/download/verity/Ignition failure modes and their timing, how to keep
logs (serial console, /var on disk via Ignition, journal-upload), and the
supported Ignition subset with its idempotency rules. ddi-installer.md links
to it instead of restating, and a test pins every console link to an
existing section.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Measured in QEMU with a /var disk: each diskless boot journals under a new
machine ID, so plain journalctl lists one boot and --merge --list-boots lists
none, while journalctl -D /var/log/journal --list-boots lists both.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at c58c77d. The failure summary and the kargs warning look good. One problem in the pull pre-check.

The HEAD pre-check can refuse a source the download would accept

systemd-import@.service.d/10-bluefin-pull-check.conf adds Requires= and After= on bluefin-pull-check@%i.service, so the image download only starts if bluefin-boot-diagnostics check-pull exits 0. In check_pull (files/initrd/usr/libexec/bluefin-boot-diagnostics:131-140), any final HEAD status other than 2xx, 405 or 501 returns 1, and OnFailure=emergency.target then reboots. A server that serves the image on GET but answers HEAD with 403 or 404 would work fine with systemd-import, but it now never boots. Presigned or signed-URL object stores behave this way, because the signature covers the method. The pre-check is meant to explain failures, so it shouldn't be able to cause one. Suggest treating any non-2xx HEAD as "cannot check the size", logging at warning and returning 0 like the 405/501 branch does, and leaving the real error to the download's own OnFailure= (which this PR already adds). The unreachable-server branch (curl rc != 0) is fine to keep fatal, since the GET would fail the same way.

What I verified

  • pytest tests/unit: 406 passed. bats tests/unit: pass. shellcheck -S warning files/initrd/usr/libexec/bluefin-boot-diagnostics: clean.
  • The script only uses bash builtins plus curl, stat, timeout, tee, systemctl and journalctl. bluefin-server-boot.bst now fails the build if bash, curl, stat, timeout or tee are missing from the initrd, and initrd-stack.bst already pulls components/curl.bst.
  • Console writes go through timeout 5 tee for each active console, so a stuck serial console can't hold the emergency path. emergency.service keeps ExecStart=- with SuccessAction=/FailureAction=reboot-force, and the delay is clamped to 10..86400 s, so a failing summary still reboots and can't hot-loop.
  • The RAM math works in KiB before multiplying, and falls back to skipping the check when /proc/meminfo or stat -f is unreadable.
  • On the UNSTABLE state: unit (run 36518954264) and docs (36518954387) pass, and build (36518954330) was still running when I checked. No failed check, just one still pending.

jeefy and others added 3 commits September 29, 2026 02:02
The pull check failed, and so stopped the boot, on any final HEAD status
other than 2xx, 405 or 501, and when curl could not reach the server. A
server that serves the image on GET but answers HEAD with 403 or 404, as
signed object-store URLs do because the signature covers the method, never
booted, although systemd-import would have downloaded the image.

The check now fails only when a 2xx HEAD gives a Content-Length that cannot
fit in /run and MemAvailable. Any other outcome (no answer, a 4xx or 5xx,
no Content-Length, unreadable RAM or /run sizes) prints one line on every
console, "BLUEFIN: cannot check that the OS image fits in RAM (<reason>);
downloading it anyway", and exits 0. If the download then fails, its own
OnFailure= puts systemd-importd's reason in the failure summary.

The check runs in a subshell and reports "does not fit" with a status of
its own, which the helper turns into exit 1; any other status, such as bash
aborting on an unset variable, exits 0, so a bug in the check cannot stop
the boot either. Since the check now fails only on RAM, its failure-summary
hint says so and no longer lists the network links.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-import@.service required bluefin-pull-check@%i.service, so any
failure of the check, or a missing check unit, failed the download's start
job. The check already stops the boot through its own
OnFailure=emergency.target when the image cannot fit, so Wants= with the
existing After= keeps it ordered before the download without making the
download depend on it.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the one-line note the pull check prints when it cannot check the
size, and replace the "CANNOT REACH THE OS IMAGE SERVER" and "ANSWERED HTTP
404" rows, which the check no longer prints, with the systemd-importd
reasons the failure summary shows for the same causes. Without a network
the download, not the check, now fails after the 300 s wait, and a machine
below the minimum RAM fails during the download when the server does not
give the size.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks, fixed in 3a0c43b, cfce154 and f155a77.

The check now fails only when it knows the image cannot fit. check_pull exits 1 only when a 2xx HEAD reports a Content-Length that does not fit in /run and MemAvailable (the existing RAM message). Every other outcome prints one line on every console (logged at warning) and exits 0, so the download runs and systemd-import@'s own OnFailure= reports any real error:

BLUEFIN: cannot check that the OS image fits in RAM (HEAD answered HTTP 403); downloading it anyway

That covers any non-2xx final status (403/404 from signed URLs, 405/501, 5xx), a 2xx without Content-Length, and unreadable RAM or /run sizes. It also covers curl failing to reach the server. You said that branch was fine to keep fatal, but I made it a note too: a failed HEAD (a timeout, DNS not ready yet) does not prove the GET fails, and when the GET does fail, the summary shows systemd-importd's Transfer failed: … with the link state. So the diagnosis is kept, and the check cannot be what fails the boot.

A broken check cannot block either.

  • check_pull runs in a subshell and reports "does not fit" with its own status, which the helper turns into exit 1. Any other status, such as bash aborting on an unset variable, exits 0.
  • The drop-in is now Wants= + After= instead of Requires=. The check stops the boot only through its own OnFailure=, and a missing or broken check unit no longer fails the download's start job.

Tests (tests/unit/test_initrd_diagnostics.py):

  • HEAD 403/404/405/500/501 → exit 0, with the note on both consoles and at <4> in the journal.
  • 2xx with no Content-Length → same.
  • Connection refused → exit 0, with the curl error in the note.
  • A malformed MemAvailable that makes bash abort the check → exit 0.
  • 2xx oversized (/run too small, low MemAvailable, or reached through a redirect) → exit 1.
  • 2xx that fits → exit 0, console quiet.
  • The drop-in has Wants= and no Requires=.

Run against the helper at c58c77d, 10 of these fail, including 403/404/500. Without the subshell, only the malformed-meminfo test fails.

Verified

  • pytest tests/unit: 410 passed, in a throwaway detached worktree at f155a77 without boot keys. Each of the three commits passes on its own.
  • bats tests/unit: 74 ok.
  • docs-checks.py: passed.
  • shellcheck -S style: clean.
  • just validate: rc 0.
  • Rebuilt the image with just export-image (only the initrd, UKI and image elements rebuilt) and checked that the netboot UKI's initrd carries the new helper and drop-in. QEMU runs with Secure Boot OVMF:
    • just dogfood-check: PASS, with PROBE diag=bluefin-boot-diagnostics: OS image 540 MiB fits: /run has 757 MiB free, MemAvailable 3305 MiB.
    • DOGFOOD_MEM=2048: still refused before the download. It printed This machine needs ~3072 MiB RAM, have 1876 MiB., the summary hint was the OS image is too big for the RAM of this machine, and the node rebooted at 71 s. The HTTP log has no GET of the image.
    • Your case: a throwaway server on the harness port that answers HEAD with 403 and serves GET. It logged "HEAD /bluefin-server_26.09.2.raw" 403, then "GET /bluefin-server_26.09.2.raw" 200 (plus SHA256SUMS and SHA256SUMS.gpg). The console showed:
               Starting Check that the OS image fits in RAM before downloading it...
      BLUEFIN: cannot check that the OS image fits in RAM (HEAD answered HTTP 403); downloading it anyway
      [  OK  ] Finished Check that the OS image fits in RAM before downloading it.
      [  OK  ] Finished Download of http://10.0.2.2:8779/bluefin-server_26.09.2.raw.
      
      The probe reported verity=verified and failed=0, and the run passed.

diskless-troubleshooting.md now describes the note. It also replaces the "CANNOT REACH" and "ANSWERED HTTP 404" rows with the systemd-importd messages the summary shows for those causes.

main has moved to fd34c54 (#275, #272) since the review, but the merge is clean (GitHub reports MERGEABLE), so I did not merge it in. CI on f155a77: unit and docs pass, build is still running.

castrojo pushed a commit that referenced this pull request Sep 29, 2026
* docs: replace stale Flatcar narrative with FSDK 26.08 facts

The OS payload is now pure FSDK 26.08 (systemd v261). Update comments in
shipped elements, scripts, and tests to remove Flatcar references and
explain why each workaround still exists:

- /etc/resolv.conf: FSDK ships no rule (upstream systemd leaves the
  symlink to the distro), and /etc is empty on every boot, so the
  tmpfiles.d rule is still required for kubelet pod sandboxes.
- firstboot credentials: the helper exists because Bluefin applies
  firstboot.hostname unconditionally and sets the live kernel hostname,
  which systemd-firstboot does not do; it is not because systemd-firstboot
  is missing from the image.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: correct patch count in bump-fsdk-version.md

patches/freedesktop-sdk/ carries six patches (0001-0006), not one.
0006 adds the Cilium/Kubernetes kernel options (VXLAN, GENEVE, tc BPF,
conntrack/ss diagnostics); dropping it breaks the kubeadm and k0s
sysexts. Update the verification checklist and the notes so a bumper
does not drop it.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: remove unused Justfile build alias

The build recipe was a forwarding alias to export-image. No doc, CI
workflow, or script calls bare 'just build'. Remove it to match the
avoid-over-engineering skill's rule against forwarding aliases.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs: add booty-integration and secure-boot-keys skills

booty-integration.md: how Booty serves releases (HTTP Boot, iPXE
chainload, per-node bluefin-node.ign, doInstall, kubeadm workers).

secure-boot-keys.md: what each key signs (PK/KEK/DB, module cert,
sysupdate signing), where keys live, how CI gets them via
BOOT_KEYS_TARBALL and SYSUPDATE_SIGNING_KEY, and rotation mechanics.

Index rows are deferred until #273 merges (it owns docs/skills/index.md).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix: stage system-container helper in the base image

files/bin/system-container was documented as shipped at
/usr/bin/system-container but no element staged it. Add
os-system-container.bst to install it into /usr/bin and wire it into
os-stack.bst. Remove the KNOWN_UNSTAGED waiver.

machinectl and systemd-nspawn ship in the base image via FSDK's
systemd component, so the helper's dependencies are present.

Closes #154

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy merged commit 476abdf into projectbluefin:main Sep 29, 2026
5 checks passed
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
Resolves the ddi-installer.md Common Rationalizations table against projectbluefin#273:
keeps projectbluefin#273's ignored-karg note on the Ignition row and this branch's
update/rollback rows.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
…refusal

Since projectbluefin#273 the initrd writes its failure summary to the console while
systemd-importd logs the refusal, so DCS/OSC sequences can land inside the
refusal line and refused() missed it (boot-test run 36560436141). Match
against the same cleaned log the harness saves.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy added a commit that referenced this pull request Sep 29, 2026
…kip image builds for docs-only changes (#270)

* test(ignition): fixture that writes a file and enables a unit

apply-marker.ign writes /etc/dogfood-ignition-marker and enables a oneshot unit through Ignition's preset; apply-marker.probe reports both from inside the booted guest, so a dogfood boot can assert that Ignition applied the config rather than only that the node booted.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci: classify pull requests that cannot change the image set

image-build-needed.py reads a PR's changed paths and answers false only for docs, Markdown outside the build inputs, unit/e2e tests and workflows that do not build the image. The unit test fails if any tracked file under a build input root (elements/, files/, include/, patches/, plugins/, scripts/, tests/fixtures/, project.conf, Justfile, build.yml) would skip the build.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(dogfood): honour DOGFOOD_PORT in dogfood-install.sh

The update probe rewrote the sysupdate transfers to a hardcoded 10.0.2.2:8765, so any DOGFOOD_PORT other than the default served the image on one port while the guest updated from another. Export the port once and substitute it into the probe.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* feat(dogfood): assert tamper refusal and probe expectations

With DOGFOOD_TAMPER, --check now passes only if the initrd's pull fails after the image, SHA256SUMS and SHA256SUMS.gpg were served and nothing booted; a systemd.unit-dropin credential copies the pull's own messages to the serial console so the log says why. DOGFOOD_EXPECT=<ERE> makes --check also require a probe line (e.g. Ignition's marker).

Robustness: key enrollment for HTTP boot waits (bounded) for the firmware to reset after systemd-boot enrolls instead of a fixed sleep, and keeps its log; the EXIT trap kills QEMU too, so an aborted run no longer leaves an orphaned guest holding the port and the caller's locks. The trap only kills PIDs that are set, since kill 0 signals the whole process group.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(build): boot-test tamper refusal, HTTP boot, Ignition and A/B rollback; skip docs-only builds

boot-test now also runs DOGFOOD_TAMPER=raw and =sums (must be refused), Ignition from the ignition.config credential and UEFI HTTP boot with bluefin-node.ign (both must show the fixture's file and enabled unit), and off main the full install -> sysupdate A->B -> boot-counted rollback. The build job exports the two higher-versioned sets it needs (<ver>.1, <ver>.2) from the warm cache; main skips them so no unpublished release-signed versions exist. Failed boot tests upload their serial logs.

A changes job skips build and boot-test for pull requests image-build-needed.py classifies as docs/tests-only; skipped jobs report as passing. On main, an opt-in step pushes all artifacts to cache.projectbluefin.io:11002 when the CASD_CLIENT_CERT variable and CASD_CLIENT_KEY secret exist (the dakota identity), so main builds stop rebuilding the kernel; it is best effort and a no-op without them. The release job is unchanged.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): CI boot coverage, build time and cache setup

ci-tooling.md is canonical for what boot-test covers, the measured build-time breakdown (kernel on the critical path), the docs-only skip, the CASD_CLIENT_CERT / CASD_CLIENT_KEY cache push, and why PR kernels and actions/cache are out of scope. ddi-installer-build.md documents DOGFOOD_EXPECT/PORT/MEM/TIMEOUT, the tamper pass criterion, how to make next/broken sets, and points at ci-tooling.md for CI coverage.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(dogfood): make DOGFOOD_TAMPER=sums test the signature, assert the reason

The sums tamper overwrote every hash in SHA256SUMS, so the pull failed on the checksum and the signature check was never exercised. It now serves the corrupted image with SHA256SUMS re-hashed to match it, which only SHA256SUMS.gpg can expose. --check requires the pull's own verdict on the serial console: 'DOWNLOAD INVALID: Checksum of <image> file did not check out' for raw, 'DOWNLOAD INVALID: Signature verification failed' for sums. Drop-ins for systemd-import@.service and systemd-importd.service (SYSTEMD_LOG_TARGET=console, journal+console) put those messages there.

The EXIT cleanup moves into a function (shellcheck SC2154 on the inline trap).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(build): version the update test sets 1.<run>.N

systemd-sysinstall writes the installed image's boot entry as a Type #1 entry with 'version 1.<ver>' (bluefin-server-commit_1.<ver>.conf). systemd-boot sorts it above a 0.<run>.1 update, so the updated node kept booting 0.<run> and the A/B step failed (reproduced locally with 0.900 -> 0.900.1). 1.<run>.N sorts above both and below every release; with it the full install, A/B update and rollback pass locally.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): tamper reasons and update test versions

Document which refusal each DOGFOOD_TAMPER mode asserts, and that next/broken sets must sort above the installed entry's 'version 1.<ver>'.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(dogfood): fail --check when Secure Boot is not enforced

Firmware that refuses systemd-boot's key enrollment boots on in setup mode, where nothing is verified, and --check still passed. CI's boot-test did exactly that on Ubuntu's OVMF: every probe reported 'secureboot=disabled (setup)' (e.g. run 36499270842). --check now requires 'PROBE secureboot=enabled', and a tamper run requires the kernel's 'Secure boot enabled' line.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(build): boot-test on Fedora's edk2 OVMF so Secure Boot is enforced

Ubuntu 26.04's OVMF 2025.11 rejects systemd-boot's PK enrollment ('Failed to write PK secure boot variable: Security violation'), so boot-test never ran with Secure Boot on, and the new HTTP boot step, which enrolls explicitly, fails outright. Fetch Fedora's edk2-ovmf 20260508-6.fc44 from Koji, check its SHA-256, extract OVMF_CODE.secboot.fd / OVMF_VARS.fd with bsdtar and point the harness at them. It is the build dogfooding uses locally, where enrollment works.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): Secure Boot enforcement in boot-test and the OVMF pin

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(build): drop the BuildStream cache push

bst artifact push --deps all oci/bluefin-server-image.bst would upload
bluefin-server/keys/boot-keys.bst, an import of files/boot-keys that on
main holds the Secure Boot db, module-signing and sysupdate private keys,
to a cache every build pulls from. Remove the push steps and the
CASD_CLIENT_CERT/CASD_CLIENT_KEY configuration; ci-tooling.md records
that pushing needs an element allow-list proven not to depend on
boot-keys.bst.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* ci(build): run the build classifier from the PR's base revision

The changes job checked out .github/scripts from the PR head, so a PR
could edit image-build-needed.py to answer false and skip build and
boot-test. Check out the base SHA instead, feed it the PR's changed
files, and build when the base has no classifier yet. The classifier
now lists itself and build.yml as gate files that always build; the
unit test asserts that and that the changes job checks out the base.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* docs(skills): the build classifier runs from the base revision

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* fix(dogfood): strip OSC and DCS sequences before matching the tamper refusal

Since #273 the initrd writes its failure summary to the console while
systemd-importd logs the refusal, so DCS/OSC sequences can land inside the
refusal line and refused() missed it (boot-test run 36560436141). Match
against the same cleaned log the harness saves.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants