Skip to content

fix(installer): rewrite the network DDI repart override without sed - #250

Merged
jeefy merged 2 commits into
projectbluefin:mainfrom
jeefy:fix/installer-sed-missing
Sep 26, 2026
Merged

jeefy merged 2 commits into
projectbluefin:mainfrom
jeefy:fix/installer-sed-missing

Conversation

@jeefy

@jeefy jeefy commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Branch: fix/installer-sed-missing. First of a stack of four installer PRs that touch the same inst.ddi_url block of the wrapper: this one → #251 (/dev/shm staging) → #252 (wait-online) → #253 (inst.creds_url). Based directly on main.

Problem

Every inst.ddi_url (PXE) install with the released installer-v26.08.0 initrd dies right after the DDI has been downloaded, verified and decompressed:

/usr/bin/bluefin-sysinstall: line 91: sed: command not found

exit 127 → FailureAction=poweroff. The wrapper uses sed to repoint CopyBlocks= in the 20-root-a.conf override it stages under /usr/lib/repart.sysinstall.d/. installer-stack.bst ships uutils coreutils, Bash, curl, zstd and systemd, and grep arrives transitively; sed is a build-depend of the installer element only and never lands in the live initrd — a mismatch #162 will make permanent when it drops it from the build too. (Confirmed against the released 26.08.0 initrd's file list: grep is present, sed/awk/tar are not.)

Reproduced in QEMU (OVMF, q35/KVM, 8 GiB) booting the released PXE kernel/initrd via iPXE with inst.ddi_url/inst.ddi_sha256; found while integrating with the jeefy/booty PXE server.

Fix

Write the override with plain Bash: copy /usr/lib/repart.d/20-root-a.conf line by line, replace only the column-0 CopyBlocks= line; every other line is unchanged apart from normalising a missing final newline. If the recipe carries no CopyBlocks= line the wrapper errors out before the disk is touched (recipe and wrapper ship from the same repo, so that is a repo bug, not something to paper over by appending a key). No new dependencies. Step 1a of the element now pins every external command the wrapper calls (grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep systemctl next to the existing five; all confirmed present in the released 26.08.0 initrd), so a missing tool fails the build instead of the install. Also split the one local x="$(...)" (SC2155) so the extracted wrapper is shellcheck -S warning clean.

Docs: ddi-installer.md now states the live initrd's real inventory (installer-stack.bst plus transitive runtime deps; sed/awk/tar absent) next to the PXE flow, points at the Step 1a tool check, and adds a Common Rationalizations row.

Verification

  • tests/unit/test_installer_contract.py:
    • test_installer_wrapper_does_not_call_sed_awk_or_tar — asserts no command position in the wrapper names sed/awk/tar (comment stripping only at # after whitespace/BOL, so # in strings does not hide code).
    • test_installer_build_pins_every_external_command_the_wrapper_calls — the Step 1a loop names each external command the wrapper uses.
    • test_installer_wrapper_copyblocks_repoint_matches_the_sed_it_replaced — runs the extracted block against the shipped files/installer/repart.d/20-root-a.conf and asserts the output equals the sed result it replaced (expected CopyBlocks= path derived from the wrapper's ROOT_COPYBLOCKS_LINE, so the stacked /dev/shm PR does not have to touch this test).
    • test_installer_wrapper_copyblocks_repoint_fails_closed_without_a_copyblocks_line — a recipe with no CopyBlocks= line, and one with an indented CopyBlocks= (deliberately not matched), both exit 1 with the wrapper's message and nothing appended.
    • tests/unit/conftest.py gains the shared, session-scoped installer_wrapper fixture (carves the heredoc out of the element) that the later PRs in the stack reuse.
    • All fail against main's element (and the fails-closed test also fails against the round-1 append fallback), pass with this branch.
  • python3 -m pytest tests/unit -q — 348 passed, 1 xfailed (pre-existing).
  • bash -n + shellcheck -S warning on the extracted wrapper — clean.
  • python3 .github/scripts/docs-checks.py + the check-* scripts — pass.
  • Not run: just validate / just build-installer (no local BuildStream).
  • QEMU (OVMF, q35/KVM, 8 GiB, iPXE from a Booty instance): the combined wrapper of this stack (this fix + /dev/shm staging + wait-online + inst.creds_url) was dropped into the released installer-v26.08.0 initrd; the install completes and the target boots. This Bash rewrite was not booted in isolation.

Review round 2

Applied from the line-by-line review:

  • Docs inventory claim reworded: the initrd holds installer-stack.bst plus transitive runtime deps (the wrapper itself uses modprobe, lsblk, mount, umount, sha256sum, readlink; grep arrives transitively) — "anything else fails" dropped. The Step 1a build-time check now pins every external command the wrapper calls, so the docs' "checked at build time" is true.
  • ROOT_COPYBLOCKS_SEEN=0 append fallback replaced by an explicit ERROR … has no CopyBlocks= line to repoint + exit 1, with an executed test for the missing-line and indented-key cases (indented keys are documented as not matched and hit the same error).
  • Nits: (sed|awk|tar) regex with [\s;|&)>]|$ lookahead and a whitespace-anchored comment stripper; the three literal-implementation asserts dropped; shlex.quote() on the substituted paths; session-scoped fixture; a comment on the block-slice coupling; commit body no longer says grep is declared in the stack or that every line is byte-identical.
  • Round-1 CI build (run 36259014114) passed; the new build is running.

jeefy and others added 2 commits September 26, 2026 12:55
The inst.ddi_url path in the released installer-v26.08.0 PXE initrd cannot
complete on any machine: after a successful download, verification and
decompression, bluefin-sysinstall runs `sed` to repoint CopyBlocks= in the
repart override and dies with

  /usr/bin/bluefin-sysinstall: line 91: sed: command not found

exit 127, FailureAction=poweroff. installer-stack.bst ships uutils
coreutils, Bash, curl, zstd and systemd, and grep arrives transitively (all
confirmed in the released initrd's file list); sed is a build-depend of the
installer element only and never lands in the live initrd.

Rewrite the override in plain Bash: copy /usr/lib/repart.d/20-root-a.conf
line by line, substituting only the column-0 CopyBlocks= line. Every other
line is unchanged apart from normalising a missing final newline. If the
recipe carries no CopyBlocks= line the wrapper now errors out before the
disk is touched (recipe and wrapper ship from the same repo, so that is a
repo bug, not something to paper over by appending a key). No new
dependencies. Also split the one `local x="$(...)"` so the extracted
wrapper is `shellcheck -S warning` clean.

Step 1a of the element now pins every external command the wrapper calls
(grep, curl, zstd, modprobe, mount, umount, sha256sum, readlink, ls, cp,
mkdir, sleep, systemctl next to the existing five), so a missing tool
fails the build instead of the install.

tests/unit/conftest.py gains a session-scoped `installer_wrapper` fixture
that carves the wrapper out of the heredoc, shared by the installer tests.
test_installer_contract.py asserts that no command position names sed, awk
or tar and that the build-time check pins the wrapper's tools; two tests
run the rewrite block against the shipped 20-root-a.conf (output equals the
sed result it replaced, expected path derived from ROOT_COPYBLOCKS_LINE)
and against recipes with no or an indented CopyBlocks= line (exit 1 with
the wrapper's message, nothing appended). All fail against the previous
element.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
State the wrapper's tool inventory next to the PXE flow description and add
a Common Rationalizations row: the initrd contains only installer-stack.bst
plus its transitive runtime deps, so sed, awk and tar fail with `command not
found` (exit 127) after the download. Use Bash builtins, or declare the tool
in the stack and add it to the Step 1a build-time tool check that pins every
external command bluefin-sysinstall calls.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy force-pushed the fix/installer-sed-missing branch from 90bc8fb to bb87bba Compare September 26, 2026 18:18

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy
jeefy merged commit d566ccb into projectbluefin:main Sep 26, 2026
5 checks passed
jeefy added a commit that referenced this pull request Sep 26, 2026
…254)

## Problem

On the installed image `bluefin-firstboot-credentials.service` fails on every boot that carries a `firstboot.*` credential:

```text
bluefin-firstboot-credentials.service: Failed at step EXEC spawning systemd-firstboot: No such file or directory
```

so `firstboot.locale`, `firstboot.locale-messages`, `firstboot.keymap`, `firstboot.timezone` and `firstboot.hostname` are silent no-ops — including the ones `systemd-sysinstall` itself propagates from the installer environment (locale/keymap/timezone, see the "Propagates installer environment settings" bullet in `ddi-installer.md`). Observed in QEMU on the installed `installer-v26.08.0` image with credentials staged on the ESP (`Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, …` followed by the EXEC failure). Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which delivers `firstboot.hostname` per host.

## Root cause

The unit runs `ExecStart=systemd-firstboot --force --welcome=no`, written when the OS payload was FSDK's systemd. #132 cut the payload over to Flatcar's `/usr` (`flatcar-usr.bst`), and **Flatcar builds systemd without `systemd-firstboot`**: `flatcar-container.tar.gz` for 4593.2.5 contains neither `/usr/bin/systemd-firstboot` nor `systemd-firstboot.service` (verified by listing the archive; the `ln -sf /dev/null …/systemd-firstboot.service` mask in `bluefin-server-ddi.bst` is therefore a no-op too, harmless). `installer-stack.bst` still has FSDK's systemd with `systemd-firstboot`, which is why the *installer* side works and the *target* side does not.

## Fix

Replace the binary with `/usr/libexec/bluefin-firstboot-credentials`, a bash + coreutils helper (both shipped by Flatcar), staged by `os-creds-prov.bst`. It reads the five credentials from `$CREDENTIALS_DIRECTORY`, validates them at least as strictly as systemd's validators (`hostname_is_valid`-style: labels `[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?`, no leading/trailing hyphen, ≤ 64 chars total; timezone must match `[A-Za-z0-9_+-]+(/…)*` **and** exist under `/usr/share/zoneinfo`; locale/keymap restricted to safe characters and < 128 bytes like `locale_is_valid()`/`keymap_is_valid()`; everything ASCII-only under `export LC_ALL=C`, since PID 1 exports the `LANG` this helper itself writes and `[[:alnum:]]` would otherwise widen with it; a NUL byte in a credential is rejected rather than silently dropped by `$(<file)`), and writes the same files `systemd-firstboot` would, in the same format — plus `/etc/hostname`. On `firstboot.hostname`: systemd ≤ v259 (Flatcar 4593.2.5 ships 257.9) has no such credential; systemd v260 added it as a *static* hostname honoured only when none is set yet, whereas this helper applies it unconditionally and additionally sets the live kernel hostname (systemd's own transient hostname credential remains `system.hostname`, consumed by PID 1):

| credential | file |
|---|---|
| `firstboot.locale` / `firstboot.locale-messages` | `/etc/locale.conf` (`LANG=` / `LC_MESSAGES=`) |
| `firstboot.keymap` | `/etc/vconsole.conf` (`KEYMAP=`) |
| `firstboot.timezone` | `/etc/localtime -> ../usr/share/zoneinfo/<tz>` |
| `firstboot.hostname` | `/etc/hostname` + live kernel hostname (`/proc/sys/kernel/hostname`) so the first DHCP request carries it — new relative to `main` (the old `ExecStartPost` set the kernel hostname but never persisted `/etc/hostname`); semantics differ from systemd v260's (always applied, not only when no static hostname exists) |

Any invalid value fails the run **before** anything is written (unit fails visibly, stamp file not created; control characters in the rejected value are `%q`-quoted in the log). The helper `sync`s before exiting so the `ExecStartPost` stamp cannot outlive the files on power loss. The unit's ordering, conditions, `ImportCredential=` lines and the `/etc/.bluefin-firstboot-credentials` stamp are unchanged; the old `ExecStartPost` hostname one-liner folds into the helper. Shebang is `/usr/bin/bash` so EXEC does not depend on the tmpfiles-created `/bin` symlink. `--root=DIR` exists for tests only.

Known, deliberate differences from `systemd-firstboot` (stated in the helper's header): trailing newlines in a credential are stripped rather than rejected; `LC_MESSAGES=` is written even when it equals `LANG=`; locales/keymaps are shape-checked, not checked against what is installed (Flatcar ships the full zoneinfo, so the timezone existence check is real).

### Why not ship the binary

Pulling FSDK's `systemd-firstboot` into the Flatcar `/usr` was the other option and was rejected: it links `libsystemd-shared-<ver>.so` from a different systemd build, so it would drag a second libsystemd-shared into the image and reintroduce exactly the FSDK/Flatcar userspace split #132 removed. The helper is ~100 lines with no dependencies beyond what the payload already has, and follows the precedent of #246's `files/os/libexec/bluefin-root-cred-sync`.

## Verification

- `tests/unit/test_credentials_provisioning.py` — unit no longer names `systemd-firstboot`, `ExecStart` points at the helper and precedes the stamp; element stages `files/os/libexec` and installs the helper 0755.
- `tests/unit/test_firstboot_credentials_behavior.py` (new, 26 tests) — executes the helper against a scratch `--root`: all five credentials produce the exact files above; a subset writes only its files; `--force`-style overwrite of existing `/etc/hostname` and `/etc/localtime`; interior hyphens/digits in labels accepted; twelve invalid inputs (`bad host`, `-leading`, `trailing-`, `foo-.bar`, `foo..bar`, `hôte`, `host\r`, 65-char hostname, `../../etc/passwd` and unknown timezone, shell metacharacters in locale, `..` in keymap) exit 1 with a named error and write **nothing**; a non-ASCII **keymap/locale** (`[[:alnum:]]` regexes) is rejected with `LC_ALL=C.UTF-8`/`LANG=C.UTF-8` in the environment — this test **fails when `export LC_ALL=C` is removed from the helper** (verified by mutation; the round-1 hostname variant did not, because `[A-Za-z0-9]` does not widen under `C.UTF-8`); 127-byte keymap accepted / 128 rejected; locale-only and messages-only `locale.conf`; invalid `locale-messages`; a NUL byte; the CR case logs `$'host\r'`; empty credential treated as absent; missing and nonexistent `CREDENTIALS_DIRECTORY` are errors; `bash -n`; `shellcheck -S warning` (skipped visibly, not silently passed, when shellcheck is absent — CI's `unit-tests.yml` does not install it).
- `python3 -m pytest tests/unit -q` — 371 passed, 1 xfailed (pre-existing). `test_files_payload_reachability` confirms the new `files/os/libexec/` path is staged.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- Not run: `just validate` / `just build-ddi` (no local BuildStream) and a rebuilt-image boot. The helper has only been exercised by the executed unit tests against a scratch `--root`; a first boot with `firstboot.*` credentials on a rebuilt DDI is the remaining proof and I am happy to run it against a CI artefact.

Docs: `ddi-installer.md` describes what the unit actually runs and writes (same files as `systemd-firstboot` plus the hostname handling, with the ≤ v259 / v260 distinction); `tpm2-credential-sealing.md` drops `systemd-firstboot` as a consumer of `passwd.hashed-password.root`.

No relation to the installer PR stack (#250, #251, #252, #253); no textual or compose overlap with #246 (`bluefin-root-cred-sync` is installed from a separate element).
jeefy added a commit that referenced this pull request Sep 26, 2026
…e before decompressing (#251)

## Problem

The `inst.ddi_url` path staged the download in `/run`. systemd PID 1 mounts `/run` as tmpfs with `size=20%` (**20 % of RAM** ; ≈1.6 GiB on an 8 GiB machine). The 26.08.0 DDI is 365 MiB compressed / 1.58 GiB raw, and `zstd --rm` removes the compressed copy only after a successful decompress, so the staging tmpfs must hold ~1.95 GiB at once. On the released `installer-v26.08.0` PXE initrd that fails on anything below ~10 GiB of RAM (1.95 GiB / 20 %):

```text
zstd: error 70 : Write error : cannot write block : No space left on device
ERROR: failed to decompress downloaded DDI.
```

Reproduced in QEMU (OVMF, q35/KVM) at 8 GiB with the released initrd; passes at 16 GiB. Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server.

## Fix

- Stage under `/dev/shm/installer`: PID 1 mounts `/dev/shm` without a `size=` option, so it gets the **kernel tmpfs default of 50 % of RAM**. The wrapper first checks `mountpoint -q /dev/shm` so it never stages on devtmpfs.
- Before decompressing, compare the raw size recorded in the zstd frame header (or 5× the compressed size when the header does not carry it — the shipped DDIs compress 2–4.5× — with a `WARN`) against **both** `df -k --output=avail` of the staging directory **and** `MemAvailable` from `/proc/meminfo` minus 256 MiB of headroom — in KiB, rounded up, so MiB truncation cannot wave through an image a few KiB too large — and abort with all the numbers instead of dying mid-write or being OOM-killed:
  `ERROR: not enough RAM-backed temporary space in /dev/shm/installer to decompress the DDI (need ~N MiB; tmpfs has M MiB free, MemAvailable is K MiB with 256 MiB headroom); add RAM to this machine.`
  The `MemAvailable` bound matters because the tmpfs cap is not proof of memory: the unpacked initrd is its own tmpfs, so `df` on `/dev/shm` can report space the kernel cannot back. `df` failure or unparsable output fails closed with a message.
- Drop the pre-created `/layer/run/installer` (the wrapper creates its own staging dir). `mountpoint`, `stat`, `df`, `tail` join the Step 1a build-time tool check.

Docs: `ddi-installer.md` replaces the "~8 GiB guidance" line with a *Memory requirements* section (why `/run` cannot hold the DDI, both copies must fit, the check needs `MemAvailable` after the download ≥ raw DDI + 256 MiB and the `/dev/shm` cap to hold both copies; 8 GiB tested and recommended for the 26.08.0 Flatcar-based DDI, smaller machines not boot-tested — the earlier "computed minimum ≈4 GiB" is gone because the tmpfs-only arithmetic could pass the check and still OOM), adds a Common Rationalizations row and fixes the "2 GiB+" DDI size to ≈1.6 GiB raw.

## Verification

- `tests/unit/test_installer_contract.py`:
  - `test_installer_network_ddi_stages_in_dev_shm_with_space_preflight` — `mountpoint` guard, staging path, no `/run/installer`, preflight ordered between `sha256sum --check` and `zstd -d`, `MemAvailable` read from `DDI_MEMINFO`.
  - `test_installer_space_preflight_uses_the_zstd_frame_header` — runs the extracted preflight block against a real zstd frame with a stand-in right-aligned `df` and a fake `/proc/meminfo` (the block's `DDI_MEMINFO=/proc/meminfo` is overridden): a 3 MiB + 1 B raw image passes with 3073 KiB free, fails with 3072 KiB, and the message carries `need ~4 MiB; tmpfs has 3 MiB free, MemAvailable is 65536 MiB`.
  - `test_installer_space_preflight_is_bounded_by_mem_available` — huge tmpfs, `MemAvailable` = headroom + 3073 KiB passes, + 3072 KiB fails; a meminfo without `MemAvailable` fails closed.
  - `test_installer_space_preflight_falls_back_to_five_times_compressed` — non-zstd payload, 5× rule, one-KiB boundary, `WARN` emitted.
  - All fail against `main`'s element; the `MemAvailable` test also fails against the round-1 tmpfs-only check (verified by mutation). The tests use the shared `installer_wrapper` fixture from #250.
- `python3 -m pytest tests/unit -q` — 352 passed, 1 xfailed (pre-existing).
- `bash -n` + `shellcheck -S warning` on the extracted wrapper — clean.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- Not run: `just validate` / `just build-installer` (no local BuildStream).
- **QEMU** (OVMF, q35/KVM, 8 GiB, iPXE from a Booty instance): the *combined* wrapper of this stack (sed rewrite + this `/dev/shm` staging + wait-online + `inst.creds_url`) was dropped into the released `installer-v26.08.0` initrd; decompression succeeds and the install completes where the released initrd hit `ENOSPC`. The **preflight abort path was not booted** (no machine small enough was tried) — it is covered by the executed unit tests only, and the KiB-rounding change post-dates the QEMU run.
jeefy added a commit that referenced this pull request Sep 26, 2026
…#252)

## Problem

The `inst.ddi_url` path guards its network wait with

```bash
if command -v systemd-networkd-wait-online >/dev/null 2>&1; then
  systemd-networkd-wait-online --timeout=90 || { ...; exit 1; }
fi
```

`systemd-networkd-wait-online` lives in `/usr/lib/systemd/`, not on `PATH`, so the guard never fires, the wait is silently skipped and `curl` races the DHCP lease. With the released `installer-v26.08.0` PXE initrd in QEMU (bridged network, KVM) the download fails ~11 s after boot:

```text
curl: (7) Could not connect to server
```

Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server; on a slower DHCP server it fails every time.

## Fix

Call `/usr/lib/systemd/systemd-networkd-wait-online --any --ipv4 --timeout=90` by absolute path and drop the guard. A missing binary is now a **build** failure — checked in step 1a next to the other critical tools — rather than a skipped wait at install time, so an FSDK relocation surfaces in CI, not on a PXE client.

**Why `--any`.** The installer's `80-dhcp.network` matches every `en*`/`eth*`/`ens*`/`enp*` interface. Without `--any`, `systemd-networkd-wait-online` blocks while *any* managed link is still in setup state `configuring` (`wait-online-manager.c` `manager_configured()`: `-EBUSY` makes it return false, so the wait runs to `--timeout` unless `--any`), and an unplugged second NIC sits at `no-carrier configuring` forever. Now that the wait actually runs, that would abort every PXE install on a multi-NIC server with an unplugged port after 90 s. With `--any` the wait returns once one managed link is `configured`. On a single-NIC machine both forms behave identically.

**Why `--ipv4`.** networkd marks a link `configured` as soon as *one* dynamic protocol finished (`link_check_ready()`), so on a LAN with router advertisements a SLAAC or DHCPv6 address makes the link configured — and `--any` satisfied — before the DHCPv4 lease, and `curl http://<IPv4>` would race again exactly as before. `--ipv4` (present since v244; FSDK 26.08 ships v261) requires the satisfying link to carry an IPv4 address, which with IPv4LL disabled in the installer's `.network` means the DHCPv4 lease. On IPv4-only LANs it changes nothing.

**Why the binary and not `systemd-networkd-wait-online.service`.** Calling it from the wrapper keeps the wait conditional on `inst.ddi_url`/`inst.creds_url` (offline USB installs must not wait); the unit route would need a `ConditionKernelCommandLine=`, an `ExecStart=` drop-in for the flags and `Requires=` to abort the install on failure.

## Verification

- `tests/unit/test_installer_contract.py::test_installer_network_ddi_waits_for_networkd_by_absolute_path` — no `command -v` guard, exactly one absolute-path call carrying `--any`, `--ipv4` and a `--timeout=` (matched as arguments, not as one literal), placed **inside** the network-pull block and before the `curl` download, build-time check present. Fails against `main`. Uses the shared `installer_wrapper` fixture from #250.
- `python3 -m pytest tests/unit -q` — 353 passed, 1 xfailed (pre-existing).
- `bash -n` + `shellcheck -S warning` on the extracted wrapper — clean.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- Not run: `just validate` / `just build-installer` (no local BuildStream).
- **QEMU** (OVMF, q35/KVM, 8 GiB, single virtio NIC, iPXE from a Booty instance): the *combined* wrapper of this stack (sed rewrite + `/dev/shm` staging + this wait + `inst.creds_url`) was dropped into the released `installer-v26.08.0` initrd; the wait blocks until the lease arrives and the download succeeds on first try. That run used the plain `--timeout=90` form; `--any` and `--ipv4` were added afterwards from the `wait-online-manager.c`/`networkd-link.c` reading above and have **not** been booted — on one IPv4-only NIC they are provably equivalent, the multi-NIC and dual-stack cases are not covered by a boot. The **timeout/abort path was not booted** either; it is asserted by the unit test only.
jeefy added a commit that referenced this pull request Sep 26, 2026
…st.creds_sha256) for PXE installs (#253)

**Stacked on #252** (wait-online) ← #251 (`/dev/shm` staging) ← #250 (sed rewrite) — only the last four commits (`feat`, `docs`, `test`, `docs`) are new here; the credentials pull shares the same staging directory and network wait, so the diff shrinks to those four as the fixes merge.

## Motivation

#229 made PXE installs self-contained by letting the installer pull the DDI over the LAN (`inst.ddi_url` / `inst.ddi_sha256`). Its feature request (#14 §4) left per-host configuration out of scope, naming the natural follow-up: installer-fetched systemd credentials. A PXE fleet still needs a hostname, an operator SSH key and usually a first-boot callback per machine, and the only delivery path today is an operator hand-mounting the ESP after the install. This PR closes that gap with the same shape as the DDI pull: two opt-in kernel parameters, mandatory verification, no new installer logic — the files are handed to the stock systemd credential consumers already documented in `docs/skills/tpm2-credential-sealing.md` (`firstboot.*`, `tmpfiles.extra`, `network.*`, `passwd.*`).

Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which serves `GET /creds/<mac>.tar` per host.

## Behaviour

| Parameter | Meaning |
|---|---|
| `inst.creds_url=<url>` | Download a **flat, uncompressed tar archive** of `systemd-creds encrypt`ed `*.cred` files (any `curl` scheme; the example uses plain `http://`) and place them in the installed system's ESP at `/loader/credentials/`. |
| `inst.creds_sha256=<hex>` | **Mandatory** when `inst.creds_url` is set. Verified before anything is unpacked or written. |

iPXE example (`${mac}` expanded by iPXE):

```text
systemd.unit=system-install.target console=tty0 console=ttyS0,115200 rw unattended inst.ddi_url=${base}/bluefin-server-ddi-<ver>.raw.zst inst.ddi_sha256=<sha256> inst.creds_url=${base}/creds/${mac}.tar inst.creds_sha256=<sha256>
```

Archive contract: an **uncompressed** ustar/pax/gnu tar (the wrapper checks the `ustar` magic at offset 257 and feeds tar on stdin, where GNU tar refuses compressed input instead of transparently decompressing a `.tar.zst` into an unbounded listing) of at most **64** top-level regular files named `<name>.cred`, `<name>` matching `[A-Za-z0-9][A-Za-z0-9._@~-]*` (`~` is systemd's own `systemd.unit-dropin.<unit>~<name>` credential namespace), each name at most **250 bytes**, each file at most **1 MiB**, the archive at most **16 MiB**, and no name repeated even ignoring case (the ESP is vfat, where `a.cred` and `A.cred` are the same file). Any other member (directory, `./` prefix, symlink, hardlink, device, FIFO, path, space/control character/leading dash or dot in the name, non-`.cred` name, over-long or duplicate name) rejects the whole archive with a wrapper-level message before anything is extracted. Build with `tar -cf <mac>.tar -C <dir> a.cred b.cred`.

Trust model: `inst.creds_sha256` arrives on the same unauthenticated PXE command line as the URL, so the checksum protects against a compromised or misconfigured *file server*, not against whoever controls DHCP/TFTP — they control both. The archive hardening is defence against operator error and a bad mirror, not a security boundary against a hostile netboot server; secrets belong in TPM2/host-key-sealed credentials.

## Guarantees

- **Absent → behaviourally unchanged.** Without `inst.creds_url` the wrapper behaves as before; the one shared change is that the ESP is now located by GPT partition type instead of "first vfat on the target" (design note 1). The commit message and in-script comment now say the same (round 1 said "byte-for-byte", which was wrong).
- **Fail closed on missing checksum**, same wording as the DDI check.
- **Nothing touches the disk before both downloads verify, and nothing unverified is used.** The archive is fetched right after the DDI (one shared `systemd-networkd-wait-online`), capped at 16 MiB (`curl --max-filesize` — curl ≥ 8.4 aborts a transfer of unknown length once it crosses the cap; re-checked with `stat` after the checksum so the bound does not depend on the curl version), `sha256sum --check`ed, checked for the `ustar` magic, and only then listed — via stdin, so tar cannot auto-decompress. The `tar -tv` listing is validated column by column *before extraction* — type must be a regular file (no directories, symlinks, hardlinks, devices, FIFOs), size numeric and ≤ 1 MiB, date/time columns where GNU tar puts them (`LC_ALL=C`), name matching the strict pattern (which also rejects the backslash sequences `--quoting-style=escape` emits for a smuggled newline or other control character) and ≤ 250 bytes, no duplicate or case-colliding names, ≤ 64 members — and cross-checked against the plain `tar -t` listing so a column shift (e.g. an owner name with a space) is caught. Only the validated names are extracted (`--no-same-owner --no-same-permissions`), pinned to `0644`, and re-checked as regular non-symlink files. Everything lands in `/dev/shm/installer/credentials/` (tmpfs) *before* `systemd-sysinstall` runs. A bad download aborts with the target untouched.
- **Placement happens after `systemd-sysinstall`** in the existing post-install ESP pass (`stage_fallback_bootloader` → `finalize_target_esp`): one `mount -t vfat` of the ESP identified by type GUID `c12a7328-f81f-11d2-ba4b-00a0c93ec93b` on the target disk, `mkdir -p loader/credentials`, `cp -- "$CREDS_DIR/$name"` for each validated name (never a glob over the extraction directory), `sync` — **first**, then the best-effort fallback-loader copy on the same mount, so a loader hiccup cannot abort the function with the mandatory step undone — then `umount`.
- **Fail closed after install too.** If `inst.creds_url` was given and the ESP cannot be found or mounted, the wrapper exits 1 (→ `FailureAction=poweroff`) instead of letting an unconfigured, unreachable host reboot.
- **Credentials must be `systemd-creds` encrypted.** `systemd-stub` stages ESP credentials under `/run/credentials/@encrypted`, so plaintext `*.cred` files are rejected by the consuming services (`status=243/CREDENTIALS`); null-key encrypted files (`systemd-creds --with-key=null encrypt --name=<name> - <name>.cred`) apply. The docs say so and show the commands.
- **One new tool**: GNU `tar` (`components/tar.bst`, present at the pinned `freedesktop-sdk-26.08.0`; runtime deps glibc, symlinks, utf-locale) in `installer-stack.bst`, also added to the build-time tool check together with `dd`, `rm`, `chmod`, `sync` (all confirmed in the released initrd). `curl`, `sha256sum`, `stat`, `mount`, `lsblk`, `udevadm` were already there (confirmed against the released 26.08.0 initrd's file list: uutils coreutils 0.11.0, GNU grep, curl, zstd, util-linux; no `sed`/`awk`/`tar`).

## Verification

- `tests/unit/test_installer_contract.py::test_installer_network_credentials_pull_contract` — parameter parsing, mandatory-checksum wording, fetch → verify → size cap → `ustar` magic → listing → unpack → `systemd-sysinstall` ordering, tar only ever reading the archive from stdin, the 16 MiB / 64 / 250-byte / 1 MiB caps, mode and name patterns, case-insensitive duplicate check, listing cross-check, 0644 pin and regular-file re-check, ESP lookup by type GUID on a single vfat mount with per-name copies (no glob) written before the loader, fail-closed message, `tar.bst` in the stack and `tar` in the build-time tool check.
- `tests/unit/test_installer_creds_pull_behavior.py` (new, 29 tests) — runs the wrapper (shared `installer_wrapper` fixture) under bash: `bash -n`, `shellcheck -S warning` when installed, and the fetch/verify/validate/unpack block against real tar archives with a stand-in `curl`. Accepted: a flat archive lands in tmpfs at `0644` regardless of archive modes/owners and the download is removed; 64 members incl. a 1 MiB member (boundary); a `systemd.unit-dropin.multi-user.target~booty.cred` member; a 250-byte name. Rejected without creating the credentials directory: checksum mismatch, non-`.cred` names, directories, `./` prefixes, symlinks, hardlinks, char/block devices, FIFOs, names with spaces / newlines / colons / leading dash / leading dot, an owner field with a space (column shift), a 1 MiB + 1 B member, 65 members, a 251-byte name, a repeated name, two names differing only by case, a 17 MiB archive, a zstd-compressed tar (rejected at the magic check; GNU tar on stdin is also shown to refuse it), empty and non-tar payloads. **Not executed here**: the curl-side `--max-filesize` (the stand-in curl ignores it; the `stat` re-check is executed) and `finalize_target_esp` (string assertions only) — stated in the module docstring. Tests skip unless GNU tar is present.
- `python3 -m pytest tests/unit -q` — 383 passed, 1 xfailed (pre-existing).
- `bash -n` + `shellcheck -S warning` on the extracted wrapper — clean.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- **End to end in QEMU** (OVMF, q35/KVM, 8 GiB, 20 GiB virtio disk, single NIC) with the *combined* wrapper of this stack and GNU tar dropped into the released `installer-v26.08.0` initrd, booted via iPXE from a Booty instance serving all four `inst.*` parameters: archive fetched and verified before any disk change, a 7-file `*.cred` archive accepted, `systemd-sysinstall` runs, `finalize_target_esp` stages the fallback loader and writes `/loader/credentials/*.cred` on `/dev/vda1`; on first boot the target logs `Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, tmpfiles.extra, …`, `tmpfiles.extra` is applied and SSH as the provisioned operator works.
- **Not booted**: the archive-hardening pass (size caps, column-by-column listing validation, cross-check, `--no-same-permissions`/`0644` pin, per-name copy) and the `--any` on wait-online were added *after* that QEMU run from review; the happy path is behaviourally identical for a well-formed archive (the executed unit tests exercise the exact block that ships), but the current wrapper has not been re-booted. The archive-rejection paths, the ESP-not-found/unmountable failure and the wait-online timeout are unit-tested only.
- Not run: `just validate` / `just build-installer` (no local BuildStream); the tested initrd was hand-assembled from the released one.

Two target-side observations from that run, both outside this PR: (a) null-key credentials are honoured by `ImportCredential=` consumers (tmpfiles, sysusers, …) but refused by PID 1 and generators (`system.hostname`, `systemd.extra-unit.*`: `Operation not supported`), so anything beyond those consumers needs host/TPM2-key encryption; (b) `bluefin-firstboot-credentials.service` fails on the released image with `Failed at step EXEC spawning systemd-firstboot`, so `firstboot.*` are no-ops there — fixed separately in `fix/firstboot-credentials-binary` (#254).

## Design notes / questions

1. **ESP lookup by partition type.** The fallback-loader pass previously took the first `vfat` partition on the target disk. I switched it to the ESP type GUID so the credentials cannot land on an unrelated vfat data partition on an install-alongside layout. Same partition in every layout the installer produces today; flagging it because it touches the non-PXE path.
2. **Mount reuse vs. separate pass.** I reused the single existing post-install mount rather than mounting twice, so the fallback loader and the credentials share one function. Happy to split it if you prefer the bootloader staging to stay strictly best-effort and isolated.
3. **`sha256sum` vs. `importctl`.** I mirrored the DDI path (`curl` + `sha256sum --check`). If you would rather move both network pulls to `importctl pull-raw --verify=checksum` with a `SHA256SUMS`, I can do that in a follow-up covering both.
4. **Encrypted-only enforcement.** The wrapper checks archive shape, not contents. Should it additionally check for the `systemd-creds` header and refuse plaintext up front?
6. **Caps.** 64 members / 1 MiB each / 16 MiB total are generous for encrypted credentials (a few KiB each) but arbitrary; happy to tune.
5. **`tar` as the only new dep.** GNU tar was the smallest thing that reads the archive; a `cpio` or a DDI-wrapped credentials volume felt heavier for a handful of small files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant