Skip to content

fix(installer): actually wait for the network before the DDI download - #252

Merged
jeefy merged 1 commit into
projectbluefin:mainfrom
jeefy:fix/installer-wait-online-path
Sep 26, 2026
Merged

jeefy merged 1 commit into
projectbluefin:mainfrom
jeefy:fix/installer-wait-online-path

Conversation

@jeefy

@jeefy jeefy commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

Branch: fix/installer-wait-online-path. Stacked on #251 (/dev/shm staging) ← #250 (sed rewrite) — only the last commit is new here; the diff shrinks to it once those merge. Next in the stack: #253 (inst.creds_url).

Problem

The inst.ddi_url path guards its network wait with

if command -v systemd-networkd-wait-online >/dev/null 2>&1; then
  systemd-networkd-wait-online --timeout=90 || { ...; exit 1; }
fi

systemd-networkd-wait-online lives in /usr/lib/systemd/, not on PATH, so the guard never fires, the wait is silently skipped and curl races the DHCP lease. With the released installer-v26.08.0 PXE initrd in QEMU (bridged network, KVM) the download fails ~11 s after boot:

curl: (7) Could not connect to server

Found while integrating with the jeefy/booty PXE server; on a slower DHCP server it fails every time.

Fix

Call /usr/lib/systemd/systemd-networkd-wait-online --any --ipv4 --timeout=90 by absolute path and drop the guard. A missing binary is now a build failure — checked in step 1a next to the other critical tools — rather than a skipped wait at install time, so an FSDK relocation surfaces in CI, not on a PXE client.

Why --any. The installer's 80-dhcp.network matches every en*/eth*/ens*/enp* interface. Without --any, systemd-networkd-wait-online blocks while any managed link is still in setup state configuring (wait-online-manager.c manager_configured(): -EBUSY makes it return false, so the wait runs to --timeout unless --any), and an unplugged second NIC sits at no-carrier configuring forever. Now that the wait actually runs, that would abort every PXE install on a multi-NIC server with an unplugged port after 90 s. With --any the wait returns once one managed link is configured. On a single-NIC machine both forms behave identically.

Why --ipv4. networkd marks a link configured as soon as one dynamic protocol finished (link_check_ready()), so on a LAN with router advertisements a SLAAC or DHCPv6 address makes the link configured — and --any satisfied — before the DHCPv4 lease, and curl http://<IPv4> would race again exactly as before. --ipv4 (present since v244; FSDK 26.08 ships v261) requires the satisfying link to carry an IPv4 address, which with IPv4LL disabled in the installer's .network means the DHCPv4 lease. On IPv4-only LANs it changes nothing.

Why the binary and not systemd-networkd-wait-online.service. Calling it from the wrapper keeps the wait conditional on inst.ddi_url/inst.creds_url (offline USB installs must not wait); the unit route would need a ConditionKernelCommandLine=, an ExecStart= drop-in for the flags and Requires= to abort the install on failure.

Verification

  • tests/unit/test_installer_contract.py::test_installer_network_ddi_waits_for_networkd_by_absolute_path — no command -v guard, exactly one absolute-path call carrying --any, --ipv4 and a --timeout= (matched as arguments, not as one literal), placed inside the network-pull block and before the curl download, build-time check present. Fails against main. Uses the shared installer_wrapper fixture from fix(installer): rewrite the network DDI repart override without sed #250.
  • python3 -m pytest tests/unit -q — 353 passed, 1 xfailed (pre-existing).
  • bash -n + shellcheck -S warning on the extracted wrapper — clean.
  • python3 .github/scripts/docs-checks.py + the check-* scripts — pass.
  • Not run: just validate / just build-installer (no local BuildStream).
  • QEMU (OVMF, q35/KVM, 8 GiB, single virtio NIC, iPXE from a Booty instance): the combined wrapper of this stack (sed rewrite + /dev/shm staging + this wait + inst.creds_url) was dropped into the released installer-v26.08.0 initrd; the wait blocks until the lease arrives and the download succeeds on first try. That run used the plain --timeout=90 form; --any and --ipv4 were added afterwards from the wait-online-manager.c/networkd-link.c reading above and have not been booted — on one IPv4-only NIC they are provably equivalent, the multi-NIC and dual-stack cases are not covered by a boot. The timeout/abort path was not booted either; it is asserted by the unit test only.

Review round 2

Applied from the line-by-line review:

  • --ipv4 added: with --any alone a SLAAC/DHCPv6-configured link satisfied the wait before the DHCPv4 lease on dual-stack LANs (and a second NIC on an RA-only segment could win the --any race); --ipv4 requires an IPv4 address on the satisfying link.
  • Test tightened: asserts the wait sits inside the network-pull if (an unconditional wait would stall every offline USB install), occurs exactly once, and matches the flags as arguments instead of one literal string.
  • Wording: the error names systemd-networkd-wait-online (any non-zero exit reaches it, not only the timeout); "-EBUSY is fatal" → "makes manager_configured() return false until the timeout"; one paragraph on why the binary is called from the script rather than via the unit.
  • Round-1 CI build (run 36259017711) passed; the new build is running.

@hivecommons-hive

hivecommons-hive Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

🐝 Possible duplicate

This PR changes exactly the same files as #251, which was opened first:

Evidence — identical changed-file set (4 file(s)):

  • docs/skills/ddi-installer.md
  • elements/oci/bluefin-server-installer.bst
  • tests/unit/conftest.py
  • tests/unit/test_installer_contract.py

This is a suggestion, not a verdict. Hive will not close, label, approve or merge anything here — a human decides.

It was produced by comparing changed-file sets across open PRs, which is a candidate generator: PRs that share a file are not necessarily the same work. Here the patches DIFFER, so this is the weaker tier: dependency-bump PRs that edit one shared manifest, and unrelated fixes to one busy file, both land in it. Read the diffs before acting.

@jeefy
jeefy force-pushed the fix/installer-wait-online-path branch from 6bc4390 to a473d6f Compare September 26, 2026 18:18

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at a473d6f (own commit: elements/oci/bluefin-server-installer.bst, tests/unit/test_installer_contract.py; the rest is the #250/#251 stack). The command -v → absolute-path diagnosis checks out, and the build-time pin at bluefin-server-installer.bst:85-88 plus the placement test cover the regression. Two low, non-blocking notes:

correctness

  • bluefin-server-installer.bst:281 — --ipv4 makes the wait fail closed on an IPv6-only segment: with no DHCPv4 server the call times out after 90 s and the install aborts, even where curl to an IPv6/AAAA-only inst.ddi_url would have succeeded. The body only states "on IPv4-only LANs it changes nothing"; if IPv6-only netboot is intentionally unsupported, that is fine, but it is now enforced.

docs-currency

  • docs/skills/ddi-installer.md:184 still reads "brings up DHCP via systemd-networkd-wait-online" (accurate), but the new operator-visible contract — abort after 90 s without a DHCPv4 lease on any matched NIC, unplugged extra NICs tolerated via --any, IPv4 lease required — is stated only in script comments (bluefin-server-installer.bst:269-280). One sentence in the network-pull section would save the next PXE debugger a trip into the heredoc.

No findings from: security, intent-alignment, style.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy
jeefy force-pushed the fix/installer-wait-online-path branch from a473d6f to 4f255a3 Compare September 26, 2026 19:36
jeefy added a commit that referenced this pull request Sep 26, 2026
…254)

## Problem

On the installed image `bluefin-firstboot-credentials.service` fails on every boot that carries a `firstboot.*` credential:

```text
bluefin-firstboot-credentials.service: Failed at step EXEC spawning systemd-firstboot: No such file or directory
```

so `firstboot.locale`, `firstboot.locale-messages`, `firstboot.keymap`, `firstboot.timezone` and `firstboot.hostname` are silent no-ops — including the ones `systemd-sysinstall` itself propagates from the installer environment (locale/keymap/timezone, see the "Propagates installer environment settings" bullet in `ddi-installer.md`). Observed in QEMU on the installed `installer-v26.08.0` image with credentials staged on the ESP (`Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, …` followed by the EXEC failure). Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which delivers `firstboot.hostname` per host.

## Root cause

The unit runs `ExecStart=systemd-firstboot --force --welcome=no`, written when the OS payload was FSDK's systemd. #132 cut the payload over to Flatcar's `/usr` (`flatcar-usr.bst`), and **Flatcar builds systemd without `systemd-firstboot`**: `flatcar-container.tar.gz` for 4593.2.5 contains neither `/usr/bin/systemd-firstboot` nor `systemd-firstboot.service` (verified by listing the archive; the `ln -sf /dev/null …/systemd-firstboot.service` mask in `bluefin-server-ddi.bst` is therefore a no-op too, harmless). `installer-stack.bst` still has FSDK's systemd with `systemd-firstboot`, which is why the *installer* side works and the *target* side does not.

## Fix

Replace the binary with `/usr/libexec/bluefin-firstboot-credentials`, a bash + coreutils helper (both shipped by Flatcar), staged by `os-creds-prov.bst`. It reads the five credentials from `$CREDENTIALS_DIRECTORY`, validates them at least as strictly as systemd's validators (`hostname_is_valid`-style: labels `[A-Za-z0-9]([A-Za-z0-9-]*[A-Za-z0-9])?`, no leading/trailing hyphen, ≤ 64 chars total; timezone must match `[A-Za-z0-9_+-]+(/…)*` **and** exist under `/usr/share/zoneinfo`; locale/keymap restricted to safe characters and < 128 bytes like `locale_is_valid()`/`keymap_is_valid()`; everything ASCII-only under `export LC_ALL=C`, since PID 1 exports the `LANG` this helper itself writes and `[[:alnum:]]` would otherwise widen with it; a NUL byte in a credential is rejected rather than silently dropped by `$(<file)`), and writes the same files `systemd-firstboot` would, in the same format — plus `/etc/hostname`. On `firstboot.hostname`: systemd ≤ v259 (Flatcar 4593.2.5 ships 257.9) has no such credential; systemd v260 added it as a *static* hostname honoured only when none is set yet, whereas this helper applies it unconditionally and additionally sets the live kernel hostname (systemd's own transient hostname credential remains `system.hostname`, consumed by PID 1):

| credential | file |
|---|---|
| `firstboot.locale` / `firstboot.locale-messages` | `/etc/locale.conf` (`LANG=` / `LC_MESSAGES=`) |
| `firstboot.keymap` | `/etc/vconsole.conf` (`KEYMAP=`) |
| `firstboot.timezone` | `/etc/localtime -> ../usr/share/zoneinfo/<tz>` |
| `firstboot.hostname` | `/etc/hostname` + live kernel hostname (`/proc/sys/kernel/hostname`) so the first DHCP request carries it — new relative to `main` (the old `ExecStartPost` set the kernel hostname but never persisted `/etc/hostname`); semantics differ from systemd v260's (always applied, not only when no static hostname exists) |

Any invalid value fails the run **before** anything is written (unit fails visibly, stamp file not created; control characters in the rejected value are `%q`-quoted in the log). The helper `sync`s before exiting so the `ExecStartPost` stamp cannot outlive the files on power loss. The unit's ordering, conditions, `ImportCredential=` lines and the `/etc/.bluefin-firstboot-credentials` stamp are unchanged; the old `ExecStartPost` hostname one-liner folds into the helper. Shebang is `/usr/bin/bash` so EXEC does not depend on the tmpfiles-created `/bin` symlink. `--root=DIR` exists for tests only.

Known, deliberate differences from `systemd-firstboot` (stated in the helper's header): trailing newlines in a credential are stripped rather than rejected; `LC_MESSAGES=` is written even when it equals `LANG=`; locales/keymaps are shape-checked, not checked against what is installed (Flatcar ships the full zoneinfo, so the timezone existence check is real).

### Why not ship the binary

Pulling FSDK's `systemd-firstboot` into the Flatcar `/usr` was the other option and was rejected: it links `libsystemd-shared-<ver>.so` from a different systemd build, so it would drag a second libsystemd-shared into the image and reintroduce exactly the FSDK/Flatcar userspace split #132 removed. The helper is ~100 lines with no dependencies beyond what the payload already has, and follows the precedent of #246's `files/os/libexec/bluefin-root-cred-sync`.

## Verification

- `tests/unit/test_credentials_provisioning.py` — unit no longer names `systemd-firstboot`, `ExecStart` points at the helper and precedes the stamp; element stages `files/os/libexec` and installs the helper 0755.
- `tests/unit/test_firstboot_credentials_behavior.py` (new, 26 tests) — executes the helper against a scratch `--root`: all five credentials produce the exact files above; a subset writes only its files; `--force`-style overwrite of existing `/etc/hostname` and `/etc/localtime`; interior hyphens/digits in labels accepted; twelve invalid inputs (`bad host`, `-leading`, `trailing-`, `foo-.bar`, `foo..bar`, `hôte`, `host\r`, 65-char hostname, `../../etc/passwd` and unknown timezone, shell metacharacters in locale, `..` in keymap) exit 1 with a named error and write **nothing**; a non-ASCII **keymap/locale** (`[[:alnum:]]` regexes) is rejected with `LC_ALL=C.UTF-8`/`LANG=C.UTF-8` in the environment — this test **fails when `export LC_ALL=C` is removed from the helper** (verified by mutation; the round-1 hostname variant did not, because `[A-Za-z0-9]` does not widen under `C.UTF-8`); 127-byte keymap accepted / 128 rejected; locale-only and messages-only `locale.conf`; invalid `locale-messages`; a NUL byte; the CR case logs `$'host\r'`; empty credential treated as absent; missing and nonexistent `CREDENTIALS_DIRECTORY` are errors; `bash -n`; `shellcheck -S warning` (skipped visibly, not silently passed, when shellcheck is absent — CI's `unit-tests.yml` does not install it).
- `python3 -m pytest tests/unit -q` — 371 passed, 1 xfailed (pre-existing). `test_files_payload_reachability` confirms the new `files/os/libexec/` path is staged.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- Not run: `just validate` / `just build-ddi` (no local BuildStream) and a rebuilt-image boot. The helper has only been exercised by the executed unit tests against a scratch `--root`; a first boot with `firstboot.*` credentials on a rebuilt DDI is the remaining proof and I am happy to run it against a CI artefact.

Docs: `ddi-installer.md` describes what the unit actually runs and writes (same files as `systemd-firstboot` plus the hostname handling, with the ≤ v259 / v260 distinction); `tpm2-credential-sealing.md` drops `systemd-firstboot` as a consumer of `passwd.hashed-password.root`.

No relation to the installer PR stack (#250, #251, #252, #253); no textual or compose overlap with #246 (`bluefin-root-cred-sync` is installed from a separate element).

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read elements/oci/bluefin-server-installer.bst at 4f255a3 in full. The one new commit here looks correct to me.

  • systemd-networkd.service is pulled in via system-install.target.wants (bst:179-180), and 80-dhcp.network is written with DHCP=yes on en* eth* ens* enp* (bst:545-546), so --any --ipv4 is the right shape: a lease on any one matched NIC satisfies the wait, and an RA-only dual-stack link cannot satisfy it early.
  • The old command -v guard could indeed never fire — nothing in the element adds /usr/lib/systemd to PATH, so the absolute-path call plus the new step 1a build-time existence check (bst:85-88) is the correct fix.
  • The wait stays inside the inst.ddi_url branch, so the embedded USB path never blocks on a network.

This is a declared stack on #251 (c021ec4, 63f40d5 are identical in both); the only new commit is 4f255a3. Merge order: #250 → #251 → this.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

🐝 Possible duplicate

This PR changes exactly the same files as #251, which was opened first:

Evidence — identical changed-file set (3 file(s)):

  • docs/skills/ddi-installer.md
  • elements/oci/bluefin-server-installer.bst
  • tests/unit/test_installer_contract.py

This is a suggestion, not a verdict. Hive will not close, label, approve or merge anything here — a human decides.

It was produced by comparing changed-file sets across open PRs, which is a candidate generator: PRs that share a file are not necessarily the same work. Here the patches DIFFER, so this is the weaker tier: dependency-bump PRs that edit one shared manifest, and unrelated fixes to one busy file, both land in it. Read the diffs before acting.

systemd-networkd-wait-online lives in /usr/lib/systemd and is not on PATH
in the installer initrd, so the `command -v` guard around it never fired,
the wait was silently skipped and curl raced the DHCP lease:

  curl: (7) Could not connect to server

11 s after boot on a bridged network, with the released installer-v26.08.0
PXE initrd. Call the binary by its absolute path and drop the guard; a
missing binary is now a build failure (checked next to the other critical
tools in step 1a) rather than a skipped wait at install time. Calling the
binary from the wrapper keeps the wait conditional on inst.ddi_url (offline
USB installs must not wait); the systemd-networkd-wait-online.service route
would need a ConditionKernelCommandLine= plus an ExecStart= drop-in for the
flags and Requires= to abort the install on failure.

Pass --any --ipv4. The installer's 80-dhcp.network matches every
en*/eth*/ens*/enp* interface, and wait-online without --any keeps waiting
while any managed link is still in setup state "configuring"
(wait-online-manager.c manager_configured(): -EBUSY makes it return false,
so the wait runs to --timeout unless --any). An unplugged second NIC sits
at "no-carrier configuring" forever, so now that the wait really runs it
would abort every PXE install on such a machine after 90 s. With --any the
wait returns once one managed link is "configured". That alone is not
enough: networkd marks a link configured as soon as one dynamic protocol
finished (link_check_ready()), so on a LAN with router advertisements a
SLAAC or DHCPv6 address makes the link configured before the DHCPv4 lease
and curl http://<IPv4> would race again; --ipv4 requires the satisfying
link to carry an IPv4 address, which with IPv4LL disabled means the DHCPv4
lease. The error text names wait-online, since any non-zero exit (not only
the timeout) reaches it.

test_installer_contract.py asserts the wrapper has no `command -v` guard,
calls the absolute path exactly once with --any, --ipv4 and a --timeout=,
inside the inst.ddi_url block and before the curl download, and that the
build-time check is present.

Assisted-by: Claude Fable 5.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy force-pushed the fix/installer-wait-online-path branch from 4f255a3 to c5b372a Compare September 26, 2026 20:26
@jeefy
jeefy merged commit c288989 into projectbluefin:main Sep 26, 2026
4 checks passed
jeefy added a commit that referenced this pull request Sep 26, 2026
…st.creds_sha256) for PXE installs (#253)

**Stacked on #252** (wait-online) ← #251 (`/dev/shm` staging) ← #250 (sed rewrite) — only the last four commits (`feat`, `docs`, `test`, `docs`) are new here; the credentials pull shares the same staging directory and network wait, so the diff shrinks to those four as the fixes merge.

## Motivation

#229 made PXE installs self-contained by letting the installer pull the DDI over the LAN (`inst.ddi_url` / `inst.ddi_sha256`). Its feature request (#14 §4) left per-host configuration out of scope, naming the natural follow-up: installer-fetched systemd credentials. A PXE fleet still needs a hostname, an operator SSH key and usually a first-boot callback per machine, and the only delivery path today is an operator hand-mounting the ESP after the install. This PR closes that gap with the same shape as the DDI pull: two opt-in kernel parameters, mandatory verification, no new installer logic — the files are handed to the stock systemd credential consumers already documented in `docs/skills/tpm2-credential-sealing.md` (`firstboot.*`, `tmpfiles.extra`, `network.*`, `passwd.*`).

Found while integrating with the [jeefy/booty](https://github.com/jeefy/booty) PXE server, which serves `GET /creds/<mac>.tar` per host.

## Behaviour

| Parameter | Meaning |
|---|---|
| `inst.creds_url=<url>` | Download a **flat, uncompressed tar archive** of `systemd-creds encrypt`ed `*.cred` files (any `curl` scheme; the example uses plain `http://`) and place them in the installed system's ESP at `/loader/credentials/`. |
| `inst.creds_sha256=<hex>` | **Mandatory** when `inst.creds_url` is set. Verified before anything is unpacked or written. |

iPXE example (`${mac}` expanded by iPXE):

```text
systemd.unit=system-install.target console=tty0 console=ttyS0,115200 rw unattended inst.ddi_url=${base}/bluefin-server-ddi-<ver>.raw.zst inst.ddi_sha256=<sha256> inst.creds_url=${base}/creds/${mac}.tar inst.creds_sha256=<sha256>
```

Archive contract: an **uncompressed** ustar/pax/gnu tar (the wrapper checks the `ustar` magic at offset 257 and feeds tar on stdin, where GNU tar refuses compressed input instead of transparently decompressing a `.tar.zst` into an unbounded listing) of at most **64** top-level regular files named `<name>.cred`, `<name>` matching `[A-Za-z0-9][A-Za-z0-9._@~-]*` (`~` is systemd's own `systemd.unit-dropin.<unit>~<name>` credential namespace), each name at most **250 bytes**, each file at most **1 MiB**, the archive at most **16 MiB**, and no name repeated even ignoring case (the ESP is vfat, where `a.cred` and `A.cred` are the same file). Any other member (directory, `./` prefix, symlink, hardlink, device, FIFO, path, space/control character/leading dash or dot in the name, non-`.cred` name, over-long or duplicate name) rejects the whole archive with a wrapper-level message before anything is extracted. Build with `tar -cf <mac>.tar -C <dir> a.cred b.cred`.

Trust model: `inst.creds_sha256` arrives on the same unauthenticated PXE command line as the URL, so the checksum protects against a compromised or misconfigured *file server*, not against whoever controls DHCP/TFTP — they control both. The archive hardening is defence against operator error and a bad mirror, not a security boundary against a hostile netboot server; secrets belong in TPM2/host-key-sealed credentials.

## Guarantees

- **Absent → behaviourally unchanged.** Without `inst.creds_url` the wrapper behaves as before; the one shared change is that the ESP is now located by GPT partition type instead of "first vfat on the target" (design note 1). The commit message and in-script comment now say the same (round 1 said "byte-for-byte", which was wrong).
- **Fail closed on missing checksum**, same wording as the DDI check.
- **Nothing touches the disk before both downloads verify, and nothing unverified is used.** The archive is fetched right after the DDI (one shared `systemd-networkd-wait-online`), capped at 16 MiB (`curl --max-filesize` — curl ≥ 8.4 aborts a transfer of unknown length once it crosses the cap; re-checked with `stat` after the checksum so the bound does not depend on the curl version), `sha256sum --check`ed, checked for the `ustar` magic, and only then listed — via stdin, so tar cannot auto-decompress. The `tar -tv` listing is validated column by column *before extraction* — type must be a regular file (no directories, symlinks, hardlinks, devices, FIFOs), size numeric and ≤ 1 MiB, date/time columns where GNU tar puts them (`LC_ALL=C`), name matching the strict pattern (which also rejects the backslash sequences `--quoting-style=escape` emits for a smuggled newline or other control character) and ≤ 250 bytes, no duplicate or case-colliding names, ≤ 64 members — and cross-checked against the plain `tar -t` listing so a column shift (e.g. an owner name with a space) is caught. Only the validated names are extracted (`--no-same-owner --no-same-permissions`), pinned to `0644`, and re-checked as regular non-symlink files. Everything lands in `/dev/shm/installer/credentials/` (tmpfs) *before* `systemd-sysinstall` runs. A bad download aborts with the target untouched.
- **Placement happens after `systemd-sysinstall`** in the existing post-install ESP pass (`stage_fallback_bootloader` → `finalize_target_esp`): one `mount -t vfat` of the ESP identified by type GUID `c12a7328-f81f-11d2-ba4b-00a0c93ec93b` on the target disk, `mkdir -p loader/credentials`, `cp -- "$CREDS_DIR/$name"` for each validated name (never a glob over the extraction directory), `sync` — **first**, then the best-effort fallback-loader copy on the same mount, so a loader hiccup cannot abort the function with the mandatory step undone — then `umount`.
- **Fail closed after install too.** If `inst.creds_url` was given and the ESP cannot be found or mounted, the wrapper exits 1 (→ `FailureAction=poweroff`) instead of letting an unconfigured, unreachable host reboot.
- **Credentials must be `systemd-creds` encrypted.** `systemd-stub` stages ESP credentials under `/run/credentials/@encrypted`, so plaintext `*.cred` files are rejected by the consuming services (`status=243/CREDENTIALS`); null-key encrypted files (`systemd-creds --with-key=null encrypt --name=<name> - <name>.cred`) apply. The docs say so and show the commands.
- **One new tool**: GNU `tar` (`components/tar.bst`, present at the pinned `freedesktop-sdk-26.08.0`; runtime deps glibc, symlinks, utf-locale) in `installer-stack.bst`, also added to the build-time tool check together with `dd`, `rm`, `chmod`, `sync` (all confirmed in the released initrd). `curl`, `sha256sum`, `stat`, `mount`, `lsblk`, `udevadm` were already there (confirmed against the released 26.08.0 initrd's file list: uutils coreutils 0.11.0, GNU grep, curl, zstd, util-linux; no `sed`/`awk`/`tar`).

## Verification

- `tests/unit/test_installer_contract.py::test_installer_network_credentials_pull_contract` — parameter parsing, mandatory-checksum wording, fetch → verify → size cap → `ustar` magic → listing → unpack → `systemd-sysinstall` ordering, tar only ever reading the archive from stdin, the 16 MiB / 64 / 250-byte / 1 MiB caps, mode and name patterns, case-insensitive duplicate check, listing cross-check, 0644 pin and regular-file re-check, ESP lookup by type GUID on a single vfat mount with per-name copies (no glob) written before the loader, fail-closed message, `tar.bst` in the stack and `tar` in the build-time tool check.
- `tests/unit/test_installer_creds_pull_behavior.py` (new, 29 tests) — runs the wrapper (shared `installer_wrapper` fixture) under bash: `bash -n`, `shellcheck -S warning` when installed, and the fetch/verify/validate/unpack block against real tar archives with a stand-in `curl`. Accepted: a flat archive lands in tmpfs at `0644` regardless of archive modes/owners and the download is removed; 64 members incl. a 1 MiB member (boundary); a `systemd.unit-dropin.multi-user.target~booty.cred` member; a 250-byte name. Rejected without creating the credentials directory: checksum mismatch, non-`.cred` names, directories, `./` prefixes, symlinks, hardlinks, char/block devices, FIFOs, names with spaces / newlines / colons / leading dash / leading dot, an owner field with a space (column shift), a 1 MiB + 1 B member, 65 members, a 251-byte name, a repeated name, two names differing only by case, a 17 MiB archive, a zstd-compressed tar (rejected at the magic check; GNU tar on stdin is also shown to refuse it), empty and non-tar payloads. **Not executed here**: the curl-side `--max-filesize` (the stand-in curl ignores it; the `stat` re-check is executed) and `finalize_target_esp` (string assertions only) — stated in the module docstring. Tests skip unless GNU tar is present.
- `python3 -m pytest tests/unit -q` — 383 passed, 1 xfailed (pre-existing).
- `bash -n` + `shellcheck -S warning` on the extracted wrapper — clean.
- `python3 .github/scripts/docs-checks.py` + the `check-*` scripts — pass.
- **End to end in QEMU** (OVMF, q35/KVM, 8 GiB, 20 GiB virtio disk, single NIC) with the *combined* wrapper of this stack and GNU tar dropped into the released `installer-v26.08.0` initrd, booted via iPXE from a Booty instance serving all four `inst.*` parameters: archive fetched and verified before any disk change, a 7-file `*.cred` archive accepted, `systemd-sysinstall` runs, `finalize_target_esp` stages the fallback loader and writes `/loader/credentials/*.cred` on `/dev/vda1`; on first boot the target logs `Received untrusted credentials: firstboot.hostname, firstboot.locale, firstboot.timezone, tmpfiles.extra, …`, `tmpfiles.extra` is applied and SSH as the provisioned operator works.
- **Not booted**: the archive-hardening pass (size caps, column-by-column listing validation, cross-check, `--no-same-permissions`/`0644` pin, per-name copy) and the `--any` on wait-online were added *after* that QEMU run from review; the happy path is behaviourally identical for a well-formed archive (the executed unit tests exercise the exact block that ships), but the current wrapper has not been re-booted. The archive-rejection paths, the ESP-not-found/unmountable failure and the wait-online timeout are unit-tested only.
- Not run: `just validate` / `just build-installer` (no local BuildStream); the tested initrd was hand-assembled from the released one.

Two target-side observations from that run, both outside this PR: (a) null-key credentials are honoured by `ImportCredential=` consumers (tmpfiles, sysusers, …) but refused by PID 1 and generators (`system.hostname`, `systemd.extra-unit.*`: `Operation not supported`), so anything beyond those consumers needs host/TPM2-key encryption; (b) `bluefin-firstboot-credentials.service` fails on the released image with `Failed at step EXEC spawning systemd-firstboot`, so `firstboot.*` are no-ops there — fixed separately in `fix/firstboot-credentials-binary` (#254).

## Design notes / questions

1. **ESP lookup by partition type.** The fallback-loader pass previously took the first `vfat` partition on the target disk. I switched it to the ESP type GUID so the credentials cannot land on an unrelated vfat data partition on an install-alongside layout. Same partition in every layout the installer produces today; flagging it because it touches the non-PXE path.
2. **Mount reuse vs. separate pass.** I reused the single existing post-install mount rather than mounting twice, so the fallback loader and the credentials share one function. Happy to split it if you prefer the bootloader staging to stay strictly best-effort and isolated.
3. **`sha256sum` vs. `importctl`.** I mirrored the DDI path (`curl` + `sha256sum --check`). If you would rather move both network pulls to `importctl pull-raw --verify=checksum` with a `SHA256SUMS`, I can do that in a follow-up covering both.
4. **Encrypted-only enforcement.** The wrapper checks archive shape, not contents. Should it additionally check for the `systemd-creds` header and refuse plaintext up front?
6. **Caps.** 64 members / 1 MiB each / 16 MiB total are generous for encrypted credentials (a few KiB each) but arbitrary; happy to tune.
5. **`tar` as the only new dep.** GNU tar was the smallest thing that reads the archive; a `cpio` or a DDI-wrapped credentials volume felt heavier for a handful of small files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant