Skip to content

Feature Request: PXE/netboot install support (standalone PXE artifacts + network DDI pull) #14

Closed

Description

@jeefy

Summary

Bluefin Server is currently installable only from physical/virtual media containing the full installer disk image (bluefin-server-installer-<ver>.raw.zst), where the OS DDI payload is embedded on the bluefin-installer-data partition. This makes it impossible to install over PXE/netboot, which is the standard provisioning path for homelabs and bare-metal fleets (the audience that a distroless, k3s-ready server image serves especially well).

This issue proposes two additive, backwards-compatible changes:

  1. Publish standalone PXE boot artifacts (kernel + installer initrd) as release assets, alongside the existing installer/DDI/UKI artifacts.
  2. Teach the installer to fetch the DDI over the network when directed to by kernel cmdline parameters, falling back to the existing embedded-partition behavior when absent.

With these in place, any PXE server (I'm building this against jeefy/booty, but it applies equally to netboot.xyz, Matchbox, dnsmasq+iPXE, etc.) can install Bluefin Server unattended over the LAN.

Motivation

  • Fleet/homelab provisioning: PXE is how bare-metal Kubernetes nodes get imaged. Bluefin Server + k3s sysext is a natural fit for exactly this workflow, but today each machine needs hand-prepared USB/virtual media.
  • No hardware media dance: Machines without easy USB access (rack servers via BMC, thin clients, mini-PC fleets) install straight from the network.
  • Everything needed already exists in the build — the kernel and initrd are built today, they're just only consumed by ukify and never exported. The DDI is already a standalone release asset. The gap is small.

Current State (as of bce0067d)

Boot artifacts are baked into the UKI only

In elements/oci/bluefin-server-installer.bst (~L252-257), the installer UKI is assembled from a kernel and cpio initrd that are never exported separately:

ukify build \
  --linux="/layer/boot/vmlinuz" \
  --initrd=/installer.cpio.gz \
  --cmdline="systemd.unit=system-install.target console=tty0 console=ttyS0,115200 rw unattended" \
  --output=/layer/boot/efi/EFI/BOOT/BOOTX64.EFI

PXE (via iPXE/GRUB) needs kernel + initrd as separate fetchable files with a caller-controlled cmdline. Chainloading the published .efi UKI directly is not sufficient because its cmdline is immutable — there is no way to inject a DDI URL (see below), and measured-boot-friendly UKIs are the point, so cmdline injection into the UKI would be the wrong direction anyway.

The installer cannot fetch the DDI over the network

The bluefin-sysinstall wrapper (installer.bst, ~L126-178) reads /proc/cmdline only to detect the unattended flag, and sources the DDI exclusively from the bluefin-installer-data GPT partition:

INSTALLER_PART="$(readlink -f /dev/disk/by-partlabel/bluefin-installer-data 2>/dev/null || true)"

Under PXE there is no installer media, therefore no bluefin-installer-data partition, and the install cannot proceed.

Proposal

Part 1: Publish standalone PXE artifacts

Export the two components that already exist at UKI-build time as release assets:

New asset Source in build Purpose
bluefin-server-pxe-vmlinuz-<ver> /layer/boot/vmlinuz PXE kernel
bluefin-server-pxe-initrd-<ver>.cpio.gz /installer.cpio.gz PXE initrd

Changes:

  • elements/oci/bluefin-server-installer.bst: copy both files into the export area alongside the UKI.
  • Justfile: new export-pxe recipe in the installer group.
  • .github/workflows/build.yml: upload both to the GitHub Release and include them in the (already GPG-signed) SHA256SUMS — consumers get artifact verification for free.

No new build work: these are byproducts of the existing installer build. The DDI (bluefin-server-ddi-<ver>.raw.zst) is already published, so no change is needed there.

Part 2: Network DDI pull in the installer

Extend bluefin-sysinstall (or better, the systemd-native machinery around it — see Design Discussion) to honor new kernel cmdline parameters:

Parameter Required Behavior
inst.ddi_url=<http(s) url> for netboot Download the (zstd-compressed) DDI instead of reading the embedded partition
inst.ddi_sha256=<hex> recommended Verify the downloaded DDI before install; abort on mismatch
inst.target_disk=/dev/... optional Explicit target disk, overriding the current first-writable-disk auto-detection

Behavior:

  1. If inst.ddi_url is absent → exactly today's behavior (embedded partition). Zero change for existing media users; offline-first design is preserved.
  2. If present → bring up networking (DHCP), fetch the DDI, verify SHA256 if provided, then feed it into the same systemd-sysinstall/systemd-repart flow the embedded path uses.
  3. Download should stream-decompress (zstd -d) rather than staging both compressed and decompressed copies in tmpfs, to keep RAM requirements sane. Minimum-RAM guidance should be documented either way.
  4. On any fetch/verification failure: fail loudly to console, do not touch the target disk.

The installer initrd would need networking + fetch tooling present in installer-stack.bst if it isn't already (systemd-networkd DHCP; fetch via importctl/systemd-pull or curl — see below).

Part 3: Documentation

  • New docs/skills/pxe-netboot.md: boot flow, cmdline parameter reference, example iPXE stanza, RAM requirements, troubleshooting (serial console, DHCP).
  • Update docs/skills/ddi-installer.md with a PXE section and cross-link.
  • Add PXE/netboot to the AGENTS.md skill routing table.

Design Discussion (maintainer input wanted)

1. Fetch mechanism: importctl / systemd-pull vs. curl.
AGENTS.md mandates the installer stay systemd-sysinstall-native with no custom scripting sprawl. systemd-importd's pull machinery (importctl pull-raw / systemd-pull) natively handles HTTP(S) download, zstd decompression, and SHA256SUMS+GPG verification — which matches the project's existing signed-manifest release format exactly. It seems like the most philosophically aligned option, at the cost of shipping importd in the installer initrd. Alternatively a small curl+zstd path in the existing bluefin-sysinstall wrapper is lighter but less "systemd-native." Which direction would you accept?

2. Cmdline namespace. inst.* follows Anaconda convention and is what PXE users will guess, but this project isn't Anaconda. Happy to use bluefin.install.* or sysinstall.* if preferred — bikeshed away.

3. UKI/measured-boot posture. Separate kernel+initrd PXE artifacts bypass the UKI's single-signed-object property. For a v1 targeting lab/fleet provisioning this seems acceptable (the installed system still boots its signed UKI), but if there's a desire to keep netboot within the UKI model long-term, UEFI HTTP boot of a netboot-variant UKI (with inst.ddi_url discovery via SMBIOS/DHCP instead of cmdline) could be a follow-up. I'd suggest not blocking v1 on it.

4. Per-host configuration (hostname, k3s join tokens, etc.) is explicitly out of scope here. A natural follow-up would be installer-fetched systemd credentials (inst.creds_url= or SMBIOS Type 11 OEM strings), which composes cleanly with this proposal.

Resulting PXE flow (for illustration)

#!ipxe
set base http://pxe-server:8080/data
kernel ${base}/bluefin-server-pxe-vmlinuz systemd.unit=system-install.target console=tty0 console=ttyS0,115200 rw unattended inst.ddi_url=${base}/bluefin-server-ddi.raw.zst inst.ddi_sha256=<sha256>
initrd ${base}/bluefin-server-pxe-initrd.cpio.gz
boot

The PXE server mirrors the three release assets (verified against the signed SHA256SUMS) and serves them over the LAN; the installer fetches the DDI locally, verifies, installs via the unchanged systemd-repart/bootctl path, and reboots into the installed OS.

Compatibility & Constraints

  • Fully backwards compatible: no cmdline params → identical behavior to today. Existing installer media unaffected.
  • Offline-first preserved: the embedded-DDI path remains the default; network pull is strictly opt-in.
  • UEFI-only: unchanged — PXE clients boot via ipxe.efi/UEFI netboot, consistent with the systemd-boot/UKI design.
  • Baseline x86_64: unchanged; artifacts are the same binaries already built.
  • Distroless target OS: untouched — all changes are installer-side.

Activity

  1. castrojo commented on Aug 15, 2026

    @castrojo
    Contributor

    Triage & scoping — feasibility, architecture, and phased plan

    Read this against current main (bce0067). Verified every claim below against the repo tree and current upstream systemd sources (systemd main, FSDK freedesktop-sdk-25.08.14).

    Feasibility verdict

    Feasible within the hard constraints, with one correction to the issue's premises: the installer already has networking. elements/oci/bluefin-server-installer.bst already enables systemd-networkd + systemd-resolved in system-install.target.wants and ships a catch-all DHCP config (80-dhcp.network for en* eth* ens* enp*). ca-certificates is already in elements/installer/installer-stack.bst, so HTTPS works too. The genuine gaps are smaller than the issue assumes:

    1. kernel + initrd are never exported (confirmed — Step 7 of the installer element only exports the .raw.zst and the target UKI);
    2. nothing in the live env fetches the DDI;
    3. the fetch needs a verifier (details in Security — this is the one real dependency add).

    No changes to systemd-sysinstall itself are needed: it reads repart definitions from /usr/lib/repart.sysinstall.d/ when populated (else /usr/lib/repart.d/), and --definitions= (v261) overrides that. That is the native seam for a network-sourced DDI: fetch the image, drop a repart definition whose CopyBlocks= points at the downloaded file, invoke systemd-sysinstall unchanged. The wrapper script (bluefin-sysinstall) stays a thin pre-flight stage; all partitioning/copy/bootctl logic remains upstream's.

    Proposed netboot chain

    Firmware PXE/HTTP-boot → iPXE (UEFI)
       ├─ kernel ${base}/bluefin-server-pxe-vmlinuz-<ver>  + caller cmdline (inst.ddi_url=…)
       └─ initrd ${base}/bluefin-server-pxe-initrd-<ver>.cpio.gz
            │
            ▼  installer initrd (existing cpio-native rootfs, /init → systemd)
       system-install.target → systemd-networkd DHCP (already wired)
            │
            ▼  bluefin-sysinstall wrapper (extended)
       if inst.ddi_url present:
         systemd-pull raw --direct --verify=signature <url> -  → /run/installer/ddi.raw (tmpfs, streamed decompress)
         write /run/repart.netboot.d/20-root-a.conf: CopyBlocks=/run/installer/ddi.raw
         exec systemd-sysinstall --definitions=/run/repart.netboot.d … (unchanged flow)
       else: exactly today's embedded-partition path
            │
            ▼
       systemd-sysinstall → systemd-repart (10-esp/20-root-a/30-var) → bootctl link/install → poweroff
    

    Notes on the chain:

    • iPXE with kernel/initrd is the right v1 vehicle precisely because the UKI cmdline is immutable, as the issue says. Chainloading the installer UKI (chain http://…/BOOTX64.EFI) also works with iPXE/UEFI HTTP Boot, but only with the baked cmdline — fine for a pure "netboot the USB media" scenario, useless for injecting a DDI URL. Both artifacts can ship; they serve different PXE servers.
    • Zero-build-change path worth validating: per systemd's docs/ROOTFS_DISCOVERY.md, UEFI firmware HTTP(S) boot downloads a disk image into a RAM disk and boots its ESP — meaning today's bluefin-server-installer-*.raw.zst may already be firmware-HTTP-bootable on supporting hardware, at the cost of the whole image residing in RAM. Lab test before documenting.
    • Later UKI-purist path: systemd-boot Type ci: delegate server builds to lab and add daily release dispatch #1 entries can reference UKIs by URL, and systemd-stub accepts SMBIOS Type 11 io.systemd.stub.kernel-cmdline-extra= (measured into PCR12). That composes with a signed netboot UKI, but SMBIOS strings are trivially settable for VMs, not bare metal — so it doesn't replace Part 1 for fleets. Defer.

    What exists vs. what is new

    Component Status Evidence
    Installer kernel /layer/boot/vmlinuz exists, not exported elements/oci/bluefin-server-installer.bst Step 4 (ukify --linux=)
    Installer initrd /installer.cpio.gz exists, not exported same element, Step 3 (cpio+gzip)
    Standalone DDI release asset already published elements/oci/bluefin-server-ddi.bst; .github/workflows/build.yml
    Signed SHA256SUMS + SHA256SUMS.gpg over all release assets already produced build.yml "Sign release SHA256SUMS manifest"
    Verification pubring exists, OS-image-only files/os/sysupdate-keys/import-pubring.pgp via elements/bluefin-server/os-sysupdate-keys.bst
    Network + DHCP in installer live env already enabled bluefin-server-installer.bst Step 1
    systemd-pull verification machinery ships in FSDK systemd (importd auto-enabled; curl/zstd/bzip2 in build-deps) FSDK elements/components/systemd-base.bst
    gpg binary in installer env missing — the one new runtime dep no gnupg anywhere in elements/ (see Security)
    Fetch + cmdline parsing in wrapper new bluefin-sysinstall currently only checks unattended
    repart.sysinstall.d override new (runtime-generated, ~5 lines) seam documented in systemd-sysinstall(8)
    PXE docs skill new docs/skills/pxe-netboot.md + index/AGENTS routing

    Security considerations (the high-risk part)

    1. DDI authenticity must reuse the existing trust root. The release format is already exactly what systemd-pull --verify=signature consumes: a flat SHA256SUMS + detached SHA256SUMS.gpg co-located with the assets, verified against /usr/lib/systemd/import-pubring.pgp. Grounding: upstream sysupdate-resource.c invokes systemd-pull raw --direct --verify=signature, and this repo's own files/os/sysupdate.d/50-root.transfer already pulls bluefin-server-ddi-@v.raw.zst from GitHub Releases through that path in production.
    2. New dependency: gpg in the installer env. Upstream src/import/pull-common.c performs signature verification by exec'ing gpg2/gpg against the pubring. Nothing in this repo ships gnupg today (checked all of elements/), so installer-stack.bst must add freedesktop-sdk.bst:components/gnupg.bst and stage the pubring (reuse files/os/sysupdate-keys/import-pubring.pgp — single source of truth). Flag for maintainers: elements/bluefin-server/os-stack.bst has no gnupg either, which suggests on-host systemd-sysupdate signature verification may hit the same missing-gpg failure. That predates this issue; please confirm — and if confirmed, the same one-line fix covers both.
    3. HTTP on the LAN is acceptable only because content is signature-verified. booty/dnsmasq-style PXE servers typically serve plain HTTP. The threat model is a malicious or MITM'd LAN server: the GPG signature on SHA256SUMS (fetched from the same server) is the control, exactly as upstream's ROOTFS_DISCOVERY trust section prescribes for network-fetched rootfs ("detached signatures (currently only GPG) or HTTPS certificate validation"). Rule: with inst.ddi_url set, verification is mandatory — signature mode by default; inst.ddi_sha256=<hex> as a standalone checksum-only fallback; no unsigned/unchecked installs. Fail closed, before any disk is touched (fetch completes and verifies before systemd-sysinstall is exec'd).
    4. Kernel/initrd integrity is anchored the same way the issue proposes: the PXE admin mirrors assets once and verifies them against the signed SHA256SUMS at mirror time. Per-boot, the payload that actually lands on disk (the DDI) is verified on-host.
    5. Secure Boot: today's installer media ships an unsigned UKI (element comment: signing "can be layered on" via ukify --secureboot-private-key at CI time), and the target OS UKI is unsigned too. Netboot v1 inherits that posture — no regression, but SB-enforcing fleets need signed artifacts (signed iPXE + signed UKI, or self-enrolled keys via sd-boot's setup-mode enrollment). Recommend explicitly scoping SB signing out of this issue and tracking it separately.
    6. Measured boot: separate kernel+initrd skips UKI measurement, but nothing is measured today (no TPM policy binds installer boot), so no regression. The installed system's first-boot credential path (os-creds-prov.bst, TPM2 sealing roadmap) is untouched.

    Phased breakdown (each independently landable)

    1. Export PXE artifacts — small. In bluefin-server-installer.bst Step 7, also copy /layer/boot/vmlinuz and /installer.cpio.gz into %{install-root} with versioned names and add them to SHA256SUMS; new export-pxe Justfile recipe (installer group); upload in build.yml. No new build work — pure export of byproducts. (One check: the current Step 3 packs a gzip initrd while the element description says cpio.zst; export whatever is actually built and reconcile the comment.)
    2. Network DDI pull in the installer — medium. installer-stack.bst: add components/gnupg.bst + stage import-pubring.pgp. Wrapper: parse inst.ddi_url / inst.ddi_sha256 / inst.target_disk (namespace bikeshed below), fetch via systemd-pull raw --direct to tmpfs with mandatory verification, write the CopyBlocks= override, exec systemd-sysinstall --definitions= unchanged. Absence of params = byte-identical behavior to today. Add a build-time assertion for systemd-pull/gpg mirroring the existing critical-tools check.
    3. Netboot smoke test — small-medium. Extend the show-me-the-future QEMU recipe (or add a sibling) to boot the exported kernel/initrd via QEMU -kernel/-initrd with a local HTTP server serving the DDI, reusing the existing OVMF flow. This is the regression net for Phase 2.
    4. Documentation — small. New docs/skills/pxe-netboot.md (chain, cmdline reference, iPXE stanza, RAM floor — see open questions — troubleshooting), PXE section in ddi-installer.md, routing rows in docs/skills/index.md + AGENTS.md.
    5. (Deferred, separate issues) Firmware HTTP-boot validation of the existing raw image (docs-only if it works on the lab); Secure Boot signing of installer/target UKIs and PXE artifacts; UKI-purist netboot via sd-boot URL entries / SMBIOS cmdline.

    Phases 1+4 alone already unblock "PXE-boot the installer with an embedded… nothing" for anyone willing to host a small custom initrd — but the real value is 1+2+3 landing together.

    Open questions / maintainer decisions needed

    • Cmdline namespace: inst.* (Anaconda muscle memory) vs bluefin.install.* vs aligning with systemd's own systemd.pull= precedent. No strong technical driver; needs an owner decision.
    • Fetch tool: systemd-pull (libexec binary; the exact invocation sysupdate uses, proven against this repo's release format) vs importctl pull-raw (nicer CLI, but pulls in the importd daemon/D-Bus and its man page lists only gz/xz/bz2 for raw pulls — the pull engine sniffs compression by magic bytes so zstd works in practice, but systemd-pull is the path already exercised by our OTA flow). Recommendation: systemd-pull. Confirm acceptable.
    • gnupg in the installer stack — confirm adding gnupg.bst (initrd size grows modestly) is acceptable, and confirm/deny the suspected missing-gpg gap for on-host sysupdate in the OS DDI.
    • RAM floor: the DDI stages in tmpfs (the whole live env is already a RAM-resident cpio rootfs), so netboot installs need roughly live-env + DDI of free RAM — expect to document ~8 GiB guidance pending measurement. Streaming directly to the target disk isn't possible without splitting repart into two phases; recommend accepting the RAM floor for v1 rather than engineering around it.
    • Secure Boot signing of release artifacts — explicitly out of scope here, or a hard requirement for this feature to merge? (Maintainer call; scoping it out matches current media posture.)
    • Does the wrapper's disk auto-detect need the installer-partition exclusion relaxed under netboot? It already no-ops cleanly when bluefin-installer-data is absent (verified in the wrapper logic), so no change needed — noting for reviewers.

    Labels

    Leaving labels untouched per the label workflow (humans triage; status/triage asks a maintainer to set kind/area + /approve). Recommendation: kind/enhancement — the spec in the issue body satisfies that label's "spec must be written in issue body" requirement. Note the repo currently has no area/* labels at all, so the triage contract can't be fully satisfied without creating one (e.g. area/installer) — also a maintainer call.

  2. castrojo commented on Sep 5, 2026

    @castrojo
    Contributor

    Status update:

    • Part 1 (Standalone PXE artifacts): Implemented and merged into main via PR [scanner] fix: publish PXE boot artifacts #23 (commit 5e263a0). Standalone installer kernel (bluefin-server-pxe-vmlinuz-*) and initrd (bluefin-server-pxe-initrd-*.cpio.gz) are exported and published in releases.
    • Part 2 (Network DDI pull): Proposed in PR fix: enable PXE DDI downloads #27 ([scanner] fix: enable PXE DDI downloads), currently hold-gated for review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions