Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 31 additions & 5 deletions docs/skills/ddi-installer.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,8 +182,10 @@ over the LAN with `inst.*` kernel parameters:

Without `inst.ddi_url`, behavior is unchanged and uses the embedded DDI. The
wrapper (`bluefin-sysinstall`) brings up DHCP via `systemd-networkd-wait-online`,
streams the DDI down with `curl`, stream-decompresses it with `zstd`, and feeds
it into the **native** `systemd-sysinstall` flow by staging a temporary
downloads the DDI with `curl` into `/dev/shm/installer/`, verifies it, checks
that the staging tmpfs and `MemAvailable` can hold the decompressed image,
decompresses it with `zstd`, and feeds it into the **native**
`systemd-sysinstall` flow by staging a temporary
`/usr/lib/repart.sysinstall.d/` override whose `20-root-a.conf` `CopyBlocks=`
points at the downloaded image — no custom installer logic. The override is
written with plain Bash line handling: the live initrd contains only what
Expand All @@ -204,8 +206,31 @@ initrd ${base}/bluefin-server-pxe-initrd-<ver>.cpio.gz
boot
```

RAM floor: the whole live env is a RAM-resident cpio rootfs and the DDI stages
in tmpfs, so plan roughly `live-env + DDI` of free RAM (~8 GiB guidance).
### Memory requirements

Everything on a PXE client lives in RAM: the live env is a RAM-resident cpio
rootfs, and the DDI is staged in tmpfs under `/dev/shm/installer/`. systemd
PID 1 mounts `/run` as tmpfs with `size=20%` (≈1.6 GiB on an 8 GiB machine),
which is why `/run` cannot hold the DDI; `/dev/shm` is mounted without a size
option and so uses the kernel tmpfs default of 50% of RAM. `zstd --rm` deletes
the compressed download only after a successful decompress, so the staging
tmpfs must hold the compressed **and** the decompressed DDI at once. Before
decompressing, the wrapper compares the raw size recorded in the zstd frame
header (or 5× the compressed size if the header does not carry it, with a
`WARN`) against both the free space `df` reports for `/dev/shm/installer` and
`MemAvailable` from `/proc/meminfo` minus 256 MiB of headroom, and aborts with
`ERROR: not enough RAM-backed temporary space ... (need ~N MiB; tmpfs has M MiB
free, MemAvailable is K MiB ...)` instead of failing mid-write with `ENOSPC` or
being OOM-killed. The `MemAvailable` bound matters because the tmpfs cap alone
is not proof of memory: the rootfs is its own tmpfs, so `df` on `/dev/shm` can
report space the kernel cannot back.

Plan for the check to pass: `MemAvailable` after the download must exceed the
raw DDI size plus 256 MiB, and the 50% `/dev/shm` cap must hold both copies.
For the 26.08.0 Flatcar-based DDI (365 MiB compressed, ≈1.6 GiB raw) 8 GiB is
the tested and recommended size; smaller machines have not been boot-tested.
The embedded (USB) install path does not stage the DDI in RAM and is
unaffected.

## Common Rationalizations

Expand All @@ -219,7 +244,8 @@ in tmpfs, so plan roughly `live-env + DDI` of free RAM (~8 GiB guidance).
| "Hardcode `root=/dev/vda2` for QEMU." | Bare metal has different device names. Always use PARTUUID. |
| "Pull the DDI from the network at install time." | Network pull is opt-in via `inst.ddi_url`; verification is mandatory and failures abort before any disk change, while the embedded installer media stays the default. |
| "The live initrd has sed/awk/tar like any Linux box." | It does not. The initrd holds only `installer-stack.bst` plus transitive runtime deps (Bash, uutils coreutils, util-linux, kmod, systemd, `grep`, `curl`, `zstd`); `sed`, `awk` and `tar` are absent and fail with `command not found` (exit 127) after the download. Use Bash builtins, or declare the tool in `installer-stack.bst` and add it to the Step 1a build-time tool check. |
| "Put the DDI in the initrd cpio." | The DDI is 2 GiB+. The initrd cpio step must run before the DDI is placed in `/layer`. |
| "`/run` can hold the downloaded DDI." | No. systemd mounts `/run` with `size=20%`; the network DDI stages in `/dev/shm/installer/` (kernel tmpfs default, 50% of RAM) and the wrapper checks tmpfs free space and `MemAvailable` before decompressing. See "Memory requirements". |
| "Put the DDI in the initrd cpio." | The DDI is ≈1.6 GiB raw (2 GiB+ once sparse space is written out). The initrd cpio step must run before the DDI is placed in `/layer`. |
| "Store the DDI in the ESP (FAT32)." | FAT32 has a 4 GiB per-file limit. Use a separate XFS partition. |
| "Add an 8 GiB minimum size floor to the DDI." | The rootfs is immutable. It never grows in-place. Content + overhead is enough. |

Expand Down
81 changes: 70 additions & 11 deletions elements/oci/bluefin-server-installer.bst
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,8 @@ config:
# live initrd has no sed, awk or tar, so the wrapper's tool set is pinned
# here rather than discovered at install time (exit 127 after download).
for tool in udevadm lsblk systemd-repart bootctl systemd-sysinstall systemctl \
grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep; do
grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep \
mountpoint stat df tail; do
if ! [ -x "/layer/usr/bin/${tool}" ] && ! [ -x "/layer/usr/sbin/${tool}" ]; then
echo "ERROR: Critical tool '${tool}' is missing from the staged rootfs at /layer!" >&2
exit 1
Expand All @@ -83,7 +84,7 @@ config:
# systemd-native initrd: /init -> PID 1
ln -sf usr/lib/systemd/systemd /layer/init
# Mount points needed by the installer service
mkdir -p /layer/efi /layer/mnt/esp /layer/mnt/root /layer/run/installer
mkdir -p /layer/efi /layer/mnt/esp /layer/mnt/root
mkdir -p /layer/usr/lib/bluefin-server
mkdir -p /layer/boot
mkdir -p /target-root/boot/EFI/Linux /tmp /var/tmp
Expand Down Expand Up @@ -248,30 +249,88 @@ config:
echo "ERROR: inst.ddi_url is set but inst.ddi_sha256 is missing; verification is mandatory for network installs." >&2
exit 1
fi
mkdir -p /run/installer
# Stage under /dev/shm, not /run: systemd PID 1 mounts /run as tmpfs
# capped at 20% of RAM, which cannot hold the decompressed DDI on an
# 8 GiB machine, while /dev/shm is a tmpfs at the kernel default of
# 50% of RAM. PID 1 mounts it as an API filesystem before any unit
# runs; refuse to stage on devtmpfs should that ever not be the case.
mountpoint -q /dev/shm || {
echo "ERROR: /dev/shm is not a mounted tmpfs; cannot stage the network DDI." >&2
exit 1
}
mkdir -p /dev/shm/installer
echo "==> Fetching DDI from ${DDI_URL}..."
if command -v systemd-networkd-wait-online >/dev/null 2>&1; then
systemd-networkd-wait-online --timeout=90 || {
echo "ERROR: network did not come up before the DDI download timeout." >&2
exit 1
}
fi
# Stream-decompress: decompress straight to the final image and remove
# the compressed copy, so only one DDI-sized blob lives in tmpfs at a time.
curl --fail --silent --show-error --location --retry 3 \
--output /run/installer/bluefin-server-ddi.raw.zst "${DDI_URL}"
printf '%s %s\n' "${DDI_SHA256}" /run/installer/bluefin-server-ddi.raw.zst \
--output /dev/shm/installer/bluefin-server-ddi.raw.zst "${DDI_URL}"
printf '%s %s\n' "${DDI_SHA256}" /dev/shm/installer/bluefin-server-ddi.raw.zst \
| sha256sum --check --status || {
echo "ERROR: downloaded DDI SHA256 verification failed." >&2
exit 1
}
# zstd --rm only drops the compressed copy after a successful decompress,
# so the staging tmpfs must hold both at once. Refuse up front with the
# numbers rather than dying mid-write with ENOSPC or an OOM kill. The
# compressed DDI already occupies its share, so what must still fit is
# the raw image: exact when the zstd frame header records it, else 5x
# the compressed size (the shipped DDIs compress 2-4.5x). Two bounds
# apply: the tmpfs cap df reports, and MemAvailable from /proc/meminfo
# (tmpfs pages are not reclaimable without swap, and the initrd has
# none), minus headroom for the live environment's own working set.
DDI_ZST_BYTES="$(stat -c %s /dev/shm/installer/bluefin-server-ddi.raw.zst)"
DDI_RAW_BYTES=$(( DDI_ZST_BYTES * 5 ))
DDI_RAW_SIZE_RE='Decompressed Size: [^(]*\(([0-9]+) B\)'
if [[ "$(zstd -lv /dev/shm/installer/bluefin-server-ddi.raw.zst 2>/dev/null || true)" =~ ${DDI_RAW_SIZE_RE} ]]; then
DDI_RAW_BYTES="${BASH_REMATCH[1]}"
else
echo "WARN: zstd frame header carries no content size; assuming 5x the compressed size (${DDI_RAW_BYTES} B)." >&2
fi
DDI_STAGE_AVAIL_KIB="$(df -k --output=avail /dev/shm/installer | tail -n1)" || {
echo "ERROR: df failed for /dev/shm/installer." >&2
exit 1
}
DDI_STAGE_AVAIL_KIB="${DDI_STAGE_AVAIL_KIB// /}"
[[ "${DDI_STAGE_AVAIL_KIB}" =~ ^[0-9]+$ ]] || {
echo "ERROR: unexpected df output for /dev/shm/installer: '${DDI_STAGE_AVAIL_KIB}'." >&2
exit 1
}
DDI_MEMINFO=/proc/meminfo
DDI_MEM_AVAIL_KIB=""
while read -r key value _; do
if [ "${key}" = "MemAvailable:" ]; then
DDI_MEM_AVAIL_KIB="${value}"
fi
done < "${DDI_MEMINFO}"
[[ "${DDI_MEM_AVAIL_KIB}" =~ ^[0-9]+$ ]] || {
echo "ERROR: could not read MemAvailable from ${DDI_MEMINFO}." >&2
exit 1
}
DDI_MEM_HEADROOM_KIB=$(( 256 * 1024 ))
DDI_RAM_AVAIL_KIB=$(( DDI_MEM_AVAIL_KIB - DDI_MEM_HEADROOM_KIB ))
DDI_HAVE_KIB="${DDI_STAGE_AVAIL_KIB}"
if [ "${DDI_RAM_AVAIL_KIB}" -lt "${DDI_HAVE_KIB}" ]; then
DDI_HAVE_KIB="${DDI_RAM_AVAIL_KIB}"
fi
DDI_NEED_KIB=$(( (DDI_RAW_BYTES + 1023) / 1024 ))
if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]; then
DDI_NEED_MIB=$(( (DDI_NEED_KIB + 1023) / 1024 ))
DDI_STAGE_MIB=$(( DDI_STAGE_AVAIL_KIB / 1024 ))
DDI_MEM_MIB=$(( DDI_MEM_AVAIL_KIB / 1024 ))
echo "ERROR: not enough RAM-backed temporary space in /dev/shm/installer to decompress the DDI (need ~${DDI_NEED_MIB} MiB; tmpfs has ${DDI_STAGE_MIB} MiB free, MemAvailable is ${DDI_MEM_MIB} MiB with 256 MiB headroom); add RAM to this machine." >&2
exit 1
fi
zstd -d -q --rm \
/run/installer/bluefin-server-ddi.raw.zst \
-o /run/installer/bluefin-server-ddi.raw || {
/dev/shm/installer/bluefin-server-ddi.raw.zst \
-o /dev/shm/installer/bluefin-server-ddi.raw || {
echo "ERROR: failed to decompress downloaded DDI." >&2
exit 1
}
ls -lh /run/installer/bluefin-server-ddi.raw
ls -lh /dev/shm/installer/bluefin-server-ddi.raw
# Feed the downloaded image into the native systemd-sysinstall flow by
# overriding only the root recipe's CopyBlocks=. systemd-sysinstall
# reads /usr/lib/repart.sysinstall.d/ when populated, else /usr/lib/
Expand All @@ -284,7 +343,7 @@ config:
# Recipe and wrapper ship from the same repo, so a recipe without a
# column-0 CopyBlocks= line is a repo bug: fail loudly before the disk
# is touched instead of guessing where the key belongs.
ROOT_COPYBLOCKS_LINE="CopyBlocks=/run/installer/bluefin-server-ddi.raw"
ROOT_COPYBLOCKS_LINE="CopyBlocks=/dev/shm/installer/bluefin-server-ddi.raw"
ROOT_COPYBLOCKS_SEEN=0
while IFS= read -r line || [ -n "${line}" ]; do
if [[ "${line}" == CopyBlocks=* ]]; then
Expand Down
154 changes: 154 additions & 0 deletions tests/unit/test_installer_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,14 @@

from __future__ import annotations

import os
import re
import shlex
import shutil
import subprocess
from pathlib import Path

import pytest
import yaml

REPO_ROOT = Path(__file__).resolve().parents[2]
Expand Down Expand Up @@ -165,6 +168,7 @@ def test_installer_build_pins_every_external_command_the_wrapper_calls() -> None
for tool in (
"grep", "curl", "zstd", "modprobe", "mount", "umount", "sha256sum",
"readlink", "lsblk", "udevadm", "systemd-sysinstall", "systemctl",
"mountpoint", "stat", "df", "tail",
):
assert tool in pinned, f"{tool} is called by bluefin-sysinstall but not pinned at build time"

Expand Down Expand Up @@ -239,6 +243,156 @@ def test_installer_wrapper_copyblocks_repoint_fails_closed_without_a_copyblocks_
assert copyblocks_line not in output.read_text(encoding="utf-8")


def test_installer_network_ddi_stages_in_dev_shm_with_space_preflight(installer_wrapper: str) -> None:
wrapper = installer_wrapper

# /run is a tmpfs capped at 20% of RAM and cannot hold the decompressed DDI
# on an 8 GiB machine; /dev/shm is at the kernel tmpfs default of 50%.
assert "mountpoint -q /dev/shm ||" in wrapper
assert "mkdir -p /dev/shm/installer" in wrapper
assert "/run/installer" not in wrapper
assert '--output /dev/shm/installer/bluefin-server-ddi.raw.zst "${DDI_URL}"' in wrapper
assert "-o /dev/shm/installer/bluefin-server-ddi.raw ||" in wrapper
assert "CopyBlocks=/dev/shm/installer/bluefin-server-ddi.raw" in wrapper

# Free space is checked against the raw size (frame header, else 5x the
# compressed size) after verification and before zstd runs, so a machine
# with too little RAM gets a clear error instead of ENOSPC mid-write or an
# OOM kill. Both the tmpfs cap (df) and MemAvailable bound the check. The
# comparison is in KiB, rounded up, so a raw image a few KiB over the free
# space is not waved through by MiB truncation.
verify = wrapper.index("sha256sum --check --status")
preflight = wrapper.index("ERROR: not enough RAM-backed temporary space in /dev/shm/installer")
decompress = wrapper.index("zstd -d -q --rm")
assert verify < preflight < decompress
assert 'DDI_ZST_BYTES="$(stat -c %s /dev/shm/installer/bluefin-server-ddi.raw.zst)"' in wrapper
assert "DDI_RAW_BYTES=$(( DDI_ZST_BYTES * 5 ))" in wrapper
assert 'DDI_STAGE_AVAIL_KIB="$(df -k --output=avail /dev/shm/installer | tail -n1)"' in wrapper
assert "DDI_MEMINFO=/proc/meminfo" in wrapper
assert 'if [ "${key}" = "MemAvailable:" ]; then' in wrapper
assert "DDI_NEED_KIB=$(( (DDI_RAW_BYTES + 1023) / 1024 ))" in wrapper
assert 'if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]; then' in wrapper


def _space_preflight_block(wrapper: str) -> str:
"""The preflight carved out of the wrapper: from the stat of the download to the fi of the size check."""
start = wrapper.index('DDI_ZST_BYTES="$(stat -c %s')
size_if = wrapper.index('if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]', start)
end = re.compile(r"^[ \t]*fi\n", re.MULTILINE).search(wrapper, size_if).end()
return wrapper[start:end]


def _run_space_preflight(
tmp_path: Path,
block: str,
payload: bytes,
avail_kib: int,
mem_available_kib: int = 64 * 1024 * 1024,
meminfo_text: str | None = None,
) -> subprocess.CompletedProcess:
"""Run the preflight with a stand-in df (right-aligned like the real one) and a fake /proc/meminfo."""
stage = tmp_path / "stage"
stage.mkdir(parents=True)
(stage / "bluefin-server-ddi.raw.zst").write_bytes(payload)
fake_bin = tmp_path / "bin"
fake_bin.mkdir()
(fake_bin / "df").write_text(
f"#!/bin/bash\nprintf ' Avail\\n %d\\n' {avail_kib}\n", encoding="utf-8"
)
(fake_bin / "df").chmod(0o755)
meminfo = tmp_path / "meminfo"
if meminfo_text is None:
meminfo_text = (
"MemTotal: 16000000 kB\n"
"MemFree: 1000000 kB\n"
f"MemAvailable: {mem_available_kib} kB\n"
"Buffers: 10 kB\n"
)
meminfo.write_text(meminfo_text, encoding="utf-8")
script = block.replace("/dev/shm/installer", shlex.quote(str(stage))).replace(
"DDI_MEMINFO=/proc/meminfo", f"DDI_MEMINFO={shlex.quote(str(meminfo))}"
)
env = dict(os.environ, PATH=f"{fake_bin}:{os.environ['PATH']}")
return subprocess.run(
["bash", "-euo", "pipefail", "-c", script],
capture_output=True,
text=True,
env=env,
)


@pytest.mark.skipif(shutil.which("zstd") is None, reason="zstd not installed")
def test_installer_space_preflight_uses_the_zstd_frame_header(
tmp_path: Path, installer_wrapper: str
) -> None:
block = _space_preflight_block(installer_wrapper)

raw = tmp_path / "ddi.raw"
raw.write_bytes(b"\0" * (3 * 1048576 + 1))
subprocess.run(["zstd", "-q", "-f", str(raw), "-o", str(tmp_path / "ddi.raw.zst")], check=True)
payload = (tmp_path / "ddi.raw.zst").read_bytes()

# A frame that records a raw size of 3 MiB + 1 byte needs 3073 KiB: it passes
# with that much free and fails with 3072 KiB (exactly 3 MiB), regardless of
# the compressed size. The message rounds up to whole MiB and names both
# bounds.
ok = _run_space_preflight(tmp_path / "ok", block, payload, avail_kib=3 * 1024 + 1)
assert ok.returncode == 0, ok.stderr
assert "WARN" not in ok.stderr
short = _run_space_preflight(tmp_path / "short", block, payload, avail_kib=3 * 1024)
assert short.returncode == 1
assert "need ~4 MiB; tmpfs has 3 MiB free, MemAvailable is 65536 MiB" in short.stderr


@pytest.mark.skipif(shutil.which("zstd") is None, reason="zstd not installed")
def test_installer_space_preflight_is_bounded_by_mem_available(
tmp_path: Path, installer_wrapper: str
) -> None:
block = _space_preflight_block(installer_wrapper)

raw = tmp_path / "ddi.raw"
raw.write_bytes(b"\0" * (3 * 1048576 + 1))
subprocess.run(["zstd", "-q", "-f", str(raw), "-o", str(tmp_path / "ddi.raw.zst")], check=True)
payload = (tmp_path / "ddi.raw.zst").read_bytes()

# The tmpfs cap is generous but real memory is not: the rootfs is its own
# tmpfs, so df on /dev/shm can report far more than the kernel can back.
# MemAvailable minus 256 MiB headroom must also cover the raw image.
headroom = 256 * 1024
ok = _run_space_preflight(
tmp_path / "ok", block, payload, avail_kib=1 << 30, mem_available_kib=headroom + 3 * 1024 + 1
)
assert ok.returncode == 0, ok.stderr
short = _run_space_preflight(
tmp_path / "short", block, payload, avail_kib=1 << 30, mem_available_kib=headroom + 3 * 1024
)
assert short.returncode == 1
assert "need ~4 MiB; tmpfs has 1048576 MiB free, MemAvailable is 259 MiB" in short.stderr

# A meminfo without a MemAvailable line fails closed too.
broken = _run_space_preflight(
tmp_path / "broken", block, payload, avail_kib=1 << 30, meminfo_text="MemTotal: 1 kB\n"
)
assert broken.returncode == 1
assert "could not read MemAvailable" in broken.stderr


def test_installer_space_preflight_falls_back_to_five_times_compressed(
tmp_path: Path, installer_wrapper: str
) -> None:
block = _space_preflight_block(installer_wrapper)

# Not a zstd frame: no header size, so 5x the 1 MiB file must fit, and the
# wrapper says so.
payload = b"x" * 1048576
ok = _run_space_preflight(tmp_path / "ok", block, payload, avail_kib=5 * 1024)
assert ok.returncode == 0, ok.stderr
assert "WARN: zstd frame header carries no content size" in ok.stderr
short = _run_space_preflight(tmp_path / "short", block, payload, avail_kib=5 * 1024 - 1)
assert short.returncode == 1
assert "need ~5 MiB; tmpfs has 4 MiB free" in short.stderr


def test_interactive_installer_uses_local_virtual_console() -> None:
installer_element = INSTALLER_ELEMENT.read_text(encoding="utf-8")

Expand Down