diff --git a/docs/skills/ddi-installer.md b/docs/skills/ddi-installer.md index 3d362fd8..2d0fd15c 100644 --- a/docs/skills/ddi-installer.md +++ b/docs/skills/ddi-installer.md @@ -182,8 +182,10 @@ over the LAN with `inst.*` kernel parameters: Without `inst.ddi_url`, behavior is unchanged and uses the embedded DDI. The wrapper (`bluefin-sysinstall`) brings up DHCP via `systemd-networkd-wait-online`, -streams the DDI down with `curl`, stream-decompresses it with `zstd`, and feeds -it into the **native** `systemd-sysinstall` flow by staging a temporary +downloads the DDI with `curl` into `/dev/shm/installer/`, verifies it, checks +that the staging tmpfs and `MemAvailable` can hold the decompressed image, +decompresses it with `zstd`, and feeds it into the **native** +`systemd-sysinstall` flow by staging a temporary `/usr/lib/repart.sysinstall.d/` override whose `20-root-a.conf` `CopyBlocks=` points at the downloaded image — no custom installer logic. The override is written with plain Bash line handling: the live initrd contains only what @@ -204,8 +206,31 @@ initrd ${base}/bluefin-server-pxe-initrd-.cpio.gz boot ``` -RAM floor: the whole live env is a RAM-resident cpio rootfs and the DDI stages -in tmpfs, so plan roughly `live-env + DDI` of free RAM (~8 GiB guidance). +### Memory requirements + +Everything on a PXE client lives in RAM: the live env is a RAM-resident cpio +rootfs, and the DDI is staged in tmpfs under `/dev/shm/installer/`. systemd +PID 1 mounts `/run` as tmpfs with `size=20%` (≈1.6 GiB on an 8 GiB machine), +which is why `/run` cannot hold the DDI; `/dev/shm` is mounted without a size +option and so uses the kernel tmpfs default of 50% of RAM. `zstd --rm` deletes +the compressed download only after a successful decompress, so the staging +tmpfs must hold the compressed **and** the decompressed DDI at once. Before +decompressing, the wrapper compares the raw size recorded in the zstd frame +header (or 5× the compressed size if the header does not carry it, with a +`WARN`) against both the free space `df` reports for `/dev/shm/installer` and +`MemAvailable` from `/proc/meminfo` minus 256 MiB of headroom, and aborts with +`ERROR: not enough RAM-backed temporary space ... (need ~N MiB; tmpfs has M MiB +free, MemAvailable is K MiB ...)` instead of failing mid-write with `ENOSPC` or +being OOM-killed. The `MemAvailable` bound matters because the tmpfs cap alone +is not proof of memory: the rootfs is its own tmpfs, so `df` on `/dev/shm` can +report space the kernel cannot back. + +Plan for the check to pass: `MemAvailable` after the download must exceed the +raw DDI size plus 256 MiB, and the 50% `/dev/shm` cap must hold both copies. +For the 26.08.0 Flatcar-based DDI (365 MiB compressed, ≈1.6 GiB raw) 8 GiB is +the tested and recommended size; smaller machines have not been boot-tested. +The embedded (USB) install path does not stage the DDI in RAM and is +unaffected. ## Common Rationalizations @@ -219,7 +244,8 @@ in tmpfs, so plan roughly `live-env + DDI` of free RAM (~8 GiB guidance). | "Hardcode `root=/dev/vda2` for QEMU." | Bare metal has different device names. Always use PARTUUID. | | "Pull the DDI from the network at install time." | Network pull is opt-in via `inst.ddi_url`; verification is mandatory and failures abort before any disk change, while the embedded installer media stays the default. | | "The live initrd has sed/awk/tar like any Linux box." | It does not. The initrd holds only `installer-stack.bst` plus transitive runtime deps (Bash, uutils coreutils, util-linux, kmod, systemd, `grep`, `curl`, `zstd`); `sed`, `awk` and `tar` are absent and fail with `command not found` (exit 127) after the download. Use Bash builtins, or declare the tool in `installer-stack.bst` and add it to the Step 1a build-time tool check. | -| "Put the DDI in the initrd cpio." | The DDI is 2 GiB+. The initrd cpio step must run before the DDI is placed in `/layer`. | +| "`/run` can hold the downloaded DDI." | No. systemd mounts `/run` with `size=20%`; the network DDI stages in `/dev/shm/installer/` (kernel tmpfs default, 50% of RAM) and the wrapper checks tmpfs free space and `MemAvailable` before decompressing. See "Memory requirements". | +| "Put the DDI in the initrd cpio." | The DDI is ≈1.6 GiB raw (2 GiB+ once sparse space is written out). The initrd cpio step must run before the DDI is placed in `/layer`. | | "Store the DDI in the ESP (FAT32)." | FAT32 has a 4 GiB per-file limit. Use a separate XFS partition. | | "Add an 8 GiB minimum size floor to the DDI." | The rootfs is immutable. It never grows in-place. Content + overhead is enough. | diff --git a/elements/oci/bluefin-server-installer.bst b/elements/oci/bluefin-server-installer.bst index 7df2c71a..2232d543 100644 --- a/elements/oci/bluefin-server-installer.bst +++ b/elements/oci/bluefin-server-installer.bst @@ -73,7 +73,8 @@ config: # live initrd has no sed, awk or tar, so the wrapper's tool set is pinned # here rather than discovered at install time (exit 127 after download). for tool in udevadm lsblk systemd-repart bootctl systemd-sysinstall systemctl \ - grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep; do + grep curl zstd modprobe mount umount sha256sum readlink ls cp mkdir sleep \ + mountpoint stat df tail; do if ! [ -x "/layer/usr/bin/${tool}" ] && ! [ -x "/layer/usr/sbin/${tool}" ]; then echo "ERROR: Critical tool '${tool}' is missing from the staged rootfs at /layer!" >&2 exit 1 @@ -83,7 +84,7 @@ config: # systemd-native initrd: /init -> PID 1 ln -sf usr/lib/systemd/systemd /layer/init # Mount points needed by the installer service - mkdir -p /layer/efi /layer/mnt/esp /layer/mnt/root /layer/run/installer + mkdir -p /layer/efi /layer/mnt/esp /layer/mnt/root mkdir -p /layer/usr/lib/bluefin-server mkdir -p /layer/boot mkdir -p /target-root/boot/EFI/Linux /tmp /var/tmp @@ -248,7 +249,16 @@ config: echo "ERROR: inst.ddi_url is set but inst.ddi_sha256 is missing; verification is mandatory for network installs." >&2 exit 1 fi - mkdir -p /run/installer + # Stage under /dev/shm, not /run: systemd PID 1 mounts /run as tmpfs + # capped at 20% of RAM, which cannot hold the decompressed DDI on an + # 8 GiB machine, while /dev/shm is a tmpfs at the kernel default of + # 50% of RAM. PID 1 mounts it as an API filesystem before any unit + # runs; refuse to stage on devtmpfs should that ever not be the case. + mountpoint -q /dev/shm || { + echo "ERROR: /dev/shm is not a mounted tmpfs; cannot stage the network DDI." >&2 + exit 1 + } + mkdir -p /dev/shm/installer echo "==> Fetching DDI from ${DDI_URL}..." if command -v systemd-networkd-wait-online >/dev/null 2>&1; then systemd-networkd-wait-online --timeout=90 || { @@ -256,22 +266,71 @@ config: exit 1 } fi - # Stream-decompress: decompress straight to the final image and remove - # the compressed copy, so only one DDI-sized blob lives in tmpfs at a time. curl --fail --silent --show-error --location --retry 3 \ - --output /run/installer/bluefin-server-ddi.raw.zst "${DDI_URL}" - printf '%s %s\n' "${DDI_SHA256}" /run/installer/bluefin-server-ddi.raw.zst \ + --output /dev/shm/installer/bluefin-server-ddi.raw.zst "${DDI_URL}" + printf '%s %s\n' "${DDI_SHA256}" /dev/shm/installer/bluefin-server-ddi.raw.zst \ | sha256sum --check --status || { echo "ERROR: downloaded DDI SHA256 verification failed." >&2 exit 1 } + # zstd --rm only drops the compressed copy after a successful decompress, + # so the staging tmpfs must hold both at once. Refuse up front with the + # numbers rather than dying mid-write with ENOSPC or an OOM kill. The + # compressed DDI already occupies its share, so what must still fit is + # the raw image: exact when the zstd frame header records it, else 5x + # the compressed size (the shipped DDIs compress 2-4.5x). Two bounds + # apply: the tmpfs cap df reports, and MemAvailable from /proc/meminfo + # (tmpfs pages are not reclaimable without swap, and the initrd has + # none), minus headroom for the live environment's own working set. + DDI_ZST_BYTES="$(stat -c %s /dev/shm/installer/bluefin-server-ddi.raw.zst)" + DDI_RAW_BYTES=$(( DDI_ZST_BYTES * 5 )) + DDI_RAW_SIZE_RE='Decompressed Size: [^(]*\(([0-9]+) B\)' + if [[ "$(zstd -lv /dev/shm/installer/bluefin-server-ddi.raw.zst 2>/dev/null || true)" =~ ${DDI_RAW_SIZE_RE} ]]; then + DDI_RAW_BYTES="${BASH_REMATCH[1]}" + else + echo "WARN: zstd frame header carries no content size; assuming 5x the compressed size (${DDI_RAW_BYTES} B)." >&2 + fi + DDI_STAGE_AVAIL_KIB="$(df -k --output=avail /dev/shm/installer | tail -n1)" || { + echo "ERROR: df failed for /dev/shm/installer." >&2 + exit 1 + } + DDI_STAGE_AVAIL_KIB="${DDI_STAGE_AVAIL_KIB// /}" + [[ "${DDI_STAGE_AVAIL_KIB}" =~ ^[0-9]+$ ]] || { + echo "ERROR: unexpected df output for /dev/shm/installer: '${DDI_STAGE_AVAIL_KIB}'." >&2 + exit 1 + } + DDI_MEMINFO=/proc/meminfo + DDI_MEM_AVAIL_KIB="" + while read -r key value _; do + if [ "${key}" = "MemAvailable:" ]; then + DDI_MEM_AVAIL_KIB="${value}" + fi + done < "${DDI_MEMINFO}" + [[ "${DDI_MEM_AVAIL_KIB}" =~ ^[0-9]+$ ]] || { + echo "ERROR: could not read MemAvailable from ${DDI_MEMINFO}." >&2 + exit 1 + } + DDI_MEM_HEADROOM_KIB=$(( 256 * 1024 )) + DDI_RAM_AVAIL_KIB=$(( DDI_MEM_AVAIL_KIB - DDI_MEM_HEADROOM_KIB )) + DDI_HAVE_KIB="${DDI_STAGE_AVAIL_KIB}" + if [ "${DDI_RAM_AVAIL_KIB}" -lt "${DDI_HAVE_KIB}" ]; then + DDI_HAVE_KIB="${DDI_RAM_AVAIL_KIB}" + fi + DDI_NEED_KIB=$(( (DDI_RAW_BYTES + 1023) / 1024 )) + if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]; then + DDI_NEED_MIB=$(( (DDI_NEED_KIB + 1023) / 1024 )) + DDI_STAGE_MIB=$(( DDI_STAGE_AVAIL_KIB / 1024 )) + DDI_MEM_MIB=$(( DDI_MEM_AVAIL_KIB / 1024 )) + echo "ERROR: not enough RAM-backed temporary space in /dev/shm/installer to decompress the DDI (need ~${DDI_NEED_MIB} MiB; tmpfs has ${DDI_STAGE_MIB} MiB free, MemAvailable is ${DDI_MEM_MIB} MiB with 256 MiB headroom); add RAM to this machine." >&2 + exit 1 + fi zstd -d -q --rm \ - /run/installer/bluefin-server-ddi.raw.zst \ - -o /run/installer/bluefin-server-ddi.raw || { + /dev/shm/installer/bluefin-server-ddi.raw.zst \ + -o /dev/shm/installer/bluefin-server-ddi.raw || { echo "ERROR: failed to decompress downloaded DDI." >&2 exit 1 } - ls -lh /run/installer/bluefin-server-ddi.raw + ls -lh /dev/shm/installer/bluefin-server-ddi.raw # Feed the downloaded image into the native systemd-sysinstall flow by # overriding only the root recipe's CopyBlocks=. systemd-sysinstall # reads /usr/lib/repart.sysinstall.d/ when populated, else /usr/lib/ @@ -284,7 +343,7 @@ config: # Recipe and wrapper ship from the same repo, so a recipe without a # column-0 CopyBlocks= line is a repo bug: fail loudly before the disk # is touched instead of guessing where the key belongs. - ROOT_COPYBLOCKS_LINE="CopyBlocks=/run/installer/bluefin-server-ddi.raw" + ROOT_COPYBLOCKS_LINE="CopyBlocks=/dev/shm/installer/bluefin-server-ddi.raw" ROOT_COPYBLOCKS_SEEN=0 while IFS= read -r line || [ -n "${line}" ]; do if [[ "${line}" == CopyBlocks=* ]]; then diff --git a/tests/unit/test_installer_contract.py b/tests/unit/test_installer_contract.py index d8108711..791ca6e8 100644 --- a/tests/unit/test_installer_contract.py +++ b/tests/unit/test_installer_contract.py @@ -2,11 +2,14 @@ from __future__ import annotations +import os import re import shlex +import shutil import subprocess from pathlib import Path +import pytest import yaml REPO_ROOT = Path(__file__).resolve().parents[2] @@ -165,6 +168,7 @@ def test_installer_build_pins_every_external_command_the_wrapper_calls() -> None for tool in ( "grep", "curl", "zstd", "modprobe", "mount", "umount", "sha256sum", "readlink", "lsblk", "udevadm", "systemd-sysinstall", "systemctl", + "mountpoint", "stat", "df", "tail", ): assert tool in pinned, f"{tool} is called by bluefin-sysinstall but not pinned at build time" @@ -239,6 +243,156 @@ def test_installer_wrapper_copyblocks_repoint_fails_closed_without_a_copyblocks_ assert copyblocks_line not in output.read_text(encoding="utf-8") +def test_installer_network_ddi_stages_in_dev_shm_with_space_preflight(installer_wrapper: str) -> None: + wrapper = installer_wrapper + + # /run is a tmpfs capped at 20% of RAM and cannot hold the decompressed DDI + # on an 8 GiB machine; /dev/shm is at the kernel tmpfs default of 50%. + assert "mountpoint -q /dev/shm ||" in wrapper + assert "mkdir -p /dev/shm/installer" in wrapper + assert "/run/installer" not in wrapper + assert '--output /dev/shm/installer/bluefin-server-ddi.raw.zst "${DDI_URL}"' in wrapper + assert "-o /dev/shm/installer/bluefin-server-ddi.raw ||" in wrapper + assert "CopyBlocks=/dev/shm/installer/bluefin-server-ddi.raw" in wrapper + + # Free space is checked against the raw size (frame header, else 5x the + # compressed size) after verification and before zstd runs, so a machine + # with too little RAM gets a clear error instead of ENOSPC mid-write or an + # OOM kill. Both the tmpfs cap (df) and MemAvailable bound the check. The + # comparison is in KiB, rounded up, so a raw image a few KiB over the free + # space is not waved through by MiB truncation. + verify = wrapper.index("sha256sum --check --status") + preflight = wrapper.index("ERROR: not enough RAM-backed temporary space in /dev/shm/installer") + decompress = wrapper.index("zstd -d -q --rm") + assert verify < preflight < decompress + assert 'DDI_ZST_BYTES="$(stat -c %s /dev/shm/installer/bluefin-server-ddi.raw.zst)"' in wrapper + assert "DDI_RAW_BYTES=$(( DDI_ZST_BYTES * 5 ))" in wrapper + assert 'DDI_STAGE_AVAIL_KIB="$(df -k --output=avail /dev/shm/installer | tail -n1)"' in wrapper + assert "DDI_MEMINFO=/proc/meminfo" in wrapper + assert 'if [ "${key}" = "MemAvailable:" ]; then' in wrapper + assert "DDI_NEED_KIB=$(( (DDI_RAW_BYTES + 1023) / 1024 ))" in wrapper + assert 'if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]; then' in wrapper + + +def _space_preflight_block(wrapper: str) -> str: + """The preflight carved out of the wrapper: from the stat of the download to the fi of the size check.""" + start = wrapper.index('DDI_ZST_BYTES="$(stat -c %s') + size_if = wrapper.index('if [ "${DDI_HAVE_KIB}" -lt "${DDI_NEED_KIB}" ]', start) + end = re.compile(r"^[ \t]*fi\n", re.MULTILINE).search(wrapper, size_if).end() + return wrapper[start:end] + + +def _run_space_preflight( + tmp_path: Path, + block: str, + payload: bytes, + avail_kib: int, + mem_available_kib: int = 64 * 1024 * 1024, + meminfo_text: str | None = None, +) -> subprocess.CompletedProcess: + """Run the preflight with a stand-in df (right-aligned like the real one) and a fake /proc/meminfo.""" + stage = tmp_path / "stage" + stage.mkdir(parents=True) + (stage / "bluefin-server-ddi.raw.zst").write_bytes(payload) + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + (fake_bin / "df").write_text( + f"#!/bin/bash\nprintf ' Avail\\n %d\\n' {avail_kib}\n", encoding="utf-8" + ) + (fake_bin / "df").chmod(0o755) + meminfo = tmp_path / "meminfo" + if meminfo_text is None: + meminfo_text = ( + "MemTotal: 16000000 kB\n" + "MemFree: 1000000 kB\n" + f"MemAvailable: {mem_available_kib} kB\n" + "Buffers: 10 kB\n" + ) + meminfo.write_text(meminfo_text, encoding="utf-8") + script = block.replace("/dev/shm/installer", shlex.quote(str(stage))).replace( + "DDI_MEMINFO=/proc/meminfo", f"DDI_MEMINFO={shlex.quote(str(meminfo))}" + ) + env = dict(os.environ, PATH=f"{fake_bin}:{os.environ['PATH']}") + return subprocess.run( + ["bash", "-euo", "pipefail", "-c", script], + capture_output=True, + text=True, + env=env, + ) + + +@pytest.mark.skipif(shutil.which("zstd") is None, reason="zstd not installed") +def test_installer_space_preflight_uses_the_zstd_frame_header( + tmp_path: Path, installer_wrapper: str +) -> None: + block = _space_preflight_block(installer_wrapper) + + raw = tmp_path / "ddi.raw" + raw.write_bytes(b"\0" * (3 * 1048576 + 1)) + subprocess.run(["zstd", "-q", "-f", str(raw), "-o", str(tmp_path / "ddi.raw.zst")], check=True) + payload = (tmp_path / "ddi.raw.zst").read_bytes() + + # A frame that records a raw size of 3 MiB + 1 byte needs 3073 KiB: it passes + # with that much free and fails with 3072 KiB (exactly 3 MiB), regardless of + # the compressed size. The message rounds up to whole MiB and names both + # bounds. + ok = _run_space_preflight(tmp_path / "ok", block, payload, avail_kib=3 * 1024 + 1) + assert ok.returncode == 0, ok.stderr + assert "WARN" not in ok.stderr + short = _run_space_preflight(tmp_path / "short", block, payload, avail_kib=3 * 1024) + assert short.returncode == 1 + assert "need ~4 MiB; tmpfs has 3 MiB free, MemAvailable is 65536 MiB" in short.stderr + + +@pytest.mark.skipif(shutil.which("zstd") is None, reason="zstd not installed") +def test_installer_space_preflight_is_bounded_by_mem_available( + tmp_path: Path, installer_wrapper: str +) -> None: + block = _space_preflight_block(installer_wrapper) + + raw = tmp_path / "ddi.raw" + raw.write_bytes(b"\0" * (3 * 1048576 + 1)) + subprocess.run(["zstd", "-q", "-f", str(raw), "-o", str(tmp_path / "ddi.raw.zst")], check=True) + payload = (tmp_path / "ddi.raw.zst").read_bytes() + + # The tmpfs cap is generous but real memory is not: the rootfs is its own + # tmpfs, so df on /dev/shm can report far more than the kernel can back. + # MemAvailable minus 256 MiB headroom must also cover the raw image. + headroom = 256 * 1024 + ok = _run_space_preflight( + tmp_path / "ok", block, payload, avail_kib=1 << 30, mem_available_kib=headroom + 3 * 1024 + 1 + ) + assert ok.returncode == 0, ok.stderr + short = _run_space_preflight( + tmp_path / "short", block, payload, avail_kib=1 << 30, mem_available_kib=headroom + 3 * 1024 + ) + assert short.returncode == 1 + assert "need ~4 MiB; tmpfs has 1048576 MiB free, MemAvailable is 259 MiB" in short.stderr + + # A meminfo without a MemAvailable line fails closed too. + broken = _run_space_preflight( + tmp_path / "broken", block, payload, avail_kib=1 << 30, meminfo_text="MemTotal: 1 kB\n" + ) + assert broken.returncode == 1 + assert "could not read MemAvailable" in broken.stderr + + +def test_installer_space_preflight_falls_back_to_five_times_compressed( + tmp_path: Path, installer_wrapper: str +) -> None: + block = _space_preflight_block(installer_wrapper) + + # Not a zstd frame: no header size, so 5x the 1 MiB file must fit, and the + # wrapper says so. + payload = b"x" * 1048576 + ok = _run_space_preflight(tmp_path / "ok", block, payload, avail_kib=5 * 1024) + assert ok.returncode == 0, ok.stderr + assert "WARN: zstd frame header carries no content size" in ok.stderr + short = _run_space_preflight(tmp_path / "short", block, payload, avail_kib=5 * 1024 - 1) + assert short.returncode == 1 + assert "need ~5 MiB; tmpfs has 4 MiB free" in short.stderr + + def test_interactive_installer_uses_local_virtual_console() -> None: installer_element = INSTALLER_ELEMENT.read_text(encoding="utf-8")