Repository navigation
feat(initrd): explain diskless boot failures and ignored Ignition kargs - #273
Conversation
The emergency drop-in printed 'journalctl -p err' to /dev/console (only the last console=, ttyS0 on the netboot UKI) and rebooted after 10 s, so the reason scrolled away and never reached the monitor. emergency.service now runs /usr/libexec/bluefin-boot-diagnostics failure-summary: each failed unit with a one-line hint (network, download, verity, Ignition stage) and the lines its processes logged, the network links when the network or download failed, and the last errors, written to every active console. It stays up for 60 s (bluefin.failure_delay credential, 10 s floor against hot loops) before the unattended reboot. Type=idle keeps the isolate's status lines out of the summary. The helper also carries the pull RAM check and the Ignition karg warning wired up in the next commits. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… pull The netboot initrd downloads the whole DDI (540 MiB) into /run, a tmpfs systemd caps at 20% of RAM. A small machine failed deep in the download, and any failed download only surfaced when the /usr devices timed out 300 s later. A drop-in makes every systemd-import@.service require bluefin-pull-check@%i.service, which takes the URL the import generator resolved (rd.systemd.pull, bootorigin, or the import.pull credential), HEADs it and fails with 'needs ~X MiB RAM, have Y' when the image plus 64 MiB does not fit in /run or MemAvailable, or with the URL and link state when the server is unreachable or answers an error. Both the check and the download go straight to the failure summary on failure. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ignition configs come only from the ignition.config / ignition.config.url credentials or bluefin-node.ign; ignition.config.url= and friends on the command line were silently ignored. bluefin-ignition-kargs.service, linked into initrd.target.wants, prints a framed warning naming the ignition.* keys it found (never their values) and how to pass the config instead; the failure summary repeats it. The UKI build now also checks that the diagnostics helper and the tools it runs are in the initrd. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ition subset) New docs/skills/diskless-troubleshooting.md is the one place for what a failing diskless node prints and how to tune the delay, the minimum RAM (DDI x5 + margin, measured in QEMU: -m 3072 refused, -m 3584 boots), the network/download/verity/Ignition failure modes and their timing, how to keep logs (serial console, /var on disk via Ignition, journal-upload), and the supported Ignition subset with its idempotency rules. ddi-installer.md links to it instead of restating, and a test pins every console link to an existing section. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Measured in QEMU with a /var disk: each diskless boot journals under a new machine ID, so plain journalctl lists one boot and --merge --list-boots lists none, while journalctl -D /var/log/journal --list-boots lists both. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed at c58c77d. The failure summary and the kargs warning look good. One problem in the pull pre-check.
The HEAD pre-check can refuse a source the download would accept
systemd-import@.service.d/10-bluefin-pull-check.conf adds Requires= and After= on bluefin-pull-check@%i.service, so the image download only starts if bluefin-boot-diagnostics check-pull exits 0. In check_pull (files/initrd/usr/libexec/bluefin-boot-diagnostics:131-140), any final HEAD status other than 2xx, 405 or 501 returns 1, and OnFailure=emergency.target then reboots. A server that serves the image on GET but answers HEAD with 403 or 404 would work fine with systemd-import, but it now never boots. Presigned or signed-URL object stores behave this way, because the signature covers the method. The pre-check is meant to explain failures, so it shouldn't be able to cause one. Suggest treating any non-2xx HEAD as "cannot check the size", logging at warning and returning 0 like the 405/501 branch does, and leaving the real error to the download's own OnFailure= (which this PR already adds). The unreachable-server branch (curl rc != 0) is fine to keep fatal, since the GET would fail the same way.
What I verified
pytest tests/unit: 406 passed.bats tests/unit: pass.shellcheck -S warning files/initrd/usr/libexec/bluefin-boot-diagnostics: clean.- The script only uses bash builtins plus curl, stat, timeout, tee, systemctl and journalctl.
bluefin-server-boot.bstnow fails the build ifbash,curl,stat,timeoutorteeare missing from the initrd, andinitrd-stack.bstalready pullscomponents/curl.bst. - Console writes go through
timeout 5 teefor each active console, so a stuck serial console can't hold the emergency path.emergency.servicekeepsExecStart=-withSuccessAction=/FailureAction=reboot-force, and the delay is clamped to 10..86400 s, so a failing summary still reboots and can't hot-loop. - The RAM math works in KiB before multiplying, and falls back to skipping the check when
/proc/meminfoorstat -fis unreadable. - On the UNSTABLE state: unit (run 36518954264) and docs (36518954387) pass, and build (36518954330) was still running when I checked. No failed check, just one still pending.
The pull check failed, and so stopped the boot, on any final HEAD status other than 2xx, 405 or 501, and when curl could not reach the server. A server that serves the image on GET but answers HEAD with 403 or 404, as signed object-store URLs do because the signature covers the method, never booted, although systemd-import would have downloaded the image. The check now fails only when a 2xx HEAD gives a Content-Length that cannot fit in /run and MemAvailable. Any other outcome (no answer, a 4xx or 5xx, no Content-Length, unreadable RAM or /run sizes) prints one line on every console, "BLUEFIN: cannot check that the OS image fits in RAM (<reason>); downloading it anyway", and exits 0. If the download then fails, its own OnFailure= puts systemd-importd's reason in the failure summary. The check runs in a subshell and reports "does not fit" with a status of its own, which the helper turns into exit 1; any other status, such as bash aborting on an unset variable, exits 0, so a bug in the check cannot stop the boot either. Since the check now fails only on RAM, its failure-summary hint says so and no longer lists the network links. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-import@.service required bluefin-pull-check@%i.service, so any failure of the check, or a missing check unit, failed the download's start job. The check already stops the boot through its own OnFailure=emergency.target when the image cannot fit, so Wants= with the existing After= keeps it ordered before the download without making the download depend on it. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the one-line note the pull check prints when it cannot check the size, and replace the "CANNOT REACH THE OS IMAGE SERVER" and "ANSWERED HTTP 404" rows, which the check no longer prints, with the systemd-importd reasons the failure summary shows for the same causes. Without a network the download, not the check, now fails after the 300 s wait, and a machine below the minimum RAM fails during the download when the server does not give the size. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks, fixed in 3a0c43b, cfce154 and f155a77. The check now fails only when it knows the image cannot fit. That covers any non-2xx final status (403/404 from signed URLs, 405/501, 5xx), a 2xx without A broken check cannot block either.
Tests (
Run against the helper at c58c77d, 10 of these fail, including 403/404/500. Without the subshell, only the malformed-meminfo test fails. Verified
|
* docs: replace stale Flatcar narrative with FSDK 26.08 facts The OS payload is now pure FSDK 26.08 (systemd v261). Update comments in shipped elements, scripts, and tests to remove Flatcar references and explain why each workaround still exists: - /etc/resolv.conf: FSDK ships no rule (upstream systemd leaves the symlink to the distro), and /etc is empty on every boot, so the tmpfiles.d rule is still required for kubelet pod sandboxes. - firstboot credentials: the helper exists because Bluefin applies firstboot.hostname unconditionally and sets the live kernel hostname, which systemd-firstboot does not do; it is not because systemd-firstboot is missing from the image. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: correct patch count in bump-fsdk-version.md patches/freedesktop-sdk/ carries six patches (0001-0006), not one. 0006 adds the Cilium/Kubernetes kernel options (VXLAN, GENEVE, tc BPF, conntrack/ss diagnostics); dropping it breaks the kubeadm and k0s sysexts. Update the verification checklist and the notes so a bumper does not drop it. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: remove unused Justfile build alias The build recipe was a forwarding alias to export-image. No doc, CI workflow, or script calls bare 'just build'. Remove it to match the avoid-over-engineering skill's rule against forwarding aliases. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: add booty-integration and secure-boot-keys skills booty-integration.md: how Booty serves releases (HTTP Boot, iPXE chainload, per-node bluefin-node.ign, doInstall, kubeadm workers). secure-boot-keys.md: what each key signs (PK/KEK/DB, module cert, sysupdate signing), where keys live, how CI gets them via BOOT_KEYS_TARBALL and SYSUPDATE_SIGNING_KEY, and rotation mechanics. Index rows are deferred until #273 merges (it owns docs/skills/index.md). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: stage system-container helper in the base image files/bin/system-container was documented as shipped at /usr/bin/system-container but no element staged it. Add os-system-container.bst to install it into /usr/bin and wire it into os-stack.bst. Remove the KNOWN_UNSTAGED waiver. machinectl and systemd-nspawn ship in the base image via FSDK's systemd component, so the helper's dependencies are present. Closes #154 Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolves the ddi-installer.md Common Rationalizations table against projectbluefin#273: keeps projectbluefin#273's ignored-karg note on the Ignition row and this branch's update/rollback rows. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…refusal Since projectbluefin#273 the initrd writes its failure summary to the console while systemd-importd logs the refusal, so DCS/OSC sequences can land inside the refusal line and refused() missed it (boot-test run 36560436141). Match against the same cleaned log the harness saves. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…kip image builds for docs-only changes (#270) * test(ignition): fixture that writes a file and enables a unit apply-marker.ign writes /etc/dogfood-ignition-marker and enables a oneshot unit through Ignition's preset; apply-marker.probe reports both from inside the booted guest, so a dogfood boot can assert that Ignition applied the config rather than only that the node booted. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: classify pull requests that cannot change the image set image-build-needed.py reads a PR's changed paths and answers false only for docs, Markdown outside the build inputs, unit/e2e tests and workflows that do not build the image. The unit test fails if any tracked file under a build input root (elements/, files/, include/, patches/, plugins/, scripts/, tests/fixtures/, project.conf, Justfile, build.yml) would skip the build. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(dogfood): honour DOGFOOD_PORT in dogfood-install.sh The update probe rewrote the sysupdate transfers to a hardcoded 10.0.2.2:8765, so any DOGFOOD_PORT other than the default served the image on one port while the guest updated from another. Export the port once and substitute it into the probe. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(dogfood): assert tamper refusal and probe expectations With DOGFOOD_TAMPER, --check now passes only if the initrd's pull fails after the image, SHA256SUMS and SHA256SUMS.gpg were served and nothing booted; a systemd.unit-dropin credential copies the pull's own messages to the serial console so the log says why. DOGFOOD_EXPECT=<ERE> makes --check also require a probe line (e.g. Ignition's marker). Robustness: key enrollment for HTTP boot waits (bounded) for the firmware to reset after systemd-boot enrolls instead of a fixed sleep, and keeps its log; the EXIT trap kills QEMU too, so an aborted run no longer leaves an orphaned guest holding the port and the caller's locks. The trap only kills PIDs that are set, since kill 0 signals the whole process group. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci(build): boot-test tamper refusal, HTTP boot, Ignition and A/B rollback; skip docs-only builds boot-test now also runs DOGFOOD_TAMPER=raw and =sums (must be refused), Ignition from the ignition.config credential and UEFI HTTP boot with bluefin-node.ign (both must show the fixture's file and enabled unit), and off main the full install -> sysupdate A->B -> boot-counted rollback. The build job exports the two higher-versioned sets it needs (<ver>.1, <ver>.2) from the warm cache; main skips them so no unpublished release-signed versions exist. Failed boot tests upload their serial logs. A changes job skips build and boot-test for pull requests image-build-needed.py classifies as docs/tests-only; skipped jobs report as passing. On main, an opt-in step pushes all artifacts to cache.projectbluefin.io:11002 when the CASD_CLIENT_CERT variable and CASD_CLIENT_KEY secret exist (the dakota identity), so main builds stop rebuilding the kernel; it is best effort and a no-op without them. The release job is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): CI boot coverage, build time and cache setup ci-tooling.md is canonical for what boot-test covers, the measured build-time breakdown (kernel on the critical path), the docs-only skip, the CASD_CLIENT_CERT / CASD_CLIENT_KEY cache push, and why PR kernels and actions/cache are out of scope. ddi-installer-build.md documents DOGFOOD_EXPECT/PORT/MEM/TIMEOUT, the tamper pass criterion, how to make next/broken sets, and points at ci-tooling.md for CI coverage. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(dogfood): make DOGFOOD_TAMPER=sums test the signature, assert the reason The sums tamper overwrote every hash in SHA256SUMS, so the pull failed on the checksum and the signature check was never exercised. It now serves the corrupted image with SHA256SUMS re-hashed to match it, which only SHA256SUMS.gpg can expose. --check requires the pull's own verdict on the serial console: 'DOWNLOAD INVALID: Checksum of <image> file did not check out' for raw, 'DOWNLOAD INVALID: Signature verification failed' for sums. Drop-ins for systemd-import@.service and systemd-importd.service (SYSTEMD_LOG_TARGET=console, journal+console) put those messages there. The EXIT cleanup moves into a function (shellcheck SC2154 on the inline trap). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci(build): version the update test sets 1.<run>.N systemd-sysinstall writes the installed image's boot entry as a Type #1 entry with 'version 1.<ver>' (bluefin-server-commit_1.<ver>.conf). systemd-boot sorts it above a 0.<run>.1 update, so the updated node kept booting 0.<run> and the A/B step failed (reproduced locally with 0.900 -> 0.900.1). 1.<run>.N sorts above both and below every release; with it the full install, A/B update and rollback pass locally. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): tamper reasons and update test versions Document which refusal each DOGFOOD_TAMPER mode asserts, and that next/broken sets must sort above the installed entry's 'version 1.<ver>'. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(dogfood): fail --check when Secure Boot is not enforced Firmware that refuses systemd-boot's key enrollment boots on in setup mode, where nothing is verified, and --check still passed. CI's boot-test did exactly that on Ubuntu's OVMF: every probe reported 'secureboot=disabled (setup)' (e.g. run 36499270842). --check now requires 'PROBE secureboot=enabled', and a tamper run requires the kernel's 'Secure boot enabled' line. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci(build): boot-test on Fedora's edk2 OVMF so Secure Boot is enforced Ubuntu 26.04's OVMF 2025.11 rejects systemd-boot's PK enrollment ('Failed to write PK secure boot variable: Security violation'), so boot-test never ran with Secure Boot on, and the new HTTP boot step, which enrolls explicitly, fails outright. Fetch Fedora's edk2-ovmf 20260508-6.fc44 from Koji, check its SHA-256, extract OVMF_CODE.secboot.fd / OVMF_VARS.fd with bsdtar and point the harness at them. It is the build dogfooding uses locally, where enrollment works. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): Secure Boot enforcement in boot-test and the OVMF pin Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci(build): drop the BuildStream cache push bst artifact push --deps all oci/bluefin-server-image.bst would upload bluefin-server/keys/boot-keys.bst, an import of files/boot-keys that on main holds the Secure Boot db, module-signing and sysupdate private keys, to a cache every build pulls from. Remove the push steps and the CASD_CLIENT_CERT/CASD_CLIENT_KEY configuration; ci-tooling.md records that pushing needs an element allow-list proven not to depend on boot-keys.bst. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci(build): run the build classifier from the PR's base revision The changes job checked out .github/scripts from the PR head, so a PR could edit image-build-needed.py to answer false and skip build and boot-test. Check out the base SHA instead, feed it the PR's changed files, and build when the base has no classifier yet. The classifier now lists itself and build.yml as gate files that always build; the unit test asserts that and that the changes job checks out the base. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(skills): the build classifier runs from the base revision Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(dogfood): strip OSC and DCS sequences before matching the tamper refusal Since #273 the initrd writes its failure summary to the console while systemd-importd logs the refusal, so DCS/OSC sequences can land inside the refusal line and refused() missed it (boot-test run 36560436141). Match against the same cleaned log the harness saves. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Motivation
Diskless boot failures were hard to diagnose and some looped with no clear message:
journalctl -p errto/dev/consoleand force-rebooted after 10 s./dev/consoleis only the lastconsole=on the netboot command line (ttyS0), so the monitor never showed the reason, and it scrolled away within seconds./run/machines, on a tmpfs that systemd caps at 20% of RAM. A small machine failed deep in the download./usrdevices timed out,systemd.default_device_timeout_sec=300later.ignition.config.url=and the otherignition.*kernel arguments were silently ignored.Change
Everything is systemd units and drop-ins plus one libexec helper,
files/initrd/usr/libexec/bluefin-boot-diagnostics. The helper uses only bash builtins, coreutils, curl, systemctl and journalctl, because the initrd has no sed or awk.Failure summary (
emergency.service.d/10-reboot.conf).failure-summarylists each failed unit with a one-line hint and what its own processes logged. For a failed download, that includes systemd-importd's reason. It also shows network link state when the network or download failed, any ignoredignition.*args, and the last errors. It writes to every active console (/sys/class/tty/console/active), then waits 60 s before the unattended reboot. Thebluefin.failure_delaycredential changes the delay (the kernel argument works only where the command line isn't locked). There is a 10 s floor so a failure can't turn into a hot loop.Type=idlekeeps the isolate's status lines out of the summary. The retry-by-reboot behavior is unchanged.RAM / reachability check before the pull. A drop-in makes every
systemd-import@.servicerequirebluefin-pull-check@%i.service. That unit takes the URL the import generator already resolved (from the unit'sDownload of <URL>description, so it coversrd.systemd.pull,bootoriginand theimport.pullcredential without deriving the URL a second time) and sends it an HTTP HEAD request. It fails withneeds ~X MiB RAM, have Yif the image plus 64 MiB won't fit in both/runandMemAvailable. It also fails with the URL and link state if the server can't be reached, or with the HTTP status on an error response.405/501and a missingContent-Lengthskip the check rather than block the boot. The same drop-in sends a failed download straight to the summary instead of waiting out the 300 s device timeout.Ignition guard rails.
bluefin-ignition-kargs.serviceprints a framed warning that names anyignition.*keys on/proc/cmdline(never their values) and says how to pass the config instead. A failedignition-<stage>.servicegets a hint that names the stage and points at the idempotency rules. The every-boot semantics are unchanged; there is no first-boot marker.Docs. New
docs/skills/diskless-troubleshooting.mdis the single source for the summary, minimum RAM, failure modes and their timing, getting logs, and the supported Ignition subset with its idempotency rules.ddi-installer.mdlinks to it instead of repeating it. It is also listed inindex.mdandAGENTS.md, andDOGFOOD_MEMis documented.Tests.
tests/unit/test_initrd_diagnostics.py(35 tests):/proc/meminfo, fake consoles, a fake sysfs net dir, and a localhttp.serveranswering HEAD (fits / too big for/run/ lowMemAvailable/ redirect / 404 / 405 / 501 / no Content-Length / connection refused);The UKI build also checks that the helper and the tools it runs are in the initrd.
Verification
python3 -m pytest tests/unit -q: 406 passed, run in a throwaway detached worktree with no boot keys.bats tests/unit: 74 ok.python3 .github/scripts/docs-checks.py: passed.flock … just validate: EXIT=0.shellcheck -S style: clean.flock /tmp/opencode/bst.lock just export-image. All QEMU runs usedflock /tmp/opencode/qemu.lockwith Secure Boot OVMF, except (c).(a) Normal diskless boot:
just dogfood-checkpasses(b) Low RAM (
DOGFOOD_MEM=2048, DDI 540 MiB)The check fails about 6 s into the boot, the summary follows, and the node reboots at 72 s:
The RAM boundary was checked on the first build of this branch, which had the same check code.
DOGFOOD_MEM=3072was refused (needs ~3072 MiB RAM, have 2816 MiB) andDOGFOOD_MEM=3584booted withPROBE failed=0.(c)
ignition.*kernel argumentsThe dogfood harness cannot add kernel arguments under Secure Boot, because the signed UKI seals the command line. So this ran through a non-Secure-Boot path: the same kernel, initrd and command line extracted from the netboot UKI, booted with
-kernel/-initrd/-appendon plain OVMF, withignition.config.url=http://…/secret-token.ign ignition.platform.id=metalappended. The unit tests also cover this. The boot reachedmulti-user.target, and the console showed:The token appears only in the kernel's own
Command line:echo, not in the warning.(d) Broken pull (
DOGFOOD_TAMPER=sums)The summary appears at 9 s, instead of after the 300 s device timeout:
Extra runs
Non-idempotent Ignition config (a file with
contentsat a path shipped in the factory/etc, run on the first build):Logs across boots (
var-on-disk.ignplus a state disk, booted twice):/varis on/dev/vdb1, and there is one journal directory per boot (a fresh machine ID each time). Plainjournalctl --list-bootsshows 1 boot,--merge --list-bootsshows 0, andjournalctl -D /var/log/journal --list-bootsshows 2. The docs recommend the last one.Pinning the machine ID through Ignition breaks things. An earlier attempt wrote
/etc/machine-idwith Ignition. That ended first-boot mode, so preset-enabled units (including the dogfood probe) never started. The docs now warn against it.Not done / notes
wait-onlinetimeout. The pull check's connection-failure branch and thewait-onlinehint are covered by the unit tests (connection refused, no carrier).system.machine_idcredential is untested. The docs name it as the way to pin the machine ID without losing first-boot mode. That comes from reading systemd v261 source (main.cdecides first boot from/etc/machine-idbeforemachine_id_setup()reads the credential); it was not run in QEMU./runwith a largersize=would cut the diskless minimum from about 3 GiB to roughly DDI + working set. This PR only diagnoses the limit.