Repository navigation
ci: boot-test tamper refusal, HTTP boot, Ignition and A/B rollback; skip image builds for docs-only changes - #270
Conversation
apply-marker.ign writes /etc/dogfood-ignition-marker and enables a oneshot unit through Ignition's preset; apply-marker.probe reports both from inside the booted guest, so a dogfood boot can assert that Ignition applied the config rather than only that the node booted. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
image-build-needed.py reads a PR's changed paths and answers false only for docs, Markdown outside the build inputs, unit/e2e tests and workflows that do not build the image. The unit test fails if any tracked file under a build input root (elements/, files/, include/, patches/, plugins/, scripts/, tests/fixtures/, project.conf, Justfile, build.yml) would skip the build. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The update probe rewrote the sysupdate transfers to a hardcoded 10.0.2.2:8765, so any DOGFOOD_PORT other than the default served the image on one port while the guest updated from another. Export the port once and substitute it into the probe. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
With DOGFOOD_TAMPER, --check now passes only if the initrd's pull fails after the image, SHA256SUMS and SHA256SUMS.gpg were served and nothing booted; a systemd.unit-dropin credential copies the pull's own messages to the serial console so the log says why. DOGFOOD_EXPECT=<ERE> makes --check also require a probe line (e.g. Ignition's marker). Robustness: key enrollment for HTTP boot waits (bounded) for the firmware to reset after systemd-boot enrolls instead of a fixed sleep, and keeps its log; the EXIT trap kills QEMU too, so an aborted run no longer leaves an orphaned guest holding the port and the caller's locks. The trap only kills PIDs that are set, since kill 0 signals the whole process group. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…back; skip docs-only builds boot-test now also runs DOGFOOD_TAMPER=raw and =sums (must be refused), Ignition from the ignition.config credential and UEFI HTTP boot with bluefin-node.ign (both must show the fixture's file and enabled unit), and off main the full install -> sysupdate A->B -> boot-counted rollback. The build job exports the two higher-versioned sets it needs (<ver>.1, <ver>.2) from the warm cache; main skips them so no unpublished release-signed versions exist. Failed boot tests upload their serial logs. A changes job skips build and boot-test for pull requests image-build-needed.py classifies as docs/tests-only; skipped jobs report as passing. On main, an opt-in step pushes all artifacts to cache.projectbluefin.io:11002 when the CASD_CLIENT_CERT variable and CASD_CLIENT_KEY secret exist (the dakota identity), so main builds stop rebuilding the kernel; it is best effort and a no-op without them. The release job is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
ci-tooling.md is canonical for what boot-test covers, the measured build-time breakdown (kernel on the critical path), the docs-only skip, the CASD_CLIENT_CERT / CASD_CLIENT_KEY cache push, and why PR kernels and actions/cache are out of scope. ddi-installer-build.md documents DOGFOOD_EXPECT/PORT/MEM/TIMEOUT, the tamper pass criterion, how to make next/broken sets, and points at ci-tooling.md for CI coverage. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… reason The sums tamper overwrote every hash in SHA256SUMS, so the pull failed on the checksum and the signature check was never exercised. It now serves the corrupted image with SHA256SUMS re-hashed to match it, which only SHA256SUMS.gpg can expose. --check requires the pull's own verdict on the serial console: 'DOWNLOAD INVALID: Checksum of <image> file did not check out' for raw, 'DOWNLOAD INVALID: Signature verification failed' for sums. Drop-ins for systemd-import@.service and systemd-importd.service (SYSTEMD_LOG_TARGET=console, journal+console) put those messages there. The EXIT cleanup moves into a function (shellcheck SC2154 on the inline trap). Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-sysinstall writes the installed image's boot entry as a Type projectbluefin#1 entry with 'version 1.<ver>' (bluefin-server-commit_1.<ver>.conf). systemd-boot sorts it above a 0.<run>.1 update, so the updated node kept booting 0.<run> and the A/B step failed (reproduced locally with 0.900 -> 0.900.1). 1.<run>.N sorts above both and below every release; with it the full install, A/B update and rollback pass locally. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Document which refusal each DOGFOOD_TAMPER mode asserts, and that next/broken sets must sort above the installed entry's 'version 1.<ver>'. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
left a comment
There was a problem hiding this comment.
Reviewed at 0f718b3. One blocking issue and one smaller note. The rest checks out.
Blocking: the opt-in cache push would upload the signing keys
.github/workflows/build.yml:196 runs just bst artifact push --deps all oci/bluefin-server-image.bst. --deps all includes build dependencies. elements/oci/bluefin-server-image.bst:52 (plus kernel-modules.bst, efi-keys.bst, os-sd-boot-signed.bst, bluefin-server-boot.bst and openzfs-signed.bst) build-depends on bluefin-server/keys/boot-keys.bst, which is kind: import of the whole files/boot-keys directory. On main this same job fills that directory from BOOT_KEYS_TARBALL and writes SYSUPDATE_SIGNING_KEY to files/boot-keys/sysupdate-signing.asc (build.yml:93-107). The element's own description says "Never add this element to a runtime stack: its artifact holds private keys."
So once CASD_CLIENT_CERT/CASD_CLIENT_KEY are configured, the first main build pushes an artifact containing the Secure Boot db key, the module-signing key and the sysupdate GPG key to cache.projectbluefin.io:11002. project.conf lists that host as a pull remote for every build. The artifact's cache key isn't secret either: it appears in the dependency metadata of every artifact that build-depends on it. Please exclude the key element from the push, for example by pushing only the elements that don't depend on the keys, or with --deps scoped to runtime plus an explicit list. Or keep push off until the cache has a private-artifact story.
Note: the classifier comes from the PR head
The changes job sparse-checks out .github/scripts from the PR head and runs image-build-needed.py from there, so a PR can edit the classifier to answer false and skip build/boot-test. A PR can already do the same by editing build.yml, so this doesn't open a new bypass. It is less visible in a diff, though, and nothing in the main ruleset requires build. If you want the gate to hold, run the classifier from the base revision and treat a change to the classifier itself as needing a build (the unit test that asserts this runs from the PR too).
What I verified
- The classifier (
image-build-needed.py):BUILD_PREFIXESis checked before the.mdexemption, sofiles/**/*.mdorelements/**changes still build. Anything unlisted builds, and an empty or 3000-file-truncated list builds. Renames are listed under both names. I found no false-negative path forelements/,files/,include/,patches/,plugins/,scripts/ortests/fixtures/. - The earlier "no build check" state was only the new
changesjob gatingbuild. At this head,changespasses andbuildis running (run 36521847750). pytest tests/unit: 389 passed.bats tests/unit: pass.actionlinton the workflows: clean apart from the unknownubuntu-26.04label in my local actionlint.shellcheck -S warning scripts/dogfood-install.sh: clean. The SC2054 warnings inscripts/dogfood-diskless.shexist on main too.
Firmware that refuses systemd-boot's key enrollment boots on in setup mode, where nothing is verified, and --check still passed. CI's boot-test did exactly that on Ubuntu's OVMF: every probe reported 'secureboot=disabled (setup)' (e.g. run 36499270842). --check now requires 'PROBE secureboot=enabled', and a tamper run requires the kernel's 'Secure boot enabled' line. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Ubuntu 26.04's OVMF 2025.11 rejects systemd-boot's PK enrollment ('Failed to write PK secure boot variable: Security violation'), so boot-test never ran with Secure Boot on, and the new HTTP boot step, which enrolls explicitly, fails outright. Fetch Fedora's edk2-ovmf 20260508-6.fc44 from Koji, check its SHA-256, extract OVMF_CODE.secboot.fd / OVMF_VARS.fd with bsdtar and point the harness at them. It is the build dogfooding uses locally, where enrollment works.
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
bst artifact push --deps all oci/bluefin-server-image.bst would upload bluefin-server/keys/boot-keys.bst, an import of files/boot-keys that on main holds the Secure Boot db, module-signing and sysupdate private keys, to a cache every build pulls from. Remove the push steps and the CASD_CLIENT_CERT/CASD_CLIENT_KEY configuration; ci-tooling.md records that pushing needs an element allow-list proven not to depend on boot-keys.bst. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The changes job checked out .github/scripts from the PR head, so a PR could edit image-build-needed.py to answer false and skip build and boot-test. Check out the base SHA instead, feed it the PR's changed files, and build when the base has no classifier yet. The classifier now lists itself and build.yml as gate files that always build; the unit test asserts that and that the changes job checks out the base. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolve scripts/dogfood-diskless.sh: keep both the tamper-run import console drop-ins and upstream's DOGFOOD_CREDS system credentials. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks for the review, @castrojo. Addressed at 75824d2: Blocking: the cache push would upload the signing keys. Fixed in 1e37fa0 ( Note: the classifier comes from the PR head. Fixed in b93e88a (
Merge. Main had moved (#272, #275) and the PR was Checks run on the merge head:
|
Both checked out pull_request.head.ref, the branch, so a push landing mid-run could pair release-dry-run's scripts with artifacts built from another commit. Check out head.sha on pull requests and github.sha otherwise, the commit release passes to gh release --target. boot-test keeps head.ref: projectbluefin#270 rewrites that job. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Combines projectbluefin#271's rollback scenarios (DOGFOOD_BROKEN, boot deadline) with this branch's DOGFOOD_PORT handling, the 1.<ver> version-order note and the tamper/RAM harness docs. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…refusal Since projectbluefin#273 the initrd writes its failure summary to the console while systemd-importd logs the refusal, so DCS/OSC sequences can land inside the refusal line and refused() missed it (boot-test run 36560436141). Match against the same cleaned log the harness saves. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Takes projectbluefin#270's changes/build/boot-test rows in the ci-tooling.md job table and keeps this branch's release and release-dry-run rows. build.yml merged cleanly; the update sets go to the separate image-update artifact, so the release set that publish-release.sh verifies is unchanged. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Runs the offline installer test after projectbluefin#270's update and rollback step, and lists it in ci-tooling.md's boot-test scenarios instead of the old table row. projectbluefin#268, which this was stacked on, is now in main. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Summary
Part A: boot coverage (
boot-test)The QEMU harness already did more than CI ran.
boot-testnow also runs:DOGFOOD_TAMPER=raw)DOWNLOAD INVALID: Checksum of <image> file did not check out, after the DDI,SHA256SUMSandSHA256SUMS.gpgwere served (so it is not a transport error). Nothing boots, and the kernel ran with Secure Boot enabled.systemd.unit-dropincredentials forsystemd-import@.serviceandsystemd-importd.service(SYSTEMD_LOG_TARGET=console,journal+console) put importd's messages on the console.DOGFOOD_TAMPER=sums)sumsmode overwrote every hash, so it failed on the checksum and never exercised the signature check. It now serves the corrupted DDI withSHA256SUMSre-hashed to match, which is what an attacker without the key can do. The pull reportsSHA256 checksum ... is valid, thengpg: BAD signature, thenDOWNLOAD INVALID: Signature verification failed, and the test asserts that last line.tests/fixtures/ignition/apply-marker.ignwrites/etc/dogfood-ignition-markerand enables a oneshot unit. The probe must printPROBE ignition marker=applied unit=active enabled=enabled ran=yes(newDOGFOOD_EXPECT).bluefin-node.ignDOGFOOD_BOOT=httpplus the same fixture served next to the UKI. This covers HTTP boot and the StubDeviceURL →bluefin-node.ignIgnition path.main)dogfood-install.sh dist/diskless dist/diskless-next dist/diskless-brokenEvery
--checknow also fails unless Secure Boot was enforced (PROBE secureboot=enabled). See "Findings" below: until now CI never booted with Secure Boot on.1.<run>.1and1.<run>.2right after the main set (just set-version+just export-image, warm cache, only 13 version-stamped elements rebuild). Locally each extra set took 129 s and 131 s including checkout. On CI the version-dependent elements were ~3–10× slower than locally in run 36499270842, so I estimate ~5–8 min per set, ~15 min in total. The installer raw is excluded from the extra upload.1.<run>.Nand not0.<run>.N: see Findings.maindoes not run this. Its extra sets would be release-signed versions that nobody publishes, and every change reachesmainthrough a PR that ran it.mainkeeps install + disk boot.dogfood-install.shhardcoded10.0.2.2:8765and ignoredDOGFOOD_PORT; fixed.sleep 2. It now does a bounded wait for the firmware reset that follows enrollment, and keepsdogfood-enroll.log.qemu.lock. The trap only kills PIDs that are set, becausekill 0signals the whole process group.Findings the new coverage surfaced
ovmf2025.11 rejects systemd-boot's PK enrollment (secure_boot_enroll_at: Failed to write PK secure boot variable: Security violation), and the guest boots on in setup mode. Run 36499270842's probes all saysecureboot=disabled (setup).boot-testnow uses Fedora'sedk2-ovmf-20260508-6.fc44(Koji URL + SHA-256 pin, extracted withbsdtar). That is byte-identical to the local build where enrollment works, and the harness now fails when Secure Boot is off. Verified locally: with Ubuntu's OVMF the new check fails withFAIL: booted without Secure Boot (PROBE secureboot=disabled (setup)).PK.auth(sbvarsignfrom FSDK's sbsigntools). Secure boot auto enrollment stopped working at some point systemd/systemd#34316 traced the same symptom to sbsigntools built against a gnu-efi whereCHAR16becamewchar_t. Real firmware may reject the auto-enroll payloads the same way.0.<run>.1update keeps booting the old version.systemd-sysinstallwrites the installed image as a Type ci: delegate server builds to lab and add daily release dispatch #1 entrybluefin-server-commit_1.<ver>.confwithversion 1.<ver>. systemd-boot sorts1.0.900above the sysupdate UKI0.900.1, so the node kept booting 0.900 (boot-entry=bluefin-server-commit_1.0.900.conf, UKIbluefin-server-0.900.1+3-0.efiuntried). Release versions (26.09.x, above1.26.09.x) are unaffected, so CI uses1.<run>.N./bluefin-server/bluefin-server-<ver>.efioutsideEFI/Linux, so sysupdate never prunes it. Once slot A is recycled it becomes a stale fallback entry.Part B: CI speed
Diagnosis. Run 36499270842 (PR, cold runner):
Build and export the image settook 23:41 → 01:41. BuildStream pulled 239 artifacts and built 85, about 3.9 h of build time over 4 cores.components/linux.bst. Its source fetch took 6 min (it is in no source cache) and the build took 1 h 43 min (23:50 → 01:33). Then kernel-modules (3.5 min), boot (2 min) and image (1 min).components/linux-module-cert.bstjunction override (our module cert) andpatches/freedesktop-sdk/0006-linux-*change its cache key, so FSDK's caches never hold it.What changed:
main — review policyruleset (id 22705159) has no required status checks, and there is no classic branch protection. I still used achangesjob rather thanpaths-ignore: skipped jobs report as passing, sobuild/boot-testcan become required later without leaving PRs pending. The job lists the PR's files through the API (renames under both names) and feeds them to.github/scripts/image-build-needed.py. Pushes and dispatches always build.tests/unit/test_image_build_needed.pyguards the invariant: every tracked file under a build-input root, plusproject.conf,Justfile,build.ymlandcheck-*.py, must build.main. It uses the same mTLS identityprojectbluefin/dakotauses: repository variableCASD_CLIENT_CERTand secretCASD_CLIENT_KEY, againsthttps://cache.projectbluefin.io:11002. When both exist, the job writes a global BuildStreamartifactsremote (push: true,override-project-caches: false) and runsbst artifact push --deps all oci/bluefin-server-image.bstafter the upload. That step iscontinue-on-error. Without the credentials it does nothing, and PRs never push. Neither exists in this repo yet; a maintainer has to add them. Once they do, main builds pull the kernel as long as its inputs and the release module cert are unchanged.gen-dev-keyscreates a new module cert on every run, so the PR kernel key never matches a cached one. Fixing that needs a stable, non-release PR module key pair (fork PRs get no secrets) plus a job that pushes that kernel. That is a key-handling decision for maintainers.actions/cache. Only the kernel source fetch (6 min) sits on the critical path, and only when the kernel rebuilds anyway. The runner cache would be several times the 2.1 GB image set, which churns the 10 GB quota, and BuildStream 2 cannot import a single artifact from a tarball. Local~/.cache/buildstream/sourcesis 894 MB.The release job is untouched; another PR is changing it.
Verification
python3 -m pytest tests/unit -q: 389 passed (throwaway detached worktree, no boot-keys).bats tests/unit: 74 ok.docs-checks.pypassed.flock bst.lock just validatepassed.ubuntu-26.04unknown-label warning.flock qemu.lock) of exactly the CI scenarios, with the final scripts on the pinned Fedora OVMF. Image sets were0.900,1.900.1and1.900.2, built in this worktree:just set-version+just export-image): 129–135 s each.Assisted-by: Claude Opus 5.5 via GitHub Copilot