Skip to content

ci(release): dry-run the publish path on pull requests; attest provenance and publish an SBOM - #269

Merged
jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:ci/release-dry-run-and-provenance
Sep 29, 2026
Merged

jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:ci/release-dry-run-and-provenance

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

The first release from main (run 36482792704) failed at the oras step, because that code had never run before merge. This PR runs the publish path on every pull request, adds provenance attestations, and ships an SBOM.

One publish path: scripts/publish-release.sh

  • verify DIR VER KEYRING: runs gpgv against the keyring and checks that every SHA256SUMS entry is present and matches (sha256sum --check --strict). It also checks that every top-level file a release would publish is listed, that each file name belongs to VER (exactly one file per expected kind; k0s has its own version axis), and that the SBOM is SPDX 2.3 with unique SPDXIDs and a creation time. It writes the attestation subjects file.
  • release DIR VER [--dry-run]: gh release create v<ver> --target <sha> (the --target is new: before, the tag landed on whatever main's HEAD was at the time). With --dry-run it prints the command instead of running it.
  • oci DIR VER REF [--plain-http] [--pull-back]: oras push (same artifact type and annotations, plus revision), then checks that :latest resolves to the same digest and compares the registry manifest's layer titles and digests with the local files. --pull-back also pulls the artifact and re-checks it.

CI (build.yml; only the release job and a new job changed)

  • release (main): checks out the repo, verifies against the committed files/os/sysupdate-keys/import-pubring.gpg, attests, creates the release, pushes to OCI, attests the digest. Adds id-token: write, attestations: write and artifact-metadata: write.
  • release-dry-run (new, pull_request, needs: build): same verify / release --dry-run / oci commands against a registry:3.1.2@sha256:… service container with --plain-http --pull-back. Only contents: read, no secrets. Uses the same setup-oras pin; tests/unit/test_release_workflow.py enforces that the pins match and that both jobs run the same commands.
  • Attestations: actions/attest@v4.2.2, pinned by SHA. attest-build-provenance v4 is now a thin wrapper around it, and upstream recommends it for new code. It runs four times: provenance and SBOM over every published file (subject-checksums), and provenance and SBOM over the OCI digest with push-to-registry: true.
  • No cosign signature: with push-to-registry, the Sigstore bundle is already stored in ghcr.io as an OCI referrer and is signed by the same keyless workflow identity. cosign verify-attestation --new-bundle-format verifies it, so a cosign sign would only repeat that identity.

SBOM

  • FSDK's collect_manifest plugin (buildstream-plugins-community, already our junction) emits SPDX 2.3 directly (output-type: spdx), so no conversion script is needed.
  • oci/bluefin-server-sbom.bst covers the runtime closure of the payload: the os and initrd stacks, the kernel, the k0s and kubeadm binaries, OpenZFS, and the KubeStellar sysext.
  • The image element stamps creationInfo.created and dedupes packages that appear twice with identical content: FSDK bootstrap elements are reached through two junction paths.
  • The image element ships the SBOM as bluefin-server_<ver>.spdx.json, inside SHA256SUMS, so the GPG signature covers it.
  • The image element also exports the keyring it trusts as sysupdate-keys/import-pubring.pgp. This is how the dry run checks a PR's throwaway key. Like efi-keys/, the directory is not published, because releases carry top-level files only.

Docs

  • systemd-sysupdate-verification.md now covers provenance, the SBOM and the verify commands (gh attestation verify …, cosign verify-attestation …), and fixes the dangling signing-and-sbom reference.
  • ci-tooling.md gets the job table, permissions and checklist updates.
  • README: one pointer line. AGENTS.md: one release-set line.

Verification (local)

  • Local export of 26.09.2 in this branch (just export-image) plus the exact CI commands against registry:3.1.2 (pinned digest) in podman:
    $ publish-release.sh verify dist/diskless 26.09.2 dist/diskless/sysupdate-keys/import-pubring.pgp
    gpgv: Good signature from "Bluefin Server dev image signing (...)"
    release set 26.09.2: 12 files match SHA256SUMS, signature verified
    $ publish-release.sh release dist/diskless 26.09.2 --dry-run
    gh release create v26.09.2 --target 8782952f… --title Bluefin\ Server\ 26.09.2 … dist/diskless/SHA256SUMS … dist/diskless/zfs_26.09.2.raw.zst
    $ publish-release.sh oci dist/diskless 26.09.2 localhost:5055/bluefin-server --plain-http --pull-back
    Pushed [registry] localhost:5055/bluefin-server:26.09.2 … Tagged latest
    pulled localhost:5055/bluefin-server@sha256:0e4a847c… back: 14 files match
    
    The same set is refused with the release keyring ("does not verify") and with the wrong version ("… is not a file of release 26.09.3").
  • SBOM: 183 SPDX packages from 114 elements (git_repo 95, local 39, tar 23, patch/patch_queue 33, git_module 8, remote 5). Examples: linux 7.2.2, systemd 261.2, glibc 2.44, gnupg 2.4.9, k0s 1.36.4, kubelet 1.34.3, containerd 2.1.5, zfs 2.4.4.
  • Other checks: just validate passes; pytest 388 passed in a detached worktree without boot-keys; bats has no failures; docs-checks.py passes; shellcheck is clean; actionlint shows only the existing ubuntu-26.04 label warning.

jeefy and others added 4 commits September 28, 2026 21:54
oci/bluefin-server-sbom.bst runs FSDK's collect_manifest plugin
(output-type: spdx) over the runtime closure of the release payload: the
/usr and initrd stacks, the kernel, and the k0s, KubeStellar, kubeadm and
OpenZFS sysext payloads. The image element stamps creationInfo.created,
drops the identical duplicate packages FSDK bootstrap elements produce,
and lists bluefin-server_<ver>.spdx.json in SHA256SUMS so the GPG
signature covers it.

It also exports the keyring the build trusts as
sysupdate-keys/import-pubring.pgp (not published, like efi-keys/) so a
pull request's release dry run can check SHA256SUMS.gpg against the
throwaway key.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
One publish path for main and for the pull-request rehearsal:
verify (gpgv against a keyring, every SHA256SUMS entry present and
matching, every published file listed, file names match the version,
SBOM is valid SPDX 2.3), release (gh release create at the built
commit, or print it with --dry-run), and oci (oras push, then compare
the registry manifest layer by layer with the local files; optionally
pull it back).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ance and SBOM

The release job now calls scripts/publish-release.sh for every publish
step and verifies the set against the committed release keyring first.
It attests SLSA build provenance and the SPDX SBOM for every published
file and for the OCI artifact digest (push-to-registry, so the Sigstore
bundles live next to the artifact as referrers).

release-dry-run runs the same verify, release --dry-run and oci commands
on every pull request, pushing to a digest-pinned registry service
container and pulling the artifact back. Read-only token, no secrets.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
systemd-sysupdate-verification.md owns the trust story and the
gh attestation / cosign verify commands, replacing the dangling
signing-and-sbom reference; ci-tooling.md lists the release-dry-run job
and the release job's new permissions.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo
castrojo previously approved these changes Sep 29, 2026

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at e31529b. I read the full diff, ran the local checks and read CI.

What I verified:

  • Token scope. release-dry-run runs only on pull_request with permissions: contents: read, no secrets, a persist-credentials: false checkout, and a digest-pinned registry:3.1.2 service on localhost. id-token, attestations, artifact-metadata and the write scopes are only on release, which is still gated on github.ref == 'refs/heads/main'. On main, verify checks against the committed files/os/sysupdate-keys/import-pubring.gpg, not the build's own copy.
  • scripts/publish-release.sh. verify runs gpgv over SHA256SUMS, then sha256sum --check --strict. It rejects duplicate and unlisted top-level files, requires exactly one file for each expected release pattern, and checks that the SBOM is SPDX-2.3 with unique SPDXIDs before emitting attestation subjects. release keeps the publish-once behaviour (gh release create refuses an existing tag) and now pins --target to the built SHA. oci checks that :<ver> and :latest resolve to the same digest and that the manifest layers match the local files byte for byte.
  • CI on this head. build, boot-test, release-dry-run, unit and docs all pass (runs 36514803017, 36514802994, 36514802942). The dry-run log shows release set 0.680: 12 files match SHA256SUMS, signature verified, the rendered gh release create command, and pulled localhost:5000/bluefin-server@sha256:370fdde… back: 14 files match.
  • Local. pytest tests/unit: 388 passed. bats tests/unit: pass. shellcheck -S warning scripts/publish-release.sh: clean. actionlint .github/workflows/*.yml: clean apart from my local actionlint not knowing the ubuntu-26.04 label.
  • SBOM dependency walk. I checked against the pinned buildstream-plugins-community 2.3.1 collect_manifest.py and BuildStream's ElementProxy.dependencies. The plugin walks the listed build-depends, then only runtime dependencies below them, so zfs/openzfs.bst's autotools build stack does not leak into the SBOM.

Non-blocking follow-ups:

  • elements/oci/bluefin-server-sbom.bst:27-30 lists k0s/k0s-bin.bst, kubeadm/kubeadm-bin.bst and zfs/openzfs.bst rather than the published oci/{k0s,kubeadm,zfs}-sysext.bst. So the local sources those sysexts ship (files/k0s/sysext, files/kubeadm/sysext, files/zfs/sysext) are not listed, while oci/kubestellar-sysext.bst is listed directly. The upstream binaries themselves are covered. Adding the sysext elements next to the -bin ones would make the "sysext payloads" line in the description exact.
  • release-dry-run checks out pull_request.head.ref (the branch name), the same as build, so a push landing mid-run could pair the scripts from one commit with the artifacts from another. Using head.sha in both jobs would close that gap, but the existing build job already behaves this way.
  • This PR and #270 both edit build.yml; whichever lands second will need a rebase.

A wall-clock creation time made every rebuild of the release set differ
and defeats the reproducibility check proposed in projectbluefin#281.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy and others added 2 commits September 29, 2026 02:01
The SBOM listed k0s-bin, kubeadm-bin and openzfs but not the sysexts
that ship them, so their local sources (files/{k0s,kubeadm,zfs}/sysext)
were missing, while the KubeStellar sysext was listed directly. A sysext
only build-depends on its payload, which collect_manifest does not
follow, so both stay listed: 183 -> 186 SPDX packages.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Both checked out pull_request.head.ref, the branch, so a push landing
mid-run could pair release-dry-run's scripts with artifacts built from
another commit. Check out head.sha on pull requests and github.sha
otherwise, the commit release passes to gh release --target.
boot-test keeps head.ref: projectbluefin#270 rewrites that job.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy

jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks @castrojo. Your review was dismissed by my later push, so here is what changed since e31529b. Head is now 941edb7.

ffa818c: SBOM timestamp from SOURCE_DATE_EPOCH. creationInfo.created now comes from BuildStream's fixed SOURCE_DATE_EPOCH (2011-11-11T11:11:11Z) instead of the build's wall clock. Rebuilding the same set no longer changes the SBOM.

8c2ff71: the SBOM lists the sysext elements. elements/oci/bluefin-server-sbom.bst now lists oci/{k0s,kubeadm,zfs}-sysext.bst, each next to the element it stages from. k0s/k0s-bin.bst, kubeadm/kubeadm-bin.bst and zfs/openzfs.bst stay listed, because a sysext only build-depends on them and collect_manifest does not follow that. The element description now says so.

  • Local rebuild of 26.09.2 (just export-image): 183 → 186 SPDX packages from 114 → 117 elements. The three new packages are exactly the sysexts' local sources, and nothing else changed:
    SPDXRef-bluefin-server-oci-k0s-sysext-0      sysext  files/k0s/sysext
    SPDXRef-bluefin-server-oci-kubeadm-sysext-0  sysext  files/kubeadm/sysext
    SPDXRef-bluefin-server-oci-zfs-sysext-0      sysext  files/zfs/sysext
    
  • The signature still verifies against the exported dev keyring:
    $ scripts/publish-release.sh verify dist/diskless 26.09.2 dist/diskless/sysupdate-keys/import-pubring.pgp
    gpgv: Good signature from "Bluefin Server dev image signing (...)"
    release set 26.09.2: 12 files match SHA256SUMS, signature verified
    
  • New test_sbom_lists_every_published_sysext_and_its_payload in tests/unit/test_build_depends.py: every *-sysext.bst the image element stages must be in the SBOM, together with the three payload elements.

941edb7: build and release-dry-run check out the commit, not the branch. Both jobs now use ref: ${{ github.event.pull_request.head.sha || github.sha }}. On push and dispatch this is github.sha instead of github.ref, so a main build is the same commit that release passes to gh release create --target. test_build_and_dry_run_check_out_the_triggering_commit checks that both jobs use this expression.

main. git merge-tree against current main (fd34c54) is clean, so I did not add a merge commit.

Checks.

  • pytest tests/unit: 390 passed, also in a detached worktree without files/boot-keys. Both new tests fail against the pre-change files.
  • bats tests/unit: 74 ok.
  • docs-checks.py passes, and just validate passes.
  • actionlint: only the ubuntu-26.04 runner-label warning.

@jeefy
jeefy requested a review from castrojo September 29, 2026 07:02
Takes projectbluefin#270's changes/build/boot-test rows in the ci-tooling.md job table
and keeps this branch's release and release-dry-run rows. build.yml merged
cleanly; the update sets go to the separate image-update artifact, so the
release set that publish-release.sh verifies is unchanged.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 7a42b26.

correctness

  • elements/oci/bluefin-server-image.bst:167 stamps creationInfo.created from ${SOURCE_DATE_EPOCH}, but inside the BuildStream sandbox that variable is the fixed default 1321009871 (BuildStream projectconfig.yaml; project.conf sets no environment: override). Every SBOM will therefore carry created: 2011-11-11T11:11:11Z. The PR body and docs/skills/systemd-sysupdate-verification.md:116 describe this as stamping a creation time, and scripts/publish-release.sh:121 only checks that the field is a string, so nothing catches it. Either feed a real timestamp (e.g. via spdx-info.created, which the plugin already supports, or derive it from the version) or document that the date is intentionally constant for reproducibility.

docs-currency

  • docs/skills/systemd-sysupdate-verification.md:119 says scripts/publish-release.sh "is the only publish path", but Justfile:138 (publish-oci) still runs its own oras push without the new revision annotation or post-push checks, and docs/skills/ddi-installer.md / AGENTS.md still point at it. Either route the recipe through the script or narrow the claim to CI.

No findings from: security, intent-alignment, style.

Confidence: 3/5 (needs attention) — 1 perspective requested changes

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy
jeefy merged commit a8e5d35 into projectbluefin:main Sep 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants