Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions .github/workflows/reproducibility.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: Reproducibility

# Builds the image set, deletes the artifacts of the final assembly steps
# (images, UKIs, initrd, sysexts, signed systemd-boot) from the local
# BuildStream cache, rebuilds them without pulling from any remote, and
# compares every output file except the GPG signatures, which embed their
# signing time. Both builds share one checkout, one set of throwaway keys and
# one image-version, so any difference comes from the assembly itself.
#
# bluefin-server/keys/efi-keys.bst is kept: sbvarsign stamps the current time
# into each Secure Boot variable update it signs (PK/KEK/db.auth), the same
# way gpg timestamps SHA256SUMS.gpg, and has no option to fix it.
on:
schedule:
- cron: '0 9 * * 1'
workflow_dispatch:

permissions:
contents: read

concurrency:
group: reproducibility
cancel-in-progress: false

jobs:
reproducibility:
runs-on: ubuntu-26.04
timeout-minutes: 300
env:
FINAL_ASSEMBLY: >-
oci/bluefin-server-image.bst
oci/bluefin-server-boot.bst
oci/bluefin-server-usr.bst
oci/k0s-sysext.bst
oci/kubestellar-sysext.bst
oci/kubeadm-sysext.bst
oci/zfs-sysext.bst
bluefin-server/os-sd-boot-signed.bst
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up just
uses: taiki-e/install-action@4cef1412cce204788f482e778a0b9187f9626a29 # v2
with:
tool: just@1.58.0

- name: Set up runner disk space
run: |
set -euo pipefail
sudo mkdir -p /mnt/podman /mnt/buildstream
sudo chown -R "$USER:$USER" /mnt/podman /mnt/buildstream
mkdir -p ~/.config/containers
cat > ~/.config/containers/storage.conf <<EOF
[storage]
driver = "overlay"
runroot = "/run/user/1001"
graphroot = "/mnt/podman"
EOF
mkdir -p ~/.cache
ln -sf /mnt/buildstream ~/.cache/buildstream

# Throwaway keys and a 0.* version: nothing from this job is published.
- name: Generate keys and set the image version
run: |
set -euo pipefail
just gen-dev-keys
just set-version "0.${GITHUB_RUN_NUMBER}"

- name: First build
run: |
set -euo pipefail
just build-image
just bst artifact checkout oci/bluefin-server-image.bst --directory /src/dist/first
(cd dist/first && find . -type f ! -name '*.gpg' -exec sha256sum {} + | sort -k2) > first.sha256
rm -rf dist/first

- name: Rebuild the final assembly
run: |
set -euo pipefail
# shellcheck disable=SC2086
just bst artifact delete ${FINAL_ASSEMBLY}
just bst build --ignore-project-artifact-remotes oci/bluefin-server-image.bst
just bst artifact checkout oci/bluefin-server-image.bst --directory /src/dist/second
(cd dist/second && find . -type f ! -name '*.gpg' -exec sha256sum {} + | sort -k2) > second.sha256

- name: Compare
run: |
set -euo pipefail
cat first.sha256
if ! diff -u first.sha256 second.sha256; then
echo "::error::The rebuilt image set differs from the first build (files listed above)."
exit 1
fi
echo "Both builds produced identical files."
11 changes: 7 additions & 4 deletions docs/skills/bump-fsdk-version.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,11 @@ There is no application version for these images. Two version axes exist:

2. Update the `ref:` in `elements/freedesktop-sdk.bst` to the new tag/commit.

3. Re-check patches still apply — FSDK ships local patches under
`patches/freedesktop-sdk/`. If a release changed the patched files, refresh or
drop them. `just validate` surfaces patch failures.
3. Re-check the local patches in `patches/freedesktop-sdk/`
([`patches/README.md`](../../patches/README.md) says why each exists and
when to drop it). `just validate` surfaces a patch that no longer applies;
it does not catch 0003-0005 pointing at a library version the new ref no
longer builds, so compare those by hand (see the README).

4. Rebuild and verify:

Expand Down Expand Up @@ -76,7 +78,8 @@ Before merging a bump:
carries the Cilium/Kubernetes kernel options** (VXLAN, GENEVE, tc BPF,
conntrack/ss diagnostics). A bumper must not drop `0006` or the kubeadm and
k0s sysexts lose their datapath. If a release changed a patched file,
refresh the patch in place; never delete `0006` because "it looks small".
refresh the patch in place; never delete `0006` because "it looks small". Why each patch exists and when it can be dropped is in
[`patches/README.md`](../../patches/README.md).
- Junction overrides are only meaningful for components your local elements
reference directly. The 25 GNOME sdk/* overrides (cairo, gtk3, pango, glib,
gdk-pixbuf…) were dead weight — none of our `base-stack`, `brew-deps` etc. ever
Expand Down
1 change: 1 addition & 0 deletions docs/skills/ci-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ sudo_cmd := if `podman info >/dev/null 2>&1 && echo 1 || echo 0` == "1" { "" } e
| `boot-test` | `build.yml` | `pull_request`, `push/main`, `workflow_dispatch` | Runs the Secure Boot QEMU checks on the exported sets (see Core Process step 4). Read-only token. |
| `release` | `build.yml` | `push/main`, `workflow_dispatch` | Publishes `dist/diskless/` as-is: an immutable GitHub Release tagged `v<image-version>` plus an ORAS OCI artifact at `ghcr.io/<owner>/bluefin-server:<ver>,latest` (one layer per file, artifact type `application/vnd.projectbluefin.server.release.v1`) (`if: ${{ !failure() && !cancelled() && github.ref == 'refs/heads/main' }}`). `contents: write` + `packages: write`. |
| `docs` | `docs-checks.yml` | `pull_request`, `push/main` | Runs markdown and skill metadata checks via `docs-checks.py`. Read-only token. |
| `reproducibility` | `reproducibility.yml` | `schedule` (Mondays 09:00 UTC), `workflow_dispatch` | Builds the image set, deletes the final-assembly artifacts, rebuilds them without remote caches and diffs every output except `*.gpg` (see "Reproducible builds" in `ddi-installer-build.md`). Throwaway keys, nothing published. Read-only token. |
| `unit` | `unit-tests.yml` | `pull_request`, `push/main` | Runs pytest and BATS unit test suites. Read-only token. |
| `check`, `propose` | `track-binaries.yml` | `schedule` (08:30 UTC), `workflow_dispatch` | `check` finds the newest patch release in each pinned series of the upstream binaries pinned by version + sha256 (Kubernetes, cri-tools, containerd, runc, CNI plugins, k0s, ORAS) with `.github/scripts/track-binaries.py`; `propose` moves each version together with its sha256 pins, verified against upstream's checksum files and the downloaded assets, and opens or updates one PR per component on `auto/track-binaries/<component>`. Minor bumps stay manual (`kubeadm-sysext.md`, `k0s-sysext.md`). Read-only `GITHUB_TOKEN`; writes use the mergeraptor app token narrowed to `contents` + `pull-requests` (+ `workflows` for ORAS, pinned in `build.yml`). Never on `pull_request`. |

Expand Down
34 changes: 33 additions & 1 deletion docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,45 @@ with the module signing certificate, and the release `SHA256SUMS` with the
image signing key. `build-image` (and `validate`) depends on `gen-dev-keys`,
which generates throwaway keys in `files/boot-keys/` (gitignored) on first
run: PK/KEK/DB, the module certificate, and the `sysupdate-signing.asc` /
`import-pubring.pgp` pair. Keys are kept unless `--force` is given. CI builds
`import-pubring.pgp` pair. Keys are kept unless `--force` is given, and a
partial set (a file of a pair or of the boot set missing or empty) is an error
rather than something to fill in. CI builds
on `main` unpack the `BOOT_KEYS_TARBALL` secret, write `SYSUPDATE_SIGNING_KEY`
to `files/boot-keys/sysupdate-signing.asc`, and copy the committed release
keyring `files/os/sysupdate-keys/import-pubring.gpg` to
`files/boot-keys/import-pubring.pgp`; pull requests get throwaway keys and
their images are never published.

**Rotating any key needs a new `image-version`.** Every key ends up in the
image bits: DB signs the UKIs and systemd-boot, the module certificate is
built into the kernel, and `import-pubring.pgp` ships in `/usr`. An image
version names one immutable set of bits, and `systemd-sysupdate` only
installs a version newer than the one it runs, so rebuilding the same version
with new keys publishes different bits under a released name and never
reaches nodes already on it. Rotate keys (`just gen-dev-keys --force`, or new
CI secrets), then `just set-version` to a version that sorts higher before
building. Nodes also need the new Secure Boot keys enrolled and, for the
image signing key, the new keyring; see
"Rotating the Signing Key" in
[systemd-sysupdate-verification.md](systemd-sysupdate-verification.md).

## Reproducible builds

With the same checkout, keys and `image-version`, rebuilding the final
assembly gives the same bytes. BuildStream exports `SOURCE_DATE_EPOCH` into
every sandbox and the assembly steps honor it: `mkfs.erofs` and
`systemd-repart` clamp file times to it, the initrd and ESP trees are clamped
before `cpio` and `mcopy` copy them, and `systemd-sbsign` (not `sbsign`)
uses it as the signing time of systemd-boot and the UKIs. `systemd-repart
--seed` fixes partition UUIDs.

Two outputs carry a signing time of their own: `SHA256SUMS.gpg` and the
`efi-keys/*.auth` updates, which `sbvarsign` stamps with the current time
(they stay fixed as long as `bluefin-server/keys/efi-keys.bst` stays
cached). `.github/workflows/reproducibility.yml` checks the rest weekly:
it builds, deletes the final-assembly artifacts, rebuilds them without remote
caches, and compares every file except `*.gpg`.

## Dogfood: boot it in QEMU

All dogfood paths boot with Secure Boot firmware (OVMF secboot). The firmware
Expand Down
7 changes: 5 additions & 2 deletions docs/skills/systemd-sysupdate-verification.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,11 @@ proves both).
```
2. Update the GitHub Actions repository secret `SYSUPDATE_SIGNING_KEY` with the
new ASCII-armored private key.
3. Rebuild and publish a release. Existing hosts will only trust updates signed
by the new key, so plan the rotation around a release boundary.
3. Rebuild and publish a release under a new `image-version` (a key rotation
is never a rebuild of an existing version; see
"Keys" in [ddi-installer-build.md](ddi-installer-build.md)). Existing hosts only
trust updates signed by the key in their keyring, so plan the rotation
around a release boundary.

For a throwaway local signing key, `just gen-dev-keys` writes
`files/boot-keys/sysupdate-signing.asc` and `files/boot-keys/import-pubring.pgp`
Expand Down
7 changes: 5 additions & 2 deletions elements/bluefin-server/os-sd-boot-signed.bst
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ config:
set -eu
src=%{indep-libdir}/systemd/boot/efi/systemd-boot%{efi-arch}.efi
install -d "%{install-root}%{indep-libdir}/systemd/boot/efi"
sbsign --key /boot-keys/DB.key --cert /boot-keys/DB.crt \
--output "%{install-root}${src}.signed" "${src}"
# systemd-sbsign, unlike sbsign, takes the signing time from
# SOURCE_DATE_EPOCH, so the signature is reproducible.
%{indep-libdir}/systemd/systemd-sbsign sign \
--private-key=/boot-keys/DB.key --certificate=/boot-keys/DB.crt \
--output="%{install-root}${src}.signed" "${src}"
sbverify --cert /boot-keys/DB.crt "%{install-root}${src}.signed"
5 changes: 4 additions & 1 deletion elements/oci/bluefin-server-boot.bst
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,8 @@ config:
rm -rf usr/share/man usr/share/doc usr/share/info usr/include usr/lib/*/pkgconfig
find . -name '*.a' -delete
du -sh /initrd
# cpio records mtimes; everything created above carries the build time.
find . -mindepth 1 -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} +
find . -mindepth 1 -printf '%P\n' | LC_ALL=C sort | cpio -o -H newc --reproducible --quiet | zstd -19 -T0 -q > /tmp/initrd.cpio.zst
ls -lh /tmp/initrd.cpio.zst

Expand All @@ -189,7 +191,8 @@ config:
--cmdline=@"/tmp/$1.cmdline" \
--os-release=@/osrel/usr/lib/os-release \
--uname="$(ls /kmods/usr/lib/modules)" \
--signtool=sbsign \
--signtool=systemd-sbsign \
--no-sign-kernel \
--secureboot-private-key=/boot-keys/DB.key \
--secureboot-certificate=/boot-keys/DB.crt \
--output="%{install-root}/$1.efi"
Expand Down
11 changes: 11 additions & 0 deletions elements/oci/bluefin-server-image.bst
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ build-depends:
- base/base-stack.bst
- freedesktop-sdk.bst:public-stacks/runtime-gnu.bst
- freedesktop-sdk.bst:components/dosfstools.bst
- freedesktop-sdk.bst:components/findutils.bst
- freedesktop-sdk.bst:components/mtools.bst
- freedesktop-sdk.bst:components/systemd.bst
- filename: oci/bluefin-server-usr.bst
Expand Down Expand Up @@ -73,6 +74,12 @@ variables:
- arch == "aarch64":
efi-arch: aa64

environment:
# dosfstools 4.2 ignores SOURCE_DATE_EPOCH and stamps each ESP's volume
# label with the current time; --invariant fixes the label time and the
# volume ID.
SYSTEMD_REPART_MKFS_OPTIONS_VFAT: --invariant

config:
commands:
- mkdir -p /tmp /var/tmp
Expand All @@ -99,6 +106,8 @@ config:
printf '[Partition]\nType=usr\nLabel=bluefin_usr_%{image-version}\nUUID=%s\nCopyBlocks=/usr-image/%{image-name}.usr.raw\n' "$(cat /tmp/usr.uuid)" > "${d}/10-usr.conf"
printf '[Partition]\nType=usr-verity\nLabel=bluefin_usr_verity_%{image-version}\nUUID=%s\nCopyBlocks=/usr-image/%{image-name}.usr-verity.raw\n' "$(cat /tmp/usr-verity.uuid)" > "${d}/20-usr-verity.conf"
printf '[Partition]\nType=esp\nFormat=vfat\nCopyFiles=%s:/\nSizeMinBytes=128M\nSizeMaxBytes=128M\n' "${esp}" > "${d}/30-esp.conf"
# mcopy keeps mtimes; the staged ESP tree carries the build time.
find "${esp}" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} +
systemd-repart --empty=create --size=auto --offline=yes --dry-run=no \
--seed=6f1b0fe2-3c2a-4f2e-9f0a-b1e5e7a5e003 \
--definitions="${d}" "%{install-root}/%{image-name}.raw"
Expand All @@ -117,6 +126,7 @@ config:
d=/tmp/netboot.repart.d
mkdir -p "${d}"
printf '[Partition]\nType=esp\nFormat=vfat\nCopyFiles=%s:/\nSizeMinBytes=160M\n' "${esp}" > "${d}/10-esp.conf"
find "${esp}" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} +
systemd-repart --empty=create --size=auto --offline=yes --dry-run=no \
--seed=6f1b0fe2-3c2a-4f2e-9f0a-b1e5e7a5e002 \
--definitions="${d}" "%{install-root}/%{netboot-name}.esp.raw"
Expand All @@ -140,6 +150,7 @@ config:
printf '[Partition]\nType=usr\nLabel=bluefin-installer-usr\nCopyBlocks=/usr-image/%{image-name}.usr.raw\n' > "${d}/10-usr.conf"
printf '[Partition]\nType=usr-verity\nLabel=bluefin-installer-usr-verity\nCopyBlocks=/usr-image/%{image-name}.usr-verity.raw\n' > "${d}/20-usr-verity.conf"
printf '[Partition]\nType=esp\nLabel=bluefin-installer\nFormat=vfat\nCopyFiles=%s:/\nSizeMinBytes=256M\n' "${esp}" > "${d}/30-esp.conf"
find "${esp}" -exec touch -h -d "@${SOURCE_DATE_EPOCH}" {} +
systemd-repart --empty=create --size=auto --offline=yes --dry-run=no \
--seed=6f1b0fe2-3c2a-4f2e-9f0a-b1e5e7a5e004 \
--definitions="${d}" "%{install-root}/bluefin-server-installer_%{image-version}.raw"
Expand Down
37 changes: 37 additions & 0 deletions patches/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# patches/

`patch_queue` sources apply every file in a directory, in file name order,
on top of the junction they belong to. `just validate` fails if one no longer
applies.

## freedesktop-sdk/

Applied to the freedesktop-sdk junction (`elements/freedesktop-sdk.bst`).
Every patch that edits an FSDK element (0002-0006) changes that element's
cache key and the keys of everything that depends on it, so those artifacts
come from the Bluefin cache or a local build, never from FSDK's cache. Keep
the queue short.

| Patch | Why | Upstream status | Drop when |
|---|---|---|---|
| `0001-project.conf-Add-GNOME-CAS-servers.patch` | Adds the GNOME artifact and source cache to FSDK's own `project.conf`, so the junction pulls from it without per-user BuildStream config. Taken from gnome-build-meta. | Downstream by design (GNOME carries it too). | GNOME's cache stops serving FSDK 26.08 artifacts. gnome-50 builds on FSDK 25.08, so check the hit rate before the next FSDK minor. |
| `0002-glib-stage1-disable-tests.patch` | `-Dtests=false` for the bootstrap glib: building its tests ran the uninstalled `glib-compile-resources` before libgio existed and failed with exit 127 in from-source bootstrap builds (projectbluefin/server#50). | Not reported upstream. | A from-source build of `components/_private/glib-stage1.bst` passes without it. |
| `0003-gobject-introspection-base-library-symlinks.patch` | Remote-execution (BuildBarn) builds: creates the uninstalled SONAME links for `libgirepository-1.0.so.1` and sets `LD_LIBRARY_PATH` before ninja runs `g-ir-compiler` (projectbluefin/server#53). | Not reported upstream. | Builds no longer run under remote execution, or they pass without it. |
| `0004-gdk-pixbuf-library-symlinks.patch` | Same failure for `gdk-pixbuf-print-mime-types` and `libgdk_pixbuf-2.0.so.0` (projectbluefin/server#55). | Not reported upstream. | As 0003. |
| `0005-appstream-library-symlinks.patch` | Same failure for `appstreamcli` and `libappstream.so.5` / `libappstream-compose.so.0` in `appstream` and `appstream-minimal` (projectbluefin/server#57). | Not reported upstream. | As 0003. |
| `0006-linux-kubernetes-cilium-networking.patch` | Builds VXLAN, GENEVE, the tc BPF classifier/action, ingress qdisc, INET_DIAG (+TCP/UDP, DIAG_DESTROY) and `NOTRACK` for Cilium and the kubeadm/k0s sysexts. Written into `fdsdk-config.sh`, so FSDK's expected-config check fails the build if Kconfig drops one. | Not proposed upstream. | FSDK's kernel config enables them. |

### Hardcoded library versions in 0003-0005

These three patches name the full library file each link points at:

| Patch | Link target | Pinned component (FSDK 26.08.0) |
|---|---|---|
| 0003 | `libgirepository-1.0.so.1.0.0` | gobject-introspection 1.86.0 |
| 0004 | `libgdk_pixbuf-2.0.so.0.4400.7` | gdk-pixbuf 2.44.7 |
| 0005 | `libappstream.so.1.1.6`, `libappstream-compose.so.1.1.6` | appstream 1.1.6 |

An FSDK point release that bumps one of these components leaves the patch
applying cleanly but its links dangling, and remote builds fail again with
exit 127. On every FSDK bump, compare these names with the component
versions in the new FSDK ref and update the patch in the same change.
Loading