Skip to content

build: reproducible image assembly, key-set safety, patch docs - #281

Merged
jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:build/reproducibility-and-keys
Sep 29, 2026
Merged

jeefy merged 8 commits into
projectbluefin:mainfrom
jeefy:build/reproducibility-and-keys

Conversation

@jeefy

@jeefy jeefy commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

5. Key rotation

  • Overwrite check. scripts/gen-dev-keys.sh already kept a complete key set without --force. But it only checked DB.key and sysupdate-signing.asc:

    • With DB.key missing, it regenerated PK, KEK and the module certificate over the files that still existed.
    • With the signing key missing, it replaced import-pubring.pgp.

    Both sets are now checked file by file, and an empty file counts as missing. A partial set exits 1 before anything is written. CI is unaffected: main unpacks a complete set and PRs start from none.

  • Tests. tests/unit/test_gen_dev_keys.py runs the script against a scratch tree with openssl/gpg stubbed:

    • a complete set is left untouched;
    • each partial set is refused without calling either tool;
    • an empty file counts as missing;
    • --force still regenerates.

    Against the old script, 6 of the 8 cases fail.

  • Docs. The canonical rule is in ddi-installer-build.md → "Keys": rotating any key needs a new image-version. Every key ends up in the image bits, versions are immutable, and systemd-sysupdate never re-installs the running version. systemd-sysupdate-verification.md step 3 links to it.

6. Reproducibility

  • What was already reproducible. BuildStream exports SOURCE_DATE_EPOCH=1321009871 into every sandbox. /usr was already reproducible: rebuilding oci/bluefin-server-usr.bst gave the same usrhash and byte-identical usr.raw/usr-verity.raw, both at 26.09.2 and at a scratch version.
  • What leaked. Building the full set twice, with the final-assembly artifacts deleted in between and remote caches ignored, showed these differences:
    • Initrd: file mtimes, which cpio --reproducible keeps. Now clamped to SOURCE_DATE_EPOCH before cpio.
    • systemd-boot and the UKIs: sbsign writes the current time into each signature's signingTime. Now signed with systemd-sbsign, which uses SOURCE_DATE_EPOCH (FSDK systemd v261, /usr/lib/systemd/systemd-sbsign). I confirmed two signatures of the same file are identical and sbverify passes. ukify needs --no-sign-kernel with this signtool; that matches current behaviour, since the kernel embedded in today's UKIs has no signature table of its own.
    • ESP images (DDI, netboot ESP, installer): the staged trees carried the build time into FAT entries (mcopy -m), and mkfs.fat from dosfstools 4.2 ignores SOURCE_DATE_EPOCH and stamps the volume label with the current time. Now the trees are clamped before systemd-repart, and SYSTEMD_REPART_MKFS_OPTIONS_VFAT=--invariant is set. The ESP volume ID changes to mkfs.fat's fixed value. Nothing references it; boot finds partitions by type and UUID.
    • Sysext .raw.zst: a random erofs UUID. The fix is in refactor: shared sysext packing recipe, arm64 binary pins, base-image cleanups #279, which owns the shared sysext recipe.
    • Signing times that remain: efi-keys/*.auth, because sbvarsign stamps the current time and has no option to change it, and SHA256SUMS.gpg.
  • Result. With this PR, a local double build at a scratch version is byte-identical for every file except the four sysexts (fixed by refactor: shared sysext packing recipe, arm64 binary pins, base-image cleanups #279) and SHA256SUMS, which lists them. The files that match are usr.raw, usr-verity.raw, the disk, netboot and installer UKIs, bluefin-server_<v>.raw, bluefin-server-installer_<v>.raw and bluefin-server-netboot_<v>.esp.raw. On refactor: shared sysext packing recipe, arm64 binary pins, base-image cleanups #279's branch, the four sysexts are byte-identical across rebuilds.
  • New workflow. .github/workflows/reproducibility.yml runs weekly (Mon 09:00 UTC) and on workflow_dispatch. It builds the image set, deletes the final-assembly artifacts, rebuilds without remote caches, and diffs the sha256 of every output except *.gpg. efi-keys.bst stays cached between the two builds, for the sbvarsign reason above. Until refactor: shared sysext packing recipe, arm64 binary pins, base-image cleanups #279 merges, it will report the four sysext files.

7. Patches

  • patches/README.md records, for each FSDK patch, why it exists, its upstream status and its drop condition.
  • It flags that 0003-0005 hard-code full library versions: libgirepository-1.0.so.1.0.0, libgdk_pixbuf-2.0.so.0.4400.7, libappstream*.so.1.1.6. Those match FSDK 26.08.0's gobject-introspection 1.86.0, gdk-pixbuf 2.44.7 and appstream 1.1.6. A component bump would leave the patch applying cleanly with dangling links.
  • Not fixed here. Rewriting those patches changes the cache keys of gobject-introspection-base, gdk-pixbuf and appstream and everything built on them, and the failure they work around only shows up under BuildBarn remote execution, which I can't reproduce locally. So bump-fsdk-version.md now links the README and adds a by-hand version comparison to the bump procedure, replacing its stale claim that only 0001 remains.

Verification

  • python3 -m pytest tests/unit -q passes: 379 in a throwaway worktree with no boot-keys.
  • bats tests/unit passes: 0 failures.
  • docs-checks.py passes.
  • actionlint reports only the ubuntu-26.04 label warning every existing workflow gets.
  • flock … just validate passes.
  • Local reproducibility loop: see 6.
  • just export-image then flock … just dogfood-check passes. This is a Secure Boot QEMU netboot, so it also shows the systemd-sbsign-signed systemd-boot and UKIs are accepted by the firmware: os=bluefin-server 26.09.2, failed=0.

Note for #269

Its SBOM step sets creationInfo.created from date -u, which puts a build-time value into SHA256SUMS. To keep the set reproducible, use date -u -d "@${SOURCE_DATE_EPOCH}" +%Y-%m-%dT%H:%M:%SZ.

jeefy and others added 7 commits September 29, 2026 00:23
A complete set was already kept, but the check looked only at DB.key and
sysupdate-signing.asc: with DB.key gone the script regenerated PK, KEK and
the module certificate over the existing files, and with the signing key
gone it replaced import-pubring.pgp. Both sets are now checked file by file
(an empty file counts as missing) and a partial set exits 1 before anything
is written.

tests/unit/test_gen_dev_keys.py runs the script against a scratch tree with
openssl and gpg stubbed: a complete set is left untouched, every partial
set is refused without calling either tool, and --force still regenerates.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
patches/README.md records why each patch exists, whether it went upstream
and when to drop it, and that 0003-0005 name full library versions
(libgirepository 1.0.0, gdk-pixbuf 0.4400.7, appstream 1.1.6) that a
component bump leaves dangling while the patch still applies.

bump-fsdk-version.md pointed at a single remaining CAS patch; it now links
the README and adds the version comparison to the bump procedure.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
cpio --reproducible fixes inode and device numbers but keeps mtimes, and
everything the boot element creates in the initrd tree (symlinks, sysusers
output, depmod indexes) carried the build time, so every rebuild gave a
different initrd and UKIs. Rebuilds now differ in no initrd byte.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
sbsign puts the current time in each Authenticode signature's signingTime
attribute; systemd-sbsign (FSDK systemd, /usr/lib/systemd) takes it from
SOURCE_DATE_EPOCH, so a rebuild with the same keys signs identically.
sbverify still checks every output. ukify needs --no-sign-kernel with this
signtool, which keeps today's behaviour: the embedded kernel was never
signed on its own, only the UKI.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The staged ESP trees carried the build time into the FAT entries (mcopy
keeps mtimes) and mkfs.fat from dosfstools 4.2, which ignores
SOURCE_DATE_EPOCH, stamped each volume label with the current time. Clamp
the trees before systemd-repart and pass --invariant to mkfs.fat through
SYSTEMD_REPART_MKFS_OPTIONS_VFAT (which also fixes the ESP volume ID).

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
reproducibility.yml builds the image set, deletes the final-assembly
artifacts, rebuilds them without remote caches and diffs every output
except *.gpg. efi-keys.bst stays cached: sbvarsign stamps the current time
into each signed variable update and has no option to fix it.
ddi-installer-build.md records what makes the assembly reproducible.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jeefy added a commit to jeefy/server that referenced this pull request Sep 29, 2026
A wall-clock creation time made every rebuild of the release set differ
and defeats the reproducibility check proposed in projectbluefin#281.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keeps main's patch bullet in bump-fsdk-version.md and adds this branch's
pointer to patches/README.md.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy merged commit 98c1b1e into projectbluefin:main Sep 29, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant