Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 54 additions & 12 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -135,10 +135,17 @@ jobs:
# FSDK's kernel is most of a cold build
# (docs/skills/ci-tooling.md#build-time-and-caches), and no upstream cache
# holds it (our module certificate and kernel config patches change its
# key). This job builds it (and Go) with the committed release module certificate and no
# signing secrets, and pushes the key-free artifacts to the project CAS unless
# a remote already holds them; every build pulls them through project.conf
# (scripts/kernel-cache.sh).
# key). These jobs build it (and Go) with no signing secrets, each against
# one committed module certificate, and push the key-free artifacts to the
# project CAS unless a remote already holds them; every build pulls them
# through project.conf (scripts/kernel-cache.sh):
# - kernel-cache: the release certificate, for release builds;
# - kernel-cache-dev: the public INSECURE dev certificate that
# `just gen-dev-keys` uses (files/dev-keys/), for pull requests, the
# nightly build and local builds. No job needs it, so it never delays
# a release.
# The cache only saves time: a failed seed must not hold back the release,
# which checks build's and boot-test's own results.
kernel-cache:
needs: changes
if: needs.changes.outputs.release == 'true'
Expand All @@ -162,20 +169,13 @@ jobs:
- name: Set up runner disk space
run: bash scripts/ci-runner-disk.sh

- name: Stage the release module certificate
run: |
mkdir -p files/boot-keys/modules
cp files/release-keys/linux-module-cert.crt files/boot-keys/modules/

# The cache only saves time: a failure here must not hold back the
# release, which checks build's and boot-test's own results.
- name: Build and push the kernel cache unless it exists
id: seed
continue-on-error: true
env:
CASD_CLIENT_CERT: ${{ vars.CASD_CLIENT_CERT }}
CASD_CLIENT_KEY: ${{ secrets.CASD_CLIENT_KEY }}
run: bash scripts/kernel-cache.sh seed
run: bash scripts/kernel-cache.sh seed release

# continue-on-error turns the job green; say so, or a cache that never
# seeds goes unnoticed while every build compiles the kernel.
Expand All @@ -186,6 +186,46 @@ jobs:
echo "::warning title=Kernel cache not seeded::${msg}"
echo "${msg}" >> "$GITHUB_STEP_SUMMARY"

kernel-cache-dev:
needs: changes
if: needs.changes.outputs.release == 'true'
# CASD_CLIENT_KEY lives in this environment, which only main may use.
environment: bst-cache
runs-on: ubuntu-26.04
timeout-minutes: 240
permissions:
contents: read
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Set up just
uses: taiki-e/install-action@e407f7bafb71fd004bc5c2da3032e5470cbb6ef0 # v2
with:
tool: just@1.58.0

- name: Set up runner disk space
run: bash scripts/ci-runner-disk.sh

- name: Build and push the dev kernel cache unless it exists
id: seed
continue-on-error: true
env:
CASD_CLIENT_CERT: ${{ vars.CASD_CLIENT_CERT }}
CASD_CLIENT_KEY: ${{ secrets.CASD_CLIENT_KEY }}
run: bash scripts/kernel-cache.sh seed dev

# continue-on-error turns the job green; say so, or a cache that never
# seeds goes unnoticed while every build compiles the kernel.
- name: Report a failed seed
if: steps.seed.outcome == 'failure'
run: |
msg="The dev kernel cache seed failed; pull requests and the nightly build compile the kernel until a release seeds it. See the step above."
echo "::warning title=Kernel cache not seeded::${msg}"
echo "${msg}" >> "$GITHUB_STEP_SUMMARY"

build:
needs: [changes, kernel-cache]
# Runs when kernel-cache is skipped (non-release builds) or failed: the
Expand Down Expand Up @@ -255,6 +295,8 @@ jobs:
exit 1
fi
cp files/release-keys/linux-module-cert.crt files/boot-keys/modules/linux-module-cert.crt
# Never the public INSECURE dev module pair (files/dev-keys/).
bash scripts/check-release-keys.sh
printf '%s\n' "${SYSUPDATE_SIGNING_KEY}" > files/boot-keys/sysupdate-signing.asc
cp files/os/sysupdate-keys/import-pubring.gpg files/boot-keys/import-pubring.pgp
else
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ All local `just` targets run BuildStream inside the FSDK `bst2` container via `j
|---|---|
| `just validate` | Merge-contract graph check — run this on every change. |
| `just test-unit` | Unit tests (pytest + bats). |
| `just gen-dev-keys` | Generate throwaway Secure Boot, module, and image (`SHA256SUMS`) signing keys in `files/boot-keys/` (gitignored). |
| `just gen-dev-keys` | Generate throwaway Secure Boot and image (`SHA256SUMS`) signing keys in `files/boot-keys/` (gitignored), plus the committed public INSECURE dev module pair from `files/dev-keys/` (`--private-module-key` for a fresh one). |
| `just set-version V` | Set `image-version` in `include/image.yml` (≤17 chars, increasing under strverscmp). |
| `just build-image` / `just export-image` | Build and export the release image set to `dist/diskless/`. |
| `just dogfood` / `just dogfood-check` | Boot `dist/diskless/` diskless in QEMU with Secure Boot (interactive / headless probe). |
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ See [`CONTRIBUTING.md`](CONTRIBUTING.md) for the contributor checklist, Conventi

## Security and release trust

- **Signed boot chain**: Secure Boot keys enroll from the ESP on first boot (`secure-boot-enroll if-safe` in VMs; on bare metal in firmware Setup Mode, from the USB installer's prompt or the systemd-boot menu); the USB installer refuses to install without Secure Boot on with these keys unless told explicitly. Local builds use throwaway keys from `just gen-dev-keys`.
- **Signed boot chain**: Secure Boot keys enroll from the ESP on first boot (`secure-boot-enroll if-safe` in VMs; on bare metal in firmware Setup Mode, from the USB installer's prompt or the systemd-boot menu); the USB installer refuses to install without Secure Boot on with these keys unless told explicitly. Local builds use throwaway Secure Boot keys from `just gen-dev-keys`, with the published INSECURE dev module signing key from `files/dev-keys/` (never in a release; see [`docs/skills/secure-boot-keys.md`](docs/skills/secure-boot-keys.md)).
- **Signed manifests**: the build signs one combined `SHA256SUMS` over the whole image set (OS images, UKIs, sysexts) inside `oci/bluefin-server-image.bst`; a release publishes `dist/diskless/` as-is to GitHub Releases and as an OCI artifact.
- **Sysupdate verification**: installed nodes verify updates against the signed manifest (`Verify=yes`), and the diskless pull checks the same signature in the initrd; see [`docs/skills/systemd-sysupdate-verification.md`](docs/skills/systemd-sysupdate-verification.md) for details.
- **Provenance and SBOM**: releases carry SLSA provenance and SPDX SBOM attestations; see [`docs/skills/systemd-sysupdate-verification.md`](docs/skills/systemd-sysupdate-verification.md) for verification.
Expand Down
33 changes: 23 additions & 10 deletions docs/skills/ci-tooling.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ one of these environments is refused before its first step.
| Environment | Secrets | Jobs |
|---|---|---|
| `release` | `BOOT_KEYS_TARBALL`, `SYSUPDATE_SIGNING_KEY` | `build`, only when `changes` says `release=true` |
| `bst-cache` | `CASD_CLIENT_KEY` (`CASD_CLIENT_CERT` is a repository variable) | `kernel-cache` (releases only) |
| `bst-cache` | `CASD_CLIENT_KEY` (`CASD_CLIENT_CERT` is a repository variable) | `kernel-cache`, `kernel-cache-dev` (releases only) |
| `trackers` | `MERGERAPTOR_APP_ID`, `MERGERAPTOR_PRIVATE_KEY` | `track-junctions`, `track-binaries`' `propose` |

`build` serves releases and every other build, so it names the environment
Expand Down Expand Up @@ -350,11 +350,19 @@ version) and the NVIDIA Container Toolkit (Go); the driver sysext adds about
so they can be made required without `paths-ignore` leaving them pending.
The unit, docs and lint workflows have no `paths:` filter for the same
reason.
- **Kernel cache in the project CAS, key-free by construction.** The
`kernel-cache` job (releases only) runs `scripts/kernel-cache.sh seed`: with
no signing secrets and the committed release module certificate
(`files/release-keys/linux-module-cert.crt`) staged as
`files/boot-keys/modules/linux-module-cert.crt`, it builds FSDK's
- **Kernel cache in the project CAS, key-free by construction.** Two jobs,
on releases only, run `scripts/kernel-cache.sh seed`, one per module
certificate the kernel can trust (it is part of the kernel's cache key):
- `kernel-cache` runs `seed release` with the committed release certificate
(`files/release-keys/linux-module-cert.crt`); `build` waits for it.
- `kernel-cache-dev` runs `seed dev` with the public INSECURE dev
certificate (`files/dev-keys/INSECURE-dev-module-key.crt`) that
`just gen-dev-keys` uses for every non-release build. No job waits for
it, so it never delays a release.

With no signing secrets and that one certificate staged as the only file of
`files/boot-keys/modules/` (`seed` refuses anything else there: that
directory's import is pushed with the kernel), each builds FSDK's
`components/linux.bst` and `components/go.bst` with a BuildStream config
that pushes artifacts and sources to `cache.projectbluefin.io:11002`,
unless `bst artifact show` reports both `available` on a remote. The push
Expand All @@ -369,7 +377,9 @@ version) and the NVIDIA Container Toolkit (Go); the driver sysext adds about
`patches/freedesktop-sdk/0001-project.conf-Add-GNOME-CAS-servers.patch`. Release builds
normalize `BOOT_KEYS_TARBALL`'s module certificate to the committed bytes
after checking it is the same certificate (and stop if not), so the keys
match. The CAS is publicly readable: `seed` refuses if
match, and run `scripts/check-release-keys.sh`, which fails the build if
the module certificate or key is the dev pair; `seed release` runs it too.
The CAS is publicly readable: `seed` refuses if
`bluefin-server/keys/boot-keys.bst` is anywhere in the graph it builds,
and the job holds no signing secret. A content grep is no guard here: FSDK
sources (Go's TLS test data and others) carry 307 PEM private keys. The
Expand All @@ -391,9 +401,12 @@ version) and the NVIDIA Container Toolkit (Go); the driver sysext adds about
`patches/` (an untested FSDK bump would otherwise merge unbuilt); the rest
run `validate`. A regression outside those paths shows up in the next
`main` build or the nightly build, both of which block nothing but the
release that contains it. When a pull request does build, it still builds
the kernel: `just gen-dev-keys` makes a new module certificate on every
run, so the PR kernel's cache key never matches anything cached.
release that contains it. When a pull request or the nightly build does
build, it pulls the kernel that `kernel-cache-dev` seeded: `just
gen-dev-keys` always stages the same public dev module certificate, so
the dev kernel's cache key is the same on every run. It compiles the
kernel (1 h 43 min) only until the next release after a kernel change
seeds it.
- **No `actions/cache`.** A full build's cache holds `boot-keys.bst`, and
pull requests can restore caches saved on `main`; the key-free kernel
cache lives in the project CAS instead.
Expand Down
9 changes: 6 additions & 3 deletions docs/skills/ddi-installer-build.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ inside the FSDK `bst2` container (`just bst`); it is not installed locally.
```bash
just validate # version invariants + resolve the shipped element graphs
just test-unit # pytest + bats
just gen-dev-keys # throwaway Secure Boot + module keys in files/boot-keys/
just gen-dev-keys # throwaway Secure Boot keys + the public dev module key in files/boot-keys/
just set-version V # set image-version in include/image.yml (<=17 chars,
# increasing under strverscmp)
just build-image # build oci/bluefin-server-image.bst
Expand Down Expand Up @@ -59,7 +59,9 @@ Every image build signs: the UKIs and systemd-boot with DB, kernel modules
with the module signing certificate, and the release `SHA256SUMS` with the
image signing key. What each key is, where it lives, what `just gen-dev-keys`
generates (throwaway keys in the gitignored `files/boot-keys/`, kept unless
`--force`, a partial set is an error), and how CI supplies the real keys:
`--force`, a partial set is an error, the module pair is the public
INSECURE dev pair so the kernel comes from the cache), and how CI supplies
the real keys:
[secure-boot-keys.md](secure-boot-keys.md). The release-signing half
(`sysupdate-signing.asc` / `import-pubring.pgp`, the committed
`files/os/sysupdate-keys/import-pubring.gpg`, and rotation):
Expand Down Expand Up @@ -243,7 +245,8 @@ nodes may already trust.
(the build fails the check in `bluefin-server-boot.bst`).
- A new release asset that is not added to `SHA256SUMS` in
`oci/bluefin-server-image.bst`.
- Keys committed anywhere outside the gitignored `files/boot-keys/`.
- Keys committed anywhere outside the gitignored `files/boot-keys/`, other
than the public INSECURE dev module pair in `files/dev-keys/`.

## Verification

Expand Down
51 changes: 44 additions & 7 deletions docs/skills/secure-boot-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Secure Boot and image signing key management for Bluefin Server. Lo
metadata:
type: reference
status: stable
last_updated: "2026-09-29"
last_updated: "2026-10-04"
---
# Secure Boot Key Management

Expand Down Expand Up @@ -32,12 +32,45 @@ built against it, and release builds stop if it differs from the one in

```bash
just gen-dev-keys # throwaway keys in files/boot-keys/ (gitignored)
just gen-dev-keys --force # rotate (rebuilds the kernel)
just gen-dev-keys --force # rotate the Secure Boot and image signing keys
just gen-dev-keys --force --private-module-key # also a fresh module key
```

`just validate` and `just build-image` depend on `gen-dev-keys`, so a fresh
clone builds out of the box.

### The INSECURE dev module key

The module signing pair of a `gen-dev-keys` key set is not generated: it is
the committed, public pair in `files/dev-keys/`
(`INSECURE-dev-module-key.{pem,crt}`; see its `README.md`). The module
certificate is part of the kernel's cache key, so with one fixed dev
certificate every non-release build (pull requests, the nightly build, local
builds) pulls the kernel that `scripts/kernel-cache.sh seed dev` pushed on
the last release instead of compiling it (see "Build time and caches" in
[ci-tooling.md](ci-tooling.md)).

A key set generated before the dev key existed keeps its own module
certificate, so the kernel still compiles; `just gen-dev-keys` says so, and
`just gen-dev-keys --force` switches it to the dev key.

- **Public on purpose.** Anyone can sign a module that a dev kernel loads
under lockdown. Dev images are also signed with throwaway Secure Boot keys
and never published. For an image you boot on hardware you care about,
use `--private-module-key` (the kernel then builds locally).
- **Never in a release.** Release builds install `BOOT_KEYS_TARBALL` and run
`scripts/check-release-keys.sh`, which fails when the module certificate or
key is the dev pair (compared by public key); `kernel-cache.sh seed
release` runs it too. The release kernel trusts only
`files/release-keys/linux-module-cert.crt`, so dev-signed modules do not
load on it.
- **Never in an artifact.** FSDK's kernel reads only
`/keys/linux-module-cert.crt` (`SYSTEM_TRUSTED_KEYS`, `MODULE_SIG_KEY ""`,
no `MODULE_SIG_ALL`): the private key is used only by
`bluefin-server/kernel-modules.bst` and the other signing elements through
`bluefin-server/keys/boot-keys.bst`, which is never pushed. No element
stages `files/dev-keys/` (`tests/unit/test_dev_module_key.py`).

## CI secrets

On `main`, the workflow unpacks two secrets from the `release` environment,
Expand All @@ -49,13 +82,15 @@ which only `main` may use (see "Environments and secrets" in
- `SYSUPDATE_SIGNING_KEY` — the ASCII-armored secret key written to
`files/boot-keys/sysupdate-signing.asc`.

Pull requests get throwaway keys from `just gen-dev-keys`; they never see the
real secrets.
Pull requests get throwaway keys from `just gen-dev-keys` (with the public
dev module pair); they never see the real secrets.

## Key rotation

1. Generate new keys (`just gen-dev-keys --force` locally, or the equivalent
in a secure environment for production).
1. Generate new keys (`just gen-dev-keys --force --private-module-key`
locally, or the equivalent in a secure environment for production). Plain
`--force` installs the public INSECURE dev module key, which
`scripts/check-release-keys.sh` rejects for release builds.
2. Update the CI secrets (`BOOT_KEYS_TARBALL`, `SYSUPDATE_SIGNING_KEY`), and
commit the new public module certificate as
`files/release-keys/linux-module-cert.crt` in the same change.
Expand All @@ -72,7 +107,9 @@ new kernel build is mandatory. Plan kernel rebuilds into the rotation window.
- [ ] `just validate` passes after key generation.
- [ ] `sbverify --cert files/boot-keys/DB.crt <uki>` passes on built UKIs.
- [ ] `gpgv --keyring files/boot-keys/import-pubring.pgp SHA256SUMS.gpg SHA256SUMS` passes.
- [ ] No private keys are committed outside the gitignored `files/boot-keys/`.
- [ ] No private keys are committed outside the gitignored `files/boot-keys/`,
except the public INSECURE dev module key in `files/dev-keys/`.
- [ ] `bash scripts/check-release-keys.sh` passes on a release key set.

## See also

Expand Down
32 changes: 32 additions & 0 deletions files/dev-keys/INSECURE-dev-module-key.crt
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
-----BEGIN CERTIFICATE-----
MIIFgzCCA2ugAwIBAgIUeo+QMr5qJbMHs5lR0LBgutFsWR0wDQYJKoZIhvcNAQEN
BQAwPzE9MDsGA1UEAww0Qmx1ZWZpbiBTZXJ2ZXIgSU5TRUNVUkUgcHVibGljIGRl
diBrZXJuZWwgbW9kdWxlIGtleTAgFw0yNjEwMDUwMTEwMDFaGA8yMTI2MDkxMTAx
MTAwMVowPzE9MDsGA1UEAww0Qmx1ZWZpbiBTZXJ2ZXIgSU5TRUNVUkUgcHVibGlj
IGRldiBrZXJuZWwgbW9kdWxlIGtleTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC
AgoCggIBALNIdFHp9VUhjTxW0PmxC8rXN6Q3qw3UScQmIoOX8aUEJJeWQi9BDNol
VMRbc24bPyMnYk2BvvGSlPkFVRhcLb9tje/Xq5ApdV3BEzvm6hEPrDA3oH7TAgI6
D+pXaJ7eY1ikTixdD0+zGYx+v2aCBdgCtiMpRpYDrh7lb4KRMYWqra1KDCG9RO9w
GKKprhgVcP1jdJE9gW/hF24o7rXkyWS0M5W3MGsrcVDcvQp3jAiX5NTh4nI8ypIk
GF0NXq40z57iu6GytqWDimEUVgTWfB8G05TkNtj1pr7326RTKSFyygwD/m4R6eT5
Epbqjl1DijkmmUCgDzWkab8yjB+tYLmsBi/W8rXkrVdMKZnBQorl3R9PRJ+05m48
j0K6ja1c/x47IRIgt6RMRL2D+AKJyyBPqlaa6yGhYBdFX5TAaVcVLZCqifR8PSpF
q6K7uxzPYTxBtQDZiKtp5GdjpBa/fUrw4/7EJcqBC56gNkpsQLYxfm0YMxkD9ww+
gBAmzE9qzoWkrufG2SUeOgYrpBmGduoEINaNI0UVHi9wPXy5GWWVSlFLTYkC08Us
7GO+sZS11a+SMl1pEGeA5f7iTcpIZi2pcWTyCQWnrKRqpxNq83IN7mHXLQTguSEE
W3ImJhuNMcdBKtkibKgP1bxmXVvguV6g/JFzQCyVuRmK/c4gYm0VAgMBAAGjdTBz
MB0GA1UdDgQWBBQKPbCbnWtFEpZo7hEczyF8M1c5LjAfBgNVHSMEGDAWgBQKPbCb
nWtFEpZo7hEczyF8M1c5LjAPBgNVHRMBAf8EBTADAQH/MAsGA1UdDwQEAwIHgDAT
BgNVHSUEDDAKBggrBgEFBQcDAzANBgkqhkiG9w0BAQ0FAAOCAgEAAWnasR9QBHib
LCBmbKckiX8MnfsT5WxvmXDGv+755ZnsxwCrcYddSU6fUPfkxXgDTxSLNzwzZODN
8xZasXs1tVzw7X+XXNq2+S1poNTUHpYbC/vGeNBv+F9AEOkpifGrGz3vt2xxKix9
FQbZywzeMUfCFwZxDyyiZcF49crXDlkWUYcigMlIwa9486xUVRkNGEW4+bQr1fne
SbmsT80VWOc42O6n5Nni82hUw5OK0tGkfuNpqjqGtktWfnmUE9yvkK++vV5BKK80
HXS9lIf8QPc7XSJ2BrxZYfGWrFiP47d1Uqyh83XPIltufQkMzBlvO0L8LRx1eqoF
3kLim/BMETmEh+qRvnjVCIK11D9bcP/BwCiuLve1kNIhBOi7PtyVdLVYS4o5N598
0fYeylKzQGt5IQNvMbe30kDAlPWQTlk49iaiDMy2JjIALyeirhKei/4bXH5pyzs8
MgD5xDv+DHSvq55hZsutjUJBVEJMHyTGSzCXjmz1xECKJ2rNakHaOhjxD1hJc8He
8FRvvh9BuRaTDGKPKg0hwUsenQiI42XI65qz49dRxs81D+34N9s/RjK2Jsi/KbUq
QxWjvVzPLHEYCuOH78cBHvugV8HRU1ax08tQxk/JmoWRi02NguAas1bQYukZQLB2
0zwcO8VeO1QxYIFng3lnbDERj3YkFl0=
-----END CERTIFICATE-----
Loading