Repository navigation
ci: cache the dev-key kernel with a committed INSECURE dev module key - #404
Conversation
There was a problem hiding this comment.
Reviewed commits 26d2941..b79c55f (the three on top of #403).
docs-currency
- info —
files/dev-keys/README.md:24,docs/skills/secure-boot-keys.md:54: "Dev images … are never published" is imprecise.build.yml:302-307uploads every non-release image set (incl. nightly) as a workflow artifact (retention-days: 1), downloadable by anyone logged in on a public repo. Suggest "never released; CI keeps them only as short-lived workflow artifacts". The trade-off itself is unchanged and sound.
No findings from: correctness, security, intent-alignment, style.
Confidence: 5/5 (safe)
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
…uilds gen-dev-keys made a fresh module certificate on every run, and the certificate is part of the kernel's cache key, so every PR full build and nightly compiled FSDK's kernel (1 h 43 min). files/dev-keys/ now holds a public, clearly named INSECURE module pair that gen-dev-keys copies into files/boot-keys/; --private-module-key keeps generating a fresh pair for images booted on real hardware. Measured: two `gen-dev-keys --force` runs (fresh DB keys) give the same linux.bst full key. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eases kernel-cache.sh seed takes a variant and stages that one committed certificate as the only file of files/boot-keys/modules/ (whose import is pushed with the kernel), refusing anything else there. A new kernel-cache-dev job seeds the dev variant on release runs; no job needs it, so releases are not delayed. scripts/check-release-keys.sh fails when a release key set's module certificate or key is the dev pair (by public key); release builds and seed release run it. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace ci-tooling.md's 'pull requests always build the kernel' with the dev kernel cache, and document the INSECURE dev pair, its release guard and why it reaches no artifact in secure-boot-keys.md. Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
b79c55f to
2be6f7f
Compare
There was a problem hiding this comment.
Read at head 2be6f7f (diff + scripts/gen-dev-keys.sh, scripts/kernel-cache.sh, scripts/check-release-keys.sh, build.yml build/seed jobs, elements/bluefin-server/keys/*.bst, the new tests). Dev pair verified consistent (openssl x509 -pubkey == openssl pkey -pubout), cert valid to 2126, only the .crt reaches linux-module-cert.bst via files/boot-keys/modules.
docs-currency
- (low)
README.md:76still says "Local builds use throwaway keys fromjust gen-dev-keys". Afterscripts/gen-dev-keys.sh:112-113the module pair is the publishedfiles/dev-keys/key, not throwaway — the user-facing README should carry the same INSECURE caveat the three updated skill docs do.
correctness
- (low) Existing clones keep compiling the kernel:
scripts/gen-dev-keys.sh:94-97exits early when a full key set exists, so a contributor with pre-PR throwaway keys never receives the dev certificate until--force. Neither that message ("pass --force to regenerate") nordocs/skills/secure-boot-keys.md:34-37says that is what makes the cache hit. One line in the early-exit message or the docs would do.
No findings from: security, intent-alignment, style.
Confidence: 3/5 (needs attention) — 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
…veat gen-dev-keys now notes on its early exit when the existing module certificate is not the dev one (the kernel builds locally until --force), secure-boot-keys.md says the same, and the README no longer calls the module key throwaway. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <scanner@hive.kubestellar.io>
There was a problem hiding this comment.
Reviewed at head bfb6323.
docs-currency
- (medium)
docs/skills/secure-boot-keys.md:90-91— "Key rotation" step 1 still says to generate new production keys withjust gen-dev-keys --force("or the equivalent in a secure environment"). After this PR that command installs the publishedfiles/dev-keys/INSECURE-dev-module-key.pemas the module key (scripts/gen-dev-keys.sh:115-116), so an operator following the doc would build a release key set around the INSECURE pair;check-release-keys.shwould then fail the release build, but the doc should say--force --private-module-keyhere. - (low)
AGENTS.md:54still describesjust gen-dev-keysas generating "throwaway Secure Boot, module, and image ... signing keys"; the module pair is no longer throwaway.
correctness
- (low)
scripts/gen-dev-keys.sh:94-100—--private-module-keywithout--forceon an existing complete key set is a silent no-op: it prints "Keys already exist" and exits 0, leaving the INSECURE module pair in place, with no hint that the flag was ignored. Worth a line in that branch (or an error) since the whole point of the flag is "hardware you care about".
style
- (low)
scripts/check-release-keys.sh:15-16— the scriptcds to the repo root before readingKEY_DIR, so a relative argument resolves against the repo root rather than the caller's cwd; the usage comment at line 8 does not say so.
No findings from: security, intent-alignment.
Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
gen-dev-keys.sh now errors when --private-module-key is passed without --force while the INSECURE dev module key is installed, instead of silently keeping it. Key rotation docs use --force --private-module-key, AGENTS.md describes the committed dev module pair, and check-release-keys.sh documents that a relative KEY_DIR is resolved against the repo root. Hive-Fix-Attempt: 2/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: scanner <scanner@hive.kubestellar.io>
Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c3908c4 to
8459737
Compare
TestRateLimitPerSource joined three times in a row from one source. The server frees the source's slot after the client already has its answer, so the next attempt could be refused as busy ("attempt 1: control plane refused: busy"). It failed main's push builds 37264198746 and 37282526257 and both full builds of #404, inside homelab/bluefin-cluster.bst's go test. Wait for the slot like the replay tests do (#391).
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Read the tree at b39bc98, not just the diff. This looks correct to me; the key-isolation claims hold up independently.
What I verified:
elements/bluefin-server/keys/linux-module-cert.bst:15imports onlyfiles/boot-keys/modules, andscripts/kernel-cache.sh:73-89(stage_cert) refuses to seed if that directory holds anything but the one certificate or if the certificate text contains a private key — so the pushed import artifact stays cert-only for both variants. The existingboot-keys.bst-in-graph guard is still in place (kernel-cache.sh:99).files/dev-keys/INSECURE-dev-module-key.{pem,crt}are a matching pair (openssl x509 -pubkey==openssl pkey -pubout), withkeyUsage=digitalSignature,extendedKeyUsage=codeSigning, and a public key distinct fromfiles/release-keys/linux-module-cert.crt.- Release path:
.github/workflows/build.yml:298-299runsscripts/check-release-keys.shafter staging the release certificate, andkernel-cache.sh:86-88runs it forseed release. The script compares public keys (check-release-keys.sh:24-36), so a re-issued certificate for the dev key is also caught. - Non-release path:
build.yml:303andreproducibility.yml:67calljust gen-dev-keys, which now installs the committed pair with the same modes (0600/0644)stage_certuses, so the dev kernel's cache key matches whatkernel-cache-devseeds. kernel-cache-dev(build.yml:189-227) is not in any job'sneeds:, so a failed dev seed cannot delay a release.- The seven touched unit test modules pass locally (120 tests).
One suggestion, not a defect: files/dev-keys/README.md and docs/skills/secure-boot-keys.md already say the pair is public; it may be worth one sentence there telling contributors not to report it to secret scanners, since generic PEM blocks tend to trip third-party scanners on forks.
Confidence: 5/5 (safe)
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Review finding M5. Rebased onto main after #402 and #403 merged.
kernel-cache-devfollows #402'sbst-cacheenvironment and failed-seed report.Problem
scripts/gen-dev-keys.shmade a new module signing certificate on every run. FSDK's kernel trusts that certificate (SYSTEM_TRUSTED_KEYS), so it is part of the kernel's cache key. As a result, everyfull-build/FSDK PR and every nightly compiled the kernel, which takes 1 h 43 min.Change
files/dev-keys/INSECURE-dev-module-key.{pem,crt}+ README. This is a public module key pair, clearly named as insecure, for non-release builds only.gen-dev-keyscopies it intofiles/boot-keys/. The other keys (PK/KEK/DB and the sysupdate signing key) are still throwaway keys generated on each run.--private-module-keygenerates a fresh module pair, for images that boot on real hardware.kernel-cache.sh seed release|dev. It stages exactly one committed certificate as the only file offiles/boot-keys/modules/. That directory's import artifact is pushed together with the kernel, so the script refuses anything else in it, including a private key.kernel-cache-devjob runsseed devon release runs, alongsidekernel-cache. No jobneeds:it, so it never delays a release.boot-keys.bst-in-graph guard is unchanged.scripts/check-release-keys.sh. It fails if the module certificate or the module key of a key set is the dev pair. It compares public keys, so a certificate re-issued for the dev key is also caught. The release "Install signing keys" step runs it after staging the release certificate, andseed releaseruns it too.Safety evidence
Cache-key stability (
just bst show --format '%{full-key}' freedesktop-sdk.bst:components/linux.bst):No private key reaches the kernel or any pushed artifact:
/keys/linux-module-cert.crt. Infiles/linux/fdsdk-config.shit setsSYSTEM_TRUSTED_KEYS, setsMODULE_SIG_KEY ""and removesMODULE_SIG_ALL.bluefin-server/kernel-modules.bstusing/boot-keys/linux-module-cert.keyfromboot-keys.bst, which is never pushed.bst show --deps all linux.bstgives 210 elements, with 0 occurrences ofboot-keys.bst. The onlybluefin-server/*element in that graph iskeys/linux-module-cert.bst.linux-module-cert.bstartifact contains onlykeys/linux-module-cert.crt(1972 bytes).3a757045…was pulled from the CAS. It is the same element, differing only in the certificate. Its 17,122 files contain:*.pem,*.keyorsigning_key*files;-----BEGIN … PRIVATE KEY-----;CONFIG_MODULE_SIG_KEY=""andMODULE_SIG_ALLnot set.vmlinuxembeds the release certificate DER and does not embed the dev certificate. Dev-signed modules therefore cannot load on release kernels.tests/unit/test_dev_module_key.pyasserts that no element's local source coversfiles/dev-keys/.Tests
test_dev_module_key.pycovers the guard:kernel-cache-dev.test_kernel_cache.pycovers:modules/are refused;seed releaserefuses the dev certificate;test_gen_dev_keys.pycovers:--forcekeeps the pair while the DB keys change;--private-module-keyproduces a different pair;Gates:
just validate; pytest (1128 passed, including a detached worktree withoutfiles/boot-keys); bats; docs-checks; shellcheck; actionlint (only the existing SC2129); zizmor (no new findings).Trade-off
A dev image's kernel trusts a published key, so root on a dev image can load a module signed with it. Dev images are signed with throwaway Secure Boot keys and are never published. This is documented in
secure-boot-keys.md, together with--private-module-key.After merge, the next release seeds the dev kernel. Until then, PR and nightly builds still compile it.