Skip to content

ci: cache the dev-key kernel with a committed INSECURE dev module key - #404

Merged
jeefy merged 8 commits into
mainfrom
ci/gha-c2-dev-kernel-cache
Oct 5, 2026
Merged

jeefy merged 8 commits into
mainfrom
ci/gha-c2-dev-kernel-cache

Conversation

@jeefy

@jeefy jeefy commented Oct 5, 2026 •

Copy link
Copy Markdown
Collaborator

Review finding M5. Rebased onto main after #402 and #403 merged. kernel-cache-dev follows #402's bst-cache environment and failed-seed report.

Problem

scripts/gen-dev-keys.sh made a new module signing certificate on every run. FSDK's kernel trusts that certificate (SYSTEM_TRUSTED_KEYS), so it is part of the kernel's cache key. As a result, every full-build/FSDK PR and every nightly compiled the kernel, which takes 1 h 43 min.

Change

  • files/dev-keys/INSECURE-dev-module-key.{pem,crt} + README. This is a public module key pair, clearly named as insecure, for non-release builds only. gen-dev-keys copies it into files/boot-keys/. The other keys (PK/KEK/DB and the sysupdate signing key) are still throwaway keys generated on each run. --private-module-key generates a fresh module pair, for images that boot on real hardware.
  • kernel-cache.sh seed release|dev. It stages exactly one committed certificate as the only file of files/boot-keys/modules/. That directory's import artifact is pushed together with the kernel, so the script refuses anything else in it, including a private key.
    • A new kernel-cache-dev job runs seed dev on release runs, alongside kernel-cache. No job needs: it, so it never delays a release.
    • The existing boot-keys.bst-in-graph guard is unchanged.
  • Release guard: scripts/check-release-keys.sh. It fails if the module certificate or the module key of a key set is the dev pair. It compares public keys, so a certificate re-issued for the dev key is also caught. The release "Install signing keys" step runs it after staging the release certificate, and seed release runs it too.

Safety evidence

Cache-key stability (just bst show --format '%{full-key}' freedesktop-sdk.bst:components/linux.bst):

gen-dev-keys --force #1 (DB.crt 9c2247…)  -> d558977d56bb294249103cecab82d3cec897e9bc04891aab17fdf10413014f45
gen-dev-keys --force #2 (DB.crt 727992…)  -> d558977d56bb294249103cecab82d3cec897e9bc04891aab17fdf10413014f45
--private-module-key                      -> 37850e830fb55bae55c3f31ddfa3a050e1b7c1fdbad4709019683a7893ee1fe5
release certificate                       -> 3a75704527028e01448daae99c5e47dbe705b6e01c68864742a440799bc41d3a (unchanged, available on the CAS)

No private key reaches the kernel or any pushed artifact:

  • FSDK's kernel reads only /keys/linux-module-cert.crt. In files/linux/fdsdk-config.sh it sets SYSTEM_TRUSTED_KEYS, sets MODULE_SIG_KEY "" and removes MODULE_SIG_ALL.
  • Modules are signed by bluefin-server/kernel-modules.bst using /boot-keys/linux-module-cert.key from boot-keys.bst, which is never pushed.
  • bst show --deps all linux.bst gives 210 elements, with 0 occurrences of boot-keys.bst. The only bluefin-server/* element in that graph is keys/linux-module-cert.bst.
  • The dev-variant linux-module-cert.bst artifact contains only keys/linux-module-cert.crt (1972 bytes).
  • The release kernel artifact 3a757045… was pulled from the CAS. It is the same element, differing only in the certificate. Its 17,122 files contain:
    • no *.pem, *.key or signing_key* files;
    • no -----BEGIN … PRIVATE KEY-----;
    • a config with CONFIG_MODULE_SIG_KEY="" and MODULE_SIG_ALL not set.
  • Its vmlinux embeds the release certificate DER and does not embed the dev certificate. Dev-signed modules therefore cannot load on release kernels.
  • tests/unit/test_dev_module_key.py asserts that no element's local source covers files/dev-keys/.

Tests

  • test_dev_module_key.py covers the guard:
    • it passes on a project pair and on a release certificate alone;
    • it fails on the dev certificate, the dev key, a certificate re-issued for the dev key, and a missing or broken certificate;
    • the committed release certificate is not the dev key;
    • workflow wiring: release builds check after staging, both seed variants run only on releases, and no job needs kernel-cache-dev.
  • test_kernel_cache.py covers:
    • each variant stages only its own certificate;
    • extra files in modules/ are refused;
    • seed release refuses the dev certificate;
    • an unknown variant gives a usage error.
  • test_gen_dev_keys.py covers:
    • the module pair equals the committed pair;
    • --force keeps the pair while the DB keys change;
    • --private-module-key produces a different pair;
    • unknown arguments are refused.

Gates: just validate; pytest (1128 passed, including a detached worktree without files/boot-keys); bats; docs-checks; shellcheck; actionlint (only the existing SC2129); zizmor (no new findings).

Trade-off

A dev image's kernel trusts a published key, so root on a dev image can load a module signed with it. Dev images are signed with throwaway Secure Boot keys and are never published. This is documented in secure-boot-keys.md, together with --private-module-key.

After merge, the next release seeds the dev kernel. Until then, PR and nightly builds still compile it.

@jeefy jeefy added the full-build Run the full image build and boot test on this pull request label Oct 5, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commits 26d2941..b79c55f (the three on top of #403).

docs-currency

  • info — files/dev-keys/README.md:24, docs/skills/secure-boot-keys.md:54: "Dev images … are never published" is imprecise. build.yml:302-307 uploads every non-release image set (incl. nightly) as a workflow artifact (retention-days: 1), downloadable by anyone logged in on a public repo. Suggest "never released; CI keeps them only as short-lived workflow artifacts". The trade-off itself is unchanged and sound.

No findings from: correctness, security, intent-alignment, style.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

jeefy and others added 3 commits October 4, 2026 23:35
…uilds

gen-dev-keys made a fresh module certificate on every run, and the certificate is part of the kernel's cache key, so every PR full build and nightly compiled FSDK's kernel (1 h 43 min). files/dev-keys/ now holds a public, clearly named INSECURE module pair that gen-dev-keys copies into files/boot-keys/; --private-module-key keeps generating a fresh pair for images booted on real hardware. Measured: two `gen-dev-keys --force` runs (fresh DB keys) give the same linux.bst full key.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eases

kernel-cache.sh seed takes a variant and stages that one committed certificate as the only file of files/boot-keys/modules/ (whose import is pushed with the kernel), refusing anything else there. A new kernel-cache-dev job seeds the dev variant on release runs; no job needs it, so releases are not delayed. scripts/check-release-keys.sh fails when a release key set's module certificate or key is the dev pair (by public key); release builds and seed release run it.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace ci-tooling.md's 'pull requests always build the kernel' with the dev kernel cache, and document the INSECURE dev pair, its release guard and why it reaches no artifact in secure-boot-keys.md.

Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy force-pushed the ci/gha-c2-dev-kernel-cache branch from b79c55f to 2be6f7f Compare October 5, 2026 04:43

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read at head 2be6f7f (diff + scripts/gen-dev-keys.sh, scripts/kernel-cache.sh, scripts/check-release-keys.sh, build.yml build/seed jobs, elements/bluefin-server/keys/*.bst, the new tests). Dev pair verified consistent (openssl x509 -pubkey == openssl pkey -pubout), cert valid to 2126, only the .crt reaches linux-module-cert.bst via files/boot-keys/modules.

docs-currency

  • (low) README.md:76 still says "Local builds use throwaway keys from just gen-dev-keys". After scripts/gen-dev-keys.sh:112-113 the module pair is the published files/dev-keys/ key, not throwaway — the user-facing README should carry the same INSECURE caveat the three updated skill docs do.

correctness

  • (low) Existing clones keep compiling the kernel: scripts/gen-dev-keys.sh:94-97 exits early when a full key set exists, so a contributor with pre-PR throwaway keys never receives the dev certificate until --force. Neither that message ("pass --force to regenerate") nor docs/skills/secure-boot-keys.md:34-37 says that is what makes the cache hit. One line in the early-exit message or the docs would do.

No findings from: security, intent-alignment, style.

Confidence: 3/5 (needs attention) — 1 perspective requested changes

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

…veat

gen-dev-keys now notes on its early exit when the existing module
certificate is not the dev one (the kernel builds locally until --force),
secure-boot-keys.md says the same, and the README no longer calls the
module key throwaway.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head bfb6323.

docs-currency

  • (medium) docs/skills/secure-boot-keys.md:90-91 — "Key rotation" step 1 still says to generate new production keys with just gen-dev-keys --force ("or the equivalent in a secure environment"). After this PR that command installs the published files/dev-keys/INSECURE-dev-module-key.pem as the module key (scripts/gen-dev-keys.sh:115-116), so an operator following the doc would build a release key set around the INSECURE pair; check-release-keys.sh would then fail the release build, but the doc should say --force --private-module-key here.
  • (low) AGENTS.md:54 still describes just gen-dev-keys as generating "throwaway Secure Boot, module, and image ... signing keys"; the module pair is no longer throwaway.

correctness

  • (low) scripts/gen-dev-keys.sh:94-100 — --private-module-key without --force on an existing complete key set is a silent no-op: it prints "Keys already exist" and exits 0, leaving the INSECURE module pair in place, with no hint that the flag was ignored. Worth a line in that branch (or an error) since the whole point of the flag is "hardware you care about".

style

  • (low) scripts/check-release-keys.sh:15-16 — the script cds to the repo root before reading KEY_DIR, so a relative argument resolves against the repo root rather than the caller's cwd; the usage comment at line 8 does not say so.

No findings from: security, intent-alignment.

Confidence: 3/5 (needs attention) — 1 medium finding; 1 perspective requested changes

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

gen-dev-keys.sh now errors when --private-module-key is passed without
--force while the INSECURE dev module key is installed, instead of
silently keeping it. Key rotation docs use --force --private-module-key,
AGENTS.md describes the committed dev module pair, and
check-release-keys.sh documents that a relative KEY_DIR is resolved
against the repo root.

Hive-Fix-Attempt: 2/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <scanner@hive.kubestellar.io>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

jeefy and others added 2 commits October 5, 2026 04:26
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jeefy
jeefy force-pushed the ci/gha-c2-dev-kernel-cache branch from c3908c4 to 8459737 Compare October 5, 2026 09:32

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

jeefy added a commit that referenced this pull request Oct 5, 2026
TestRateLimitPerSource joined three times in a row from one source. The server frees the source's slot after the client already has its answer, so the next attempt could be refused as busy ("attempt 1: control plane refused: busy"). It failed main's push builds 37264198746 and 37282526257 and both full builds of #404, inside homelab/bluefin-cluster.bst's go test. Wait for the slot like the replay tests do (#391).

Assisted-by: Claude Opus 5.5 via GitHub Copilot

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Assisted-by: Claude Opus 5.5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the tree at b39bc98, not just the diff. This looks correct to me; the key-isolation claims hold up independently.

What I verified:

  • elements/bluefin-server/keys/linux-module-cert.bst:15 imports only files/boot-keys/modules, and scripts/kernel-cache.sh:73-89 (stage_cert) refuses to seed if that directory holds anything but the one certificate or if the certificate text contains a private key — so the pushed import artifact stays cert-only for both variants. The existing boot-keys.bst-in-graph guard is still in place (kernel-cache.sh:99).
  • files/dev-keys/INSECURE-dev-module-key.{pem,crt} are a matching pair (openssl x509 -pubkey == openssl pkey -pubout), with keyUsage=digitalSignature, extendedKeyUsage=codeSigning, and a public key distinct from files/release-keys/linux-module-cert.crt.
  • Release path: .github/workflows/build.yml:298-299 runs scripts/check-release-keys.sh after staging the release certificate, and kernel-cache.sh:86-88 runs it for seed release. The script compares public keys (check-release-keys.sh:24-36), so a re-issued certificate for the dev key is also caught.
  • Non-release path: build.yml:303 and reproducibility.yml:67 call just gen-dev-keys, which now installs the committed pair with the same modes (0600/0644) stage_cert uses, so the dev kernel's cache key matches what kernel-cache-dev seeds.
  • kernel-cache-dev (build.yml:189-227) is not in any job's needs:, so a failed dev seed cannot delay a release.
  • The seven touched unit test modules pass locally (120 tests).

One suggestion, not a defect: files/dev-keys/README.md and docs/skills/secure-boot-keys.md already say the pair is public; it may be worth one sentence there telling contributors not to report it to secret scanners, since generic PEM blocks tend to trip third-party scanners on forks.

Confidence: 5/5 (safe)

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@jeefy
jeefy merged commit 158de34 into main Oct 5, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-build Run the full image build and boot test on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant