Skip to content

ci: upload key-free BuildStream artifacts to cache.projectbluefin.io from main #299

Closed

Description

@jeefy

Summary

CI rebuilds FSDK's kernel (components/linux.bst) from source on every run: 62–93 min of a ~110 min build step, about 85–90% of the build job. On main its cache key doesn't change between runs (main runs 36640985380 and 36651111965 both built kernel key 641192ff), but CI only pulls from caches, so every run starts cold.

Pushing a safe subset of our artifacts to cache.projectbluefin.io from main should bring main's build step from ~108 min down to roughly 12–18 min. PR builds get the same benefit later, once they use a fixed dev module key (separate decision; the lab's bluefin-server-build-pipeline already does this).

What we need from a cache / org admin

  1. Upload credential for projectbluefin/server. This uses the same mTLS scheme dakota uses (vars.CASD_CLIENT_CERT + secrets.CASD_CLIENT_KEY, uploading to https://cache.projectbluefin.io:11002). Ideally a server-specific client cert, so it can be revoked on its own. Please add it to this repo's bst-cache environment (already created; deployments restricted to main):
    • environment variable CASD_CLIENT_CERT (PEM client certificate)
    • environment secret CASD_CLIENT_KEY (PEM client key)
  2. Capacity sign-off. We'd upload the kernel plus roughly 15 FSDK/bootstrap elements that FSDK's cache doesn't hold (go, binutils-stage1, elfutils, lvm2-base, …) and our own key-free elements (ignition, openzfs, k0s/kubeadm binaries, …). Most of these only change when FSDK is bumped or a pin moves. Measured sizes will be posted in the implementing PR.
  3. Confirm that artifacts uploaded on :11002 are served from :11001. :11001 is the read-only endpoint already in project.conf, so PR builds and local builds would pull them with no config change.

Safety design (implemented in the follow-up PR)

  • Upload only from pushes to main. The build job uses the bst-cache environment only for push events on refs/heads/main. The environment's deployment rule allows only main (enforced by GitHub), so a PR that edits the workflow still can't get the credential. Fork PRs never see secrets at all.
  • Nothing key-derived is uploaded. The upload step runs after the build, with an explicit allow-list: the full graph minus bluefin-server/keys/boot-keys.bst, anything that uses files/boot-keys, and everything that depends on those (build or runtime). A unit test enforces this. The upload config is never active during bst build; otherwise BuildStream would upload every artifact it builds, including the keys.
  • Missing credentials mean no upload. Until the environment has both values, the step logs that it is skipping and the build behaves exactly as it does today.
  • Reproducibility is unaffected. reproducibility.yml already rebuilds the final artifacts with --ignore-project-artifact-remotes.

Prior art

  • dakota: .github/actions/generate-bst-ci-config/action.yml (config generation, mTLS, upload flag), .github/workflows/build.yml (how the secrets are passed and gated).
  • lab: argo/workflow-templates/bluefin-server-build-pipeline.yaml, which builds this repo with a fixed dev key "so a stable dev key keeps the kernel … in the BuildBarn CAS".

Activity

  1. added
    hive/covered-by-prHive verified that an open PR references or claims this issue; still actionable until confirmed
    on Sep 30, 2026
  2. kylerankin commented on Sep 30, 2026

    @kylerankin
    Contributor

    The shippable follow-up is ready: PR #302 (non-draft, base main).

    It implements the safety design from the issue:

    • .github/scripts/bst-push-elements.py — explicit key-free allow-list, fails closed if any element is key-derived (boot-key element, files/boot-keys, or a transitive build/run dependency of either).
    • Upload steps in .github/workflows/build.yml, gated to pushes on refs/heads/main behind the bst-cache environment; clean skip with no behavior change when credentials are absent.
    • tests/unit/test_bst_push_allowlist.py enforces the invariant on the real graph (207 passing).

    The three admin asks are left for you: create/point the bst-cache environment at a revocable server-specific client cert, capacity sign-off, and confirm :11001 vs :11002.

    — hive: backend=pi model=lemonade/Ornith-1.5-35B-A3B-GGUF-Q6_K


    🐝 Hive Agent: contributor | SHA: 4892da5

  3. Danathar commented on Sep 30, 2026

    @Danathar
    Contributor

    Heads-up: there are two open PRs for this issue, and neither links to the other.

    Both are now conflicting with main. Choosing one approach before either gets rebased would save a round of work.

  4. removed
    hive/covered-by-prHive verified that an open PR references or claims this issue; still actionable until confirmed
    on Sep 30, 2026
  5. dtg01100 commented on Oct 2, 2026

    @dtg01100
    Contributor

    smoke test


    🐝 Hive Agent: contributor | SHA: 9994af8

  6. jeefy commented on Oct 3, 2026

    @jeefy
    CollaboratorAuthor

    Status: #365 (ahmedadan) now seeds the kernel and Go into cache.projectbluefin.io:11002 over mTLS from the kernel-cache job, and builds pull them through project.conf's remotes. That's the main win this issue was after, so #301 (closed) is no longer needed. Open follow-up from #355: move CASD_CLIENT_KEY into a GitHub Environment that only main can deploy to (this repo already has bst-cache, restricted to main), attach that environment to the kernel-cache job, and delete the repository-level secret. That needs the key's value, so an admin has to do it. Then this can close.

  7. jeefy commented on Oct 6, 2026

    @jeefy
    CollaboratorAuthor

    Done. #365 seeds the kernel and Go into the project cache from main. #404 also seeds the dev-key kernel, so PR and nightly builds can reuse it. The remaining step, moving the CASD_CLIENT_KEY value into the bst-cache environment, is documented in docs/skills/ci-tooling.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions