Skip to content

fix(cache): complete artifact replication without blocking releases - #670

Merged
castrojo merged 8 commits into
mainfrom
fix/cache-warming-background
Nov 15, 2026
Merged

castrojo merged 8 commits into
mainfrom
fix/cache-warming-background

Conversation

@castrojo

@castrojo castrojo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Production failure

Both first-main seed steps in run 43639399092 failed despite green jobs: release could not push the uncached linux-module-cert import; dev could not push bootstrap/base-sdk/binary-seed-x86_64. Cached targets made the default build plan skip build-only dependencies. continue-on-error masked both failures.

The first image in #640's merge-group run 43655195547 pulled 218 cached artifacts but still built 98 locally. Its expensive compiler artifacts included:

Artifact Observed build time
OpenZFS 15m15s
Ruby 13m59s
systemd base (components/_private/systemd-base.bst) 11m18s
NVIDIA modules 7m44s

These builds overlap; the times are not additive. This is measured uncached compilation, not evidence that cache transfers are stuck.

Fix

  • Run bst build --deps all before bst artifact push --deps all, materializing the complete key-free dependency closure even when roots are cached.
  • Keep six public seed roots: FSDK kernel, Go and GCC stage 2, the Brew machine, unsigned OpenZFS and unsigned NVIDIA modules. The actual Ruby and private systemd-base compiler elements are already transitive dependencies; no speculative or nonexistent systemd root is needed.
  • Move both main-only producers into cache-warm.yml with separate cache-warm-* concurrency, removing both the direct image-build dependency and the workflow-wide release-queue blockage.
  • Remove obsolete continue-on-error/warning wrappers: independent producer failures are red without gating releases.
  • Preserve main-only bst-cache credentials, both public certificate variants, private-key graph guards and every image/boot/publication gate.
  • Rebase onto merged chore(deps): update github-actions (non-major) #640 and preserve its updated action pins in the relocated producers.

Verification

  • Native pinned BuildStream 2.8.1 and two isolated CAS servers reproduced the cached-parent replication failure, then passed with build --deps all. With upstream stopped, an empty consumer materialized parent and build-only dependency payloads from destination.
  • Separately cold-pulled the real GCC bootstrap payload from only cache.projectbluefin.io:11001, then materialized executable ELF cc1 bytes. Payload proof, not metadata-only availability.
  • Current six-root publisher contract: 21 regression tests passed; shell syntax, actionlint and docs checks passed. The broader producer/CI contract selection also passed 97 tests with this executable selection.
  • Earlier full producer-fix validation: just test-unit passed 1936 Python tests with 30 environment-dependent skips and all 158 Bats tests; coverage 84.78%. Isolated Git configuration prevents workstation hooks from changing fixture behavior.
  • Native graph proof: https://github.com/projectbluefin/server/actions/runs/43662005909 . just validate passed; both release and dev certificate variants resolved every seed root plus the measured Ruby and components/_private/systemd-base.bst compiler artifacts, with the private-key import absent. The temporary proof branch is not part of this PR.

Sequencing and remaining gate

#664 and #640 are merged. #591 is independently in the normal merge queue; this PR does not gate it. No administrative bypass or self-approval.

Non-author review remains required; Danathar, daegalus and hanthor are already requested. After merge, both independent main producers must genuinely succeed before claiming the new artifacts are populated or a build-time improvement is measured. A green warning wrapper is not producer success.

@jeefy jeefy added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 14, 2026
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive

hivecommons-hive Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor
6ca1f0c

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @castrojo, head 6ca1f0c19941) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 2 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/build.yml
    • .github/workflows/cache-warm.yml
  • ci_gate_weakening — CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/build.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1 similar comment
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo
castrojo changed the base branch from fix/fsdk-containers-cache-alignment to main November 14, 2026 21:23
@castrojo castrojo added the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
@castrojo
castrojo marked this pull request as ready for review November 14, 2026 21:34
@castrojo
castrojo requested a review from Danathar November 14, 2026 21:34
@castrojo

Copy link
Copy Markdown
Contributor Author

Full local just test-unit now passes on 0ce055a: 1936 pytest tests passed, 30 environment-dependent skips, all 158 Bats tests passed, coverage 84.78% (84% required). Run with isolated Git global/system configuration so workstation commit hooks do not reject temporary fixture commits; no tests or source changed to obtain this result.

Requested a non-author OWNERS maintainer review. Keep held until both production seed jobs from #664 complete and a fresh-cache GCC payload pull/checkout from the project CAS succeeds. No queue entry or administrative bypass requested. The moved warmers have no changes dependency; explicit main-ref guards are retained because workflow_dispatch can target other refs. Existing actionlint and required-check policy tests cover reintroduction into the release workflow.

@jeefy
jeefy requested review from daegalus and hanthor November 14, 2026 21:34
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added dco All commits are DCO signed-off stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Nov 14, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

@castrojo this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the branch conflicts with the base and needs a rebase

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (4 waiting): fixes are handed out before new work as soon as you have capacity.

@castrojo

Copy link
Copy Markdown
Contributor Author

Cold production payload proof passed for GCC bootstrap key 1ec8073d05ef6f2bb9c15f903d266632f439e00cef81d026690ebd8c8ec4394f. Used a new empty cache with project recommendations overridden and https://cache.projectbluefin.io:11001 as the only artifact server. Native artifact pull --deps none completed successfully in 9m09s, followed by artifact checkout --deps none --no-integrate in 4s. Materialized cross/libexec/gcc/x86_64-unknown-linux-gnu/16.2.0/cc1 is a regular executable x86-64 ELF, 401875008 bytes; no build ran.

The initial combined command hit its 600s limit after the pull had succeeded because checkout defaults to runtime dependencies. Finished checkout of the intended GCC-only artifact without re-pulling it. This proves actual payload bytes, not just remote metadata. Both seed-job completion logs and non-author maintainer review remain outstanding; keep held.

@castrojo
castrojo force-pushed the fix/cache-warming-background branch from 0ce055a to 6bed72c Compare November 14, 2026 22:44
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo castrojo changed the title fix(ci): keep cache warming off the release critical path fix(cache): complete artifact replication without blocking releases Nov 14, 2026
@castrojo

castrojo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

@Danathar please review current head 6687800. The final test-only commit restores the semantic build-selection contract: dependency mode all and all four required cache targets, without pinning target order or long-option spelling. All 21 cache tests passed locally; fresh native CI, including unit, validate, docs, actionlint and zizmor, passed on this exact head.

Both old production seed steps failed on uncached build-only dependencies despite green job status. The native cached-parent fixture reproduces that failure and passes after materializing the full graph, including destination-only payload checkouts with upstream stopped. The full local gate on the unchanged production implementation passed: 1936 pytest, 30 environment-dependent skips, 158 Bats, 84.78% coverage.

The advisory-warming decoupling is implemented and verified in this PR; landing and production verification remain outstanding. Non-author maintainer review and /lgtm are required before lifting the hold and using the normal merge queue. After merge, both new independent main producers must genuinely succeed before refreshing and unholding #640, followed by #591. Do not infer success from the old green-but-failed producer jobs or bypass these gates.

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Nov 14, 2026
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo

Copy link
Copy Markdown
Contributor Author

/hold cancel

Removing the extra manual hold I introduced. Head 6687800 is review-ready with green CI; the required independent review remains the existing review gate and does not need a second manual hold. I am not granting LGTM to my own PR.

This cache-producer follow-up is independent of #640 and #591, which are already unheld and progressing through their native checks/queue. Both producers will need real completion verification when this PR lands.

@castrojo castrojo removed the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
@hivecommons-hive hivecommons-hive Bot added role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026
castrojo and others added 6 commits November 14, 2026 20:00
Image builds wait for graph classification and boot-tool preflight, not the four-hour cache seeder. Preserve all publication, boot-test, image-check and credential boundaries. A cold run may duplicate compilation while warming, but a slow producer cannot prevent release startup.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
The failing unit explicitly required the kernel-cache build dependency removed by this PR. Delete that scheduling assertion rather than re-pin its inverse; preserve release-only seed and no-signing-key security checks. The active cache repair at 47ce75e already has successful unit checks and is unchanged.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
Removing build.needs alone leaves the four-hour producers inside the workflow-wide main concurrency group, delaying later releases. Move the existing two seed jobs into a main-only cache-warm workflow with its own concurrency group; do not add producers or weaken environment restrictions. Migrate the credential, runner-disk and gate contracts and remove obsolete exceptions. Retain all image, boot and publication checks.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
…rgets

Both first-main seed runs failed while pushing uncached build-only dependencies: the release certificate import and the bootstrap binary-seed element. A cached parent makes the default build plan skip these dependencies. Select --deps all before the matching all-dependencies push. Native pinned BuildStream reproduced the failed cached-parent push, then proved destination-only payload checkouts after upstream was stopped.

Cache warming now lives outside release CI, so remove continue-on-error and redundant warning steps instead of turning failed seed jobs green. Preserve credential boundaries and update the failure-visibility contract. Remove an exact-argv snapshot; keep credential and failed-payload safety checks.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
Parse the emitted build selection and require dependency mode all plus the four published targets. Do not pin target ordering or long-option spelling. This guards the cached-parent regression verified by the native two-CAS proof. Production code is unchanged; the focused kernel-cache suite passed all 21 cases.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
The observed warm-kernel image build still compiled OpenZFS for 15m15s and NVIDIA modules for 7m44s. Seed their key-free compiler outputs and full dependency closures; keep signing stages private.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@castrojo
castrojo force-pushed the fix/cache-warming-background branch from 6687800 to 3790eaa Compare November 15, 2026 01:06
@jeefy

jeefy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The native seed-graph check proved Ruby is already in the Brew closure, but systemd-base is absent. Add the measured 11m18s compiler target explicitly instead of assuming it is transitively seeded.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@jeefy

jeefy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Nov 15, 2026
The measured compiler is components/_private/systemd-base.bst, not components/systemd-base.bst. Restore the valid six-root selection and verify the actual compiler in its transitive graph.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@jeefy

jeefy commented Nov 15, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo
castrojo added this pull request to the merge queue Nov 15, 2026
Merged via the queue into main with commit 2bd0dc5 Nov 15, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants