Repository navigation
fix(build): repair shared caching and boot clock epoch - #664
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Danathar
left a comment
There was a problem hiding this comment.
Stand-in review (the hive's reviewers are out of quota). Commenting, not approving yet. The diagnosis fits #650's evidence: about 340 remote pull misses, with fsdk-containers.bst:freedesktop-sdk.bst:… built from source as a separate graph. Collapsing that junction should end the timeouts. But this needs @jeefy's call, for three reasons:
- It reverses a deliberate design. The removed
junctions: duplicates:block inproject.confwas documented as intentional: "Host /usr and the independently packaged Brew machine never share a rootfs." The duplicate let the Brew machine keep fsdk-containers' own FSDK. - It introduces an FSDK version skew. With
overrides: freedesktop-sdk.bst: freedesktop-sdk.bst, the fsdk-containers elements server uses (oci/brew-nspawn.bstinhomebrew/image.bst, andbrew/brew-runtime.bstandbrew/brew-prefix.bstin the SBOM) build against server's FSDK, 26.08.3 (d0c0b597). The bumped fsdk-containers refb79382bdpins 26.08.4 (114ae909). So the Brew machine would ship a different FSDK from what fsdk-containers builds and tests. - It overlaps a hold. The 26.08.3 → 26.08.4 bump is #543, which jeefy put on hold. Landing #543 first, or together with this, would remove the skew.
Worth recording either way: the zizmor failure here is #651 (the install-action pin comments), not this change. The full build is still running. A finish well inside the 5-hour limit, with mostly remote hits, would confirm the cache diagnosis.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@castrojo this PR is back in your court ( Failing checks:
The same checks have failed on 2 heads so far (latest Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (9 waiting): fixes are handed out before new work as soon as you have capacity. |
Override freedesktop-sdk.bst onto host freedesktop-sdk.bst in elements/fsdk-containers.bst, bump junction ref to b79382bd, and remove dead duplicate junction declaration from project.conf. This eliminates 311 duplicate elements (including the 64-element FSDK bootstrap), collapses the homebrew build closure from 713 to 361 elements, and aligns glibc/toolchain cache keys so builds pull directly from cache.projectbluefin.io rather than compiling from source for 5 hours. Assisted-by: GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo <castrojo@users.noreply.github.com>
95defd9 to
d6e0145
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/check-dco |
|
Stand-in re-review at d6e0145 (rewritten since 95defd9):
No objection beyond that. My earlier #543-hold note was for the owners, and this is now castrojo's own change. |
…check-image Assisted-by: Gemini-3.8-flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo <castrojo@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Consolidated the cache source fixes here in f96222b: one shared SDK graph, Brew machine cache population, project CAS first, and all seven install-action pin comments corrected. No dependency PR is a prerequisite now. Verification: just validate passed; the actual 369-element seed graph contains no boot-keys element and no duplicate nested SDK; 47 focused tests passed; shell syntax, ruff, actionlint, docs checks and online zizmor passed. Existing CI logs show kernel and Go remote pulls succeeded; brew-nspawn is the remaining uncached seed target. #640, #667, #591 and #665 are held; pre-fix queue builds were cancelled and the queue was emptied. Removed full-build here to avoid duplicating the mandatory full build: this corrected head will get its full image, boot and image checks as the first and only merge-queue entry. No branch protection bypass. |
47ce75e
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @castrojo, head
Hive did not add the |
|
Re-review at f96222b:
No further concerns from me. |
|
Checked the remaining review concerns against the actual code and graph:
|
Remote availability can advertise an artifact whose CAS payload cannot be pulled. Let BuildStream perform native pull/build/push checks instead of reporting seed success from metadata alone. Make the observed missing GCC stage2 compiler an explicit seed target, rather than relying on a build dependency being needed. Assisted-by: GPT-6.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo <castrojo@users.noreply.github.com>
|
The reproduced metadata-only seed bug is fixed and tested in stacked #669 (dfb3337): native BuildStream validates payloads, and GCC stage2 is an explicit target. The regression failed before the change and passed afterward; 46 focused tests, graph safety, just validate, shell syntax, ruff and docs checks passed. #669 is draft and is not queued, preserving this running build. Dependencies remain held until both cache fixes land and actual seed payload pulls succeed. |
A native two-CAS test confirms that building a cached target leaves build-only dependencies outside the destination cache. Explicit artifact push --deps all pulls and replicates those dependencies. Only report successful seeding after replication succeeds. Assisted-by: GPT-6.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo <castrojo@users.noreply.github.com>
|
/hold The full build completed successfully (16:32:29–18:09:35 UTC) and image-check passed. Boot failures share a concrete clock defect: RTC initializes to 2026-11-14, then systemd logs "System time was further ahead than 15y after built-in epoch, clock reset to Thu 2011-11-10 15:00:00 UTC". GPG consequently rejects the newly generated signing key as created 5483 days in the future. Fixing the stale systemd epoch; do not requeue this head or relax signature verification. |
PID 1 resets a correct 2026 RTC to the SDK build epoch in 2011 once its 15-year window expires, causing GPG to reject matching image keys as future-dated. Produce /usr/lib/clock-epoch from the pinned SDK year/month and restore its mtime after initrd clamping and before repart. Keep signature verification and SDK cache keys unchanged. Assisted-by: GPT-6.1 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: castrojo <castrojo@users.noreply.github.com>
|
All source corrections are now on this branch at 47ce75e, including the #669 metadata/payload and explicit dependency-replication fixes. #669 is superseded. The boot root cause is fixed with systemd native /usr/lib/clock-epoch, using a reproducible SDK release-month floor and preserving its mtime through cpio/repart. No SDK recompilation or signature-time bypass. Proof before requeue:
Dependencies stay held. This corrected head needs its cheap PR checks, then exactly one full merge-queue build. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: castrojo The full list of commands accepted by this bot can be found here. Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/hold cancel 47ce75e includes the reproduced clock-epoch root fix and all cache corrections. Native RTC/signature verification and reproducible timestamp proofs passed; enable normal merge-queue processing only after required checks pass. Dependencies remain held. |
|
Re-review at 47ce75e (dfb3337, fbca5f6, 47ce75e):
Nothing blocking beyond that one question. The full build/image-check/boot-test still hasn't run on the PR. |
|
Remaining advisory concerns checked without restarting this build:
|
|
Concern-resolution checkpoint: #670 is now complete and CI-green at 0ce055a, including required unit (https://github.com/projectbluefin/server/actions/runs/43634713042/job/132568871482), Go, actionlint, zizmor, docs and graph validation. The original failed unit was specific to #670 at 39f2371: its obsolete assertion required the dependency being removed. #664 at 47ce75e already had successful unit checks and was not restarted. The full scheduling fix moves the existing two warmers into cache-warm.yml with separate concurrency; merely removing build.needs would still leave subsequent releases blocked by the old workflow-wide group. Credential/environment boundaries and image/boot/publication gates remain. Local checks: 102 focused tests plus staged Renovate pin coverage passed. #670 stays draft behind this PR to preserve the running full build. Main payload proof and landing #670 are still prerequisites before unholding dependencies. #640 also needs its action-pin updates migrated to the moved warmer jobs after the base updates; that is recorded on #640. |
Brings in the boot clock epoch fix (projectbluefin#664). Without it the guest clock resets to 2011, gpg rejects the dev signing key as future-dated and the diskless and install boot tests fail on "Signature verification failed". Conflicts: - workflows: take main's install-action pin and its removal of the kernel-cache jobs; keep homebrew/seed.bst in the element list. - project.conf: take main's; it already drops the duplicate junction. - elements/fsdk-containers.bst: stays deleted; nothing builds the Brew machine any more. - scripts/kernel-cache.sh: drop the fsdk-containers.bst:oci/brew-nspawn.bst shared seed target, which no longer resolves. - docs: keep the base-image Homebrew text; drop the Brew machine notes. Assisted-by: github-copilot/claude-fable-5.2 via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Root causes
Fixes on this PR
Observed verification
Landing order
All fixes are on this branch at 47ce75e. #665, #667 and #669 are closed as superseded. #640 and #591 remain held until this PR lands, both main seed variants succeed and project-CAS payload pulls work. Cheap PR checks first; exactly one full merge-queue build of this corrected head, no administrative bypass.