Skip to content

fix(build): repair shared caching and boot clock epoch - #664

Merged
castrojo merged 6 commits into
mainfrom
fix/fsdk-containers-cache-alignment
Nov 14, 2026
Merged

castrojo merged 6 commits into
mainfrom
fix/fsdk-containers-cache-alignment

Conversation

@castrojo

@castrojo castrojo commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Root causes

  1. An independent fsdk-containers SDK graph caused avoidable cold bootstrap builds.
  2. The seed script trusted remote metadata without validating payloads and did not explicitly replicate all build dependencies into the project CAS.
  3. PID 1 reset a correct 2026 RTC to its 2011 built-in epoch after the 15-year sanity window expired. GPG then rejected matching signing keys as future-dated; this broke diskless installs and installed updates.

Fixes on this PR

  • Share the server SDK junction; retain separate Brew/host filesystems.
  • Seed kernel, Go, GCC stage2 and Brew; let native BuildStream validate/pull/rebuild payloads, then explicitly artifact push --deps all to the authenticated project CAS.
  • Prefer the project CAS while retaining upstream fallbacks. Preserve key-free graph guards and main-only write credentials.
  • Correct all seven action-pin version comments so this fix has no dependency-PR prerequisite.
  • Generate native /usr/lib/clock-epoch from the pinned SDK year/month. Restore its mtime after initrd clamping and before repart; retain reproducibility and all signature checks. No systemd/SDK recompilation.
  • Remove the obsolete image-check waiver and ignore deliberately broken Homebrew test-fixture ELFs.

Observed verification

  • Previous full run 43616111754: build succeeded in 97 minutes; image-check and release-dry-run passed. Three boot buckets failed from the identified clock reset. That run does NOT verify this new clock-fix head.
  • Native targeted guest: original signed DDI accepted with NTP disabled; RTC remained 2026 instead of resetting to 2011. Direct-kernel KVM with Secure Boot off, stopped before switch-root; full Secure Boot/update gates still required.
  • Native cpio and repart/erofs: clock-epoch mtime preserved at 1785542400; repeated outputs byte-identical. Assembly proof used host tools; guest signature proof used the original SDK guest.
  • Actual modified os-release element built successfully. Regression executes its real recipe and checks both epoch content and mtime for SDK 26.08 and 27.02.
  • 26 focused clock/cache/reproducibility tests, pinned ruff, docs checks and just validate passed after the clock change.
  • Native two-CAS test: explicit dependency push made a fresh client able to retrieve payloads with upstream stopped. Cache/workflow tests previously passed 47 cases.

Landing order

All fixes are on this branch at 47ce75e. #665, #667 and #669 are closed as superseded. #640 and #591 remain held until this PR lands, both main seed variants succeed and project-CAS payload pulls work. Cheap PR checks first; exactly one full merge-queue build of this corrected head, no administrative bypass.

@jeefy
jeefy requested review from ahmedadan and daegalus November 14, 2026 01:01
@jeefy jeefy added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 14, 2026
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo castrojo added the full-build Run the full image build and boot test on this pull request label Nov 14, 2026
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login no-dco A commit is missing Signed-off-by — run `git rebase --signoff` role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stand-in review (the hive's reviewers are out of quota). Commenting, not approving yet. The diagnosis fits #650's evidence: about 340 remote pull misses, with fsdk-containers.bst:freedesktop-sdk.bst:… built from source as a separate graph. Collapsing that junction should end the timeouts. But this needs @jeefy's call, for three reasons:

  1. It reverses a deliberate design. The removed junctions: duplicates: block in project.conf was documented as intentional: "Host /usr and the independently packaged Brew machine never share a rootfs." The duplicate let the Brew machine keep fsdk-containers' own FSDK.
  2. It introduces an FSDK version skew. With overrides: freedesktop-sdk.bst: freedesktop-sdk.bst, the fsdk-containers elements server uses (oci/brew-nspawn.bst in homebrew/image.bst, and brew/brew-runtime.bst and brew/brew-prefix.bst in the SBOM) build against server's FSDK, 26.08.3 (d0c0b597). The bumped fsdk-containers ref b79382bd pins 26.08.4 (114ae909). So the Brew machine would ship a different FSDK from what fsdk-containers builds and tests.
  3. It overlaps a hold. The 26.08.3 → 26.08.4 bump is #543, which jeefy put on hold. Landing #543 first, or together with this, would remove the skew.

Worth recording either way: the zizmor failure here is #651 (the install-action pin comments), not this change. The full build is still running. A finish well inside the 5-hour limit, with mostly remote hits, would confirm the cache diagnosis.

@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

@castrojo this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • zizmor

The same checks have failed on 2 heads so far (latest faf8e309f7cb).

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (9 waiting): fixes are handed out before new work as soon as you have capacity.

Override freedesktop-sdk.bst onto host freedesktop-sdk.bst in
elements/fsdk-containers.bst, bump junction ref to b79382bd, and remove
dead duplicate junction declaration from project.conf.

This eliminates 311 duplicate elements (including the 64-element FSDK
bootstrap), collapses the homebrew build closure from 713 to 361 elements,
and aligns glibc/toolchain cache keys so builds pull directly from
cache.projectbluefin.io rather than compiling from source for 5 hours.

Assisted-by: GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@castrojo
castrojo force-pushed the fix/fsdk-containers-cache-alignment branch from 95defd9 to d6e0145 Compare November 14, 2026 14:35
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo

Copy link
Copy Markdown
Contributor Author

/check-dco

@hivecommons-hive hivecommons-hive Bot added dco All commits are DCO signed-off stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed no-dco A commit is missing Signed-off-by — run `git rebase --signoff` stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Nov 14, 2026
@Danathar

Copy link
Copy Markdown
Contributor

Stand-in re-review at d6e0145 (rewritten since 95defd9):

  • The mechanism is right. overrides: freedesktop-sdk.bst: freedesktop-sdk.bst in elements/fsdk-containers.bst collapses the nested FSDK onto server's, which is the standard BuildStream way to do it, and the junctions: duplicates: allowance in project.conf goes away with it. The ref moves to b79382bd (the shared printing-layer commit).
  • Version skew, now a deliberate choice: server pins FSDK 26.08.3 (elements/freedesktop-sdk.bst), while fsdk-containers at b79382bd pins 26.08.4. With the override, fsdk-containers' elements build against 26.08.3 here, which is fine as long as nothing in them needs 26.08.4. Bumping server to 26.08.4 first (or alongside) would remove the question.
  • The only failing check is zizmor. That's the shared ref-version-mismatch failure on every server PR, fixed by ci: name the exact install-action tag its pin points to #667. It's not this change.
  • The full build hasn't run on this head (13 checks skipped). I'd want a green full build before merging, since this changes how every fsdk-containers element resolves.

No objection beyond that. My earlier #543-hold note was for the owners, and this is now castrojo's own change.

@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026
…check-image

Assisted-by: Gemini-3.8-flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo

Copy link
Copy Markdown
Contributor Author

Consolidated the cache source fixes here in f96222b: one shared SDK graph, Brew machine cache population, project CAS first, and all seven install-action pin comments corrected. No dependency PR is a prerequisite now.

Verification: just validate passed; the actual 369-element seed graph contains no boot-keys element and no duplicate nested SDK; 47 focused tests passed; shell syntax, ruff, actionlint, docs checks and online zizmor passed. Existing CI logs show kernel and Go remote pulls succeeded; brew-nspawn is the remaining uncached seed target.

#640, #667, #591 and #665 are held; pre-fix queue builds were cancelled and the queue was emptied. Removed full-build here to avoid duplicating the mandatory full build: this corrected head will get its full image, boot and image checks as the first and only merge-queue entry. No branch protection bypass.

@castrojo
castrojo enabled auto-merge November 14, 2026 16:28
@castrojo
castrojo added this pull request to the merge queue Nov 14, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor
47ce75e

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @castrojo, head 47ce75e01338) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 4 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/build.yml
    • .github/workflows/reproducibility.yml
    • .github/workflows/track-junctions.yml
    • .github/workflows/unit-tests.yml
  • remote_code_execution — adds a remote-fetch-and-execute, decoded payload, reverse shell or embedded credential (Adds curl|sh, eval of encoded payloads, reverse shells or embedded private keys)
    • tests/unit/test_clock_epoch.py

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act labels Nov 14, 2026
@Danathar

Copy link
Copy Markdown
Contributor

Re-review at f96222b:

  • Sentinel sensitive_path (4 workflows): benign. All four workflow edits are comment-only, # v2 → # v2.90.2 on the same taiki-e/install-action@3a3daafd… pin (tag v2.90.2 is that commit). Nothing that runs changes. These are the same seven edits as ci: name the exact install-action tag its pin points to #667, and identical hunks merge cleanly in either order, so whichever lands second just becomes a no-op there.
  • project.conf: the project CAS (cache.projectbluefin.io:11001) is now queried before gbm.gnome.org:11003 for artifacts and sources. Same two pull-only caches, same connection settings, new order.
  • kernel-cache.sh: the seed set gains fsdk-containers.bst:oci/brew-nspawn.bst, and the new test_seed_builds_when_only_brew_is_missing covers the case where only Brew is uncached. Push still goes through the client certificate to :11002.
  • zizmor passes now. The full build / image-check / boot-test jobs are still skipped on the PR, so the verification you list (just validate, the 369-element seed graph, 47 tests) is the evidence for the collapse. That matches what I'd want to see.

No further concerns from me.

@castrojo

Copy link
Copy Markdown
Contributor Author

Checked the remaining review concerns against the actual code and graph:

  • The 369-element Brew/kernel/Go seed closure DOES include freedesktop-sdk.bst:bootstrap/build/gcc-stage2.bst (verified after stripping ANSI from bst show output).
  • Every push to main sets release=true in changes, so both existing seed jobs already run on every main push. There is no release-tag-only seed gap. The merge-queue build stays key-free and cannot use the main-only bst-cache credentials; I am preserving that boundary.
  • The metadata-only skip is a real separate correctness problem: artifact show reported GCC available, but a direct artifact pull retrieved no payload. Preparing a small seed correction independently so this running image build is not restarted: explicitly target GCC stage2 and let native bst build validate/pull/rebuild instead of returning success from metadata alone.
  • ci: name the exact install-action tag its pin points to #667 and fix(cache): seed brew-nspawn to CAS and prioritize Bluefin remote #665 are now closed as superseded, preventing redundant re-enqueue. No dependency update will be unheld until the cache corrections are landed and seed payloads verified.

Remote availability can advertise an artifact whose CAS payload cannot be pulled. Let BuildStream perform native pull/build/push checks instead of reporting seed success from metadata alone. Make the observed missing GCC stage2 compiler an explicit seed target, rather than relying on a build dependency being needed.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Contributor Author

The reproduced metadata-only seed bug is fixed and tested in stacked #669 (dfb3337): native BuildStream validates payloads, and GCC stage2 is an explicit target. The regression failed before the change and passed afterward; 46 focused tests, graph safety, just validate, shell syntax, ruff and docs checks passed. #669 is draft and is not queued, preserving this running build. Dependencies remain held until both cache fixes land and actual seed payload pulls succeed.

A native two-CAS test confirms that building a cached target leaves build-only dependencies outside the destination cache. Explicit artifact push --deps all pulls and replicates those dependencies. Only report successful seeding after replication succeeds.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 14, 2026
@castrojo

Copy link
Copy Markdown
Contributor Author

/hold

The full build completed successfully (16:32:29–18:09:35 UTC) and image-check passed. Boot failures share a concrete clock defect: RTC initializes to 2026-11-14, then systemd logs "System time was further ahead than 15y after built-in epoch, clock reset to Thu 2011-11-10 15:00:00 UTC". GPG consequently rejects the newly generated signing key as created 5483 days in the future. Fixing the stale systemd epoch; do not requeue this head or relax signature verification.

@jeefy jeefy added the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
PID 1 resets a correct 2026 RTC to the SDK build epoch in 2011 once its 15-year window expires, causing GPG to reject matching image keys as future-dated. Produce /usr/lib/clock-epoch from the pinned SDK year/month and restore its mtime after initrd clamping and before repart. Keep signature verification and SDK cache keys unchanged.

Assisted-by: GPT-6.1 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: castrojo <castrojo@users.noreply.github.com>
@castrojo

Copy link
Copy Markdown
Contributor Author

All source corrections are now on this branch at 47ce75e, including the #669 metadata/payload and explicit dependency-replication fixes. #669 is superseded.

The boot root cause is fixed with systemd native /usr/lib/clock-epoch, using a reproducible SDK release-month floor and preserving its mtime through cpio/repart. No SDK recompilation or signature-time bypass.

Proof before requeue:

  • Original built guest retained RTC 2026-11-14 and systemd-importd reported Signature verification succeeded for the original signed DDI, with NTP disabled. Direct-kernel experiment, Secure Boot off; stopped before switch-root. Full Secure Boot/installed-update checks still required.
  • Native cpio and repart/erofs retained clock-epoch mtime 1785542400; repeated images were byte-identical. Assembly proof used host tools, while guest signature proof used the original SDK guest.
  • Real os-release element built successfully. Regression checks both epoch content and mtime for SDK 26.08 and 27.02.
  • 26 focused clock/cache/reproducibility tests, pinned ruff, docs checks and just validate passed. Earlier replication fix also passed 47 focused checks and native two-CAS replication with upstream stopped.

Dependencies stay held. This corrected head needs its cheap PR checks, then exactly one full merge-queue build.

@jeefy

jeefy commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: castrojo

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@castrojo castrojo changed the title fix(build): collapse fsdk-containers FSDK junction onto host FSDK fix(build): repair shared caching and boot clock epoch Nov 14, 2026
@castrojo

Copy link
Copy Markdown
Contributor Author

/hold cancel

47ce75e includes the reproduced clock-epoch root fix and all cache corrections. Native RTC/signature verification and reproducible timestamp proofs passed; enable normal merge-queue processing only after required checks pass. Dependencies remain held.

@castrojo castrojo removed the hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2026
@castrojo
castrojo added this pull request to the merge queue Nov 14, 2026
@Danathar

Copy link
Copy Markdown
Contributor

Re-review at 47ce75e (dfb3337, fbca5f6, 47ce75e):

  • Clock epoch: the fix is right. systemd takes /usr/lib/clock-epoch's mtime as the earliest time PID 1 will allow, and the build's clamp was resetting it to the SDK's 2011. Deriving it from installer-version (26.08.3 → 2026-08-01) is reproducible, and test_release_version.py keeps that YY.MM.N shape. Re-touching it after the initrd clamp (bluefin-server-boot.bst) is needed.
  • Worth confirming it's intended: in bluefin-server-usr.bst the new line exports SOURCE_DATE_EPOCH=$(cat …/clock-epoch) for the whole systemd-repart run. repart clamps every file's mtime to SOURCE_DATE_EPOCH, so this moves all /usr mtimes to the installer month (2026-08-01), not just clock-epoch's. It's still reproducible, but if the goal is only to keep that one file's mtime, re-touching it after repart (or limiting the export to that file) would keep the rest of the image's clamp as it was.
  • Cache: making seed push payloads that bst build has verified, rather than trusting available metadata, is the safer contract. Seeding on every push to main costs CI time on each merge; that's your call.

Nothing blocking beyond that one question. The full build/image-check/boot-test still hasn't run on the PR.

@castrojo

Copy link
Copy Markdown
Contributor Author

Remaining advisory concerns checked without restarting this build:

  • Pinned BuildStream natively executes manual commands with sh -c -e. A malformed-date build aborted at date with exit 1, so the claimed silent-success epoch bug is not present; no arbitrary 10-digit check added.
  • The release-startup dependency is real: build currently needs the 240-minute kernel-cache job. Verified removal is prepared in stacked draft fix(cache): complete artifact replication without blocking releases #670 (39f2371); it preserves all preflight/boot/image/publication gates and has 45 focused tests, actionlint, online zizmor, docs checks and just validate passing.
  • fix(cache): complete artifact replication without blocking releases #670 is not queued and does not change this running head. Retarget and land it after this build/merge, before resuming dependencies. Until then, the first main run still uses the old seeder dependency. Cold compilation can overlap warming after decoupling; no unmeasured speedup is claimed.

@castrojo

Copy link
Copy Markdown
Contributor Author

Concern-resolution checkpoint: #670 is now complete and CI-green at 0ce055a, including required unit (https://github.com/projectbluefin/server/actions/runs/43634713042/job/132568871482), Go, actionlint, zizmor, docs and graph validation. The original failed unit was specific to #670 at 39f2371: its obsolete assertion required the dependency being removed. #664 at 47ce75e already had successful unit checks and was not restarted.

The full scheduling fix moves the existing two warmers into cache-warm.yml with separate concurrency; merely removing build.needs would still leave subsequent releases blocked by the old workflow-wide group. Credential/environment boundaries and image/boot/publication gates remain. Local checks: 102 focused tests plus staged Renovate pin coverage passed.

#670 stays draft behind this PR to preserve the running full build. Main payload proof and landing #670 are still prerequisites before unholding dependencies. #640 also needs its action-pin updates migrated to the moved warmer jobs after the base updates; that is recorded on #640.

Merged via the queue into main with commit e0a63e8 Nov 14, 2026
43 checks passed
jeefy added a commit to jeefy/server that referenced this pull request Nov 16, 2026
Brings in the boot clock epoch fix (projectbluefin#664). Without it the guest clock
resets to 2011, gpg rejects the dev signing key as future-dated and the
diskless and install boot tests fail on "Signature verification failed".

Conflicts:
- workflows: take main's install-action pin and its removal of the
  kernel-cache jobs; keep homebrew/seed.bst in the element list.
- project.conf: take main's; it already drops the duplicate junction.
- elements/fsdk-containers.bst: stays deleted; nothing builds the Brew
  machine any more.
- scripts/kernel-cache.sh: drop the fsdk-containers.bst:oci/brew-nspawn.bst
  shared seed target, which no longer resolves.
- docs: keep the base-image Homebrew text; drop the Brew machine notes.

Assisted-by: github-copilot/claude-fable-5.2 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants