Skip to content

ci(canary): prove pipeline changes on the real workflow in minutes - #257

Merged
hanthor merged 3 commits into
mainfrom
ci/canary
Sep 25, 2026
Merged

hanthor merged 3 commits into
mainfrom
ci/canary

Conversation

@hanthor

@hanthor hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member

Top-priority item: a canary so every pipeline change is proven in minutes on the real jobs, before any full run.

What

.github/workflows/canary.yml calls rebuild-rpms.yml itself (new workflow_call trigger), not a copy, over a fixed set that spans two stages with a reverse dependency: libtalloc, libtdb (stage 0) and libtevent (stage 1, BuildRequires: libtalloc-devel). All three run-time close over Hummingbird alone (checked against the published repodata and Hummingbird's primary.xml), and built in about 4.3 min each in recent runs.

job asserts
pass1 builds, precedence, Hummingbird-only transaction over every binary built (consumer: built), publish + cosign sign + provenance attestation to utah-packages:canary-pr<N>. Never latest: both prepare and publish refuse it.
verify-publish reads the pushed digest with Utah's own scripts/check-repo-availability.py (fetched from Utah main at run time, repository_metadata(): layer 0 < 64 MiB, only repository/repodata/*), asserts exactly two layers with the small one first and that the repodata carries exactly the canary set, then cosign verify against the workflow identity and gh attestation verify.
pass2 same set, skip_publish: every build job must be a cache hit, zero compiles.
pass3 same set + python-typing-inspection with its spec perturbed in the checkout: it compiles, the canary set must still hit.
verify-cache reads the run's jobs API and checks the above per step (Build the verified source… skipped + Restore package RPM cache success), writing a table to the job summary.
Canary the required check. Always runs; passes with nothing built when the PR touches no pipeline path (.github/workflows/, .github/actions/, tools/, config/, Containerfile*); fails for fork PRs that do (they cannot publish).

pass2 and pass3 only need pass1's cache entries, so they run in parallel.

The cache is salted with a digest of the build path (tools/canary.py salt: build-stage.yml, the load actions, hummingbird.repo, source_pipeline.py, dist_bump.py, package_cache_key.py). A PR that changes how packages compile compiles for real in pass1; anything else reuses the previous canary's builds. Dispatch with compile=true to force it.

Production behaviour is unchanged

Every new input defaults to what a production run does:

  • empty cache_salt is left out of the key payload entirely, so production keys are byte-for-byte what ci(buildroot): factory-owned build-root mirror, and a cache key that survives a publish #252 made them (new test recomputes the unsalted key).
  • artifact_prefix empty → identical artifact names.
  • call-only booleans are negative (no_factory, skip_publish) so null on schedule/dispatch reads as false; GitHub's null == false is true, which would have silently disabled publish.
  • the publish seed is now needs.prepare.outputs.seed_image, which equals factory_image unless a publish_tag is given (repeated mistake 14 kept).

One real bug fixed on the way

build-stage.yml downloaded earlier stages with pattern: ${{ steps.prior.outputs.pattern }} but no step had id: prior, so the pattern was empty and download-artifact pulled every artifact of the run into work/prior, including same-stage siblings that finished first (repeated mistake 3) and the build-root tarball. A prior step now names rpm-s{0,…,N-1}-*. This can surface a mis-staged package that only built because a same-stage sibling leaked in; that is the intended rule, and item 1 (solved waves) removes the hand staging anyway.

After merge

I'll make Canary a required status check on main.

Proof

The canary run on this PR is the proof; numbers to follow in a comment.

🤖 Generated with Claude Code

A pipeline change used to be proven by dispatching the full factory and
reading it five to ten hours later. canary.yml calls rebuild-rpms.yml
itself (now also a workflow_call target) over libtalloc, libtdb (stage 0)
and libtevent (stage 1, BuildRequires libtalloc-devel):

- pass1 builds, checks precedence and a Hummingbird-only transaction over
  every binary built, and publishes, signs and attests
  utah-packages:canary-<pr>, never latest. verify-publish reads the digest
  with Utah's own check-repo-availability.py fetched from Utah main,
  asserts two layers with repodata first and exactly the canary set, and
  verifies the cosign signature and the provenance attestation.
- pass2 repeats the set and must compile nothing (cache key stable).
- pass3 adds python-typing-inspection with a perturbed recipe: it compiles,
  every other package must still hit (no wholesale invalidation).

The "Canary" job always runs and is the required check; it passes without
building when a PR touches no pipeline path.

Every new rebuild-rpms/build-stage input defaults to production behaviour:
an empty cache salt leaves production keys byte-for-byte unchanged (tested),
and call-only booleans are negative so null on schedule/dispatch reads false.

Also fixes build-stage.yml downloading every artifact of the run into
work/prior: it read steps.prior.outputs.pattern with no step of that id, so
same-stage siblings leaked into the stages repository (repeated mistake 3).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
hanthor and others added 2 commits September 25, 2026 17:51
…ulkan-loader

samba.org answered the runner with bytes that did not match the talloc lock
while serving the right tarball everywhere else; the source pipeline rightly
refuses that. A canary has to fail on the pipeline, not on an upstream host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
@hanthor

hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Canary proof: run 36170094324, green

job start → end (UTC) duration
changes 17:56:15 → 17:56:22 7 s
pass1 (prepare → 2 waves → precedence → publish/sign/attest) 17:56:26 → 18:07:39 11 m 13 s
verify-publish (Utah's reader, 2 layers, cosign verify, gh attestation verify) 18:07:43 → 18:07:56 13 s
pass2 ∥ pass3 18:07:43 → 18:14:20 6 m 37 s
verify-cache 18:14:24 → 18:14:29 5 s
Canary (gate) → 18:14:37 18 m 22 s from first job; 19 m 44 s from trigger

Cache verdict (from the run's own jobs API):

pass libical vulkan-headers vulkan-loader python-typing-inspection
pass1 compiled compiled compiled
pass2 cache hit cache hit cache hit
pass3 cache hit cache hit cache hit compiled (perturbed)

Published ghcr.io/projectbluefin/utah-packages:canary-pr257, never latest.

About the 15-minute target: pass1's critical path is libical (5 m 11 s with its %check), then vulkan-loader, which sits at config stage 4 and waits for the whole of wave 0. Once item 1 solves waves from BuildRequires it moves to wave 1. The remaining lever is swapping libical for a faster stage-0 package. I'll tune this after item 1 lands, measured on the canary.

Two things the first attempts turned up:

  • pull_request events carry github.event.before, and a force push makes that commit unreachable, so git diff killed prepare. Only a push range is diffed now, and an unreachable base degrades to "no range".
  • www.samba.org answered the runner for talloc-2.5.0.tar.gz with bytes that do not match the lock, while serving the locked tarball everywhere else. The source pipeline correctly refuses a mismatch without falling back, so libtalloc/libtevent can fail in the factory for the same reason. The canary set is now lookaside-only.

@hanthor
hanthor marked this pull request as ready for review September 25, 2026 18:15
@hanthor
hanthor merged commit f1ed995 into main Sep 25, 2026
56 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant