Skip to content

feat(ci): cache key for per-package build reuse, and the rule for when to use it - #178

Merged
hanthor merged 20 commits into
mainfrom
claude/package-build-cache
Sep 20, 2026
Merged

hanthor merged 20 commits into
mainfrom
claude/package-build-cache

Conversation

@hanthor

@hanthor hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member

Draft solution for #177. Key and selection rule only — no workflow wiring, deliberately, because the key is the part that has to be right and it wants reviewing before anything depends on it.

The problem, in one line

The factory's only skip mechanism is the atomically-published consumer repository, so a failing publish gate freezes the skip witness and ~282 of 349 packages rebuild every run. The witness has been @sha256:9cb66729 with 67 packages since 2026-08-30. webkitgtk built 20 times on 19 September at 182 minutes a build, with a recipe unchanged since 2026-09-05. Full measurements in #177.

What this adds

tools/package_cache_key.py — the tag under which one package's built RPMs are cached, modelled on projectbluefin/utah's scripts/kernel-cache-tag.sh.

key = sha256({
  schema, package, recipe, buildroot, factory, resolved_root, disttag
})[:32]

rebuild_plan.cacheable(build, changed, stale) — which of the packages plan() already selected may be satisfied from the cache rather than compiled.

The two design decisions that matter

1. The resolved build root is part of the key, so the key is computed after builddep.

Keying on the recipe alone would be wrong and would reintroduce precisely the incoherence the publish gate exists to catch — a stage-5 package builds in a root holding earlier stages' fresh output, so the same spec can legitimately produce different binaries. Resolving first costs about a minute; the compile costs 182.

2. The chain across stages only holds because releases are deterministic — and I checked.

If an unchanged recipe produced a fresh release each build, every stage's resolved root would differ from last run's and nothing would ever hit. It doesn't: 167 of 349 specs use %autorelease, which rpmautospec derives from the committed packages/<name>/changelog, not a build counter. Confirmed empirically — libical-3.0.20-1.hum1.bfin came out with the identical NEVR in run 35413902261 (01:50) and run 35445712318 (13:25), twelve hours and many commits apart.

That is why the changelog is inside the recipe digest, with a test saying so. The whole design rests on it.

Why a hit is safe to trust

A hit is materialised as the stage's ordinary artifact. precedence, the transaction check and publish see an identical repository either way. The cache changes how the RPMs were obtained, never what the run validates. Nothing installs from it; only plan() consults it. The consumer repo stays atomic.

cacheable() — and the one exclusion that is not belt-and-braces

It narrows nothing: plan() still decides what to build.

  • changed excluded — the author of a recipe edit is owed a real build. The key would miss anyway (editing the recipe changes its digest), so this is belt and braces; it keeps the promise legible rather than resting on the key being right.
  • stale excluded — this one is load-bearing. A stale published package requires a soname nothing provides any more, and its recipe may not have moved, so its key can hit — and hitting would hand back the very build that is broken, defeating the repair the run exists to perform. That is the one case where a hit is worse than a miss, so it is named rather than left to chance.

Verification

Key changes on every input — package, recipe, buildroot digest, factory digest, resolved root, disttag, and the schema version. Does not change on transaction order or duplicates in the resolved root (dnf reports installs in an order that varies harmlessly). An empty resolved root is refused rather than keyed, since a key over nothing collides across genuinely different roots.

Recipe digest covers every file with paths hashed alongside contents, so a renamed patch is a change. An empty recipe directory raises.

19 new tests, 338 in the suite. check_workflow_quoting.py, validate.py (349 source RPMs) and the factory contract all clean.

Approaches rejected, recorded so they are not re-proposed

Publishing each tier on completion. Already tried; the publish job carries its epitaph — needs stopped at rebuild4 and publication validated a repository that could not contain later stages' output (nothing provides libepoxy.so.0()(64bit) needed by gtk4 ... from factory). Stages order providers before consumers, so a tier prefix is incoherent by construction: new providers beside carried-forward consumers, and any soname bump breaks every published consumer — the libheif.so.1 / libglycin-2.so.0 class visible in #143's publish log.

Demoting packages that fail to an earlier tier. A stage number is not why a package failed, and demotion gives it a root with fewer factory packages. Every ICU failure fixed today was caused by building too early relative to a provider (bluez 0→1, gvfs 1→3), so demotion moves the wrong way and would silently violate the ordering IcuLinkerOrderingTests asserts. It also oscillates, at 182 minutes a step.

And the decisive one: building is not installability. Run 35413902261 had all 331 builds green and was still unpublishable, because nautilus compiled perfectly and required libicuuc.so.77. Selecting on build success discards the signal the gate exists to provide.

Next, if this shape is agreed

Workflow wiring, as a separate PR: build-stage.yml records the resolved NEVRs after builddep, computes the key, checks ghcr.io/projectbluefin/utah-packages-cache:<key>, and on a hit unpacks the RPMs into work/result and uploads them as the stage artifact exactly as a build would; on a miss it builds and pushes the cache entry. prepare passes cacheable through alongside the stage lists.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M


Generated by Claude Code

claude and others added 18 commits September 18, 2026 02:13
The factory's stated purpose is to track the latest upstream release rather
than whatever Fedora tagged -- docs/targeting-hummingbird.md is explicit that
"a spec is a build recipe, not a release feed". Automation covered 17 of 340
locks, all of them download.gnome.org. 5%.

This adds the next tranche that has a real release feed, taking it to 63:

    download.gnome.org  17  cache.json per-module index   (unchanged)
    github.com          36  tags, or releases where published
    gitlab (3 hosts)    10  repository tags API

The remaining 277 are locked on Fedora's lookaside cache or on a bare
directory listing. A lookaside URL carries its own SHA-512 in the path, so
there is nothing to poll for "latest" -- reaching those needs each project's
real upstream feed identified per package, which is a different job and is
filed separately.

Three shapes are recognised, measured against the real inventory rather than
guessed at: github /archive/<tag>/ (25), github /releases/download/<tag>/
(11), and gitlab /-/archive/ or /-/releases/<tag>/downloads/ (10). Two
details came from reading the locks instead of assuming: evtest is pinned over
http rather than https, and two freedesktop projects use the GitLab release
shape rather than the archive one. Both would have been silent misses -- the
first draft of the pattern found 7 of the 10 GitLab locks and said nothing
about the other three.

Where this differs from the GNOME path, and why:

  Release selection. GNOME has release cycles, so that path only moves inside
  the cycle a lock already names. A forge project has no cycle, so the major
  stands in for it: a newer stable release sharing the current major is an
  "update" and may be applied, one that crosses a major is "review" only and
  is never written. A major can move a soname and break every consumer in the
  graph; no comparison of version strings can make that call.

  URL rewriting. GNOME entries are rebuilt from a known template. A forge URL
  has no template this tool can own, because projects choose their own tag and
  asset names, so forge_planned_entry substitutes the old version for the new
  and the old digest for the new across url, filename, sha256_url and
  fallback_urls. Substituting the bare version fixes a v-prefixed tag in the
  same pass, since "v2.2.1" contains "2.2.1". It refuses outright when the
  current version does not appear in the URL, rather than writing a new
  version beside an old tarball.

  Prereleases. A maintainer's draft or prerelease flag on a GitHub release is
  honoured before the version string is examined, because some prereleases
  carry no alpha or beta suffix at all.

GITHUB_TOKEN is wired into the workflow and is not optional: anonymous GitHub
allows 60 requests an hour against 36 locks, so an unauthenticated run
rate-limits part way down the list and reports the rest as unreachable --
which reads as "nothing to bump". A non-list API response is raised as an
error for the same reason; treating GitHub's rate-limit object as an empty
feed would report every package as current.

The workflow is renamed bump-upstream-sources.yml, since it is no longer only
GNOME. Nothing referenced the old filename.

Verified. 287 tests pass, 20 of them new, all offline through the existing
fake opener. The GitLab path was also exercised against the live API: it
returns 19, 50, 50, 13 and 75 versions for libcamera, wayland-protocols,
shared-mime-info, evtest and accountsservice, and correctly reports each as
already at its newest stable -- so "nothing to bump" there is a real answer
and not an empty list. validate reports 340 source RPMs, the factory contract
is satisfied across 2463 files, and the quoting check clears all 12 workflows.

NOT verified here: the GitHub path against the live API. api.github.com
answers 403 in this environment -- a plain curl does too, so it is the network
and not this code. Its tag, release, draft/prerelease and rate-limit
behaviours are covered by unit tests through the fake opener, and the first
real exercise will be the workflow's own run with a token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Every run was a full rebuild. prepare decides what is already built by
reading a repository listing, and the URL it read -- the same one every
build root enabled as [factory] -- was the GitHub Pages mirror that README
says was removed. The deployment was never taken down, so it still answers,
frozen at 67 packages from before the 343-recipe contract. Both runs 375
and 377 say it plainly:

    published repo has 67 source packages
    will build 333 of 340 packages

Nothing publish wrote could ever reach that listing, so no run could ever
skip more than seven recipes. Eight hours for a one-line spec fix, and
webkitgtk alone five of them on the serial path at stage 5, with no change
to its recipe.

The image publish writes is the only thing that moves, so it is now the
only thing either half reads:

- prepare resolves ghcr.io/<owner>/utah-packages to a digest once (the ref's
  own tag first, then latest; main reads only latest), extracts /repository
  out of it, and runs the plan against that on disk. No image at all is a
  warning and an empty witness, which rebuild_plan.py already treats as
  "skip nothing".
- every build job, and preflight, extract the same digest with the new
  load-factory-repo action and enable it as file:///work/factory. Same
  bytes as the plan read, and nothing fetched over HTTP mid-build. The
  vars.FACTORY_REPO override is gone: a second source of truth is how the
  two halves first came to disagree.

Skipping correctly makes a second gap load-bearing. A fix to mutter would
build mutter alone and leave the published gnome-shell linked against the
libmutter it just replaced. The plan now reads Provides and Requires out of
the same primary.xml and drags every published dependent of anything being
rebuilt, transitively, in inventory order so the waves still hold. It is
the runtime closure; the repository carries no SRPMs, so a devel-only
BuildRequires is the residual gap.

Also on the critical path: debuginfo. The upload step excludes every
debuginfo and debugsource RPM by pattern, so they were built, stripped and
compressed only to be discarded -- 25 minutes of find-debuginfo and 10 of
compression on webkitgtk alone. Both lanes now build with
debug_package %{nil}.

And two things that made the runs read worse than they were: precedence
and publish ran after a failed prepare and died on a missing artifact
(372, 379), so they now require prepare to have succeeded; and a push to a
pull request left the previous run going for hours (369, 370, 374 were all
cancelled by hand), so pull_request runs now cancel their predecessor.
main and the schedule are untouched.

Verified: 293 tests pass, including new ones for the dependency closure
and for the workflow wiring; both workflows and the action parse; the
quoting check passes. The first run after this lands is still a near-full
build, because latest holds the 60-package image from before #70 -- the
last one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…n .debug files

Run 380 failed ten stage-0 packages -- enchant2, hidapi, libsoup3, libvorbis,
libxmlb, openjpeg, opus, protobuf3, taglib, wavpack -- all with the same
error at the same step:

    error: Installed (but unpackaged) file(s) found:
       /usr/lib/debug/usr/lib64/libhidapi-hidraw.so.0.15.0-...x86_64.debug
       /usr/src/debug/hidapi-0.15.0-4.hum1.bfin.x86_64/hidapi/hidapi.h

Those ten are exactly the specs that build a MinGW half and say
%{?mingw_debug_package}; the eleventh such spec, abseil-cpp, has its mingw
bcond off and passed. Nothing else in the run failed.

debug_package %{nil} only removes the native -debuginfo subpackage. It does
not stop find-debuginfo, which __spec_install_post runs whenever
__debug_package is set -- and Fedora's %mingw_debug_package sets it itself
(mingw-filesystem, macros.mingw line 20: %global __debug_package 1). So on
these ten the native extraction still ran, wrote .debug files into the
buildroot, and the subpackage that used to claim them was gone.

The fix empties __debug_install_post as well, in both lanes. That stops the
native extraction at its source. The MinGW halves are unaffected: their
.debug files come from the explicit %mingw_debug_install_post call in
%install and go into the mingw32/mingw64 -debuginfo subpackages, which still
exist and which the upload filter drops like every other debuginfo RPM.
Specs that reference __debug_install_post themselves (libcamera, libxcrypt,
grub2) all guard it with %{?__debug_package:...}, which stays unset for them.

Reproduced and fixed locally with a five-line spec that sets __debug_package
the way the mingw macro does: debug_package %{nil} alone dies on the same
"Installed (but unpackaged)" .debug file; with __debug_install_post %{nil}
added it writes the RPM. The workflow test now pins both defines in both
lanes. 300 tests pass; the quoting check clears all 12 workflows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Run 381 got past every build stage but failed the publish job's
"Validate Hummingbird-only consumer transaction" for two reasons.

First, mesa-dri-drivers links libSPIRV-Tools.so and Hummingbird does
not ship spirv-tools, so dropping the recipe (56b2973) made the runtime
contract unresolvable. The recipe is back, moved to Fedora Rawhide's
2026.3 (SDK 1.4.357.0) because Fedora 44's spirv-headers 1.5.5 carries
GatherModes, which spirv-tools 2026.2 cannot compile against. The
tarball comes from the Fedora lookaside with its sha512 verified; the
two downstream patches no longer apply and are gone with the spec.

Second, several packages in the 67-package witness were built against
sonames the factory no longer provides (libheif -> libavcodec.so.62,
gvfs -> libbluray.so.3, mozc -> libabsl 2601, gnome-shell ->
libedataserver-1.2.so.27). Their current specs already disable those
inputs, but nothing changed in the recipes, so the plan trusted the
published build and skipped them. The dependents-dragging rule cannot
see this because the missing provider is absent from the witness.

rebuild_plan now reads the published repo's Requires and Provides and
marks a source stale when one of its binaries requires something that
neither the published repo nor Hummingbird provides. rebuild_matrix
fetches Hummingbird's primary.xml (baseurl from config/hummingbird.repo)
for the external provides, prints the reason for each stale rebuild,
and skips the rule with a warning if the fetch fails. Stale packages
drag their dependents like any other rebuild.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…ade again

git log on main is a record of fixes that undid earlier fixes: fifteen
commits fighting one ICU split, three digest repins that each died within
days, a recipe dropped and restored, the same libheif bcond added and
retired twice, four soname breaks that were one staging mistake, and two
empty commits pushed to re-run CI.

docs/skills/repeated-mistakes.md names each recurring pattern, the commits
that made and unmade it, and the rule that settled it, with a checklist to
run before pushing a fix. It is indexed in docs/SKILL.md and AGENTS.md so
the router loads it for stage, pin, bcond, workflow and recipe-removal
work.

The triage skill still told readers to repin the build-root digest on exit
125 "manifest unknown"; f1f1e4e showed that cannot work for an image whose
digests live hours, and moved the build root to a once-per-run pull. Both
rows now say so instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Eight recipes imported from Fedora Rawhide dist-git, each pinned by commit
in .hummingbird-upstream.json and source-locked to its upstream release:

  wpa_supplicant 2.11, wireless-regdb 2026.09.03, iw 6.17
    Hummingbird ships NetworkManager-wifi, whose metadata requires
    wireless-regdb and (wpa_supplicant or iwd), and ships none of the
    three, so Wi-Fi cannot be installed on Utah at all (#136, utah#97).
    wireless-regdb in turn requires iw. The wpa_gui bcond is forced off:
    nothing in the contract wants it and the build root has no qt-devel.

  libdaemon 0.14
    Hummingbird's avahi daemon requires libdaemon.so.0 and ships no
    libdaemon, so geoclue retries forever against no daemon (utah#104).

  inih 62 (stage 0), exiv2 0.28.9 (1), libgexiv2 0.16.2 (2), libportal
  0.10.0 (5)
    The factory built nautilus with no way to install it: it needs
    libgexiv2-0.16.so.4, libportal.so.1 and libportal-gtk4.so.1, none in
    any repository (utah#100). exiv2 links inih, gexiv2 links exiv2, and
    libportal links gtk3 and gtk4 (gtk4 is stage 4), hence the stages;
    nautilus itself is at 6. libportal's Qt5 backend is off, since
    Hummingbird has no Qt5 and only the GTK4 backend is consumed.

inih and exiv2 are GitHub archives and gexiv2 is on download.gnome.org;
the bootstrap tool could not fetch them through this environment's proxy,
so their locks are written by hand with the sha512 Fedora's sources file
records, verified against the byte-identical lookaside copy, which is
also the fallback URL, as for the other GitHub-hosted recipes.

The runtime contract now asks for NetworkManager-wifi, wpa_supplicant,
wireless-regdb, avahi and nautilus, so publication proves the three
closures resolve from factory plus Hummingbird before Utah can pin them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…d not work

pipewire was the one failure in 333 jobs of the last run, and it failed
the same way it did before b6b5b4a:

    39/52 pipewire:pw-test-endpoint  FAIL  5.01s  killed by signal 14 SIGALRM
    Ok: 51   Fail: 1

b6b5b4a serialized %check and claimed that fixed it. It did not. The log
shows --num-processes 1 in the meson test command line and the identical
failure at the identical 5.01s, so contention was never the cause and the
serialization was only making the longest job in stage 6 slower. It is
reverted here. Reporting that fix without a green job to point at is the
mistake docs/skills/repeated-mistakes.md records as pattern 13, made
again by me.

The real cause is in the test. src/tests/test-endpoint.c:441 arms

    alarm(5); /* watchdog; terminate after 5 seconds */

around setup that costs more than five seconds on this build root, so the
process kills itself; meson never gets to apply a timeout, which is why
--timeout-multiplier is not the lever either. The budget is out of line
with the test's own siblings rather than with anything the test does:
pw-test-stream and pw-test-filter are declared in the same test_apps list
in src/tests/meson.build, with the same environment, and took 12.02s and
18.03s in this same run on this same machine.

So the watchdog is patched to 60 seconds. Nothing is skipped, disabled or
made non-fatal: every assertion still runs and still has to pass. The run
is still bounded, by meson rather than by the alarm -- these tests declare
no timeout: kwarg, and meson's default is 30 seconds (KwargInfo('timeout',
int, default=30) in mesonbuild/interpreter/type_checking.py), so a genuine
hang is killed at 30s, before the new alarm can fire. The alarm goes from
being the tighter guard to being the redundant one.

The patch was applied to the locked 1.6.8 tarball and checked rather than
assumed: patch -p1 applies clean and leaves alarm(60) at line 441.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Review finding on dae6241, and it is this change's own bug one layer down.

prepare resolves the skip witness as the branch tag first and latest second.
publish seeded from a hardcoded ghcr.io/<owner>/utah-packages:latest. So the
two disagreed about what is already built: on a second push to a pull request,
anything skipped because the *branch* tag carried it was in neither the seed
nor that run's artifacts, and the republished branch image simply lost it --
or regressed to the stale latest copy for packages the first push had rebuilt.

That is exactly the failure this change removed vars.FACTORY_REPO to prevent,
reintroduced at the far end of the same pipeline. publish already needs
prepare, so its resolved digest was there to use all along.

The empty case is handled rather than assumed: when no image is published at
all, prepare emits an empty witness and warns, and the seed now says so and
publishes a new repository instead of trying to pull "".

The concurrency group is scoped to pull requests too. The group covered every
ref while only cancel-in-progress was conditional, so main and schedule runs
queued behind one another and GitHub could drop a superseded pending run --
a behaviour change the block's own comment says is not wanted, since
concurrent waves are safe by design.

tests/test_factory_witness.py asserts the seed reads prepare's output and that
no workflow names the latest tag directly. Reverting the seed to :latest fails
it, so it is a gate and not a restatement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
The publish gate on run 35413902261 refused to publish, correctly. All 331
build jobs passed; the Hummingbird-only consumer transaction then failed:

    Problem 2: package nautilus-gsconnect ... requires
      gnome-shell-extension-gsconnect = 72-3.hum1.bfin, but none of the
      providers can be installed
      - package gnome-shell-extension-gsconnect ... requires /usr/bin/ffmpeg,
        but none of the providers can be installed
      - nothing provides libplacebo.so.360()(64bit) needed by
        ffmpeg-free-9.0.1-1.hum1.bfin.x86_64 from factory

libplacebo exists in neither runtime repository. Hummingbird does not package
it -- I read its primary.xml and there is no libplacebo of any version among
the 20050 packages -- and the factory has no recipe. So the BuildRequires was
satisfied by Fedora 44 in the build root alone, and the built ffmpeg-free
carried a runtime dependency nothing could ever answer. That is precisely the
build-only-becomes-runtime failure this factory exists to prevent, and it took
gsconnect down with it because that needs /usr/bin/ffmpeg.

Two ways out: add libplacebo to the factory, or stop linking it. libplacebo
would need glslang, shaderc and spirv-headers first, none of which the factory
builds, so that is four new recipes for Vulkan-backed libavfilter paths
(placebo scaling and tone mapping) that nothing in Utah's desktop asks for
yet. Turning the bcond off is one line and makes ffmpeg-free installable now;
the comment records what it costs and what re-enabling needs.

Release goes to 2 so the rebuilt RPM is not the same NVR with different
dependencies.

This does not address the other half of the gate failure -- nautilus requiring
libicuuc.so.77 against a runtime that excludes libicu 77. That one needs a
decision rather than a patch, and I have written the evidence up on the PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
The other half of the publish failure on run 35413902261. All 331 builds
passed; the Hummingbird-only transaction then refused:

    package nautilus-51~beta-1.hum1.bfin from factory requires
      libicuuc.so.77, but none of the providers can be installed
      - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
        filtered out by exclude filtering

I read Hummingbird's primary.xml rather than reasoning from the comment
history, and it settles the question. Hummingbird has migrated to ICU 78:

    harfbuzz-icu 14.1.0-1.1 .. 14.2.0-1   libicuuc.so.77
    harfbuzz-icu 14.2.1-3   .. 14.3.1-1   libicuuc.so.78
    boost-locale 1.90.0-6.1, -7           libicuuc.so.77
    boost-locale 1.90.0-9 .. -13          libicuuc.so.78
    libtinysparql 3.11.1-2                libicuuc.so.78

Every current build links 78; only superseded ones link 77. So my earlier note
on the PR -- that Hummingbird is itself split across both ICUs -- was wrong: I
had counted superseded builds alongside current ones. The consumer exclusion of
libicu 77 is therefore correct, and nothing current needs 77.

What went wrong is that the build root never made the same choice. The two
libicu builds share a package name, so dnf installs exactly one and nothing
said which. A root that resolved a superseded copy pulled libicu 77;
libicu-devel followed it to 77.1 to match; and the package built there linked
ICU 77, which its consumers then refuse. Two halves of one decision, disagreeing
-- pattern 2 in docs/skills/repeated-mistakes.md.

So HB_REPO_EXCLUDE gains the same exclusion, in the same spelling, with release
as its own field because dnf splits a spec on dashes before globbing (1f88491).

This is not the blanket exclusion 012cb6a reverted. That one reached Fedora,
whose build-only deps do legitimately link ICU 77 -- raptor2, localsearch and
pipewire-libs-extra transitively -- and it is scoped to the Hummingbird
repository here. HB_EXCLUDE already keeps Fedora ICU out of the build root
entirely, and the rendered mock config shows both halves of that:

    excludepkgs=...,libicu,icu,...                    (fedora, fedora-updates)
    excludepkgs=ruby3.3-...,ruby3.4-...,libicu-77.*-*hum1   (hummingbird)

tools/mock_config.py carries it too, because tests/test_mock_config.py already
asserts the mock root and the container root encode one policy -- that test
caught this change when only the workflow had it, which is the guard working.
New IcuAgreementTests then asserts the build root and the consumer transaction
name the same ICU, so they cannot drift apart again.

309 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Third place the same decision lives, and the one that would have let this
failure repeat even with the other two fixed.

rebuild_matrix builds `external` from every capability Hummingbird provides,
libicu 77 included, and stale_from_primary asks whether a published binary's
Requires are satisfied by factory plus that set. So a published package linked
against libicuuc.so.77 looks satisfiable, is skipped as fresh, and then fails
the consumer transaction -- which excludes libicu 77 -- at publish. The model
that exists to predict that transaction disagreed with it.

Run 35413902261 is what that costs: 331 green builds, precedence green, then
publish refused, and nothing published.

provides_from_primary now takes the builds the transaction refuses and skips
their capabilities, driven by one module constant:

    EXCLUDED_EXTERNAL = (("libicu", "77."),)

A package linked against the refused build is then reported stale, which makes
it rebuild against the current root -- the only cure -- rather than being
carried forward to fail the gate.

The tests use a primary.xml shaped like the real thing, with libicu 77.1 and
78.3 under one name, and assert all three directions: 77 provides nothing, a
published package needing .so.77 is stale, and one needing .so.78 is not, so
the rule does not condemn everything that touches ICU. Emptying
EXCLUDED_EXTERNAL fails three of them.

313 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
… .so.77

I added libicu-77.*-*hum1 to HB_REPO_EXCLUDE two commits ago and it made the
factory strictly worse. bluez failed at stage 0 of run 35443117478:

    package libical-3.0.20-7.fc44 from fedora requires libicuuc.so.77, but
    none of the providers can be installed
    - package libicu-77.1-2.fc44 from fedora is filtered out by exclude filtering
    - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
      filtered out by exclude filtering

The reasoning that produced it was checked against the wrong population. I read
Hummingbird primary.xml, found that every current Hummingbird build links
libicuuc.so.78 and only superseded ones link .so.77, and concluded that
excluding 77 could not strand anything. That conclusion is sound about
Hummingbird and says nothing about the build root, which also carries Fedora
binaries Hummingbird has never rebuilt. libical is one of them and requires
.so.77 outright.

My own commit message contained the refutation: HB_EXCLUDE already keeps Fedora
libicu out of the root, so Hummingbird superseded libicu-77 was the only
remaining provider of .so.77. Removing it left none. I quoted that line as
evidence the exclusion was safe when it is the reason it was not. 012cb6a
reverted a blanket libicu exclusion for this same underlying reason, and
scoping it to the Hummingbird repository did not change the reason -- it only
changed which repository the last provider happened to live in.

Trading a publish-gate failure for a stage-0 failure is a bad trade: the gate
failure came after 331 successful builds and named one package, this one loses
every build behind it. So the build-root half comes out, in the workflow and in
tools/mock_config.py, and both places now carry a note saying why it must not
come back.

The publish-side half stays. EXCLUDED_EXTERNAL in tools/rebuild_plan.py and the
consumer exclusion are about what may be installed on a Hummingbird system,
where nothing current links ICU 77; the build root is a different question with
a different answer. IcuAgreementTests asserted the two halves were symmetric;
it now asserts the asymmetry and says why, so the next person to notice that
Hummingbird has migrated to 78 does not make this change again.

The underlying problem is unsolved and stated as such in both comments: a
factory package can still link ICU 77 because Fedora build deps pull it into
the root, and the consumer transaction then refuses that package. It has to be
fixed without removing the .so.77 provider the root legitimately needs.

315 tests pass. check_workflow_quoting.py clean; the new comment contains no
apostrophe.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
This is the actual cause of the publish failure the ICU exclusions were
flailing at, and it is an ordering bug, not an exclusion one.

Hummingbird ships libicu 77.1 beside 78.3 under one name, so a build root
installs exactly one. dnf takes the newest unless something in the root
requires libicuuc.so.77, and then it downgrades -- libicu-devel follows the
base to match, because icu.spec pins it exactly:

    Requires: lib%{name}%{?_isa} = %{version}-%{release}

so there is no devel-only split to exploit, which was the other fix I
considered. Whatever builds in that root then links ICU 77, and the
Hummingbird-only consumer transaction refuses it.

The chain in run 35413902261 runs entirely through our own staging:

  Fedora samba-core-libs-4.24.6-1.fc44 requires libicuuc.so.77
    -> localsearch (stage 4) resolved samba from Fedora, because the factory
       samba was stage 6 and did not exist yet. Its root installed
       libicu-77.1-2.1.hum1 over five available 78.3 builds, including our own
       libicu-78.3-1.hum1.bfin, and localsearch-3.12~beta-1.hum1.bfin came out
       requiring .so.77.
    -> nautilus (stage 6) requires localsearch, so its root downgraded too, and
       nautilus-51~beta-1.hum1.bfin required .so.77.
    -> publish step 6 refused nautilus. All 331 builds had passed.

The factory samba is not the problem: it built against libicu-78.3-1.hum1.bfin
with zero .so.77 anywhere in its root. Its own entry says it was rebuilt
precisely to stop requiring .so.77. It was simply built four stages too late to
be used by the packages that needed it.

samba moves to stage 2 and libtevent, its only factory build dependency above
stage 0, to stage 1. Both are safe by their own dependencies: libtevent is
built from libtalloc and make, both stage 0; samba from libtevent plus
libtalloc, libtdb, icu and libxcrypt, all stage 0 or now 1. Nothing moves
later, so no existing consumer ordering can break -- pulling a provider earlier
only ever adds packages to the roots that follow it.

tests/test_icu_staging.py makes this a gate. It asserts samba precedes both
known consumers and that samba still follows everything it is built from, and
it fails against the previous stage numbers. The docstring carries the whole
chain, because the next person to meet an ICU 77 requirement will otherwise
reach for an exclusion again -- which is what de4ac4d had to revert, since
Fedora packages the root legitimately needs (libical) require .so.77 and
excluding it strands them outright.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Found by asking which factory packages consume samba, which I should have done
in the same breath as moving it. gvfs BuildRequires libsmbclient-devel -- a
samba subpackage -- and sat at stage 1, so it resolved libsmbclient from Fedora
and its root installed libicu-77.1-2.1.hum1, exactly as localsearch did:

    [ 34/485] Installing libicu-0:77.1-2.1.  36.4 MiB
    [132/485] Installing libicu-devel-0:77.   5.1 MiB

That was already true before the previous commit -- gvfs stage 1 against samba
stage 6 -- so this is not fallout from moving samba, it is the same bug one
package over that moving samba to stage 2 did not by itself reach.

gvfs goes to stage 3. It is unconstrained in both directions: no factory package
BuildRequires gvfs at all, and its only build dependency above stage 0 is samba,
now at 2. Everything else it needs -- libnfs, libsoup3, libsecret, udisks2, gcr,
ibus, libcdio, libplist, libimobiledevice, gnome-online-accounts, libtalloc --
is stage 0.

The dependency scan behind both moves now resolves pkgconfig() BuildRequires as
well as plain names, which the first pass skipped; re-running it over samba
confirms its factory dependencies are still only libtevent (1) and stage-0
packages, so stage 2 remains correct.

tests/test_icu_staging.py covers gvfs in both directions and fails against the
previous stage number.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Three contaminated packages had been found three different slow ways --
localsearch and nautilus from the publish failure, gvfs by asking who consumes
samba, and each of those took a round trip. So I asked the general question
instead: which factory packages link ICU at all, and does anything consume one
of them from its own stage or earlier?

Nine factory packages BuildRequire ICU (enchant2, evolution-data-server,
libical, libphonenumber, localsearch, mozjs140, re2, samba, webkitgtk). Exactly
one unfixed ordering violation fell out:

    bluez (stage 0) BuildRequires libical-devel from libical (stage 0)

Same stage, so bluez cannot see the factory libical and takes Fedora's, which
requires libicuuc.so.77. Confirmed against run 35413902261, where bluez root
installed libical-0:3.0.20-7.fc44 from fedora and then libicu-0:77.1-2.1.hum1.
The factory own libical is clean -- it built against libicu-78.3-8.hum1 -- it
was simply not available in time, exactly as samba was not available in time
for gvfs and localsearch.

bluez moves to stage 1. Nothing constrains it in either direction: no factory
package BuildRequires bluez, and its only factory build dependencies are
libical and make, both stage 0.

The more useful half is the new IcuLinkerOrderingTests, which derives this
rather than restating it. It scans every recipe for an ICU BuildRequires,
then for every factory consumer of one of those packages asserts the consumer
builds strictly later. Verified it fails against bluez previous stage with the
offending pair named. Not every same-stage pair would be a live bug -- it only
bites when Fedora build of the provider actually requires .so.77 -- but every
such pair is a place where a Fedora ICU can enter a build root unnoticed, and
the fix costs one stage number. Finding the fourth case by hand was not going
to be cheaper than encoding the rule.

319 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…to use it

The factory rebuilds about 282 of its 349 packages on every run, because its
only skip mechanism is the atomically-published consumer repository and
publication needs every stage plus precedence plus the Hummingbird-only
transaction. A failing gate therefore freezes the witness, and the witness has
been ghcr.io/projectbluefin/utah-packages:latest @sha256:9cb66729 since
2026-08-30 with 67 of 349 packages in it. webkitgtk, which is not among them,
built 20 times on 19 September at 182 minutes a build, with a recipe unchanged
since 2026-09-05 and an inventory entry byte-identical across every commit since.
Issue #177 has the measurements.

This is the first half of the fix: a key for a cache that answers "have we
already built this exact thing", kept strictly apart from "is the repository
coherent enough to install from". Nothing installs from the cache. A hit is
materialised as the stage's ordinary artifact, so precedence, the transaction
check and publish see an identical repository either way -- the cache changes how
the RPMs were obtained, never what the run validates. That is the property that
makes a hit safe to trust.

Keying on the recipe alone would be wrong, and would reintroduce exactly the
incoherence the publish gate exists to catch. A stage-5 package builds in a root
holding earlier stages fresh output, so the resolved build root is part of the
key, which means the key is computed after dnf has resolved builddep rather than
before: the resolution costs about a minute and the compile costs 182.

The chain only holds across stages if an unchanged recipe yields an unchanged
NEVR, or every root would differ from last run and nothing would ever hit. It
does. 167 of 349 specs use %autorelease, which rpmautospec derives from the
committed packages/<name>/changelog rather than from a build counter, and
libical-3.0.20-1.hum1.bfin came out identical in run 35413902261 and run
35445712318, twelve hours and many commits apart. The changelog is therefore
part of the recipe digest, and there is a test saying so.

Recipes are hashed whole, every file, paths alongside contents so a rename is a
change, for the reason utah's kernel-cache-tag.sh gives: it can only ever
rebuild something that did not need rebuilding, never reuse something stale.

rebuild_plan.cacheable() is the second half: which selected packages may consult
the cache. It narrows nothing -- plan() still decides what to build. It excludes
`changed`, because the author of a recipe edit is owed a real build (the key
would miss anyway, so this is belt and braces), and `stale`, which matters:
a stale published package requires a soname nothing provides any more and its
recipe may not have moved, so its key can hit and hitting would hand back the
very build that is broken. That is the one case where a hit is worse than a
miss, so it is named rather than left to chance.

Verified the key changes on every input -- package, recipe, buildroot digest,
factory digest, resolved root, disttag, schema -- and does not change on
transaction order or duplicates in the root. An empty resolved root is refused
rather than keyed, because a key over nothing collides across genuinely
different roots.

Not in this commit: the workflow wiring that publishes and consults the cache.
That is a separate change and wants this key reviewed first.

19 new tests, 338 in the suite. check_workflow_quoting, validate.py and the
factory contract all clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Review suggestion, and better than the bare hash I had. A tag of
nautilus-e02585b1... can be listed, filtered and reasoned about in the registry
without a side database, which is what you want when the question is "why did
this miss". A bare hash says nothing.

The prefix adds no uniqueness -- the package name is already inside the hash,
and a test asserts that two packages with otherwise identical inputs get
different keys, so the prefix can never be the only thing separating them. It is
metadata, not identity.

Checked before adopting it, rather than after: all 349 recipe names are valid
OCI tag components, and the longest tag is 64 of the 128 characters
(gnome-shell-extension-gsconnect). But RPM names are laxer than tags -- gtk+ and
libstdc++ are legal packages and illegal tag components -- so cache_tag refuses
a name a tag cannot carry instead of emitting a mangled one. A silently mangled
tag would be a permanent cache miss that reads as a correctness bug. No recipe
needs it today; an import could.

The one thing this must not do is share an image name with utah-packages.
That name is the consumer repository, and prepare resolves utah-packages:latest
and utah-packages:<branch> and treats whatever it gets as a whole repository.
A cache entry holds one package's RPMs. If they shared a tag namespace, a
resolve landing on a cache entry would compute `published` from a single
package -- and publish seeds from that same image, so it would drop the other
66. The docstring says so at the point where someone would be tempted.

--key-only prints the bare key for callers that want it, and still builds the
tag first so the grammar is validated whichever output was asked for.

26 tests, 345 in the suite. One pre-existing test asserted the CLI printed a
bare key; it now asserts the tag, since that is the contract the workflow will
use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Adopted — <package>-<key> is better than the bare hash I had, and it is in d6c2e39.

nautilus-e02585b12eb2490ae4242ce4ef899011
gnome-shell-extension-gsconnect-6c3c74fd580a11503552445fd9818704

The argument for it that convinced me is operational rather than aesthetic: a bare hash is unlistable. With the prefix you can ask the registry "what cached builds of nautilus exist" and get an answer without a side database, which is exactly the question you have when a build you expected to hit didn't.

One clarification on "one image name", because it matters. One cache image name, yes — but it must not be utah-packages itself. That name is the consumer repository, and prepare resolves utah-packages:latest and utah-packages:<branch> and then treats whatever it gets as a whole repository. A cache entry holds one package's RPMs. If the two shared a tag namespace and a resolve ever landed on a cache entry, published would be computed from a single package — and publish seeds from that same image, so it would drop the other 66. That is a silent, repo-destroying failure, so the docstring now says it at the point where someone would be tempted. Cache entries go under utah-packages-cache.

Two things I checked rather than assumed before adopting it:

  • All 349 recipe names are valid OCI tag components, and the longest tag is 64 of the 128 characters allowed (gnome-shell-extension-gsconnect). So the scheme fits with room to spare.
  • RPM names are laxer than OCI tags. gtk+ and libstdc++ are legal package names and illegal tag components. None are in the inventory today, but an import could bring one, and a silently mangled tag would be a permanent cache miss that reads as a correctness bug. cache_tag() now refuses such a name rather than emitting something the registry will mangle, with a subtest over every real recipe name plus explicit cases for gtk+, libstdc++, foo/bar and a leading dash.

The prefix adds no uniqueness, and that is deliberate: the package name is already inside the hash. A test asserts two packages with otherwise identical inputs get different keys, so the prefix can never be the only thing separating them — it is metadata, not identity. If it were identity, anything resolving by key alone could be served the wrong RPMs.

--key-only prints the bare key for callers that want it, and still builds the tag first so the grammar is validated either way.

26 tests in this file, 345 in the suite, check_workflow_quoting clean. One pre-existing test asserted the CLI printed a bare key; it now asserts the tag, since that is the contract the workflow wiring will use.


Generated by Claude Code

Right, and this reverses the previous commit. The prefix bought nothing the key
does not already carry -- the package name is inside the hash, and a test
asserts two packages with otherwise identical inputs get different keys -- while
introducing a failure mode the bare key cannot have.

That failure mode was mine to begin with. RPM names are laxer than OCI tags:
gtk+ and libstdc++ are legal packages and illegal tag components, so the prefix
needed a grammar check, a refusal path and a subtest over all 349 names, all to
guard a hazard that only existed because of the prefix. A 32-character hex
digest always matches [a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}, so with the key as the
tag there is nothing to validate and nothing an import can break. Verified
across 20000 generated keys, and gtk+ now hashes like any other name.

The legibility argument was the only one for the prefix, and it was aimed at
the wrong place. Browsing a tag list is not how you answer "why did this miss" --
the build log already prints the package and the key it computed, which is what
you actually read. For the registry side, the push step sets labels, which
survive without constraining the tag grammar:

    org.opencontainers.image.title      = <package>
    org.opencontainers.image.version    = <nevr>
    org.opencontainers.image.revision   = <commit>
    org.projectbluefin.factory.disttag  = <.humN.bfin[.N]>

skopeo inspect answers the question, and a tag listing still answers "how many
entries exist", which is the other thing you would want from it.

The note about not sharing an image name with utah-packages stays, because that
hazard is real and independent of the tag shape: prepare resolves
utah-packages:latest and :<branch> and treats what it gets as a whole
repository, and publish seeds from that same image.

22 tests, 341 in the suite. The removed tag tests are replaced by
TagIsTheKeyTests, which pins the property that makes the prefix unnecessary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Scope: Implements tools/package_cache_key.py and rebuild_plan.cacheable() for per-package cache key calculation based on resolved build root, recipe digest, buildroot image, and factory state.
  • In-flight check: Branch branches off #133 and carries 32 commits from #133. CI runs currently in progress.
  • Outstanding: Awaiting completion of CI runs and independent peer review (+2) once underlying branch structure is settled.

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

I cancelled this PR's rebuild run (35458244044), and the reason is a small argument for the PR itself.

That run was 283 jobs, queued since 17:29. It was triggered because this branch touches tools/, which is_global_change treats as invalidating everything. But nothing in this PR can be validated by a rebuild: it adds tools/package_cache_key.py, a cacheable() helper that no workflow calls yet, and tests. The validation is the unit suite, which runs as its own check.

So it was 283 jobs of pure waste, and not harmlessly so — utah-packages had four full rebuilds queued simultaneously (ci/rebuild-reverse-dependencies, main, this branch, and feat/printing-parity-cups), roughly 1,200 jobs of demand, which is what has been starving the account's runners. A Collect image digests job on projectbluefin/utah's testing build — the one that gates the ISO end-to-end test — has been queued 47 minutes behind it.

I left the other three alone. main's rebuild is the one that could actually advance the published witness, and the other two are contributors' work; cancelling those to speed up mine is not a trade I get to make.

This is the loop described in #177 seen from the other side. It is not only that an unchanged webkitgtk is rebuilt every run at 182 minutes — currently 240 and counting on the run in flight. It is that a documentation-and-tests PR schedules 283 builds, and several such PRs together saturate the runners for everything else in the org, including the release-critical work.

Worth noting as possible follow-up, separate from this PR: is_global_change treating any tools/ edit as a full invalidation is very coarse. A change to tools/package_cache_key.py cannot affect what mock produces for webkitgtk, and neither can a change to tools/rebuild_plan.py that only adds an unreferenced function. Narrowing that — to the tools that actually participate in a build root — would be a second, independent saving. I have not proposed it here because it wants its own reasoning about which tools are build-affecting, and getting that wrong silently serves stale RPMs, which is the same hazard the cache key is careful about.

CI on this PR still needs to pass on the checks that matter; I will re-run those rather than the rebuild.


Generated by Claude Code

castrojo pushed a commit that referenced this pull request Sep 27, 2026
…ance (#186)

* feat(supply-chain): lock factory buildroots and require recipe provenance

A rebuild is reviewable only if it can say what it consumed. Three inputs
decide the output -- the recipe, the source archive, and the build root -- and
each was described by something that could move underneath a run without the
run noticing.

**The build root.** config/buildroot-lock.json declares it by digest, and the
preflight job now records what the root actually contained: every package
NEVRA, the digest prepare resolved, and the digest the lock expected. rpm -qa
runs inside the root and the snapshot is assembled on the runner, so the root
needs no interpreter -- installing one so it could inventory itself would
change the set being inventoried.

The rebuild does not pass --strict, deliberately. quay.io/fedora/fedora:44 is
republished several times a day and each previous digest is collected, so
failing on a moved tag recreates the outage repeated-mistakes.md section 7
records: thirty-seven jobs dead mid-run on a pin that was correct when the run
started. The mismatch is recorded in the snapshot and the manifest instead, so
a moved tag is visible rather than asserted away. --strict is there for an
operator asking whether a root is exactly what was locked.

The digest is now written twice -- in the lock and in BUILDROOT_IMAGE -- and
two copies of one digest drift. Renovate's build-root-image manager covers the
lock file as well, so both move in one pull request, and tools/validate.py
fails the gate when any workflow pins a locked buildroot to a digest the lock
does not name. That check runs before the recipe tally, because a package
merely missing a Packit entry used to return 1 first and hide the drift.

**The recipe.** Provenance was already written at import and validated
loosely. It is now mandatory and checked by form: a rawhide import must pin
full 40-character commit and tree SHAs (an abbreviated SHA is ambiguous and is
not what git resolved), and a direct-upstream import must name a non-empty
remote and carry no dist-git snapshot. validate.py reports the split it
validated -- 346 rawhide, 3 upstream -- so the direct-upstream form, the one
easiest to leave unchecked, is visible in the gate output.

**The source.** source_pipeline.py records how each accepted archive was
verified: signature, sha256-manifest, or the locked sha512 alone. A checksum
is not a signature, so factory_manifest.py counts each kind and names the
packages in the checksum-only classes -- an exception list to shorten, not a
property to grep for.

**The output.** factory_manifest.py writes manifest.json beside the
repository: built RPMs, source reports and their verification summary,
buildroot snapshots, the lock, and the published OCI reference and digest. It
is written once before the container build and again once the digest exists,
and uploaded as the factory-manifest artifact. Reports are classified by shape
-- schema+name+packages for a snapshot, package for a source report -- so
unrelated JSON beside the repository is not folded in.

docs/skills/supply-chain-provenance.md carries the conventions, including why
--strict is not the rebuild default and why the duplicated digest is not
resolved by deleting either copy.

Carries the work from #158, rebased onto the #178 baseline the maintainer
asked it to be re-cut against, with that review's findings closed.

Closes #25

Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(tests): keep the buildroot snapshot tests out of the prepare job's env

The prepare job exports BUILDROOT_IMAGE for the whole job and runs the
unit tests inside it. cmd_snapshot reads that variable as the running
image when no --image or --digest is passed, so test_cmd_snapshot saw
the factory's real digest instead of the lock's fake one and failed its
own strict check. Strip the factory's variables in setUp so the tests
only see what they pass.

Also double the % signs in the --packages-from help text: argparse
%-formats help strings, and Python 3.14 checks that eagerly at
add_argument time, which made every buildroot_lock test error locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK

* fix(supply-chain): keep workflow expressions out of run scripts and stop the snapshot defaulting to the lock

Two review findings on the buildroot lock.

The two factory manifest steps in rebuild-rpms.yml inlined
${{ needs.prepare.outputs.build_list }} inside single quotes in run:,
and the second one inlined steps.oci.outputs.ref and .digest as well.
An expression in run: is substituted before bash parses the script, so
a quote in a package directory name would end the argument early, in
the publish job, which holds packages: write and id-token: write. The
values now go through env: and the script reads "$BUILD_LIST",
"$OCI_REF" and "$OCI_DIGEST", the way preflight already passes
build_list. Every expression this change adds to the workflow is now
in env: or with:.

cmd_snapshot's fallback chain ended in `or locked_image`, so an empty
BUILDROOT_DIGEST made the snapshot record the lock's own digest as the
image it ran in, which is exactly the provenance the snapshot exists
to check rather than assert. With nothing resolved it now records
image: null and warns; under --strict that is an error, because an
operator asking whether the root is exactly what was locked cannot be
told yes. locked_image is unchanged. test_cmd_snapshot passes a digest
now that --strict refuses an unknown image, and two tests cover the
null case and the strict refusal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YQCk7xJDvhtPoc4mWPzVCK
Signed-off-by: Danathar <doug.baggett@gmail.com>

* fix(lock): follow the fedora:44 buildroot digest Renovate moved on main

The merge from main brought the Renovate bump of BUILDROOT_IMAGE in
rebuild-rpms.yml to sha256:a659125..., and this branch's
config/buildroot-lock.json still named the previous digest, so the
PR's own drift gate (tools/validate.py) failed. Point the lock at the
digest the workflow pins. Once this lands, renovate.json's
build-root-image manager moves both copies in one PR.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(supply-chain): drop snapshot env fallbacks and make the drift gate total

Review findings on #186.

cmd_snapshot resolved actual_image through undocumented fallbacks to
ACTUAL_BUILDROOT_IMAGE and BUILDROOT_IMAGE, contradicting the module
docstring and docs/skills/supply-chain-provenance.md, which both promise
image:null when no --image/--digest/BUILDROOT_DIGEST is given. Since
validate.py forces the workflow's BUILDROOT_IMAGE to equal the lock digest,
running snapshot inside the prepare job silently re-asserted the lock's own
pin as the resolved image -- the failure mode this branch exists to remove.
Both fallbacks are gone; --image, --digest and BUILDROOT_DIGEST are the only
inputs. tests/test_supply_chain_lock.py setUp no longer strips the two
variables (it acknowledged the hazard rather than removing it), and a new
test exports both to prove they are ignored.

check_buildroot_drift keyed workflow pins by full repo:tag, so a pin moved
to a different tag (workflow fedora:45@X vs lock fedora:44@Y) shared no key
with the lock and was skipped, and a deleted pin left nothing to compare:
validate passed in both cases. The comparison is now per repository, and a
locked buildroot that no workflow pins is drift rather than an absence of
evidence. Nothing is required when .github/workflows does not exist.

just check and just test both pass: 441 tests, 2127 subtests; validate
reports 342 source RPMs (339 rawhide, 3 upstream).

Refs #186

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix(supply-chain): make the snapshot honest about what it could not read

Three reviewer findings, all about the tool misreporting its own confidence.

compare() crashed instead of reporting when a lock entry had no `nevra`:
`item.get("nevra")` put None into the expected set, it landed in `missing`,
and `", ".join(missing[:20])` died with a TypeError. load_lock only validates
that `packages` is a list -- the per-entry requirement lives in
tools/validate.py -- so `buildroot_lock.py snapshot --strict` run standalone
against a hand-edited lock hit this. The gate failed either way, but a
traceback does not tell an operator which entry to fix. Malformed entries are
now counted and named as a lock defect, and deliberately not reported as
missing packages: they are not a divergence in the build root.

parse_packages() dropped malformed `rpm -qa` lines without a word. For an
attestation input that is the wrong default -- a truncated or corrupted
capture makes the snapshot under-report what the root contained, and nothing
downstream could tell. It now warns on stderr naming the dropped lines, and
--strict refuses the snapshot outright: a capture that under-reports cannot
answer "is this root exactly what was locked?", because the lines it could not
read might be the divergence. The scheduled rebuild is unaffected; it does not
pass --strict, and records what it saw.

Also drops the unused `prefix` parameter from factory_manifest.reports().

443 tests pass; `tools/validate.py` clean.

Assisted-by: Claude Opus 5 via Claude Code
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

* fix(supply-chain): stop the manifest attesting a previous run's buildroot

`buildroots` is the one part of the manifest that attests rather than
describes: it says which root this run built in. Publish seeds
`repository/` from the previously published factory image, that image was
built `COPY repository /repository` with the last run's `reports/` inside
it, and preflight -- which writes the replacement -- is skipped outright
when `build_list` is `[]` and downloads `continue-on-error`. A cleanup-only
run, or any run whose preflight failed, therefore reached the manifest step
with the previous run's `buildroot-fedora-44.json` on disk and folded it in
with nothing marking it as someone else's: the manifest attesting a root
this run never measured, which is the failure class the lock exists to
close.

Two independent stops, because either one alone is a step-ordering
assumption:

- the seed step deletes `repository/reports/buildroot-*.json` right after
  the `podman cp`, so only this run's `preflight-buildroot` artifact can put
  one back. Source reports stay: they describe the seeded RPMs, which really
  are in the repository being published.
- `factory_manifest.py --buildroot-digest` keeps a snapshot only when the
  image it resolved carries the digest `prepare` resolved. Both manifest
  writes pass it.

Discarded, not silently dropped. A rejected snapshot is named under
`buildroots_discarded` with its report path, image and a reason, and warned
about on stderr -- "no root was measured" and "a root was measured and not
attested" are different answers to the question asked after a bad package
ships. With no digest to compare against (standalone use, or a run whose own
resolution failed) nothing is judged; the prune covers that case instead.

Tests cover both halves, including the workflow wiring: the prune exists and
precedes the preflight download, and both manifest writes pass the digest.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(supply-chain): make the buildroot lock follow the mirror pin

main moved the build root to the factory mirror (#252, #255): prepare now
pulls exactly what config/buildroot-image pins, no workflow carries
BUILDROOT_IMAGE, and Renovate no longer tracks quay.io/fedora/fedora:44.
The drift gate here compared the lock against workflow pins, so after the
merge it failed with "no workflow pins quay.io/fedora/fedora", and, as
hanthor noted on #252, it would have failed every weekly refresh PR,
because refresh-buildroot.yml only runs buildroot_pin.py set.

- config/buildroot-lock.json names the mirror pin (moved with the tool).
- tools/validate.py compares every locked root with config/buildroot-image,
  tag and digest; a missing pin is drift.
- tools/buildroot_pin.py set moves the lock beside the pin. It parses the
  lock before writing either file, and leaves a locked packages list alone
  so snapshot --strict reports the divergence.
- renovate.json goes back to main: the lock is no longer a Renovate target.
- The validate output quoted in docs/architecture.md is the real line.

* fix(supply-chain): publish the manifest from publish-repository.yml, and snapshot only the pinned image

Publishing moved out of rebuild-rpms.yml into publish-repository.yml on
main, one call per early wave plus the final one, so the manifest steps
move with it:

- the seed step still drops the previous run's buildroot-*.json, and the
  preflight snapshot is downloaded after it (now artifact-prefixed, as
  every artifact of a canary pass must be);
- built RPMs now land in built/ and only RPMs are copied into the
  repository, so the source reports of the packages this publication
  replaced are carried into repository/reports; a pruned source loses its
  report with its RPMs;
- both manifest writes judge snapshots against a new buildroot_digest
  input, which every publish job passes; every publish job also needs
  preflight, so an early one cannot run before the snapshot exists;
- the artifact is factory-manifest, or factory-manifest-wave<N> for an
  early publication, since names are unique per run.

The last review on 522a9fc found the snapshot described no real root: it
was taken after preflight installed @buildsys-build from live repos, which
is neither the pulled image nor any wave's root. rpm -qa now runs first,
before preflight changes anything, so the NEVRA list is exactly what the
recorded digest names. The skill doc says what it is not: each hermetic
package's root is its lock-s<N>-<package> artifact, and the container
lane's is the post-builddep rpm -qa the cache key hashes.

---------

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Signed-off-by: Danathar <doug.baggett@gmail.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants