Skip to content

Build the whole Utah contract: 343 recipes, Firefox as a Flatpak, four stage failures fixed - #70

Merged
hanthor merged 64 commits into
mainfrom
claude/great-allen-pt5nwv
Sep 18, 2026
Merged

hanthor merged 64 commits into
mainfrom
claude/great-allen-pt5nwv

Conversation

@hanthor

@hanthor hanthor commented Sep 10, 2026

Copy link
Copy Markdown
Member

Takes the contract from #51 and fixes everything that failed in its last run
(34415980185), so a single run can build what a Utah image actually needs.

That run's numbers: 285 build jobs, 279 succeeded, 5 failed, gcc cancelled
with the run
. publish was skipped because the run ended cancelled, so
nothing from it reached the repository. Of the five failures, one was a real
recipe defect and four were the build root not containing what the recipe
needed.

Firefox comes out

Utah ships Firefox as the org.mozilla.firefox Flatpak. The factory built the
browser only because Bluefin's copied [fedora] section names it, and
projectbluefin/utah already lists firefox as unavailable while carrying the
Flatpak in contracts/bluefin-desktop.toml.

It was not a cheap recipe to keep: a 2.5-hour PGO/LTO job, the longest of the
285, carrying its own 12G swap workaround in the build workflow — and it
failed anyway, in %install:

cp: cannot create regular file
'objdir/dist/bin/browser/chrome/browser/content/browser/default-bookmarks.html'

So the recipe, its source lock, its .packit.yaml entry and the swap step that
existed for it alone are gone, and firefox joins [unavailable] in the
runtime contract — which is what keeps the Bluefin-requested name from failing
the transaction. mozjs140 is untouched: it is the embeddable JavaScript
runtime and unpacks a firefox tarball of its own.

The build root could not see the published repository

This is the interesting one. prepare skips any package the published
repository already carries at the same version — but the [factory] repo block
in build-stage.yml is guarded by vars.FACTORY_REPO, which is unset on
pull_request runs. The block never ran, so a skipped package was not "already
available", it was absent. 59 of the 344 recipes were in that state.

That is why pipewire-libs-extra failed on

No match for argument: pkgconfig(libfreeaptx)

while libfreeaptx-devel sat published and skipped. FACTORY_REPO now
defaults to the same GitHub Pages URL prepare reads to decide what to skip,
so the two halves of that decision cannot disagree. The repository variable
still wins when set.

Two recipes were staged before what they need

package stage why
xorg-x11-server-Xwayland 3 → 5 needs wayland-client >= 1.26.0, build root had Fedora's 1.25.0; this factory builds wayland at 4
evolution-ews 6 → 10 needs libebook/libecal/libedata-* >= 3.61.3; evolution-data-server builds at 9

Xwayland's three stage-0 build consumers (libayatana-appindicator,
libayatana-indicator, zenity) resolve it from Fedora and never saw ours, so
moving it later changes nothing for them.

gstreamer1-plugins-bad-free declines chromaprint

On exactly the reasoning ffmpeg.spec already records: Fedora's
libchromaprint links libavutil.so.60, this factory's ffmpeg 9.0.1 provides
so.61, and the factory excludes the Fedora libav by name — so
libchromaprint-devel is uninstallable here, permanently, not as a bootstrap
phase. The spec disabled chromaprint only under %if %{without extras}; it is
now its own bcond, off, with the BuildRequires and the plugin in %files
following it. Nothing in the runtime contract links AcoustID fingerprinting.

gcc was cancelled with the run rather than failing, and needs only a re-run.

Relationship to the open PRs

Verification

tools/validate.py reports 343 source RPMs, the runtime contract resolves to
71 packages against the pinned base image, the workflow quoting check passes
across all 11 workflows, both changed workflows parse, and all 73 tests pass.

Not verified locally: the build itself. That is what this PR's run is for.
The chromaprint bcond and the FACTORY_REPO default can only be confirmed by
Actions building against a real root.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M


Generated by Claude Code

hanthor and others added 30 commits September 6, 2026 22:57
The factory builds 193 packages; a Utah image needs 345. The missing
152 are what a real dnf transaction against Hummingbird names, found by
composing the image and reading the resolver until it stopped
complaining: ffmpeg, bluez, SDL3, flatpak, appstream, gnome-keyring,
Xwayland and its five X11 helpers, hunspell and its dictionaries, gcr3,
openjph, setools, hidapi, desktop-file-utils, soxr, and the rest of the
runtime closure behind them.

Every recipe arrives in this repository's shape rather than dist-git's:
one spec, sources, changelog, patches, and the upstream provenance file.
The tmt plans, monitoring.toml, per-package .packit.yaml and .gitignore
that the scaffolding purge removed are stripped on the way in, so
nothing lands that the purge would only have to remove again.

Three recipes carried build-time defines in the old factory, which the
Packit inventory has no field for. They move into the spec, where they
work whichever builder runs:

  mdadm       ABRT off -- its only unresolved dependency was
              libreport-filesystem, for a crash-reporter hook, in an
              image that ships no crash reporter
  v4l-utils   Qt off -- qv4l2 and qvidcap need Qt6, and the build root
              cannot resolve it while Hummingbird offers both libicu-77
              and a qt6-qtbase that requires libicuuc.so.78
  protobuf    Java off -- the bindings need a JDK the build root does
              not carry and nothing here consumes them

color-filesystem has no upstream archive at all: it creates four
directories and an rpm macro file, and dist-git records no sources for
it. Rather than invent a URL to satisfy the shape of a lock, an entry
may now declare `no_upstream_source`, which fetches and verifies
nothing. colord requires it, and colord is what gtk3 and the desktop
hang off.

webkit2gtk4.1 is deliberately absent. It is a second entry that builds
webkitgtk's recipe with a different define, and the inventory has no
alias field; webkitgtk itself is here.

Verified: `tools/validate.py` reports 345, the inventory reports every
recipe source-locked and Packit-configured, and all 42 tests pass. Not
verified: none of these has been built under Mock. The counts the
inventory tests assert are updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
PACKAGE_KEY did not accept "_", so a package whose name contains one
parses as no package at all. package_names() drives both the rebuild
matrix and the inventory's packit_configured check, which means such a
package is silently never built and nothing reports it missing.

No name in the current 193 has an underscore, so this has been latent.
It surfaces the moment lm_sensors or volume_key is added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
GitHub caps a matrix at 256 jobs, and it does not fail when one would exceed
that -- it expands to nothing. A stage holding more than 256 packages produces
zero build jobs, silently, and every later stage then builds against an empty
buildroot.

This branch's cold stage 0 is 315 packages, so run 34171159421 spawned no
rebuild0 jobs at all: 14 jobs at stage 1, 3 at stage 2, 1 each at 3 and 4, and
none at 0. The stage-1 failures all read as missing dependencies --

  vulkan-loader:       No match for argument: vulkan-headers = 1.4.350.0
  pipewire-libs-extra: No match for argument: pkgconfig(libfreeaptx)
  gdk-pixbuf2:         glycin-2 found 2.1.0 but need >= 2.2.alpha.7

-- and all three name packages this factory builds at stage 0. They were not
missing; they were never built.

Hand each stage to build-stage.yml in chunks of 250 rather than in one piece,
and fan the caller out over them. Stage 0 becomes two chunks (250 + 65); every
other stage is unchanged at one. The flat stageN outputs stay, so the
already-published skip logic and the empty-stage guards are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
samba BuildRequires libtevent-devel >= 0.17.2 and python3-tevent >= 0.17.2.
Fedora 44 ships 0.17.1, so those have to come from this factory's own
libtevent 0.17.2 -- but libtevent sat at the same stage 1 as samba, and a
stage cannot see its own output. samba failed with:

  No match for argument: libtevent-devel >= 0.17.2

Move samba to stage 2. libtalloc 2.5.0, its other version-floored dependency,
is already at stage 0, so nothing else needs to move.

The rest of that job's "missing dependency" errors were not ordering at all --
vulkan-headers, glycin, libfreeaptx and liblc3plus are all stage 0 packages
that never built, because stage 0 exceeded GitHub's matrix cap and expanded to
zero jobs. That is fixed separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Thirteen stage-0 jobs failed to resolve their buildroot, and none of them were
missing a package. Each hit the same shape: this factory rebuilds a library
with a new soname, Fedora's dependent still wants the old one, and the rebuilt
dependent -- which we also carry -- sits in the same stage, where a sibling job
cannot see it.

  libheif 1.23.1 (so.2)  vs  Fedora glycin-loaders wanting libheif.so.1
    -> pulseaudio, rest, sdl2-compat, tecla, wayland, wireplumber,
       xdg-user-dirs-gtk  (reached via graphviz -> gdk-pixbuf2 -> glycin)
  abseil-cpp 20260526    vs  Fedora webrtc-audio-processing wanting
                             libabsl_strings.so.2601
    -> libepoxy, libglvnd, libayatana-ido, xdg-desktop-portal-gtk
  pipewire 1.6.8         vs  Fedora pipewire-jack-audio-connection-kit-devel
    -> gstreamer1-plugins-good

Our glycin 2.2, webrtc-audio-processing 2.1 and pipewire 1.6.8 are all stage 0
and all build fine; they were simply invisible to their siblings. Move the
dependents to stage 1 so they see them.

nautilus-python goes to stage 2 rather than 1: it needs localsearch, which is
itself stage 1 -- Fedora's localsearch 3.11 wants libavformat.so.62, which our
ffmpeg 9.0.1 displaces.

Moving wayland to stage 1 is safe: the only versioned consumer is
wayland-protocols (>= 1.25.0), and Fedora 44 ships wayland-devel 1.25.0.

Also fix a lowercase month in dracut's changelog, which rpmbuild rejects
outright: "bad date in %changelog: Mon aug 10 2026". 10 Aug 2026 is a Monday,
so only the capitalisation was wrong.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Six stage-0 builds -- gcc, git, iputils, libvdpau, appstream and libtheora --
died with nothing useful in a tail:

  Running transaction
  Transaction failed: Rpm transaction failed.

dnf resolved the buildroot happily; rpm then refused to install it, and the
reason prints after the failure line rather than before it:

  file /usr/bin/erb conflicts between attempted installs of
    ruby4.0-default-gems-4.0.6-37.3.hum1 and ruby3.4-default-gems-3.4.10-31.7.hum1

Both sides are .hum1. The existing exclusions are per-repo -- fedora.excludepkgs
and updates.excludepkgs -- so they can express "prefer Hummingbird's copy over
Fedora's" but not "prefer one Hummingbird package over another". Hummingbird now
carries ruby3.4-default-gems and ruby4.0-default-gems, whose file lists collide.

Exclude the superseded ruby globally with --exclude, leaving 4.0 as the only
candidate in any repo. The per-repo lists are left alone: they still do the job
they were written for.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Both failed with a file that is sitting in the recipe directory:

  error: Bad file: /work/staged/mdadm/9CE05640D703A136.asc: No such file

.gitignore carries `packages/*/*.asc`, meant for downloaded signatures. A
keyring a spec names as a SourceN is not a downloaded artefact -- it is part
of the recipe, read straight out of _sourcedir:

  %gpgverify --keyring='%{SOURCE11}' --signature='%{SOURCE10}' --data=-

So the file existed locally, was never committed, and CI checked out without
it. git, wayland and zsh already carry theirs because someone force-added them
one at a time; mdadm and setxkbmap were simply missed, and nothing reported it.

Commit the three missing keyrings and narrow the ignore so the next one is not
lost the same way: keep ignoring downloaded .asc signatures, but never a file
named as a key ID or carrying "key"/"keyring" in its name.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Xwayland's meson.build asks for wayland-client >= 1.26; Fedora 44 ships
1.25.0, so only this factory's wayland 1.26.0 satisfies it:

  meson.build:189:4: ERROR: Dependency lookup for wayland-client with method
  'pkg-config' failed: Invalid version, need 'wayland-client' ['>= 1.26...']

Both sat at stage 0, where a job cannot see its sibling's output, so Xwayland
could never have found it. wayland has since moved to stage 1, so Xwayland
goes to stage 2.

The floor is declared in upstream's meson.build rather than the spec, which is
why scanning BuildRequires does not find it -- the spec's wayland-devel carries
no version at all.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The openssh regress suite failed every parallel test job:

  Privilege separation user sshd does not exist
  FATAL: sshd_proxy broken
  make: *** [parallel_test.Makefile:3: t-exec-03] Error 2

The recipe carries openssh-server-systemd-sysusers.conf and creates the user
from %pre at install time. A buildroot never installs the package it is
building, so the user is absent exactly where the test suite expects it.

Apply any sysusers config the recipe stages, before rpmbuild runs. This is
generic -- any package shipping one gets the users it declares -- and it
supplies what a real build root has rather than disabling the test that
noticed the gap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
%prep died with all three hunks rejected:

  3 out of 3 hunks FAILED -- saving rejects to file src/nasm.inc.rej
  error: Bad exit status from /var/tmp/rpm-tmp.bC0pTE (%prep)

The patch was not stale. xvidcore ships src/nasm.inc with CRLF line endings --
all 213 lines of it -- while the patch carried LF-only context, so no context
line ever matched. Nothing about the change itself was wrong.

Regenerated against the pristine 1.3.7 tarball, with CRLF preserved on both
sides of the diff. Same three hunks, same 23 inserted lines, same GNUHEADER
macro and the two call sites. Verified by applying it to an untouched
src/nasm.inc and comparing the result against the intended file byte for byte.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Git normalised the CRLF context out of the regenerated xvidcore patch the
moment it was committed -- the blob went in with zero carriage returns, so
the fix would have failed in CI exactly as before, and the working tree
would still have looked correct.

Mark patches, diffs, keys and signatures as -text so git leaves them alone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
flac failed %check with the suite naming the cause itself:

  ERROR: iterator claims file is writable when tester thinks it should not
  be; are you running as root?
  80% tests passed, 2 tests failed out of 10

The test is right. It chmods a file read-only and asserts the write fails,
and root writes it regardless of the mode bits. tools/build-rpm.sh already
fixed this for a local build, in as many words:

  # Match mock's build user. FLAC's read-only-file tests correctly fail when
  # rpmbuild runs as root, which can write files regardless of their mode bits.
  useradd --create-home --home-dir /builddir mockbuild
  build_rpm() { runuser -u mockbuild -- rpmbuild "$@"; }

The CI lane never got the same treatment and kept building as root out of
/root/rpmbuild. So the two lanes disagreed about who runs the build, and a
package could pass locally and fail here for a reason belonging to neither
the recipe nor the code.

Create the same mockbuild user, move _topdir to /builddir/rpmbuild, and run
both rpmbuild invocations through runuser. Fedora builds this way; so should
both of ours.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
evolution-ews sat at stage 0 while evolution-data-server, which supplies every
pkgconfig module it asks for, is at stage 1:

  No match for argument: pkgconfig(camel-1.2) >= 3.61.3
  No match for argument: pkgconfig(evolution-data-server-1.2) >= 3.61.3
  No match for argument: pkgconfig(evolution-mail-3.0) >= 3.61.3

Move it to stage 2. evolution-data-server itself builds fine.

tailscale vendors go-json-experiment/json, whose alias.go forwards to the
standard library encoding/json/v2 when the jsonv2 experiment is on. Hummingbird
ships Go 1.27, whose v2 API has moved past that vendored shim:

  vendor/github.com/go-json-experiment/json/alias.go:618:21:
    undefined: json.SkipFunc
  vendor/github.com/go-json-experiment/json/alias.go:957:14:
    undefined: json.DiscardUnknownMembers

GOEXPERIMENT=nojsonv2 takes the build tag out, so the module uses the
implementation tailscale ships and tests against.

libnice lost one test of forty-two, on the clock rather than on a failure:

  42/42 libnice:test-new-trickle  TIMEOUT  30.03s  killed by signal 15 SIGTERM
  Fail: 0   Timeout: 1

test-new-trickle negotiates ICE candidates against timers and wants more than
the 30s meson allows by default on a loaded builder. Raise the multiplier
rather than drop the test that ran out of time.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The Packit SRPM pilot has failed on every run since the monorepo passed 256
packages, and it failed invisibly: discover goes green, reports "discovered
345 packages", and the srpm matrix beneath it produces no jobs at all. GitHub
caps a matrix at 256 and expands a larger one to nothing rather than
rejecting it.

The discover step already guards against a matrix that runs nothing -- it
asserts the list is non-empty -- but a non-empty list of 345 is exactly the
case that guard cannot see.

Split the per-package steps into a reusable packit-srpm-chunk.yml and fan the
pilot out over chunks of 250, the same shape rebuild-rpms.yml uses. Per-package
parallelism is unchanged; only the number of matrices grows. Chunking lives in
packit_workflow.py rather than inline in YAML so it can be tested, and a test
now asserts every chunk fits under the cap and that chunking loses no package.

This is what the pilot was built to answer and has never reached: whether
packit srpm works against these recipes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Packit stays, scoped to what it does well: turning a recipe into an SRPM and
proving the spec is well formed, in three or four seconds a package. That is a
cheap gate -- dracut sat behind a lowercase month in a %changelog date, which
an SRPM step catches at once instead of after a build root has been installed
and the package compiled.

Packit is not replacing the binary lane, and the disttags say why. packit srpm
yields flac-1.5.0-9.fc43; this factory ships
gnome-shell-extension-gsconnect-72-3.hum1.bfin. Rebuilding against Hummingbird
so sonames match the image is the entire reason the factory exists, and a
Fedora-chroot SRPM does not answer it.

Copr is the shape in which Packit could take the binary lane, and we are not
pursuing it now: it would need a custom chroot carrying the Hummingbird
repository plus the exclusion rules the lane already encodes. That is a
migration of the build root, not a change of build driver. Written down so the
option is not rediscovered from scratch.

Also records two things that were true but undocumented -- Packit does no
source acquisition here either, and .packit.yaml carries no jobs section, so
the service does no work on a pull request -- notes the matrix cap that kept
the pilot from ever running, and refreshes the counts from 193 to 345.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Both were solved on fix/repeatable-local-builds (#45) and lost in the port to
the Packit shape. Bringing the recipes across rather than rediscovering them.

libheif is the one blocking ten packages in the current run. Its ffmpegdec
plugin links Fedora libav, and the moment the factory promises its own ffmpeg
9.x the Fedora copy is excluded by name -- so the accumulator libheif becomes
uninstallable and takes every buildroot that reaches gdk-pixbuf2, glycin,
graphviz or doxygen with it. That is exactly what opus, libthai, libvpx,
mpg123, openjpeg, spandsp, libnice, libnma and libtalloc are failing on:

  package graphviz requires libgdk_pixbuf-2.0.so.0, but none of the providers
  can be installed ... glycin-loaders requires libheif.so.1 ...

The draft answers it with a narrow %bcond ffmpeg 0: the aom and dav1d decoder
plugins stay, which is what actually decodes AVIF and HEIC here, and only the
ffmpeg HEVC path goes. Narrower than %bcond bootstrap, which would also drop
rav1e, SvtEnc and sdl2.

tailscale builds from the official static release payload rather than a
vendored Go tree, so it no longer depends on the Hummingbird Go version merely
to reproduce an upstream artifact. This supersedes the GOEXPERIMENT=nojsonv2
workaround added earlier today, which was treating a symptom: Go 1.27 has
moved past the vendored go-json-experiment shim. The spec goes from 1192 lines
to 71, the vendor tarball script and the 1.92.5 systray patch go with it, and
the source lock points at tailscale_1.98.8_amd64.tgz.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
…ildroot

git was the last failing package, and not for a reason belonging to git. Six
tests of t9168-git-svn-partially-globbed-names failed, every one of them on

  svn: E155007: ... is not a working copy

with the cause further up the log:

  svn: E200029: atomic initialization could not be performed
  svn: E200030: SQLite compiled for 3.53.4, but running with 3.51.2

Hummingbird builds subversion against sqlite 3.53.4. Fedora 44 ships 3.51.2
and wins the buildroot, so subversion cannot open a repository at all, every
checkout and copy fails, and it surfaces as a git test failure.

sqlite therefore joins libicu, gpgme and qt6-qtbase in the list of names
Fedora must not answer for -- the same mechanism, for the same reason, written
down where the others already are.

The subversion versions are otherwise identical (1.14.5 in both), which is
what rules out a version incompatibility in the test itself and makes skipping
it the wrong answer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
spandsp was the last failure after the draft port, and again not for a reason
belonging to spandsp. Its buildroot reaches doxygen, and:

  package libheif-1.23.1-1.hum1.bfin from stages requires
    libopenjph.so.0.25()(64bit), but none of the providers can be installed

Same shape as the ffmpeg plugin, one layer down. The factory promises openjph
0.31, which provides libopenjph.so.0.31. libheif builds in the same stage, so
it cannot see that and links Fedora openjph 0.25 instead -- and at install time
the Fedora copy is excluded by name, because the factory answers for openjph.
libheif is then uninstallable and takes gdk-pixbuf2, glycin, graphviz and
doxygen down with it.

Ordering cannot settle this without restratifying five deep -- openjph,
libheif, glycin, gdk-pixbuf2, then everything that reaches doxygen -- which is
the entire stage budget for one plugin.

Decline the feature, as the ffmpeg bcond already does. This drops the HTJ2K
encoder plugin alone: AVIF and HEIC still decode through aom and dav1d, and
JPEG 2000 still works through OpenJPEG, which is a different library and stays
enabled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The prior-artifact download ran unconditionally, so a stage 0 job collected
whatever rpm-* artifacts its sibling stage 0 jobs had already uploaded. The
buildroot then depended on the order those siblings finished in, which is why
the failing set moved between runs of the same commit.

It also breaks the stage contract in the direction that matters. A stage
cannot see its own output -- that is the assumption every stage assignment in
config/upstream-sources.json is made against -- and here stage 0 saw part of
it, non-deterministically.

The failure names nothing near the cause. Collecting a sibling openjph 0.31
puts openjph into the by-name Fedora exclusion list, so:

  package libopenjph-0.25.3-3.fc44 from fedora is filtered out by exclude
  package libheif-1.21.2-1.fc44 requires libopenjph.so.0.25 ...
  glycin-loaders requires libheif.so.1 ... gdk-pixbuf2 requires libglycin-2
  graphviz requires libgdk_pixbuf-2.0.so.0 ... doxygen requires graphviz

and opus, mesa-demos, passim and msgraph fail on a package none of them names.

Guard the step on the stage number. Stage 0 resolves against Fedora and
Hummingbird alone, which is reproducible and is what its stage assignments
already assume.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
…-processing

Reading #45's config/build-lanes.toml rather than rediscovering it. It already
records why these three cannot build in the fast lane, and the reasoning holds
under the numeric stages this branch uses.

libheif now builds with %bcond bootstrap on, permanently. It build-requires
sdl2, and sdl2 -> SDL3 -> libdecor -> gtk3 -> gdk-pixbuf2 -> glycin-libs ->
glycin-loaders -> libheif closes a cycle. Fedora never trips it because its
repository is self-consistent; this accumulator is not. Bootstrap costs exactly
one binary, the heif-view example -- the encoders and every library stay.

That also retires the %bcond openjph I added earlier today, which dropped the
HTJ2K encoder to dodge the same problem. Ordering is the better answer:
libheif moves to the wave after openjph, so it links the openjph this factory
builds rather than the Fedora 0.25 it then excludes by name. The plugin comes
back.

ffmpeg and webrtc-audio-processing move with it, for the reason the draft
gives: they link libraries the fast lane rebuilds with moved sonames, so
building beside them leaves the consumer linked to sonames that never reach the
repository.

Their own consumers follow one wave later -- pipewire links
webrtc-audio-processing, and libepoxy, libglvnd, libayatana-ido,
xdg-desktop-portal-gtk and pipewire-libs-extra reach it through pipewire-libs.

Stages stay within the five the workflow allows: 296/24/16/6/3.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Adding sqlite to the Fedora exclusion list changed nothing, because excluding
a repository does not touch a package that is already installed. The base
image arrives carrying Fedora sqlite-libs 3.51.2, so the resolver preferred
Hummingbird while the build root kept Fedora, and subversion -- which
Hummingbird builds against 3.53.4 -- still could not open a repository:

  svn: E200029: atomic initialization could not be performed
  svn: E200030: SQLite compiled for 3.53.4, but running with 3.51.2

which surfaced, six tests later and one package away, as
t9168-git-svn-partially-globbed-names failing in the git build.

Upgrade each name in HB_EXCLUDE that the image already has, so the decision
that Hummingbird answers for it reaches the build root and not merely the
resolver. Scoped to those names rather than a general upgrade: they are
exactly the ones already declared, and a wholesale upgrade would move a build
root that 238 packages currently build against.

Also prints the resulting sqlite alongside the openssl line, so the next
mismatch of this kind is visible in the log rather than inferred from a test
failure three layers up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Guarding stage 0 fixed the worst case; the same race remained above it.
Artifacts are named rpm-s<stage>-<package> and the download pattern was rpm-*,
so a stage 2 job collected rpm-s2-* as well -- whichever of its own siblings
had finished uploading by then.

That is the same defect as the stage 0 one, and it fails the same way: the
buildroot depends on the order siblings complete in, so the same commit
produces different failures on different runs, each naming a package that is
not the cause.

Ask for strictly earlier stages: stage 3 collects rpm-s[0-2]-*. Every stage
now sees exactly what its stage assignment promises -- everything built
before it, and nothing built beside it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Seven of the eight stage 0 failures were not package failures at all:

  Failed to FinalizeArtifact: Received non-retryable error: (403) Forbidden

grub2, gtk2, iio-sensor-proxy, liblc3, xkbcomp, xmodmap and xrdb each built
their RPMs and then lost them on upload. Every job uploads its debuginfo and
debugsource alongside the RPMs, which no BuildRequires ever names and which
are more than half the bytes -- #45 already skips them when seeding a local
prior for exactly that reason. Exclude them from the artifact so the run stops
spending its budget on payload nothing downstream reads.

python-evdev is the one real failure. Its %generate_buildrequires runs
%pyproject_buildrequires, and that macro has to execute before the
dependencies it generates can be resolved -- so with nothing named statically
the first pass has nothing to run it with:

  pyproject-rpm-macros is needed by python-evdev
  python3-devel is needed by python-evdev
  python3dist(packaging) is needed by python-evdev

and the retry loop then reinstalls what is already installed, three times, and
gives up. Name them, the same way python-typing-inspection asks for
python3-packaging.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
Sixteen of twenty-three stage 1 packages failed on one knot, and openjph tied
it. This PR added openjph; libheif is the only recipe in the factory that
build-requires it, and Hummingbird ships no libopenjph at all -- only Fedora
does, at 0.25.

Carrying openjph 0.31 therefore excluded Fedora libopenjph by name, which made
Fedora libheif uninstallable, which made this factory own glycin-loaders
uninstallable, which took gdk-pixbuf2 and everything behind it down:

  glycin-libs-2.2~beta from stages requires glycin-loaders = 2.2~beta
  glycin-loaders-2.2~beta from stages requires libheif.so.1
  libheif-1.21.2-1.fc44 from fedora requires libopenjph.so.0.25
  libopenjph-0.25.3-3.fc44 from fedora is filtered out by exclude filtering

-- evolution-data-server, ffmpeg, flatpak, gdk-pixbuf2, gstreamer1-plugins-bad-free,
gstreamer1-plugins-good, libtevent, localsearch, pulseaudio, rest, sdl2-compat,
tecla, wayland, webkitgtk, wireplumber and xdg-user-dirs-gtk.

Ordering cannot untie it within five stages. openjph, libheif, glycin,
gdk-pixbuf2, then everything that reaches gdk-pixbuf2, is five waves before
webkitgtk, gjs and gnome-shell have anywhere left to go.

Declining the encoder removes the dependency instead: libheif no longer needs
openjph at build or at run time, Fedora libopenjph is never excluded, Fedora
libheif stays installable for glycin to link, and nothing has to move. libheif
returns to stage 0.

The cost is HTJ2K encoding. AVIF and HEIC decode through aom and dav1d, which
is what this image actually shows, and JPEG 2000 keeps working through
OpenJPEG -- a different library, still enabled. Counts drop to 344.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
…ed out

Sixteen stage 1 packages failed on the openjph knot, and the reason ordering
could not untie it was that five waves is not enough to express this
dependency graph. openjph, libheif, glycin, gdk-pixbuf2 and then everything
reaching gdk-pixbuf2 is already five, before webkitgtk, gjs,
evolution-data-server, flatpak or gnome-shell have anywhere to go.

fix/repeatable-local-builds solved that with sub-lanes -- stage0_late,
stage0_late_b, stage0_late_c, stage1_heavy, stage2_late, stage3_late -- each
its own wave with its own boundary. Rather than invent a second answer, take
that lane order and flatten it into consecutive numbers here:

  1  libheif                     (stage0_late)
  2  webrtc-audio-processing     (stage0_late_b)
  3  ffmpeg, gstreamer1-plugins-good, xorg-x11-server-Xwayland (stage0_late_c)
  5  webkitgtk                   (stage1_heavy)
  7  gjs                         (stage2_late)
  9  evolution-data-server, flatpak, gnome-shell (stage3_late)

with the old numeric waves shifted into 4, 6, 8 and 10 around them. The
reasoning behind each boundary is recorded in that file and holds unchanged:
WebRTC before any consumer whose buildroot resolves PipeWire; libheif before
its late consumers; Flatpak after the malcontent-bootstrap provider it needs
and before the full malcontent build that needs Flatpak.

rebuild-rpms.yml grows rebuild5 through rebuild10 on the same chunked
reusable-workflow pattern, and KNOWN_STAGES widens to match so the inventory
still refuses a stage no job can resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
The factory built Firefox only because Bluefin's copied [fedora] section
names it. Utah does not want the RPM: projectbluefin/utah already lists
firefox in its unavailable set and carries org.mozilla.firefox in
contracts/bluefin-desktop.toml.

Building it was not free. The recipe took a 2.5-hour PGO/LTO job in the
last full run -- the longest of the 285 -- carried its own 12G swap
workaround in the build workflow, and still failed in %install:

    cp: cannot create regular file
    'objdir/dist/bin/browser/chrome/browser/content/browser/default-bookmarks.html'

So the recipe, its source lock, its .packit.yaml entry and the swap step
that existed for it alone all come out. firefox joins [unavailable] in
the runtime contract, which is what keeps the Bluefin-requested name from
failing the transaction. mozjs140 is untouched -- it is the embeddable
JavaScript runtime and unpacks a firefox tarball of its own.

Inventory 193 -> 192; the four test counts follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
…ures

Merges the 151-recipe contract from #51 (which carries #52's matrix-cap
sharding and eleven stages) and fixes every non-Firefox failure from run
34415980185.

**The build root could not see the published repository.** `prepare` skips
any package already published at the same version, but the [factory] repo
block is guarded by `vars.FACTORY_REPO`, which is unset on pull_request
runs -- so it never ran, and a skipped package was simply absent. That is
why pipewire-libs-extra died on

    No match for argument: pkgconfig(libfreeaptx)

while libfreeaptx-devel sat published and skipped. FACTORY_REPO now
defaults to the same GitHub Pages URL `prepare` reads to decide what to
skip, so the two halves of that decision cannot disagree.

**Two recipes were staged before what they need.**

- xorg-x11-server-Xwayland 3 -> 5. It needs wayland-client >= 1.26.0 and
  the buildroot had Fedora's 1.25.0; this factory builds wayland at 4.
  Its three stage-0 build consumers resolve Xwayland from Fedora, so
  moving it later changes nothing for them.
- evolution-ews 6 -> 10. It needs libebook/libecal/libedata-* >= 3.61.3,
  and evolution-data-server builds at 9.

**gstreamer1-plugins-bad-free declines chromaprint**, on exactly the
reasoning ffmpeg.spec already records: Fedora's libchromaprint links
libavutil.so.60, this factory's ffmpeg 9.0.1 provides so.61, and the
factory excludes the Fedora libav by name, so libchromaprint-devel is
uninstallable here. The spec disabled chromaprint only under
`%if %{without extras}`; it is now its own bcond, off, with the
BuildRequires and the plugin in %files following it.

gcc was cancelled with the run, not failed, and needs only a re-run.

Inventory 344 -> 343 with Firefox dropped; validate, the runtime
contract, the quoting check and all 73 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
spirv-tools is the one failure in run 35200615124, and it cannot be fixed
here. 3d2da97 set SPIRV_WERROR=OFF, which worked -- the build got past
-Werror=switch and died one layer down on a real API break:

    core_tables_body.inc:1713:10: error: SPV_OPERAND_TYPE_GATHER_MODES
      was not declared in this scope

core_tables_body.inc is generated from the external spirv-headers grammar.
The newer headers emit an operand type that spirv-tools 2026.2 does not
define, so the generated table references a symbol that does not exist. That
is version skew between two separately released halves of one project, not a
warning to relax: the only fixes are a spirv-tools release that knows the
symbol or a vendored older headers copy, and neither belongs in this factory.

Rebuilding it here buys nothing. Nothing in the runtime contract asks for it
-- absent from bluefin-packages.toml, runtime-contract.toml and
factory-contract.json. Its only consumer in the tree is mesa, through
pkgconfig(SPIRV-Tools), and mesa sits in the same stage 0: it resolves
spirv-tools-devel out of the build root before our rebuild exists and can
never consume it. So the recipe is dead weight that fails a stage.

Dropping it follows 20eae7a (Firefox), 4d2af41 (openjph) and 4bb0711 (gcc):
the factory scope is the desktop stack Hummingbird does not ship, and a
recipe no consumer here can use is out of scope.

Removing spirv-tools from the rebuild set:
- config/upstream-sources.json: drop the entry
- .packit.yaml: drop the block
- packages/spirv-tools/: delete the recipe, patches and sources
- tests/: track the 341 -> 340 package count, and 340 -> 339 for the
  selection test that already skips dracut

267 tests pass, validate reports 340 source RPMs, the factory contract is
satisfied across 2463 tracked files, and the quoting check clears all 12
workflows. The plan drops stage 0 from 290 to 289 packages and leaves every
other stage untouched -- stage 6 still holds pipewire and samba.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
The ONNX fix in fa7c3fc worked. pipewire got through builddep, through the
build, and into %check for the first time in this factory -- and 51 of its 52
tests passed. The one failure:

    39/52 pipewire:pw-test-endpoint   FAIL   5.01s   killed by signal 14 SIGALRM
    Ok: 51   Fail: 1

That is not a timeout meson imposed. src/tests/test-endpoint.c:441 arms its
own watchdog:

    alarm(5); /* watchdog; terminate after 5 seconds */

and then drives five sequential pw_main_loop_run round trips through a real
pw_context with the session-manager modules loaded. The budget lives inside
the test binary, which is why --timeout-multiplier is the wrong lever here and
is deliberately not what this commit reaches for: meson never gets to apply a
timeout, because the process SIGALRMs itself first. Failing at 5.01s against a
5-second alarm is the signature of a process that ran out of wall clock, not
one that deadlocked.

What is movable is the contention. %meson_test defaults to one process per
core, so 52 tests share a 4-vCPU runner while this one needs five daemon
handshakes inside five seconds. --num-processes 1 gives it the box.

Nothing is skipped and nothing is made non-fatal. The same 52 tests run and
the same exit status still fails the build -- tests_nonfatal is untouched, and
it would be the wrong answer even though the spec offers it. This is the
approach e6cf24a already took for librsvg2's remaining suites, where a
parallel run was likewise what produced the failures.

The cost is wall clock in one job. pipewire's %check took 7m22s parallel; one
process at a time will be slower, and that is an acceptable trade against a
stage 6 failure that blocks the publish gate.

Verified from the primary source rather than inferred: the failing test name
was read off run 35200615124 job 105241322510, and alarm(5) was read out of
the pipewire-1.6.8 tarball this factory locks, whose sha512 matches
config/upstream-sources.json.

267 tests pass, validate reports 340 source RPMs, the factory contract is
satisfied across 2463 tracked files, and the quoting check clears all 12
workflows.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 17, 2026

Copy link
Copy Markdown
Member Author

b6b5b4a did not fix pipewire, and it disproved my diagnosis

Serializing %check was the wrong fix, and the run that tested it says so cleanly. Stage 6 of run 35236148265:

38/52 pipewire:pw-test-protocol-native    OK      6.01s
39/52 pipewire:pw-test-endpoint           FAIL    5.00s   killed by signal 14 SIGALRM
40/52 pipewire:pw-test-interfaces         OK      0.00s
41/52 pipewire:pw-test-stream             OK     12.02s
42/52 pipewire:pw-test-filter             OK     18.03s
43/52 pipewire:pw-test-security-context   OK      6.01s
50/52 pipewire:test-context               OK     12.02s
Ok: 51   Fail: 1

--num-processes 1 took effect — the tests ran strictly one at a time. And with the whole 4-vCPU runner to itself, pw-test-endpoint still died at 5.00s, against 5.01s when it was competing with 51 other tests.

That rules out starvation. The neighbouring tests are the control: pw-test-filter gets 18 seconds and passes, pw-test-stream and test-context get 12 and pass, and they build real pw_contexts in the same container. Wall-clock capacity was never the constraint. A test that fails at its 5-second alarm whether it has one core or four isn't slow — it's hung, and the watchdog is simply what kills it.

Which also kills the fallback I'd proposed. Patching alarm(5) upward would have produced a test that hangs for 60 seconds and then fails. I'd have shipped a second wrong fix on a worse hypothesis.

The test emits nothing at all — meson records <testcase name="pipewire:pw-test-endpoint" time="5.0048"><failure /></testcase>, empty, no stdout — so the hang is somewhere in the session-manager endpoint path (module-session-manager: client-endpoint, endpoint, endpoint-stream, endpoint-link) that src/tests/test-endpoint.c drives through five sequential pw_main_loop_run round trips.

What this leaves

--num-processes 1 should come out — it's justified by a hypothesis that's now disproved and only makes %check slower. I've not pushed a bare revert, because a 5-6 hour traversal for a no-op change isn't worth a runner; it should ride along with whatever actually lands.

Some facts that bear on the options, measured rather than assumed:

  • pipewire is not in Utah's install contract. contract() over bluefin.toml + utah.toml resolves 79 packages and pipewire is not among them; only pipewire-libs-extra, which is a separate recipe here and built green. Runtime pipewire comes from the Hummingbird base.
  • One factory recipe could consume our rebuild: gnome-shell at stage 10, via BuildRequires: pkgconfig(libpipewire-0.3) >= %{pipewire_version}. Every other in-tree consumer — mutter (6), wireplumber (4), xdg-desktop-portal (0), SDL3 (0), gstreamer1-plugins-good (3) — is at or before stage 6 and resolves pipewire from the build root regardless.
  • gnome-shell built green in run 35200615124 with pipewire red, so Hummingbird's pipewire already satisfies that constraint.

So the honest options are:

  1. Drop the pipewire recipe and let Hummingbird supply it, keeping pipewire-libs-extra. Precedent: spirv-tools, gcc (fix(gcc): drop gcc from the rebuild set, let Hummingbird supply it #90), Firefox. The measurements above say nothing breaks. But this is the audio stack of a desktop factory, and unlike spirv-tools there is a real consumer relationship with gnome-shell, so it is a scope decision rather than a cleanup.
  2. Report it upstream to PipeWire with this evidence. Right thing to do regardless, but it does not unblock the publish gate.
  3. Root-cause the hang properly — which needs a container runtime to reproduce interactively, and I have none in this environment, so anything I did here would be guesswork.

tests_nonfatal is not on that list. The spec offers it and it would turn this green in one line, and it is exactly the wrong answer.

Everything else in the contract is proven. Run 35200615124 completed all eleven stages with this as the only failure in 338 jobs — samba, webkitgtk, nautilus, mesa, flatpak, evolution-data-server, librsvg2, gjs, gdm and gnome-shell all built, and precedence passed. The publish gate is correctly holding on this one test.


Generated by Claude Code

@hanthor
hanthor merged commit 5116be9 into main Sep 18, 2026
680 of 681 checks passed
@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Correcting myself on this, before anyone acts on it.

My comment above offered "drop the pipewire recipe and let Hummingbird supply it" as option 1, on the reasoning that pipewire is absent from Utah's resolved install set. That reasoning was wrong, and the option is not available. I checked that the name wasn't in Utah's explicit list and inferred Hummingbird supplied it — without ever checking Hummingbird.

Measured since:

source pipewire
public-hummingbird-x86_64-rpms no pipewire packages at all
pinned factory image pipewire 1.6.8-3.hum1.bfin + 24 subpackages (-alsa, -pulseaudio, -libs, -utils, …)

This factory is the only source of the audio stack for Utah. And pipewire-libs-extra, which is named in bluefin.toml and does resolve into Utah's install set, is built from the same pipewire-1.6.8.tar.gz with Requires: pipewire >= %{version} — so dropping the recipe makes it unsatisfiable and fails the image build. pipewire reaches the image as a dependency of that, from here.

What stands from the earlier comment: the diagnosis. Serialized on an idle runner it still dies at 5.00s while neighbouring tests take 12s and 18s and pass, so it is a hang and not starvation, and raising the watchdog is no more a fix than serializing was.

Also worth recording: the reason this surfaced only now is that prepare skips a package the published repository already carries at the same version. 1.6.8-3.hum1.bfin is published, so the recipe was being skipped until fa7c3fc and b6b5b4a touched it and forced a rebuild. The published build even carries pipewire-module-filter-chain-onnx, so it was built with ONNX enabled before the abseil soname drift that fa7c3fc worked around.

Full write-up, with the measurements and the three real options, in #132.


Generated by Claude Code

hanthor pushed a commit that referenced this pull request Sep 18, 2026
Every run was a full rebuild. prepare decides what is already built by
reading a repository listing, and the URL it read -- the same one every
build root enabled as [factory] -- was the GitHub Pages mirror that README
says was removed. The deployment was never taken down, so it still answers,
frozen at 67 packages from before the 343-recipe contract. Both runs 375
and 377 say it plainly:

    published repo has 67 source packages
    will build 333 of 340 packages

Nothing publish wrote could ever reach that listing, so no run could ever
skip more than seven recipes. Eight hours for a one-line spec fix, and
webkitgtk alone five of them on the serial path at stage 5, with no change
to its recipe.

The image publish writes is the only thing that moves, so it is now the
only thing either half reads:

- prepare resolves ghcr.io/<owner>/utah-packages to a digest once (the ref's
  own tag first, then latest; main reads only latest), extracts /repository
  out of it, and runs the plan against that on disk. No image at all is a
  warning and an empty witness, which rebuild_plan.py already treats as
  "skip nothing".
- every build job, and preflight, extract the same digest with the new
  load-factory-repo action and enable it as file:///work/factory. Same
  bytes as the plan read, and nothing fetched over HTTP mid-build. The
  vars.FACTORY_REPO override is gone: a second source of truth is how the
  two halves first came to disagree.

Skipping correctly makes a second gap load-bearing. A fix to mutter would
build mutter alone and leave the published gnome-shell linked against the
libmutter it just replaced. The plan now reads Provides and Requires out of
the same primary.xml and drags every published dependent of anything being
rebuilt, transitively, in inventory order so the waves still hold. It is
the runtime closure; the repository carries no SRPMs, so a devel-only
BuildRequires is the residual gap.

Also on the critical path: debuginfo. The upload step excludes every
debuginfo and debugsource RPM by pattern, so they were built, stripped and
compressed only to be discarded -- 25 minutes of find-debuginfo and 10 of
compression on webkitgtk alone. Both lanes now build with
debug_package %{nil}.

And two things that made the runs read worse than they were: precedence
and publish ran after a failed prepare and died on a missing artifact
(372, 379), so they now require prepare to have succeeded; and a push to a
pull request left the previous run going for hours (369, 370, 374 were all
cancelled by hand), so pull_request runs now cancel their predecessor.
main and the schedule are untouched.

Verified: 293 tests pass, including new ones for the dependency closure
and for the workflow wiring; both workflows and the action parse; the
quoting check passes. The first run after this lands is still a near-full
build, because latest holds the 60-package image from before #70 -- the
last one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
This was referenced Sep 18, 2026
hanthor added a commit that referenced this pull request Sep 20, 2026
…n to use it (#178)

* feat(sources): poll git forges for releases, not just GNOME

The factory's stated purpose is to track the latest upstream release rather
than whatever Fedora tagged -- docs/targeting-hummingbird.md is explicit that
"a spec is a build recipe, not a release feed". Automation covered 17 of 340
locks, all of them download.gnome.org. 5%.

This adds the next tranche that has a real release feed, taking it to 63:

    download.gnome.org  17  cache.json per-module index   (unchanged)
    github.com          36  tags, or releases where published
    gitlab (3 hosts)    10  repository tags API

The remaining 277 are locked on Fedora's lookaside cache or on a bare
directory listing. A lookaside URL carries its own SHA-512 in the path, so
there is nothing to poll for "latest" -- reaching those needs each project's
real upstream feed identified per package, which is a different job and is
filed separately.

Three shapes are recognised, measured against the real inventory rather than
guessed at: github /archive/<tag>/ (25), github /releases/download/<tag>/
(11), and gitlab /-/archive/ or /-/releases/<tag>/downloads/ (10). Two
details came from reading the locks instead of assuming: evtest is pinned over
http rather than https, and two freedesktop projects use the GitLab release
shape rather than the archive one. Both would have been silent misses -- the
first draft of the pattern found 7 of the 10 GitLab locks and said nothing
about the other three.

Where this differs from the GNOME path, and why:

  Release selection. GNOME has release cycles, so that path only moves inside
  the cycle a lock already names. A forge project has no cycle, so the major
  stands in for it: a newer stable release sharing the current major is an
  "update" and may be applied, one that crosses a major is "review" only and
  is never written. A major can move a soname and break every consumer in the
  graph; no comparison of version strings can make that call.

  URL rewriting. GNOME entries are rebuilt from a known template. A forge URL
  has no template this tool can own, because projects choose their own tag and
  asset names, so forge_planned_entry substitutes the old version for the new
  and the old digest for the new across url, filename, sha256_url and
  fallback_urls. Substituting the bare version fixes a v-prefixed tag in the
  same pass, since "v2.2.1" contains "2.2.1". It refuses outright when the
  current version does not appear in the URL, rather than writing a new
  version beside an old tarball.

  Prereleases. A maintainer's draft or prerelease flag on a GitHub release is
  honoured before the version string is examined, because some prereleases
  carry no alpha or beta suffix at all.

GITHUB_TOKEN is wired into the workflow and is not optional: anonymous GitHub
allows 60 requests an hour against 36 locks, so an unauthenticated run
rate-limits part way down the list and reports the rest as unreachable --
which reads as "nothing to bump". A non-list API response is raised as an
error for the same reason; treating GitHub's rate-limit object as an empty
feed would report every package as current.

The workflow is renamed bump-upstream-sources.yml, since it is no longer only
GNOME. Nothing referenced the old filename.

Verified. 287 tests pass, 20 of them new, all offline through the existing
fake opener. The GitLab path was also exercised against the live API: it
returns 19, 50, 50, 13 and 75 versions for libcamera, wayland-protocols,
shared-mime-info, evtest and accountsservice, and correctly reports each as
already at its newest stable -- so "nothing to bump" there is a real answer
and not an empty list. validate reports 340 source RPMs, the factory contract
is satisfied across 2463 files, and the quoting check clears all 12 workflows.

NOT verified here: the GitHub path against the live API. api.github.com
answers 403 in this environment -- a plain curl does too, so it is the network
and not this code. Its tag, release, draft/prerelease and rate-limit
behaviours are covered by unit tests through the fake opener, and the first
real exercise will be the workflow's own run with a token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* fix(ci): skip against the published image, not a retired Pages mirror

Every run was a full rebuild. prepare decides what is already built by
reading a repository listing, and the URL it read -- the same one every
build root enabled as [factory] -- was the GitHub Pages mirror that README
says was removed. The deployment was never taken down, so it still answers,
frozen at 67 packages from before the 343-recipe contract. Both runs 375
and 377 say it plainly:

    published repo has 67 source packages
    will build 333 of 340 packages

Nothing publish wrote could ever reach that listing, so no run could ever
skip more than seven recipes. Eight hours for a one-line spec fix, and
webkitgtk alone five of them on the serial path at stage 5, with no change
to its recipe.

The image publish writes is the only thing that moves, so it is now the
only thing either half reads:

- prepare resolves ghcr.io/<owner>/utah-packages to a digest once (the ref's
  own tag first, then latest; main reads only latest), extracts /repository
  out of it, and runs the plan against that on disk. No image at all is a
  warning and an empty witness, which rebuild_plan.py already treats as
  "skip nothing".
- every build job, and preflight, extract the same digest with the new
  load-factory-repo action and enable it as file:///work/factory. Same
  bytes as the plan read, and nothing fetched over HTTP mid-build. The
  vars.FACTORY_REPO override is gone: a second source of truth is how the
  two halves first came to disagree.

Skipping correctly makes a second gap load-bearing. A fix to mutter would
build mutter alone and leave the published gnome-shell linked against the
libmutter it just replaced. The plan now reads Provides and Requires out of
the same primary.xml and drags every published dependent of anything being
rebuilt, transitively, in inventory order so the waves still hold. It is
the runtime closure; the repository carries no SRPMs, so a devel-only
BuildRequires is the residual gap.

Also on the critical path: debuginfo. The upload step excludes every
debuginfo and debugsource RPM by pattern, so they were built, stripped and
compressed only to be discarded -- 25 minutes of find-debuginfo and 10 of
compression on webkitgtk alone. Both lanes now build with
debug_package %{nil}.

And two things that made the runs read worse than they were: precedence
and publish ran after a failed prepare and died on a missing artifact
(372, 379), so they now require prepare to have succeeded; and a push to a
pull request left the previous run going for hours (369, 370, 374 were all
cancelled by hand), so pull_request runs now cancel their predecessor.
main and the schedule are untouched.

Verified: 293 tests pass, including new ones for the dependency closure
and for the workflow wiring; both workflows and the action parse; the
quoting check passes. The first run after this lands is still a near-full
build, because latest holds the 60-package image from before #70 -- the
last one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* fix(ci): stop the native find-debuginfo too, or the MinGW specs orphan .debug files

Run 380 failed ten stage-0 packages -- enchant2, hidapi, libsoup3, libvorbis,
libxmlb, openjpeg, opus, protobuf3, taglib, wavpack -- all with the same
error at the same step:

    error: Installed (but unpackaged) file(s) found:
       /usr/lib/debug/usr/lib64/libhidapi-hidraw.so.0.15.0-...x86_64.debug
       /usr/src/debug/hidapi-0.15.0-4.hum1.bfin.x86_64/hidapi/hidapi.h

Those ten are exactly the specs that build a MinGW half and say
%{?mingw_debug_package}; the eleventh such spec, abseil-cpp, has its mingw
bcond off and passed. Nothing else in the run failed.

debug_package %{nil} only removes the native -debuginfo subpackage. It does
not stop find-debuginfo, which __spec_install_post runs whenever
__debug_package is set -- and Fedora's %mingw_debug_package sets it itself
(mingw-filesystem, macros.mingw line 20: %global __debug_package 1). So on
these ten the native extraction still ran, wrote .debug files into the
buildroot, and the subpackage that used to claim them was gone.

The fix empties __debug_install_post as well, in both lanes. That stops the
native extraction at its source. The MinGW halves are unaffected: their
.debug files come from the explicit %mingw_debug_install_post call in
%install and go into the mingw32/mingw64 -debuginfo subpackages, which still
exist and which the upload filter drops like every other debuginfo RPM.
Specs that reference __debug_install_post themselves (libcamera, libxcrypt,
grub2) all guard it with %{?__debug_package:...}, which stays unset for them.

Reproduced and fixed locally with a five-line spec that sets __debug_package
the way the mingw macro does: debug_package %{nil} alone dies on the same
"Installed (but unpackaged)" .debug file; with __debug_install_post %{nil}
added it writes the RPM. The workflow test now pins both defines in both
lanes. 300 tests pass; the quoting check clears all 12 workflows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Revert "fix(spirv-tools): drop the recipe, let the build root supply it"

This reverts commit 56b2973.

* Restore spirv-tools at 2026.3 and rebuild stale published packages

Run 381 got past every build stage but failed the publish job's
"Validate Hummingbird-only consumer transaction" for two reasons.

First, mesa-dri-drivers links libSPIRV-Tools.so and Hummingbird does
not ship spirv-tools, so dropping the recipe (56b2973) made the runtime
contract unresolvable. The recipe is back, moved to Fedora Rawhide's
2026.3 (SDK 1.4.357.0) because Fedora 44's spirv-headers 1.5.5 carries
GatherModes, which spirv-tools 2026.2 cannot compile against. The
tarball comes from the Fedora lookaside with its sha512 verified; the
two downstream patches no longer apply and are gone with the spec.

Second, several packages in the 67-package witness were built against
sonames the factory no longer provides (libheif -> libavcodec.so.62,
gvfs -> libbluray.so.3, mozc -> libabsl 2601, gnome-shell ->
libedataserver-1.2.so.27). Their current specs already disable those
inputs, but nothing changed in the recipes, so the plan trusted the
published build and skipped them. The dependents-dragging rule cannot
see this because the missing provider is absent from the witness.

rebuild_plan now reads the published repo's Requires and Provides and
marks a source stale when one of its binaries requires something that
neither the published repo nor Hummingbird provides. rebuild_matrix
fetches Hummingbird's primary.xml (baseurl from config/hummingbird.repo)
for the external provides, prints the reason for each stale rebuild,
and skips the rule with a warning if the fetch fails. Stale packages
drag their dependents like any other rebuild.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* docs: record the mistakes the history already made, so they are not made again

git log on main is a record of fixes that undid earlier fixes: fifteen
commits fighting one ICU split, three digest repins that each died within
days, a recipe dropped and restored, the same libheif bcond added and
retired twice, four soname breaks that were one staging mistake, and two
empty commits pushed to re-run CI.

docs/skills/repeated-mistakes.md names each recurring pattern, the commits
that made and unmade it, and the rule that settled it, with a checklist to
run before pushing a fix. It is indexed in docs/SKILL.md and AGENTS.md so
the router loads it for stage, pin, bcond, workflow and recipe-removal
work.

The triage skill still told readers to repin the build-root digest on exit
125 "manifest unknown"; f1f1e4e showed that cannot work for an image whose
digests live hours, and moved the build root to a once-per-run pull. Both
rows now say so instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Build the Wi-Fi supplicant, libdaemon and the nautilus closure

Eight recipes imported from Fedora Rawhide dist-git, each pinned by commit
in .hummingbird-upstream.json and source-locked to its upstream release:

  wpa_supplicant 2.11, wireless-regdb 2026.09.03, iw 6.17
    Hummingbird ships NetworkManager-wifi, whose metadata requires
    wireless-regdb and (wpa_supplicant or iwd), and ships none of the
    three, so Wi-Fi cannot be installed on Utah at all (#136, utah#97).
    wireless-regdb in turn requires iw. The wpa_gui bcond is forced off:
    nothing in the contract wants it and the build root has no qt-devel.

  libdaemon 0.14
    Hummingbird's avahi daemon requires libdaemon.so.0 and ships no
    libdaemon, so geoclue retries forever against no daemon (utah#104).

  inih 62 (stage 0), exiv2 0.28.9 (1), libgexiv2 0.16.2 (2), libportal
  0.10.0 (5)
    The factory built nautilus with no way to install it: it needs
    libgexiv2-0.16.so.4, libportal.so.1 and libportal-gtk4.so.1, none in
    any repository (utah#100). exiv2 links inih, gexiv2 links exiv2, and
    libportal links gtk3 and gtk4 (gtk4 is stage 4), hence the stages;
    nautilus itself is at 6. libportal's Qt5 backend is off, since
    Hummingbird has no Qt5 and only the GTK4 backend is consumed.

inih and exiv2 are GitHub archives and gexiv2 is on download.gnome.org;
the bootstrap tool could not fetch them through this environment's proxy,
so their locks are written by hand with the sha512 Fedora's sources file
records, verified against the byte-identical lookaside copy, which is
also the fallback URL, as for the other GitHub-hosted recipes.

The runtime contract now asks for NetworkManager-wifi, wpa_supplicant,
wireless-regdb, avahi and nautilus, so publication proves the three
closures resolve from factory plus Hummingbird before Utah can pin them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Patch pw-test-endpoint's watchdog, and drop the serialization that did not work

pipewire was the one failure in 333 jobs of the last run, and it failed
the same way it did before b6b5b4a:

    39/52 pipewire:pw-test-endpoint  FAIL  5.01s  killed by signal 14 SIGALRM
    Ok: 51   Fail: 1

b6b5b4a serialized %check and claimed that fixed it. It did not. The log
shows --num-processes 1 in the meson test command line and the identical
failure at the identical 5.01s, so contention was never the cause and the
serialization was only making the longest job in stage 6 slower. It is
reverted here. Reporting that fix without a green job to point at is the
mistake docs/skills/repeated-mistakes.md records as pattern 13, made
again by me.

The real cause is in the test. src/tests/test-endpoint.c:441 arms

    alarm(5); /* watchdog; terminate after 5 seconds */

around setup that costs more than five seconds on this build root, so the
process kills itself; meson never gets to apply a timeout, which is why
--timeout-multiplier is not the lever either. The budget is out of line
with the test's own siblings rather than with anything the test does:
pw-test-stream and pw-test-filter are declared in the same test_apps list
in src/tests/meson.build, with the same environment, and took 12.02s and
18.03s in this same run on this same machine.

So the watchdog is patched to 60 seconds. Nothing is skipped, disabled or
made non-fatal: every assertion still runs and still has to pass. The run
is still bounded, by meson rather than by the alarm -- these tests declare
no timeout: kwarg, and meson's default is 30 seconds (KwargInfo('timeout',
int, default=30) in mesonbuild/interpreter/type_checking.py), so a genuine
hang is killed at 30s, before the new alarm can fire. The alarm goes from
being the tighter guard to being the redundant one.

The patch was applied to the locked 1.6.8 tarball and checked rather than
assumed: patch -p1 applies clean and leaves alarm(60) at line 441.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Seed publish from the image prepare witnessed, not from :latest

Review finding on dae6241, and it is this change's own bug one layer down.

prepare resolves the skip witness as the branch tag first and latest second.
publish seeded from a hardcoded ghcr.io/<owner>/utah-packages:latest. So the
two disagreed about what is already built: on a second push to a pull request,
anything skipped because the *branch* tag carried it was in neither the seed
nor that run's artifacts, and the republished branch image simply lost it --
or regressed to the stale latest copy for packages the first push had rebuilt.

That is exactly the failure this change removed vars.FACTORY_REPO to prevent,
reintroduced at the far end of the same pipeline. publish already needs
prepare, so its resolved digest was there to use all along.

The empty case is handled rather than assumed: when no image is published at
all, prepare emits an empty witness and warns, and the seed now says so and
publishes a new repository instead of trying to pull "".

The concurrency group is scoped to pull requests too. The group covered every
ref while only cancel-in-progress was conditional, so main and schedule runs
queued behind one another and GitHub could drop a superseded pending run --
a behaviour change the block's own comment says is not wanted, since
concurrent waves are safe by design.

tests/test_factory_witness.py asserts the seed reads prepare's output and that
no workflow names the latest tag directly. Reverting the seed to :latest fails
it, so it is a gate and not a restatement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Build ffmpeg without libplacebo, which no consumer can provide

The publish gate on run 35413902261 refused to publish, correctly. All 331
build jobs passed; the Hummingbird-only consumer transaction then failed:

    Problem 2: package nautilus-gsconnect ... requires
      gnome-shell-extension-gsconnect = 72-3.hum1.bfin, but none of the
      providers can be installed
      - package gnome-shell-extension-gsconnect ... requires /usr/bin/ffmpeg,
        but none of the providers can be installed
      - nothing provides libplacebo.so.360()(64bit) needed by
        ffmpeg-free-9.0.1-1.hum1.bfin.x86_64 from factory

libplacebo exists in neither runtime repository. Hummingbird does not package
it -- I read its primary.xml and there is no libplacebo of any version among
the 20050 packages -- and the factory has no recipe. So the BuildRequires was
satisfied by Fedora 44 in the build root alone, and the built ffmpeg-free
carried a runtime dependency nothing could ever answer. That is precisely the
build-only-becomes-runtime failure this factory exists to prevent, and it took
gsconnect down with it because that needs /usr/bin/ffmpeg.

Two ways out: add libplacebo to the factory, or stop linking it. libplacebo
would need glslang, shaderc and spirv-headers first, none of which the factory
builds, so that is four new recipes for Vulkan-backed libavfilter paths
(placebo scaling and tone mapping) that nothing in Utah's desktop asks for
yet. Turning the bcond off is one line and makes ffmpeg-free installable now;
the comment records what it costs and what re-enabling needs.

Release goes to 2 so the rebuilt RPM is not the same NVR with different
dependencies.

This does not address the other half of the gate failure -- nautilus requiring
libicuuc.so.77 against a runtime that excludes libicu 77. That one needs a
decision rather than a patch, and I have written the evidence up on the PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Make the build root pick the same ICU the consumer transaction demands

The other half of the publish failure on run 35413902261. All 331 builds
passed; the Hummingbird-only transaction then refused:

    package nautilus-51~beta-1.hum1.bfin from factory requires
      libicuuc.so.77, but none of the providers can be installed
      - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
        filtered out by exclude filtering

I read Hummingbird's primary.xml rather than reasoning from the comment
history, and it settles the question. Hummingbird has migrated to ICU 78:

    harfbuzz-icu 14.1.0-1.1 .. 14.2.0-1   libicuuc.so.77
    harfbuzz-icu 14.2.1-3   .. 14.3.1-1   libicuuc.so.78
    boost-locale 1.90.0-6.1, -7           libicuuc.so.77
    boost-locale 1.90.0-9 .. -13          libicuuc.so.78
    libtinysparql 3.11.1-2                libicuuc.so.78

Every current build links 78; only superseded ones link 77. So my earlier note
on the PR -- that Hummingbird is itself split across both ICUs -- was wrong: I
had counted superseded builds alongside current ones. The consumer exclusion of
libicu 77 is therefore correct, and nothing current needs 77.

What went wrong is that the build root never made the same choice. The two
libicu builds share a package name, so dnf installs exactly one and nothing
said which. A root that resolved a superseded copy pulled libicu 77;
libicu-devel followed it to 77.1 to match; and the package built there linked
ICU 77, which its consumers then refuse. Two halves of one decision, disagreeing
-- pattern 2 in docs/skills/repeated-mistakes.md.

So HB_REPO_EXCLUDE gains the same exclusion, in the same spelling, with release
as its own field because dnf splits a spec on dashes before globbing (1f88491).

This is not the blanket exclusion 012cb6a reverted. That one reached Fedora,
whose build-only deps do legitimately link ICU 77 -- raptor2, localsearch and
pipewire-libs-extra transitively -- and it is scoped to the Hummingbird
repository here. HB_EXCLUDE already keeps Fedora ICU out of the build root
entirely, and the rendered mock config shows both halves of that:

    excludepkgs=...,libicu,icu,...                    (fedora, fedora-updates)
    excludepkgs=ruby3.3-...,ruby3.4-...,libicu-77.*-*hum1   (hummingbird)

tools/mock_config.py carries it too, because tests/test_mock_config.py already
asserts the mock root and the container root encode one policy -- that test
caught this change when only the workflow had it, which is the guard working.
New IcuAgreementTests then asserts the build root and the consumer transaction
name the same ICU, so they cannot drift apart again.

309 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Stop the staleness model counting an ICU the gate will refuse

Third place the same decision lives, and the one that would have let this
failure repeat even with the other two fixed.

rebuild_matrix builds `external` from every capability Hummingbird provides,
libicu 77 included, and stale_from_primary asks whether a published binary's
Requires are satisfied by factory plus that set. So a published package linked
against libicuuc.so.77 looks satisfiable, is skipped as fresh, and then fails
the consumer transaction -- which excludes libicu 77 -- at publish. The model
that exists to predict that transaction disagreed with it.

Run 35413902261 is what that costs: 331 green builds, precedence green, then
publish refused, and nothing published.

provides_from_primary now takes the builds the transaction refuses and skips
their capabilities, driven by one module constant:

    EXCLUDED_EXTERNAL = (("libicu", "77."),)

A package linked against the refused build is then reported stale, which makes
it rebuild against the current root -- the only cure -- rather than being
carried forward to fail the gate.

The tests use a primary.xml shaped like the real thing, with libicu 77.1 and
78.3 under one name, and assert all three directions: 77 provides nothing, a
published package needing .so.77 is stale, and one needing .so.78 is not, so
the rule does not condemn everything that touches ICU. Emptying
EXCLUDED_EXTERNAL fails three of them.

313 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Revert the build-root ICU exclusion: it stranded the last provider of .so.77

I added libicu-77.*-*hum1 to HB_REPO_EXCLUDE two commits ago and it made the
factory strictly worse. bluez failed at stage 0 of run 35443117478:

    package libical-3.0.20-7.fc44 from fedora requires libicuuc.so.77, but
    none of the providers can be installed
    - package libicu-77.1-2.fc44 from fedora is filtered out by exclude filtering
    - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
      filtered out by exclude filtering

The reasoning that produced it was checked against the wrong population. I read
Hummingbird primary.xml, found that every current Hummingbird build links
libicuuc.so.78 and only superseded ones link .so.77, and concluded that
excluding 77 could not strand anything. That conclusion is sound about
Hummingbird and says nothing about the build root, which also carries Fedora
binaries Hummingbird has never rebuilt. libical is one of them and requires
.so.77 outright.

My own commit message contained the refutation: HB_EXCLUDE already keeps Fedora
libicu out of the root, so Hummingbird superseded libicu-77 was the only
remaining provider of .so.77. Removing it left none. I quoted that line as
evidence the exclusion was safe when it is the reason it was not. 012cb6a
reverted a blanket libicu exclusion for this same underlying reason, and
scoping it to the Hummingbird repository did not change the reason -- it only
changed which repository the last provider happened to live in.

Trading a publish-gate failure for a stage-0 failure is a bad trade: the gate
failure came after 331 successful builds and named one package, this one loses
every build behind it. So the build-root half comes out, in the workflow and in
tools/mock_config.py, and both places now carry a note saying why it must not
come back.

The publish-side half stays. EXCLUDED_EXTERNAL in tools/rebuild_plan.py and the
consumer exclusion are about what may be installed on a Hummingbird system,
where nothing current links ICU 77; the build root is a different question with
a different answer. IcuAgreementTests asserted the two halves were symmetric;
it now asserts the asymmetry and says why, so the next person to notice that
Hummingbird has migrated to 78 does not make this change again.

The underlying problem is unsolved and stated as such in both comments: a
factory package can still link ICU 77 because Fedora build deps pull it into
the root, and the consumer transaction then refuses that package. It has to be
fixed without removing the .so.77 provider the root legitimately needs.

315 tests pass. check_workflow_quoting.py clean; the new comment contains no
apostrophe.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Build samba before the packages that would otherwise take Fedora ICU 77

This is the actual cause of the publish failure the ICU exclusions were
flailing at, and it is an ordering bug, not an exclusion one.

Hummingbird ships libicu 77.1 beside 78.3 under one name, so a build root
installs exactly one. dnf takes the newest unless something in the root
requires libicuuc.so.77, and then it downgrades -- libicu-devel follows the
base to match, because icu.spec pins it exactly:

    Requires: lib%{name}%{?_isa} = %{version}-%{release}

so there is no devel-only split to exploit, which was the other fix I
considered. Whatever builds in that root then links ICU 77, and the
Hummingbird-only consumer transaction refuses it.

The chain in run 35413902261 runs entirely through our own staging:

  Fedora samba-core-libs-4.24.6-1.fc44 requires libicuuc.so.77
    -> localsearch (stage 4) resolved samba from Fedora, because the factory
       samba was stage 6 and did not exist yet. Its root installed
       libicu-77.1-2.1.hum1 over five available 78.3 builds, including our own
       libicu-78.3-1.hum1.bfin, and localsearch-3.12~beta-1.hum1.bfin came out
       requiring .so.77.
    -> nautilus (stage 6) requires localsearch, so its root downgraded too, and
       nautilus-51~beta-1.hum1.bfin required .so.77.
    -> publish step 6 refused nautilus. All 331 builds had passed.

The factory samba is not the problem: it built against libicu-78.3-1.hum1.bfin
with zero .so.77 anywhere in its root. Its own entry says it was rebuilt
precisely to stop requiring .so.77. It was simply built four stages too late to
be used by the packages that needed it.

samba moves to stage 2 and libtevent, its only factory build dependency above
stage 0, to stage 1. Both are safe by their own dependencies: libtevent is
built from libtalloc and make, both stage 0; samba from libtevent plus
libtalloc, libtdb, icu and libxcrypt, all stage 0 or now 1. Nothing moves
later, so no existing consumer ordering can break -- pulling a provider earlier
only ever adds packages to the roots that follow it.

tests/test_icu_staging.py makes this a gate. It asserts samba precedes both
known consumers and that samba still follows everything it is built from, and
it fails against the previous stage numbers. The docstring carries the whole
chain, because the next person to meet an ICU 77 requirement will otherwise
reach for an exclusion again -- which is what de4ac4d had to revert, since
Fedora packages the root legitimately needs (libical) require .so.77 and
excluding it strands them outright.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Build gvfs after samba too: it takes ICU 77 by the same route

Found by asking which factory packages consume samba, which I should have done
in the same breath as moving it. gvfs BuildRequires libsmbclient-devel -- a
samba subpackage -- and sat at stage 1, so it resolved libsmbclient from Fedora
and its root installed libicu-77.1-2.1.hum1, exactly as localsearch did:

    [ 34/485] Installing libicu-0:77.1-2.1.  36.4 MiB
    [132/485] Installing libicu-devel-0:77.   5.1 MiB

That was already true before the previous commit -- gvfs stage 1 against samba
stage 6 -- so this is not fallout from moving samba, it is the same bug one
package over that moving samba to stage 2 did not by itself reach.

gvfs goes to stage 3. It is unconstrained in both directions: no factory package
BuildRequires gvfs at all, and its only build dependency above stage 0 is samba,
now at 2. Everything else it needs -- libnfs, libsoup3, libsecret, udisks2, gcr,
ibus, libcdio, libplist, libimobiledevice, gnome-online-accounts, libtalloc --
is stage 0.

The dependency scan behind both moves now resolves pkgconfig() BuildRequires as
well as plain names, which the first pass skipped; re-running it over samba
confirms its factory dependencies are still only libtevent (1) and stage-0
packages, so stage 2 remains correct.

tests/test_icu_staging.py covers gvfs in both directions and fails against the
previous stage number.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Find the remaining ICU-77 consumers by rule instead of one at a time

Three contaminated packages had been found three different slow ways --
localsearch and nautilus from the publish failure, gvfs by asking who consumes
samba, and each of those took a round trip. So I asked the general question
instead: which factory packages link ICU at all, and does anything consume one
of them from its own stage or earlier?

Nine factory packages BuildRequire ICU (enchant2, evolution-data-server,
libical, libphonenumber, localsearch, mozjs140, re2, samba, webkitgtk). Exactly
one unfixed ordering violation fell out:

    bluez (stage 0) BuildRequires libical-devel from libical (stage 0)

Same stage, so bluez cannot see the factory libical and takes Fedora's, which
requires libicuuc.so.77. Confirmed against run 35413902261, where bluez root
installed libical-0:3.0.20-7.fc44 from fedora and then libicu-0:77.1-2.1.hum1.
The factory own libical is clean -- it built against libicu-78.3-8.hum1 -- it
was simply not available in time, exactly as samba was not available in time
for gvfs and localsearch.

bluez moves to stage 1. Nothing constrains it in either direction: no factory
package BuildRequires bluez, and its only factory build dependencies are
libical and make, both stage 0.

The more useful half is the new IcuLinkerOrderingTests, which derives this
rather than restating it. It scans every recipe for an ICU BuildRequires,
then for every factory consumer of one of those packages asserts the consumer
builds strictly later. Verified it fails against bluez previous stage with the
offending pair named. Not every same-stage pair would be a live bug -- it only
bites when Fedora build of the provider actually requires .so.77 -- but every
such pair is a place where a Fedora ICU can enter a build root unnoticed, and
the fix costs one stage number. Finding the fourth case by hand was not going
to be cheaper than encoding the rule.

319 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Add the cache key for per-package build reuse, and the rule for when to use it

The factory rebuilds about 282 of its 349 packages on every run, because its
only skip mechanism is the atomically-published consumer repository and
publication needs every stage plus precedence plus the Hummingbird-only
transaction. A failing gate therefore freezes the witness, and the witness has
been ghcr.io/projectbluefin/utah-packages:latest @sha256:9cb66729 since
2026-08-30 with 67 of 349 packages in it. webkitgtk, which is not among them,
built 20 times on 19 September at 182 minutes a build, with a recipe unchanged
since 2026-09-05 and an inventory entry byte-identical across every commit since.
Issue #177 has the measurements.

This is the first half of the fix: a key for a cache that answers "have we
already built this exact thing", kept strictly apart from "is the repository
coherent enough to install from". Nothing installs from the cache. A hit is
materialised as the stage's ordinary artifact, so precedence, the transaction
check and publish see an identical repository either way -- the cache changes how
the RPMs were obtained, never what the run validates. That is the property that
makes a hit safe to trust.

Keying on the recipe alone would be wrong, and would reintroduce exactly the
incoherence the publish gate exists to catch. A stage-5 package builds in a root
holding earlier stages fresh output, so the resolved build root is part of the
key, which means the key is computed after dnf has resolved builddep rather than
before: the resolution costs about a minute and the compile costs 182.

The chain only holds across stages if an unchanged recipe yields an unchanged
NEVR, or every root would differ from last run and nothing would ever hit. It
does. 167 of 349 specs use %autorelease, which rpmautospec derives from the
committed packages/<name>/changelog rather than from a build counter, and
libical-3.0.20-1.hum1.bfin came out identical in run 35413902261 and run
35445712318, twelve hours and many commits apart. The changelog is therefore
part of the recipe digest, and there is a test saying so.

Recipes are hashed whole, every file, paths alongside contents so a rename is a
change, for the reason utah's kernel-cache-tag.sh gives: it can only ever
rebuild something that did not need rebuilding, never reuse something stale.

rebuild_plan.cacheable() is the second half: which selected packages may consult
the cache. It narrows nothing -- plan() still decides what to build. It excludes
`changed`, because the author of a recipe edit is owed a real build (the key
would miss anyway, so this is belt and braces), and `stale`, which matters:
a stale published package requires a soname nothing provides any more and its
recipe may not have moved, so its key can hit and hitting would hand back the
very build that is broken. That is the one case where a hit is worse than a
miss, so it is named rather than left to chance.

Verified the key changes on every input -- package, recipe, buildroot digest,
factory digest, resolved root, disttag, schema -- and does not change on
transaction order or duplicates in the root. An empty resolved root is refused
rather than keyed, because a key over nothing collides across genuinely
different roots.

Not in this commit: the workflow wiring that publishes and consults the cache.
That is a separate change and wants this key reviewed first.

19 new tests, 338 in the suite. check_workflow_quoting, validate.py and the
factory contract all clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Tag cache entries <package>-<key>, under a cache image name of its own

Review suggestion, and better than the bare hash I had. A tag of
nautilus-e02585b1... can be listed, filtered and reasoned about in the registry
without a side database, which is what you want when the question is "why did
this miss". A bare hash says nothing.

The prefix adds no uniqueness -- the package name is already inside the hash,
and a test asserts that two packages with otherwise identical inputs get
different keys, so the prefix can never be the only thing separating them. It is
metadata, not identity.

Checked before adopting it, rather than after: all 349 recipe names are valid
OCI tag components, and the longest tag is 64 of the 128 characters
(gnome-shell-extension-gsconnect). But RPM names are laxer than tags -- gtk+ and
libstdc++ are legal packages and illegal tag components -- so cache_tag refuses
a name a tag cannot carry instead of emitting a mangled one. A silently mangled
tag would be a permanent cache miss that reads as a correctness bug. No recipe
needs it today; an import could.

The one thing this must not do is share an image name with utah-packages.
That name is the consumer repository, and prepare resolves utah-packages:latest
and utah-packages:<branch> and treats whatever it gets as a whole repository.
A cache entry holds one package's RPMs. If they shared a tag namespace, a
resolve landing on a cache entry would compute `published` from a single
package -- and publish seeds from that same image, so it would drop the other
66. The docstring says so at the point where someone would be tempted.

--key-only prints the bare key for callers that want it, and still builds the
tag first so the grammar is validated whichever output was asked for.

26 tests, 345 in the suite. One pre-existing test asserted the CLI printed a
bare key; it now asserts the tag, since that is the contract the workflow will
use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* Drop the package-name prefix: the key is the tag

Right, and this reverses the previous commit. The prefix bought nothing the key
does not already carry -- the package name is inside the hash, and a test
asserts two packages with otherwise identical inputs get different keys -- while
introducing a failure mode the bare key cannot have.

That failure mode was mine to begin with. RPM names are laxer than OCI tags:
gtk+ and libstdc++ are legal packages and illegal tag components, so the prefix
needed a grammar check, a refusal path and a subtest over all 349 names, all to
guard a hazard that only existed because of the prefix. A 32-character hex
digest always matches [a-zA-Z0-9_][a-zA-Z0-9._-]{0,127}, so with the key as the
tag there is nothing to validate and nothing an import can break. Verified
across 20000 generated keys, and gtk+ now hashes like any other name.

The legibility argument was the only one for the prefix, and it was aimed at
the wrong place. Browsing a tag list is not how you answer "why did this miss" --
the build log already prints the package and the key it computed, which is what
you actually read. For the registry side, the push step sets labels, which
survive without constraining the tag grammar:

    org.opencontainers.image.title      = <package>
    org.opencontainers.image.version    = <nevr>
    org.opencontainers.image.revision   = <commit>
    org.projectbluefin.factory.disttag  = <.humN.bfin[.N]>

skopeo inspect answers the question, and a tag listing still answers "how many
entries exist", which is the other thing you would want from it.

The note about not sharing an image name with utah-packages stays, because that
hazard is real and independent of the tag shape: prepare resolves
utah-packages:latest and :<branch> and treats what it gets as a whole
repository, and publish seeds from that same image.

22 tests, 341 in the suite. The removed tag tests are replaced by
TagIsTheKeyTests, which pins the property that makes the prefix unnecessary.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

* fix(ci): address package cache review findings

---------

Co-authored-by: Claude <noreply@anthropic.com>
hanthor pushed a commit to hanthor/utah-packages that referenced this pull request Sep 28, 2026
tools/rebuild_plan.py (from projectbluefin#70) reads spec_release() too and treats None as
"no comparable release"; "" made it concatenate a suffix onto nothing and
fail test_an_autorelease_recipe_falls_back_to_the_version. The plan now
also short-circuits on None explicitly instead of relying on BumpError.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
hivecommons-hive Bot added a commit to kylerankin/utah-packages that referenced this pull request Oct 29, 2026
…ries

ffmpeg's recipe was first imported in projectbluefin#70; projectbluefin#414 only added
ffmpegthumbnailer, so keep pr = 414 just for the thumbnailer names.

Hive-Fix-Attempt: 3/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants