Skip to content

fix(ci): skip against the published image, drag dependents, stop building debuginfo - #133

Closed
hanthor wants to merge 16 commits into
mainfrom
claude/great-allen-pt5nwv
Closed

hanthor wants to merge 16 commits into
mainfrom
claude/great-allen-pt5nwv

Conversation

@hanthor

@hanthor hanthor commented Sep 18, 2026 •

Copy link
Copy Markdown
Member

Why every run took 5–8 hours

prepare decides what is already built by reading a repository listing, and every build root enabled the same URL as [factory]. That URL was the GitHub Pages mirror the README says was removed. The deployment was never taken down: it still answers, frozen at 67 packages from before the 343-recipe contract, and nothing publish writes can reach it. Runs 375 and 377 both log:

published repo has 67 source packages
will build 333 of 340 packages

So every push, PR and nightly was a full traversal of all eleven waves — 282 stage-0 jobs, then webkitgtk alone for 5.2 hours at stage 5 with nothing else running, then the tail. Eight hours for a one-line spec fix.

What changes

The image is the witness (rebuild-rpms.yml, build-stage.yml, new .github/actions/load-factory-repo)

  • prepare resolves ghcr.io/<owner>/utah-packages to a digest once — the ref's own tag first, then latest; main reads only latest — extracts /repository from it and runs the plan against that on disk.
  • Every build job and preflight extract the same digest and enable it as file:///work/factory. The plan and the build root read the same bytes; nothing is fetched over HTTP mid-build. vars.FACTORY_REPO is gone — a second source of truth is how the halves first disagreed.
  • No image at all is a warning and an empty witness, which rebuild_plan.py already treats as "skip nothing".

Rebuilt packages drag their published dependents (tools/rebuild_plan.py, tools/rebuild_matrix.py)

  • Once skipping works, a fix to mutter would build mutter alone and leave the published gnome-shell linked against the libmutter it replaced. The plan now reads Provides/Requires from the same primary.xml and rebuilds the transitive runtime closure of anything being rebuilt, in inventory order so waves still hold. The repo carries no SRPMs, so a devel-only BuildRequires is the residual gap.

No debuginfo — the upload already excludes every *-debuginfo-* / *-debugsource-* by pattern, so they were built, stripped and compressed only to be discarded: 25 min of find-debuginfo plus 10 min of compression on webkitgtk alone. Both lanes now pass --define "debug_package %{nil}".

Hygiene

  • precedence and publish require prepare to have succeeded; they used to run after a failed prepare and die on a missing artifact (372, 379).
  • pull_request runs cancel their predecessor on a new push (369, 370, 374 were all cancelled by hand, one four hours in). main and the schedule are untouched.

Also carries the earlier commit on this branch: feat(sources): poll git forges for releases, not just GNOME.

Verification

  • 293 tests pass (python3 -m unittest discover -s tests), including new tests/test_factory_witness.py (workflow wiring) and DependentsTests in tests/test_rebuild_plan.py (closure).
  • Both workflows and the action parse; tools/check_workflow_quoting.py and tools/factory_contract.py pass.
  • Reproduced the root cause locally: rebuild_plan.plan() against the Pages listing with an empty diff still schedules 275 of 340, all "absent from the listing".

Expect one more near-full build: the first run after this reads latest, which is still the ~60-package image from before #70 (run 377 on main is publishing the 333-package one now). After that, a one-recipe PR is a handful of jobs.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

The factory's stated purpose is to track the latest upstream release rather
than whatever Fedora tagged -- docs/targeting-hummingbird.md is explicit that
"a spec is a build recipe, not a release feed". Automation covered 17 of 340
locks, all of them download.gnome.org. 5%.

This adds the next tranche that has a real release feed, taking it to 63:

    download.gnome.org  17  cache.json per-module index   (unchanged)
    github.com          36  tags, or releases where published
    gitlab (3 hosts)    10  repository tags API

The remaining 277 are locked on Fedora's lookaside cache or on a bare
directory listing. A lookaside URL carries its own SHA-512 in the path, so
there is nothing to poll for "latest" -- reaching those needs each project's
real upstream feed identified per package, which is a different job and is
filed separately.

Three shapes are recognised, measured against the real inventory rather than
guessed at: github /archive/<tag>/ (25), github /releases/download/<tag>/
(11), and gitlab /-/archive/ or /-/releases/<tag>/downloads/ (10). Two
details came from reading the locks instead of assuming: evtest is pinned over
http rather than https, and two freedesktop projects use the GitLab release
shape rather than the archive one. Both would have been silent misses -- the
first draft of the pattern found 7 of the 10 GitLab locks and said nothing
about the other three.

Where this differs from the GNOME path, and why:

  Release selection. GNOME has release cycles, so that path only moves inside
  the cycle a lock already names. A forge project has no cycle, so the major
  stands in for it: a newer stable release sharing the current major is an
  "update" and may be applied, one that crosses a major is "review" only and
  is never written. A major can move a soname and break every consumer in the
  graph; no comparison of version strings can make that call.

  URL rewriting. GNOME entries are rebuilt from a known template. A forge URL
  has no template this tool can own, because projects choose their own tag and
  asset names, so forge_planned_entry substitutes the old version for the new
  and the old digest for the new across url, filename, sha256_url and
  fallback_urls. Substituting the bare version fixes a v-prefixed tag in the
  same pass, since "v2.2.1" contains "2.2.1". It refuses outright when the
  current version does not appear in the URL, rather than writing a new
  version beside an old tarball.

  Prereleases. A maintainer's draft or prerelease flag on a GitHub release is
  honoured before the version string is examined, because some prereleases
  carry no alpha or beta suffix at all.

GITHUB_TOKEN is wired into the workflow and is not optional: anonymous GitHub
allows 60 requests an hour against 36 locks, so an unauthenticated run
rate-limits part way down the list and reports the rest as unreachable --
which reads as "nothing to bump". A non-list API response is raised as an
error for the same reason; treating GitHub's rate-limit object as an empty
feed would report every package as current.

The workflow is renamed bump-upstream-sources.yml, since it is no longer only
GNOME. Nothing referenced the old filename.

Verified. 287 tests pass, 20 of them new, all offline through the existing
fake opener. The GitLab path was also exercised against the live API: it
returns 19, 50, 50, 13 and 75 versions for libcamera, wayland-protocols,
shared-mime-info, evtest and accountsservice, and correctly reports each as
already at its newest stable -- so "nothing to bump" there is a real answer
and not an empty list. validate reports 340 source RPMs, the factory contract
is satisfied across 2463 files, and the quoting check clears all 12 workflows.

NOT verified here: the GitHub path against the live API. api.github.com
answers 403 in this environment -- a plain curl does too, so it is the network
and not this code. Its tag, release, draft/prerelease and rate-limit
behaviours are covered by unit tests through the fake opener, and the first
real exercise will be the workflow's own run with a token.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Every run was a full rebuild. prepare decides what is already built by
reading a repository listing, and the URL it read -- the same one every
build root enabled as [factory] -- was the GitHub Pages mirror that README
says was removed. The deployment was never taken down, so it still answers,
frozen at 67 packages from before the 343-recipe contract. Both runs 375
and 377 say it plainly:

    published repo has 67 source packages
    will build 333 of 340 packages

Nothing publish wrote could ever reach that listing, so no run could ever
skip more than seven recipes. Eight hours for a one-line spec fix, and
webkitgtk alone five of them on the serial path at stage 5, with no change
to its recipe.

The image publish writes is the only thing that moves, so it is now the
only thing either half reads:

- prepare resolves ghcr.io/<owner>/utah-packages to a digest once (the ref's
  own tag first, then latest; main reads only latest), extracts /repository
  out of it, and runs the plan against that on disk. No image at all is a
  warning and an empty witness, which rebuild_plan.py already treats as
  "skip nothing".
- every build job, and preflight, extract the same digest with the new
  load-factory-repo action and enable it as file:///work/factory. Same
  bytes as the plan read, and nothing fetched over HTTP mid-build. The
  vars.FACTORY_REPO override is gone: a second source of truth is how the
  two halves first came to disagree.

Skipping correctly makes a second gap load-bearing. A fix to mutter would
build mutter alone and leave the published gnome-shell linked against the
libmutter it just replaced. The plan now reads Provides and Requires out of
the same primary.xml and drags every published dependent of anything being
rebuilt, transitively, in inventory order so the waves still hold. It is
the runtime closure; the repository carries no SRPMs, so a devel-only
BuildRequires is the residual gap.

Also on the critical path: debuginfo. The upload step excludes every
debuginfo and debugsource RPM by pattern, so they were built, stripped and
compressed only to be discarded -- 25 minutes of find-debuginfo and 10 of
compression on webkitgtk alone. Both lanes now build with
debug_package %{nil}.

And two things that made the runs read worse than they were: precedence
and publish ran after a failed prepare and died on a missing artifact
(372, 379), so they now require prepare to have succeeded; and a push to a
pull request left the previous run going for hours (369, 370, 374 were all
cancelled by hand), so pull_request runs now cancel their predecessor.
main and the schedule are untouched.

Verified: 293 tests pass, including new ones for the dependency closure
and for the workflow wiring; both workflows and the action parse; the
quoting check passes. The first run after this lands is still a near-full
build, because latest holds the 60-package image from before #70 -- the
last one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor hanthor changed the title feat(sources): poll git forges for releases, not just GNOME fix(ci): skip against the published image, drag dependents, stop building debuginfo Sep 18, 2026
…n .debug files

Run 380 failed ten stage-0 packages -- enchant2, hidapi, libsoup3, libvorbis,
libxmlb, openjpeg, opus, protobuf3, taglib, wavpack -- all with the same
error at the same step:

    error: Installed (but unpackaged) file(s) found:
       /usr/lib/debug/usr/lib64/libhidapi-hidraw.so.0.15.0-...x86_64.debug
       /usr/src/debug/hidapi-0.15.0-4.hum1.bfin.x86_64/hidapi/hidapi.h

Those ten are exactly the specs that build a MinGW half and say
%{?mingw_debug_package}; the eleventh such spec, abseil-cpp, has its mingw
bcond off and passed. Nothing else in the run failed.

debug_package %{nil} only removes the native -debuginfo subpackage. It does
not stop find-debuginfo, which __spec_install_post runs whenever
__debug_package is set -- and Fedora's %mingw_debug_package sets it itself
(mingw-filesystem, macros.mingw line 20: %global __debug_package 1). So on
these ten the native extraction still ran, wrote .debug files into the
buildroot, and the subpackage that used to claim them was gone.

The fix empties __debug_install_post as well, in both lanes. That stops the
native extraction at its source. The MinGW halves are unaffected: their
.debug files come from the explicit %mingw_debug_install_post call in
%install and go into the mingw32/mingw64 -debuginfo subpackages, which still
exist and which the upload filter drops like every other debuginfo RPM.
Specs that reference __debug_install_post themselves (libcamera, libxcrypt,
grub2) all guard it with %{?__debug_package:...}, which stays unset for them.

Reproduced and fixed locally with a five-line spec that sets __debug_package
the way the mingw macro does: debug_package %{nil} alone dies on the same
"Installed (but unpackaged)" .debug file; with __debug_install_post %{nil}
added it writes the RPM. The workflow test now pins both defines in both
lanes. 300 tests pass; the quoting check clears all 12 workflows.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Run 381 got past every build stage but failed the publish job's
"Validate Hummingbird-only consumer transaction" for two reasons.

First, mesa-dri-drivers links libSPIRV-Tools.so and Hummingbird does
not ship spirv-tools, so dropping the recipe (56b2973) made the runtime
contract unresolvable. The recipe is back, moved to Fedora Rawhide's
2026.3 (SDK 1.4.357.0) because Fedora 44's spirv-headers 1.5.5 carries
GatherModes, which spirv-tools 2026.2 cannot compile against. The
tarball comes from the Fedora lookaside with its sha512 verified; the
two downstream patches no longer apply and are gone with the spec.

Second, several packages in the 67-package witness were built against
sonames the factory no longer provides (libheif -> libavcodec.so.62,
gvfs -> libbluray.so.3, mozc -> libabsl 2601, gnome-shell ->
libedataserver-1.2.so.27). Their current specs already disable those
inputs, but nothing changed in the recipes, so the plan trusted the
published build and skipped them. The dependents-dragging rule cannot
see this because the missing provider is absent from the witness.

rebuild_plan now reads the published repo's Requires and Provides and
marks a source stale when one of its binaries requires something that
neither the published repo nor Hummingbird provides. rebuild_matrix
fetches Hummingbird's primary.xml (baseurl from config/hummingbird.repo)
for the external provides, prints the reason for each stale rebuild,
and skips the rule with a warning if the fetch fails. Stale packages
drag their dependents like any other rebuild.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

Run 381 got all 278 builds green; the failure moved to the publish job's "Validate Hummingbird-only consumer transaction", which is the real gate and had never been reached before. Two causes, both fixed in bf72d53:

  1. libSPIRV-Tools.so missing for mesa-dri-drivers. Dropping spirv-tools earlier in this PR was wrong (Hummingbird doesn't ship it). The recipe is restored at Rawhide's 2026.3 / SDK 1.4.357.0, which compiles against Fedora 44's spirv-headers 1.5.5 (2026.2 fails on GatherModes). Tarball from the Fedora lookaside, sha512 verified.
  2. Stale published builds. libheif (needs libavcodec.so.62), gvfs (libbluray.so.3), mozc (libabsl_*.so.2601) and gnome-shell (libedataserver-1.2.so.27) in the witness were built against sonames the factory no longer provides. Their specs were unchanged, so the plan trusted the published build and skipped them; the dependents rule can't catch this because the missing provider isn't in the witness at all. rebuild_plan now marks a source stale when a published binary Requires something neither the factory nor Hummingbird provides (Hummingbird's primary.xml is fetched via the baseurl in config/hummingbird.repo), and rebuilds it plus its dependents.

Watching the new run; will merge once publish validation passes.


Generated by Claude Code

…ade again

git log on main is a record of fixes that undid earlier fixes: fifteen
commits fighting one ICU split, three digest repins that each died within
days, a recipe dropped and restored, the same libheif bcond added and
retired twice, four soname breaks that were one staging mistake, and two
empty commits pushed to re-run CI.

docs/skills/repeated-mistakes.md names each recurring pattern, the commits
that made and unmade it, and the rule that settled it, with a checklist to
run before pushing a fix. It is indexed in docs/SKILL.md and AGENTS.md so
the router loads it for stage, pin, bcond, workflow and recipe-removal
work.

The triage skill still told readers to repin the build-root digest on exit
125 "manifest unknown"; f1f1e4e showed that cannot work for an image whose
digests live hours, and moved the build root to a once-per-run pull. Both
rows now say so instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Eight recipes imported from Fedora Rawhide dist-git, each pinned by commit
in .hummingbird-upstream.json and source-locked to its upstream release:

  wpa_supplicant 2.11, wireless-regdb 2026.09.03, iw 6.17
    Hummingbird ships NetworkManager-wifi, whose metadata requires
    wireless-regdb and (wpa_supplicant or iwd), and ships none of the
    three, so Wi-Fi cannot be installed on Utah at all (#136, utah#97).
    wireless-regdb in turn requires iw. The wpa_gui bcond is forced off:
    nothing in the contract wants it and the build root has no qt-devel.

  libdaemon 0.14
    Hummingbird's avahi daemon requires libdaemon.so.0 and ships no
    libdaemon, so geoclue retries forever against no daemon (utah#104).

  inih 62 (stage 0), exiv2 0.28.9 (1), libgexiv2 0.16.2 (2), libportal
  0.10.0 (5)
    The factory built nautilus with no way to install it: it needs
    libgexiv2-0.16.so.4, libportal.so.1 and libportal-gtk4.so.1, none in
    any repository (utah#100). exiv2 links inih, gexiv2 links exiv2, and
    libportal links gtk3 and gtk4 (gtk4 is stage 4), hence the stages;
    nautilus itself is at 6. libportal's Qt5 backend is off, since
    Hummingbird has no Qt5 and only the GTK4 backend is consumed.

inih and exiv2 are GitHub archives and gexiv2 is on download.gnome.org;
the bootstrap tool could not fetch them through this environment's proxy,
so their locks are written by hand with the sha512 Fedora's sources file
records, verified against the byte-identical lookaside copy, which is
also the fallback URL, as for the other GitHub-hosted recipes.

The runtime contract now asks for NetworkManager-wifi, wpa_supplicant,
wireless-regdb, avahi and nautilus, so publication proves the three
closures resolve from factory plus Hummingbird before Utah can pin them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 18, 2026

Copy link
Copy Markdown
Member Author

a4c4f19 adds eight recipes so the runtime contract can grow past the desktop stack:

inih, exiv2 and gexiv2 tarballs could not be fetched through this environment's proxy, so their locks are hand-written with Fedora's recorded sha512, verified against the lookaside copy (also the fallback URL). config/runtime-contract.toml now asks for NetworkManager-wifi, wpa_supplicant, wireless-regdb, avahi and nautilus, so publish validation proves those closures before Utah pins the image.


Generated by Claude Code

…d not work

pipewire was the one failure in 333 jobs of the last run, and it failed
the same way it did before b6b5b4a:

    39/52 pipewire:pw-test-endpoint  FAIL  5.01s  killed by signal 14 SIGALRM
    Ok: 51   Fail: 1

b6b5b4a serialized %check and claimed that fixed it. It did not. The log
shows --num-processes 1 in the meson test command line and the identical
failure at the identical 5.01s, so contention was never the cause and the
serialization was only making the longest job in stage 6 slower. It is
reverted here. Reporting that fix without a green job to point at is the
mistake docs/skills/repeated-mistakes.md records as pattern 13, made
again by me.

The real cause is in the test. src/tests/test-endpoint.c:441 arms

    alarm(5); /* watchdog; terminate after 5 seconds */

around setup that costs more than five seconds on this build root, so the
process kills itself; meson never gets to apply a timeout, which is why
--timeout-multiplier is not the lever either. The budget is out of line
with the test's own siblings rather than with anything the test does:
pw-test-stream and pw-test-filter are declared in the same test_apps list
in src/tests/meson.build, with the same environment, and took 12.02s and
18.03s in this same run on this same machine.

So the watchdog is patched to 60 seconds. Nothing is skipped, disabled or
made non-fatal: every assertion still runs and still has to pass. The run
is still bounded, by meson rather than by the alarm -- these tests declare
no timeout: kwarg, and meson's default is 30 seconds (KwargInfo('timeout',
int, default=30) in mesonbuild/interpreter/type_checking.py), so a genuine
hang is killed at 30s, before the new alarm can fire. The alarm goes from
being the tighter guard to being the redundant one.

The patch was applied to the locked 1.6.8 tarball and checked rather than
assumed: patch -p1 applies clean and leaves alarm(60) at line 441.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

The run on a4c4f19 finished: 331 of 333 jobs green, one failure, publish skipped behind it. All eight new recipes built — wpa_supplicant, wireless-regdb, iw, libdaemon, inih, exiv2, libgexiv2 and libportal are all in the green set, as is spirv-tools at 2026.3 and every package the stale-published rule dragged in.

The single failure is pipewire, and it is the same one b6b5b4a claimed to have fixed:

39/52 pipewire:pw-test-endpoint  FAIL  5.01s  killed by signal 14 SIGALRM
Ok: 51   Fail: 1

That commit serialized %check and said contention was starving the test's watchdog. The log disproves it: --num-processes 1 is in the meson command line and the failure is identical, to the hundredth of a second. Reporting a fix without a green job is pattern 13 in docs/skills/repeated-mistakes.md, and I wrote both the pattern and the mistake.

dae6241 does it properly. src/tests/test-endpoint.c:441 arms alarm(5) around setup that costs more than five seconds here, so the process kills itself before meson can apply any timeout. The budget is out of line with the test's own siblings, not with the machine: pw-test-stream and pw-test-filter are in the same test_apps list with the same environment and took 12.02s and 18.03s in this very run. A downstream patch raises the watchdog to 60s, and the serialization is reverted since it did nothing but slow the longest job in stage 6.

Nothing is skipped or made non-fatal, and the run is still bounded: these tests declare no timeout: kwarg, so meson's default of 30 seconds (KwargInfo('timeout', int, default=30)) kills a genuine hang before the new alarm can fire. The alarm goes from being the tighter guard to the redundant one. The patch was applied to the locked 1.6.8 tarball and verified, not assumed.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head dae6241; ran the suite locally — 305 tests pass)

1. publish seeds from :latest while prepare skips against the branch tag — a PR's second push can publish a branch image that lost packages the plan just proved "already published". (medium-high)

prepare resolves the skip witness as $branch first, then latest (.github/workflows/rebuild-rpms.yml:190-192), but the publish seed is still hardcoded to ghcr.io/<owner>/utah-packages:latest (.github/workflows/rebuild-rpms.yml:649). On a PR's second push, anything skipped because the branch tag carries it — but latest does not — is absent from both the seed and the run's artifacts, so the republished branch image drops it (or regresses it to the stale latest copy for packages the first push rebuilt as stale). The body's own transition state makes this concrete: while latest is still the ~67-package image, a first push builds ~340 into <branch>, and a second push then skips ~everything unchanged and seeds from the 67-package latest — the Hummingbird-only transaction check fails, or worse passes with a silently regressed branch image. This is the same two-sources-of-truth disagreement the PR fixes elsewhere; the seed should read needs.prepare.outputs.factory_image (the digest the plan witnessed), not latest.

Note: changed (diff vs. PR base) does shield the PR's own modified/added recipes across pushes, so the loss is confined to packages the first push built for other reasons — stale rebuilds, and everything built while latest lags the branch tag. That is still exactly the "second push skips what the first push built" scenario the body advertises.

2. Body says "main and the schedule are untouched" by the concurrency change — the group applies to them too. (low)

The concurrency.group includes all refs (.github/workflows/rebuild-rpms.yml:67-68); only cancel-in-progress is PR-conditional. On main/schedule, new runs now queue behind an in-flight run instead of running concurrently (the header comment at lines 58-60 documents concurrent waves as safe), and GitHub keeps only one pending run per group, so an intermediate queued main run is superseded. The newest run still lands, so the end state is correct — but this is a behavior change on main, contrary to the body and the adjacent comment. If serializing main is intended, say so in the comment; if not, scope the group key to PRs.

Scope / limits of this review: the branch also carries ~4,400 lines of imported package content (wpa_supplicant, exiv2, libdaemon, wireless-regdb, spirv-tools, etc. — commits a4c4f19, bf72d53, dae6241) that the PR body does not describe; I verified they don't touch the CI logic but did not review the specs/patches themselves for correctness.

Otherwise the core logic checks out: dependents_from_primary/reverse_closure/stale_from_primary (tools/rebuild_plan.py:86-216) handle self-edges, rich deps, and rpmlib caps correctly; the file:///work/factory vs. host-path split is handled at each consumer (rebuild-rpms.yml prepare step, preflight's docker -e FACTORY_REPO with /work mounted, build-stage.yml:91); prepare and publish sanitize the branch tag identically; and the new closure/witness behavior is under test.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Finding 1 is right and it is the same bug this PR exists to fix, one layer down. I am not pushing the fix yet, for a reason I want to state plainly rather than have look like inaction.

The seed disagrees with the witness. prepare resolves the skip witness as $branch then latest (rebuild-rpms.yml:190-192), while publish seeds from a hardcoded ghcr.io/<owner>/utah-packages:latest (rebuild-rpms.yml:649). So on a second push, anything skipped because the branch tag carries it is in neither the seed nor the run's artifacts, and the republished branch image loses it. That is two sources of truth for "what is already built" — exactly what vars.FACTORY_REPO was removed for in this same diff, reintroduced at the other end of the pipeline. The fix is the one you name: seed from needs.prepare.outputs.factory_image, the digest the plan actually witnessed.

Your note that changed shields the PR's own modified recipes is right too, and it is why this has not bitten yet — the exposure is packages the first push rebuilt for other reasons, which is precisely what the stale-rebuild rule in this PR creates more of.

Why I am not pushing it right now: run 35413902261 is six hours into a near-full 329-package build and is currently inside webkitgtk, the five-hour job. cancel-in-progress is true for pull requests, so a push cancels it. That run is also the only thing that will tell me whether the pw-test-endpoint watchdog patch actually works — I claimed that fix once before without a green job behind it, and I am not doing it twice. I would rather spend the two hours than throw the evidence away and start another near-full build, since latest is still the pre-#70 image and the next run would be just as long.

So: the seed fix goes in as soon as that run reports, and before this merges either way. If publish produces a regressed branch image in the meantime, that is contained to the branch tag and the same push fixes it.

Finding 2 is also correct and the body is wrong. The concurrency.group covers every ref; only cancel-in-progress is PR-conditional. So main and the schedule now queue behind an in-flight run rather than running concurrently, and GitHub drops superseded pending runs — the end state is right, but it is a behaviour change, and the adjacent comment says concurrent waves are safe. I will scope the group key to pull requests rather than serialize main, since nothing here argues for serializing it, and correct the body.

On scope: you are right that the ~4,400 lines of imported package content are not described in the body. They are the eight Wi-Fi-closure recipes from a4c4f191 plus libdaemon; I will split the description so a reviewer can see which commits are CI and which are recipes.


Generated by Claude Code

hanthor and others added 2 commits September 19, 2026 11:37
Review finding on dae6241, and it is this change's own bug one layer down.

prepare resolves the skip witness as the branch tag first and latest second.
publish seeded from a hardcoded ghcr.io/<owner>/utah-packages:latest. So the
two disagreed about what is already built: on a second push to a pull request,
anything skipped because the *branch* tag carried it was in neither the seed
nor that run's artifacts, and the republished branch image simply lost it --
or regressed to the stale latest copy for packages the first push had rebuilt.

That is exactly the failure this change removed vars.FACTORY_REPO to prevent,
reintroduced at the far end of the same pipeline. publish already needs
prepare, so its resolved digest was there to use all along.

The empty case is handled rather than assumed: when no image is published at
all, prepare emits an empty witness and warns, and the seed now says so and
publishes a new repository instead of trying to pull "".

The concurrency group is scoped to pull requests too. The group covered every
ref while only cancel-in-progress was conditional, so main and schedule runs
queued behind one another and GitHub could drop a superseded pending run --
a behaviour change the block's own comment says is not wanted, since
concurrent waves are safe by design.

tests/test_factory_witness.py asserts the seed reads prepare's output and that
no workflow names the latest tag directly. Reverting the seed to :latest fails
it, so it is a gate and not a restatement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
The publish gate on run 35413902261 refused to publish, correctly. All 331
build jobs passed; the Hummingbird-only consumer transaction then failed:

    Problem 2: package nautilus-gsconnect ... requires
      gnome-shell-extension-gsconnect = 72-3.hum1.bfin, but none of the
      providers can be installed
      - package gnome-shell-extension-gsconnect ... requires /usr/bin/ffmpeg,
        but none of the providers can be installed
      - nothing provides libplacebo.so.360()(64bit) needed by
        ffmpeg-free-9.0.1-1.hum1.bfin.x86_64 from factory

libplacebo exists in neither runtime repository. Hummingbird does not package
it -- I read its primary.xml and there is no libplacebo of any version among
the 20050 packages -- and the factory has no recipe. So the BuildRequires was
satisfied by Fedora 44 in the build root alone, and the built ffmpeg-free
carried a runtime dependency nothing could ever answer. That is precisely the
build-only-becomes-runtime failure this factory exists to prevent, and it took
gsconnect down with it because that needs /usr/bin/ffmpeg.

Two ways out: add libplacebo to the factory, or stop linking it. libplacebo
would need glslang, shaderc and spirv-headers first, none of which the factory
builds, so that is four new recipes for Vulkan-backed libavfilter paths
(placebo scaling and tone mapping) that nothing in Utah's desktop asks for
yet. Turning the bcond off is one line and makes ffmpeg-free installable now;
the comment records what it costs and what re-enabling needs.

Release goes to 2 so the rebuilt RPM is not the same NVR with different
dependencies.

This does not address the other half of the gate failure -- nautilus requiring
libicuuc.so.77 against a runtime that excludes libicu 77. That one needs a
decision rather than a patch, and I have written the evidence up on the PR.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Run 35413902261 finished. All 331 build jobs passed — I enumerated every one rather than inferring from stage order — precedence passed, and then publish refused at step 6, the Hummingbird-only consumer transaction. That is the gate doing exactly its job: nothing was published, and the two problems it caught are real.

Also, for the record on the thing I owed: pipewire built green (7m56s, 24 RPMs), with %check unconditional in the spec. The watchdog patch works.

Problem 2: ffmpeg — fixed in 13e52bc

nothing provides libplacebo.so.360()(64bit) needed by
ffmpeg-free-9.0.1-1.hum1.bfin.x86_64 from factory

libplacebo is in neither runtime repository. I read Hummingbird's primary.xml: no libplacebo of any version among its 20050 packages, and the factory has no recipe. So the BuildRequires: pkgconfig(libplacebo) was satisfied by Fedora 44 in the build root alone, and the built RPM carried a runtime dependency nothing could ever answer — the build-only-becomes-runtime failure this factory exists to prevent. It took gnome-shell-extension-gsconnect with it, which needs /usr/bin/ffmpeg.

Adding libplacebo would mean glslang, shaderc and spirv-headers first — four recipes, for Vulkan-backed libavfilter paths nothing in Utah asks for. So %bcond placebo 0, Release bumped to 2, with a comment recording the cost and what re-enabling needs.

Problem 1: nautilus and ICU — evidence, not a patch

package nautilus-51~beta-1.hum1.bfin.x86_64 from factory requires
  libicuuc.so.77()(64bit), but none of the providers can be installed
  - package libicu-77.1-2.1.hum1.x86_64 from public-hummingbird-x86_64-rpms
    is filtered out by exclude filtering

This is the ICU story in docs/skills/repeated-mistakes.md, and I do not want to reopen fifteen commits' worth of it by guessing. What the repodata actually says:

  • Hummingbird ships both libicu 77.1 and 78.3 (and libicu-devel for both).
  • They declare no Conflicts and no Obsoletes, and provide disjoint sonames — but they share a package name, so dnf installs exactly one. That is why the original split had "no resolution": not a declared conflict, just one-version-per-name.
  • 28 Hummingbird packages require libicuuc.so.77, including every boost-* and harfbuzz-icu. libtinysparql requires .so.78.

So Hummingbird is itself split across both ICUs, and the runtime exclusion of 77 is only tenable while the contract pulls nothing from that set of 28. nautilus is the first package to break that: something in its build root — harfbuzz-icu or a boost-*, both of which need .so.77 — pulled libicu 77, which made libicu-devel resolve to 77, so nautilus compiled against ICU 77 headers and now requires .so.77 at runtime.

Two candidate fixes, both with consequences I would rather you weighed:

  1. Force the build root to ICU 78 by excluding libicu-77.*-*hum1 from Hummingbird in build-stage.yml as well. Then nautilus links 78 and matches the runtime. The risk is that any build root genuinely needing harfbuzz-icu or boost-* becomes unresolvable — and 012cb6a already reverted a blanket exclusion once for a related reason, though that one also hit Fedora, which HB_EXCLUDE now excludes anyway.
  2. Drop the runtime exclusion and let the contract decide. This only works if nothing in the contract needs .so.78 at the same time — libtinysparql does, and localsearch is in the image, so I expect this one fails.

My reading is that (1) is the coherent direction and (2) reproduces the original split, but the buildroot blast radius of (1) is exactly what bit before, so it is your call rather than mine.

Also pushed: the seed fix from your review

76bf7ad seeds publish from needs.prepare.outputs.factory_image instead of a hardcoded :latest, handles the empty-witness case, scopes concurrency.group to pull requests, and adds a test that fails if the seed goes back to :latest. I held it until the run finished so a push would not cancel six hours of building.

Ran 306 tests ... OK and factory contract satisfied (2532 tracked files checked) at 13e52bc.


Generated by Claude Code

The other half of the publish failure on run 35413902261. All 331 builds
passed; the Hummingbird-only transaction then refused:

    package nautilus-51~beta-1.hum1.bfin from factory requires
      libicuuc.so.77, but none of the providers can be installed
      - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
        filtered out by exclude filtering

I read Hummingbird's primary.xml rather than reasoning from the comment
history, and it settles the question. Hummingbird has migrated to ICU 78:

    harfbuzz-icu 14.1.0-1.1 .. 14.2.0-1   libicuuc.so.77
    harfbuzz-icu 14.2.1-3   .. 14.3.1-1   libicuuc.so.78
    boost-locale 1.90.0-6.1, -7           libicuuc.so.77
    boost-locale 1.90.0-9 .. -13          libicuuc.so.78
    libtinysparql 3.11.1-2                libicuuc.so.78

Every current build links 78; only superseded ones link 77. So my earlier note
on the PR -- that Hummingbird is itself split across both ICUs -- was wrong: I
had counted superseded builds alongside current ones. The consumer exclusion of
libicu 77 is therefore correct, and nothing current needs 77.

What went wrong is that the build root never made the same choice. The two
libicu builds share a package name, so dnf installs exactly one and nothing
said which. A root that resolved a superseded copy pulled libicu 77;
libicu-devel followed it to 77.1 to match; and the package built there linked
ICU 77, which its consumers then refuse. Two halves of one decision, disagreeing
-- pattern 2 in docs/skills/repeated-mistakes.md.

So HB_REPO_EXCLUDE gains the same exclusion, in the same spelling, with release
as its own field because dnf splits a spec on dashes before globbing (1f88491).

This is not the blanket exclusion 012cb6a reverted. That one reached Fedora,
whose build-only deps do legitimately link ICU 77 -- raptor2, localsearch and
pipewire-libs-extra transitively -- and it is scoped to the Hummingbird
repository here. HB_EXCLUDE already keeps Fedora ICU out of the build root
entirely, and the rendered mock config shows both halves of that:

    excludepkgs=...,libicu,icu,...                    (fedora, fedora-updates)
    excludepkgs=ruby3.3-...,ruby3.4-...,libicu-77.*-*hum1   (hummingbird)

tools/mock_config.py carries it too, because tests/test_mock_config.py already
asserts the mock root and the container root encode one policy -- that test
caught this change when only the workflow had it, which is the guard working.
New IcuAgreementTests then asserts the build root and the consumer transaction
name the same ICU, so they cannot drift apart again.

309 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Correcting myself on the ICU half, and fixing it in 67150e9 — I read Hummingbird's primary.xml instead of reasoning from the comment history, and it answers the question rather than leaving it a judgement call.

I was wrong that Hummingbird is split across both ICUs. I had counted superseded builds alongside current ones. Per build:

harfbuzz-icu 14.1.0-1.1 .. 14.2.0-1    libicuuc.so.77
harfbuzz-icu 14.2.1-3   .. 14.3.1-1    libicuuc.so.78
boost-locale 1.90.0-6.1, -7            libicuuc.so.77
boost-locale 1.90.0-9   .. -13         libicuuc.so.78
libtinysparql 3.11.1-2                 libicuuc.so.78

Hummingbird has migrated to ICU 78. Every current build links 78; only superseded ones link 77. Checked across the whole repository: taking the newest build of each of its 20050 packages, zero require an ICU 77 soname.

That makes the consumer exclusion of libicu 77 correct, and it makes option 1 from my previous comment safe — the blast radius I warned about is empty. So the fix is the one this factory's own rules point at: the build root must make the same choice the consumer transaction does.

Both libicu builds share a package name, so dnf installs exactly one and nothing said which. A root that resolved a superseded copy pulled libicu 77, libicu-devel followed it to 77.1 to match, and the package built there linked ICU 77 — which its consumers then refuse. Two halves of one decision, disagreeing: pattern 2 in docs/skills/repeated-mistakes.md.

HB_REPO_EXCLUDE now carries libicu-77.*-*hum1, release as its own field per 1f88491. This is not the blanket exclusion 012cb6a reverted — that one reached Fedora, whose build-only deps legitimately link ICU 77, and this is scoped to the Hummingbird repository. HB_EXCLUDE already keeps Fedora's ICU out of the build root entirely, which the rendered mock config shows in one view:

excludepkgs=...,libicu,icu,...                          (fedora, fedora-updates)
excludepkgs=ruby3.3-…,ruby3.4-…,libicu-77.*-*hum1       (hummingbird)

Two things about how this landed that are worth recording:

  • tests/test_mock_config.py caught me. It asserts the mock root and the container root encode one policy, and it failed the moment only the workflow had the exclusion. That is the guard doing its job on the person adding a policy, which is the only time it matters.
  • New IcuAgreementTests asserts the build root and the consumer transaction name the same ICU in the same spelling, so these two cannot drift apart again — which is how this bug existed in the first place.

309 tests pass. Both halves of the publish failure now have fixes on the branch, and the run from these pushes is the test of them.


Generated by Claude Code

Third place the same decision lives, and the one that would have let this
failure repeat even with the other two fixed.

rebuild_matrix builds `external` from every capability Hummingbird provides,
libicu 77 included, and stale_from_primary asks whether a published binary's
Requires are satisfied by factory plus that set. So a published package linked
against libicuuc.so.77 looks satisfiable, is skipped as fresh, and then fails
the consumer transaction -- which excludes libicu 77 -- at publish. The model
that exists to predict that transaction disagreed with it.

Run 35413902261 is what that costs: 331 green builds, precedence green, then
publish refused, and nothing published.

provides_from_primary now takes the builds the transaction refuses and skips
their capabilities, driven by one module constant:

    EXCLUDED_EXTERNAL = (("libicu", "77."),)

A package linked against the refused build is then reported stale, which makes
it rebuild against the current root -- the only cure -- rather than being
carried forward to fail the gate.

The tests use a primary.xml shaped like the real thing, with libicu 77.1 and
78.3 under one name, and assert all three directions: 77 provides nothing, a
published package needing .so.77 is stale, and one needing .so.78 is not, so
the rule does not condemn everything that touches ICU. Emptying
EXCLUDED_EXTERNAL fails three of them.

313 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Found a third place the ICU decision lives, and it is the one that would have let this failure repeat even with the other two fixed. 0d0bec3.

rebuild_matrix builds external from every capability Hummingbird provides — libicu 77 included — and stale_from_primary then asks whether a published binary's Requires are satisfied by factory plus that set. So a published package linked against libicuuc.so.77 looks satisfiable, gets skipped as fresh, and fails the consumer transaction at publish. The model whose entire job is to predict that transaction disagreed with it.

That matters beyond nautilus. nautilus was rebuilt this run, so the build-root fix reaches it. Any other published package already linked against ICU 77 would be skipped, carried forward untouched, and would fail the gate again — with the fix in place and no indication why.

provides_from_primary now skips the capabilities of builds the transaction refuses, from one constant:

EXCLUDED_EXTERNAL = (("libicu", "77."),)

A published package linked against the refused build is then reported stale, which makes it rebuild against the current root — which the docstring already says is the only cure.

The tests use a primary.xml shaped like the real repository, libicu 77.1 and 78.3 under one name, and assert all three directions: the refused build provides nothing, a published package needing .so.77 is stale, and one needing .so.78 is not — so the rule does not condemn everything that touches ICU. Emptying EXCLUDED_EXTERNAL fails three of them.

So the one decision now holds in all three places that act on it, each with a test:

where what it does guarded by
build root (build-stage.yml, tools/mock_config.py) builds link ICU 78 IcuAgreementTests, test_mock_config.py
staleness model (tools/rebuild_plan.py) an ICU-77 published build is stale, so it rebuilds ExcludedExternalTests
publish gate (rebuild-rpms.yml) refuses ICU 77 outright unchanged, and now agreed with

313 tests pass; factory contract satisfied (2532 tracked files checked).


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (read at head 67150e9; the dispatched SHA 13e52bc has been superseded).

No blockers. The witness/seed invariant holds: publish seeds from the exact digest prepare skipped against (.github/workflows/rebuild-rpms.yml:665), and both prepare and publish sanitize the tag from github.ref_name identically (rebuild-rpms.yml:193, rebuild-rpms.yml:792), so a skipped∪built set is always complete. Verified config/hummingbird.repo:8 has a literal baseurl (no $basearch), so hummingbird_baseurl (tools/rebuild_matrix.py:110) works against the live repomd.

Two low-severity findings:

  1. tests/test_factory_witness.py:99 — the if __name__ == "__main__": unittest.main() block sits before IcuAgreementTests (line 102). Running the file directly exits inside unittest.main() before that class is defined, so its three tests silently never run in that invocation mode (discover mode is unaffected). Move the guard to the end of the file.

  2. tools/rebuild_plan.py:181 (stale_from_primary) judges only against factory + Hummingbird provides, but the build root also carries Fedora repos (build-stage.yml). A published package that legitimately links a Fedora-only runtime lib will be marked stale on every run, rebuilt, and come out requiring the same capability — a permanent skip-defeat for that package, silently. Worth a log line or a known-external allowance if any such package exists.

Also noting (name-only matching): stale_from_primary/dependents_from_primary ignore version constraints on Requires, so libfoo >= 2.0 counts as satisfied by any provider version — under-detection, conservative direction, fine as-is.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 0d0bec3)

High — the forge auth header is a committed redaction artifact, and it breaks GitHub polling in CI. tools/upstream_bump.py:237:

headers["Authorization"] = f"******"

That is an f-string with no placeholder — the literal string ****** is sent as the Authorization header. It should be f"Bearer {token}". This looks like a secret-masking pass rewrote the source before it was committed.

Consequences: bump-upstream-sources.yml always supplies GITHUB_TOKEN to both invocations (lines 66, 73), so token is truthy in CI and every GitHub API request goes out as Authorization: ******. GitHub returns 401 Bad credentials for a malformed header — it does not fall back to anonymous — so all 36 GitHub-hosted locks land in the except branch of forge_proposal (tools/upstream_bump.py:585-588) and are reported as unreachable. The exact failure mode the docstring at tools/upstream_bump.py:230-234 says the token exists to prevent ("reads as nothing to bump") is what this ships. The 293 passing tests don't catch it because every forge test injects opener and none inspects the Request headers (tests/test_upstream_bump.py, ForgeVersionsTests).

Everything else checked holds up:

  • plan() dependents drag + stale rules (tools/rebuild_plan.py:328-364) preserve inventory order, handle the empty-witness and dropped-recipe cases, and are well tested including the icu-77 exclusion regression.
  • The host/container file:// split is handled correctly — prepare rewrites to file://$PWD/work/factory on the host (rebuild-rpms.yml:225-231) while build jobs get file:///work/factory matching the /work mount (load-factory-repo/action.yml:73-76).
  • The branch-tag sanitization in prepare's resolve step matches publish's tag derivation, and publish now seeds from needs.prepare.outputs.factory_image (rebuild-rpms.yml:661), closing the latest-vs-branch seed disagreement flagged at the previous revision.

Not verifiable from the diff: the claim that __debug_install_post %{nil} leaves the ten MinGW specs' own debuginfo intact rests on run 380; I could not independently confirm it.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

… .so.77

I added libicu-77.*-*hum1 to HB_REPO_EXCLUDE two commits ago and it made the
factory strictly worse. bluez failed at stage 0 of run 35443117478:

    package libical-3.0.20-7.fc44 from fedora requires libicuuc.so.77, but
    none of the providers can be installed
    - package libicu-77.1-2.fc44 from fedora is filtered out by exclude filtering
    - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
      filtered out by exclude filtering

The reasoning that produced it was checked against the wrong population. I read
Hummingbird primary.xml, found that every current Hummingbird build links
libicuuc.so.78 and only superseded ones link .so.77, and concluded that
excluding 77 could not strand anything. That conclusion is sound about
Hummingbird and says nothing about the build root, which also carries Fedora
binaries Hummingbird has never rebuilt. libical is one of them and requires
.so.77 outright.

My own commit message contained the refutation: HB_EXCLUDE already keeps Fedora
libicu out of the root, so Hummingbird superseded libicu-77 was the only
remaining provider of .so.77. Removing it left none. I quoted that line as
evidence the exclusion was safe when it is the reason it was not. 012cb6a
reverted a blanket libicu exclusion for this same underlying reason, and
scoping it to the Hummingbird repository did not change the reason -- it only
changed which repository the last provider happened to live in.

Trading a publish-gate failure for a stage-0 failure is a bad trade: the gate
failure came after 331 successful builds and named one package, this one loses
every build behind it. So the build-root half comes out, in the workflow and in
tools/mock_config.py, and both places now carry a note saying why it must not
come back.

The publish-side half stays. EXCLUDED_EXTERNAL in tools/rebuild_plan.py and the
consumer exclusion are about what may be installed on a Hummingbird system,
where nothing current links ICU 77; the build root is a different question with
a different answer. IcuAgreementTests asserted the two halves were symmetric;
it now asserts the asymmetry and says why, so the next person to notice that
Hummingbird has migrated to 78 does not make this change again.

The underlying problem is unsolved and stated as such in both comments: a
factory package can still link ICU 77 because Fedora build deps pull it into
the root, and the consumer transaction then refuses that package. It has to be
fixed without removing the .so.77 provider the root legitimately needs.

315 tests pass. check_workflow_quoting.py clean; the new comment contains no
apostrophe.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

I have to correct the ICU change I posted three comments ago, and reverse half of it.

I said the one ICU decision now held in all three places and showed a table to that effect. The build-root half of it was wrong, and it made the factory worse rather than better. bluez failed at stage 0 of run 35443117478:

package libical-3.0.20-7.fc44 from fedora requires libicuuc.so.77, but none
of the providers can be installed
 - package libicu-77.1-2.fc44 from fedora is filtered out by exclude filtering
 - package libicu-77.1-2.1.hum1 from public-hummingbird-x86_64-rpms is
   filtered out by exclude filtering

What I got wrong. I read Hummingbird's primary.xml and established that every current Hummingbird build links libicuuc.so.78, and only superseded builds link .so.77. That is correct, and I checked it carefully — but it is a fact about Hummingbird, and I used it to justify a claim about the build root. The build root is not only Hummingbird. It also carries Fedora binaries Hummingbird has never rebuilt, and libical-3.0.20-7.fc44 requires libicuuc.so.77 outright. I checked the newest-per-name population in Hummingbird and never checked the Fedora side of the same root.

The sharper version: my own commit message contains the refutation. I wrote

HB_EXCLUDE already keeps Fedora ICU out of the build root entirely

and offered it as evidence the exclusion was safely scoped. It is the exact reason it was not. With Fedora's libicu already excluded so Hummingbird wins the name, Hummingbird's superseded libicu-77 was the last remaining provider of .so.77 anywhere in the root. Excluding it left none. I quoted the sentence that disproved my change as support for it.

012cb6a reverted a blanket libicu exclusion for this same underlying reason. Scoping it to the Hummingbird repository did not change the reason — it only changed which repository the last provider happened to live in.

What I did. Reverted the build-root half in de4ac4d, in both .github/workflows/build-stage.yml and tools/mock_config.py. Trading a publish-gate failure for a stage-0 failure is a bad trade: the gate failure came after all 331 builds passed and named one package; this one loses every build behind it.

The publish-side half stays. EXCLUDED_EXTERNAL in tools/rebuild_plan.py and the consumer exclusion are about what may be installed on a Hummingbird system, where nothing current links ICU 77. The build root is a different question with a different answer, and the two are now deliberately asymmetric.

IcuAgreementTests asserted the symmetry. It now asserts the asymmetry, with the reason in the docstring, plus a check that the mock root agrees — so the next person who notices Hummingbird has migrated to 78 does not make this change again. Both code comments say explicitly that libicu-77.*-*hum1 does not belong in the build-root exclusion and what happens if it is added.

What is still open. The original problem is unsolved, and I have marked it as such in both comments rather than leaving a fixed-looking gap: a factory package can still link ICU 77, because Fedora build deps pull libicu-77 into the root and libicu-devel follows it to match, and the Hummingbird-only consumer transaction then refuses the result — which is how nautilus-51~beta-1.hum1.bfin failed publication in run 35413902261. It needs a fix that does not remove the .so.77 provider the root legitimately needs. The obvious candidate is constraining libicu-devel rather than libicu, so the headers a package compiles against are always 78 while the runtime 77 stays installable for Fedora's sake; I have not verified that dnf will hold libicu-devel-78 beside libicu-77, and I am not shipping it untested on the back of this.

315 tests pass. check_workflow_quoting.py clean, and the new comment contains no apostrophe.


Generated by Claude Code

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Follow-up on the open ICU problem, so the next attempt does not have to rediscover this. Two candidate fixes, both ruled out by evidence rather than by argument.

Constraining libicu-devel instead of libicu does not work. The idea was to let the runtime ICU 77 stay installable for Fedora's sake while forcing the headers a package compiles against to be 78. packages/icu/icu.spec settles it:

%package  -n lib%{name}-devel
Requires: lib%{name}%{?_isa} = %{version}-%{release}

The devel subpackage requires an exact version-release match on the base, so libicu-devel-78 cannot be held beside libicu-77. dnf will resolve libicu to one build and the devel package follows it or the transaction fails. There is no split to exploit.

Fixing it by stage ordering is not a small change either. bluez declares BuildRequires: libical-devel and both are stage 0, so they build in parallel and bluez resolves libical from Fedora — which is where the .so.77 requirement enters. That looks like a staging bug until you count: 295 of 349 packages are stage 0, and there are 424 BuildRequires edges where the provider's stage is not strictly lower than the consumer's, across 207 distinct packages (SDL3→mesa, colord-gtk→colord, accountsservice→libxcrypt, and so on).

That is not 424 bugs. It means stage 0 is not "has no factory dependencies" — it is "builds correctly against Fedora plus Hummingbird alone", and the stage numbers are a deliberate exception mechanism for the cases where the distro version is not good enough. Reordering stages to make the factory build against its own output throughout is a different architecture, not a fix, and I am not going to propose it off the back of one ICU failure.

Where the lever actually is. The factory already builds its own libical — run 35413902261 produced libical-3.0.20-1.hum1.bfin. If that build links ICU 78, then consumers pulling libical from the factory stop requiring .so.77, and the contamination chain is cut at its source without excluding anything. The question I could not answer from the logs is which libicu NVR was resolved into libical's own build root, because the dnf transaction detail is not in the retained output. Adding a line that records the resolved libicu NVR per build root would make this answerable directly rather than by inference — that seems worth doing regardless of how the fix lands, and it is cheap.

Nothing here changes the revert in de4ac4d, which stands on its own: the build-root exclusion traded one gate failure for losing every build behind it.


Generated by Claude Code

This is the actual cause of the publish failure the ICU exclusions were
flailing at, and it is an ordering bug, not an exclusion one.

Hummingbird ships libicu 77.1 beside 78.3 under one name, so a build root
installs exactly one. dnf takes the newest unless something in the root
requires libicuuc.so.77, and then it downgrades -- libicu-devel follows the
base to match, because icu.spec pins it exactly:

    Requires: lib%{name}%{?_isa} = %{version}-%{release}

so there is no devel-only split to exploit, which was the other fix I
considered. Whatever builds in that root then links ICU 77, and the
Hummingbird-only consumer transaction refuses it.

The chain in run 35413902261 runs entirely through our own staging:

  Fedora samba-core-libs-4.24.6-1.fc44 requires libicuuc.so.77
    -> localsearch (stage 4) resolved samba from Fedora, because the factory
       samba was stage 6 and did not exist yet. Its root installed
       libicu-77.1-2.1.hum1 over five available 78.3 builds, including our own
       libicu-78.3-1.hum1.bfin, and localsearch-3.12~beta-1.hum1.bfin came out
       requiring .so.77.
    -> nautilus (stage 6) requires localsearch, so its root downgraded too, and
       nautilus-51~beta-1.hum1.bfin required .so.77.
    -> publish step 6 refused nautilus. All 331 builds had passed.

The factory samba is not the problem: it built against libicu-78.3-1.hum1.bfin
with zero .so.77 anywhere in its root. Its own entry says it was rebuilt
precisely to stop requiring .so.77. It was simply built four stages too late to
be used by the packages that needed it.

samba moves to stage 2 and libtevent, its only factory build dependency above
stage 0, to stage 1. Both are safe by their own dependencies: libtevent is
built from libtalloc and make, both stage 0; samba from libtevent plus
libtalloc, libtdb, icu and libxcrypt, all stage 0 or now 1. Nothing moves
later, so no existing consumer ordering can break -- pulling a provider earlier
only ever adds packages to the roots that follow it.

tests/test_icu_staging.py makes this a gate. It asserts samba precedes both
known consumers and that samba still follows everything it is built from, and
it fails against the previous stage numbers. The docstring carries the whole
chain, because the next person to meet an ICU 77 requirement will otherwise
reach for an exclusion again -- which is what de4ac4d had to revert, since
Fedora packages the root legitimately needs (libical) require .so.77 and
excluding it strands them outright.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Found it. The ICU problem is an ordering bug, not an exclusion one, and the whole chain runs through our own staging. Fixed in bcd5e33.

I traced it by reading the build roots of run 35413902261 rather than reasoning about it, which is what I should have done before writing either exclusion.

The chain.

nautilus (stage 6) resolved ICU 77. Its root shows dnf choosing libicu-77.1-2.1.hum1 over five available 78.3 builds, including our own libicu-78.3-1.hum1.bfin from the stages repo at priority 1. Something forced the downgrade:

- package localsearch-3.12~beta-1.hum1.bfin.x86_64 from stages requires
  libicui18n.so.77()(64bit), but none of the providers can be installed

So the requirement came from a factory-built package. Going up one level, into localsearch's own root (stage 4):

- package samba-core-libs-2:4.24.6-1.fc44.x86_64 from updates requires
  libicuuc.so.77()(64bit), but none of the providers can be installed

Fedora's samba. localsearch is stage 4 and the factory's samba is stage 6, so when localsearch built, our samba did not exist yet and its root took Fedora's — which requires .so.77. libicu downgraded to 77, libicu-devel followed the base to match, localsearch linked ICU 77, and then every later root that depends on localsearch inherited it. nautilus at stage 6 was one of them, and publish step 6 refused it.

The factory's samba is not the problem. Its root installed libicu-78.3-1.hum1.bfin with zero occurrences of .so.77. Its own entry in config/upstream-sources.json says it was rebuilt precisely so it would stop requiring .so.77. It was simply scheduled four stages too late to be used by the packages that needed it.

The fix. samba 6 → 2, and libtevent (its only factory build dependency above stage 0) 4 → 1. Both are safe by their own dependency stages: libtevent is built from libtalloc and make, both stage 0; samba from libtevent plus libtalloc, libtdb, icu, libxcrypt, all stage 0 or now 1. Nothing moves later, so no existing consumer ordering can break — pulling a provider earlier only ever adds packages to the roots that follow it.

tests/test_icu_staging.py makes it a gate: samba must precede both known consumers, and must still follow everything it is built from. It fails against the previous stage numbers.

On the two fixes I ruled out earlier. Constraining libicu-devel is impossible — icu.spec has Requires: lib%{name}%{?_isa} = %{version}-%{release}, so devel cannot be held at 78 beside a base at 77. And the 424 cross-stage BuildRequires edges I counted are not the thing to fix; they are the design. Only the specific packages that rebuild a Fedora package carrying an ICU-77 requirement need ordering, and that is a set of one.

I have left the test's docstring carrying the full chain, because the reflex on meeting an ICU 77 requirement is to exclude it — which is exactly what I did twice and what de4ac4d had to revert, since Fedora packages the build root legitimately needs (libical) require .so.77 and excluding it strands them outright.

318 tests pass. The run on this head will show whether publish step 6 now passes; that is the real verdict, and I will report it either way.


Generated by Claude Code

Found by asking which factory packages consume samba, which I should have done
in the same breath as moving it. gvfs BuildRequires libsmbclient-devel -- a
samba subpackage -- and sat at stage 1, so it resolved libsmbclient from Fedora
and its root installed libicu-77.1-2.1.hum1, exactly as localsearch did:

    [ 34/485] Installing libicu-0:77.1-2.1.  36.4 MiB
    [132/485] Installing libicu-devel-0:77.   5.1 MiB

That was already true before the previous commit -- gvfs stage 1 against samba
stage 6 -- so this is not fallout from moving samba, it is the same bug one
package over that moving samba to stage 2 did not by itself reach.

gvfs goes to stage 3. It is unconstrained in both directions: no factory package
BuildRequires gvfs at all, and its only build dependency above stage 0 is samba,
now at 2. Everything else it needs -- libnfs, libsoup3, libsecret, udisks2, gcr,
ibus, libcdio, libplist, libimobiledevice, gnome-online-accounts, libtalloc --
is stage 0.

The dependency scan behind both moves now resolves pkgconfig() BuildRequires as
well as plain names, which the first pass skipped; re-running it over samba
confirms its factory dependencies are still only libtevent (1) and stage-0
packages, so stage 2 remains correct.

tests/test_icu_staging.py covers gvfs in both directions and fails against the
previous stage number.

318 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (read at current head a7f93afa; the two commits past the dispatched head de4ac4d1 touch only config/upstream-sources.json and tests/test_icu_staging.py).

1. Medium — publish seeding from the prepare-time digest reintroduces a lost-update window on main.
publish now seeds from needs.prepare.outputs.factory_image (rebuild-rpms.yml:662), the digest resolved at run start. The publish concurrency group serializes publishers per ref, but the comments it relies on — "the later publisher re-seeds from the repository that its predecessor just made current" (rebuild-rpms.yml:633) and "concurrent work cannot lose RPMs" (rebuild-rpms.yml:59-60) — are now false: the later publisher re-seeds from whatever latest was when its own run started. main runs are deliberately not cancelled or queued, so a push run and the nightly can overlap. If run A publishes a same-NEVR rebuild — exactly what the new drag closure and stale detection produce (tools/rebuild_plan.py reverse_closure/stale_from_primary) — a concurrently started run B skips that package (its witness carries the same NEVR), seeds from the pre-A digest, and republishes the old binary, silently undoing A's drag rebuild. The branch-tag fix in the same commit is correct; on main the seed needs to be the then-current latest (or a merge of it with the prepare digest), or the stale comments/invariant need to be an accepted, documented trade-off.

2. Low — dependent edges through file-path Requires are resolved against a partial file list.
stale_from_primary skips /-prefixed Requires because "primary.xml lists only a subset of files" (tools/rebuild_plan.py:196), but dependents_from_primary resolves file Requires against that same subset (tools/rebuild_plan.py:116-118). A published binary requiring a file outside primary.xml's subset (anything not under bin dirs or /etc) produces no edge, so its dependent is silently not dragged — a missed rebuild, the non-conservative direction for this feature.

3. Low — new test classes are unreachable under direct execution.
icu_primary and ExcludedExternalTests are appended after the if __name__ == "__main__": unittest.main() guard (tests/test_rebuild_plan.py:415-438), so python3 tests/test_rebuild_plan.py exits before they are defined. unittest discover (what CI runs) is unaffected; move them above the guard to avoid a silent gap.

Not verified: the rpm-macro interaction claims (__debug_install_post vs %mingw_debug_package, build-stage.yml:392-431) rest on run evidence (380) I could not reproduce; the reasoning is plausible and the mock lane mirrors it.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Three contaminated packages had been found three different slow ways --
localsearch and nautilus from the publish failure, gvfs by asking who consumes
samba, and each of those took a round trip. So I asked the general question
instead: which factory packages link ICU at all, and does anything consume one
of them from its own stage or earlier?

Nine factory packages BuildRequire ICU (enchant2, evolution-data-server,
libical, libphonenumber, localsearch, mozjs140, re2, samba, webkitgtk). Exactly
one unfixed ordering violation fell out:

    bluez (stage 0) BuildRequires libical-devel from libical (stage 0)

Same stage, so bluez cannot see the factory libical and takes Fedora's, which
requires libicuuc.so.77. Confirmed against run 35413902261, where bluez root
installed libical-0:3.0.20-7.fc44 from fedora and then libicu-0:77.1-2.1.hum1.
The factory own libical is clean -- it built against libicu-78.3-8.hum1 -- it
was simply not available in time, exactly as samba was not available in time
for gvfs and localsearch.

bluez moves to stage 1. Nothing constrains it in either direction: no factory
package BuildRequires bluez, and its only factory build dependencies are
libical and make, both stage 0.

The more useful half is the new IcuLinkerOrderingTests, which derives this
rather than restating it. It scans every recipe for an ICU BuildRequires,
then for every factory consumer of one of those packages asserts the consumer
builds strictly later. Verified it fails against bluez previous stage with the
offending pair named. Not every same-stage pair would be a live bug -- it only
bites when Fedora build of the provider actually requires .so.77 -- but every
such pair is a place where a Fedora ICU can enter a build root unnoticed, and
the fix costs one stage number. Finding the fourth case by hand was not going
to be cheaper than encoding the rule.

319 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Two more ICU fixes, and then I stopped finding them one at a time.

gvfs (4161310's predecessor, a7f93af). After moving samba I asked the obvious next question — who consumes samba? gvfs BuildRequires libsmbclient-devel and sat at stage 1, so it took Fedora's libsmbclient and its root installed libicu-77.1-2.1.hum1, same as localsearch. That was already true before I touched samba (gvfs 1 against samba 6); moving samba to stage 2 did not by itself reach it. gvfs → stage 3. Nothing constrains it: no factory package BuildRequires gvfs at all, and its only dependency above stage 0 is samba.

Then I asked the general question instead. Three contaminated packages had been found three different slow ways, and the fourth was going to cost another round trip. So: which factory packages link ICU at all, and does anything consume one of them from its own stage or earlier?

Nine recipes BuildRequire ICU — enchant2, evolution-data-server, libical, libphonenumber, localsearch, mozjs140, re2, samba, webkitgtk. Exactly one unfixed violation fell out:

bluez (stage 0) BuildRequires libical-devel from libical (stage 0)

Same stage, so bluez cannot see the factory's libical and takes Fedora's, which requires libicuuc.so.77. Confirmed against run 35413902261 — bluez's root installed libical-0:3.0.20-7.fc44 from fedora and then libicu-0:77.1-2.1.hum1. The factory's own libical is clean; it built against libicu-78.3-8.hum1. It was simply not available in time, exactly as samba was not available in time for gvfs and localsearch. bluez → stage 1, unconstrained in both directions.

IcuLinkerOrderingTests now derives this rather than restating it. It scans every recipe for an ICU BuildRequires, then asserts every factory consumer of one of those packages builds strictly later, naming the offending pairs when it fails:

AssertionError: Lists differ:
['bluez (stage 0) BuildRequires libical-devel from libical (stage 0)'] != []

Not every same-stage pair would be a live bug — it only bites when Fedora's build of the provider actually requires .so.77 — but every such pair is a place where a Fedora ICU can enter a factory build root unnoticed, and the fix costs one stage number. That seemed the right trade against discovering the fifth case from another failed publish.

Current ordering: libical 0 → bluez 1, libtevent 1 → samba 2 → gvfs 3 → localsearch 4 → nautilus 6.

319 tests pass. The run on this head is the verdict; I will report it either way, including if it turns up a chain I have not accounted for.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (reviewed head 4161310, which the branch moved to after a7f93af was dispatched).

Medium — main-lane publish can now regress latest under concurrent runs. The seed step now pulls needs.prepare.outputs.factory_image — the digest resolved when this run started — instead of the then-current :latest (.github/workflows/rebuild-rpms.yml:649-662). On main there is no cancellation and the workflow-level concurrency group is run-unique (rebuild-rpms.yml:70-74), so two main runs can overlap (e.g. a push during the nightly). If run A publishes X-new while run B is in flight, B — whose witness and seed are both the pre-A digest, and whose older checkout matches X-old so it skipped X — publishes a seed containing X-old, moving latest backwards and dropping A's build. Publish's own concurrency group serializes the tag write but does not refresh the seed (rebuild-rpms.yml:635-637), and precedence only compares against Fedora/Hummingbird, not the prior factory image. The header comment at rebuild-rpms.yml:59 ("each publication re-seeds from the then-current repository, so concurrent work cannot lose RPMs") is now contradicted by the code below it. The branch-tag fix is right for PR lanes; on main consider re-resolving latest at publish time (inside the serialized section) or merging seed + run-start witness.

Low — dependents map can under-drag on file-path Requires. dependents_from_primary resolves Requires: /usr/bin/foo through the files listed in primary.xml (tools/rebuild_plan.py:117), but primary.xml carries only a subset of files — a limitation stale_from_primary documents and works around (tools/rebuild_plan.py:197) while the dependents docstring names only the devel-BuildRequires gap. A dependent whose only edge is a file path outside that subset won't be dragged. Worth a note in the docstring at minimum.

Everything else I traced checks out: the file:///work/factory URL is correct for both the host (prepare re-derives file://$PWD/work/factory, rebuild-rpms.yml:227-233) and the containerized preflight/build steps (-v "$PWD/work:/work"); prepare's branch-tag sanitization matches publish's tag derivation exactly; reverse_closure and inventory-order filtering in plan() are correct; the empty-witness path degrades to build-everything as claimed.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 4161310…a05a8)

1. File-based Provides are never collected from real repodata — tools/rebuild_plan.py:113-116 and :157-158 (medium)
In primary.xml as createrepo_c writes it, <file> elements are children of <format>, not of <package>. Both dependents_from_primary (package.iterfind(f"{{{COMMON_NS}}}file"), rebuild_plan.py:113) and provides_from_primary (rebuild_plan.py:157) search only direct children of <package>, so against a real repository they collect zero file provides (verified: iterfind on <package> returns [] for format-nested <file>; package.iter(...) finds them).

Consequence: a published binary with Requires: /usr/bin/foo finds no provider in dependents_from_primary, so that edge is silently dropped and the dependent is not dragged — the exact under-rebuild class this PR exists to close. (stale_from_primary is unaffected because it already skips /-prefixed requires, rebuild_plan.py:216.)

The test that covers this, test_provides_include_shipped_files (tests/test_rebuild_plan.py:352-356), constructs the fixture with .replace(b"</format>", b"</format><file>...</file>"), i.e. it encodes the same wrong layout, so it passes while real repos misparse. Fix is one-line per site (fmt.iterfind(...) or package.iter(...)) plus correcting the fixture.

2. Diff carries substantially more than the body states (info)
The body describes CI/skip/debuginfo work plus one sources commit, but the PR also adds/rewrites ~10 package recipes (wpa_supplicant, exiv2, iw, inih, libdaemon, libgexiv2, libportal, wireless-regdb, spirv-tools revert, pipewire watchdog patch, ffmpeg libplacebo drop — commits a4c4f19, bf72d53, dae6241, 13e52bc et al.). None of these are reviewed by the body's stated verification; reviewers should be aware the change surface is much larger than "fix(ci)".

Not verified here: the claim that --define "__debug_install_post %{nil}" leaves MinGW -debuginfo subpackages intact (build-stage.yml:395-424) rests on the cited run 380 evidence and could not be tested locally.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Setting expectations for the run on this head before it finishes, because I have just established that a green publish is not solely in my gift.

The publish gate is failing repo-wide, not just here — #152, #143 and #141 are all red on it. I pulled #143's publish log expecting to find the ICU chain and found a different one:

 10 requires libheif.so.1
 10 requires libglycin-2.so.0
 10 requires libgdk_pixbuf-2.0.so.0
  5 requires libgstvideo-1.0.so.0
  5 requires libgstallocators-1.0.so.0

across gdk-pixbuf2, gnome-online-accounts, gtk3, gtk4, gstreamer1-plugins-base, gvfs-goa and others. Not an ICU soname among them.

So there are at least two independent classes of publish-gate failure:

  1. ICU 77 contamination — a factory package links libicuuc.so.77 because Fedora build deps dragged ICU 77 into its root. That is what the staging fixes here address (libical 0 → bluez 1, libtevent 1 → samba 2 → gvfs 3 → localsearch 4 → nautilus 6).
  2. Stale soname consumers — a published package still requiring a soname nothing provides any more, because the provider was rebuilt and the consumer was not. That is fix(plan): unblock publish gate by triggering rebuild of stale soname consumers #171's territory, and the %autorelease hole it identifies in is_published is a plausible mechanism for it.

These are different bugs with different fixes and neither subsumes the other. Which means: if publish on this head still fails, the first thing to check is which class, and a soname failure would not be evidence against the ICU work — it would be #171's case, on which I left a review.

I would rather say this now than after the fact. The honest claim for this branch is that it removes one cause of the gate failing; whether the gate then passes depends on whether the other is also resolved on this head.


Generated by Claude Code

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

The staging fix is confirmed working, by the one observation the whole theory rested on.

bluez built green at its new stage 1 in run 35445712318 (job 105912047692). Its build root, side by side with the same package at stage 0 in run 35413902261:

before (stage 0) after (stage 1)
libical resolved from libical 3.0.20-7.fc44 — fedora libical 3.0.20-1.hum1.bfin — stages
ICU installed libicu-0:77.1-2.1.hum1 libicu-0:78.3-1.hum1
libicu-devel 77.1-2.1.hum1 78.3-1.hum1

That is the causal chain end to end. Moving bluez one stage later means the factory's own libical exists by the time bluez resolves, so bluez takes it instead of Fedora's; the factory's libical is linked against ICU 78; nothing in the root requires libicuuc.so.77; dnf therefore installs ICU 78, and libicu-devel follows the base as it must. No exclusion anywhere.

I checked the six remaining so.77 matches in that log before claiming this, because a leftover requirement would invalidate it. All six are my own build-stage.yml comment text being echoed by bash -x — the comment explaining why the exclusion must not come back. Not a single actual requirement.

Run state: 285 of 286 jobs green, zero failures. libtevent also green at its new stage 1; samba is building now at stage 2, then gvfs at 3, localsearch at 4, nautilus at 6. Each of those is a further instance of the same prediction and I will check them the same way rather than assuming.

One correction to my own earlier reporting in this thread: I said progress was "98/276" for a while. That was wrong — GitHub's jobs endpoint caps at 100 per page and I was not paginating. The real figures are above, from pages 1–4.

Still open, as flagged earlier: publish has a second, independent failure class — stale soname consumers (libheif.so.1, libglycin-2.so.0, libgdk_pixbuf-2.0.so.0), which is #171's territory. If the gate fails on those, it is not this work regressing.


Generated by Claude Code

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Scope: Major overhaul of factory skipping against published OCI image, dependent dragging, and debuginfo exclusion.
  • Findings: File-based Provides are format-nested in primary.xml (), causing iterfind on to miss file provides in real repodata.
  • In-flight check: Multiple matrix jobs still building across stages.
  • Outstanding: Author needs to fix file-provides parsing in rebuild_plan.py, followed by CI completion and independent review (+2).

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @hanthor: this "fix(ci)" PR carries 16 commits including 9 new packages (wpa_supplicant with 15 vendored patches, exiv2, libdaemon, wireless-regdb, …); a maintainer should decide whether to split it or accept the combined scope.

intent-alignment
The title and body describe the CI skip/dependents/debuginfo fix, and that part is implemented as described (.github/actions/load-factory-repo/action.yml, rebuild-rpms.yml:185-216, build-stage.yml:392-425, tools/rebuild_plan.py:81-243). But the branch also carries undisclosed work the body never mentions: nine new package recipes plus .packit.yaml entries (e.g. packages/wpa_supplicant/wpa_supplicant.spec, .packit.yaml:1915-1920), packages/ffmpeg/ffmpeg.spec and packages/pipewire/pipewire.spec changes, ICU-77 staging logic (tests/test_icu_staging.py, config/runtime-contract.toml), and docs/skills/repeated-mistakes.md. Only the feat(sources) commit is disclosed as a passenger.

security

  • I could not audit the new wpa_supplicant recipe: 904-line spec plus 15 patches touching OpenSSL/PKCS11/WEP behavior (packages/wpa_supplicant/wpa_supplicant-OpenSSL-Use-pkcs11-provider-when-OPENSSL_NO_ENGINE-i.patch and siblings). They read like Fedora's standard set but I did not verify provenance — human eyes warranted on a network supplicant.
  • Low: rebuild-rpms.yml "Resolve the published factory image" step interpolates ${{ github.ref_name }} directly into the run: script (branch=$(echo "${{ github.ref_name }}" | tr …)). Safe on pull_request (N/merge) but a push branch name is shell-evaluated before the sanitizing tr; pass it via env: instead. Exploit needs write access, hence low.
  • The new factory-repo trust path is sound: digest-pinned, publish gated off fork PRs (rebuild-rpms.yml:629-630), token only ever piped to docker login.

docs-currency

  • The renamed bump-upstream-sources.yml still opens its PRs with branch: bump/gnome-sources, a GNOME-only title, and a body claiming "Opened by .github/workflows/bump-gnome-sources.yml" and "the tarball GNOME actually serves" (.github/workflows/bump-upstream-sources.yml:93-101) — stale now that it polls GitHub/GitLab too, and it names a workflow file that no longer exists.
  • packages/libportal/README.md and packages/spirv-tools/README.md are placeholder stubs ("The libportal package").
  • README.md:44-51 and AGENTS.md correctly document the new skip behavior.

No findings from: style.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

The ICU chain is verified end to end. nautilus — the package whose libicuuc.so.77 requirement failed the publish gate in run 35413902261 — built green at stage 6 and is clean.

From its build root (job 105962840910):

localsearch  3.12~beta-1.hum1.bfin   stages        ← the factory's, not Fedora's
Installing libicu-0:78.3-1.hum1
Installing libicu-devel-0:78.3-1.hum1

Its built RPM requires libicuuc.so.78 and libicui18n.so.78. The log does contain two .so.77 strings; both are the build-stage.yml comment being echoed by bash -x — filtering the echo prefix leaves zero real occurrences, the same false positive I checked for on every package in this chain.

All five, each verified by reading the actual build root rather than inferring from a green check:

package stage dependency resolved from ICU installed
bluez 0 → 1 libical from stages 78.3
samba 6 → 2 libtevent from stages 78.3
gvfs 1 → 3 libsmbclient from stages 78.3
localsearch 4 samba-core-libs from stages 78.3
nautilus 6 localsearch from stages 78.3

The chain that produced the original failure is closed at every link: Fedora's samba-core-libs and libical require .so.77, and every factory package that used to resolve them from Fedora now resolves them from stages, so the requirement never enters a build root.

Run state: 320 jobs, 309 green, zero failures. Remaining stages, then precedence, then publish step 6 — the Hummingbird-only consumer transaction that has been the gate all along.

Standing caveat, unchanged and worth repeating before the result lands: publish has a second, independent failure class — stale soname consumers (libheif.so.1, libglycin-2.so.0, libgdk_pixbuf-2.0.so.0), visible in #143's publish log and addressed by #171, not by this work. If the gate fails on those, it is not the ICU fix regressing. If it fails on an ICU soname, that would be a chain I have not accounted for, and the method for finding it is in IcuLinkerOrderingTests.


Generated by Claude Code

@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

1 similar comment
@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member Author

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

@hanthor hanthor closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants