Skip to content

fix(plan): unblock publish gate by triggering rebuild of stale soname consumers - #171

Closed
mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/soname-closure-rebuilds
Closed

mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/soname-closure-rebuilds

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Resolves #153

Cause

The published repository at https://projectbluefin.github.io/utah-packages/ contains stale consumer builds that depend on older shared library sonames that are no longer available in the published set or the Hummingbird base:

  • mozc / ibus-mozc requiring libabsl_*.so.2601.0.0 (due to abseil-cpp update)
  • libheif requiring libavcodec.so.62 and libopenjph.so.0.25 (bconds disabled in spec)
  • mesa requiring libSPIRV-Tools.so (the spirv-tools recipe was dropped in 56b2973 so mesa must link the buildroot copy)
  • gvfs requiring libnfs.so and libbluray.so.3 (bconds disabled in spec)

Because libheif, mesa, and gvfs use %autorelease, expected_release returns None and is_published previously fell back to version-only matching, which skipped them even on scheduled runs.

Fix

  1. In tools/rebuild_plan.py, make is_published return False if entry.get('rebuild_reason') is present, preventing stale %autorelease packages from being skipped.
  2. In config/upstream-sources.json, add rebuild_reason for gvfs, libheif, mesa, and mozc, as well as dist_bump (count: 1, baseline: 21) for mozc.
  3. Added unit test in tests/test_rebuild_plan.py verifying rebuild_reason invalidates published status.
  4. Documented failure signature and remediation in .agents/skills/build-failure-triage/SKILL.md.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 9b18393

…tale consumer rebuilds

The published repository contains stale consumer builds linking older
sonames that are no longer satisfied by the repository (or the Hummingbird
runtime closure):
- mozc / ibus-mozc wanting libabsl_*.so.2601.0.0 (abseil-cpp updated)
- libheif wanting libavcodec.so.62 and libopenjph.so.0.25 (ffmpeg/openjph bconds disabled)
- mesa wanting libSPIRV-Tools.so (spirv-tools recipe removed)
- gvfs wanting libnfs.so and libbluray.so.3 (nfs/bluray bconds disabled)

Because libheif, mesa, and gvfs use %autorelease, expected_release returns
None and is_published previously fell back to matching the version alone,
treating them as published and skipping them even on scheduled runs.

- Update tools/rebuild_plan.py so is_published returns False if
  entry.get('rebuild_reason') is set.
- Add test coverage in tests/test_rebuild_plan.py.
- Add rebuild_reason for gvfs, libheif, mesa, and mozc (and dist_bump for
  mozc's literal Release) in config/upstream-sources.json.
- Document consumer soname closure failure pattern in
  .agents/skills/build-failure-triage/SKILL.md.

Fixes projectbluefin#153

Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (at 9b18393)

The fix works as described: is_published now returns False when rebuild_reason is set (tools/rebuild_plan.py:113), the mozc dist_bump baseline "21" matches the spec's literal Release: 21%{?dist} (packages/mozc/mozc.spec:9), so mozc republishes as ...bfin.1 and supersedes the stale build. Full suite passes (317 tests).

One finding:

[medium] rebuild_reason never expires — permanent rebuild/republish loop. Nothing removes or retires the field: unlike dist_bump, whose baseline retires the counter when the release moves (tools/dist_bump.py:90-94), is_published returns False unconditionally whenever rebuild_reason is present (tools/rebuild_plan.py:113-114). Once merged, gvfs, libheif, mesa, and mozc will be rebuilt on every scheduled run indefinitely — and since %autorelease (gvfs/libheif/mesa) and the fixed dist_bump count (mozc, config/upstream-sources.json:1842-1845) produce the same NEVR each time, each run republishes identical NEVRs with different bits. mesa alone makes this expensive. The SKILL.md entry added at .agents/skills/build-failure-triage/SKILL.md:161 documents adding the field but not removing it. Suggest either (a) noting in SKILL.md that rebuild_reason must be removed once the rebuilt packages are published, or (b) an expiry mechanism analogous to dist_bump's baseline.

Test adequacy is otherwise fine — the new test (tests/test_rebuild_plan.py:128-134) covers exactly the branch added.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looked at this against the workflow and the run at 9b18393, since it's the PR that unblocks the publish gate for everything queued behind it.

What checks out. The four stale consumers are correctly identified, and all four rebuilt green on this PR (gvfs, libheif, mozc in rebuild1, mesa in rebuild0); 266 of 267 jobs are green with only webkitgtk still building. The mozc dist_bump is the right tool where Release: is a literal.

Two things to know before merging.

  1. The consumer transaction is still unproven. publish — the job that runs "Validate Hummingbird-only consumer transaction", i.e. the thing failing in #153 — is gated on github.event.pull_request.head.repo.full_name == github.repository, and this is a fork PR, so it never ran here. The first time we learn whether the rebuilt mesa really links against the buildroot spirv-tools (rather than the dropped recipe) is the push to main. Worth saying so in the PR body, and watching that run.

  2. rebuild_reason becomes a permanent nightly-rebuild flag. The is_published change returns False whenever the key is present, forever. The existing convention on main (samba, the only entry that has it today) is that the entry edit trips changed and forces one rebuild on the push; after that publishes, nightly runs see the new version/release and skip it. With this change, samba and these four — including mesa, one of the heaviest builds in the inventory — are rebuilt on every 06:41 schedule until someone remembers to delete the key, and nothing in the diff or the skill doc says to. For the one-time invalidation this PR needs, the JSON edit alone already does the job on the merge push.

    If the goal is specifically to cover an %autorelease package whose entry was edited between nightlies without a push (the "schedule has no diff range" case in the is_published comment), a narrower fix is to key the invalidation on something that changes once — e.g. a rebuild_reason that includes a stamp compared against the published build — or to document that the key must be removed in the same PR that confirms the republish. Either way, the skill-doc row should tell the reader the flag is not free.

Not a repo maintainer here, so this is a comment rather than a review decision.

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diagnosis is right and this is a real class of publish-gate failure — distinct from the ICU one I have been chasing in #133, so the two are complementary rather than competing. Stale %autorelease consumers falling through is_published's version-only fallback is exactly the hole you describe, and mozc/libheif/mesa/gvfs are a plausible set.

The one-line change in tools/rebuild_plan.py is what I would not merge as written:

if entry.get("rebuild_reason") is not None:
    return False

This is unconditional and permanent, and nothing ever removes the field. rebuild_reason is a prose annotation, not a trigger with an expiry. Once it is set, that package can never again be recognised as published, so it rebuilds on every run, forever — including the scheduled ones. For mesa in particular that is not a small cost, and it applies even after the rebuild you wanted has happened and the good build is sitting in the published repository. The skip optimisation is the reason the factory finishes at all; this quietly removes five packages from it with no way back except deleting the annotation, which nobody will think to do because it reads as documentation.

The behaviour you actually want is already there, and you get it for free. changed_entries compares whole inventory entries:

return {
    entry["name"]
    for entry in after.get("packages", [])
    if old.get(entry["name"]) != entry
}

Its docstring is about precisely this failure — a stage move that left Version and Release untouched, so the package matched the published listing, was skipped, and came back from the seeded image as the very build the move existed to replace. So adding rebuild_reason to those four entries already changes the entry, already invalidates the match, and already forces exactly one rebuild. The is_published change is not needed to achieve the immediate goal; its only additional effect is to make that rebuild permanent.

I would drop the rebuild_plan.py hunk and its test, and keep the inventory edits. That is a one-shot rebuild of the four packages, which is what the issue asks for.

If you want a durable "this must not be skipped until it has actually been fixed" marker — which is a reasonable thing to want — it needs to carry the condition it expires under, not just the prose. Pairing it with the published release it was recorded against, so it stops firing once the published build is newer than the broken one, makes it self-clearing. Either shape is fine; what does not work is a flag that is true forever by construction.

Two smaller points:

  • samba is currently the only entry in the repository carrying rebuild_reason, so this change silently changes samba's behaviour too, which the description does not mention. That matters right now: I have just moved samba from stage 6 to stage 2 on the #133 branch to fix the ICU 77 contamination chain (Fedora's samba-core-libs requires libicuuc.so.77, which was dragging localsearch and then nautilus onto ICU 77 and failing the consumer transaction). Under this change samba would also rebuild on every run from now on.
  • Expect a trivial conflict with that branch in config/upstream-sources.json — different entries (samba/libtevent stage fields versus your four rebuild_reason additions), so it should merge cleanly, but whichever lands second will need a rebase.

CI is currently unstable on this head, which should be resolved before merge regardless of the above.


Generated by Claude Code

@Danathar

Copy link
Copy Markdown
Contributor

@castrojo @hanthor — I reviewed this today (comment above) and only have read access here, so it needs one of you. Short version: the fix is sound and all four stale packages rebuilt green, but (1) the publish job that runs the failing consumer transaction never runs on a fork PR, so the actual gate is unproven until merge, and (2) the is_published change turns rebuild_reason into a permanent nightly-rebuild flag for every entry that has it, including mesa and samba. I'd merge it with the second point fixed or at least documented.

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member

Evidence that strengthens this PR's case, found while checking whether my own ICU work would be enough to turn publish green. It is not, and this is why.

publish is failing repo-wide — #152, #143 and #141 are all red on it. I pulled #143's publish log expecting the ICU chain I have been chasing and found yours instead:

 10 requires libheif.so.1
 10 requires libglycin-2.so.0
 10 requires libgdk_pixbuf-2.0.so.0
  5 requires libgstvideo-1.0.so.0
  5 requires libgstallocators-1.0.so.0
  4 requires libgstgl-1.0.so.0

across gdk-pixbuf2, gnome-online-accounts, gnome-tweaks, gtk3, gtk4, gstreamer1-plugins-base and gvfs-goa. Not an ICU soname among them — these are exactly the stale consumers this PR is about, and libheif is one of the four you add a rebuild_reason for.

So the gate has at least two independent failure classes: ICU 77 contamination from Fedora build deps entering a build root (staging fixes, #133) and stale soname consumers that were never rebuilt after their provider was (this PR). Neither subsumes the other, and I now expect both are needed before publish goes green. I have said the same on #133 so that a soname failure there is not misread as the ICU work not having landed.

That does not change my review — the is_published hunk still makes rebuild_reason a permanent never-skip, and changed_entries already gives you the one-shot rebuild for free by comparing whole inventory entries. But it does raise the priority: this is not a tidy-up, it is one of the two things standing between the factory and a published repository.

If it helps, I am happy to take the inventory half of this onto #133 so the two fixes ride together, and leave you the is_published design question separately — your call, and I would rather not touch your PR uninvited.


Generated by Claude Code

Address review feedback from @hanthor: keep the one-shot rebuild_reason
inventory modifications in config/upstream-sources.json (which invalidate
the whole-entry comparison in changed_entries for a one-time rebuild), but
drop the permanent False return in is_published() to avoid triggering rebuilds
on every subsequent run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback: dropped the unconditional rebuild_reason check in tools/rebuild_plan.py and its test in tests/test_rebuild_plan.py in commit b921e24. The rebuild_reason entries in config/upstream-sources.json are preserved, which causes changed_entries() to flag those four packages for a single rebuild without permanently bypassing is_published.

🐝 Hive Agent: contributor | SHA: unknown

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Advisory — read access only here.) Re-checked after b921e24: the is_published change and its test are gone, so rebuild_reason is back to being documentation on the entry, and the four entries (gvfs, libheif, mesa, mozc with its dist_bump) get their one rebuild through changed_entries() on the merge push — no permanent nightly rebuild of mesa or samba. That resolves my second point. My first point stands as a fact rather than a request: publish doesn't run on fork PRs, so the consumer transaction is only proven on the push to main. hanthor's log from #143 (libheif.so.1, libglycin, libgdk_pixbuf, the gstreamer trio) confirms these are the stale consumers this PR targets, and that ICU 77 (#133) is a separate, second cause — so both need to land before publish goes green. I'd merge this one.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Status: Danathar approved commit b921e24 which removed the permanent rebuild_reason logic from rebuild_plan.py, leaving the one-time rebuild triggered via changed_entries on merge push.
  • Review requirements: Awaiting @hanthor to dismiss or convert their CHANGES_REQUESTED review following the b921e24 update, plus resolution of the transient libxshmfence build check failure in rebuild0 before merge.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment
The PR body is stale relative to the head. Fix items 1 and 3 (the is_published early-return in tools/rebuild_plan.py and the unit test in tests/test_rebuild_plan.py) were reverted in head commit b921e24 ("drop unconditional rebuild_reason in is_published"); the net diff is only config/upstream-sources.json and .agents/skills/build-failure-triage/SKILL.md. As merged, rebuild_reason is read by nothing in tools/ — it works purely as a one-shot inventory-entry change picked up by changed_entries (tools/rebuild_plan.py:136-158), the same mechanism the existing samba entry relies on (config/upstream-sources.json:2200). Please update the body so it describes the actual fix.

docs-currency
(info) The new SKILL.md row (.agents/skills/build-failure-triage/SKILL.md:161) says adding rebuild_reason forces "rebuild and invalidation", wording from the reverted is_published change. It is now a one-shot trigger: it fires only on the push that edits the entry, so if that run fails, a subsequent scheduled run (no BASE_SHA) skips the %autorelease packages again and the entry must be edited again to re-trigger. Worth a clause in the remediation column.

Mechanism otherwise verified: mozc dist_bump baseline "21" matches the spec's literal Release: 21%{?dist} (packages/mozc/mozc.spec:9) and the documented baseline semantics (tools/dist_bump.py:11-25).

No findings from: correctness, security, style.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

@hanthor hanthor closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[shared-ci] publish / Validate Hummingbird-only consumer transaction failing across projectbluefin/utah-packages

4 participants