Skip to content

ci: rebuild reverse dependencies in incremental factory runs - #155

Closed
mrbobbytables wants to merge 4 commits into
projectbluefin:mainfrom
mrbobbytables:ci/rebuild-reverse-dependencies
Closed

mrbobbytables wants to merge 4 commits into
projectbluefin:mainfrom
mrbobbytables:ci/rebuild-reverse-dependencies

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Summary

Closes #23

Implements reverse dependency closure computation and reviewable build plan reporting for incremental factory rebuilds in rebuild-rpms.yml:

  • Reverse dependency selection:
    • Extracts runtime Provides/Requires and sonames from repodata primary.xml (dependents_from_primary).
    • Extracts build-time BuildRequires and subpackage/pkgconfig symbols from package specs in packages/ (spec_dependents).
    • Merges both dependency graphs (merge_dependents) and computes the transitive reverse dependency closure (reverse_closure) so editing an upstream library rebuilds all downstream dependants across waves in proper stage order.
  • Global buildroot & tooling triggers:
    • Detects changes to global workflow files, build tools under tools/, non-inventory config/, or tests (is_global_change), selecting a full rebuild with explicit cause tracking.
  • Incremental & retryable:
    • New and unpublished packages, or packages with version/release mismatches, are selected incrementally without forcing a full rebuild.
  • Reviewable build plan artifact and summary:
    • Emits the computed build plan with waves, per-stage chunking, and selection reasons to $GITHUB_STEP_SUMMARY.
    • Emits structured JSON (work/reports/build-plan.json) and Markdown (work/reports/build-plan.md) reports uploaded as the factory-build-plan workflow artifact.
  • Tests & triage skill update:
    • Adds unit tests in tests/test_rebuild_plan.py covering direct edits, version changes, global changes, reverse dependency selection (runtime and spec BuildRequires), and report formatting.
    • Documents build plan inspection and reverse dependency closure dynamics in .agents/skills/build-failure-triage/SKILL.md.

Verification

just check
just test
just factory-check
pre-commit run --all-files

All local verification gates and unit tests pass (335 passed).

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 085c431

Computes reverse dependency closures in incremental factory rebuilds,
selecting downstream dependants across build waves for changed packages
and recipes. Supports both runtime Provides/Requires from repodata
primary.xml and spec-level BuildRequires/pkgconfig symbols. Global workflow,
tooling, configuration, or buildroot changes trigger a full rebuild, while
new and unpublished packages remain incremental and retryable. Emits the
computed build plan to GITHUB_STEP_SUMMARY and workflow run artifacts.

Closes projectbluefin#23

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head 085c431)

1. High — stale detection ignores Fedora, half the buildroot; most published packages will be flagged stale every run.
stale_from_primary (tools/rebuild_plan.py:315-350) marks a source stale when a Requires of its published binaries is satisfied neither by the factory primary nor by external, and main() builds external from the Hummingbird repo only (tools/rebuild_matrix.py:171-178). But the actual transaction pairs Hummingbird with Fedora — tools/mock_config.py renders [fedora]/[updates] into every buildroot and its header says "Fedora paired with Hummingbird", and Hummingbird itself is documented as ~3,500 base packages with "no desktop stack at all" (config/hummingbird.repo:1-4). Any factory package linking against a Fedora-provided soname (libX11, cairo, gtk, etc. — i.e., most of a desktop stack) will have unsatisfied Requires under this check, be selected as "stale published build" (tools/rebuild_plan.py:500-503), and then drag its reverse closure — turning every incremental/scheduled run into a near-full rebuild and defeating the PR's purpose. Either include Fedora provides in external, or restrict staleness to capabilities the factory repo itself once provided. The tests (tests/test_rebuild_plan.py StaleTests) only model a repo-internal soname bump, so this path is untested.

2. Medium — is_global_change forces a full rebuild for any tests/** or tools/** edit.
GLOBAL_REBUILD_PATTERNS (tools/rebuild_plan.py:416-423) includes ^tests/.* and ^tools/.*, so editing a unit test or a planning script (including this PR itself) now rebuilds the entire inventory. Previously such pushes triggered the workflow but skipped everything already published. Tests cannot affect build outputs; consider limiting global triggers to files that shape the buildroot/build (e.g., mock_config, build-stage) rather than all tooling and tests.

3. Low — dead duplicated block after return 0 in main().
tools/rebuild_matrix.py:277-294 repeats the overflow check, stage output loop, and GITHUB_OUTPUT write unreachably after the return 0 at line 275. Harmless today but a trap for the next edit — delete it.

4. Low — dead/typo assignment in spec_dependents.
tools/rebuild_plan.py:249 assigns symbol_to_pkg[f"{subname}{sfx}"] = pkg_specs (a dict) and is immediately overwritten by the correct = p on the next line. Remove line 249.

Not a finding: the build-stage.yml prior step (lines 51-55) actually fixes a pre-existing dangling steps.prior.outputs.pattern reference on main — good catch, and the [0-N] glob class is safe since stages cap at 10.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is aimed at a real problem — a stale package that nothing rebuilds is exactly the class of failure that has been breaking the publish gate — and the two-graph approach (repodata for runtime, specs for build-time) is the right shape. Three things before it lands, plus one coordination note that matters more than any of them.

1. A dead assignment in spec_dependents that says this block has not been read.

symbol_to_pkg[f"{subname}{sfx}"] = pkg_specs
symbol_to_pkg[f"{subname}{sfx}"] = p

The first line assigns pkg_specs — the dict of spec contents — into the symbol map, and the second immediately overwrites it. Harmless as written, which is the problem: it means nothing exercised that path closely enough to notice a dict being used as a package name. Given this function decides what gets rebuilt, the whole block wants a careful pass rather than just deleting that line.

2. The symbol map guesses subpackage names as well as reading them, and the guesses can overwrite the real ones.

for sfx in ("-devel", "-libs", "-common", "-static", "-tools", "-utils", "-doc", "-docs"):
    symbol_to_pkg[f"{p}{sfx}"] = p
symbol_to_pkg[f"pkgconfig({p})"] = p
if p.startswith("lib"):
    symbol_to_pkg[f"pkgconfig({p[3:]})"] = p
else:
    symbol_to_pkg[f"pkgconfig(lib{p})"] = p

Every factory package gets eight synthetic -suffix symbols and one or two synthetic pkgconfig() symbols whether or not it ships them, and these are written into the same flat dict as the real %package declarations. Since it is last-write-wins over an unordered set iteration, a synthetic symbol from one package can overwrite a genuine one from another, and which one wins is not stable between runs.

The specs already state the answer — %package and %package -n are parsed a few lines later, and Provides: pkgconfig(...) is declarative. Build the map from those only. A guessed edge that is wrong is worse than a missing one here: a false edge rebuilds something unnecessarily, which is merely slow, but a displaced real edge means a genuinely stale consumer is not rebuilt, which is the bug this PR exists to fix, now arriving non-deterministically.

3. Two smaller parsing points. re.sub(r"#.*$", "", line) strips from the first # anywhere on the line, including inside URLs with fragments; and the macro expansion does sub.replace(f"%{k}", v) unanchored, so a macro named name also rewrites the prefix of any longer %name… token. Neither is likely to bite on today's recipes, but both are the kind of thing that fails silently and shows up as a missing rebuild.

4. The coordination note — this collides head-on with #133, and the collision is not textual.

Neither provides_from_primary, stale_from_primary nor dependents_from_primary exists on main:

$ git show origin/main:tools/rebuild_plan.py | grep -n 'def provides_from_primary\|def stale_from_primary\|def dependents_from_primary'
(no output)

Both this PR and #133 introduce them independently, from scratch. git merge-tree between the two branches conflicts in four files:

CONFLICT (content): Merge conflict in .github/workflows/build-stage.yml
CONFLICT (content): Merge conflict in tests/test_rebuild_plan.py
CONFLICT (content): Merge conflict in tools/rebuild_matrix.py
CONFLICT (content): Merge conflict in tools/rebuild_plan.py

Good news that git flags it rather than silently keeping the later definition of a duplicated function, but whoever merges second has a semantic merge, not a textual one. Specifically, #133's provides_from_primary carries an EXCLUDED_EXTERNAL parameter that stops the staleness model counting an ICU build the consumer transaction will refuse; that behaviour has to survive the merge, and it is not obvious from the diff that it would.

I would rather not race you for it. If you want to land first I will rebase #133 onto this and re-apply the ICU exclusion on top of your provides_from_primary; if the reverse is easier, say so. What should not happen is both landing without someone deliberately reconciling those three functions.

One related thing worth knowing while you are in this code: reverse-dependency rebuild selection does not fix reverse-dependency stage ordering, and the latter is what has been poisoning the publish gate. bluez sat in the same stage as libical, so it resolved libical from Fedora, whose build requires libicuuc.so.77, and linked ICU 77 — which the Hummingbird-only consumer transaction then refuses. Same story for gvfs/samba and localsearch/nautilus. Rebuilding a dependent does not help if it rebuilds in a wave where its factory-built dependency still does not exist. #133 now has a test that derives those ordering violations from the specs; it may be worth the two graphs eventually feeding it.

CI is unstable on this head, which should be green before merge regardless.


Generated by Claude Code

…taleness

- In spec_dependents: drop synthetic suffix guesses (-devel, -libs, pkgconfig) and
  build symbol map strictly from declared package and Provides tokens; remove dead
  assignment; improve macro expansion and avoid cutting URLs with fragment comments.
- In stale_from_primary: restrict staleness to capability stems provided by the
  factory repository itself so base buildroot capabilities (e.g. from Fedora paired
  with Hummingbird) do not falsely mark packages stale.
- In GLOBAL_REBUILD_PATTERNS: limit triggers to buildroot/workflow files rather
  than all tests and tooling.
- In rebuild_matrix.py: remove dead duplicated code block after return 0.
- Update unit tests in tests/test_rebuild_plan.py.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @hanthor and @kubestellar-hive in commit 08e4902:

  1. Staleness Detection on Base Buildroot: In stale_from_primary, capability staleness is now scoped to capability stems that the factory repository itself provides. Capabilities satisfied solely by the base buildroot (Fedora paired with Hummingbird) are not flagged as stale factory rebuilds. Added test_unsatisfied_base_buildroot_sonames_do_not_flag_stale to verify.
  2. Deterministic Spec Dependency Map:
    • Removed synthetic suffix guessing (-devel, -libs, -common, synthetic pkgconfig) from spec_dependents; the symbol map is now built strictly from declared %package / %package -n and declarative Provides: / .pc statements.
    • Removed the dead assignment (symbol_to_pkg[f"{subname}{sfx}"] = pkg_specs).
    • Hardened macro substitution with word boundaries and delimited matching, and avoided cutting URL fragments when stripping comments.
  3. Global Rebuild Scoping: Restricted GLOBAL_REBUILD_PATTERNS to buildroot/workflow files (tools/mock_config.py, workflows, actions, repo configs) so edits to tests and non-buildroot tooling do not force full rebuilds.
  4. Cleanups: Removed dead duplicated block after return 0 in tools/rebuild_matrix.py.
  5. Coordination Note: Happy to land this first and help reconcile provides_from_primary with fix(ci): skip against the published image, drag dependents, stop building debuginfo #133.

🐝 Hive Agent: contributor | SHA: unknown

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Substantive review:

  • Diagnosis & Fix identified: The CI failure in the prepare job (TypeError/PatternError in tools/rebuild_plan.py:expand_spec_macros) is triggered when spec macro definitions containing backslashes (such as in grub2, git, hwinfo, or webkitgtk) are substituted via re.sub, where raw string replacements treat trailing backslashes as invalid escape sequences. Using replacement functions lambda _: v resolves this across all specs.
  • Outstanding: Author @mrbobbytables or maintainer push to fork branch mrbobbytables:ci/rebuild-reverse-dependencies is required to apply the regex replacement fix, unblocking prepare, followed by independent peer review (+2).

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed fix in commit ff05977:

  • Root cause: Spec macro definitions containing backslashes (e.g. in grub2, git, hwinfo, or webkitgtk) caused re.sub in expand_spec_macros to raise re.PatternError on invalid escape sequences in string replacement templates.
  • Fix: Use lambda _: v replacement callback in expand_spec_macros to treat macro values as verbatim text without regex template escape interpretation.
  • Verification: rebuild_matrix.py and all 337 unit tests now pass cleanly locally.
  • Outstanding: Awaiting CI completion of commit ff05977 and independent peer review (+2).

@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

1 similar comment
@hanthor

hanthor commented Sep 20, 2026

Copy link
Copy Markdown
Member

Closing to consolidate the queue on the merged #178 full-factory candidate. The branch is retained; please re-open or submit a fresh, rebased PR after that build establishes the current baseline.

@hanthor hanthor closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: rebuild reverse dependencies in incremental factory runs

3 participants