Skip to content

ci: rebuild reverse dependencies in incremental factory runs - #195

Closed
mrbobbytables wants to merge 1 commit into
projectbluefin:mainfrom
mrbobbytables:ci/rebuild-reverse-deps
Closed

mrbobbytables wants to merge 1 commit into
projectbluefin:mainfrom
mrbobbytables:ci/rebuild-reverse-deps

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Closes #23

Summary

Implements reverse dependency closure computation and reviewable build plan reporting for incremental factory rebuilds in rebuild-rpms.yml:

  • Reverse dependency selection:
    • Spec analysis (spec_dependents) parses declared packages, subpackages (%package, %package -n), Provides:, and pkgconfig(...) identifiers across recipes in packages/ to map factory BuildRequires edges.
    • Repodata analysis (dependents_from_primary) extracts runtime Provides/Requires and sonames from primary.xml.
    • Merges both dependency graphs (merge_dependents) and computes the transitive reverse dependency closure (reverse_closure), ensuring editing an upstream or staged library selects all downstream factory dependants across stages in proper wave order.
  • Global buildroot & tooling triggers:
    • Detects changes to global workflow files (rebuild-rpms.yml, build-stage.yml), build actions, tools/mock_config.py, and repo configurations (is_global_change), selecting a full or policy-defined rebuild while recording the exact triggering paths.
  • Incremental & retryable state:
    • New and unpublished packages, or packages with version/release mismatches, are selected incrementally without forcing a full rebuild.
  • Reviewable build plan CI artifact & summary:
    • Emits the computed build plan with waves, per-stage chunking, and selection reasons to $GITHUB_STEP_SUMMARY.
    • Writes structured JSON (work/reports/build-plan.json) and Markdown (work/reports/build-plan.md) reports uploaded as the factory-build-plan workflow artifact.
  • Self-improvement:
    • Updates .agents/skills/build-failure-triage/SKILL.md to document inspecting the factory-build-plan artifact and triaging reverse dependency closures.
  • Comprehensive test suite:
    • Tests in tests/test_rebuild_plan.py cover direct edits, version changes, new packages, global rebuild triggers, spec BuildRequires dependents, and report formatting.

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 755644a

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: Calculates and rebuilds reverse dependency closures in incremental factory runs, with build plan artifacts and test coverage. Currently has merge conflicts against main in .github/workflows/rebuild-rpms.yml and tools/rebuild_matrix.py due to the recent workflow restructure (removing PR triggers from rebuild-rpms.yml and adding package pruning). Awaiting author rebase/conflict resolution on fork branch, followed by second independent review.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • spec_dependents crashes on a %package line whose argument is empty after the whitespace (e.g. a trailing space, or %package -n with a macro that strips to nothing): sub.split()[0] raises IndexError (tools/rebuild_plan.py:303,305 — reproduced at head 755644a). Because this runs unconditionally in prepare over every spec, one malformed spec kills the whole plan job. Guard the empty case.
  • Report classification: stale published packages (selected via name in stale, unchanged and published) fall into the reverse_deps bucket in tools/rebuild_matrix.py:213-222, so build-plan.json's reverse_deps list and the "Reverse dependency closures" count include packages that were actually selected for staleness. Reasons per package are right; the aggregate buckets are not.

intent-alignment

  • Issue #23's acceptance criteria include "source-policy" / "global source/build tooling" changes selecting a rebuild, but GLOBAL_IGNORE_PATTERNS (^tools/(?!mock_config\.py$).*, tools/rebuild_plan.py:391) deliberately ignores tools/source_pipeline.py and tools/generated_sources.py. Editing source-fetch/verification policy selects nothing. If that's intentional, worth saying so; otherwise those paths belong in GLOBAL_REBUILD_PATTERNS.

style

  • tools/rebuild_plan.py:345 — the elif token.startswith("pkgconfig("): branch is dead code: its inner token in symbol_to_pkg check is exactly the preceding if condition.

docs-currency

  • docs/architecture.md:155 (prepare row: "Selects the packages that are new, changed, or requested by a full rebuild") describes exactly the selection behavior this PR changes (reverse-dep closure, global triggers, build-plan artifact) and was not updated, unlike SKILL.md.

Verified at head 755644a: all 50 tests in tests/test_rebuild_plan.py pass, and spec_dependents parses the real 349-package tree cleanly (863 edges). The new prior step in build-stage.yml also fixes the previously dangling steps.prior.outputs.pattern reference, correctly excluding the new factory-build-plan artifact from work/prior.

No findings from: security.

— hive: agent=reviewer backend=copilot model=claude-fable-5

hivecommons-hive Bot pushed a commit to mrbobbytables/utah-packages that referenced this pull request Sep 21, 2026
Review findings on projectbluefin#195.

spec_dependents crashed with IndexError on a '%package ' line carrying only
whitespace, or '%package -n' with no name: sub.split()[0] indexed an empty
list. That runs in the prepare job, so one malformed spec took the whole
pipeline down. Reproduced at head with a one-line spec; an unparseable
directive declares no subpackage and is now skipped, and a test covers all
three shapes alongside a real subpackage to prove the surviving edge.

rebuild_matrix bucketed every selected-but-published-and-unchanged package
as a reverse dependency, so stale selections inflated reverse_deps in
build-plan.json and the markdown summary. Stale packages now have their own
bucket in both, since a stale build is downstream of nothing this run
edited -- its build root moved under it. Per-package reasons were already
correct and are unchanged.

GLOBAL_IGNORE_PATTERNS ignored all of tools/ but mock_config.py, so edits
to source_pipeline.py or generated_sources.py selected no rebuild, against
the projectbluefin#23 criterion that global source tooling must. Both are now global
triggers: build-stage.yml runs source_pipeline.py for every package, and
generated_sources.py produces the Source0 bytes for the recipes with no
verbatim upstream archive, so either changes what a build's sources are
without any recipe diff recording it.

Also removed the unreachable 'elif token.startswith("pkgconfig(")' branch,
which re-tested the membership the preceding if had already decided, and
updated docs/architecture.md's prepare row, which still described selection
as 'new, changed, or requested by a full rebuild' with no mention of the
reverse-dep closure, the global triggers or the factory-build-plan artifact.

just check and just test pass: 413 tests, 2133 subtests.

Refs projectbluefin#195

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@castrojo
castrojo requested a review from hanthor September 21, 2026 03:27
- tools/rebuild_plan.py: calculate reverse dependency closures for changed packages
- tools/rebuild_matrix.py: bucket reverse dependencies and stale rebuilds separately
- .github/workflows/build-stage.yml: stage artifacts excluding build plan
- .github/workflows/rebuild-rpms.yml: upload factory-build-plan artifact
- tests/test_rebuild_plan.py: comprehensive unit test coverage
- docs/architecture.md: update pipeline description
- .agents/skills/build-failure-triage/SKILL.md: skill documentation

Co-authored-by: Bob Killen <mrbobbytables@users.noreply.github.com>
@castrojo
castrojo force-pushed the ci/rebuild-reverse-deps branch from caf771a to f00d9b9 Compare September 21, 2026 03:38

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • High — prune_sources output deleted but the workflow still consumes it. This PR removes prunable_sources and the outputs["prune_sources"] write from tools/rebuild_matrix.py, but at head .github/workflows/rebuild-rpms.yml:90 still maps steps.matrix.outputs.prune_sources, which is now the empty string. Two consequences:

    1. The publish gate at rebuild-rpms.yml:610 (needs.prepare.outputs.prune_sources != '[]') is now always true, so the publish job runs on every run — including no-op runs — republishing the repository each time. The deleted docstring in tools/rebuild_plan.py existed precisely to keep cleanup-only runs idempotent.
    2. The Hummingbird-owned RPM pruning at rebuild-rpms.yml:703-720 becomes a permanent silent no-op (jq -r '.[]' <<<"" yields an empty array), so RPMs from recipes handed over to Hummingbird now persist in the published OCI repository forever. config/hummingbird-provided-sources.json is left orphaned.

    Either restore the output or remove its consumers in the same change.

intent-alignment

  • Medium — undisclosed removal of the source-pruning feature and unrelated doc deletions. Issue #23 is about reverse-dependency selection; nothing in it or the PR body mentions deleting prunable_sources, HUMMINGBIRD_OWNED, and HummingbirdOwnershipTests (tests/test_rebuild_plan.py), nor deleting the "Trigger and merge-queue policy" section from docs/architecture.md (18 lines) or the "job name in the run list is the chunk, not the package" triage guidance from .agents/skills/build-failure-triage/SKILL.md (20 lines). The SKILL.md guidance documented a real triage trap (issue #212) and is unrelated to this feature. Conversely, the prior step added to build-stage.yml:65-69 fixes a dangling steps.prior.outputs.pattern reference that exists in base (build-stage.yml:81) — a good fix, but also undisclosed.

docs-currency

  • Low — stale operator-facing text left behind. The comment at rebuild-rpms.yml:704-710 still says "The prepare step computes this list from the repository it actually witnessed" — no longer true once prune_sources stops being emitted. Also docs/architecture.md:155, rewritten by this PR, still claims prepare "emits five stage lists" and that "a package declaring a stage above 4 has no job to run in," while the workflow has rebuild0–rebuild10 (rebuild-rpms.yml:329-489) and STAGES = 11 (tools/rebuild_plan.py:54).

No findings from: security, style.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is the reverse-dependency closure and build-plan report for rebuild-rpms.yml. The tests pass at f00d9b9 and on a trial merge with main, and the closure logic reads correctly, but one removal breaks the publish job and is not mentioned in the PR body.

tools/rebuild_matrix.py no longer writes outputs["prune_sources"] and tools/rebuild_plan.py drops prunable_sources() and its test. The workflow still consumes that output at head:

.github/workflows/rebuild-rpms.yml:90    prune_sources: ${{ steps.matrix.outputs.prune_sources }}
.github/workflows/rebuild-rpms.yml:610   || needs.prepare.outputs.prune_sources != '[]')
.github/workflows/rebuild-rpms.yml:703   PRUNE_SOURCES: ${{ needs.prepare.outputs.prune_sources }}

With the output missing, prune_sources is the empty string. Two effects I can see: the publish gate at line 610 becomes always true, so publish runs on no-op runs and republishes the repository every time; and the Hummingbird-owned prune loop at lines 711-720 gets an empty list, so RPMs from recipes deferred to Hummingbird stop being removed from the seeded repository. tests/test_factory_witness.py still asserts those three workflow lines, which is why CI stays green while the producer is gone.

Either restore the prune_sources output (keep prunable_sources and the HUMMINGBIRD_OWNED read in rebuild_matrix.py) or remove the three consumers and the witness test together, and say which in the PR body. Two smaller things while you are there: the comment at rebuild-rpms.yml:704-710 still says prepare computes the prune list, and docs/architecture.md:155 still says "five stage lists" and "a stage above 4" while STAGES is 11.

@hivecommons-hive hivecommons-hive Bot added the 3-human-queue Work admitted to the human-maintained queue. label Sep 21, 2026

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the deleted prune_sources output is still consumed by the workflow. Please restore or migrate the consumer.

hanthor added a commit that referenced this pull request Sep 25, 2026
… merge

A push to main that changes packages/** or config/** now runs the factory,
and the run builds only what changed since the published image plus the
direct BuildRequires dependents of what changed, in waves solved from real
BuildRequires. Everything else stays published and untouched.

- tools/extract_buildrequires.sh runs rpmspec over every recipe in the
  build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each
  BuildRequires to the factory recipes that provide it (subpackage names,
  explicit Provides, and the published generated provides), orders a build
  set by longest path, and reverses the edges for selection. A config
  stage only orders the members of a BuildRequires cycle
  (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored.
  The solved graph is 12 waves deep where hand stages went to 10, so the
  workflow now has rebuild0..rebuild13.
- tools/factory_state.py records on the published image, as the label
  org.projectbluefin.factory.state, the input digest of every build in it
  (recipe files, inventory entry, build-root pin, Hummingbird repo) and the
  packages that failed at their current digest. prepare selects against it,
  so a run is the union of every change since the last publication, and a
  push holds a package that already failed at exactly these inputs while
  the daily run retries it. An image without the label falls back to the
  diff and NEVR comparison. A label adds no layer.
- Reverse dependencies are direct only: following BuildRequires
  transitively dragged 167 packages into a one-line libical fix, through
  tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle.
  A consumer two hops away that really breaks surfaces as a stale published
  build and is rebuilt then.
- Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the
  same minute as bump-upstream-sources.yml; a merged bump now builds via
  push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized
  with cancel-in-progress false; a replaced pending run loses nothing
  because the next run selects against the published state.
- prepare writes the plan, with the wave and the reason for every package,
  to the job summary, and uploads the graph as the build-graph artifact.

The canary gains pass5: an incremental run against pass1's image, pinned
by digest, with vulkan-headers perturbed. It must select exactly
vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config
stage 4), and leave libical alone; verify-publish now also checks the
state label.

Supersedes #195, which computed a transitive closure from a regex parse of
the specs and dropped the Hummingbird-owned prune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
hanthor added a commit that referenced this pull request Sep 25, 2026
… merge

A push to main that changes packages/** or config/** now runs the factory,
and the run builds only what changed since the published image plus the
direct BuildRequires dependents of what changed, in waves solved from real
BuildRequires. Everything else stays published and untouched.

- tools/extract_buildrequires.sh runs rpmspec over every recipe in the
  build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each
  BuildRequires to the factory recipes that provide it (subpackage names,
  explicit Provides, and the published generated provides), orders a build
  set by longest path, and reverses the edges for selection. A config
  stage only orders the members of a BuildRequires cycle
  (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored.
  The solved graph is 12 waves deep where hand stages went to 10, so the
  workflow now has rebuild0..rebuild13.
- tools/factory_state.py records on the published image, as the label
  org.projectbluefin.factory.state, the input digest of every build in it
  (recipe files, inventory entry, build-root pin, Hummingbird repo) and the
  packages that failed at their current digest. prepare selects against it,
  so a run is the union of every change since the last publication, and a
  push holds a package that already failed at exactly these inputs while
  the daily run retries it. An image without the label falls back to the
  diff and NEVR comparison. A label adds no layer.
- Reverse dependencies are direct only: following BuildRequires
  transitively dragged 167 packages into a one-line libical fix, through
  tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle.
  A consumer two hops away that really breaks surfaces as a stale published
  build and is rebuilt then.
- Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the
  same minute as bump-upstream-sources.yml; a merged bump now builds via
  push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized
  with cancel-in-progress false; a replaced pending run loses nothing
  because the next run selects against the published state.
- prepare writes the plan, with the wave and the reason for every package,
  to the job summary, and uploads the graph as the build-graph artifact.

The canary gains pass5: an incremental run against pass1's image, pinned
by digest, with vulkan-headers perturbed. It must select exactly
vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config
stage 4), and leave libical alone; verify-publish now also checks the
state label.

Supersedes #195, which computed a transitive closure from a regex parse of
the specs and dropped the Hummingbird-owned prune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
hanthor added a commit that referenced this pull request Sep 25, 2026
… merge (#259)

A push to main that changes packages/** or config/** now runs the factory,
and the run builds only what changed since the published image plus the
direct BuildRequires dependents of what changed, in waves solved from real
BuildRequires. Everything else stays published and untouched.

- tools/extract_buildrequires.sh runs rpmspec over every recipe in the
  build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each
  BuildRequires to the factory recipes that provide it (subpackage names,
  explicit Provides, and the published generated provides), orders a build
  set by longest path, and reverses the edges for selection. A config
  stage only orders the members of a BuildRequires cycle
  (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored.
  The solved graph is 12 waves deep where hand stages went to 10, so the
  workflow now has rebuild0..rebuild13.
- tools/factory_state.py records on the published image, as the label
  org.projectbluefin.factory.state, the input digest of every build in it
  (recipe files, inventory entry, build-root pin, Hummingbird repo) and the
  packages that failed at their current digest. prepare selects against it,
  so a run is the union of every change since the last publication, and a
  push holds a package that already failed at exactly these inputs while
  the daily run retries it. An image without the label falls back to the
  diff and NEVR comparison. A label adds no layer.
- Reverse dependencies are direct only: following BuildRequires
  transitively dragged 167 packages into a one-line libical fix, through
  tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle.
  A consumer two hops away that really breaks surfaces as a stale published
  build and is rebuilt then.
- Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the
  same minute as bump-upstream-sources.yml; a merged bump now builds via
  push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized
  with cancel-in-progress false; a replaced pending run loses nothing
  because the next run selects against the published state.
- prepare writes the plan, with the wave and the reason for every package,
  to the job summary, and uploads the graph as the build-graph artifact.

The canary gains pass5: an incremental run against pass1's image, pinned
by digest, with vulkan-headers perturbed. It must select exactly
vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config
stage 4), and leave libical alone; verify-publish now also checks the
state label.

Supersedes #195, which computed a transitive closure from a regex parse of
the specs and dropped the Hummingbird-owned prune.


Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@hanthor

hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member

Superseded by #259, which solves waves and reverse dependencies from rpmspec BuildRequires in the build root (direct dependents; transitive closure measured at 167 packages for a libical fix) and selects against a per-package state label on the published image.

@hanthor hanthor closed this Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3-human-queue Work admitted to the human-maintained queue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: rebuild reverse dependencies in incremental factory runs

4 participants