Repository navigation
ci: rebuild reverse dependencies in incremental factory runs - #195
mrbobbytables wants to merge 1 commit into
Conversation
castrojo
left a comment
There was a problem hiding this comment.
Reviewed: Calculates and rebuilds reverse dependency closures in incremental factory runs, with build plan artifacts and test coverage. Currently has merge conflicts against main in .github/workflows/rebuild-rpms.yml and tools/rebuild_matrix.py due to the recent workflow restructure (removing PR triggers from rebuild-rpms.yml and adding package pruning). Awaiting author rebase/conflict resolution on fork branch, followed by second independent review.
There was a problem hiding this comment.
correctness
spec_dependentscrashes on a%packageline whose argument is empty after the whitespace (e.g. a trailing space, or%package -nwith a macro that strips to nothing):sub.split()[0]raisesIndexError(tools/rebuild_plan.py:303,305 — reproduced at head 755644a). Because this runs unconditionally inprepareover every spec, one malformed spec kills the whole plan job. Guard the empty case.- Report classification: stale published packages (selected via
name in stale, unchanged and published) fall into thereverse_depsbucket in tools/rebuild_matrix.py:213-222, sobuild-plan.json'sreverse_depslist and the "Reverse dependency closures" count include packages that were actually selected for staleness. Reasons per package are right; the aggregate buckets are not.
intent-alignment
- Issue #23's acceptance criteria include "source-policy" / "global source/build tooling" changes selecting a rebuild, but
GLOBAL_IGNORE_PATTERNS(^tools/(?!mock_config\.py$).*, tools/rebuild_plan.py:391) deliberately ignorestools/source_pipeline.pyandtools/generated_sources.py. Editing source-fetch/verification policy selects nothing. If that's intentional, worth saying so; otherwise those paths belong inGLOBAL_REBUILD_PATTERNS.
style
- tools/rebuild_plan.py:345 — the
elif token.startswith("pkgconfig("):branch is dead code: its innertoken in symbol_to_pkgcheck is exactly the precedingifcondition.
docs-currency
- docs/architecture.md:155 (
preparerow: "Selects the packages that are new, changed, or requested by a full rebuild") describes exactly the selection behavior this PR changes (reverse-dep closure, global triggers, build-plan artifact) and was not updated, unlike SKILL.md.
Verified at head 755644a: all 50 tests in tests/test_rebuild_plan.py pass, and spec_dependents parses the real 349-package tree cleanly (863 edges). The new prior step in build-stage.yml also fixes the previously dangling steps.prior.outputs.pattern reference, correctly excluding the new factory-build-plan artifact from work/prior.
No findings from: security.
— hive: agent=reviewer backend=copilot model=claude-fable-5
Review findings on projectbluefin#195. spec_dependents crashed with IndexError on a '%package ' line carrying only whitespace, or '%package -n' with no name: sub.split()[0] indexed an empty list. That runs in the prepare job, so one malformed spec took the whole pipeline down. Reproduced at head with a one-line spec; an unparseable directive declares no subpackage and is now skipped, and a test covers all three shapes alongside a real subpackage to prove the surviving edge. rebuild_matrix bucketed every selected-but-published-and-unchanged package as a reverse dependency, so stale selections inflated reverse_deps in build-plan.json and the markdown summary. Stale packages now have their own bucket in both, since a stale build is downstream of nothing this run edited -- its build root moved under it. Per-package reasons were already correct and are unchanged. GLOBAL_IGNORE_PATTERNS ignored all of tools/ but mock_config.py, so edits to source_pipeline.py or generated_sources.py selected no rebuild, against the projectbluefin#23 criterion that global source tooling must. Both are now global triggers: build-stage.yml runs source_pipeline.py for every package, and generated_sources.py produces the Source0 bytes for the recipes with no verbatim upstream archive, so either changes what a build's sources are without any recipe diff recording it. Also removed the unreachable 'elif token.startswith("pkgconfig(")' branch, which re-tested the membership the preceding if had already decided, and updated docs/architecture.md's prepare row, which still described selection as 'new, changed, or requested by a full rebuild' with no mention of the reverse-dep closure, the global triggers or the factory-build-plan artifact. just check and just test pass: 413 tests, 2133 subtests. Refs projectbluefin#195 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
- tools/rebuild_plan.py: calculate reverse dependency closures for changed packages - tools/rebuild_matrix.py: bucket reverse dependencies and stale rebuilds separately - .github/workflows/build-stage.yml: stage artifacts excluding build plan - .github/workflows/rebuild-rpms.yml: upload factory-build-plan artifact - tests/test_rebuild_plan.py: comprehensive unit test coverage - docs/architecture.md: update pipeline description - .agents/skills/build-failure-triage/SKILL.md: skill documentation Co-authored-by: Bob Killen <mrbobbytables@users.noreply.github.com>
caf771a to
f00d9b9
Compare
There was a problem hiding this comment.
correctness
-
High —
prune_sourcesoutput deleted but the workflow still consumes it. This PR removesprunable_sourcesand theoutputs["prune_sources"]write fromtools/rebuild_matrix.py, but at head.github/workflows/rebuild-rpms.yml:90still mapssteps.matrix.outputs.prune_sources, which is now the empty string. Two consequences:- The publish gate at
rebuild-rpms.yml:610(needs.prepare.outputs.prune_sources != '[]') is now always true, so the publish job runs on every run — including no-op runs — republishing the repository each time. The deleted docstring intools/rebuild_plan.pyexisted precisely to keep cleanup-only runs idempotent. - The Hummingbird-owned RPM pruning at
rebuild-rpms.yml:703-720becomes a permanent silent no-op (jq -r '.[]' <<<""yields an empty array), so RPMs from recipes handed over to Hummingbird now persist in the published OCI repository forever.config/hummingbird-provided-sources.jsonis left orphaned.
Either restore the output or remove its consumers in the same change.
- The publish gate at
intent-alignment
- Medium — undisclosed removal of the source-pruning feature and unrelated doc deletions. Issue #23 is about reverse-dependency selection; nothing in it or the PR body mentions deleting
prunable_sources,HUMMINGBIRD_OWNED, andHummingbirdOwnershipTests(tests/test_rebuild_plan.py), nor deleting the "Trigger and merge-queue policy" section fromdocs/architecture.md(18 lines) or the "job name in the run list is the chunk, not the package" triage guidance from.agents/skills/build-failure-triage/SKILL.md(20 lines). The SKILL.md guidance documented a real triage trap (issue #212) and is unrelated to this feature. Conversely, thepriorstep added tobuild-stage.yml:65-69fixes a danglingsteps.prior.outputs.patternreference that exists in base (build-stage.yml:81) — a good fix, but also undisclosed.
docs-currency
- Low — stale operator-facing text left behind. The comment at
rebuild-rpms.yml:704-710still says "The prepare step computes this list from the repository it actually witnessed" — no longer true onceprune_sourcesstops being emitted. Alsodocs/architecture.md:155, rewritten by this PR, still claims prepare "emits five stage lists" and that "a package declaring a stage above 4 has no job to run in," while the workflow hasrebuild0–rebuild10(rebuild-rpms.yml:329-489) andSTAGES = 11(tools/rebuild_plan.py:54).
No findings from: security, style.
— hive: agent=reviewer backend=copilot model=claude-fable-5
Danathar
left a comment
There was a problem hiding this comment.
This PR is the reverse-dependency closure and build-plan report for rebuild-rpms.yml. The tests pass at f00d9b9 and on a trial merge with main, and the closure logic reads correctly, but one removal breaks the publish job and is not mentioned in the PR body.
tools/rebuild_matrix.py no longer writes outputs["prune_sources"] and tools/rebuild_plan.py drops prunable_sources() and its test. The workflow still consumes that output at head:
.github/workflows/rebuild-rpms.yml:90 prune_sources: ${{ steps.matrix.outputs.prune_sources }}
.github/workflows/rebuild-rpms.yml:610 || needs.prepare.outputs.prune_sources != '[]')
.github/workflows/rebuild-rpms.yml:703 PRUNE_SOURCES: ${{ needs.prepare.outputs.prune_sources }}
With the output missing, prune_sources is the empty string. Two effects I can see: the publish gate at line 610 becomes always true, so publish runs on no-op runs and republishes the repository every time; and the Hummingbird-owned prune loop at lines 711-720 gets an empty list, so RPMs from recipes deferred to Hummingbird stop being removed from the seeded repository. tests/test_factory_witness.py still asserts those three workflow lines, which is why CI stays green while the producer is gone.
Either restore the prune_sources output (keep prunable_sources and the HUMMINGBIRD_OWNED read in rebuild_matrix.py) or remove the three consumers and the witness test together, and say which in the PR body. Two smaller things while you are there: the comment at rebuild-rpms.yml:704-710 still says prepare computes the prune list, and docs/architecture.md:155 still says "five stage lists" and "a stage above 4" while STAGES is 11.
hanthor
left a comment
There was a problem hiding this comment.
Requesting changes: the deleted prune_sources output is still consumed by the workflow. Please restore or migrate the consumer.
… merge A push to main that changes packages/** or config/** now runs the factory, and the run builds only what changed since the published image plus the direct BuildRequires dependents of what changed, in waves solved from real BuildRequires. Everything else stays published and untouched. - tools/extract_buildrequires.sh runs rpmspec over every recipe in the build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each BuildRequires to the factory recipes that provide it (subpackage names, explicit Provides, and the published generated provides), orders a build set by longest path, and reverses the edges for selection. A config stage only orders the members of a BuildRequires cycle (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored. The solved graph is 12 waves deep where hand stages went to 10, so the workflow now has rebuild0..rebuild13. - tools/factory_state.py records on the published image, as the label org.projectbluefin.factory.state, the input digest of every build in it (recipe files, inventory entry, build-root pin, Hummingbird repo) and the packages that failed at their current digest. prepare selects against it, so a run is the union of every change since the last publication, and a push holds a package that already failed at exactly these inputs while the daily run retries it. An image without the label falls back to the diff and NEVR comparison. A label adds no layer. - Reverse dependencies are direct only: following BuildRequires transitively dragged 167 packages into a one-line libical fix, through tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle. A consumer two hops away that really breaks surfaces as a stale published build and is rebuilt then. - Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the same minute as bump-upstream-sources.yml; a merged bump now builds via push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized with cancel-in-progress false; a replaced pending run loses nothing because the next run selects against the published state. - prepare writes the plan, with the wave and the reason for every package, to the job summary, and uploads the graph as the build-graph artifact. The canary gains pass5: an incremental run against pass1's image, pinned by digest, with vulkan-headers perturbed. It must select exactly vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config stage 4), and leave libical alone; verify-publish now also checks the state label. Supersedes #195, which computed a transitive closure from a regex parse of the specs and dropped the Hummingbird-owned prune. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
… merge A push to main that changes packages/** or config/** now runs the factory, and the run builds only what changed since the published image plus the direct BuildRequires dependents of what changed, in waves solved from real BuildRequires. Everything else stays published and untouched. - tools/extract_buildrequires.sh runs rpmspec over every recipe in the build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each BuildRequires to the factory recipes that provide it (subpackage names, explicit Provides, and the published generated provides), orders a build set by longest path, and reverses the edges for selection. A config stage only orders the members of a BuildRequires cycle (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored. The solved graph is 12 waves deep where hand stages went to 10, so the workflow now has rebuild0..rebuild13. - tools/factory_state.py records on the published image, as the label org.projectbluefin.factory.state, the input digest of every build in it (recipe files, inventory entry, build-root pin, Hummingbird repo) and the packages that failed at their current digest. prepare selects against it, so a run is the union of every change since the last publication, and a push holds a package that already failed at exactly these inputs while the daily run retries it. An image without the label falls back to the diff and NEVR comparison. A label adds no layer. - Reverse dependencies are direct only: following BuildRequires transitively dragged 167 packages into a one-line libical fix, through tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle. A consumer two hops away that really breaks surfaces as a stale published build and is rebuilt then. - Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the same minute as bump-upstream-sources.yml; a merged bump now builds via push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized with cancel-in-progress false; a replaced pending run loses nothing because the next run selects against the published state. - prepare writes the plan, with the wave and the reason for every package, to the job summary, and uploads the graph as the build-graph artifact. The canary gains pass5: an incremental run against pass1's image, pinned by digest, with vulkan-headers perturbed. It must select exactly vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config stage 4), and leave libical alone; verify-publish now also checks the state label. Supersedes #195, which computed a transitive closure from a regex parse of the specs and dropped the Hummingbird-owned prune. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
… merge (#259) A push to main that changes packages/** or config/** now runs the factory, and the run builds only what changed since the published image plus the direct BuildRequires dependents of what changed, in waves solved from real BuildRequires. Everything else stays published and untouched. - tools/extract_buildrequires.sh runs rpmspec over every recipe in the build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each BuildRequires to the factory recipes that provide it (subpackage names, explicit Provides, and the published generated provides), orders a build set by longest path, and reverses the edges for selection. A config stage only orders the members of a BuildRequires cycle (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored. The solved graph is 12 waves deep where hand stages went to 10, so the workflow now has rebuild0..rebuild13. - tools/factory_state.py records on the published image, as the label org.projectbluefin.factory.state, the input digest of every build in it (recipe files, inventory entry, build-root pin, Hummingbird repo) and the packages that failed at their current digest. prepare selects against it, so a run is the union of every change since the last publication, and a push holds a package that already failed at exactly these inputs while the daily run retries it. An image without the label falls back to the diff and NEVR comparison. A label adds no layer. - Reverse dependencies are direct only: following BuildRequires transitively dragged 167 packages into a one-line libical fix, through tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle. A consumer two hops away that really breaks surfaces as a stale published build and is rebuilt then. - Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the same minute as bump-upstream-sources.yml; a merged bump now builds via push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized with cancel-in-progress false; a replaced pending run loses nothing because the next run selects against the published state. - prepare writes the plan, with the wave and the reason for every package, to the job summary, and uploads the graph as the build-graph artifact. The canary gains pass5: an incremental run against pass1's image, pinned by digest, with vulkan-headers perturbed. It must select exactly vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config stage 4), and leave libical alone; verify-publish now also checks the state label. Supersedes #195, which computed a transitive closure from a regex parse of the specs and dropped the Hummingbird-owned prune. Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
Superseded by #259, which solves waves and reverse dependencies from rpmspec BuildRequires in the build root (direct dependents; transitive closure measured at 167 packages for a libical fix) and selects against a per-package state label on the published image. |
Closes #23
Summary
Implements reverse dependency closure computation and reviewable build plan reporting for incremental factory rebuilds in
rebuild-rpms.yml:spec_dependents) parses declared packages, subpackages (%package,%package -n),Provides:, andpkgconfig(...)identifiers across recipes inpackages/to map factoryBuildRequiresedges.dependents_from_primary) extracts runtimeProvides/Requiresand sonames fromprimary.xml.merge_dependents) and computes the transitive reverse dependency closure (reverse_closure), ensuring editing an upstream or staged library selects all downstream factory dependants across stages in proper wave order.rebuild-rpms.yml,build-stage.yml), build actions,tools/mock_config.py, and repo configurations (is_global_change), selecting a full or policy-defined rebuild while recording the exact triggering paths.$GITHUB_STEP_SUMMARY.work/reports/build-plan.json) and Markdown (work/reports/build-plan.md) reports uploaded as thefactory-build-planworkflow artifact..agents/skills/build-failure-triage/SKILL.mdto document inspecting thefactory-build-planartifact and triaging reverse dependency closures.tests/test_rebuild_plan.pycover direct edits, version changes, new packages, global rebuild triggers, specBuildRequiresdependents, and report formatting.— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:755644a