Skip to content

feat(build): rebuild what changed and what BuildRequires it, on every merge - #259

Merged
hanthor merged 1 commit into
mainfrom
ci/incremental-builds
Sep 25, 2026
Merged

hanthor merged 1 commit into
mainfrom
ci/incremental-builds

Conversation

@hanthor

@hanthor hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member

Item 1 of the incremental-factory work. Stacked on #258 (incremental publish); retargets to main when that merges. Supersedes #195.

What a merge now does

A push to main that touches packages/** or config/** runs the factory. Prepare selects:

  1. every package whose input digest differs from the one recorded on the published image;
  2. every stale published build (unsatisfied Requires, as before);
  3. the direct BuildRequires dependents of both.

It orders them into waves solved from real BuildRequires. Everything else stays published and is not touched. The plan, with each package's wave and the reason it was picked, goes to the job summary. The graph is uploaded as the build-graph artifact.

How

piece what it does
tools/extract_buildrequires.sh rpmspec -q --buildrequires / --provides / --qf %{NAME} over every recipe, in the build root, 4-way parallel. All 397 specs parse; 4 min on 2 local cores, so about 2 min on the runner.
tools/build_graph.py maps each BuildRequires capability to the factory recipes that provide it: subpackage names and explicit Provides from rpmspec, plus the generated provides (sonames, pkgconfig(), files) from the published primary.xml. Solves waves by longest path over the edges inside the build set. stage is consulted only between members of one strongly connected component (the malcontent-bootstrap → flatpak → malcontent cycle-breaker). A cycle that config does not order builds side by side and is named in the log.
tools/factory_state.py the org.projectbluefin.factory.state label on the published image: inputs = input digest of every published build (recipe files + inventory entry + config/buildroot-image + config/hummingbird.repo); failed = packages whose last attempt at their current digest failed. Publish merges it: replaced → new digest, failed → keep the old entry and record the failure, trusted (everything prepare did not select) → current digest, pruned → removed. A label adds no layer, and the canary asserts the image is still exactly two layers.

With the state label, a run builds the union of every change since the last publication, however many runs the queue replaced. An image without the label (the first run after merge) falls back to the old diff plus NEVR comparison. The first labelled publish learns the state of untouched packages from prepare's trusted list.

Why direct reverse dependencies, not transitive

Measured on the real graph: following BuildRequires transitively selects 167 packages for a libical fix and 178 for vulkan-headers. The edges run through tool BuildRequires (47 recipes BuildRequire git for %autosetup -S git, 27 desktop-file-utils) and a 55-member BuildRequires cycle (cairo, gtk3, gobject-introspection, …). Taking direct dependents only, the median change selects 1 other package, 116 recipes have none, and the largest (gobject-introspection) selects 82. That is how Fedora and Hummingbird rebuild for an ABI change. A consumer two hops away that really does break shows up as a stale published build (an unsatisfied Requires) and is rebuilt then. Without the graph, the old transitive runtime closure is still the fallback.

Waves

The solved graph is 12 waves deep (hwdata → libdisplay-info → mesa → libglvnd → mesa-libGLU → gstreamer1-plugins-base → pulseaudio → pipewire → ffmpeg → localsearch → nautilus → nautilus-python). The hand stages went to 10, put 321 packages in stage 0 where the graph puts 194, and had missed real edges. The workflow now declares rebuild0…rebuild13, with publish_gate.STAGES, rebuild_plan.STAGES and the tests updated to match. One consequence: a package that used to build next to a same-stage provider now builds after it, which is repeated mistake 3's own rule. It may surface failures that were hidden before. Under incremental publish those stay per-package.

Triggers

trigger runs
push to main, packages/** or config/** incremental; holds a package that already failed at exactly its current inputs
daily 03:17 UTC (was 06:41, the same minute as bump-upstream-sources.yml) incremental, plus a retry of every failed package
weekly Sunday 01:23 UTC full, against the cache (safety net)
dispatch as daily, or full

Top-level concurrency: rebuild-rpms-<ref>-<prefix|factory>, cancel-in-progress: false, so every merge's run finishes. GitHub keeps one run pending and replaces an older pending run with a newer one. Nothing is lost when that happens, because the newer run selects against the published state. A merged bump PR now reaches the factory through the push trigger.

Canary

The canary gains pass5, an incremental run against pass1's image pinned by digest (a new factory_tag input that also takes sha256:…), with vulkan-headers perturbed. verify-incremental asserts it selected exactly vulkan-headers (wave 0) and vulkan-loader (wave 1, not its config stage 4), and left libical alone. Without the state label this pass would select nothing: there is no diff range and the NEVR did not move. verify-publish now also checks that the label records the set.

Also

  • fix: stage_outputs/overflow take the solved waves; reverse_closure takes a depth.
  • docs: AGENTS.md conventions, contributing.md, architecture.md (jobs, triggers, stage count, and the "Stage assignment" row moved out of "agreed, not yet built"), package-build-cache.md (per-merge triggers are now the design), repeated-mistakes 3 and 4, triage skill.

Proof

Canary on this PR, then a real one-package change on main with timings. Numbers to follow.

🤖 Generated with Claude Code

@hanthor
hanthor force-pushed the ci/incremental-builds branch from 74e5559 to 221dc34 Compare September 25, 2026 18:34
@hanthor
hanthor changed the base branch from ci/incremental-publish to main September 25, 2026 18:34
… merge

A push to main that changes packages/** or config/** now runs the factory,
and the run builds only what changed since the published image plus the
direct BuildRequires dependents of what changed, in waves solved from real
BuildRequires. Everything else stays published and untouched.

- tools/extract_buildrequires.sh runs rpmspec over every recipe in the
  build root (prepare, ~2 min on 4 cores); tools/build_graph.py maps each
  BuildRequires to the factory recipes that provide it (subpackage names,
  explicit Provides, and the published generated provides), orders a build
  set by longest path, and reverses the edges for selection. A config
  stage only orders the members of a BuildRequires cycle
  (malcontent-bootstrap -> flatpak -> malcontent); elsewhere it is ignored.
  The solved graph is 12 waves deep where hand stages went to 10, so the
  workflow now has rebuild0..rebuild13.
- tools/factory_state.py records on the published image, as the label
  org.projectbluefin.factory.state, the input digest of every build in it
  (recipe files, inventory entry, build-root pin, Hummingbird repo) and the
  packages that failed at their current digest. prepare selects against it,
  so a run is the union of every change since the last publication, and a
  push holds a package that already failed at exactly these inputs while
  the daily run retries it. An image without the label falls back to the
  diff and NEVR comparison. A label adds no layer.
- Reverse dependencies are direct only: following BuildRequires
  transitively dragged 167 packages into a one-line libical fix, through
  tool BuildRequires (47 recipes BuildRequire git) and a 55-package cycle.
  A consumer two hops away that really breaks surfaces as a stale published
  build and is rebuilt then.
- Triggers: push (packages/**, config/**), daily 03:17 (was 06:41, the
  same minute as bump-upstream-sources.yml; a merged bump now builds via
  push), weekly full rebuild Sunday 01:23. Runs on a ref are serialized
  with cancel-in-progress false; a replaced pending run loses nothing
  because the next run selects against the published state.
- prepare writes the plan, with the wave and the reason for every package,
  to the job summary, and uploads the graph as the build-graph artifact.

The canary gains pass5: an incremental run against pass1's image, pinned
by digest, with vulkan-headers perturbed. It must select exactly
vulkan-headers and vulkan-loader, in solved waves 0 and 1 (not config
stage 4), and leave libical alone; verify-publish now also checks the
state label.

Supersedes #195, which computed a transitive closure from a regex parse of
the specs and dropped the Hummingbird-owned prune.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
@hanthor
hanthor force-pushed the ci/incremental-builds branch from 221dc34 to 701f1e7 Compare September 25, 2026 18:55
@hanthor

hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Canary proof: run 36176501278, gate and workflow green

  • pass1: waves solved from BuildRequires. vulkan-loader built in rebuild1, not at its config stage 4, and prepare (graph extraction over all 397 specs included) took 1 m 19 s. verify-publish confirmed the image carries the org.projectbluefin.factory.state label for the set, still two layers.
  • pass5 (incremental against pass1's digest, vulkan-headers perturbed): selected exactly vulkan-headers (wave 0) and vulkan-loader (wave 1, "BuildRequires vulkan-headers") and left libical alone. verify-incremental ✓.
  • pass2/pass3 cache ✓; pass4 incremental publish + flaky %check retry ✓.
  • First job → gate: 18:55:42 → 19:11:59, 16 m 17 s.

Merging this adds the push trigger, which covers packages/** and config/** only. The merge itself touches neither, so it starts no factory run and leaves the in-flight run 36167998553 alone.

@hanthor
hanthor marked this pull request as ready for review September 25, 2026 19:13
@hanthor
hanthor merged commit 3b32158 into main Sep 25, 2026
108 of 109 checks passed
castrojo added a commit that referenced this pull request Sep 26, 2026
…ut format

Assisted-by: Gemini via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
castrojo added a commit that referenced this pull request Sep 27, 2026
…ild_matrix.py (#192)

* test: executed coverage for tools/rebuild_matrix.py — tests/test_rebuild_matrix.py

tools/rebuild_matrix.py is the driver .github/workflows/rebuild-rpms.yml runs
to decide which packages a rebuild builds, and it reported 0% of its 104
statements. tests/test_factory_witness.py names the file but only reads it as
text, so nothing ever executed it.

Adds tests/test_rebuild_matrix.py: 31 cases over changed_recipes,
changed_inventory, fetch_primary, hummingbird_baseurl, fetch_published and
main(). subprocess, urllib.request.urlopen and the module's ROOT are redirected
at the module object, so no case touches the network or a real git repository.

Covered decisions that had nothing holding them in place: the base-sha guard
that keeps a scheduled run from shelling out to an empty diff range; the
inventory reader that proves nothing rather than everything when the old config
is absent or unparseable; the mirror failure that rebuilds everything instead
of skipping; the stale and dependent-drag rebuild reasons; the >250-package
chunk split; and the SystemExit on a stage with no job.

Coverage for the module goes 0% -> 99%; the remaining line is the __main__
guard. Suite goes 396 -> 427 passed.

Assisted-by: Claude Opus 5 via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>

* fix(tests): adapt test_rebuild_matrix.py to hummingbird-provided-sources and prune repo fetch

* test: adapt test_rebuild_matrix.py to post-#259 wave and trusted output format

Assisted-by: Gemini via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: castrojo <castrojo@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant