Skip to content

fix: shard rebuild stages past GitHub's 256-job matrix cap - #52

Closed
hanthor wants to merge 2 commits into
mainfrom
fix/rebuild-matrix-cap
Closed

hanthor wants to merge 2 commits into
mainfrom
fix/rebuild-matrix-cap

Conversation

@hanthor

@hanthor hanthor commented Sep 6, 2026

Copy link
Copy Markdown
Member

The bug

GitHub caps a workflow matrix at 256 jobs. When a stage exceeds it, the
matrix produces no jobs at all — not an error, not a skip — and because the
later stages gate on !cancelled(), they run anyway against an empty
buildroot
.

A full rebuild of the current inventory queues 266 packages in stage 0, so
rebuild0 silently vanished and every stage-1 build resolved against Fedora
instead of the factory. That is why a large rebuild shows failures like
gdk-pixbuf2 needing glycin >= 2.2.alpha.7 but getting Fedora's 2.1.5: the
glycin this repo builds at stage 0 never happened. The failures are the empty
buildroot, not the recipes — the one package that did land a stage-1 job built
and passed.

Main hasn't hit this because almost everything is already published, so its
incremental stage 0 stays small. It bites the moment a stage needs to build
more than 256 packages at once — a big PR, or workflow_dispatch with
full=true.

The fix

The per-package build body moves verbatim into a reusable workflow,
rebuild-stage.yml, called once per ≤250-package chunk. prepare emits each
stage as a list of chunks; the caller matrix iterates the chunks (two jobs for
a 266-package stage) and the reusable workflow expands the per-package matrix
within each. No single matrix exceeds the cap, and per-package granularity and
parallelism are unchanged.

A side effect: the five near-identical 217-line stage blocks collapse to one
reusable definition, so the file drops 1582 → 585 lines. The fragile
bash -exc script the comments warn must never contain an apostrophe now
exists once instead of five times, and check_workflow_quoting.py — which
checked only rebuild-rpms.yml — now checks every workflow, since the script
moved.

Verification

Local: validate.py, all 42 tests, both workflows parse, the quoting check
passes, and the chunk handoff round-trips (266 → 250 + 16, each ≤256).

Not verified locally, needs a CI run: reusable-workflow + matrix +
artifact-merge behaviour. This is the change that can only be confirmed by
Actions actually running it.

Unblocks #51 (adding 152 recipes), whose CI failed on exactly this — its cold
stage 0 is 266.

🤖 Generated with Claude Code

https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9

GitHub caps a workflow matrix at 256 jobs. When a stage exceeds it the
matrix produces no jobs at all -- not an error, not a skip -- and because
the later stages gate on !cancelled(), they run anyway against an empty
buildroot. A full rebuild of the 345-package inventory queues 266 in
stage 0, so rebuild0 silently vanished and every stage-1 build resolved
against Fedora instead of the factory: gdk-pixbuf2 got Fedora's glycin
2.1.5 rather than the 2.2 this repo builds, and so on down the wave.

The per-package build body moves verbatim into a reusable workflow,
rebuild-stage.yml, called once per <=250-package chunk. prepare now emits
each stage as a list of chunks; the caller matrix iterates the chunks
(two jobs for a 266-package stage) and the reusable workflow expands the
per-package matrix within each, so no single matrix exceeds the cap and
per-package granularity is kept. 250 leaves headroom for the caller job.

This removes the five near-identical 217-line stage blocks -- the file
drops from 1582 lines to 585 -- so the fragile bash build script, the one
the comments warn must never contain an apostrophe, now exists once
rather than five times. check_workflow_quoting.py followed it: it checked
only rebuild-rpms.yml and now checks every workflow, since the script
lives in the reusable one.

Verified locally: validate.py, all 42 tests, both workflows parse, the
quoting check passes, and the chunk handoff round-trips (266 -> 250 + 16,
each <=256). Not verified: reusable-workflow + matrix + artifact-merge
behaviour, which only a CI run confirms.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015VF8ttMH6WGQHeQ9hdMSY9
@hivecommons-hive hivecommons-hive Bot added the lgtm Approved by a Hive merger/owner for auto-merge on green CI label Sep 7, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by @castrojo for Hive auto-merge on green CI.

Signed-off-by: sec-check[bot] <sec-check[bot]@users.noreply.github.com>
@hanthor

hanthor commented Sep 8, 2026

Copy link
Copy Markdown
Member Author

Superseded by #51, which now carries an equivalent fix on its own branch.

#51 refactors the rebuild into a reusable build-stage.yml and fans each stage
out over chunks of 250 via stage<N>_chunks, so no single matrix exceeds the
256-job cap. Verified in CI rather than by inspection: before the change,
rebuild0 produced zero jobs against 266 stage-0 packages; after it, the
same run spawned all 266 across two chunks (250 + 16).

That is the same outcome this PR was opened for, so merging both would only
conflict — #51 and this branch refactor the same file in parallel, and a merge
of #52 into #51 produced conflicts in rebuild-rpms.yml and
tools/check_workflow_quoting.py.

Two things from here are worth keeping regardless of which lands:

Closing as superseded rather than merging. Reopen if you would rather land this
shape and rebase #51 onto it.

@hanthor hanthor closed this Sep 8, 2026
hanthor added a commit to hanthor/utah-packages that referenced this pull request Sep 28, 2026
…ures

Merges the 151-recipe contract from projectbluefin#51 (which carries projectbluefin#52's matrix-cap
sharding and eleven stages) and fixes every non-Firefox failure from run
34415980185.

**The build root could not see the published repository.** `prepare` skips
any package already published at the same version, but the [factory] repo
block is guarded by `vars.FACTORY_REPO`, which is unset on pull_request
runs -- so it never ran, and a skipped package was simply absent. That is
why pipewire-libs-extra died on

    No match for argument: pkgconfig(libfreeaptx)

while libfreeaptx-devel sat published and skipped. FACTORY_REPO now
defaults to the same GitHub Pages URL `prepare` reads to decide what to
skip, so the two halves of that decision cannot disagree.

**Two recipes were staged before what they need.**

- xorg-x11-server-Xwayland 3 -> 5. It needs wayland-client >= 1.26.0 and
  the buildroot had Fedora's 1.25.0; this factory builds wayland at 4.
  Its three stage-0 build consumers resolve Xwayland from Fedora, so
  moving it later changes nothing for them.
- evolution-ews 6 -> 10. It needs libebook/libecal/libedata-* >= 3.61.3,
  and evolution-data-server builds at 9.

**gstreamer1-plugins-bad-free declines chromaprint**, on exactly the
reasoning ffmpeg.spec already records: Fedora's libchromaprint links
libavutil.so.60, this factory's ffmpeg 9.0.1 provides so.61, and the
factory excludes the Fedora libav by name, so libchromaprint-devel is
uninstallable here. The spec disabled chromaprint only under
`%if %{without extras}`; it is now its own bcond, off, with the
BuildRequires and the plugin in %files following it.

gcc was cancelled with the run, not failed, and needs only a re-run.

Inventory 344 -> 343 with Firefox dropped; validate, the runtime
contract, the quoting check and all 73 tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lgtm Approved by a Hive merger/owner for auto-merge on green CI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant