Repository navigation
feat(publish): publish each good build, keep a failed package's previous one - #258
Merged
Merged
Conversation
…ous one Publication was all-or-nothing: one failed package (a flaky fish test) held back every other one, and 3 of 117 full runs published in four weeks. It now follows Fedora and Hummingbird: - publish seeds from the verified previous image and replaces, by source package name, the RPMs of every package this run built. A dropped or renamed subpackage leaves with its source; nothing else changes. - a package that failed, or built but lost precedence, keeps its previous published build, or stays absent if it never built. precedence names the losing sources in an output instead of failing the job. - the Hummingbird-only consumer transaction over the whole candidate still gates the tag; if it does not resolve, nothing moves. - a new report job, which runs whether or not publish did, names every failed package from the run's own artifact list in the job summary and, on main, opens, updates or closes "Factory: packages failing on main". - a run in which nothing built and nothing is pruned publishes nothing. tools/publish_gate.py encodes the rule (publish_allowed, assemble, failures_from_artifacts, render_report) and assert_gate_enforced now rejects a wave result in the publish job's if: while still requiring it to wait for every wave. tests/test_atomic_publish.py becomes test_incremental_publish.py: the failure-mode tests are inverted where the rule changed, and the repodata-first layer, seed-verification and transaction-order tests are kept as they were. The canary gains pass4: libical fails on purpose, and verify-failure checks the canary tag still moved, still carries pass1's libical, and that the run reported exactly libical as failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
…lishes around it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
hanthor
force-pushed
the
ci/incremental-publish
branch
from
September 25, 2026 18:16
50f6730 to
fca9f2a
Compare
Member
Author
Canary proof: run 36172410377, gate
|
| job | start → end (UTC) | |
|---|---|---|
| pass1 (build, precedence, publish + sign + attest, report) | 18:18:08 → 18:25:57 | 7 m 49 s |
| verify-publish (Utah's reader, 2 layers, cosign, attestation) | 18:26:02 → 18:26:15 | ✓ |
pass4 (vulkan-loader fails on purpose) |
18:26:01 → 18:30:34 | build job red by design; precedence ✓, publish ✓, report ✓ |
| verify-failure | 18:30:38 → 18:30:46 | ✓ failed == ["vulkan-loader"], new digest ≠ pass1's, image still carries pass1's vulkan-loader (Utah reader) |
| pass2 ∥ pass3 → verify-cache | → 18:32:01 | ✓ zero compiles / only the perturbed package compiled |
| Canary | 17:58 → 18:32:07 | 14 m 09 s first job to gate |
That is the incremental rule working end to end. One package failed, and the run still published a new signed, attested image that kept the failed package's previous build and named it as the one failure.
The workflow run is red because pass4's injected failure is a real failed job. Only the Canary gate is the required check. I'll mark injected failures continue-on-error in the next PR so the run reads green as well.
hanthor
marked this pull request as ready for review
September 25, 2026 18:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 2 of the incremental-factory work. Stacked on #257 (canary); retargets to
mainwhen that merges.Rule change
Publication was atomic: every selected wave had to succeed. One flaky test (fish) held back every package, and 3 of 117 full runs published in four weeks. Now, the way Fedora and Hummingbird do it:
precedencefails, nothing publishesprecedencesucceeds with alosersoutputUnchanged: the verified seed (cosign, prepare-time witness, repeated mistake 14), the two-layer repodata-first image (#254) and its test, sign + attest, the publish concurrency group.
Reporting
A new
reportjob (always(), after publish) names every selected package with norpm-s*-<pkg>artifact, plus precedence losers, in the job summary with what consumers get for each, and emits one::errorper package. Onmain(and never for the canary) it opens "Factory: packages failing on main", edits it each run, comments when the failing set changes, and closes it on the first run with no failures. If publish itself failed (transaction), the report says Not published.Tests
tools/publish_gate.pygainsassemble,apply,failures_from_artifacts,render_report;publish_allowednow takes wave results but ignores them.assert_gate_enforcedstill requiresneeds:on every wave (publish must not run around a wave still building) but rejects anyneeds.rebuildN.resultin theif:, and checks seed → replace → transaction → publish step order.tests/test_atomic_publish.py→tests/test_incremental_publish.py(git mv): the failed-wave tests are inverted to the new rule rather than deleted; the precedence, transaction, fork, layer-order and seed-verification tests are kept.Canary sub-case
The canary gains
pass4: the same set withvulkan-loader(the stage-4 consumer) failing on purpose (inject_failure, canary-only input), published to the canary tag.verify-failureasserts the tag moved, the image (read with Utah's own reader) still carries pass1'svulkan-loader, and the run'sfailedoutput is exactly["vulkan-loader"].Not in this PR
Proof
Canary run on this PR: numbers to follow.
🤖 Generated with Claude Code