Skip to content

feat(publish): publish each good build, keep a failed package's previous one - #258

Merged
hanthor merged 3 commits into
mainfrom
ci/incremental-publish
Sep 25, 2026
Merged

hanthor merged 3 commits into
mainfrom
ci/incremental-publish

Conversation

@hanthor

@hanthor hanthor commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Item 2 of the incremental-factory work. Stacked on #257 (canary); retargets to main when that merges.

Rule change

Publication was atomic: every selected wave had to succeed. One flaky test (fish) held back every package, and 3 of 117 full runs published in four weeks. Now, the way Fedora and Hummingbird do it:

case before now
package builds published only if everything built replaces its own previous build, by source name (a dropped/renamed subpackage leaves with it)
package fails to build nothing publishes keeps its previous published build, or stays absent if it never built; named in the summary and tracking issue
package loses precedence precedence fails, nothing publishes kept out, keeps previous build, named as failed; precedence succeeds with a losers output
Hummingbird-only transaction does not resolve nothing publishes nothing publishes (unchanged; this is what protects Utah)
nothing built, nothing to prune n/a nothing is republished

Unchanged: the verified seed (cosign, prepare-time witness, repeated mistake 14), the two-layer repodata-first image (#254) and its test, sign + attest, the publish concurrency group.

Reporting

A new report job (always(), after publish) names every selected package with no rpm-s*-<pkg> artifact, plus precedence losers, in the job summary with what consumers get for each, and emits one ::error per package. On main (and never for the canary) it opens "Factory: packages failing on main", edits it each run, comments when the failing set changes, and closes it on the first run with no failures. If publish itself failed (transaction), the report says Not published.

Tests

tools/publish_gate.py gains assemble, apply, failures_from_artifacts, render_report; publish_allowed now takes wave results but ignores them. assert_gate_enforced still requires needs: on every wave (publish must not run around a wave still building) but rejects any needs.rebuildN.result in the if:, and checks seed → replace → transaction → publish step order.

tests/test_atomic_publish.py → tests/test_incremental_publish.py (git mv): the failed-wave tests are inverted to the new rule rather than deleted; the precedence, transaction, fork, layer-order and seed-verification tests are kept.

Canary sub-case

The canary gains pass4: the same set with vulkan-loader (the stage-4 consumer) failing on purpose (inject_failure, canary-only input), published to the canary tag. verify-failure asserts the tag moved, the image (read with Utah's own reader) still carries pass1's vulkan-loader, and the run's failed output is exactly ["vulkan-loader"].

Not in this PR

  • If a library builds but a consumer rebuilt against it fails, the old consumer stays next to the new library; a soname bump then fails the transaction and nothing publishes (the stated rule). A hold-back pass (retry the candidate without providers whose consumers failed) is a possible follow-up.

Proof

Canary run on this PR: numbers to follow.

🤖 Generated with Claude Code

hanthor and others added 3 commits September 25, 2026 18:16
…ous one

Publication was all-or-nothing: one failed package (a flaky fish test)
held back every other one, and 3 of 117 full runs published in four weeks.
It now follows Fedora and Hummingbird:

- publish seeds from the verified previous image and replaces, by source
  package name, the RPMs of every package this run built. A dropped or
  renamed subpackage leaves with its source; nothing else changes.
- a package that failed, or built but lost precedence, keeps its previous
  published build, or stays absent if it never built. precedence names the
  losing sources in an output instead of failing the job.
- the Hummingbird-only consumer transaction over the whole candidate still
  gates the tag; if it does not resolve, nothing moves.
- a new report job, which runs whether or not publish did, names every
  failed package from the run's own artifact list in the job summary and,
  on main, opens, updates or closes "Factory: packages failing on main".
- a run in which nothing built and nothing is pruned publishes nothing.

tools/publish_gate.py encodes the rule (publish_allowed, assemble,
failures_from_artifacts, render_report) and assert_gate_enforced now
rejects a wave result in the publish job's if: while still requiring it to
wait for every wave. tests/test_atomic_publish.py becomes
test_incremental_publish.py: the failure-mode tests are inverted where the
rule changed, and the repodata-first layer, seed-verification and
transaction-order tests are kept as they were.

The canary gains pass4: libical fails on purpose, and verify-failure checks
the canary tag still moved, still carries pass1's libical, and that the run
reported exactly libical as failed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
…lishes around it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QMTLsnqW2jtz5QsNC3W3qv
@hanthor

hanthor commented Sep 25, 2026

Copy link
Copy Markdown
Member Author

Canary proof: run 36172410377, gate Canary green

job start → end (UTC)
pass1 (build, precedence, publish + sign + attest, report) 18:18:08 → 18:25:57 7 m 49 s
verify-publish (Utah's reader, 2 layers, cosign, attestation) 18:26:02 → 18:26:15 ✓
pass4 (vulkan-loader fails on purpose) 18:26:01 → 18:30:34 build job red by design; precedence ✓, publish ✓, report ✓
verify-failure 18:30:38 → 18:30:46 ✓ failed == ["vulkan-loader"], new digest ≠ pass1's, image still carries pass1's vulkan-loader (Utah reader)
pass2 ∥ pass3 → verify-cache → 18:32:01 ✓ zero compiles / only the perturbed package compiled
Canary 17:58 → 18:32:07 14 m 09 s first job to gate

That is the incremental rule working end to end. One package failed, and the run still published a new signed, attested image that kept the failed package's previous build and named it as the one failure.

The workflow run is red because pass4's injected failure is a real failed job. Only the Canary gate is the required check. I'll mark injected failures continue-on-error in the next PR so the run reads green as well.

@hanthor
hanthor marked this pull request as ready for review September 25, 2026 18:32
@hanthor
hanthor merged commit bc0604b into main Sep 25, 2026
79 of 80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant