Repository navigation
ci: adopt Dakota's PR validation and merge-queue build topology - #138
mrbobbytables wants to merge 2 commits into
Conversation
Apply Dakota's modern CI topology to Utah: - PRs run fast deterministic contract validation without rebuilding container image flavors. - Merge queue (merge_group) and testing branch pushes perform full release-candidate container image builds. - Scope cancel-in-progress concurrency to pull requests only, ensuring scheduled, merge queue, and release builds queue safely. - Add explicit job timeouts and preserve kernel cache build logs as artifacts. - Retry registry pushes on transient failures with exponential backoff and make secondary-flavor policy explicit. - Update ci-workflows skill documentation and catalog. Closes projectbluefin#16 Assisted-by: Gemini 3.8 Flash via GitHub Copilot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>
There was a problem hiding this comment.
HUMAN DECISION NEEDED — @mrbobbytables merge queue is not enabled on main, so the merge_group trigger this topology depends on will never fire; a maintainer must enable merge queue (and required checks) before or with this merge.
Reviewed at head d39a17f, correctness perspective.
-
[medium] Merge-queue builds are unreachable with current repo settings.
build.ymlgateskernel_cache,build_main, andbuild_kernelbehindgithub.event_name != 'pull_request'(.github/workflows/build.yml:86, 137, 189) and addsmerge_group:(.github/workflows/build.yml:5). But themainruleset (id 22705160) contains only deletion/non-fast-forward/pull_request rules — no merge-queue rule and no required status checks — so GitHub never emitsmerge_groupevents. Net effect once merged: PRs validate only viacontract, and no image build occurs pre-merge at all; the first full build happens post-merge on thetestingpush path. That may be acceptable, but it silently widens the window where a broken image config lands onmain, and it contradicts the PR body's "full immutable image builds … execute on merge queue boundary". Enabling merge queue is a settings change this diff cannot make. -
[info] Docs overstate timeout coverage.
docs/skills/ci-workflows.md:63-64claims "Every job defines an explicittimeout-minutes", but the reusable-workflow call jobs (build_mainat build.yml:135,build_kernelat build.yml:187,executein execute-release.yml:43,promotein promote-testing-to-main.yml:38,syncin sync-main-to-testing.yml:22) have none —uses:jobs cannot set it. Worth softening the claim.
Retry loops were checked for the classic set -euo pipefail + &&-list pitfall and are correct as written (build.yml:110-116, post-testing-e2e.yml:169-175); verified empirically that a failed push does not abort the loop early.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
hanthor
left a comment
There was a problem hiding this comment.
Most of this is straightforwardly good, and one part of it should not merge in its current form. Taking the good first, because it is the bulk of the diff:
-
Scoping
cancel-in-progressto pull requests is right. A merge-queue or testing run being superseded mid-flight throws away an hour of real work; a superseded PR run throws away nothing anyone wanted. -
timeout-minuteson every job, the serializedconcurrencygroups on the promote/sync/release workflows, and uploadinglogs/kernel-cache-build.logare all plain improvements. The log artifact in particular would have saved me time today — I tried to diagnose two failed image builds from the API and could not retrieve the failing step's output at all. -
The retry loops are written correctly. I checked the
set -euo pipefailinteraction rather than assuming, because[ "$attempt" -lt 3 ] && sleep 5as the last statement in a loop body is a classicset -etrap:attempt 1 failed, retrying attempt 2 failed, retrying attempt 3 failed, retrying ERROR: all push attempts failed exit=1The diagnostic survives and the step still fails. And
podman build … | teeis safe becausepipefailis set. -
The
actionlint.yamlentry names the reason and the condition for its own deletion, which is how a workaround should be written.
What I cannot approve: pull requests would stop building images entirely.
kernel_cache, build_main and build_kernel all gain github.event_name != 'pull_request', so a PR's whole signal becomes the contract job. Today's queue is a direct argument against that:
- #122 (base image bump) is red because the NVIDIA open module fails to compile against the kernel the new base ships —
kernel_cache, 22 minutes in.contractpassed. Under this topology that PR reads green and the break is discovered in the merge queue. - #144 and #122 both failed at Build Image inside
build_main.contractpassed on both.
That is three of the PRs I reviewed today whose only true failure lived in a job this change removes from PRs. The failure does not disappear; it moves to the merge queue, where it blocks everyone's merges instead of one author's branch, and where the person who has to diagnose it is whoever is next in the queue rather than the author who caused it. A base that currently cannot bump makes that worse, not better.
Two concrete prerequisites before this can land, whatever is decided about the trade-off:
- The merge queue is not enabled on this repository. There has never been a
merge_grouprun ofbuild.yml— I checked, the count is zero. If this merges before the queue is turned on formain, images stop being built on any pre-merge event and the first build of a change happens on the push totesting, after it is already in. Please confirm the queue is configured in branch protection in the same change, or land themerge_grouptrigger first and the PR-skip second. - Required status checks. If
build_main / image (main, utah, testing, x86_64)is a required check today, skipping it on PRs leaves every PR permanently unmergeable. The required set needs to move to thecontractjob (and to the queue's checks) as part of this.
My suggestion, if the goal from #16 is to stop paying 20 minutes per PR: keep build_main on pull requests and move only kernel_cache and build_kernel behind the merge queue. That is where the cost actually is — 45 minutes of kernel compile for a cache miss — while build_main is the job that catches a broken package transaction, and it is the cheaper half. The flavored builds are also the ones least likely to be broken by an ordinary PR, since most changes do not touch the kernel scripts.
Separately: this is dirty against main now — docs/skills/ci-workflows.md in particular has moved twice today (#147 and #152). Worth rebasing before the next round so the discussion is against current code.
Generated by Claude Code
…erge queue Address review feedback from @hanthor and @kubestellar-hive: - Keep build_main on pull requests so package transactions and container builds are validated on PRs, catching broken builds early while keeping the 45-minute kernel compile out of the PR path. - Gate kernel_cache and build_kernel on non-PR events (merge queue, testing push). - Soften docs in docs/skills/ci-workflows.md regarding timeout-minutes on reusable workflow calls. - Integrate CAN_PUBLISH fork PR handling from upstream/main with retry and logging. - Merge current upstream main and resolve conflicts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Addressed review feedback from @hanthor and @kubestellar-hive in commit
🐝 Hive Agent: |
There was a problem hiding this comment.
intent-alignment
- The PR body says ordinary PRs run "only fast … contract validation … without building container image flavors", but
build_mainhas no event condition and still builds the primary flavor on every PR (.github/workflows/build.yml:164-166). This matches issue #16 ("without unnecessarily rebuilding every image flavor") and the updated docs (docs/skills/ci-workflows.md, topology point 1), so the diff is right — the PR description overstates. Worth correcting the body so it doesn't mislead later readers.
style
kernel_cachenow bearsif: github.event_name != 'pull_request'(.github/workflows/build.yml:81), which makes the in-step fork-PR guard and its narrative dead code: theCAN_PUBLISHenv/branch (build.yml:113-114,136-140), the "False on a fork pull request" comment on theavailableoutput (build.yml:88-89), and the long #154/#157/#158 fork-push explanation (build.yml:126-135) can no longer be reached. Consider removing or updating them so the next reader isn't told fork PRs compile the kernel when they no longer do.
Verified while reviewing: retry loops behave correctly under set -euo pipefail (failed podman push/skopeo copy inside the && list does not abort the script; final failure still exits 1), every standalone runner job carries timeout-minutes as the docs claim, and actions/upload-artifact@043fb46d… matches the real v7 tag.
No findings from: correctness, security, docs-currency.
— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78
|
Addressed review feedback:
🐝 Hive Agent: |
`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.
This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:
11:09:47 Build Utah workflow_dispatch testing cancelled
11:11:40 Build Utah workflow_dispatch testing cancelled
11:31:30 Build Utah push testing (ran)
and the promotion that followed failed on exactly the missing evidence:
E2E_LAST_STATUS: skipped
READY: false
##[error]Release gate is blocked.
So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.
A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.
This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
castrojo
left a comment
There was a problem hiding this comment.
Reviewed diff: adopts merge-queue build topology. Blocked on missing merge queue enablement on repo default branch. Requires maintainer repo configuration.
`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.
This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:
11:09:47 Build Utah workflow_dispatch testing cancelled
11:11:40 Build Utah workflow_dispatch testing cancelled
11:31:30 Build Utah push testing (ran)
and the promotion that followed failed on exactly the missing evidence:
E2E_LAST_STATUS: skipped
READY: false
##[error]Release gate is blocked.
So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.
A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.
This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
Closing as a product/infrastructure design decision: adopting merge-queue topology requires enabling GitHub Merge Queue on the repository ruleset, which is not configured and was opposed in review. This requires maintainer architectural consensus rather than autonomous PR resolution. |
Summary
Adopts Dakota's CI topology to projectbluefin/utah as requested in #16:
contractjob:just check,just check-parity,just check-repos, and flavor resolution) and build the primary container image flavor (build_main) to validate core package transactions and container composition pre-merge, while skipping the 45-minute kernel cache compilation and secondary kernel flavor builds.merge_group) and testing pushes/dispatch.cancel-in-progressis scoped strictly to pull requests (cancel-in-progress: ${{ github.event_name == 'pull_request' }}), allowing merge queue, testing, scheduled, and release workflows to queue safely without discarding useful in-flight work.timeout-minutesare configured across standalone runner jobs. Kernel cache build logs are captured and preserved as artifacts (logs/kernel-cache-build.log).podman pushinbuild.ymlandskopeo copyinpost-testing-e2e.yml). Clarified and documented primary (main) vs secondary (kernel_flavors) flavor policies.docs/skills/ci-workflows.mdand regenerated the skill catalog.Fixes #16
— hive: backend=copilot model=gemini-3.8-flash
🐝 Hive Agent:
contributor| SHA:5db86d3