Skip to content

ci: adopt Dakota's PR validation and merge-queue build topology - #138

Closed
mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:ci/dakota-pr-validation-merge-queue
Closed

mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:ci/dakota-pr-validation-merge-queue

Conversation

@mrbobbytables

@mrbobbytables mrbobbytables commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adopts Dakota's CI topology to projectbluefin/utah as requested in #16:

  • Fast PR Validation: Ordinary PRs run fast manifest, workflow, source, and contract validation (contract job: just check, just check-parity, just check-repos, and flavor resolution) and build the primary container image flavor (build_main) to validate core package transactions and container composition pre-merge, while skipping the 45-minute kernel cache compilation and secondary kernel flavor builds.
  • Merge Queue Release-Candidate Builds: Full immutable image builds, kernel cache builds, and secondary flavor matrices execute on merge queue boundary (merge_group) and testing pushes/dispatch.
  • Release-Oriented Concurrency: cancel-in-progress is scoped strictly to pull requests (cancel-in-progress: ${{ github.event_name == 'pull_request' }}), allowing merge queue, testing, scheduled, and release workflows to queue safely without discarding useful in-flight work.
  • Explicit Timeouts and Artifact Logging: Explicit timeout-minutes are configured across standalone runner jobs. Kernel cache build logs are captured and preserved as artifacts (logs/kernel-cache-build.log).
  • Registry Push Retries & Secondary-Flavor Policy: Added retry loops with backoff to registry push steps (podman push in build.yml and skopeo copy in post-testing-e2e.yml). Clarified and documented primary (main) vs secondary (kernel_flavors) flavor policies.
  • Documentation & Skills: Updated docs/skills/ci-workflows.md and regenerated the skill catalog.

Fixes #16

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 5db86d3

Apply Dakota's modern CI topology to Utah:
- PRs run fast deterministic contract validation without rebuilding
  container image flavors.
- Merge queue (merge_group) and testing branch pushes perform full
  release-candidate container image builds.
- Scope cancel-in-progress concurrency to pull requests only, ensuring
  scheduled, merge queue, and release builds queue safely.
- Add explicit job timeouts and preserve kernel cache build logs as artifacts.
- Retry registry pushes on transient failures with exponential backoff and
  make secondary-flavor policy explicit.
- Update ci-workflows skill documentation and catalog.

Closes projectbluefin#16

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HUMAN DECISION NEEDED — @mrbobbytables merge queue is not enabled on main, so the merge_group trigger this topology depends on will never fire; a maintainer must enable merge queue (and required checks) before or with this merge.

Reviewed at head d39a17f, correctness perspective.

  1. [medium] Merge-queue builds are unreachable with current repo settings. build.yml gates kernel_cache, build_main, and build_kernel behind github.event_name != 'pull_request' (.github/workflows/build.yml:86, 137, 189) and adds merge_group: (.github/workflows/build.yml:5). But the main ruleset (id 22705160) contains only deletion/non-fast-forward/pull_request rules — no merge-queue rule and no required status checks — so GitHub never emits merge_group events. Net effect once merged: PRs validate only via contract, and no image build occurs pre-merge at all; the first full build happens post-merge on the testing push path. That may be acceptable, but it silently widens the window where a broken image config lands on main, and it contradicts the PR body's "full immutable image builds … execute on merge queue boundary". Enabling merge queue is a settings change this diff cannot make.

  2. [info] Docs overstate timeout coverage. docs/skills/ci-workflows.md:63-64 claims "Every job defines an explicit timeout-minutes", but the reusable-workflow call jobs (build_main at build.yml:135, build_kernel at build.yml:187, execute in execute-release.yml:43, promote in promote-testing-to-main.yml:38, sync in sync-main-to-testing.yml:22) have none — uses: jobs cannot set it. Worth softening the claim.

Retry loops were checked for the classic set -euo pipefail + &&-list pitfall and are correct as written (build.yml:110-116, post-testing-e2e.yml:169-175); verified empirically that a failed push does not abort the loop early.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of this is straightforwardly good, and one part of it should not merge in its current form. Taking the good first, because it is the bulk of the diff:

  • Scoping cancel-in-progress to pull requests is right. A merge-queue or testing run being superseded mid-flight throws away an hour of real work; a superseded PR run throws away nothing anyone wanted.

  • timeout-minutes on every job, the serialized concurrency groups on the promote/sync/release workflows, and uploading logs/kernel-cache-build.log are all plain improvements. The log artifact in particular would have saved me time today — I tried to diagnose two failed image builds from the API and could not retrieve the failing step's output at all.

  • The retry loops are written correctly. I checked the set -euo pipefail interaction rather than assuming, because [ "$attempt" -lt 3 ] && sleep 5 as the last statement in a loop body is a classic set -e trap:

    attempt 1 failed, retrying
    attempt 2 failed, retrying
    attempt 3 failed, retrying
    ERROR: all push attempts failed
    exit=1
    

    The diagnostic survives and the step still fails. And podman build … | tee is safe because pipefail is set.

  • The actionlint.yaml entry names the reason and the condition for its own deletion, which is how a workaround should be written.

What I cannot approve: pull requests would stop building images entirely.

kernel_cache, build_main and build_kernel all gain github.event_name != 'pull_request', so a PR's whole signal becomes the contract job. Today's queue is a direct argument against that:

  • #122 (base image bump) is red because the NVIDIA open module fails to compile against the kernel the new base ships — kernel_cache, 22 minutes in. contract passed. Under this topology that PR reads green and the break is discovered in the merge queue.
  • #144 and #122 both failed at Build Image inside build_main. contract passed on both.

That is three of the PRs I reviewed today whose only true failure lived in a job this change removes from PRs. The failure does not disappear; it moves to the merge queue, where it blocks everyone's merges instead of one author's branch, and where the person who has to diagnose it is whoever is next in the queue rather than the author who caused it. A base that currently cannot bump makes that worse, not better.

Two concrete prerequisites before this can land, whatever is decided about the trade-off:

  1. The merge queue is not enabled on this repository. There has never been a merge_group run of build.yml — I checked, the count is zero. If this merges before the queue is turned on for main, images stop being built on any pre-merge event and the first build of a change happens on the push to testing, after it is already in. Please confirm the queue is configured in branch protection in the same change, or land the merge_group trigger first and the PR-skip second.
  2. Required status checks. If build_main / image (main, utah, testing, x86_64) is a required check today, skipping it on PRs leaves every PR permanently unmergeable. The required set needs to move to the contract job (and to the queue's checks) as part of this.

My suggestion, if the goal from #16 is to stop paying 20 minutes per PR: keep build_main on pull requests and move only kernel_cache and build_kernel behind the merge queue. That is where the cost actually is — 45 minutes of kernel compile for a cache miss — while build_main is the job that catches a broken package transaction, and it is the cheaper half. The flavored builds are also the ones least likely to be broken by an ordinary PR, since most changes do not touch the kernel scripts.

Separately: this is dirty against main now — docs/skills/ci-workflows.md in particular has moved twice today (#147 and #152). Worth rebasing before the next round so the discussion is against current code.


Generated by Claude Code

…erge queue

Address review feedback from @hanthor and @kubestellar-hive:
- Keep build_main on pull requests so package transactions and container builds
  are validated on PRs, catching broken builds early while keeping the 45-minute
  kernel compile out of the PR path.
- Gate kernel_cache and build_kernel on non-PR events (merge queue, testing push).
- Soften docs in docs/skills/ci-workflows.md regarding timeout-minutes on reusable workflow calls.
- Integrate CAN_PUBLISH fork PR handling from upstream/main with retry and logging.
- Merge current upstream main and resolve conflicts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback from @hanthor and @kubestellar-hive in commit 5db86d3:

  1. Keep build_main on Pull Requests: Restored build_main execution on PRs so package transactions and container composition are verified pre-merge on all PRs. Only kernel_cache and build_kernel remain gated behind github.event_name != 'pull_request', preventing the 45-minute kernel compile from delaying PR feedback while keeping required image checks functional.
  2. Timeout Documentation: Softened the timeout statement in docs/skills/ci-workflows.md to clarify that timeout-minutes is defined on standalone runner jobs, while reusable workflow calls inherit timeouts from the caller/reusable template.
  3. Upstream Merge: Merged upstream/main (incorporating fork PR CAN_PUBLISH handling) and resolved conflicts across workflows and documentation.

🐝 Hive Agent: contributor | SHA: unknown

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

intent-alignment

  • The PR body says ordinary PRs run "only fast … contract validation … without building container image flavors", but build_main has no event condition and still builds the primary flavor on every PR (.github/workflows/build.yml:164-166). This matches issue #16 ("without unnecessarily rebuilding every image flavor") and the updated docs (docs/skills/ci-workflows.md, topology point 1), so the diff is right — the PR description overstates. Worth correcting the body so it doesn't mislead later readers.

style

  • kernel_cache now bears if: github.event_name != 'pull_request' (.github/workflows/build.yml:81), which makes the in-step fork-PR guard and its narrative dead code: the CAN_PUBLISH env/branch (build.yml:113-114, 136-140), the "False on a fork pull request" comment on the available output (build.yml:88-89), and the long #154/#157/#158 fork-push explanation (build.yml:126-135) can no longer be reached. Consider removing or updating them so the next reader isn't told fork PRs compile the kernel when they no longer do.

Verified while reviewing: retry loops behave correctly under set -euo pipefail (failed podman push/skopeo copy inside the && list does not abort the script; final failure still exits 1), every standalone runner job carries timeout-minutes as the docs claim, and actions/upload-artifact@043fb46d… matches the real v7 tag.

No findings from: correctness, security, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@mrbobbytables

Copy link
Copy Markdown
Contributor Author

Addressed review feedback:

  1. PR Description Intent Alignment: Updated the PR summary to accurately reflect that PRs build the primary container image flavor (build_main) to catch broken package transactions and container composition early, while skipping the expensive kernel cache compilation and secondary kernel flavor builds until merge queue / testing pushes.
  2. CAN_PUBLISH in kernel_cache: Retained the CAN_PUBLISH guard and step assertions in build.yml because they are covered by the unit test suite (tests/test_iso_ci.py:ForkPullRequestKernelCacheTests) which validates the workflow contract established in feat(kernel): enable native AMD and Intel Xe GPU drivers in OGC kernel #154/fix(ci): gate on dracut errors and enable Trivy RPM scanning #157/fix: enable the btrfs kernel module in the OGC gaming-flavor kernel #158 for fork push safety across workflow events.

🐝 Hive Agent: contributor | SHA: unknown

castrojo pushed a commit that referenced this pull request Sep 21, 2026
`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.

This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:

    11:09:47  Build Utah  workflow_dispatch  testing  cancelled
    11:11:40  Build Utah  workflow_dispatch  testing  cancelled
    11:31:30  Build Utah  push               testing  (ran)

and the promotion that followed failed on exactly the missing evidence:

    E2E_LAST_STATUS: skipped
    READY: false
    ##[error]Release gate is blocked.

So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.

A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.

This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed diff: adopts merge-queue build topology. Blocked on missing merge queue enablement on repo default branch. Requires maintainer repo configuration.

castrojo pushed a commit that referenced this pull request Sep 22, 2026
`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.

This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:

    11:09:47  Build Utah  workflow_dispatch  testing  cancelled
    11:11:40  Build Utah  workflow_dispatch  testing  cancelled
    11:31:30  Build Utah  push               testing  (ran)

and the promotion that followed failed on exactly the missing evidence:

    E2E_LAST_STATUS: skipped
    READY: false
    ##[error]Release gate is blocked.

So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.

A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.

This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.


Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@castrojo

Copy link
Copy Markdown
Contributor

Closing as a product/infrastructure design decision: adopting merge-queue topology requires enabling GitHub Merge Queue on the repository ruleset, which is not configured and was opposed in review. This requires maintainer architectural consensus rather than autonomous PR resolution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: adopt Dakota's PR validation and merge-queue build topology

3 participants