Skip to content

ci: stop a testing build cancelling the one before it, which is losing the ISO evidence the release gate needs - #185

Merged
castrojo merged 1 commit into
mainfrom
claude/testing-builds-not-cancelled
Sep 22, 2026
Merged

castrojo merged 1 commit into
mainfrom
claude/testing-builds-not-cancelled

Conversation

@hanthor

@hanthor hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member

What is happening

build.yml sets cancel-in-progress: true for every ref. A run on testing is not a redundant one — it produces the images, and it is what dispatches post-testing-e2e, the ISO validation the release gate reads. When two pushes to testing land close together, the first build is killed mid-flight and its E2E dispatch dies with it.

Observed today, not theorised:

11:09:47  Build Utah  workflow_dispatch  testing  cancelled
11:11:40  Build Utah  workflow_dispatch  testing  cancelled
11:31:30  Build Utah  push               testing  (ran)

and the promotion that followed failed on precisely the missing evidence:

env:
  E2E_LAST_STATUS: skipped
  READY: false
##[error]Release gate is blocked. Review the promotion PR gate section for the failing check.

So the release gate was not wrong and the images were not broken. The run that would have produced the proof had been cancelled by the next one. That is a self-inflicted wound on the path from "image builds" to "image promoted", and it is the sort of failure that reads as a mysterious gate problem rather than as what it is.

The change

One line: cancellation applies to pull requests only. A PR genuinely does supersede itself — a new push means the previous run is answering a question nobody is asking. testing, schedules and dispatches now run to completion. The group still includes github.ref, so a testing build does not queue behind an unrelated branch's.

Relationship to #138

This is the uncontroversial half of the concurrency change proposed there. I requested changes on #138 over a different part of it — removing image builds from pull requests, which would have hidden two real failures I hit today — and this piece should not wait on that discussion, because it is actively costing the project its ISO validation. Credit for spotting the concurrency problem belongs to that PR; if it lands first, this becomes a no-op.

Verification

  • Ran 197 tests ... OK (skipped=1), with two new cases pinning the expression and the ref-scoped group.

🤖 Generated with Claude Code

https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M


Generated by Claude Code

@hanthor

hanthor commented Sep 19, 2026

Copy link
Copy Markdown
Member Author

Correcting the causal claim in my own PR body, because I got it wrong and it is the kind of wrong that sends the next reader down a dead end.

I wrote that today's promotion failed "on exactly the missing evidence" the cancelled builds would have produced. The gate's own rows say E2E is not missing — it is disabled by the caller (run_e2e: false in promote-testing-to-main.yml):

Check Status Details
Digest resolution ❌ failed Failed to resolve one or more digests from testing.
Cosign signatures ❌ failed Skipped because digest resolution failed.
E2E ⏭️ skipped E2E check disabled by caller.

The real blocker is digest resolution, and it is failing because no :testing tag exists at all:

utah                   :testing -> HTTP 404
utah-nvidia            :testing -> HTTP 404
utah-gaming            :testing -> HTTP 404
utah-nvidia-gaming     :testing -> HTTP 404

That tag is written by post-testing-e2e.yml's promote-to-testing job, which needs the luks matrix to pass; two of four flavors fail, so it is skipped. Full chain in my comment on #181.

What this does not change: the case for this PR. Cancelling a testing build still throws away a build and the ISO validation it dispatches, and two were cancelled within two minutes today. It is simply not the cause of this promotion failure, and I should not have said it was without reading the gate rows first. The one-line change stands on its own merits; only the second paragraph of the description is wrong, and I would rather leave it corrected here than quietly rewrite it.


Generated by Claude Code

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

Danathar
Danathar previously approved these changes Sep 19, 2026

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I checked: the two cancelled testing runs at 11:09 and 11:11 today are real, and the promotion that followed did fail with E2E_LAST_STATUS: skipped. The one-line change — cancel-in-progress: ${{ github.event_name == 'pull_request' }} — keeps the superseding behaviour for PRs, where it is wanted, and stops a testing push from killing the previous build and the E2E dispatch it owns. The concurrency group still includes github.ref, so a testing build does not queue behind an unrelated PR. Two tests pin both properties. All 14 checks green.

Same heads-up as on #184: the two PRs touch the same two files, so one of them will need a rebase after the other lands.

`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.

This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:

    11:09:47  Build Utah  workflow_dispatch  testing  cancelled
    11:11:40  Build Utah  workflow_dispatch  testing  cancelled
    11:31:30  Build Utah  push               testing  (ran)

and the promotion that followed failed on exactly the missing evidence:

    E2E_LAST_STATUS: skipped
    READY: false
    ##[error]Release gate is blocked.

So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.

A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.

This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased branch on origin/main, resolved merge conflict in tests/test_iso_ci.py, and verified with 'just check' (286 tests passed). contract and build_main CI jobs have completed successfully. Awaiting completion of kernel_cache and approval from an independent maintainer to land.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from correctness, security, intent-alignment, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5

@Danathar Danathar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at bf69a94. My earlier approval at b913021 was dismissed when castrojo rebased this onto main. I re-diffed the PR's own patch (head against its merge base) at both SHAs: the build.yml change and the ConcurrencyTests class are byte-identical, and the only difference is the hunk context in tests/test_iso_ci.py because the class now lands after ForkPullRequestKernelCacheTests from #184. python3 tests/run_suite.py at bf69a94 passes (286 tests, 1 skipped) and all CI checks are green.

@castrojo castrojo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second review: verified cancel-in-progress restriction to pull_request events in build.yml to preserve testing artifacts.

@castrojo
castrojo merged commit 6cf26a8 into main Sep 22, 2026
14 checks passed
@castrojo
castrojo deleted the claude/testing-builds-not-cancelled branch September 22, 2026 18:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants