Repository navigation
ci: stop a testing build cancelling the one before it, which is losing the ISO evidence the release gate needs - #185
Conversation
|
Correcting the causal claim in my own PR body, because I got it wrong and it is the kind of wrong that sends the next reader down a dead end. I wrote that today's promotion failed "on exactly the missing evidence" the cancelled builds would have produced. The gate's own rows say E2E is not missing — it is disabled by the caller (
The real blocker is digest resolution, and it is failing because no That tag is written by What this does not change: the case for this PR. Cancelling a testing build still throws away a build and the ISO validation it dispatches, and two were cancelled within two minutes today. It is simply not the cause of this promotion failure, and I should not have said it was without reading the gate rows first. The one-line change stands on its own merits; only the second paragraph of the description is wrong, and I would rather leave it corrected here than quietly rewrite it. Generated by Claude Code |
Danathar
left a comment
There was a problem hiding this comment.
What I checked: the two cancelled testing runs at 11:09 and 11:11 today are real, and the promotion that followed did fail with E2E_LAST_STATUS: skipped. The one-line change — cancel-in-progress: ${{ github.event_name == 'pull_request' }} — keeps the superseding behaviour for PRs, where it is wanted, and stops a testing push from killing the previous build and the E2E dispatch it owns. The concurrency group still includes github.ref, so a testing build does not queue behind an unrelated PR. Two tests pin both properties. All 14 checks green.
Same heads-up as on #184: the two PRs touch the same two files, so one of them will need a rebase after the other lands.
`cancel-in-progress: true` applies to every ref, so two pushes to `testing`
close together kill the first build mid-flight. That build is not a redundant
one: it produces the images, and it dispatches the ISO end-to-end validation
that the release gate reads. Killing it throws the evidence away.
This is not theoretical. Today, two testing builds were cancelled within two
minutes of each other:
11:09:47 Build Utah workflow_dispatch testing cancelled
11:11:40 Build Utah workflow_dispatch testing cancelled
11:31:30 Build Utah push testing (ran)
and the promotion that followed failed on exactly the missing evidence:
E2E_LAST_STATUS: skipped
READY: false
##[error]Release gate is blocked.
So the gate was not wrong and the images were not broken -- the run that would
have produced the proof had been cancelled by the next one.
A pull request genuinely does supersede itself, so cancellation stays there.
The group still includes github.ref, so a testing build does not queue behind
an unrelated branch's.
This is the uncontroversial half of the concurrency change proposed in #138. I
asked for changes on that PR over a different part of it -- taking image builds
off pull requests -- and this piece should not wait on that discussion, because
it is currently costing the project its ISO validation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
b913021 to
bf69a94
Compare
castrojo
left a comment
There was a problem hiding this comment.
Rebased branch on origin/main, resolved merge conflict in tests/test_iso_ci.py, and verified with 'just check' (286 tests passed). contract and build_main CI jobs have completed successfully. Awaiting completion of kernel_cache and approval from an independent maintainer to land.
Danathar
left a comment
There was a problem hiding this comment.
Re-approving at bf69a94. My earlier approval at b913021 was dismissed when castrojo rebased this onto main. I re-diffed the PR's own patch (head against its merge base) at both SHAs: the build.yml change and the ConcurrencyTests class are byte-identical, and the only difference is the hunk context in tests/test_iso_ci.py because the class now lands after ForkPullRequestKernelCacheTests from #184. python3 tests/run_suite.py at bf69a94 passes (286 tests, 1 skipped) and all CI checks are green.
castrojo
left a comment
There was a problem hiding this comment.
Second review: verified cancel-in-progress restriction to pull_request events in build.yml to preserve testing artifacts.
What is happening
build.ymlsetscancel-in-progress: truefor every ref. A run ontestingis not a redundant one — it produces the images, and it is what dispatchespost-testing-e2e, the ISO validation the release gate reads. When two pushes totestingland close together, the first build is killed mid-flight and its E2E dispatch dies with it.Observed today, not theorised:
and the promotion that followed failed on precisely the missing evidence:
So the release gate was not wrong and the images were not broken. The run that would have produced the proof had been cancelled by the next one. That is a self-inflicted wound on the path from "image builds" to "image promoted", and it is the sort of failure that reads as a mysterious gate problem rather than as what it is.
The change
One line: cancellation applies to pull requests only. A PR genuinely does supersede itself — a new push means the previous run is answering a question nobody is asking.
testing, schedules and dispatches now run to completion. The group still includesgithub.ref, so a testing build does not queue behind an unrelated branch's.Relationship to #138
This is the uncontroversial half of the concurrency change proposed there. I requested changes on #138 over a different part of it — removing image builds from pull requests, which would have hidden two real failures I hit today — and this piece should not wait on that discussion, because it is actively costing the project its ISO validation. Credit for spotting the concurrency problem belongs to that PR; if it lands first, this becomes a no-op.
Verification
Ran 197 tests ... OK (skipped=1), with two new cases pinning the expression and the ref-scoped group.🤖 Generated with Claude Code
https://claude.ai/code/session_01W6BYLUD6KBzF8rBFXKjP7M
Generated by Claude Code