Skip to content

πŸ› fix(ci): label events no longer cancel required checks on the same head - #18580

Merged
kubestellar-prow[bot] merged 1 commit into
v5from
fix/18578-ci-label-cancel
Nov 17, 2026
Merged

kubestellar-prow[bot] merged 1 commit into
v5from
fix/18578-ci-label-cancel

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #18578

Problem

verify PR contents (pr-verifier.yml), Sign-off survives the squash (dco-squash-attribution.yml) and changelog-fragment-guard are required on v5 and re-run on label events. All three used one concurrency group per PR. GitHub scores every check-run of a required name on a commit, so a single CANCELLED sibling leaves the PR BLOCKED, even with a green run beside it, and Tide retries it at the head of the queue.

Two concurrency behaviors produced those cancelled siblings:

  • cancel-in-progress (pr-verifier, dco-squash-attribution): each label flip (lgtm, approved, size, stage) cancelled the run in flight.
  • queue replacement (all three, including changelog-fragment-guard, which already had cancel-in-progress off for labels): a group holds at most one pending run, and the next run to queue cancels it even with cancel-in-progress: false. A burst of three or four labels therefore still cancelled queued runs.

Before / after

Before After
Group <workflow>-<PR> pr-verifier / dco: <workflow>-<PR>-<head SHA>; changelog guard: unchanged per-PR group
Label / edit / reopen re-checks share the group: cancel the in-flight run, or are dropped from the queue group gets -<action>-<run_id>: each re-check runs alone, never cancels and is never cancelled
Push to a new commit cancels old-commit runs (pr-verifier, dco) pr-verifier / dco: old-commit runs finish (10 / 5 min caps), with their verdict on the old SHA, which never merges. changelog guard: still cancels on synchronize
cancel-in-progress true on PR events (pr-verifier, dco) false (pr-verifier, dco). In a per-SHA group it could only ever cancel a same-head run

Trade-off: with the head SHA in the group, a new push can no longer cancel the previous commit's verifier and squash-attribution runs, because they are in a different group now. They run to completion instead. Both jobs are short and timeout-capped.

Label dependence (step 2 of the issue)

Other required checks (step 3)

Read-only review of the v5 required_status_checks. gate, build-and-test, test, coverage, golangci-lint, gosec, govulncheck come from docker.yml, v2-ci.yml, v2-tests.yml and go-security-analysis.yml. All of them use the default pull_request types, so they have no label triggers. DCO comes from an app. No change is needed there.

Guard

.github/scripts/test-required-check-concurrency.py, in Python + PyYAML unittest like test-native-queue-workflows.py, runs from pr-auto-update-test.yml. That workflow now also triggers on any .github/workflows/*.yml change. The guard parses every workflow, finds the jobs that produce a v5 required context, and evaluates their real concurrency expressions with a small GitHub-expression evaluator for a push followed by lgtm+approved-style label bursts on one head SHA. It fails if any two of those runs share a group, and the message names "cancels it in flight" or "replaces it in the queue". Self-tests check that the pre-#18578 shapes (per-PR group with cancel, head-SHA-only group, changelog guard's queue-replacement shape) are all flagged. The evaluator raises on any construct it cannot model; it never passes silently.

Workflow security: no change to what any workflow checks out or runs, and no change to permissions or action pins. Only concurrency: blocks, comments, and the test workflow's path filter and one step changed. actionlint is clean on every edited workflow.

Self-test on this PR

When every other check is green, this PR is labeled lgtm and approved together. Expected: both required checks stay green with no cancelled runs. The result is reported in a comment below.

πŸ€– Generated with Claude Code

…head

pr-verifier.yml, dco-squash-attribution.yml and changelog-fragment-guard.yml
shared a per-PR concurrency group across label events. Label bursts cancelled
the run in flight (cancel-in-progress) or dropped queued runs (queue
replacement), leaving CANCELLED check-runs for required contexts and jamming
Tide.

Key the verifier and squash-attribution groups on the head SHA, and give
labeled/unlabeled (plus edited/reopened/ready_for_review where triggered)
runs their own run_id-suffixed group so same-head re-checks never cancel or
get cancelled. Add test-required-check-concurrency.py, which evaluates every
required-check workflow's concurrency for a push plus a label burst and fails
on any shared group.

Fixes #18578

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 17, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

ℹ️ Sentinel: informational, author is trusted (owner), merge is not blocked. Head 32b06680af28 matched:

  • sensitive_path: changes 4 sensitive path(s) (.github/workflows/changelog-fragment-guard.yml, .github/workflows/dco-squash-attribution.yml, .github/workflows/pr-auto-update-test.yml, .github/workflows/pr-verifier.yml)

No sentinel-alert label was added. The finding is recorded in the dashboard Security tab and /api/sentinel; set sentinel.trusted_authors_block: true to block trusted authors too.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review at 32b0668: no in-scope blocker.

The concurrency groups in pr-verifier.yml, dco-squash-attribution.yml and changelog-fragment-guard.yml do what #18578 asks. Re-checks of the same head each get their own group, and nothing cancels a same-head run.

Advisory (low): .github/scripts/test-required-check-concurrency.py defines RECHECK_ACTIONS = ("labeled", "unlabeled"), so the guard never simulates edited, reopened or ready_for_review. The workflows list those actions in their own-group expression. A regression that dropped one of them would pass the guard. Consider deriving the re-check actions from each workflow's types:.

Note: label-dependent verdicts (dco merge-method/merge, changelog no-changelog) from a stale label run now complete instead of being cancelled. This is fine if GitHub takes the latest run per name. If it scores every run, as the PR body says, a stale red run could block. I did not verify this.

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by:

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective: no exploitable regression found in pr-verifier.yml, dco-squash-attribution.yml or changelog-fragment-guard.yml at 32b0668. Permissions and triggers are untouched. The new group key adds only the head SHA, a fixed action name and run_id. The SHA is hex and server-assigned, so a fork cannot inject into the key.

Advisory, low severity: pr-verifier.yml is pull_request_target with checks: write. After this change edited and ready_for_review get a unique group, and cancel-in-progress is false on every event. A fork author can edit their own PR title or body repeatedly, and each edit queues an uncancellable run of up to 10 minutes. Before this PR those edits cancelled the previous run. Cost is bounded by the job's timeout and GitHub's per-repo concurrency limits, but it is new cost amplification by a low-trust actor. If this matters, drop edited from the unique-group list and keep it in the shared per-SHA group. The verdict is not label-dependent, per the PR's own comment.

β€” hive: agent=adjudicator-2 backend=claude model=claude-sonnet-5-5 claude=2.1.325

@kubestellar-prow
kubestellar-prow Bot merged commit d0f587f into v5 Nov 17, 2026
105 of 108 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/18578-ci-label-cancel branch November 17, 2026 02:21
@clubanderson

Copy link
Copy Markdown
Member Author

Self-test result. lgtm and approved were added together at 01:35:30Z, after the push-time runs at 01:23Z. These are the check-runs on the head SHA:

  • Sign-off survives the squash (pull_request, so it ran this PR's new config): 5 runs, 5 success, 0 cancelled. The two label runs at 01:36:00Z and 01:36:11Z overlapped and both finished.
  • changelog-fragment-guard (pull_request, new config): 5 runs, 5 success, 0 cancelled. The label runs at 01:36:16Z and 01:36:36Z both finished.
  • verify PR contents: 2 success, 3 cancelled (01:23:04Z, 01:23:11Z, 01:35:36Z). This is expected for this PR. pr-verifier.yml runs on pull_request_target, which always uses the base branch (v5) copy of the workflow, so this PR's own verifier runs used the old cancelling config. The fix takes effect for verifier runs on PRs opened or labeled after this merge.

Tide merged once the surviving runs were green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

πŸ› CI: label events cancel in-flight required checks (verify PR contents, Sign-off survives the squash), jamming Tide

1 participant