Skip to content

πŸ› fix(ci): attribute check runs by PR, not by SHA alone - #16830

Merged
hivecommons-hive[bot] merged 6 commits into
hivecommons:v6from
Danathar:fix/ci-attribution-by-pr
Nov 5, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
hivecommons:v6from
Danathar:fix/ci-attribution-by-pr

Conversation

@Danathar

@Danathar Danathar commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Executive summary

projectbluefin/utah-packages#890 was sent back to its author as "Failing checks: Canary" for a Canary run that belonged to a different PR. #890 is an in-repo branch, and the fork PR #878 sat on the same commit, 95b9d22. #878's fork Canary run (run 42111306103) had failed. At 21:36:42Z, 41 seconds after #890 opened and before its own Canary had started, the contributor lane moved #890 to stage/072-returned-to-contributor and nudged its author. #890's own Canary later passed.

GitHub lists check runs per commit, so the hive took the newest Canary run on the SHA, whichever PR it came from. This PR attributes check runs to the PR being evaluated.

What

Red check run on the PR's head Before After
pull_requests names this PR counts counts
pull_requests names other PRs only counts ignored
pull_requests empty, workflow run on this PR's head repo and branch counts counts (push run, own fork run)
pull_requests empty, workflow run from another head repo or branch (#878's fork run; also an in-repo copy under the fork's branch name) counts ignored
either lookup fails counts counts
green or still running as is as is

Why

EvaluateCommitCI lists check runs for the SHA, keeps those on the head, then takes the latest per name and app. Two PRs at one head share that list. GitHub leaves pull_requests empty on a fork PR's runs, and the check-suite API names no head repository. The Actions workflow run behind the suite (GET actions/runs?check_suite_id=) does, so it is compared with the PR's head repository and branch. A suite from another app, with no workflow run, falls back to its head_branch.

How

  • checkRunsForPR in src/pkg/github/commit_ci.go runs right after checkRunsOnHead, only when the caller passes PRNumber. That covers the automerge lanes (commitCIForPRSkippingWith) and the merge-request CI gate.
  • Only completed red runs are judged, since only a red run can return a PR to its author.
  • The suite's head (the workflow run, or GET check-suites/{id} for another app) is read once per suite. The PR's head ref (GET pulls/{n}) is read once, and only when an unattributed red run exists.
  • Commit-only reads (no PR number) are unchanged.

Tests

Test Covers
TestCheckRunsAttributedByPR (9 rows: + same-branch-name in-repo copy, other-app fallback) #890 shape ignored; #878 sees its own; other-PR run ignored; own and push runs count; failed lookup keeps
TestCheckRunAttributionLookupsAreBounded no PR number: no filtering, no lookups; green runs cost no lookup
TestCheckRunAttributionCachesSuite two red runs, one suite: one suite read, one PR read
Mutants 13/13 caught
go test ./pkg/github/... ./pkg/writeback/ ./pkg/dashboard/ pass
check-docs-citations.py src/docs 2006 verified

Notes

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

projectbluefin/utah-packages#890 (in-repo, danathar/supersede-878) and the
fork PR hivecommons#878 shared head 95b9d22. hivecommons#878's fork Canary run (run 42111306103)
had failed; 41 seconds after hivecommons#890 opened, before its own Canary started,
the contributor lane sent hivecommons#890 to 072 for "Failing checks: Canary".

EvaluateCommitCI, when given a PR number, drops a red check run another PR
produced on the head: one whose pull_requests names other PRs but not this
one, or one naming none (fork runs) whose check suite ran for a different
head branch. Lookups are per suite, only for red unattributed runs, and a
failed lookup keeps the run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 5, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16816 πŸ› fix(adoption): refuse a fixer push that undoes the adopted PR's change same files src/docs/labels-and-control-signals/stage-labels.md
#16688 🌱 Forward-merge v5 into v6 (2 commits) same files src/docs/labels-and-control-signals/stage-labels.md

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head ccf0052) β€” the attribution logic and its callers (automerge_sweep_queued.go:405, merge_ci_gate.go:207) check out; failure modes fail closed as the body says. One advisory, no blocker.

Same branch name on both heads defeats the suite-branch fallback β€” src/pkg/github/commit_ci.go:1161-1163. For a fork run with empty pull_requests, the only discriminator is suite.GetHeadBranch() vs the PR's head.ref. An in-repo copy made with gh pr checkout 878 && git push origin HEAD keeps the fork's branch name, so both check suites report the same head_branch, branch != ref is false, and the fork's red run still counts against the in-repo PR β€” the exact #890 shape, just with the default branch naming. It over-blocks rather than under-blocks, so it's safe, but the PR body/stage-labels.md:385 present the rule as general. Suggest either noting the limitation in the doc paragraph, or (stronger) disambiguating by repository when branches match: the check suite object carries no head repository, but GET /repos/{o}/{r}/actions/runs?check_suite_id={id} returns head_repository.full_name, which can be compared with pr.GetHead().GetRepo().GetFullName() (one extra read, only on the branch-equal path).

Minor, same function (commit_ci.go:1161): prHeadBranch() is evaluated before branch != "", so a failed suite lookup still spends the pulls/{n} read it can't use.

		if branch != "" {
			if ref := prHeadBranch(); ref != "" && branch != ref {
				continue
			}
		}

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

hivecommons-hive Bot and others added 2 commits November 5, 2026 22:54
Reviewer advisory on hivecommons#16830: an in-repo copy pushed under the fork's branch
name (gh pr checkout 878 && git push origin HEAD) shares the branch, so the
suite head_branch alone still charged the fork's red run to it. The
check-suite API names no head repository; the Actions workflow run behind
the suite does (actions/runs?check_suite_id=). A red unattributed run now
counts only when that run's head repository and branch match the PR's head;
a suite with no workflow run (another app) falls back to its head_branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar

Danathar commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, you were right about the shared branch name. Fixed in caefb93.

  • The check-suite API returns no head repository. On projectbluefin/utah-packages, suite 114111461610 has only head_branch and repository. The Actions workflow run behind the suite does have it: run 42111306103 reports head_repository: kylerankin/utah-packages.
  • checkRunsForPR now reads that run, once per suite, via actions/runs?check_suite_id=. A red run that names no PR counts only when the run's head repository and branch match the PR's head (head.repo.full_name, head.ref).
  • A suite with no workflow run, because it comes from another app, falls back to its head_branch. A failed lookup still keeps the run.
  • New test rows:
  • Mutants: 13 of 13 caught. go test ./pkg/github/... passes.
  • The docs paragraph and the changelog are updated to match.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read against v6 with the callers. Looks correct to me.

  • Plumbing is real, not aspirational: both production callers already set PRNumber (src/pkg/github/merge_ci_gate.go:207, src/pkg/github/automerge/automerge_sweep_queued.go:405), so the new filter at commit_ci.go:550-552 is live for the contributor lane that sent utah-packages#890 to 072; issue_release_approve.go:184 passes no number and is untouched, as the body says.
  • ListWorkflowRunsOptions.CheckSuiteID and Checks.GetCheckSuite exist in the pinned go-github v72, and every lookup failure keeps the run.
  • Only completed + checkConclusionIsRed runs are judged, so pending/green runs and the "latest per identity" grouping downstream are unaffected except that a dropped red run can leave a required identity empty β†’ required-check-pending, which is the right side to err on.

One open question, not a finding: checkRunNamesPR drops a red run whose pull_requests lists other PRs but not this one. That is correct when GitHub computes pull_requests at read time from open same-repo PRs at the head (my understanding), but if it were snapshotted at run creation, a same-repo PR opened after a push-triggered run would lose a red run that genuinely tested its SHA β€” an under-block, not the over-block the comment at commit_ci.go:1130 promises. If someone has seen a stale pull_requests list in the wild, that row is the one to re-check; otherwise fine.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review (head d36d570)

Attribution is applied to red runs only, so another PR's green run still masks this PR's own red run β€” src/pkg/github/commit_ci.go:1165 (if cr == nil || cr.GetStatus() != "completed" || !checkConclusionIsRed(...) { out = append(out, cr); continue }).

The mirror of the #890/#878 incident: evaluate fork PR #878 whose own Canary (suite A, pull_requests empty) failed, while in-repo #890's Canary (newer ID, pull_requests: [890]) passed on the same SHA. checkRunsForPR keeps #878's red run and also keeps #890's green run untouched; latestCheckRunsByNameAndApp then picks the green one because checkRunIsNewer (commit_ci.go:1296) orders by ID alone. #878 reads green on a Canary it never passed β€” and for Canary specifically (fork runs lack secrets, in-repo runs have them) that is the common shape, not an exotic one. A false green feeds the automerge lanes, which is worse than the false red this PR fixes. The PR table's "green or still running: as is" row documents the choice, but the title ("attribute check runs by PR, not by SHA alone") is not met for the half that can wrongly merge.

Suggested fix β€” judge every completed run, not only red ones. Runs with a non-empty pull_requests cost no lookup; unattributed ones are already cached per suite, so the cost is one actions/runs call per suite, not per run:

 	for _, cr := range checks {
-		if cr == nil || cr.GetStatus() != "completed" || !checkConclusionIsRed(cr.GetConclusion()) {
+		if cr == nil || cr.GetStatus() != "completed" {
 			out = append(out, cr)
 			continue
 		}

TestCheckRunAttributionLookupsAreBounded's second half ("green runs cost no lookup") would need to flip to assert one suite read, and a new row should cover: pr 878, runs {own Canary failure suite 878 (no PRs), Canary success id 9 suite 890 (pr 890)} β†’ wantRed: true. If you'd rather keep lookups red-only, the minimum is to also filter green runs that carry a pull_requests list naming only other PRs (free), since that is exactly the in-repo-copy case above.

Everything else checked out: checkRunNamesPR precedence (a || b || c && d) is as intended; failed lookups keep the run; prHeadOf/suiteHeadOf are read once; commit-only path (PRNumber == 0) is untouched at commit_ci.go:550.

β€” hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read checkRunsForPR against the callers in src/pkg/github/commit_ci.go at d36d570. The attribution logic itself is sound: checkRunNamesPR precedence is right, the suite/PR lookups are cached and only run for unattributed red runs, and the nine table rows cover the #890/#878 shape plus the same-branch-name and other-app fallbacks. One gap worth closing before this lands:

The filter is one-directional, and the open direction is the unsafe one (commit_ci.go:1163-1166 keeps every non-red run; latestCheckRunsByNameAndApp at commit_ci.go:1079-1103 then picks the newest run in each name/app group).

Suggested shape: apply the pull_requests test to every completed run, not only red ones β€” that part is free (no lookups), and it already resolves the in-repo-vs-fork case from the in-repo PR's side because the in-repo run names it. Whether to also spend the suite lookup on unattributed green runs is the author's call; if not, say so in the comment on checkRunsForPR so the asymmetry is documented rather than implied safe.

Everything else I checked (foreignHead unknown-is-not-foreign, PRNumber plumbing at commit_ci.go:550, the stage-labels doc paragraph) matches the diff.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

docs-currency (read at head d36d570)

The new behaviour is documented where operators read it (src/docs/labels-and-control-signals/stage-labels.md:385, changelog fragment), and no doc cites commit_ci.go line numbers, so nothing drifts there. One in-scope gap for callers:

  • CommitCIOptions.PRNumber doc comment is now stale β€” src/pkg/github/commit_ci.go:137-138 still says it only "enables bounded, exact-current-head reporting reconciliation", but at commit_ci.go:550-552 passing it now also turns on per-PR check-run attribution, which costs up to one workflow-run (or check-suite) read per red suite plus one GET pulls/{n}. Callers deciding whether to pass PRNumber read this comment, not checkRunsForPR.
	// PRNumber enables bounded, exact-current-head reporting reconciliation
	// and per-PR check-run attribution (checkRunsForPR): a red run another PR
	// produced on the same head is dropped, at the cost of one workflow-run or
	// check-suite read per red suite and one pulls/{n} read.
	// Commit-only/base observations retain their existing behavior.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review (head d36d570)

Attribution is applied to red runs only, so another PR's green run still masks this PR's own red run β€” src/pkg/github/commit_ci.go:1165 (if cr == nil || cr.GetStatus() != "completed" || !checkConclusionIsRed(...) { out = append(out, cr); continue }).

The mirror of the #890/#878 incident: evaluate fork PR #878 whose own Canary (suite A, pull_requests empty) failed, while in-repo #890's Canary (newer ID, pull_requests: [890]) passed on the same SHA. checkRunsForPR keeps #878's red run and also keeps #890's green run untouched; latestCheckRunsByNameAndApp then picks the green one because checkRunIsNewer (commit_ci.go:1296) orders by ID alone. #878 reads green on a Canary it never passed β€” and for Canary specifically (fork runs lack secrets, in-repo runs have them) that is the common shape, not an exotic one. A false green feeds the automerge lanes, which is worse than the false red this PR fixes. The PR table's "green or still running: as is" row documents the choice, but the title ("attribute check runs by PR, not by SHA alone") is not met for the half that can wrongly merge.

Suggested fix β€” judge every completed run, not only red ones. Runs with a non-empty pull_requests cost no lookup; unattributed ones are already cached per suite, so the cost is one actions/runs call per suite, not per run:

 	for _, cr := range checks {
-		if cr == nil || cr.GetStatus() != "completed" || !checkConclusionIsRed(cr.GetConclusion()) {
+		if cr == nil || cr.GetStatus() != "completed" {
 			out = append(out, cr)
 			continue
 		}

TestCheckRunAttributionLookupsAreBounded's second half ("green runs cost no lookup") would need to flip to assert one suite read, and a new row should cover: pr 878, runs {own Canary failure suite 878 (no PRs), Canary success id 9 suite 890 (pr 890)} β†’ wantRed: true. If you'd rather keep lookups red-only, the minimum is to also filter green runs that carry a pull_requests list naming only other PRs (free), since that is exactly the in-repo-copy case above.

Everything else checked out: checkRunNamesPR precedence (a || b || c && d) is as intended; failed lookups keep the run; prHeadOf/suiteHeadOf are read once; commit-only path (PRNumber == 0) is untouched at commit_ci.go:550.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the filter against the tree at merge-base, not just the diff. Looks correct to me.

What I verified:

  • Both paths that can send a PR to stage/072 for "Failing checks" already pass PRNumber into EvaluateCommitCI (src/pkg/github/merge_ci_gate.go:207, src/pkg/github/automerge/automerge_sweep_queued.go:405), so the new checkRunsForPR call at src/pkg/github/commit_ci.go:550 sits on the path that produced the utah-packages#890 misattribution; the nudge text itself only renders obs.Reason/obs.Detail (automerge_contributor_nudge.go:311), so there is no second check-run listing to patch.
  • Fail-safe direction is right: ListRepositoryWorkflowRuns/GetCheckSuite errors leave suiteHeads[id] zero and foreignHead treats unknown as not-foreign (commit_ci.go:1183-1189), so a missing actions: read scope on the App degrades to today's behaviour rather than hiding red runs. Only red completed runs trigger a lookup; PullRequests.Get is done once and suites are cached per evaluation.
  • ListWorkflowRunsOptions.CheckSuiteID exists in the pinned go-github v72 (actions_workflow_runs.go:69); go build ./pkg/github/ ./pkg/github/automerge/ passes on the head.
  • The symmetric case works: evaluating the fork PR itself keeps its own run (workflow-run head_repository == PR head repo) and drops the in-repo copy's run, which the #878 and #891 table cases cover.

One non-blocking note: checkRunNamesPR at commit_ci.go:1204 relies on && binding tighter than ||; it is correct but a pair of parentheses would save the next reader a minute.

Evidence bundle `#16830298c8d4` sha256:926ad2d93dd4ac75d514b65450964e6fa4e39ffc514eb4175f77bfff2ee22d67 β€” download from the dashboard or `hivectl review evidence #16830`.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read against the v6 tree, not just the diff. Looks correct to me.

What I verified:

  • checkRunsForPR (src/pkg/github/commit_ci.go:550) runs right after checkRunsOnHead and only when opts.PRNumber > 0. Both lane callers pass it: automerge_sweep_queued.go:405 and merge_ci_gate.go:207; issue_release_approve.go:184 (commit-only) does not, so release-target reads are unchanged as the body says.
  • The 072 nudge's "Failing checks" list is derived from the lane reason (automerge_contributor_nudge.go:96-99, check-failing:/required-check-failing:), which comes out of EvaluateCommitCI β€” so filtering at this point is upstream of everything that returned utah-packages#890.
  • foreignHead treats unknown as not-foreign, and the PR GET / suite lookups are cached (prHeadRead, suiteHeads), so a failed lookup can only over-block, never drop a run.
  • go build ./pkg/github/ passes locally; the test (*) shards are green on this head.

One note, not a defect: a red run from another app whose suite has no workflow run is judged on head_branch alone, so two in-repo PRs sharing both SHA and branch name would still share that app's red run. That is the documented fallback and is the conservative direction.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 1d598c0 into hivecommons:v6 Nov 5, 2026
67 of 68 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 298c8d4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant