Skip to content
1 change: 1 addition & 0 deletions changelog.d/fixed-held-workflow-runs-refresh-and-stage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- A PR whose workflow runs GitHub holds for approval, and that the hive will not approve (an untrusted or fork head, or auto-approval off), now parks at 085 with reason `workflow-approval-required:held-runs:<run ids>` instead of showing an unrelated lane reason such as "author association" or "waiting for a verdict". The approval probe also runs on green heads that are missing a required check (a fork head whose `pull_request_target` check passed), so the missing-check base refresh no longer merges the base into such a PR again and again, and it now updates a PR once per set of missing checks instead of once per check.
1 change: 1 addition & 0 deletions src/cmd/hive/merge_eligibility.go
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,7 @@ func stageLabelPolicyFunc(cfg *config.Config) func() automerge.StageLabelPolicy
BaseBranchRed: stageBaseBranchRed,
BaseRedRemembered: stageBaseRedRemembered,
ForkGuard: stageForkGuard,
HeldWorkflowRuns: stageHeldWorkflowRuns,
DCORequired: stage.DCO.RequiredFor,
SquashOnly: cfg.AutoMerge.SquashOnlyFor,
QuietNotifications: cfg.GitHub.Notifications.Quiet(),
Expand Down
29 changes: 21 additions & 8 deletions src/cmd/hive/missing_check_base_refresh.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ type missingCheckRefresher interface {
// already updated a branch for. The second one is the loop guard: a check
// that a fresh head still does not run (a path filter, a GITHUB_TOKEN-made
// PR, see #11877) is not chased with another update every time the base
// moves. After a restart both are empty, which costs at most one more
// update per PR and check.
// moves. One update marks every check missing at the time, so a PR missing
// many checks is updated once, not once per check. After a restart both are
// empty, which costs at most one more update per PR.
type missingCheckRefreshState struct {
mu sync.Mutex
lastRun time.Time
Expand All @@ -53,11 +54,12 @@ var missingCheckRefresh = &missingCheckRefreshState{}
// the base tip.
//
// It leaves alone a PR that is a draft, held, exempt or parked on a human
// decision, red, conflicting, waiting on workflow approval, a fork with
// "allow edits by maintainers" off, a dependency bot's branch, returned to
// its contributor (072) or adopted by the hive (074). It shares the stale-red
// refresh's interval and caps, and updates a PR at most once per (head, base
// tip) and once per missing check.
// decision, red, conflicting, waiting on workflow approval (GitHub holds its
// runs, and an update would only make a new head whose runs are held again),
// a fork with "allow edits by maintainers" off, a dependency bot's branch,
// returned to its contributor (072) or adopted by the hive (074). It shares
// the stale-red refresh's interval and caps, and updates a PR at most once
// per (head, base tip) and once per set of missing checks.
//
// It returns the escalation keys of the PRs it updated.
func refreshMissingCheckBehindPRs(
Expand Down Expand Up @@ -182,7 +184,18 @@ func refreshMissingCheckBehindPRs(
}
continue
}
st.refreshed[memoKey+"|"+check] = true
// One update answers every missing check that already runs today (a
// run on the base tip or another open PR): a fresh head that still
// lacks them will not grow them on the next base move either, so
// none of them earns another update (projectbluefin/server#686 has
// 12 missing checks). A missing check with no such evidence yet did
// not justify this update; it keeps its own once the workflow that
// reports it lands on the base (#18945 review).
for _, name := range pending {
Comment thread
Danathar marked this conversation as resolved.
Resolved
if base.state.Reported[name] || seenOnPRs[bk][name] {
st.refreshed[memoKey+"|"+name] = true
}
}
perRepo[repo]++
total++
if logger != nil {
Expand Down
77 changes: 77 additions & 0 deletions src/cmd/hive/missing_check_base_refresh_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -248,3 +248,80 @@ func TestMissingCheckRefreshSkip(t *testing.T) {
})
}
}

// projectbluefin/server#686: a fork PR whose pull_request_target check (Prow)
// passed while GitHub holds every pull_request run for approval. The probe
// now runs on such a green head, and a held head is never base-refreshed:
// the update would only make a new head whose runs are held again.
func TestRefreshMissingCheckBehindPRs_HeldForkHeadLeftAlone(t *testing.T) {
resetMissingCheckRefresh(t)
pr := missingCheckPR(686, "h1")
pr.CIActivity.Checks = []github.PRActivityCheck{{Name: "prow / prow", State: "completed", Conclusion: "success", Head: "h1"}}
pr.CIActivity.WorkflowApproval = &github.PRWorkflowApproval{Pending: 4, RunIDs: []int64{1, 2, 3, 4}}
f := &fakeMissingCheckRefresher{bases: baseRunning("base1", "dashboard-build"), out: github.StaleRedRefreshOutcome{Updated: true, Settled: true}}
refreshMissingCheckBehindPRs(context.Background(), escalationTestConfig(), f, actionableWith(pr), nil, discardLogger(), time.Now())
if len(f.calls) != 0 || len(f.baseReads) != 0 {
t.Fatalf("calls %d, base reads %v; want a head with held runs left alone with no API call", len(f.calls), f.baseReads)
}
}

// A PR missing many required checks is updated once, not once per check:
// the one update answers every check missing at the time, so a later base
// tip does not buy another update for the next check on the list.
func TestRefreshMissingCheckBehindPRs_OncePerSetOfMissingChecks(t *testing.T) {
resetMissingCheckRefresh(t)
cfg := escalationTestConfig()
missing := []string{"build", "docs", "lint", "unit"}
pr := func(head string) github.PullRequest {
p := missingCheckPR(686, head)
p.Protection.MissingRequiredChecks = append([]string(nil), missing...)
return p
}
f := &fakeMissingCheckRefresher{bases: baseRunning("base1", missing...), out: github.StaleRedRefreshOutcome{Updated: true, Settled: true}}
now := time.Date(2026, 11, 18, 2, 9, 0, 0, time.UTC)
for i, tip := range []string{"base1", "base2", "base3", "base4"} {
f.bases = baseRunning(tip, missing...)
refreshMissingCheckBehindPRs(context.Background(), cfg, f, actionableWith(pr("head"+tip)), nil, discardLogger(), now.Add(time.Duration(i)*staleRedRefreshInterval))
}
if len(f.calls) != 1 {
t.Fatalf("refresh calls = %d over four base tips with %d missing checks, want 1", len(f.calls), len(missing))
}
// A check that becomes required later still gets its one update.
f.bases = baseRunning("base5", append(missing, "e2e")...)
next := pr("headbase5")
next.Protection.MissingRequiredChecks = append(next.Protection.MissingRequiredChecks, "e2e")
refreshMissingCheckBehindPRs(context.Background(), cfg, f, actionableWith(next), nil, discardLogger(), now.Add(5*staleRedRefreshInterval))
if len(f.calls) != 2 || f.checks[1] != "e2e" {
t.Fatalf("refresh calls = %d (checks %v), want a second update for the newly required e2e", len(f.calls), f.checks)
}
}

// A missing check with no evidence that it runs anywhere did not justify the
// update, so it is not memoised with the ones that did: once its workflow
// lands on the base, the PR earns one update for it (#18945 review).
func TestRefreshMissingCheckBehindPRs_UnevidencedCheckKeepsItsUpdate(t *testing.T) {
resetMissingCheckRefresh(t)
cfg := escalationTestConfig()
pr := func(head string) github.PullRequest {
p := missingCheckPR(686, head)
p.Protection.MissingRequiredChecks = []string{"build", "e2e"}
return p
}
f := &fakeMissingCheckRefresher{bases: baseRunning("base1", "build"), out: github.StaleRedRefreshOutcome{Updated: true, Settled: true}}
now := time.Date(2026, 11, 18, 9, 0, 0, 0, time.UTC)
refreshMissingCheckBehindPRs(context.Background(), cfg, f, actionableWith(pr("h1")), nil, discardLogger(), now)
if len(f.calls) != 1 || f.checks[0] != "build" {
t.Fatalf("first pass: calls=%d checks=%v, want one update for build", len(f.calls), f.checks)
}
// e2e's workflow now runs on the base; build still has its update spent.
f.bases = baseRunning("base2", "build", "e2e")
refreshMissingCheckBehindPRs(context.Background(), cfg, f, actionableWith(pr("h2")), nil, discardLogger(), now.Add(staleRedRefreshInterval))
if len(f.calls) != 2 || f.checks[1] != "e2e" {
t.Fatalf("second pass: calls=%d checks=%v, want one more update for e2e", len(f.calls), f.checks)
}
f.bases = baseRunning("base3", "build", "e2e")
refreshMissingCheckBehindPRs(context.Background(), cfg, f, actionableWith(pr("h3")), nil, discardLogger(), now.Add(2*staleRedRefreshInterval))
if len(f.calls) != 2 {
t.Fatalf("third pass: calls=%d, want no more updates", len(f.calls))
}
}
18 changes: 17 additions & 1 deletion src/cmd/hive/stage_shared_ci.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@ type stageSharedCIEntry struct {
// is from a fork; set only when every failing check does
// (PullRequest.ForkGuardChecks).
forkGuard []string
// heldRuns names the workflow runs GitHub holds for approval on the head
// that the hive did not approve (PullRequest.HeldWorkflowRunIDs).
heldRuns []int64
}

// stageSharedCIIndex holds, per "repo#number", the open [shared-ci] incident
Expand All @@ -46,7 +49,8 @@ func publishStageSharedCIIncidents(actionable *github.ActionableResult) {
e.baseRed = append([]string(nil), pr.BaseRedChecks...)
}
e.forkGuard = pr.ForkGuardChecks()
if e.incident > 0 || len(e.baseRed) > 0 || len(e.forkGuard) > 0 {
e.heldRuns = pr.HeldWorkflowRunIDs()
if e.incident > 0 || len(e.baseRed) > 0 || len(e.forkGuard) > 0 || len(e.heldRuns) > 0 {
idx[stageSharedCIKey(pr.Repo, pr.Number)] = e
}
}
Expand Down Expand Up @@ -116,6 +120,18 @@ func stageForkGuard(repo string, number int, head string) []string {
return append([]string(nil), e.forkGuard...)
}

// stageHeldWorkflowRuns answers StageLabelPolicy.HeldWorkflowRuns with the
// same repo/head matching as stageSharedCIIncident: the workflow runs GitHub
// holds for approval on the head that the hive did not approve. A new head
// is not covered until the next enumeration probes it.
func stageHeldWorkflowRuns(repo string, number int, head string) []int64 {
e, ok := stageSharedCILookup(repo, number, head)
if !ok || len(e.heldRuns) == 0 {
return nil
}
return append([]int64(nil), e.heldRuns...)
}

func stageSharedCILookup(repo string, number int, head string) (stageSharedCIEntry, bool) {
idx := stageSharedCIIndex.Load()
if idx == nil {
Expand Down
22 changes: 22 additions & 0 deletions src/cmd/hive/stage_shared_ci_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,3 +86,25 @@ func TestStageBaseRedRemembered(t *testing.T) {
t.Fatalf("after an empty publish: %v", got)
}
}

func TestStageHeldWorkflowRunsIndex(t *testing.T) {
t.Cleanup(func() { stageSharedCIIndex.Store(nil) })
held := &github.PRWorkflowApproval{Pending: 2, RunIDs: []int64{44261912124, 44257752419}, Denied: true}
undecided := &github.PRWorkflowApproval{Pending: 1, RunIDs: []int64{11}}
act := &github.ActionableResult{}
act.PRs.Items = []github.PullRequest{
{Repo: "acme/server", Number: 686, HeadSHA: "aaa", CIActivity: &github.PRCIActivity{Head: "aaa", WorkflowApproval: held}},
{Repo: "acme/server", Number: 687, HeadSHA: "bbb", CIActivity: &github.PRCIActivity{Head: "bbb", WorkflowApproval: undecided}},
}
publishStageSharedCIIncidents(act)

if got := stageHeldWorkflowRuns("acme/server", 686, "aaa"); len(got) != 2 || got[0] != 44261912124 {
t.Fatalf("held runs = %v", got)
}
if got := stageHeldWorkflowRuns("acme/server", 686, "new-head"); got != nil {
t.Fatalf("a new head must not inherit the old head's held runs: %v", got)
}
if got := stageHeldWorkflowRuns("acme/server", 687, "bbb"); got != nil {
t.Fatalf("runs the approval sweep has not decided on read as held: %v", got)
}
}
2 changes: 1 addition & 1 deletion src/docs/github-write-surface.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ writes no entry.
| Shared-CI rerun (`rerun-failed-jobs`, or `rerun` for a `cancelled` run; `hive-shared-ci-rerun-<n>` + `hive-shared-ci-lifted-<n>` comment) | `shared_ci_rerun.go` | yes (`shared-ci-rerun`) | yes |
| Transient-CI rerun (`rerun-failed-jobs` on a hive or trusted-bot PR whose failing runs carry `hive-ci-transient`, once per head, no comment) | `transient_ci_rerun.go` | yes (`transient-ci-rerun`) | yes |
| Stale-red base refresh (`update-branch` on a red PR behind its base whose failing checks are green on the base tip, once per head and base tip, no comment) | `stale_red_base_refresh.go` | yes (`stale-red-base-refresh`) | yes |
| Missing-check base refresh (`update-branch` on a PR behind its base whose head never ran a required check that runs on the base tip or another open PR, once per head and base tip and once per check, no comment) | `missing_check_base_refresh.go` | yes (`missing-check-base-refresh`) | yes |
| Missing-check base refresh (`update-branch` on a PR behind its base whose head never ran a required check that runs on the base tip or another open PR, once per head and base tip and once per set of missing checks, never while GitHub holds the head's runs for approval, no comment) | `missing_check_base_refresh.go` | yes (`missing-check-base-refresh`) | yes |
| Dependency-bot rebase request (ticks only Renovate's rebase/retry checkbox line in the PR description, once per head, on a bot PR the bot stopped rebasing; write-back kind `bot_rebase_request`; Dependabot is never asked, because it refuses commands from apps) | `dependency_bot_rebase.go` | yes (`dependency_bot_rebase_requested`) | yes |
| Open-PR dispatch guard (`has-open-pr` label on a refused issue) and duplicate-fix sweep (one `hive:duplicate-fix` comment; `hive/superseded` close of the later hive PR, or `needs-human` + `Parked` fact on escalation) | `open_pr_links.go`, `pr_duplicate_fix_sweep.go` | yes (`duplicate-fix-*`) | yes |
| Duplicate-PR sweep suggestions | `duplicate_sweep.go` | yes (`duplicate-sweep-suggested`) | yes (target = surviving PR) |
Expand Down
Loading
Loading