Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-ok-to-test-refusal-wordings.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The contributor lane stops posting `/ok-to-test` on a fork PR once the merge bot refuses the App, in cncf/prow-github-actions' wording too. Only upstream Prow's "Cannot trigger testing until a trusted user" was recognised, so repositories that run Prow commands through cncf/prow-github-actions ("Cannot /ok-to-test because Error: <app> is not a reviewer or approver …") got the command again on every new head, refused every time, and never got the needs-human note asking a person to post it. A command for an earlier head that left no `ok-to-test` label and no presubmit after 10 minutes now also counts as refused, whatever the bot said. The same-repo held-runs path reads the same refusal forms.
2 changes: 1 addition & 1 deletion src/docs/fork-pr-presubmit-stall-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ gh pr view <n> --repo <org>/<repo> --comments | grep -i -E "tide|not mergeable|r

## 2. Start the presubmits

The contributor lane now posts `/ok-to-test` itself, once per head, on a fork PR it admitted whose author is not an org member or collaborator ([#17474](https://github.com/hivecommons/hive/issues/17474); see [hive-merge.md](hive-merge.md)). This step is needed only when that did not help: the PR carries the hive's `needs-human` note saying Prow rejected the App (`Cannot trigger testing until a trusted user`), or the lane did not admit the PR.
The contributor lane now posts `/ok-to-test` itself, once per head, on a fork PR it admitted whose author is not an org member or collaborator ([#17474](https://github.com/hivecommons/hive/issues/17474); see [hive-merge.md](hive-merge.md)). This step is needed only when that did not help: the PR carries the hive's `needs-human` note saying Prow rejected the App (`Cannot trigger testing until a trusted user`, or cncf/prow-github-actions' `Cannot /ok-to-test because …`) or that an earlier post had no effect (no `ok-to-test` label and no presubmit after 10 minutes), or the lane did not admit the PR.

A human with write access comments on the PR. This is Prow's `/ok-to-test` (trigger plugin, shows as Tide `Jobs X, Y have not succeeded`), not the same-repo `hive-ok-to-test` GitHub Actions workflow (shows as `workflow-approval-required:<ctx>`; see [github-app-setup.md](github-app-setup.md)), which never covers fork heads:

Expand Down
18 changes: 12 additions & 6 deletions src/docs/hive-merge.md
Original file line number Diff line number Diff line change
Expand Up @@ -1112,13 +1112,19 @@ author's `author_association` is not `OWNER`, `MEMBER` or `COLLABORATOR`, the
PR has no `ok-to-test` label and Prow has reported no presubmit status on the
head (a status other than `tide` from the account that reports `tide`), the
lane posts `/ok-to-test` once per head with the same per-head marker and
mutation-boundary dedupe as the trusted-lane command. If Prow replies
`Cannot trigger testing until a trusted user`, the lane logs it once with
mutation-boundary dedupe as the trusted-lane command. The lane treats the
command as refused when the merge bot says so, in upstream Prow's words
(`@<app>: Cannot trigger testing until a trusted user …`) or
cncf/prow-github-actions' (`Cannot /ok-to-test because Error: <app> is not a
reviewer or approver …`), each naming the App. It also treats it as refused,
whatever the reply, when its command for an earlier head is more than 10
minutes old and the PR still has no `ok-to-test` label and Prow reported no
presubmit status on that head. A refused command is logged once with
`skipped=prow-rejects-bot-commands`, leaves one
`<!-- hive-ok-to-test-rejected -->` needs-human note and never posts the
command on that PR again; making the App a Prow trusted user is the
repository's Prow configuration. The merge-request relay's read-only
evaluation posts nothing.
`<!-- hive-ok-to-test-rejected -->` needs-human note and is never posted on
that PR again; making the App a Prow trusted user is the repository's Prow
configuration. The same refusal reader serves the same-repo held-runs
command. The merge-request relay's read-only evaluation posts nothing.

A PR merges through this tier only when all of the following hold:

Expand Down
9 changes: 9 additions & 0 deletions src/pkg/github/automerge/automerge_contributor_lane_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,9 @@ type contributorLaneFixture struct {
// headStatuses is what GET commits/sha7/statuses (every commit status,
// with its creator) serves; nil serves an empty list.
headStatuses []map[string]any
// earlierStatuses is what GET commits/<sha>/statuses serves for heads
// other than sha7; a head without an entry serves an empty list.
earlierStatuses map[string][]map[string]any
// baseRef, when set, gives the PR base a branch name and serves the
// branch, protection and compare routes the current-base gate
// (automerge_current_base.go) and release hold need; empty leaves the
Expand Down Expand Up @@ -379,6 +382,12 @@ func newContributorLaneAPI(t *testing.T, fx *contributorLaneFixture) *httptest.S
statuses = []map[string]any{}
}
json.NewEncoder(w).Encode(statuses)
case r.Method == http.MethodGet && strings.HasPrefix(r.URL.Path, "/repos/acme/widget/commits/") && strings.HasSuffix(r.URL.Path, "/statuses"):
statuses := fx.earlierStatuses[strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/repos/acme/widget/commits/"), "/statuses")]
if statuses == nil {
statuses = []map[string]any{}
}
json.NewEncoder(w).Encode(statuses)
case r.Method == http.MethodGet && (r.URL.Path == "/repos/acme/widget/commits/sha7/check-runs" ||
(fx.recheckHeadSHA != "" && r.URL.Path == "/repos/acme/widget/commits/"+fx.recheckHeadSHA+"/check-runs")):
if fx.noCIEvidence {
Expand Down
54 changes: 44 additions & 10 deletions src/pkg/github/automerge/automerge_contributor_prow_oktotest.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,15 +66,16 @@ func (c *Engine) requestProwPresubmitsForFork(ctx context.Context, displayRepo,
if !scanned {
return
}
if state.rejected {
unanswered := !state.rejected && c.okToTestUnanswered(ctx, displayRepo, owner, repo, number, state.earlierPosts)
if state.rejected || unanswered {
c.markWorkflowApprovalUntrusted(owner, repo, number)
args := []any{"repo", displayRepo, "pr", number, "sha", headSHA, "skipped", hgithub.PRApprovalCommandsSkippedProw}
args := []any{"repo", displayRepo, "pr", number, "sha", headSHA, "skipped", hgithub.PRApprovalCommandsSkippedProw, "unanswered", unanswered}
const msg = "contributor-lane ok-to-test not posted again: Prow rejected the App's command on this fork PR; a trusted user must post it"
if state.rejectionNoted {
c.debug(msg, args...)
return
}
c.postWorkflowApprovalUntrustedNote(ctx, displayRepo, owner, repo, number, headSHA, prowPresubmitsRejectedNote(strings.TrimSpace(c.transport.AppBotLogin())))
c.postWorkflowApprovalUntrustedNote(ctx, displayRepo, owner, repo, number, headSHA, prowPresubmitsRejectedNote(strings.TrimSpace(c.transport.AppBotLogin()), state.refusal))
c.warn(msg, args...)
return
}
Expand All @@ -87,6 +88,29 @@ func (c *Engine) requestProwPresubmitsForFork(ctx context.Context, displayRepo,
c.postWorkflowApprovalCommand(ctx, displayRepo, owner, repo, number, headSHA)
}

// okToTestUnanswered reports whether the App's command for an earlier head
// evidently did nothing, whatever the merge bot's wording (or none): it was
// posted at least workflowApprovalGrace ago, the PR still has no `ok-to-test`
// label (the caller returned on one; the trigger plugin's label persists
// across pushes) and Prow reported no presubmit on that head. The newest such
// post is checked; a status lookup failure answers false.
func (c *Engine) okToTestUnanswered(ctx context.Context, displayRepo, owner, repo string, number int, posts []okToTestPost) bool {
var last okToTestPost
now := c.clock()
for _, post := range posts {
if post.at.IsZero() || now.Sub(post.at) < workflowApprovalGrace {
continue
}
if last.at.IsZero() || post.at.After(last.at) {
last = post
}
}
if last.sha == "" {
return false
}
return !c.prowPresubmitStatusPresent(ctx, displayRepo, owner, repo, number, last.sha)
}

// prowPresubmitStatusPresent reports whether Prow already reported a job on
// headSHA: a commit status other than Tide's own, created by the account that
// created Tide's status. Without a Tide status nothing on the head can be
Expand Down Expand Up @@ -130,20 +154,30 @@ func (c *Engine) prowPresubmitStatusPresent(ctx context.Context, displayRepo, ow
}

// prowPresubmitsRejectedNote is the one needs-human comment posted when Prow
// refuses the App's `/ok-to-test` on a non-member fork PR.
func prowPresubmitsRejectedNote(appLogin string) string {
// refuses the App's `/ok-to-test` on a non-member fork PR. refusal is the
// merge bot's wording; "" means it never acted on an earlier post.
func prowPresubmitsRejectedNote(appLogin, refusal string) string {
if appLogin == "" {
appLogin = "the Hive App"
}
var b strings.Builder
b.WriteString(workflowApprovalRejectedMarker)
b.WriteString("\nneeds-human: Prow has not run this fork PR's presubmits. Its author is not one of Prow's trusted users, so the trigger plugin waits for `")
b.WriteString(workflowApprovalCommand)
b.WriteString("`; Hive posted it and Prow rejected `")
b.WriteString(appLogin)
b.WriteString("` (\"")
b.WriteString(prowOkToTestRejection)
b.WriteString("\"). Hive will not post the command again on this PR.\n\nA maintainer can comment `")
if refusal != "" {
b.WriteString("`; Hive posted it and Prow rejected `")
b.WriteString(appLogin)
b.WriteString("` (\"")
b.WriteString(refusal)
b.WriteString("\")")
} else {
b.WriteString("`; Hive posted it for an earlier head and nothing came of it: no `")
b.WriteString(prowOkToTestLabel)
b.WriteString("` label and no presubmit status, so Prow does not accept it from `")
b.WriteString(appLogin)
b.WriteString("`")
}
b.WriteString(". Hive will not post the command again on this PR.\n\nA maintainer can comment `")
b.WriteString(workflowApprovalCommand)
b.WriteString("` (plus `/test all` for the current head), or make `")
b.WriteString(appLogin)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,97 @@ func TestContributorLaneStopsProwOkToTestAfterRejection(t *testing.T) {
}
}

// cncf/prow-github-actions answers in its own words and names the App with no
// @ (projectbluefin/chairlift#802: five posts, five refusals, no note). That
// reply stops the lane exactly like Prow's, and the note quotes it.
func TestContributorLaneStopsProwOkToTestAfterProwActionsRefusal(t *testing.T) {
fx := prowOkToTestFixture()
fx.prComments = []map[string]any{
appComment(2, workflowApprovalCommandBody("oldhead")),
prowComment(3, "Cannot /ok-to-test because Error: "+testHiveAppBotLogin+" is not a reviewer or approver for any changed file"),
}
runProwOkToTestTicks(t, &fx, 3)
runProwOkToTestTicks(t, &fx, 2)
if fx.comments != 1 {
t.Fatalf("comments = %d %q, want only the one needs-human note", fx.comments, fx.allComments)
}
note := fx.allComments[0]
if !strings.HasPrefix(note, workflowApprovalRejectedMarker) || !strings.Contains(note, prowActionsOkToTestRefusal) || strings.Contains(note, prowOkToTestRejection) {
t.Fatalf("note = %q, want the rejected marker quoting the action's wording, not Prow's", note)
}
}

// commentAt stamps a fixture comment with its created_at.
func commentAt(comment map[string]any, at time.Time) map[string]any {
comment["created_at"] = at.Format(time.RFC3339)
return comment
}

// Whatever a merge bot replies (or if it says nothing), a command posted for
// an earlier head that is past the grace, left no ok-to-test label and drew
// no Prow presubmit on that head did nothing: the lane does not post again
// on a new head and leaves the one note. A command that did start
// presubmits, or one still inside the grace, is not treated as refused.
func TestContributorLaneTreatsUnansweredProwOkToTestAsRefused(t *testing.T) {
clockStart := time.Date(2026, 11, 9, 12, 0, 0, 0, time.UTC)
command := workflowApprovalCommandBody("sha7")
for _, tc := range []struct {
name string
postedAt time.Time
mutate func(fx *contributorLaneFixture)
ticks int
wantPost bool
wantNote bool
}{
{name: "earlier post past the grace with no effect", postedAt: clockStart.Add(-30 * time.Minute), ticks: 3, wantNote: true},
{name: "earlier post still inside the grace", postedAt: clockStart.Add(-2 * time.Minute), ticks: 1, wantPost: true},
{
name: "earlier post that drew a Prow presubmit on its head",
postedAt: clockStart.Add(-30 * time.Minute),
mutate: func(fx *contributorLaneFixture) {
fx.earlierStatuses = map[string][]map[string]any{"oldhead": {
prowStatus(prowTideContext, prowOkToTestTestBot),
prowStatus("pull-widget-unit", prowOkToTestTestBot),
}}
},
ticks: 3,
wantPost: true,
},
{
// The trigger plugin's label persists across pushes: a command
// that took effect is never mistaken for a refusal.
name: "ok-to-test label present",
postedAt: clockStart.Add(-30 * time.Minute),
mutate: func(fx *contributorLaneFixture) { fx.labels = []string{"size/S", "ok-to-test"} },
ticks: 3,
},
} {
t.Run(tc.name, func(t *testing.T) {
fx := prowOkToTestFixture()
fx.prComments = []map[string]any{commentAt(appComment(2, workflowApprovalCommandBody("oldhead")), tc.postedAt)}
if tc.mutate != nil {
tc.mutate(&fx)
}
runProwOkToTestTicks(t, &fx, tc.ticks)
posted, notes := 0, 0
for _, body := range fx.allComments {
switch {
case body == command:
posted++
case strings.HasPrefix(body, workflowApprovalRejectedMarker):
notes++
if strings.Contains(body, prowOkToTestRejection) || !strings.Contains(body, "earlier head") {
t.Fatalf("note = %q, want the no-effect wording, not a quoted refusal", body)
}
}
}
if (posted == 1) != tc.wantPost || posted > 1 || (notes == 1) != tc.wantNote || notes > 1 {
t.Fatalf("posts=%d notes=%d (comments %q), want post=%v note=%v", posted, notes, fx.allComments, tc.wantPost, tc.wantNote)
}
})
}
}

// The merge-request relay evaluates the lane read-only and posts nothing.
func TestContributorLaneRelayEvaluationDoesNotPostProwOkToTest(t *testing.T) {
fx := prowOkToTestFixture()
Expand Down
74 changes: 68 additions & 6 deletions src/pkg/github/automerge/automerge_held_runs.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,11 @@ const (
// prowOkToTestRejection is the reply Prow's trigger plugin posts when the
// commenter is not one of its trusted users.
prowOkToTestRejection = "Cannot trigger testing until a trusted user"
// prowActionsOkToTestRefusal opens the reply cncf/prow-github-actions
// posts instead ("Cannot /ok-to-test because Error: <login> is not a
// reviewer or approver for any changed file"); the reason after it names
// the commenter without an @.
prowActionsOkToTestRefusal = "Cannot " + workflowApprovalCommand + " because "
// prowApprovalNotifier appears in every comment Prow's approve plugin
// posts; its presence means `/ok-to-test` on this PR is Prow's command.
prowApprovalNotifier = "[APPROVALNOTIFIER]"
Expand Down Expand Up @@ -256,7 +261,7 @@ func (c *Engine) workflowApprovalAttention(ctx context.Context, displayRepo, own
// hand the command to a person once.
c.markWorkflowApprovalUntrusted(owner, repo, number)
if !state.rejectionNoted {
c.postWorkflowApprovalUntrustedNote(ctx, displayRepo, owner, repo, number, headSHA, workflowApprovalUntrustedNote(strings.TrimSpace(c.transport.AppBotLogin()), state.rejected))
c.postWorkflowApprovalUntrustedNote(ctx, displayRepo, owner, repo, number, headSHA, workflowApprovalUntrustedNote(strings.TrimSpace(c.transport.AppBotLogin()), state.refusal))
}
c.warn("automerge workflow approval cannot be requested by the App; merge bot trusts only its own users",
"repo", displayRepo, "pr", number, "sha", headSHA, "held", held, "context", contextName, "rejected", state.rejected, "prow", state.prowManaged)
Expand Down Expand Up @@ -345,12 +350,65 @@ type workflowApprovalComments struct {
postedForHead bool
// rejected: the merge bot replied that the commenter is not trusted.
rejected bool
// refusal is the merge bot's wording for that reply, quoted in the
// needs-human note: prowOkToTestRejection or prowActionsOkToTestRefusal.
refusal string
// earlierPosts are the App's commands for heads other than this one.
earlierPosts []okToTestPost
// rejectionNoted: the App already posted its needs-human note.
rejectionNoted bool
// prowManaged: Prow's approve plugin comments on this PR.
prowManaged bool
}

// okToTestPost is one of the App's /ok-to-test comments: the head its marker
// names and when it was posted (zero when GitHub did not say).
type okToTestPost struct {
sha string
at time.Time
}

// okToTestRefusal returns the merge bot's wording when body refuses the
// App's /ok-to-test, or "" when it does not. Upstream Prow's trigger plugin
// @-mentions the commenter; cncf/prow-github-actions opens its reply with
// "Cannot /ok-to-test because" and names the commenter bare. Either reply
// must name the App, so a refusal of someone else's command does not count.
// Like Prow's check, the author is not restricted beyond not being the App.
func okToTestRefusal(body, appLogin string) string {
names := func(withAt bool) bool {
if appLogin == "" {
return true
}
login := strings.ToLower(appLogin)
if withAt {
login = "@" + login
}
return strings.Contains(strings.ToLower(body), login)
}
switch {
case strings.Contains(body, prowOkToTestRejection) && names(true):
return prowOkToTestRejection
case strings.HasPrefix(strings.TrimSpace(body), prowActionsOkToTestRefusal) && names(false):
return prowActionsOkToTestRefusal + "…"
}
return ""
}

// okToTestPostSHA returns the head an App comment's command marker names,
// or "" when body carries no marker.
func okToTestPostSHA(body string) string {
i := strings.Index(body, workflowApprovalCommandMarkerPrefix)
if i < 0 {
return ""
}
rest := body[i+len(workflowApprovalCommandMarkerPrefix):]
j := strings.Index(rest, " -->")
if j < 0 {
return ""
}
return strings.ToLower(strings.TrimSpace(rest[:j]))
}

// workflowApprovalCommentState reads the PR's comments when a scan is due.
// The second result is false when nothing was read (not due, or the API
// failed), in which case the caller keeps its in-process knowledge.
Expand Down Expand Up @@ -380,6 +438,8 @@ func (c *Engine) workflowApprovalCommentState(ctx context.Context, displayRepo,
if appLogin != "" && strings.EqualFold(login, appLogin) {
if strings.Contains(body, marker) {
state.postedForHead = true
} else if sha := okToTestPostSHA(body); sha != "" {
state.earlierPosts = append(state.earlierPosts, okToTestPost{sha: sha, at: comment.GetCreatedAt().Time})
}
if strings.Contains(body, workflowApprovalRejectedMarker) {
state.rejectionNoted = true
Expand All @@ -389,8 +449,9 @@ func (c *Engine) workflowApprovalCommentState(ctx context.Context, displayRepo,
if strings.Contains(body, prowApprovalNotifier) {
state.prowManaged = true
}
if strings.Contains(body, prowOkToTestRejection) && (appLogin == "" || strings.Contains(strings.ToLower(body), "@"+strings.ToLower(appLogin))) {
if refusal := okToTestRefusal(body, appLogin); refusal != "" {
state.rejected = true
state.refusal = refusal
}
}
if resp == nil || resp.NextPage == 0 {
Expand Down Expand Up @@ -468,8 +529,9 @@ func (c *Engine) postWorkflowApprovalCommand(ctx context.Context, displayRepo, o
}

// workflowApprovalUntrustedNote is the one needs-human comment posted when
// the merge bot refuses the App's command.
func workflowApprovalUntrustedNote(appLogin string, rejected bool) string {
// the merge bot refuses the App's command; refusal is the bot's wording, ""
// when it never replied.
func workflowApprovalUntrustedNote(appLogin, refusal string) string {
if appLogin == "" {
appLogin = "the Hive App"
}
Expand All @@ -478,11 +540,11 @@ func workflowApprovalUntrustedNote(appLogin string, rejected bool) string {
b.WriteString("\nneeds-human: GitHub Actions is holding this PR's workflow runs for approval, and Hive cannot start them. `")
b.WriteString(workflowApprovalCommand)
b.WriteString("` on this repository belongs to Prow, which only accepts it from its trusted users")
if rejected {
if refusal != "" {
b.WriteString("; it rejected `")
b.WriteString(appLogin)
b.WriteString("` (\"")
b.WriteString(prowOkToTestRejection)
b.WriteString(refusal)
b.WriteString("\")")
} else {
b.WriteString(", and `")
Expand Down
Loading
Loading