Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-16287-open-pr-guard-timeout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Open-PR guard bounds its listing deadline and backs off per repo after a failure, logging a classified reason once per window (#16287)
1 change: 1 addition & 0 deletions changelog.d/fixed-open-pr-guard-rate-limit.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The open-PR pre-dispatch guard no longer hot-loops or fails open while the GitHub installation is rate-limited. A rate-limited open-PR listing (primary `RateLimitError` or secondary `AbuseRateLimitError`) is now remembered per repo until the limit resets (`Rate.Reset` / `Retry-After`, capped at an hour, one minute when unknown) and replayed without a GitHub call, warned about once per repo per window instead of once per candidate — one hive logged 47,000 `could not list open PRs` warnings in an hour. While the listing is rate-limited the guard refuses the dispatch (dispatch reason `open-PR guard deferred: GitHub rate limit`, withheld reason `open_pr_guard_rate_limited`) instead of admitting issues an open PR may already link; other listing failures still fail open.
12 changes: 9 additions & 3 deletions src/docs/operator-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1192,9 +1192,15 @@ between. Two mechanisms close that gap
already links this issue (claim skipped)* with the PR as evidence
(reason `open_pr_linked`). A failed listing fails **open**: the
ledger-backed guards still apply, and a flaky listing must not stall every
claim. The gate also refuses an issue the confirming read says is already
**closed** (`issue closed`), so an issue closed between enumerations is
never dispatched from the cached snapshot.
claim; the failure is remembered per repo for a minute, so the listing is
not retried and the warning not repeated per candidate. A **rate-limited**
listing fails **closed** instead: the guard cannot see an already-linked
PR, so every candidate in the repo is deferred until the limit resets
(dispatch reason `open-PR guard deferred: GitHub rate limit`, withheld
reason `open_pr_guard_rate_limited`), with no GitHub call and one
warning per repo until then. The gate also refuses an issue the
confirming read says is already **closed** (`issue closed`), so an issue
closed between enumerations is never dispatched from the cached snapshot.
- **Webhook claim linking.** A `pull_request` delivery (opened, reopened,
edited, synchronize, ready-for-review, draft, closed) is parsed with the
same closing-keyword and branch-name grammar and written into the claim
Expand Down
20 changes: 16 additions & 4 deletions src/pkg/dashboard/contribute_admission.go
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,8 @@ type contributorAdmissionDecision struct {
reason string
claim ghpkg.IssueClaim
// openPR carries the live open PR behind an open_pr_linked refusal
// (hivecommons/hive#14799). Zero for every other reason.
// (hivecommons/hive#14799), or the reset time behind an
// open_pr_guard_rate_limited one. Zero for every other reason.
openPR ghpkg.OpenPRLink
// churn carries the pull-request history behind an issue_churn refusal
// (#7995): which PRs merged, which closed unmerged. Zero-valued for every
Expand All @@ -127,6 +128,11 @@ const (
// It is what open_pr_claim becomes once the ledger catches up; the
// distinct code tells an operator the guard, not the ledger, caught it.
contributorAdmissionReasonOpenPRLinked = "open_pr_linked"
// contributorAdmissionReasonOpenPRGuardRateLimited: GitHub rate-limited
// the guard's open-PR listing, so it cannot tell whether a PR already
// links this issue. The candidate is deferred until the limit resets
// rather than dispatched blind.
contributorAdmissionReasonOpenPRGuardRateLimited = "open_pr_guard_rate_limited"
// contributorAdmissionReasonIssueClaim: someone has claimed the issue on
// the issue itself — a `hive:claim` marker comment or an assignee — and
// the claim has not expired (hivecommons/hive#8380). It covers the window
Expand Down Expand Up @@ -206,8 +212,12 @@ func (h *ContributeWSHub) evaluateContributorNeutralAdmission(sweep *contributor
// a PR opened since is only visible on the forge. One listing per repo
// per sweep, cached by the client.
if link, ok := h.openPRLinkingIssue(candidate); ok {
reason := contributorAdmissionReasonOpenPRLinked
if link.RateLimited() {
reason = contributorAdmissionReasonOpenPRGuardRateLimited
}
return contributorAdmissionDecision{
reason: contributorAdmissionReasonOpenPRLinked,
reason: reason,
openPR: link,
}
}
Expand Down Expand Up @@ -324,8 +334,10 @@ func mergedReferenceClaim(c ghpkg.IssueClaim) bool {
// openPRLinkingIssue asks the Dependencies.OpenPRLinkingIssue hook whether an
// open PR already links the candidate (hivecommons/hive#14799). Repo
// spellings are tried in the same order and for the same reason as
// issueClaimedByOpenPR. No hook wired (tests, or a hive booted without
// GitHub credentials) reports no link and selection proceeds as before.
// issueClaimedByOpenPR; a rate-limit deferral on the first spelling answers
// for both (one canonical listing backs them). No hook wired (tests, or a
// hive booted without GitHub credentials) reports no link and selection
// proceeds as before.
func (h *ContributeWSHub) openPRLinkingIssue(candidate contributorAdmissionCandidate) (ghpkg.OpenPRLink, bool) {
if h == nil || h.server == nil || h.server.deps == nil || h.server.deps.OpenPRLinkingIssue == nil {
return ghpkg.OpenPRLink{}, false
Expand Down
3 changes: 3 additions & 0 deletions src/pkg/dashboard/contribute_select.go
Original file line number Diff line number Diff line change
Expand Up @@ -905,6 +905,9 @@ func (h *ContributeWSHub) selectTaskPass(c *ContributorConnection, skippedUnmint
h.logger.Info("[contribute-ws] claim skipped: open PR already linked",
"repo", repo.Full, "number", number,
"pr", decision.openPR.PRNumber, "pr_url", decision.openPR.PRURL, "pr_author", decision.openPR.PRAuthor)
case contributorAdmissionReasonOpenPRGuardRateLimited:
// Not logged per item: the client warns once per repo
// per rate-limit window, and every candidate shares it.
case contributorAdmissionReasonIssueClaim:
// #8380: someone claimed the issue on the issue itself; the
// claim expires on its own, so the log names when.
Expand Down
44 changes: 25 additions & 19 deletions src/pkg/dashboard/contribute_withheld.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,25 +71,26 @@ const (
// has to carry a second copy of this vocabulary (the #6902 "one admission path"
// rule applies to the WORDS as much as to the decisions).
var withheldReasonLabels = map[string]string{
contributorAdmissionReasonOpenPRClaim: "An open pull request already claims this issue",
contributorAdmissionReasonOpenPRLinked: "An open pull request already links this issue (claim skipped)",
contributorAdmissionReasonIssueClaim: "Someone has claimed this issue; the claim expires on its own",
contributorAdmissionReasonMergedClaimStale: "Fixed by a merged pull request; the issue is still open — close it or say what remains",
contributorAdmissionReasonIssueChurn: "Too many pull requests on one issue — needs maintainer triage",
contributorAdmissionReasonWorkflowBlocked: "Workflow label: blocked",
contributorAdmissionReasonLabelSkipped: "Contribute skip label",
contributorAdmissionReasonDependencyBlocked: "A dependency is still open",
contributorAdmissionReasonDependencyUnknown: "A dependency could not be resolved",
withheldReasonDisabledRepo: "Repository is disabled for contribution",
withheldReasonRepoUnmintable: "Repository excluded after a GitHub App token mint failure",
withheldReasonTracker: "Tracker / umbrella issue",
withheldReasonCooldown: "Completion cooldown",
withheldReasonNoWorkNeeded: "Previous no_work_needed verdict",
completionVerdictNeedsDecision: "Waiting on a maintainer decision",
withheldReasonFailureCooldown: "Failure cooldown",
withheldReasonInFlight: "Already in flight",
withheldReasonContributorFilter: "Rejected by a contributor filter",
withheldReasonAssignedToOther: "Assigned to another contributor",
contributorAdmissionReasonOpenPRClaim: "An open pull request already claims this issue",
contributorAdmissionReasonOpenPRLinked: "An open pull request already links this issue (claim skipped)",
contributorAdmissionReasonOpenPRGuardRateLimited: "GitHub rate limit: the open-PR check is deferred until it resets",
contributorAdmissionReasonIssueClaim: "Someone has claimed this issue; the claim expires on its own",
contributorAdmissionReasonMergedClaimStale: "Fixed by a merged pull request; the issue is still open — close it or say what remains",
contributorAdmissionReasonIssueChurn: "Too many pull requests on one issue — needs maintainer triage",
contributorAdmissionReasonWorkflowBlocked: "Workflow label: blocked",
contributorAdmissionReasonLabelSkipped: "Contribute skip label",
contributorAdmissionReasonDependencyBlocked: "A dependency is still open",
contributorAdmissionReasonDependencyUnknown: "A dependency could not be resolved",
withheldReasonDisabledRepo: "Repository is disabled for contribution",
withheldReasonRepoUnmintable: "Repository excluded after a GitHub App token mint failure",
withheldReasonTracker: "Tracker / umbrella issue",
withheldReasonCooldown: "Completion cooldown",
withheldReasonNoWorkNeeded: "Previous no_work_needed verdict",
completionVerdictNeedsDecision: "Waiting on a maintainer decision",
withheldReasonFailureCooldown: "Failure cooldown",
withheldReasonInFlight: "Already in flight",
withheldReasonContributorFilter: "Rejected by a contributor filter",
withheldReasonAssignedToOther: "Assigned to another contributor",
}

// withheldReasonLabel returns the operator-facing prose for a reason code,
Expand All @@ -112,6 +113,7 @@ func isConvergenceWithheldReason(reason string) bool {
switch reason {
case contributorAdmissionReasonOpenPRClaim,
contributorAdmissionReasonOpenPRLinked,
contributorAdmissionReasonOpenPRGuardRateLimited,
contributorAdmissionReasonIssueClaim,
contributorAdmissionReasonMergedClaimStale,
contributorAdmissionReasonIssueChurn,
Expand Down Expand Up @@ -341,6 +343,10 @@ func withheldFromAdmissionDecision(c withheldCandidate, d contributorAdmissionDe
item.Detail = fmt.Sprintf("Claim skipped: open pull request #%d already links this issue", d.openPR.PRNumber)
}
return item
case contributorAdmissionReasonOpenPRGuardRateLimited:
item := newWithheldItem(c, d.reason)
item.Detail = fmt.Sprintf("GitHub rate-limited the open-PR listing; dispatch deferred until %s", formatRunTime(d.openPR.RateLimitedUntil))
return item
case contributorAdmissionReasonMergedClaimStale:
item := newWithheldItem(c, d.reason)
item.ClaimURL = d.claim.PRURL
Expand Down
20 changes: 20 additions & 0 deletions src/pkg/dashboard/contribute_withheld_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,26 @@ func withheldCases() []withheldCase {
}
},
},
{
// A rate-limited open-PR listing cannot rule out a linking PR,
// so the candidate is deferred to the reset, not dispatched.
name: "open-PR guard rate-limited",
reason: contributorAdmissionReasonOpenPRGuardRateLimited,
arrange: func(t *testing.T, hub *ContributeWSHub, s *Server) {
s.deps.IssueClaimed = func(string, int) (ghpkg.IssueClaim, bool) { return ghpkg.IssueClaim{}, false }
s.deps.OpenPRLinkingIssue = func(repo string, number int, labels []string) (ghpkg.OpenPRLink, bool) {
if number == 601 {
return ghpkg.OpenPRLink{Issue: 601, RateLimitedUntil: time.Now().Add(30 * time.Minute)}, true
}
return ghpkg.OpenPRLink{}, false
}
},
evidence: func(t *testing.T, item AdmissionWithheldItem) {
if item.ClaimURL != "" || !strings.Contains(item.Detail, "deferred until") {
t.Errorf("claim_url = %q detail = %q; want no PR and the reset time", item.ClaimURL, item.Detail)
}
},
},
{
// #8003: chairlift#55 — fixed by merged #133 four days ago, still
// open, re-offered every 72h. Withheld under its own reason that
Expand Down
4 changes: 3 additions & 1 deletion src/pkg/dashboard/deps.go
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,9 @@ type Dependencies struct {
// refreshed by the eval cycle, so a PR a person opened by hand stayed
// invisible to the contribute lane until the next scan and the issue was
// dispatched again in that window. labels are the issue's current labels
// so the label write is idempotent. Nil disables the live check; the
// so the label write is idempotent. ok with link.RateLimited() means
// GitHub rate-limited the listing and the candidate is deferred until
// link.RateLimitedUntil. Nil disables the live check; the
// client honours the scanner open-PR switch itself.
OpenPRLinkingIssue func(repo string, number int, labels []string) (ghpkg.OpenPRLink, bool)

Expand Down
3 changes: 2 additions & 1 deletion src/pkg/github/issue_claim_gate.go
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,8 @@ func (c *Client) issueClaimGate(ctx context.Context, repo string, number int) (i
// using IssueClaimBlockReason: the agent whose own PR links the issue holds
// its claim legitimately, and a guard that released it would un-stage the
// very PR it exists to protect. Off with scanner.skip_issues_with_open_pr:
// false; a failed listing fails open with a warning.
// false; a failed listing fails open with a warning, except a rate-limited
// one, which refuses with ClaimBlockReasonOpenPRRateLimited until the reset.
func (c *Client) IssueDispatchBlockReason(ctx context.Context, repo string, number int) (string, error) {
gate, err := c.issueClaimGate(ctx, repo, number)
if err != nil || gate.reason != "" {
Expand Down
Loading