From da7d5860967a30f40b035a71378f22a2655d7aaf Mon Sep 17 00:00:00 2001 From: Danathar Date: Wed, 28 Oct 2026 14:38:08 -0400 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20fix(github):=20back=20off=20a=20?= =?UTF-8?q?header-less=20secondary=20403=20for=20a=20minute,=20not=20an=20?= =?UTF-8?q?hour?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A secondary-limit 403 that named no wait (no Retry-After) assumed slowStartDefaultRetryAfter = 1h, plus the 10m slow-start window. For those 70 minutes every low-class request queued more than 5s was shed, including the automerge sweep's commit-status read. On the projectbluefin hive (2026-10-28, build 7f89f9d) one /search/issues 403 at 17:22Z after a restart meant no merges until at least 18:10Z. finpilot#599 (approved, 19/19 green) was skipped with reason "status-check" ("request shed ... class low, queue 5.454s ahead, limit 5s") while core quota stood at 87%. - slowstart.go: a secondary 403 now uses GitHub's documented order. A parseable Retry-After (seconds or an HTTP date) wins. Next comes X-RateLimit-Reset, but only with X-RateLimit-Remaining: 0. Otherwise the back-off is one minute, doubling on each repeat within 30 minutes of the last back-off (1m, 2m, 4m, 8m, capped at 15m). The rest of a burst refused by the same block does not escalate. X-RateLimit-Reset with quota left is the primary bucket's reset and is ignored. The slow-start window still follows every back-off. - automerge: once a lane (queued, self-authored, trusted-author or contributor) reaches a candidate PR, that PR's evaluation and merge run at normal pacing class (mergeDecisionPriority). The tick and floor pass gates and the pre-candidate reads stay low class. The lanes evaluate one request at a time, so this adds at most one slot and waits at most 30s, not the pile-up hivecommons/hive#13382 introduced the low class for. - EngageRateLimitCautionForTest lets other packages test behaviour under caution. - Docs: operator-reference "Rate-limit pacing priority". Shifted automerge_sweep.go line citations kept on the same lines. Tests: header-less 403 gives caution of 1m + window; escalation, same-burst, cap and reset; named waits honoured (Retry-After seconds and date, Reset only with Remaining 0); a sweep merges a green App PR while the caution queue is 5.5s deep. Each rule was mutation-checked. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar --- .../fixed-secondary-403-short-caution.md | 1 + src/docs/design/github-mention-triggers.md | 2 +- src/docs/labels-and-control-signals.md | 6 +- src/docs/operator-reference.md | 18 +- .../automerge/automerge_contributor_lane.go | 2 +- src/pkg/github/automerge/automerge_sweep.go | 27 ++- .../github/automerge/sweep_caution_test.go | 37 ++++ src/pkg/github/slowstart.go | 171 ++++++++++++++---- src/pkg/github/slowstart_test.go | 125 +++++++++++++ 9 files changed, 342 insertions(+), 47 deletions(-) create mode 100644 changelog.d/fixed-secondary-403-short-caution.md diff --git a/changelog.d/fixed-secondary-403-short-caution.md b/changelog.d/fixed-secondary-403-short-caution.md new file mode 100644 index 0000000000..537d5b04c2 --- /dev/null +++ b/changelog.d/fixed-secondary-403-short-caution.md @@ -0,0 +1 @@ +- One GitHub secondary-limit 403 no longer stalls merges for over an hour. A secondary 403 with no `Retry-After` used to assume a full hour, then add the 10-minute slow-start window. For those 70 minutes every low-priority call that would queue more than 5s was shed, including the automerge sweep's commit-status read. On the projectbluefin hive (2026-10-28) one search refusal after a restart meant no PR merged for over an hour, even an approved PR with 19/19 checks green, while the primary quota stood at 87%. Such a 403 now backs off for `Retry-After` when GitHub sends it, or until `X-RateLimit-Reset` when `X-RateLimit-Remaining` is `0`. Otherwise it waits one minute, GitHub's documented rule, and doubles on each repeat up to 15 minutes. Separately, once an automerge lane reaches a candidate PR, the PR's evaluation and merge run at normal pacing priority, so its status and check-run reads are never shed as housekeeping. diff --git a/src/docs/design/github-mention-triggers.md b/src/docs/design/github-mention-triggers.md index bb95356f1b..647fd246fb 100644 --- a/src/docs/design/github-mention-triggers.md +++ b/src/docs/design/github-mention-triggers.md @@ -136,7 +136,7 @@ comment (or issue) node id, so a mention seen twice is handled once. A mention is a comment or issue body containing `@` where `` is the installation's bot login, the value the hive already knows -as `appBotLogin` (`isHiveAppReviewAuthor`, `src/pkg/github/automerge/automerge_sweep.go:1221`). +as `appBotLogin` (`isHiveAppReviewAuthor`, `src/pkg/github/automerge/automerge_sweep.go:1242`). The text after the mention is the request. Two optional forms route it: ``` diff --git a/src/docs/labels-and-control-signals.md b/src/docs/labels-and-control-signals.md index 73d596a71e..e77234e7dd 100644 --- a/src/docs/labels-and-control-signals.md +++ b/src/docs/labels-and-control-signals.md @@ -8,9 +8,9 @@ Most hard gates run before an agent sees work. GitHub issue enumeration reads la Some sweeps bypass that enumeration and list items themselves, but every one of them gates on the same hold predicate as enumeration (`Client.IsHeldLabels` / `Client.isHeld`, `src/pkg/github/client.go:1969-1977`, `src/pkg/github/client.go:2679-2681`), so the generic hold substrings and the exact dashboard `hive-pause/` label hold everywhere: -- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; both the queued (`lgtm`) and the self-authored sweep skip a held PR before fetching it (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:959-964`, `src/pkg/github/automerge/automerge_sweep.go:966-1004`, `src/pkg/github/automerge/automerge_sweep.go:1029-1046`). +- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; both the queued (`lgtm`) and the self-authored sweep skip a held PR before fetching it (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`). - The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:797-815`, `src/cmd/hive/claimescalation.go`). -- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; the queued (`lgtm`) sweep skips a held PR before fetching it, and the self-authored sweep skips held PRs plus `needs-human` before merging (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:959-964`, `src/pkg/github/automerge/automerge_sweep.go:966-1004`, `src/pkg/github/automerge/automerge_sweep.go:1029-1046`). +- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; the queued (`lgtm`) sweep skips a held PR before fetching it, and the self-authored sweep skips held PRs plus `needs-human` before merging (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`). - The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:848-854`). - The SHA-hold sweep lists primary-repo `kind/bug` issues itself, adds literal `hold` plus a marker notice when no SHA is present (for reporters in `issue_filter.reporter_trust` trusted associations it adds the soft `issues.sha_hold_trusted_label`, default `needs-sha`, instead, #12002), and removes `hold` only when the current hold is its own: the newest `hold` label event is a `labeled` by the App bot and an App-authored SHA-hold notice was posted at or after it (`EnforceSHAHold` / `shaHoldIsOwn` in `src/pkg/github/client.go`). - PR-request watching applies server-side PR holds after a PR is opened; the agent's `--label hold` flag is deliberately discarded by the wrapper (`src/pkg/github/pr_request_watcher.go:452-524`, `bin/hive-open-pr.sh:142-150`). @@ -101,7 +101,7 @@ Some hive-authored labels look like provenance but are read as signals, so they | `tracker`, `meta-tracker`, `tracking`, `epic` (last segment) | Issues | Tracker detector, PR claim rewrite, contributor queue | Marks coordination-only/tracker issues; agents see tracker tags, contributors do not get them. | Soft for agents; hard for contributors | Human/Hive | Enumeration/contributor admission | Remove label or close tracker | Tracker detector (`src/pkg/github/client.go:2095-2161`) | | `epic` (any case; `governor.claims.never_claim_labels`) | Issues | Issue-claim admission, issue stage derivation | Umbrella issue is never a unit of agent work: every automated claim producer (relay `claim`, start signal, contributor lane) is refused with `never-claim label: