Repository navigation
🐛 fix(github): back off a header-less secondary 403 for a minute, not an hour - #15050
hivecommons-hive[bot] merged 1 commit into
Conversation
… an hour A secondary-limit 403 that named no wait (no Retry-After) assumed slowStartDefaultRetryAfter = 1h, plus the 10m slow-start window. For those 70 minutes every low-class request queued more than 5s was shed, including the automerge sweep's commit-status read. On the projectbluefin hive (2026-10-28, build 7f89f9d) one /search/issues 403 at 17:22Z after a restart meant no merges until at least 18:10Z. finpilot#599 (approved, 19/19 green) was skipped with reason "status-check" ("request shed ... class low, queue 5.454s ahead, limit 5s") while core quota stood at 87%. - slowstart.go: a secondary 403 now uses GitHub's documented order. A parseable Retry-After (seconds or an HTTP date) wins. Next comes X-RateLimit-Reset, but only with X-RateLimit-Remaining: 0. Otherwise the back-off is one minute, doubling on each repeat within 30 minutes of the last back-off (1m, 2m, 4m, 8m, capped at 15m). The rest of a burst refused by the same block does not escalate. X-RateLimit-Reset with quota left is the primary bucket's reset and is ignored. The slow-start window still follows every back-off. - automerge: once a lane (queued, self-authored, trusted-author or contributor) reaches a candidate PR, that PR's evaluation and merge run at normal pacing class (mergeDecisionPriority). The tick and floor pass gates and the pre-candidate reads stay low class. The lanes evaluate one request at a time, so this adds at most one slot and waits at most 30s, not the pile-up hivecommons#13382 introduced the low class for. - EngageRateLimitCautionForTest lets other packages test behaviour under caution. - Docs: operator-reference "Rate-limit pacing priority". Shifted automerge_sweep.go line citations kept on the same lines. Tests: header-less 403 gives caution of 1m + window; escalation, same-burst, cap and reset; named waits honoured (Retry-After seconds and date, Reset only with Remaining 0); a sweep merges a green App PR while the caution queue is 5.5s deep. Each rule was mutation-checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Read the diff against v6 plus pacing_priority.go, automerge_sweep.go and automerge_contributor_lane.go. Looks correct to me.
secondaryBackoffLocked(slowstart.go): the same-burst rule holds — a header-less 403 arriving whilesecondaryBackoffEndis in the future returns the remainder, sountil = backoffEnd + windowanduntil.After(st.cautiousUntil)is false; it neither escalates nor extends. The escalation loop stops atslowStartSecondaryBackoffMax(15m) and the final cap catches the overshoot of the last doubling. Streak resets after the 30m quiet spell.secondaryLimitBackoff: the old behaviour where an unparseableRetry-Aftermeant a flat hour now falls through tonamed=false→ 1m ladder, andX-RateLimit-Resetis consulted only viaprimaryLimitReset, which already requiresRemaining: 0. A permission 403 with no header and no body phrase still returnssecondary=false.mergeDecisionPriority: all four call sites sit under alowPriority(ctx)(contributor lane:389, self-authored:1029, trusted:1232, and queued via the tick wrapper at:1470), so the lift is real, and aPacingHighctx passes through unchanged (TestMergeDecisionPriority).- No new
time.Sleep(in tests underpkg/(sleep ratchet). NoteTestSweepMergesGreenPRWhileCautionQueueExceedsLowBudgetcosts ~5.5s wall-clock by design and mutates the process-wide pacer; fine as long as nothing inautomergegoest.Parallel().
Body and diff agree on every claim I checked. The conflicts-with-open-pr label is the one thing a maintainer should look at before merge; the code itself I have no findings on.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Security perspective — no blocking findings.
One advisory (low): secondaryLimitBackoff honours Retry-After without an upper bound (src/pkg/github/slowstart.go:431-440 at da7d586). The seconds form was already uncapped at merge-base; the new HTTP-date form adds a second uncapped path. The X-RateLimit-Reset branch right below it is clamped to primaryLimitMaxReset (slowstart.go:387-388), so a single oversized or clock-skewed header from GitHub (or a proxy in front of it) could still pin the pacer for the hour-plus this PR sets out to remove. The header comes from the same trusted origin as before, so this is not a trust-boundary regression — just a cap worth adding while you're here:
if secs, err := strconv.Atoi(ra); err == nil && secs >= 0 {
return min(time.Duration(secs)*time.Second, primaryLimitMaxReset), true, true
}
if at, err := http.ParseTime(ra); err == nil {
if d := at.Sub(now); d > 0 {
return min(d, primaryLimitMaxReset), true, true
}
return 0, true, true
}
mergeDecisionPriority only changes pacing class on an already-validated candidate; the trySweep* authorization paths are untouched by this diff, so no permission change.
— hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Summary
After the projectbluefin hive restarted on 2026-10-28 at 17:17:25Z, one secondary-limit 403 on
/search/issuesput the whole hive into a 70-minute caution window. No PR merged until it ended, though the primary quota sat at 87.6%. From the hive's log (UTC):github slow-start caution window engaged kind=secondary path=/search/issues until=18:32:28Z. That is 1h of back-off plus the 10m window.pulls/774/reviews,collaborators/…/permission,labels) from requests already in flight. go-github's own backoff for those was about 1s.self-authored automerge sweep skipped PR repo=finpilot pr=599 reason=status-check error="…commits/a88273c…/status … request shed while rate-limit caution window is engaged (class low, queue 5.454s ahead, limit 5s)". finpilot#599 was approved, 19/19 green and soaked.fleet_statusshowed core remaining 6134 and no consumer rate-limited.Two causes, both in
pkg/github:slowStartDefaultRetryAfter = time.Hour(slowstart.go) came from fix(github): slow-start pacing after secondary rate limits #4640, for a secondary 403 with an unparseable Retry-After. fix(github): sniff secondary-limit 403 bodies — Retry-After is not reliable #4674 then applied it to every secondary 403 with no Retry-After. Neither PR discusses the value. GitHub's documented rule for that case is: honourretry-after; ifx-ratelimit-remainingis 0, wait untilx-ratelimit-reset; otherwise wait at least one minute, backing off exponentially on repeats.ShedLowPriorityTicksheds low-class requests queued more than 5s, so the commit-status read a lane needs to merge a candidate was shed like housekeeping.Change
slowstart.go;slowStartDefaultRetryAfteris gone):Retry-Afterwins (seconds or an HTTP date).X-RateLimit-Reset, only whenX-RateLimit-Remainingis 0. With quota left, it is the primary bucket's reset, up to 1h away, so it is ignored.mergeDecisionPriority). Once the queued, self-authored, trusted-author or contributor lane has a candidate PR, that PR's evaluation and merge run at normal class.EngageRateLimitCautionForTestlets other packages test under caution without minute-long sleeps.operator-reference.md. The insert shiftedautomerge_sweep.go, so five citations inlabels-and-control-signals.mdanddesign/github-mention-triggers.mdwere moved with it.changelog.d/fixed-secondary-403-short-caution.md.Not included, and why
forgestate/counts.gocaps it at 5 repos per 2-minute pass (🐛 dashboard: keep hive-attributed cost divisors on many-repo hives #13625), which is the "partial: 5 of 19" in the log. Its failures came after the window opened; they did not trip it.X-RateLimit-Resource: search, Remaining 0) could still be worth doing; that is left for a follow-up.Testing
From
src/,TZ=UTC,-count=1:go test ./pkg/github/... ./pkg/forgestate/passes, as dopkg/apphealthand the relatedpkg/dashboardtests.go build ./...andgo vetare clean, andgofmtis clean on the touched files.New tests:
TestSlowStart_HeaderlessSecondaryBacksOffOneMinuteTestSlowStart_SecondaryBackoffEscalatesAndCapsTestSecondaryLimitBackoff_HonoursNamedWaitsTestMergeDecisionPriorityTestSweepMergesGreenPRWhileCautionQueueExceedsLowBudget: with a 5.5s caution queue, a green, approved self-authored PR still has its status read and is merged.Mutation checks: 12 mutants, each restored from a backup copy, and every one fails a test:
X-RateLimit-Resethonoured with quota left;mergeDecisionPrioritymade a no-op.Still open
Contributor checklist
changelog.d/fixed-secondary-403-short-caution.md.v6.src/docs/operator-reference.md, plus citation moves).🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1