Skip to content

🐛 fix(github): back off a header-less secondary 403 for a minute, not an hour - #15050

Merged
hivecommons-hive[bot] merged 1 commit into
hivecommons:v6from
Danathar:fix/secondary-403-short-caution
Oct 28, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
hivecommons:v6from
Danathar:fix/secondary-403-short-caution

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

After the projectbluefin hive restarted on 2026-10-28 at 17:17:25Z, one secondary-limit 403 on /search/issues put the whole hive into a 70-minute caution window. No PR merged until it ended, though the primary quota sat at 87.6%. From the hive's log (UTC):

  • 17:22:28 github slow-start caution window engaged kind=secondary path=/search/issues until=18:32:28Z. That is 1h of back-off plus the 10m window.
  • 17:22:34 a burst of secondary 403s on core endpoints (pulls/774/reviews, collaborators/…/permission, labels) from requests already in flight. go-github's own backoff for those was about 1s.
  • 18:04:22 self-authored automerge sweep skipped PR repo=finpilot pr=599 reason=status-check error="…commits/a88273c…/status … request shed while rate-limit caution window is engaged (class low, queue 5.454s ahead, limit 5s)". finpilot#599 was approved, 19/19 green and soaked.
  • No merge from 17:17 to at least 18:10Z. fleet_status showed core remaining 6134 and no consumer rate-limited.

Two causes, both in pkg/github:

Change

  • Back-off length (slowstart.go; slowStartDefaultRetryAfter is gone):
    • A parseable Retry-After wins (seconds or an HTTP date).
    • Next, X-RateLimit-Reset, only when X-RateLimit-Remaining is 0. With quota left, it is the primary bucket's reset, up to 1h away, so it is ignored.
    • Otherwise 60s, doubling on each re-trip within 30 minutes of the previous back-off ending: 1m, 2m, 4m, 8m, capped at 15m. 403s from the same burst, which arrive while the back-off runs, neither escalate nor extend it. A trip after a quiet spell starts over at 1m.
    • The 10m slow-start window still follows each back-off.
  • Merge decisions are not shed (mergeDecisionPriority). Once the queued, self-authored, trusted-author or contributor lane has a candidate PR, that PR's evaluation and merge run at normal class.
  • Test helper: EngageRateLimitCautionForTest lets other packages test under caution without minute-long sleeps.
  • Docs: the "Rate-limit pacing priority" section of operator-reference.md. The insert shifted automerge_sweep.go, so five citations in labels-and-control-signals.md and design/github-mention-triggers.md were moved with it.
  • Changelog: changelog.d/fixed-secondary-403-short-caution.md.

Not included, and why

  • The boot-time count reconcile is already staggered. forgestate/counts.go caps it at 5 repos per 2-minute pass (🐛 dashboard: keep hive-attributed cost divisors on many-repo hives #13625), which is the "partial: 5 of 19" in the log. Its failures came after the window opened; they did not trip it.
  • Scoping caution to the endpoint family that tripped would not have prevented this. GitHub shares the secondary concurrency and points-per-minute limits across REST and GraphQL, and the core endpoints above got their own secondary 403s six seconds after the search one. Scoping caution for a search primary limit (X-RateLimit-Resource: search, Remaining 0) could still be worth doing; that is left for a follow-up.

Testing

From src/, TZ=UTC, -count=1:

  • go test ./pkg/github/... ./pkg/forgestate/ passes, as do pkg/apphealth and the related pkg/dashboard tests.
  • go build ./... and go vet are clean, and gofmt is clean on the touched files.
  • The changelog, docs-links and docs-citations checks pass.

New tests:

  • TestSlowStart_HeaderlessSecondaryBacksOffOneMinute
  • TestSlowStart_SecondaryBackoffEscalatesAndCaps
  • TestSecondaryLimitBackoff_HonoursNamedWaits
  • TestMergeDecisionPriority
  • TestSweepMergesGreenPRWhileCautionQueueExceedsLowBudget: with a 5.5s caution queue, a green, approved self-authored PR still has its status read and is merged.

Mutation checks: 12 mutants, each restored from a backup copy, and every one fails a test:

  • no escalation, no same-burst rule, no cap, no streak reset;
  • X-RateLimit-Reset honoured with quota left;
  • the 1m–15m ladder skipped, no HTTP-date parsing, the old 1h default;
  • a Remaining-0 reset ignored;
  • the self-authored call site reverted;
  • mergeDecisionPriority made a no-op.

Still open

  • Untested call sites. The trusted-author, contributor and queued sites have no test of their own.
  • The floor pass is still skipped while the queue is more than 5s deep; event passes still run.
  • Not run on a live hive.

Contributor checklist

  • Changelog: changelog.d/fixed-secondary-403-short-caution.md.
  • Targets v6.
  • The title uses the emoji convention.
  • Commits include DCO sign-off.
  • Docs updated (src/docs/operator-reference.md, plus citation moves).
  • No secrets, credentials or local runtime state are committed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

… an hour

A secondary-limit 403 that named no wait (no Retry-After) assumed
slowStartDefaultRetryAfter = 1h, plus the 10m slow-start window. For those
70 minutes every low-class request queued more than 5s was shed, including
the automerge sweep's commit-status read. On the projectbluefin hive
(2026-10-28, build 7f89f9d) one /search/issues 403 at 17:22Z after a restart
meant no merges until at least 18:10Z. finpilot#599 (approved, 19/19 green)
was skipped with reason "status-check" ("request shed ... class low, queue
5.454s ahead, limit 5s") while core quota stood at 87%.

- slowstart.go: a secondary 403 now uses GitHub's documented order. A
  parseable Retry-After (seconds or an HTTP date) wins. Next comes
  X-RateLimit-Reset, but only with X-RateLimit-Remaining: 0. Otherwise the
  back-off is one minute, doubling on each repeat within 30 minutes of the
  last back-off (1m, 2m, 4m, 8m, capped at 15m). The rest of a burst refused
  by the same block does not escalate. X-RateLimit-Reset with quota left is
  the primary bucket's reset and is ignored. The slow-start window still
  follows every back-off.
- automerge: once a lane (queued, self-authored, trusted-author or
  contributor) reaches a candidate PR, that PR's evaluation and merge run at
  normal pacing class (mergeDecisionPriority). The tick and floor pass gates
  and the pre-candidate reads stay low class. The lanes evaluate one request
  at a time, so this adds at most one slot and waits at most 30s, not the
  pile-up hivecommons#13382 introduced the low class for.
- EngageRateLimitCautionForTest lets other packages test behaviour under
  caution.
- Docs: operator-reference "Rate-limit pacing priority". Shifted
  automerge_sweep.go line citations kept on the same lines.

Tests: header-less 403 gives caution of 1m + window; escalation, same-burst,
cap and reset; named waits honoured (Retry-After seconds and date, Reset
only with Remaining 0); a sweep merges a green App PR while the caution
queue is 5.5s deep. Each rule was mutation-checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 28, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 28, 2026
@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 28, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed — no findings from this perspective.

— hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#14945 🐛 fix: release label-blocked serialized lane fronts during sweeps CONFLICT src/docs/labels-and-control-signals.md
#15029 ✨ feat(contribute): record the blocker a relay's blocked verdict names same files src/docs/labels-and-control-signals.md
#14947 🐛 fix: reject auto-merge queues from untrusted merger identities same files src/pkg/github/automerge/automerge_sweep.go
#15035 🐛 fix(automerge): hold merges whose closing keywords would close the wrong thing same files src/docs/labels-and-control-signals.md
src/pkg/github/automerge/automerge_contributor_lane.go
src/pkg/github/automerge/automerge_sweep.go

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v6 plus pacing_priority.go, automerge_sweep.go and automerge_contributor_lane.go. Looks correct to me.

  • secondaryBackoffLocked (slowstart.go): the same-burst rule holds — a header-less 403 arriving while secondaryBackoffEnd is in the future returns the remainder, so until = backoffEnd + window and until.After(st.cautiousUntil) is false; it neither escalates nor extends. The escalation loop stops at slowStartSecondaryBackoffMax (15m) and the final cap catches the overshoot of the last doubling. Streak resets after the 30m quiet spell.
  • secondaryLimitBackoff: the old behaviour where an unparseable Retry-After meant a flat hour now falls through to named=false → 1m ladder, and X-RateLimit-Reset is consulted only via primaryLimitReset, which already requires Remaining: 0. A permission 403 with no header and no body phrase still returns secondary=false.
  • mergeDecisionPriority: all four call sites sit under a lowPriority(ctx) (contributor lane :389, self-authored :1029, trusted :1232, and queued via the tick wrapper at :1470), so the lift is real, and a PacingHigh ctx passes through unchanged (TestMergeDecisionPriority).
  • No new time.Sleep( in tests under pkg/ (sleep ratchet). Note TestSweepMergesGreenPRWhileCautionQueueExceedsLowBudget costs ~5.5s wall-clock by design and mutates the process-wide pacer; fine as long as nothing in automerge goes t.Parallel().

Body and diff agree on every claim I checked. The conflicts-with-open-pr label is the one thing a maintainer should look at before merge; the code itself I have no findings on.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective — no blocking findings.

One advisory (low): secondaryLimitBackoff honours Retry-After without an upper bound (src/pkg/github/slowstart.go:431-440 at da7d586). The seconds form was already uncapped at merge-base; the new HTTP-date form adds a second uncapped path. The X-RateLimit-Reset branch right below it is clamped to primaryLimitMaxReset (slowstart.go:387-388), so a single oversized or clock-skewed header from GitHub (or a proxy in front of it) could still pin the pacer for the hour-plus this PR sets out to remove. The header comes from the same trusted origin as before, so this is not a trust-boundary regression — just a cap worth adding while you're here:

		if secs, err := strconv.Atoi(ra); err == nil && secs >= 0 {
			return min(time.Duration(secs)*time.Second, primaryLimitMaxReset), true, true
		}
		if at, err := http.ParseTime(ra); err == nil {
			if d := at.Sub(now); d > 0 {
				return min(d, primaryLimitMaxReset), true, true
			}
			return 0, true, true
		}

mergeDecisionPriority only changes pacing class on an already-validated candidate; the trySweep* authorization paths are untouched by this diff, so no permission change.

— hive: agent=adjudicator-3 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit f950551 into hivecommons:v6 Oct 28, 2026
67 of 70 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor
Evidence bundle `#15050da7d586` sha256:513e6498efb5868e4c2d574227c213ba614017436b995046c2c1527c2569a6f6 — download from the dashboard or `hivectl review evidence #15050`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at da7d586.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login conflicts-with-open-pr Open PR has a merge conflict with another open PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant