Skip to content

🐛 fix(claims): open-PR guard backs off until rate-limit reset and refuses claims meanwhile - #175

Merged
hanthor merged 2 commits into
tuna/v6from
fix/open-pr-guard-ratelimit
Nov 13, 2026
Merged

hanthor merged 2 commits into
tuna/v6from
fix/open-pr-guard-ratelimit

Conversation

@hanthor

@hanthor hanthor commented Nov 6, 2026

Copy link
Copy Markdown
Member

Production evidence

When the GitHub installation's limit runs out, go-github stops making requests and fails every call locally with *github.RateLimitError ("403 API rate limit of 6450 still exceeded until , not making remote request"). openPRIndexFor (src/pkg/github/open_pr_links.go) cached only successful listings. As a result:

  • every claim candidate, and both repo spellings in contribute_admission.go openPRLinkingIssue, ran the listing again right away;
  • OpenPRLinkGuard logged open-PR guard: could not list open PRs, admitting the claim on ledger evidence alone on every call: ~47,000 warnings in one hour on one hive, ~144k on another;
  • the guard failed open. While rate-limited, it admitted issues that an open PR already links, so they could be claimed and dispatched.

Change

  1. Cherry-pick of upstream 🐛 fix(claims): bound and classify the open-PR guard's listing timeout hivecommons/hive#16303 (977d7ffd7, which tuna/v6 did not have yet). It adds the listing timeout and its classification, plus a one-minute per-repo failure cache. I kept it as its own commit with -x so the later merge from upstream stays clean.
  2. Rate-limit backoff. A listing that fails with a primary RateLimitError or a secondary AbuseRateLimitError is cached per canonical repo until the limit resets. The reset comes from Rate.Reset, Retry-After, or the response headers (reuses retryableGitHubDelay). The backoff is capped at 1h and falls back to 1m when the reset time is unknown. Until then the cached error is returned without calling GitHub. Other errors keep the 1m backoff, so a listing that keeps failing can't hot-loop.
  3. Rate limits fail closed. While the listing is rate-limited, the guard can't see whether a PR already links the issue. Any work it would dispatch needs the same exhausted limit anyway, so waiting for the reset costs nothing. Behaviour for each caller:
    • OpenPRLinkGuard returns ok=true with OpenPRLink.RateLimitedUntil set. PR fields are empty and no has-open-pr label is written.
    • IssueDispatchBlockReason / openPRLinkBlockReason return a new distinct reason, ClaimBlockReasonOpenPRRateLimited = open-PR guard deferred: GitHub rate limit.
    • The contribute lane (the OpenPRLinkingIssue hook in cmd/hive/main.go, whose signature is unchanged) withholds the candidate under a new reason, open_pr_guard_rate_limited. The Withheld detail shows the reset time. A deferral on the first repo spelling also covers the second spelling.
    • Non-rate-limit failures (timeouts, 5xx) still fail open, as the doc comment already promised.
  4. Logging. The client warns once per repo per backoff window. The contribute lane does not log each deferred item.
  5. The doc comments, src/docs/operator-reference.md, and a changelog.d fragment are updated.

Tests

New tests in src/pkg/github/open_pr_links_test.go. The test server can now answer with a primary 403 (X-RateLimit-Remaining: 0, X-RateLimit-Reset) or a secondary 403 (Retry-After). The tests use the cache's injectable now:

  • TestOpenPRGuard_RateLimitedListingIsNotRetriedUntilReset: 20 guard calls over 19 fake minutes, alternating o/r and r, make 1 listing request and produce 1 warning.
  • TestOpenPRGuard_RateLimitedListingRefusesClaim: the dispatch is refused with ClaimBlockReasonOpenPRRateLimited and no label is written.
  • TestOpenPRGuard_RateLimitedListingRetriesAfterReset: the issue is still deferred 1s before the reset. 1s after it, the guard lists again and refuses on the real PR (open PR already linked: #14775).
  • TestOpenPRGuard_SecondaryRateLimitRefusesForRetryAfter: the backoff follows Retry-After, not the 1m failure TTL.
  • In the dashboard, a withheld-table case covers open_pr_guard_rate_limited.

Before the fix, run against tuna/v6 @ 230f74501. The new-API assertions were rewritten against the base API so the file compiles there:

open_pr_links_test.go:297: listing attempts = 20, want 1 until the rate limit resets
--- FAIL: TestOpenPRGuard_RateLimitedListingIsNotRetriedUntilReset (45.01s)
open_pr_links_test.go:313: reason = "", <nil>; want "open-PR guard deferred: GitHub rate limit" while rate-limited
--- FAIL: TestOpenPRGuard_RateLimitedListingRefusesClaim
--- FAIL: TestOpenPRGuard_RateLimitedListingRetriesAfterReset
--- FAIL: TestOpenPRGuard_SecondaryRateLimitRefusesForRetryAfter

With only the upstream hivecommons#16303 cherry-pick, the same four tests still fail: 20 listings, because the 1m TTL expires between calls, and the claim is still admitted.

After the fix:

cd src && go build ./... && go vet ./pkg/github/... ./pkg/dashboard/... ./cmd/hive/... && go test ./pkg/github/... ./pkg/dashboard/...
ok  github.com/hivecommons/hive/pkg/github            65.293s
ok  github.com/hivecommons/hive/pkg/github/automerge   7.754s
ok  github.com/hivecommons/hive/pkg/github/requestwatch
ok  github.com/hivecommons/hive/pkg/dashboard         130.282s
ok  github.com/hivecommons/hive/pkg/dashboard/{collect,theme,webstatic}
go test ./cmd/hive/ -run 'OpenPR|ClaimsAdmission|Admission'   ok

Nothing has been deployed.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Open PR overlap check

Base branch: tuna/v6

Conflicts are checked with every PR merged onto the current tip of tuna/v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#172 ✨ Refresh v6 pilot and enable upstream ambient code-rain testing same files src/docs/operator-reference.md
src/pkg/dashboard/deps.go
src/pkg/github/open_pr_links.go

@tuna-reef-hive tuna-reef-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Nov 6, 2026
@tuna-reef-hive

tuna-reef-hive Bot commented Nov 6, 2026

Copy link
Copy Markdown

@hanthor this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • check-failure

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@tuna-reef-hive tuna-reef-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 6, 2026
@tuna-reef-hive

tuna-reef-hive Bot commented Nov 7, 2026

Copy link
Copy Markdown

@hanthor is now trusted for contributor-lane merging in this hive (last 5 merged PRs: no reverts or sentinel alerts). Applies to their open and future PRs here; CI, review verdicts, diff fences, soak and branch protection still apply.

@tuna-reef-hive tuna-reef-hive Bot added stage/stalled In its current stage longer than expected and removed stage/stalled In its current stage longer than expected labels Nov 8, 2026
@hanthor hanthor added the merge-method/merge Hive merges this PR with a merge commit (mixed authorship, each commit self-signed); never squash label Nov 13, 2026
hivecommons-hive Bot and others added 2 commits November 13, 2026 03:51
…hivecommons#16303)

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
(cherry picked from commit 977d7ff)
…uses claims meanwhile

While the GitHub installation is rate-limited, go-github fails every call
locally with a RateLimitError. openPRIndexFor cached only successful
listings, so every claim candidate (and both repo spellings in the
contribute lane) re-listed immediately and OpenPRLinkGuard warned
"could not list open PRs, admitting the claim on ledger evidence alone"
each time — 47k warnings/hour on one hive, 144k on another — and the guard
failed OPEN, admitting issues an open PR already links.

- Remember a rate-limited listing (primary RateLimitError or secondary
  AbuseRateLimitError) per canonical repo until the reset (Rate.Reset /
  Retry-After, capped at 1h; 1m when unknown) and replay it without a
  GitHub call. Other failures keep the one-minute backoff from hivecommons#16303.
- Rate-limited listings fail CLOSED: OpenPRLinkGuard returns ok with
  OpenPRLink.RateLimitedUntil set (no label written); IssueDispatchBlockReason
  refuses with ClaimBlockReasonOpenPRRateLimited; the contribute lane
  withholds the candidate as open_pr_guard_rate_limited. Non-rate-limit
  failures still fail open.
- Warn once per repo per backoff window, not per candidate.

Signed-off-by: James Reilly <jreilly1821@gmail.com>
@hanthor
hanthor force-pushed the fix/open-pr-guard-ratelimit branch from 4479a62 to e782500 Compare November 13, 2026 03:51
@hanthor

hanthor commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Decision (for @hanthor): merge with a merge commit.

Why the sign-off check failed: commit 1 is an upstream cherry-pick (hivecommons/hive 977d7ff, hivecommons#16303) authored and signed off by hivecommons-hive[bot]. A squash would re-attribute it to hanthor under the bot's sign-off. The legitimate fix is the repo's existing merge-method/merge election: the bot commit lands unchanged with its own author and sign-off, and the rate-limit fix keeps James Reilly's own sign-off. No sign-off was rewritten or forged, and the checker is unchanged.

What ran (locally, on head e782500 rebased onto tuna/v6 efb4c5d). CI can't run because GitHub has blocked Actions on tuna-os/hive (Actions has been disabled for this repository, the same block as compass/iced/rdocx/athens; support ticket pending):

  • check-squash-signoff-attribution.sh with PR_MERGE_METHOD=merge → passed (self-test test-check-squash-signoff-attribution.sh OK)
  • go build ./..., go vet ./... → clean; gofmt -l on changed Go files → clean
  • go test ./pkg/github/ ./pkg/dashboard/ → dashboard ok. In github, the only failure is TestReopenedClaimPersistsAcrossLoad, which fails identically on tuna/v6 HEAD. It's a pre-existing clock-dependent test, fixed upstream in b7279fe (🌱 test(github): the reopened-claim reload test reads the ledger at a fixed instant hivecommons/hive#18055), which the upstream refresh brings in.
  • check-changelog-fragment.sh tuna/v6 → OK (fragments present)

This closes a production fail-open: the claim guard admitted claims while rate-limited. Merging doesn't redeploy the spokes, which run a pinned binary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco All commits are DCO signed-off merge-method/merge Hive merges this PR with a merge commit (mixed authorship, each commit self-signed); never squash role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant