Repository navigation
🐛 fix(claims): open-PR guard backs off until rate-limit reset and refuses claims meanwhile - #175
Conversation
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@hanthor this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
@hanthor is now trusted for contributor-lane merging in this hive (last 5 merged PRs: no reverts or sentinel alerts). Applies to their open and future PRs here; CI, review verdicts, diff fences, soak and branch protection still apply. |
…hivecommons#16303) Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> (cherry picked from commit 977d7ff)
…uses claims meanwhile While the GitHub installation is rate-limited, go-github fails every call locally with a RateLimitError. openPRIndexFor cached only successful listings, so every claim candidate (and both repo spellings in the contribute lane) re-listed immediately and OpenPRLinkGuard warned "could not list open PRs, admitting the claim on ledger evidence alone" each time — 47k warnings/hour on one hive, 144k on another — and the guard failed OPEN, admitting issues an open PR already links. - Remember a rate-limited listing (primary RateLimitError or secondary AbuseRateLimitError) per canonical repo until the reset (Rate.Reset / Retry-After, capped at 1h; 1m when unknown) and replay it without a GitHub call. Other failures keep the one-minute backoff from hivecommons#16303. - Rate-limited listings fail CLOSED: OpenPRLinkGuard returns ok with OpenPRLink.RateLimitedUntil set (no label written); IssueDispatchBlockReason refuses with ClaimBlockReasonOpenPRRateLimited; the contribute lane withholds the candidate as open_pr_guard_rate_limited. Non-rate-limit failures still fail open. - Warn once per repo per backoff window, not per candidate. Signed-off-by: James Reilly <jreilly1821@gmail.com>
4479a62 to
e782500
Compare
|
Decision (for @hanthor): merge with a merge commit. Why the sign-off check failed: commit 1 is an upstream cherry-pick (hivecommons/hive 977d7ff, hivecommons#16303) authored and signed off by What ran (locally, on head e782500 rebased onto tuna/v6 efb4c5d). CI can't run because GitHub has blocked Actions on tuna-os/hive (
This closes a production fail-open: the claim guard admitted claims while rate-limited. Merging doesn't redeploy the spokes, which run a pinned binary. |
Production evidence
When the GitHub installation's limit runs out, go-github stops making requests and fails every call locally with
*github.RateLimitError("403 API rate limit of 6450 still exceeded until , not making remote request").openPRIndexFor(src/pkg/github/open_pr_links.go) cached only successful listings. As a result:contribute_admission.goopenPRLinkingIssue, ran the listing again right away;OpenPRLinkGuardloggedopen-PR guard: could not list open PRs, admitting the claim on ledger evidence aloneon every call: ~47,000 warnings in one hour on one hive, ~144k on another;Change
977d7ffd7, whichtuna/v6did not have yet). It adds the listing timeout and its classification, plus a one-minute per-repo failure cache. I kept it as its own commit with-xso the later merge from upstream stays clean.RateLimitErroror a secondaryAbuseRateLimitErroris cached per canonical repo until the limit resets. The reset comes fromRate.Reset,Retry-After, or the response headers (reusesretryableGitHubDelay). The backoff is capped at 1h and falls back to 1m when the reset time is unknown. Until then the cached error is returned without calling GitHub. Other errors keep the 1m backoff, so a listing that keeps failing can't hot-loop.OpenPRLinkGuardreturnsok=truewithOpenPRLink.RateLimitedUntilset. PR fields are empty and nohas-open-prlabel is written.IssueDispatchBlockReason/openPRLinkBlockReasonreturn a new distinct reason,ClaimBlockReasonOpenPRRateLimited=open-PR guard deferred: GitHub rate limit.OpenPRLinkingIssuehook incmd/hive/main.go, whose signature is unchanged) withholds the candidate under a new reason,open_pr_guard_rate_limited. The Withheld detail shows the reset time. A deferral on the first repo spelling also covers the second spelling.src/docs/operator-reference.md, and achangelog.dfragment are updated.Tests
New tests in
src/pkg/github/open_pr_links_test.go. The test server can now answer with a primary 403 (X-RateLimit-Remaining: 0,X-RateLimit-Reset) or a secondary 403 (Retry-After). The tests use the cache's injectablenow:TestOpenPRGuard_RateLimitedListingIsNotRetriedUntilReset: 20 guard calls over 19 fake minutes, alternatingo/randr, make 1 listing request and produce 1 warning.TestOpenPRGuard_RateLimitedListingRefusesClaim: the dispatch is refused withClaimBlockReasonOpenPRRateLimitedand no label is written.TestOpenPRGuard_RateLimitedListingRetriesAfterReset: the issue is still deferred 1s before the reset. 1s after it, the guard lists again and refuses on the real PR (open PR already linked: #14775).TestOpenPRGuard_SecondaryRateLimitRefusesForRetryAfter: the backoff followsRetry-After, not the 1m failure TTL.open_pr_guard_rate_limited.Before the fix, run against
tuna/v6@230f74501. The new-API assertions were rewritten against the base API so the file compiles there:With only the upstream hivecommons#16303 cherry-pick, the same four tests still fail: 20 listings, because the 1m TTL expires between calls, and the claim is still admitted.
After the fix:
Nothing has been deployed.