Repository navigation
π fix(dashboard): Let hive decide preview names GitHub rate limits, not hive unavailable - #16951
Conversation
β¦not "hive unavailable" The preview gathers the item live from GitHub and answered any failure with 502. On a hosted hive the Ingress swaps every 502 and 503 for its canned "hive unavailable" body, so while the App was out of quota every preview read "Could not preview <item> (hive unavailable)" with the hive up. The plan now maps the gather error: a primary or secondary rate limit (or a request the pacer shed) answers 429 with Retry-After and the time it lifts; any other GitHub answer (404, permission 403), a closed item or a missing snapshot answers 424 with the real message; only a request that never got an answer stays 502. pkg/github gains RateLimitRetryDelay, built on the existing rate-limit classifier and retry-delay reader. The dashboard shows the server's message and, for a 429 without a time, "retry after HH:MMZ" from Retry-After. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
Read the diff against the v6 tree and the retry helpers it reuses. Looks correct to me; go build ./pkg/github/ ./pkg/dashboard/ is clean on this head.
errors.Asreaches the go-github types because every gather failure is%w-wrapped (api_repo_item_decide_delegated.go:658,676,703,707), and go-github already turns the "403 API rate limit β¦ exceeded" answer in your log into*gh.RateLimitError, so that case hits the 429 branch atdelegated_gather_error.go:44-46.RateLimitRetryDelay(github_retryable.go:24-31) passesfallback=0, fallbackMax=0, explicitMax=1hintoretryableGitHubDelay; when GitHub names no reset the clamp at:98-100yields 0, which the caller replaces with the one-minute floor β so a bare 429 never emitsRetry-After: 0.- Status split reads right: a permission 403 or 404 is
*gh.ErrorResponsewithout the rate-limit shape β 424 with GitHub's words;net.Error/ deadline β 502, which is the one case the Ingress rewrite is still appropriate for. - The JS only appends its own "retry after" when the server message lacks an
HH:MMZ(tracker-actions.js:1785), so the Go-side times are never doubled.
One note, not a defect: an explicit reset further than an hour out is clamped to 1h by explicitMax, so "resets at" could read early in that case. Primary-limit windows are β€1h, so it should not occur in practice.
β hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the mapping against the tree rather than the diff alone; this looks correct to me.
src/pkg/dashboard/delegated_gather_error.go:38callsRateLimitRetryDelay, which in turn callsretryableGitHubDelay(err, now, 0, 0, time.Hour)(src/pkg/github/github_retryable.go:27). Withfallback=0, fallbackMax=0, the clamp atgithub_retryable.go:99-100yields0whenever GitHub named neither a reset norRetry-After, so the one-minute fallback atdelegated_gather_error.go:40is actually reached; with an explicit reset it is preserved up to 1 h, which covers the primary window.*gh.RateLimitError/*gh.AbuseRateLimitErrorare matched before the*gh.ErrorResponsebranch, and a bare 429 is caught by theStatusCode != 429guard, so no rate-limit shape falls through to 424.net.Error/context.DeadlineExceededkeep 502 (delegated_gather_error.go:62); the hosted Ingress only rewrites502,503(src/pkg/hub/saas_provision.go:3706), so 429 and 424 reach the dialog intact.- The JS path:
r.json().catch(() => ({}))on the Ingress HTML page givesmsg='', andstageTrackerDelegatePlanErrorTextthen returnshive unavailablefor 502/503 β matches the documented wording.
One observation, not a defect: context.Canceled (client went away mid-gather) now answers 424 instead of 502. The client is gone by then so nothing reads it.
CI is green on 12dd36f. A maintainer can take this as-is.
β hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Summary
While the hive's GitHub App is out of quota, every Let hive decide preview failed with "(hive unavailable)", even though the hive was up. The preview answered any GitHub read failure with 502, and the hosted Ingress swaps every 502 and 503 for its own "hive is restarting" body. The preview now answers 429 with a
Retry-Afterheader and the time the limit lifts, and other GitHub refusals answer 424 with GitHub's own words. The dialog shows that message. 502 is left for a GitHub that could not be reached at all.What went wrong
On 2026-11-06 around 17:00-17:15Z, the projectbluefin hive showed "Could not preview projectbluefin/testsuite#1026 (hive unavailable)" on every item. Its logs at the same time said
403 API rate limit of 7000 still exceeded until 2026-11-06 12:32:06 -0500 EST.The chain:
handleRepoItemDecideDelegatedPlanreads the item live from GitHub and turned any error into502 Bad Gateway.custom-http-errors: "502,503"(src/pkg/hub/saas_provision.go), so ingress-nginx replaced that 502 with the canned{"error":"hive unavailable", β¦}fromsrc/deploy/k8s/error-pages.yaml.data.error, which was now "hive unavailable".What changes
The plan handler maps the read error (wrapped errors included, via
errors.As):*github.RateLimitError)Retry-Afterto the reset*github.AbuseRateLimitError) or a bare 429ErrPacingShed)With no time from GitHub, the wait is one minute (GitHub's documented minimum).
pkg/githubgainsRateLimitRetryDelay. It reuses the existingisRateLimitErrclassifier and theretryableGitHubDelayreader of reset time andRetry-After.Dashboard: the preview error shows the server's message. A 429 whose message has no time gets "; retry after HH:MMZ" from
Retry-After. The Ingress's HTML 502 page (not JSON) now reads "hive unavailable" instead of "HTTP 502".Docs: the API reference row for
β¦/decide-delegated/planand the Let hive decide steps in the activity-monitor page.The
β¦/decide-delegatedPOST and the job status GET do not have this bug. The POST answers 202 and the job carries the error text; the GET reads the job from memory. Neither was changed.Files
src/pkg/dashboard/delegated_gather_error.gosrc/pkg/dashboard/api_repo_item_decide_delegated_plan.gosrc/pkg/github/github_retryable.goRateLimitRetryDelaysrc/pkg/dashboard/static-src/js/tracker-actions.jsstageTrackerDelegatePlanErrorText, used by the plan POSTsrc/pkg/dashboard/static/index.htmlsrc/pkg/dashboard/stage_tracker_row_actions_test.gosrc/docs/api-reference/packs-and-acmm.md,src/docs/dashboard-sections/activity-monitor.mdchangelog.d/fixed-delegate-plan-github-rate-limit.mdTests
TestDelegatedDecisionPlanRateLimitedAnswers429WithResetRetry-After, reset time in the messagecode=502 β¦ the Ingress rewrites this to "hive unavailable"(both cases)TestDelegatedGatherErrorNeverAnswers502ForGitHubRefusalsRetry-After; no rate limit or GitHub refusal answers 502/503TestStageTrackerDelegatePlanRateLimitMessage(node)index.html does not define stageTrackerDelegatePlanErrorText()TestRateLimitRetryDelay(pkg/github)The rate-limit test cleans up the shared pacer the way
apphealthdoes. It sendsX-RateLimit-Limit: 60so the shared core-bucket snapshot is not touched.Also run: the delegated-decision, decision, stage-tracker and settings-leaf tests in
pkg/dashboard, the rate-limit tests inpkg/github,go veton both, the docs citation, link and API reference checks, and the changelog check.Found, not changed
Other 502/503 answers (same class as #4241). The write handlers have the same pattern, but changing them is not mechanical. They run under
recordOperatorAction, andoperatorActionOutcome(src/pkg/dashboard/operator_actions.go:468-473) records a 5xx asfailedbut does not record a 4xx at all. Remapping them would drop failed actions from the row's history. So they are left for a separate change:api_repo_item_decision.goapi_repo_item_accept_triage.goapi_repo_item_decision_reroute.goitem_blocker.go,api_repo_hold.go,api_review_queue_superseded.go,copilot_auth.goplan_handlers.go,api_review_pipeline.go,api_agents.goBoth delegated handlers also answer "GitHub client not configured" with 503, which the Ingress hides the same way. That is a configuration state, not a GitHub failure, so it is not changed here.
High-priority pacing and last-good fallback. A pacing class exists (
src/pkg/github/pacing_priority.go:39-47:PacingHigh,PacingNormal,PacingLow). The preview runs asPacingNormal. But a class only orders the hive's own queue while the slow-start caution window is engaged. It cannot help with primary exhaustion: go-github refuses every core call before the request leaves the process. A last-good snapshot exists per repo for the enumeration (src/pkg/github/enumerate_lastgood.go:36, 2 h max age; cache type at:54), and the dashboard keeps the stage-tracker snapshot (s.status). Both live only in memory, so a restart empties them, as in this incident. A fallback preview would also lack most of what the rules read (CI per head, reviews, files, author permission, contributor activity). A follow-up would need: (1) a per-item cache of the last gathereddelegatedecision.Input, kept longer than the 45 s plan cache; (2) on a rate-limit error, a plan from that input marked stale with its age; (3) Let hive do that disabled while stale, since the decision itself needs live reads and writes.What spent the quota. Nothing spreads out the GitHub work after a restart. These figures are read from the code, not measured.
runLoopWithwaits 10 s (src/cmd/hive/run_loop_deps.go:12). It then starts every sweep worker and runs the first full enumeration in the same moment (src/cmd/hive/main.go:6464-6476). Each worker fires on its first tick becauselast.IsZero()(src/cmd/hive/sweep_workers.go:59), so about 19 sweeps start together with no jitter. The merged-PR backfill also runs on every boot (src/cmd/hive/main.go:2790). All the caches that would make this cheap are in memory and start cold, so there are no free 304s after a restart: the ETag cache (src/pkg/github/etagcache.go, no disk writes), the PR-detail cache, the claim cache and the last-good enumeration. The biggest spenders, by rough estimate: the first enumeration with CI enrichment (a PR GET, check runs and sometimes comments or workflow runs for each open PR, plus one comment listing per actionable issue when claims are on); the red-main sweep; the supersession sweep (oneIssues.Getper claimed issue); and the automerge sweep's first pass. For 20 repos that is likely 1.5k-3k core requests in the first few minutes, paid again on each restart. Two restarts 2.5 minutes apart, on a quota already spent once that day, fit the log. With no last-good copy, a fully refused pass returns theall N repos failederror (src/pkg/github/client.go:1924-1926). The budget modes (src/pkg/github/apibudget.go: conserve below 800 remaining, critical below 250) start as normal at boot, because nothing has been observed yet. Per-caller REST accounting exists (/api/gh-rate-limits,src/pkg/github/restaccounting.go), but it is also in memory, so each restart erased the record of what spent the budget. A follow-up could stagger first ticks with jitter after boot, or persist the ETag cache.Open-PR guard fails open.
OpenPRLinkGuard(src/pkg/github/open_pr_links.go:249-286) logsopen-PR guard: could not list open PRs, admitting the claim on ledger evidence aloneand admits the claim when the listing fails. This is intended. The doc comment says "a failure fails OPEN with a warning: the ledger-backed guards downstream still stand, and a flaky listing must not stall every claim in the hive".TestOpenPRGuard_ListingFailureFailsOpenandTestOpenPRGuard_ListingFailureIsNotRetriedWithinWindow(open_pr_links_test.go:198,:212) pin it. It came with #14812. The repeat every minute isopenPRLinkFailureTTL(open_pr_links.go:51). The contribute lane calls the guard with labels from the snapshot (src/cmd/hive/main.go:1169-1176). There is no fresh issue read first, so the dispatch can go ahead while every GitHub call fails. The ledger it relies on "learns of a hand-opened PR one scan late" (src/pkg/github/issue_claim_gate.go:193-195), and scans also fail while rate-limited. So yes: an issue that gained an open PR since the last good scan can be handed out again, and a second PR can follow. The downstream catches are after the fact: the duplicate-fix sweep (src/pkg/github/pr_duplicate_fix_sweep.go:179) and the merge train's one-PR-per-issue hold (src/pkg/github/automerge/automerge_merge_train_duplicate.go:114). That hold parks the pair for a person, which is the outcome the guard was added to prevent. Failing closed only on rate-limit errors (keep failing open on timeouts) would be a small follow-up.π€ Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1