Skip to content

πŸ› fix(dashboard): Let hive decide preview names GitHub rate limits, not hive unavailable - #16951

Merged
hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/delegate-plan-github-rate-limit
Nov 6, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/delegate-plan-github-rate-limit

Conversation

@Danathar

@Danathar Danathar commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Summary

While the hive's GitHub App is out of quota, every Let hive decide preview failed with "(hive unavailable)", even though the hive was up. The preview answered any GitHub read failure with 502, and the hosted Ingress swaps every 502 and 503 for its own "hive is restarting" body. The preview now answers 429 with a Retry-After header and the time the limit lifts, and other GitHub refusals answer 424 with GitHub's own words. The dialog shows that message. 502 is left for a GitHub that could not be reached at all.

What went wrong

On 2026-11-06 around 17:00-17:15Z, the projectbluefin hive showed "Could not preview projectbluefin/testsuite#1026 (hive unavailable)" on every item. Its logs at the same time said 403 API rate limit of 7000 still exceeded until 2026-11-06 12:32:06 -0500 EST.

The chain:

  1. handleRepoItemDecideDelegatedPlan reads the item live from GitHub and turned any error into 502 Bad Gateway.
  2. The hosted Ingress sets custom-http-errors: "502,503" (src/pkg/hub/saas_provision.go), so ingress-nginx replaced that 502 with the canned {"error":"hive unavailable", …} from src/deploy/k8s/error-pages.yaml.
  3. The dialog printed data.error, which was now "hive unavailable".

What changes

  • The plan handler maps the read error (wrapped errors included, via errors.As):

    GitHub said Answer Message
    Primary rate limit (*github.RateLimitError) 429 GitHub API rate limit for this hive is exhausted; resets at 17:32Z, Retry-After to the reset
    Secondary limit (*github.AbuseRateLimitError) or a bare 429 429 GitHub secondary rate limit reached for this hive; retry after 17:12Z
    The hive's own pacer shed the request (ErrPacingShed) 429 This hive is pacing its GitHub requests after a rate limit; retry after …
    Any other GitHub answer (404, permission 403, …), closed item, no snapshot 424 The real error text
    No answer at all (network error, timeout) 502 The real error text (the Ingress still rewrites it)

    With no time from GitHub, the wait is one minute (GitHub's documented minimum).

  • pkg/github gains RateLimitRetryDelay. It reuses the existing isRateLimitErr classifier and the retryableGitHubDelay reader of reset time and Retry-After.

  • Dashboard: the preview error shows the server's message. A 429 whose message has no time gets "; retry after HH:MMZ" from Retry-After. The Ingress's HTML 502 page (not JSON) now reads "hive unavailable" instead of "HTTP 502".

  • Docs: the API reference row for …/decide-delegated/plan and the Let hive decide steps in the activity-monitor page.

The …/decide-delegated POST and the job status GET do not have this bug. The POST answers 202 and the job carries the error text; the GET reads the job from memory. Neither was changed.

Files

File Change
src/pkg/dashboard/delegated_gather_error.go New: the error-to-status mapping
src/pkg/dashboard/api_repo_item_decide_delegated_plan.go One line: use the mapping instead of 502
src/pkg/github/github_retryable.go New RateLimitRetryDelay
src/pkg/dashboard/static-src/js/tracker-actions.js New stageTrackerDelegatePlanErrorText, used by the plan POST
src/pkg/dashboard/static/index.html Regenerated
src/pkg/dashboard/stage_tracker_row_actions_test.go The new JS helper joins the extraction list
src/docs/api-reference/packs-and-acmm.md, src/docs/dashboard-sections/activity-monitor.md Error answers documented
changelog.d/fixed-delegate-plan-github-rate-limit.md Changelog

Tests

Test What it proves Without the change
TestDelegatedDecisionPlanRateLimitedAnswers429WithReset Real handler, real GitHub client, fake GitHub answering a primary and a secondary rate-limit 403: 429, Retry-After, reset time in the message Fails: code=502 … the Ingress rewrites this to "hive unavailable" (both cases)
TestDelegatedGatherErrorNeverAnswers502ForGitHubRefusals Every class above: status, message, Retry-After; no rate limit or GitHub refusal answers 502/503 Does not build (mapping absent)
TestStageTrackerDelegatePlanRateLimitMessage (node) The dialog shows the server's 429 message, Retry-After fallback, the Ingress's HTML 502 Fails: index.html does not define stageTrackerDelegatePlanErrorText()
TestRateLimitRetryDelay (pkg/github) Primary, secondary, bare 429 are rate limits with GitHub's wait; 404 and permission 403 are not Does not build (helper absent)

The rate-limit test cleans up the shared pacer the way apphealth does. It sends X-RateLimit-Limit: 60 so the shared core-bucket snapshot is not touched.

Also run: the delegated-decision, decision, stage-tracker and settings-leaf tests in pkg/dashboard, the rate-limit tests in pkg/github, go vet on both, the docs citation, link and API reference checks, and the changelog check.

Found, not changed

Other 502/503 answers (same class as #4241). The write handlers have the same pattern, but changing them is not mechanical. They run under recordOperatorAction, and operatorActionOutcome (src/pkg/dashboard/operator_actions.go:468-473) records a 5xx as failed but does not record a 4xx at all. Remapping them would drop failed actions from the row's history. So they are left for a separate change:

File 502 sites Notes
api_repo_item_decision.go 15 Decision POST: GitHub reads and writes (comment, close, merge, labels)
api_repo_item_accept_triage.go 7
api_repo_item_decision_reroute.go 6 Reroute and adopt-now
item_blocker.go, api_repo_hold.go, api_review_queue_superseded.go, copilot_auth.go 4 each
plan_handlers.go, api_review_pipeline.go, api_agents.go 3 each
17 more files 1-2 each

Both delegated handlers also answer "GitHub client not configured" with 503, which the Ingress hides the same way. That is a configuration state, not a GitHub failure, so it is not changed here.

High-priority pacing and last-good fallback. A pacing class exists (src/pkg/github/pacing_priority.go:39-47: PacingHigh, PacingNormal, PacingLow). The preview runs as PacingNormal. But a class only orders the hive's own queue while the slow-start caution window is engaged. It cannot help with primary exhaustion: go-github refuses every core call before the request leaves the process. A last-good snapshot exists per repo for the enumeration (src/pkg/github/enumerate_lastgood.go:36, 2 h max age; cache type at :54), and the dashboard keeps the stage-tracker snapshot (s.status). Both live only in memory, so a restart empties them, as in this incident. A fallback preview would also lack most of what the rules read (CI per head, reviews, files, author permission, contributor activity). A follow-up would need: (1) a per-item cache of the last gathered delegatedecision.Input, kept longer than the 45 s plan cache; (2) on a rate-limit error, a plan from that input marked stale with its age; (3) Let hive do that disabled while stale, since the decision itself needs live reads and writes.

What spent the quota. Nothing spreads out the GitHub work after a restart. These figures are read from the code, not measured. runLoopWith waits 10 s (src/cmd/hive/run_loop_deps.go:12). It then starts every sweep worker and runs the first full enumeration in the same moment (src/cmd/hive/main.go:6464-6476). Each worker fires on its first tick because last.IsZero() (src/cmd/hive/sweep_workers.go:59), so about 19 sweeps start together with no jitter. The merged-PR backfill also runs on every boot (src/cmd/hive/main.go:2790). All the caches that would make this cheap are in memory and start cold, so there are no free 304s after a restart: the ETag cache (src/pkg/github/etagcache.go, no disk writes), the PR-detail cache, the claim cache and the last-good enumeration. The biggest spenders, by rough estimate: the first enumeration with CI enrichment (a PR GET, check runs and sometimes comments or workflow runs for each open PR, plus one comment listing per actionable issue when claims are on); the red-main sweep; the supersession sweep (one Issues.Get per claimed issue); and the automerge sweep's first pass. For 20 repos that is likely 1.5k-3k core requests in the first few minutes, paid again on each restart. Two restarts 2.5 minutes apart, on a quota already spent once that day, fit the log. With no last-good copy, a fully refused pass returns the all N repos failed error (src/pkg/github/client.go:1924-1926). The budget modes (src/pkg/github/apibudget.go: conserve below 800 remaining, critical below 250) start as normal at boot, because nothing has been observed yet. Per-caller REST accounting exists (/api/gh-rate-limits, src/pkg/github/restaccounting.go), but it is also in memory, so each restart erased the record of what spent the budget. A follow-up could stagger first ticks with jitter after boot, or persist the ETag cache.

Open-PR guard fails open. OpenPRLinkGuard (src/pkg/github/open_pr_links.go:249-286) logs open-PR guard: could not list open PRs, admitting the claim on ledger evidence alone and admits the claim when the listing fails. This is intended. The doc comment says "a failure fails OPEN with a warning: the ledger-backed guards downstream still stand, and a flaky listing must not stall every claim in the hive". TestOpenPRGuard_ListingFailureFailsOpen and TestOpenPRGuard_ListingFailureIsNotRetriedWithinWindow (open_pr_links_test.go:198, :212) pin it. It came with #14812. The repeat every minute is openPRLinkFailureTTL (open_pr_links.go:51). The contribute lane calls the guard with labels from the snapshot (src/cmd/hive/main.go:1169-1176). There is no fresh issue read first, so the dispatch can go ahead while every GitHub call fails. The ledger it relies on "learns of a hand-opened PR one scan late" (src/pkg/github/issue_claim_gate.go:193-195), and scans also fail while rate-limited. So yes: an issue that gained an open PR since the last good scan can be handed out again, and a second PR can follow. The downstream catches are after the fact: the duplicate-fix sweep (src/pkg/github/pr_duplicate_fix_sweep.go:179) and the merge train's one-PR-per-issue hold (src/pkg/github/automerge/automerge_merge_train_duplicate.go:114). That hold parks the pair for a person, which is the outcome the guard was added to prevent. Failing closed only on rate-limit errors (keep failing open on timeouts) would be a small follow-up.

πŸ€– Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…not "hive unavailable"

The preview gathers the item live from GitHub and answered any failure
with 502. On a hosted hive the Ingress swaps every 502 and 503 for its
canned "hive unavailable" body, so while the App was out of quota every
preview read "Could not preview <item> (hive unavailable)" with the hive
up.

The plan now maps the gather error: a primary or secondary rate limit
(or a request the pacer shed) answers 429 with Retry-After and the time
it lifts; any other GitHub answer (404, permission 403), a closed item or
a missing snapshot answers 424 with the real message; only a request that
never got an answer stays 502. pkg/github gains RateLimitRetryDelay,
built on the existing rate-limit classifier and retry-delay reader. The
dashboard shows the server's message and, for a 429 without a time,
"retry after HH:MMZ" from Retry-After.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 6, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16949 πŸ› fix(github): delegated issue decisions sanitise reporter-authored option text same files src/pkg/dashboard/api_repo_item_decide_delegated_plan.go
#16688 🌱 Forward-merge v5 into v6 (2 commits) same files src/pkg/dashboard/static/index.html

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v6 tree and the retry helpers it reuses. Looks correct to me; go build ./pkg/github/ ./pkg/dashboard/ is clean on this head.

  • errors.As reaches the go-github types because every gather failure is %w-wrapped (api_repo_item_decide_delegated.go:658,676,703,707), and go-github already turns the "403 API rate limit … exceeded" answer in your log into *gh.RateLimitError, so that case hits the 429 branch at delegated_gather_error.go:44-46.
  • RateLimitRetryDelay (github_retryable.go:24-31) passes fallback=0, fallbackMax=0, explicitMax=1h into retryableGitHubDelay; when GitHub names no reset the clamp at :98-100 yields 0, which the caller replaces with the one-minute floor β€” so a bare 429 never emits Retry-After: 0.
  • Status split reads right: a permission 403 or 404 is *gh.ErrorResponse without the rate-limit shape β†’ 424 with GitHub's words; net.Error / deadline β†’ 502, which is the one case the Ingress rewrite is still appropriate for.
  • The JS only appends its own "retry after" when the server message lacks an HH:MMZ (tracker-actions.js:1785), so the Go-side times are never doubled.

One note, not a defect: an explicit reset further than an hour out is clamped to 1h by explicitMax, so "resets at" could read early in that case. Primary-limit windows are ≀1h, so it should not occur in practice.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the mapping against the tree rather than the diff alone; this looks correct to me.

  • src/pkg/dashboard/delegated_gather_error.go:38 calls RateLimitRetryDelay, which in turn calls retryableGitHubDelay(err, now, 0, 0, time.Hour) (src/pkg/github/github_retryable.go:27). With fallback=0, fallbackMax=0, the clamp at github_retryable.go:99-100 yields 0 whenever GitHub named neither a reset nor Retry-After, so the one-minute fallback at delegated_gather_error.go:40 is actually reached; with an explicit reset it is preserved up to 1 h, which covers the primary window.
  • *gh.RateLimitError / *gh.AbuseRateLimitError are matched before the *gh.ErrorResponse branch, and a bare 429 is caught by the StatusCode != 429 guard, so no rate-limit shape falls through to 424.
  • net.Error / context.DeadlineExceeded keep 502 (delegated_gather_error.go:62); the hosted Ingress only rewrites 502,503 (src/pkg/hub/saas_provision.go:3706), so 429 and 424 reach the dialog intact.
  • The JS path: r.json().catch(() => ({})) on the Ingress HTML page gives msg='', and stageTrackerDelegatePlanErrorText then returns hive unavailable for 502/503 β€” matches the documented wording.

One observation, not a defect: context.Canceled (client went away mid-gather) now answers 424 instead of 502. The client is gone by then so nothing reads it.

CI is green on 12dd36f. A maintainer can take this as-is.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 729baa9 into hivecommons:v6 Nov 6, 2026
70 of 71 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#169510490c30` sha256:fe9a44fa053798d0adbb7eb93a6f4afcc90bc1c577fd8035293f2f8428b15018 β€” download from the dashboard or `hivectl review evidence #16951`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 0490c30.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant