Skip to content

🐛 fix(github): historical PR accounting waits while the GitHub API budget is low - #17075

Merged
hivecommons-hive[bot] merged 1 commit into
hivecommons:v6from
Danathar:fix/gh-api-budget-1107
Nov 7, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
hivecommons:v6from
Danathar:fix/gh-api-budget-1107

Conversation

@Danathar

@Danathar Danathar commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Summary

The historical PR accounting worker reads the merger and the reviews of each recently merged or closed PR, for the throughput and review counts. With this change it waits while the GitHub API budget is in conserve or critical. It keeps its queue and catches up once the budget is back to normal.

Before this, it ran a batch every ten seconds whatever the budget said. After a restart it replays every PR merged or closed in the last 72 hours, at about two REST calls each. On the projectbluefin hive today, that replay spent about 400 calls in five minutes. Most of them came after the budget had already dropped below the conserve threshold.

What went wrong

The projectbluefin hive has 20 repos and one App installation, with a limit of 7000 calls an hour. It was upgraded to 7b9eeb8 at 18:21Z on 2026-11-07. Its log and its own REST accounting (/api/status ghRateLimits.top_consumers) show this:

  • From 18:24:53Z to 18:29:19Z the log has 30 historical PR enrichment batch lines. Each covers 8 PRs, so about 230 PRs in all.
  • The REST accounting for that process lists 216 charged calls for hive:pr_terminal_reviews, with none answered from the ETag cache. It lists 195 charged calls for hive:pr_terminal_merged_by, with one answered from the cache. That makes 411 calls. Both numbers stayed the same for the rest of the hour, so this is a one-off cost of the restart.
  • The log line github api budget mode changed shows the budget going to conserve at 18:26:19Z (remaining 793) and to critical at 18:28:46Z (remaining 214). The worker kept going until 18:29:19Z. About 20 of its 30 batches ran after the budget was already in conserve, which is about 280 calls.
  • From about 18:38Z until the reset at 18:54:37Z, every REST call the hive made failed with 403 API rate limit of 7000 still exceeded, and all 20 repos failed to enumerate.

The eval loop already skips its optional enrichment steps in conserve and critical (APIBudgetEnrichmentSteps). This worker was added later, in #16289. It runs on its own ticker and never checked the budget. It feeds only accounting: nothing plans, claims or dispatches from it, so making it wait is safe.

This PR does not fix the hive's steady-state use, which was also over budget before the restart. That is a separate and larger problem.

What changes

  • enrichTerminalPRBatch asks APIBudgetMode() first. Outside normal it returns without making any GitHub call. Pending tasks stay queued and keep coalescing by PR, as before.
  • The worker logs historical PR enrichment paused for GitHub API budget once when it pauses, with the mode, the remaining count and the queue size. It logs historical PR enrichment resumed once when it starts again. It does not log every ten seconds.
  • The API budget section of the operator reference now says this.

Files

  • src/pkg/github/pr_terminal_enrichment.go: the budget check and the log lines.
  • src/pkg/github/pr_terminal_enrichment_test.go: the new test.
  • src/docs/operator-reference/sweeps-and-operations.md: one paragraph in "GitHub API budget governor".
  • changelog.d/fixed-historical-pr-enrichment-api-budget.md

Tests

  • The new test TestTerminalEnrichmentPausesOutsideNormalAPIBudget queues two historical PRs. It runs a batch at remaining 675 (conserve) and again at 214 (critical), and checks that no request reaches the test server and that both tasks are still queued. Then it runs a batch at 6000 (normal) and checks that both tasks drain.
    • On v6 tip it fails with remaining=675: historical enrichment made 2 GitHub requests outside normal budget mode.
    • With this change it passes.
  • All the existing TestTerminalEnrichment* tests pass, and so does the full ./pkg/github/ package. go vet ./pkg/github/ is clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

The terminal-PR enrichment worker (merged-by and review reads for the
throughput and review counts) ran a batch every ten seconds regardless of
the GitHub API budget. After a restart the 72-hour settle replay queues
every recent merged or closed PR at about two REST calls each; on the
projectbluefin hive on 2026-11-07 that was 411 calls in five minutes, about
280 of them after the budget had entered conserve.

The worker now checks APIBudgetMode before each batch and makes no GitHub
call outside normal mode. The queue is kept and drained once the budget
recovers. The pause and the resume are each logged once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 7, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 7, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 2161745 into hivecommons:v6 Nov 7, 2026
72 of 75 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#17075b436e08` sha256:deaeda5d6395fd4259b70053b4f0ac38a62801df3f92dbebeeb5e49e3f63768a — download from the dashboard or `hivectl review evidence #17075`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at b436e08.

hivecommons-hive Bot added a commit that referenced this pull request Nov 7, 2026
… sweeps name their REST calls (#17101)

Several sweeps built their list query's since from time.Now() on every
pass (un-park comment reads, closed-issue stage sweep and review backlog
split, shared-CI rerun, same-component sibling check). The URL changed on
every call, so the ETag cache, keyed on the full URL, never matched and
every pass was charged. The since sent to GitHub is now rounded down to
the UTC hour (cacheableSince); callers keep filtering on their exact
cutoff, so the wider window misses nothing and answers nothing twice.

Untagged REST calls from the sweep workers, the governor cycle, the
forgestate loop, the metrics collector, the request watchers, the mention
poller and dashboard requests now carry WithRESTCaller names, so
/api/gh-rate-limits no longer lumps ~4,400 calls an hour under "hive".

Refs #17075


Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Danathar <Danathar@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant