Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-historical-pr-enrichment-api-budget.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- Historical PR accounting (the merged- and closed-PR enrichment worker that reads each settled PR's merger and reviews) now pauses while the GitHub API budget is in `conserve` or `critical` mode, keeps its queue, and catches up once the budget is back to `normal`; the pause and the resume are each logged once. Before this it kept running a batch every ten seconds whatever the budget said. After a restart the 72-hour settle replay re-queues every recent merged or closed PR, about two REST calls each, and on a hosted hive with 20 repos on 2026-11-07 that replay spent about 400 calls in five minutes while the hourly budget was already under the conserve threshold, helping push it to `critical` and then to an exhausted bucket. Nothing plans, claims or dispatches from this accounting, so delaying it is safe.
2 changes: 2 additions & 0 deletions src/docs/operator-reference/sweeps-and-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -905,6 +905,8 @@ Mode exits use hysteresis: Hive leaves conserve/critical only after quota rises

**After a restart the periodic sweeps start one at a time.** Each sweep checks the budget mode when it starts a pass. After a restart every in-memory cache is empty. If all the sweeps started in the same second, each would list every repo in full, and all of them would decide against the same budget reading. On 2026-11-06 a hosted hive with 20 repos used up its 7000-request hourly limit about five minutes after a restart. Now the merge sweep runs at once. The other sweeps make their first pass at evenly spaced points over the first five minutes, in a fixed order. A sweep with a short interval never waits longer than that interval. So a later sweep sees the budget mode the earlier passes left behind, and it can get free `304` answers for lists an earlier sweep already fetched. After the first pass, each sweep keeps its own interval.

**Historical PR accounting waits for a normal budget.** The worker that reads each recently merged or closed PR's merger and reviews (for the throughput and review counts) runs only in `normal` mode. In `conserve` and `critical` it makes no GitHub calls; its queue is kept and drained once the budget is back to `normal`. The hive logs `historical PR enrichment paused for GitHub API budget` once when it pauses and `historical PR enrichment resumed` once when it starts again. This matters most after a restart: the 72-hour settle replay queues every recent merged or closed PR again, at about two calls each.

- Recovery from a provider spending limit is automatic, via a probe. Withholding kicks also withholds the inference calls that would reveal the provider is serving again, so the hive suppresses only while the last refusal is recent and then lets a single kick through to test the gateway; the probe re-arms suppression the moment it is released, so at most one probe run flies per interval. A still-clipped key refuses the probe and suppression resumes for another interval; once the provider's window resets the probe succeeds, normal kicking resumes with no operator action, and a one-time recovery notification is sent (the entering notification is likewise sent once per clip, not once per cycle). Tune with `governor.provider_budget.probe_interval_s` (default 1800 — 30 minutes):

```yaml
Expand Down
34 changes: 34 additions & 0 deletions src/pkg/github/pr_terminal_enrichment.go
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ type terminalEnrichmentState struct {
pending map[string]terminalEnrichmentTask
sequence uint64
retries retryTracker
// budgetPaused records that the last batch was skipped for the GitHub
// API budget, so the pause and the resume are each logged once.
budgetPaused bool
}

// StartPRTerminalEnrichment isolates historical accounting from current-work
Expand Down Expand Up @@ -165,6 +168,9 @@ func (c *Client) enrichTerminalPR(ctx context.Context, task terminalEnrichmentTa
}

func (c *Client) enrichTerminalPRBatch(ctx context.Context, budget time.Duration, limit int) {
if !c.terminalEnrichmentBudgetAllows() {
return
}
started := time.Now()
ctx, cancel := context.WithTimeout(ctx, budget)
defer cancel()
Expand Down Expand Up @@ -213,3 +219,31 @@ func (c *Client) enrichTerminalPRBatch(ctx context.Context, budget time.Duration
"duration", time.Since(started), "budget", budget, "limit", limit)
}
}

// terminalEnrichmentBudgetAllows reports whether historical PR accounting may
// spend GitHub calls now. It is the most deferrable REST work the hive does:
// nothing plans, claims or dispatches from it. So, like the eval loop's
// enrichment steps, it runs only in normal API budget mode. In conserve and
// critical the queue is kept, coalesced as usual, and drained once the budget
// is back to normal. After a restart the 72-hour settle replay re-queues
// every recent merged or closed PR (about two calls each), and on the
// projectbluefin hive on 2026-11-07 that replay spent ~400 calls while the
// budget was already below the conserve threshold.
func (c *Client) terminalEnrichmentBudgetAllows() bool {
mode, snap := c.APIBudgetMode()
s := &c.prTerminalEnrichment
s.mu.Lock()
wasPaused := s.budgetPaused
s.budgetPaused = mode != APIBudgetNormal
pending := len(s.pending)
s.mu.Unlock()
if c.logger != nil && wasPaused != (mode != APIBudgetNormal) {
if wasPaused {
c.logger.Info("historical PR enrichment resumed: GitHub API budget back to normal", "pending", pending)
} else {
c.logger.Info("historical PR enrichment paused for GitHub API budget",
"mode", mode.String(), "remaining", snap.Remaining, "pending", pending)
}
}
return mode == APIBudgetNormal
}
51 changes: 51 additions & 0 deletions src/pkg/github/pr_terminal_enrichment_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -296,3 +296,54 @@ func TestTerminalEnrichmentRestartReplaysUnprocessedRecentHistory(t *testing.T)
t.Fatalf("restart did not replay the real historical timestamp: %+v", observed)
}
}

// Historical accounting is deferrable work. On the projectbluefin hive
// (2026-11-07) a restart replayed ~200 settled PRs at eight per ten seconds,
// about 400 REST calls, while the core budget was already in conserve and
// then critical. The worker must wait for normal mode, keep its queue, and
// make no GitHub request until then.
func TestTerminalEnrichmentPausesOutsideNormalAPIBudget(t *testing.T) {
resetRESTAccountingForTest(t, time.Now())
var requests atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests.Add(1)
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte("[]"))
}))
defer srv.Close()
c := NewClientForTest(srv.URL, "torch-spyre", []string{"spyre-inference"}, testLogger())
c.SetAPIBudgetThresholds(800, 250)
setCore := func(remaining int) {
c.rateLimits.mu.Lock()
c.rateLimits.windows = map[string]rateLimitWindow{githubCoreRateLimitResource: {
limit: 7000, remaining: remaining, reset: time.Now().Add(30 * time.Minute), observedAt: time.Now(),
}}
c.rateLimits.mu.Unlock()
}
deferTerminalTasks(c)
var reviewed, observed atomic.Int32
c.SetPRReviewObservedHook(func(PRReviewObservation) { reviewed.Add(1) })
c.SetPRTerminalObservedHook(func(PRTerminalObservation) { observed.Add(1) })
at := time.Now()
c.observeTerminalPR(context.Background(), historicalTask(1, at))
c.observeTerminalPR(context.Background(), historicalTask(2, at))

for _, remaining := range []int{675, 214} { // conserve, then critical
setCore(remaining)
c.enrichTerminalPRBatch(context.Background(), terminalEnrichmentTestTimeout, terminalEnrichmentBatch)
if got := requests.Load(); got != 0 {
t.Fatalf("remaining=%d: historical enrichment made %d GitHub requests outside normal budget mode", remaining, got)
}
if observed.Load() != 0 || len(c.prTerminalEnrichment.pending) != 2 {
t.Fatalf("remaining=%d: paused enrichment consumed accounting: observed=%d pending=%d",
remaining, observed.Load(), len(c.prTerminalEnrichment.pending))
}
}

setCore(6000)
c.enrichTerminalPRBatch(context.Background(), terminalEnrichmentTestTimeout, terminalEnrichmentBatch)
if observed.Load() != 2 || len(c.prTerminalEnrichment.pending) != 0 || requests.Load() == 0 {
t.Fatalf("enrichment did not resume in normal mode: observed=%d pending=%d requests=%d",
observed.Load(), len(c.prTerminalEnrichment.pending), requests.Load())
}
}
Loading