Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-secondary-403-short-caution.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- One GitHub secondary-limit 403 no longer stalls merges for over an hour. A secondary 403 with no `Retry-After` used to assume a full hour, then add the 10-minute slow-start window. For those 70 minutes every low-priority call that would queue more than 5s was shed, including the automerge sweep's commit-status read. On the projectbluefin hive (2026-10-28) one search refusal after a restart meant no PR merged for over an hour, even an approved PR with 19/19 checks green, while the primary quota stood at 87%. Such a 403 now backs off for `Retry-After` when GitHub sends it, or until `X-RateLimit-Reset` when `X-RateLimit-Remaining` is `0`. Otherwise it waits one minute, GitHub's documented rule, and doubles on each repeat up to 15 minutes. Separately, once an automerge lane reaches a candidate PR, the PR's evaluation and merge run at normal pacing priority, so its status and check-run reads are never shed as housekeeping.
2 changes: 1 addition & 1 deletion src/docs/design/github-mention-triggers.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ comment (or issue) node id, so a mention seen twice is handled once.

A mention is a comment or issue body containing `@<app-login>` where
`<app-login>` is the installation's bot login, the value the hive already knows
as `appBotLogin` (`isHiveAppReviewAuthor`, `src/pkg/github/automerge/automerge_sweep.go:1221`).
as `appBotLogin` (`isHiveAppReviewAuthor`, `src/pkg/github/automerge/automerge_sweep.go:1242`).
The text after the mention is the request. Two optional forms route it:

```
Expand Down
6 changes: 3 additions & 3 deletions src/docs/labels-and-control-signals.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ Most hard gates run before an agent sees work. GitHub issue enumeration reads la

Some sweeps bypass that enumeration and list items themselves, but every one of them gates on the same hold predicate as enumeration (`Client.IsHeldLabels` / `Client.isHeld`, `src/pkg/github/client.go:1969-1977`, `src/pkg/github/client.go:2679-2681`), so the generic hold substrings and the exact dashboard `hive-pause/<hive-id>` label hold everywhere:

- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; both the queued (`lgtm`) and the self-authored sweep skip a held PR before fetching it (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:959-964`, `src/pkg/github/automerge/automerge_sweep.go:966-1004`, `src/pkg/github/automerge/automerge_sweep.go:1029-1046`).
- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; both the queued (`lgtm`) and the self-authored sweep skip a held PR before fetching it (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`).
- The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:797-815`, `src/cmd/hive/claimescalation.go`).
- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; the queued (`lgtm`) sweep skips a held PR before fetching it, and the self-authored sweep skips held PRs plus `needs-human` before merging (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:959-964`, `src/pkg/github/automerge/automerge_sweep.go:966-1004`, `src/pkg/github/automerge/automerge_sweep.go:1029-1046`).
- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; the queued (`lgtm`) sweep skips a held PR before fetching it, and the self-authored sweep skips held PRs plus `needs-human` before merging (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`).
- The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:848-854`).
- The SHA-hold sweep lists primary-repo `kind/bug` issues itself, adds literal `hold` plus a marker notice when no SHA is present (for reporters in `issue_filter.reporter_trust` trusted associations it adds the soft `issues.sha_hold_trusted_label`, default `needs-sha`, instead, #12002), and removes `hold` only when the current hold is its own: the newest `hold` label event is a `labeled` by the App bot and an App-authored SHA-hold notice was posted at or after it (`EnforceSHAHold` / `shaHoldIsOwn` in `src/pkg/github/client.go`).
- PR-request watching applies server-side PR holds after a PR is opened; the agent's `--label hold` flag is deliberately discarded by the wrapper (`src/pkg/github/pr_request_watcher.go:452-524`, `bin/hive-open-pr.sh:142-150`).
Expand Down Expand Up @@ -101,7 +101,7 @@ Some hive-authored labels look like provenance but are read as signals, so they
| `tracker`, `meta-tracker`, `tracking`, `epic` (last segment) | Issues | Tracker detector, PR claim rewrite, contributor queue | Marks coordination-only/tracker issues; agents see tracker tags, contributors do not get them. | Soft for agents; hard for contributors | Human/Hive | Enumeration/contributor admission | Remove label or close tracker | Tracker detector (`src/pkg/github/client.go:2095-2161`) |
| `epic` (any case; `governor.claims.never_claim_labels`) | Issues | Issue-claim admission, issue stage derivation | Umbrella issue is never a unit of agent work: every automated claim producer (relay `claim`, start signal, contributor lane) is refused with `never-claim label: <label>`, and a claim or start mark that exists on one never derives `stage/020`/`stage/030` — the epic stays `stage/010-actionable` ([#12999](https://github.com/hivecommons/hive/issues/12999)). | Hard for agent claims | Human | Claim admission, stage derivation | Remove label or set `never_claim_labels: []` | `src/pkg/github/issue_claim_gate.go`, `src/pkg/github/issue_stage_labels.go`, `src/pkg/claims/neverclaim.go` |
| `hive-plan`, `hive-design` | Issues | Planning label sweep | `hive-plan` mints/decomposes an epic; `hive-design` asks architect for design first. Only works when planning-from-label is enabled and ACMM >= 5. | Gate when enabled | Human | Planning sweep | Remove label or process plan/design | `planning.plan_from_label`, `planning.plan_labels`, `planning.design_labels` (`src/pkg/config/planning_config.go:35-56`, `src/pkg/planning/issue.go:483-510`) |
| `lgtm` | PRs | Queued auto-merge sweep | Queue label for human/owner merge action; the sweep also requires its normal authorization and skips held/exempt PRs. Adding by hand is not enough if the App approval/authorization is absent. | Permit | Dashboard/merger | Auto-merge sweep | Remove label; head changes can de-queue | `governor.labels.automerge`, default `lgtm` (`src/pkg/config/governor_config.go:175-184`, `src/pkg/config/repo_policy.go:11-16`, `src/pkg/github/automerge/automerge_sweep.go:966-1036`) |
| `lgtm` | PRs | Queued auto-merge sweep | Queue label for human/owner merge action; the sweep also requires its normal authorization and skips held/exempt PRs. Adding by hand is not enough if the App approval/authorization is absent. | Permit | Dashboard/merger | Auto-merge sweep | Remove label; head changes can de-queue | `governor.labels.automerge`, default `lgtm` (`src/pkg/config/governor_config.go:175-184`, `src/pkg/config/repo_policy.go:11-16`, `src/pkg/github/automerge/automerge_sweep.go:987-1057`) |
| `reviewer-passed` | PRs | Reviewer lane/escalation reconciliation | Marks one reviewer pass/de-escalation; with no `needs-human`, un-escalates. | Workflow | Reviewer lane/Hive | Reviewer/escalation pass | Remove only if intentionally re-reviewing | Fixed label (`src/pkg/escalation/escalation.go:1185-1205`) |
| `reviewer-recommend-close` | PRs | Reviewer lane | Reviewer recommends closing rather than continuing automated repair. | Workflow | Reviewer lane | Reviewer pass | Human decides/clears | Fixed label (`src/pkg/scheduler/reviewer_lane.go:296-305`) |
| `reviewer-handoff` | PRs | Reviewer lane | Reviewer handed the PR to a maintainer: the change is fine but only a maintainer can unblock it; the verdict comment's first line is the maintainer's next step. `needs-human` stays; the dashboard reason and the verdict comment both tell the maintainer to remove this label together with `needs-human`. | Workflow | Reviewer lane | Reviewer pass | Human does the step, then a trusted merger's `/hive approve` on a hive-authored PR removes it, lifts a Hive-added `needs-human` and adds `reviewer-passed` ([#13309](https://github.com/hivecommons/hive/issues/13309)); otherwise a person removes it and `needs-human` (left on, a later re-escalation repeats the hand-off reason and the reviewer lane skips the PR) | Fixed label (`src/pkg/scheduler/reviewer_lane.go:59-66`) |
Expand Down
18 changes: 16 additions & 2 deletions src/docs/operator-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1717,8 +1717,16 @@ only the two list calls per repo move behind it.

After a GitHub rate-limit refusal (a secondary-limit 403 or a primary-limit
reset) the shared transport enters a slow-start caution window and paces
requests about 2s apart. Pacing is prioritised so periodic housekeeping cannot
starve the governor:
requests about 2s apart. The window lasts the limit's back-off plus 10
minutes. A primary refusal backs off until its `X-RateLimit-Reset`. A
secondary 403 backs off for its `Retry-After` (seconds or an HTTP date), or
until `X-RateLimit-Reset` when `X-RateLimit-Remaining` is `0`. Otherwise it
follows GitHub's rule for secondary limits: wait one minute, doubling on each
repeat within 30 minutes of the last back-off (1m, 2m, 4m, 8m, capped at
15m). The rest of the burst refused by the same block does not count as a
repeat. Before this, a header-less secondary 403 assumed a full hour, so one
search refusal kept the hive pacing for 70 minutes. Pacing is prioritised so
periodic housekeeping cannot starve the governor:

- the reconcile pass (`EnumerateActionable`) runs **high** priority and is
bounded only by its own deadline;
Expand All @@ -1732,6 +1740,12 @@ starve the governor:
most once a minute per caller) and retried on its next tick;
- the self-authored automerge sweep backs its tick off to at least 60s while
the window is engaged and resumes its adaptive interval when it ends;
- once an automerge lane (queued, self-authored, trusted-author, contributor)
reaches a candidate PR, that PR's evaluation and merge (the PR read, its
status and check-run reads, the merge call) run at **normal** priority, so
a green, approved PR is never skipped with `status-check` because its
status read was shed. The lanes evaluate one request at a time, so this adds
at most one slot to the queue;
- GitHub App installation-token mints (and the other App-JWT endpoints) do
not go through the pacer at all: they are a separate rate-limit bucket, and
a mint that queued behind paced traffic failed every request that needed
Expand Down
2 changes: 1 addition & 1 deletion src/pkg/github/automerge/automerge_contributor_lane.go
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ func (c *Engine) SweepContributorLaneAutoMerges(ctx context.Context, opts AutoMe
continue
}
result.Candidates++
event, reason, err := c.trySweepContributorLanePR(ctx, displayRepo, owner, repoName, number, policy)
event, reason, err := c.trySweepContributorLanePR(mergeDecisionPriority(ctx), displayRepo, owner, repoName, number, policy)
if err != nil {
c.warn("contributor-lane automerge sweep skipped PR", "repo", displayRepo, "pr", number, "reason", reason, "error", err)
result.Skipped++
Expand Down
27 changes: 24 additions & 3 deletions src/pkg/github/automerge/automerge_sweep.go
Original file line number Diff line number Diff line change
Expand Up @@ -501,6 +501,27 @@ func lowPriority(ctx context.Context) context.Context {
return hgithub.WithPacingClass(ctx, hgithub.PacingLow)
}

// mergeDecisionPriority lifts one candidate PR's evaluation and merge out of
// the low pacing class the lane loop runs at. The lanes stay housekeeping
// where it matters: the tick or floor pass still yields to a deep queue
// (ShedLowPriorityTick) and backs off to its caution floor, and the
// pre-candidate reads stay low. But once a PR passed the cheap prefilter,
// shedding its status / check-run read is not load shedding, it is a missed
// merge: on the projectbluefin hive (2026-10-28) an approved, 19/19-green
// finpilot#599 was skipped with reason "status-check" ("request shed … class
// low, queue 5.454s ahead, limit 5s") pass after pass for the whole caution
// window. The lanes evaluate candidates one request at a time, so normal
// class adds at most one slot to the queue and waits at most
// slowStartNormalMaxQueue for it — it cannot rebuild the 2026-10-21 pile-up
// the low class was introduced for (hivecommons/hive#13382). A ctx the caller
// already classed (high, or explicitly normal) is left alone.
func mergeDecisionPriority(ctx context.Context) context.Context {
if hgithub.PacingClassOf(ctx) != hgithub.PacingLow {
return ctx
}
return hgithub.WithPacingClass(ctx, hgithub.PacingNormal)
}

// selfAuthoredSweepBudgetShare caps the fraction of the App's hourly REST
// budget this sweep alone may consume. The sweep is a background nicety; the
// agents' own gh calls, the governor's eval cycle, the merge-eligible writer
Expand Down Expand Up @@ -949,7 +970,7 @@ func (c *Engine) SweepQueuedAutoMerges(ctx context.Context, opts AutoMergeSweepO
result.Skipped++
continue
}
event, reason, err := c.trySweepQueuedPR(ctx, repo, owner, repoName, issue.GetNumber(), label)
event, reason, err := c.trySweepQueuedPR(mergeDecisionPriority(ctx), repo, owner, repoName, issue.GetNumber(), label)
if err != nil {
c.warn("automerge sweep skipped PR", "repo", repo, "pr", issue.GetNumber(), "reason", reason, "error", err)
result.Skipped++
Expand Down Expand Up @@ -1146,7 +1167,7 @@ func (c *Engine) SweepSelfAuthoredAutoMerges(ctx context.Context, opts AutoMerge
}
result.Candidates++
repoCandidates++
event, reason, err := c.trySweepSelfAuthoredPR(ctx, repo, owner, repoName, number, branchUpdateAttempts < selfAuthoredSweepMaxBranchUpdates)
event, reason, err := c.trySweepSelfAuthoredPR(mergeDecisionPriority(ctx), repo, owner, repoName, number, branchUpdateAttempts < selfAuthoredSweepMaxBranchUpdates)
if event.BranchUpdated || reason == "updated-branch" || reason == "update-branch" {
branchUpdateAttempts++
}
Expand Down Expand Up @@ -1291,7 +1312,7 @@ func (c *Engine) SweepTrustedAuthorAutoMerges(ctx context.Context, opts AutoMerg
continue
}
result.Candidates++
event, reason, err := c.trySweepTrustedAuthorPR(ctx, displayRepo, owner, repoName, number, policy)
event, reason, err := c.trySweepTrustedAuthorPR(mergeDecisionPriority(ctx), displayRepo, owner, repoName, number, policy)
if err != nil {
c.warn("trusted-author automerge sweep skipped PR", "repo", displayRepo, "pr", number, "reason", reason, "error", err)
result.Skipped++
Expand Down
37 changes: 37 additions & 0 deletions src/pkg/github/automerge/sweep_caution_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,13 @@ package automerge

import (
"bytes"
"context"
"log/slog"
"strings"
"testing"
"time"

hgithub "github.com/hivecommons/hive/pkg/github"
)

func TestCautionSweepInterval(t *testing.T) {
Expand Down Expand Up @@ -64,3 +67,37 @@ func TestSweepTickGateBacksOffAndShedsUnderCaution(t *testing.T) {
t.Fatalf("shed consulted outside caution: calls = %d", shedCalls)
}
}

// A candidate's evaluation and merge run at normal pacing class, lifted out
// of the lane's low class; a class the caller chose is left alone.
func TestMergeDecisionPriority(t *testing.T) {
low := lowPriority(context.Background())
if got := hgithub.PacingClassOf(mergeDecisionPriority(low)); got != hgithub.PacingNormal {
t.Fatalf("low lane ctx → %s, want normal", got)
}
high := hgithub.WithPacingClass(context.Background(), hgithub.PacingHigh)
if got := hgithub.PacingClassOf(mergeDecisionPriority(high)); got != hgithub.PacingHigh {
t.Fatalf("high ctx → %s, want high kept", got)
}
}

// During a caution window whose queue is deeper than the low class may wait
// (projectbluefin hive 2026-10-28: "class low, queue 5.454s ahead, limit
// 5s"), an approved green App PR the pass reaches is still read and merged:
// its status / check-run reads are not shed as housekeeping.
func TestSweepMergesGreenPRWhileCautionQueueExceedsLowBudget(t *testing.T) {
var merged []int
src := newFakePRSource("acme/widget", greenCachedPR(11))
c, _ := newCachedSweepEngine(t, src, greenAPIPRs(11), &merged)
// 5.5s ahead: over the 5s low budget, inside the 30s normal one. Later
// slots are 1ms apart so the pass costs one wait, not a slot per call.
t.Cleanup(hgithub.EngageRateLimitCautionForTest(5500*time.Millisecond, time.Millisecond))

result, err := c.SweepSelfAuthoredAutoMerges(context.Background(), AutoMergeSweepOptions{})
if err != nil {
t.Fatalf("sweep: %v", err)
}
if len(merged) != 1 || merged[0] != 11 {
t.Fatalf("merged=%v skipped=%d, want PR 11 merged during the caution window", merged, result.Skipped)
}
}
Loading
Loading