Skip to content

🐛 scanner backlog starved: review-fix kicks defer the governor kick every cycle #15166

Closed

Description

@clubanderson

Symptom

Operator (2026-10-28 19:19 ET): "is the scanner working properly? seeing lots of actionables". The hosted spoke's Activity Monitor shows 13 issues at stage/010-actionable with no claim, the oldest for 11h (a hive/partially-addressed tracker), several for 20m+ after a pod rollout-restart released their claims. The scanner is up ("running"), inference is fine, GitHub quota is fine. It is simply never being handed the backlog.

Evidence (hosted spoke, v5 29bbea4, /data/logs/hive.log)

Scanner kicks delivered today ("audit: agent kicked","name":"scanner"): 43 governor/backlog kicks vs 50 [review-fix:…] kicks (+4 FIX-BEFORE-NEW). Since the last backlog kick at 18:14:02 there have been zero backlog kicks and six fix kicks:

18:14:02 SECURITY CANARY… (governor/backlog kick, 61773 bytes, 17 issues listed)
18:37:54 [review-fix:hivecommons/hive#15126]
18:43:26 🔴 FIX-BEFORE-NEW — hivecommons/hive#15129
18:45:34 [review-fix:hivecommons/hive#15129]
18:48:55 [review-fix:hivecommons/hive#15139]
18:54:43 [review-fix:hivecommons/hive#15105]
19:09:15 [review-fix:hivecommons/hive#15020]
19:19:55 [review-fix:hivecommons/hive#15105]
19:25:27 [review-fix:hivecommons/hive#15105]

Each cycle the governor assembles both kinds for the scanner and only the fix kick lands:

19:08:31 "kick rendered","agent":"scanner","kick_bytes":55061,"issues_total":22,"issues_listed":22
19:08:31 "governor eval complete","agents_due":["architect","reviewer","reviewer-2","scanner"]
19:08:40 "audit: governor kicking agent","agent":"scanner"   (x3 — two fix kicks + the governor kick)
19:09:15 "audit: agent kicked","name":"scanner","preview":"[review-fix:hivecommons/hive#15020]…"
19:10:57 "agent busy: deferring kick","agent":"scanner"
19:10:57 "agent busy: deferring kick","agent":"scanner"
19:16:20 "kick turn ended","agent":"scanner","seconds_since_kick":424
19:19:12 "kick rendered","agent":"scanner","kick_bytes":54989,"issues_total":17
19:19:19 "audit: governor kicking agent","agent":"scanner"   (x2)
19:19:55 "audit: agent kicked","name":"scanner","preview":"[review-fix:hivecommons/hive#15105]…"
19:19:55 "agent busy: deferring kick","agent":"scanner"

agent busy: deferring kick for the scanner appears 30+ times today; the pending-fix-already-dispatched guard never fires for the adopted PR.

Root cause — three correct-by-themselves rules compose into starvation

  1. src/cmd/hive/review_kick_order.go:32 orderReviewFixKicksFirst puts every review-fix kick ahead of the governor kick for the same agent (🐛 fix: keep review hold release across base-branch merges that do not change the PR diff #13100 — correct: a maintainer's instruction must not time out behind queue work).
  2. src/cmd/hive/eval_cycle_seams.go:585-597 dispatchAgentKicks: a delivered kick (or a busy bounce) sets deferLane = true, so the governor kick behind it in the same lane is logged agent busy: deferring kick and dropped — not failed, no RecordKickFailure, no counter, and the governor kick is re-rendered from scratch next cycle with no age.
  3. src/pkg/review/dispatch.go:1428-1434 dispatchFix: for an adopted contributor PR (stage/074-adopted-by-hive) adoptedHiveOwnedFix bypasses the pending check, so a changes_requested adopted PR is re-planned as a fix kick every cycle until escalation.MaxReEngagements (6) — by design (TestAdoptedPendingFixIsRekickedEveryCycle, 🐛 Keep adopted changes-requested PRs in the fixer work list #12987). ✨ feat(hub): surface "Pause hive" in the hub spoke dropdown menu #15105 (Karibhou, adopted, coverage check failing) was fix-kicked at 18:54, 19:19 and 19:25; 🐛 fix(automerge): hold merges whose closing keywords would close the wrong thing #15079 got 4 kicks and 🐛 fix: recover orphaned supersede adoptions #15057 2 earlier today.

Result: as long as any adopted-or-hive PR authored by the scanner carries a fresh changes_requested (on this spoke that is nearly always), the scanner's governor kick is deferred every cycle. Nothing bounds the streak, so the issue backlog (including [selfreport], ci-flake, and human-filed triage/accepted bugs — none of which are routed to another agent; the kick template lists all 17) is never offered to the agent.

Secondary effect: the 18:14 backlog turn had claimed #14598/#15106/#15109/#15116/#15120/… and spawned sub-agents when the 18:55 rollout killed it; issue-claims: released agent claims on relaunch (18:56:45) returned 22 issues to 010 and, with no backlog kick since, none were re-taken. The tracker shows them as "Triaged and eligible; waiting for the scanner to pick it up" with no hint that the scanner is being fed only fix kicks.

Expected

A fix kick may displace the governor kick, but not indefinitely. After the governor kick for an agent has been deferred behind a delivered fix kick N cycles in a row (N=2), the governor kick goes first for that cycle and the fix kick is re-planned next cycle — the same thing that already happens to an undelivered fix kick today. The governor kick carries the agent's own red PRs (FIX-BEFORE-NEW) so fix work is reordered, not lost.

Not the cause (checked)

  • Inference: inference route cleared at 18:56:45 is the normal relaunch reset; copilot usage recorded for the scanner streams continuously after each kick.
  • GitHub quota: no 429/secondary-limit lines in the window; kick rendered lists 17–22 issues each cycle.
  • Issue filters: ci-flake and [selfreport] issues are in the scanner's rendered kick; #14598 is a tracker "waiting on 1 open child issue(s)" (caption already says so).

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Oct 28, 2026
  2. github-actions commented on Oct 28, 2026

    @github-actions
    Contributor

    @clubanderson this issue was closed by @hivecommons-hive[bot] as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it — check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you — you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions