Skip to content

πŸ› [v6] army: stop attributing adopted PRs to a remote agent's rework queue and show why a fix is not dispatched - #17857

Merged
clubanderson merged 1 commit into
v6from
topup/v6-army-rework-queue
Nov 12, 2026
Merged

clubanderson merged 1 commit into
v6from
topup/v6-army-rework-queue

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Bug

The dashboard "Your army" card showed a remote agent (Karibhou, omp, max parallel 1, preempt) as idle Β· 33 PRs awaiting rework (not yet dispatched) Β· fix queue 1: hive#17739, … β€” a red wall of ~33 links, ~27 annotated adopted by hive, many for PRs that are long closed or merged (hive#15029 and #15086 are CLOSED since 2026-10-28, #17739 MERGED, hivecommons.github.io#373 CLOSED).

Root causes

  1. Stale queue. fleetReworkFor built the list from every stage-store record by author in 072-returned-to-contributor and 074-adopted-by-hive, with no open-state check. Stage-store records for merged/closed PRs are never Forget-ed when a PR leaves the open listing (only on terminal verdicts) and age out solely via Prune(14d), so closed PRs still carrying stage/074 were listed and counted for two weeks, and PRs the hive had taken over were still attributed to the relay.
  2. Dispatch stall. Fix-pr dispatches are booked per user in fixqueue.Scheduler, but the connection that holds them (c.fixTasks) is per socket. When the socket drops (1006 flaps), the dispatch stays "in flight" with no holder: Dispatch/DispatchAtBoundary refuse (capacity 1 busy), and selectFixTask answers task_unavailable fix_pr_in_flight β€” the relay gets neither the fix nor new work, forever. The card hid the idle reason behind idle Β· N PRs awaiting rework.

Fix (per Andy's clarified spec)

  • Only open PRs authored by the agent in 072 and not dispatched are listed/counted. Open state comes from the forge cache (ForgeState().PR(...).Open()); when the cache does not know the PR, a stage record the lane has not re-observed for an hour is treated as gone. Adopted (074) PRs are dropped β€” they are the hive's. A queued fix whose PR is merged/closed is removed from the fix queue.
  • pending_rework_blocked names the dispatcher's actual reason (offline, no fix_pr_interrupt_v1, scope not acknowledged, not queued, orphaned in-flight dispatch, backoff, focus window, slots busy, awaiting ready), with the relay's last idle reason appended.
  • Orphaned in-flight fixes are released (no attempt counted, fixqueue.Release) at the relay's next ready; a task_complete for such a dispatch arriving on a reconnected socket settles it.
  • UI: at most 3 links inline, the rest behind <details class="clanker-rework-more">show N more</details> (open state persisted in ccOpenPanes); tokens only, no new deps.

Tests

Go: adopted/merged/closed/unobserved excluded and removed from the queue; authored+open+072 included; blocked-reason strings (offline, orphan β†’ released at ready); completion on a reconnected socket; fixqueue.Release. JS harness: block-reason render, 5 pending β†’ 3 inline + "show 2 more", open-state persistence, ≀3 no disclosure. Docs (contributor-relay.md, stage-labels.md) and changelog fragment updated.


Top-up of #17853 (v5 β†’ v6). On v6 the existing open-PR index from #14715/#17412 stays the primary open-state source; the forge cache and the pendingReworkUnseenBound fallback decide where the index cannot vouch for a repo. Adopted (074) PRs are dropped per Andy's clarified spec; the v6 tests that expected them listed were updated accordingly. fixqueue.Release sits beside v6's Return on the shared unwind.

… and show why a fix is not dispatched (#17853)

The contribute fleet card listed every stage-store record by author in 072
or 074 as the relay's pending rework, so merged/closed PRs still carrying
stage/074-adopted-by-hive (the store prunes them only after 14 days) and
PRs the hive had taken over piled up as '33 PRs awaiting rework'. Only open,
not-adopted, not-dispatched PRs count now: the forge cache decides open vs
merged/closed when it knows the PR, otherwise a record the stage lane has
not re-observed for an hour is treated as gone. A queued fix whose PR is
gone leaves the fix queue instead of being handed to the relay.

The idle-with-queue stall: a fix dispatched to a socket that dropped stays
booked in flight with nobody holding it, so capacity 1 is busy forever and
the relay gets neither the fix nor new work (task_unavailable
fix_pr_in_flight, which the card hid behind 'idle Β· N awaiting rework').
The relay's next ready now releases such orphans (no attempt counted) and a
task_complete for the dispatch on a reconnected socket settles it. The card
prints the dispatcher's actual reason (pending_rework_blocked) instead of
'not yet dispatched', and links at most three PRs inline with the rest
behind a show-N-more disclosure.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 12, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v6 at 7f27bba plus the callers (contribute_select.go:487-498, contribute_ws.go:2255-2276, contribute_ws.go:4474, contribute_fleet.go:250-380). The change does what the body says, and the two lock-ordering questions it raises check out. Looks correct to me; two low-severity open questions for the author, neither blocking.

Verified

  • releaseOrphanedFixes β†’ heldFixTaskIDs takes h.mu.RLock and every c.mu (contribute_fixqueue.go:707-734). Both selectTask callers reach it with h.mu released (contribute_ws.go:2267-2276 unlocks before the call) and no c.mu held (repoScopeGateReason at contribute_select.go:489 locks c.mu itself first, so a held c.mu would already deadlock there). No self-deadlock.
  • FleetSnapshot gathers heldFix in its own pass before any row's c.mu is taken (contribute_fleet.go:264-280), so fleetReworkFor never holds two connection locks at once. forgetFixOwner/fixQueueMu is a leaf lock (contribute_fixqueue.go:404-410).
  • completeFixTask fallback (contribute_fixqueue.go:665-690) settles a dispatch the current socket never saw by matching sched.InFlight(user) on taskID β€” the 1006-flap case in the body. fixqueue.Release (fixqueue.go:379-387) shares unwind with Requeue/Return and leaves Attempts/NotBefore untouched, as documented; TestReleaseReturnsDispatchWithoutAttempt covers it.
  • Signed-off, changelog fragment present, contributor-relay.md/stage-labels.md updated for pending_rework_blocked.

Open questions (low)

  1. fleetReworkFor now mutates on a read path: sched.Remove + forgetFixOwner from the dashboard snapshot (contribute_fleet_rework.go:316-322) whenever reworkCandidateOpen says the PR is gone. Where neither the open-PR index vouches nor the forge cache knows the PR, that decision falls to owedRework's unseen bound (contribute_fleet_rework.go:160-172), so a PR the stage lane merely stopped observing for an hour is dropped from the fix queue, not just from the card. The body says the re-nudge path re-queues it, so the consequence is a delayed fix rather than a lost one β€” worth confirming that is intended for the queue and not only the display.
  2. Narrow race: dispatchFixesHeld books sched.Dispatch (contribute_fixqueue.go:~461-463) before it writes c.fixTasks[d.ID] (:588-591). A second ready from another live socket of the same user in that window sees the dispatch in flight but unheld and Releases it, so the same PR could be served twice. Requires two concurrent readys for one user; probably acceptable for a max-parallel-1 relay, noting it so the reasoning is on record.

I have not approved or changed anything; a maintainer should sign off.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 03fa276 into v6 Nov 12, 2026
83 of 87 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the topup/v6-army-rework-queue branch November 12, 2026 06:05
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the diff against the v6 merge-base (8c35975f2) plus the lock paths it crosses. The Go change looks correct to me; one body/diff gap worth a decision.

Body claims a UI change this diff does not contain. The body's "UI: at most 3 links inline, the rest behind <details class="clanker-rework-more">… open state persisted in ccOpenPanes" and the "JS harness" tests are not in this PR: the 13 changed files are Go, docs and the changelog fragment, with nothing under src/pkg/dashboard/static-src/ or static/. At the PR head, git grep pending_rework_blocked -- src/pkg/dashboard/static-src src/pkg/dashboard/static returns nothing, and v6 has no clanker-rework-more either. So the new FleetRework.PendingReworkBlocked field (contribute_fleet_rework.go:119) is emitted by the server and rendered by nothing. Either the v5β†’v6 top-up dropped the UI commit, or it is intended as a follow-up β€” the body should say which, since the operator-facing "shows why a fix is not dispatched" in the title currently only happens in the JSON.

Verified on the Go side

  • No new deadlock. FleetSnapshot holds h.mu.RLock + one c.mu (contribute_fleet.go:258-280) when fleetReworkFor calls sched.Remove and forgetFixOwner (contribute_fleet_rework.go:376-377); forgetFixOwner takes only fixQueueMu (contribute_fixqueue.go:405), which is a leaf lock everywhere it is used (:166,:360-368,:392,:823-891 all release before touching h.mu/c.mu). selectFixTask β†’ releaseOrphanedFixes β†’ heldFixTaskIDs locks h.mu.RLock and every c.mu (contribute_fixqueue.go:703-717); both selectTask callers (contribute_ws.go:2267-2276, :3845) have released h.mu first and do not hold c.mu, consistent with the pre-existing c.mu.Lock() at :587 in the same dispatch path.
  • The held-task pre-pass in FleetSnapshot locks connections one at a time before the main loop, so no two c.mu are ever held together, as the comment promises.
  • reworkCandidateOpen (contribute_fleet_rework.go:155-169) keeps the v6 open-PR index as the first authority and only falls to the forge cache / owedRework bound where it cannot vouch β€” matches the body's "top-up" note. Dropping 074 records removes the old ByAuthor(user, AdoptedByHive) loop entirely, and FleetReworkStateAdopted is kept as a constant so nothing referencing it breaks.
  • fixqueue.Release (fixqueue.go:380-382) routes through unwind with countAttempt=false, backOff=false, so a released item keeps its prior NotBefore and Attempts; Requeue/Return behaviour is unchanged.
  • completeFixTask (contribute_fixqueue.go:665-690) settling a dispatch the current socket never saw only does so when the scheduler still books that taskID for this user, so a stray task_complete for an unknown ID still returns false.

One side-effect note, not a defect: FleetSnapshot is documented "read-only" (contribute_fleet.go:252-253) but now prunes merged/closed items from the fix queue as it renders. The pruning is correct; the doc comment should say the snapshot does it.

@github-actions

Copy link
Copy Markdown
Contributor

Mutation testing

Head a3649088a.

⏱️ The 900s mutation budget ran out, so 1 package(s) were not fully mutated and the results below do not cover them:

  • pkg/fixqueue: skipped (not started)

❌ gremlins did not produce a report for 1 package(s), so no verdict can be derived from this run (see the job log):

  • pkg/dashboard: gremlins exited 137

Verdict: error (#17349).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant