Skip to content

πŸ› fix: route undeliverable fixer kicks to fallback agents - #16541

Merged
hivecommons-hive[bot] merged 6 commits into
hivecommons:v5from
thatguywob:fix/16536-fixer-fallback
Nov 4, 2026
Merged

hivecommons-hive[bot] merged 6 commits into
hivecommons:v5from
thatguywob:fix/16536-fixer-fallback

Conversation

@thatguywob

Copy link
Copy Markdown
Contributor

When the primary fixer repeatedly refuses kicks during a long turn, red PR repairs can sit indefinitely behind it. Add opt-in review.fixer_fallback_agents and review.fixer_fallback_after_undeliverable (default 3 only with a non-empty fallback list) so review fixes and stale CI-red PR re-engagements try the first available fallback after the existing fixer candidates.

Selection also recognizes a busy fixer's transcript silence using its existing busy_no_activity_threshold. Routing is logged, the primary resumes receiving eligible repairs after recovery, and turns are never interrupted. Existing pending fixes and attempt budgets remain intact; temporary fleet backpressure stays retryable. Unset configuration preserves existing behavior. The dashboard review API, example YAML, review-swarm guide, and changelog are updated.

Validation: added regression coverage for thresholds, transcript liveness, ordered fallback availability, recovery, pending-fix accounting, retry without a human hold, and dashboard configuration updates. gofmt -l and git diff --check are clean. Tests are deferred to CI as required by AGENTS.md; no local test suite was run.

Fixes #16536

β€” hive: backend=codex model=gpt-6.1-sol effort=high


🐝 Hive Agent: contributor | SHA: f226ee1c8

Add opt-in ordered repair fallbacks for review fixes and stale red PR re-engagements. Use refusal counts and existing transcript silence thresholds without interrupting turns, and keep temporary fleet backpressure retryable.

Hive-Run: hivecommons#16536
Hive-Plan: hivecommons#16526/fixer-lane
Hive-Spec: issue-16536#acceptance-criteria
Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 4, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 4, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • test (dashboard shuffle 3/3)

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

Add ordered fallback agents and the undeliverable kick threshold to the Features tab so new review settings satisfy dashboard parity. Extend the existing controls coverage.

Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • test (rest 3/4)

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read agentAvailability/fixerUndeliverable/resolveRedPRFixer in src/cmd/hive/main_sweeps.go, selectFixerAgent/dispatchFix in src/pkg/review/dispatch.go, the config accessors, and the crux side they depend on. The core routing looks correct to me.

Verified against the tree:

  • KicksUndeliverable is reset to 0 on a delivered kick (src/crux/agent/busy_visibility.go:268-290), so the primary genuinely resumes receiving repairs after recovery, as the body says.
  • Process carries KicksUndeliverable, LastTranscriptActivity, ProviderErrorBackoffUntil and Config (src/crux/process.go:21-74), and GetStatusFast is on the crux.Runtime interface (src/crux/runtime.go:91).
  • With the fallback list unset, EffectiveFixerFallbackAfterUndeliverable() is 0, reviewFixerAvailability returns nil, and agentAvailability returns at the old cadenceReachable contract β€” so unset config is a true no-op.

One low-severity note: when the liveness filter rejects every candidate but a capability-only pass would have found one, dispatchFix correctly skips the human hold but still records the exclusion as FixDispatchExcludedNoEnabledFixer (src/pkg/review/dispatch.go:1602). That reason feeds the once-per-head WARN in auditWithheldReviewFixes (src/cmd/hive/main_sweeps.go:2893-2903), so a temporarily busy fixer logs as "no-enabled-fixer", which reads as a configuration fault. A distinct reason (e.g. fixer-busy) would keep that log honest. Not a blocker.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

Move fallback controls into the dashboard source fragments and regenerate index.html. Use the existing spacing utility to preserve the style ratchet baseline.

Signed-off-by: thatguywob <thatguywob@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16492 ✨ feat(dashboard): changed-paragraph reader kiosk view (part 1/3) same files src/pkg/dashboard/static/index.html
#16554 πŸ› Activity Monitor: a 071/072 row with a pushed or checking head keeps its CI track lit same files src/pkg/dashboard/static/index.html
#16553 πŸ› fix: expose deferred PR fixer queue positions same files src/pkg/dashboard/static/index.html
#16514 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/pkg/config/config.go
src/pkg/dashboard/static/index.html
#16500 ✨ feat(dashboard): replace Matrix presentation with Code activity same files src/pkg/config/config.go
src/pkg/dashboard/static/index.html
#16513 ✨ feat(dashboard): feed published checkpoint diff regions to Code activity same files src/pkg/dashboard/static/index.html
#16456 ✨ feat(github): enforce pin-only exception in merge-request relay same files src/pkg/config/config.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • mobile-shots

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@thatguywob the hive has adopted this PR (stage/074-adopted-by-hive): your hive has been offline since the nudge. The hive will rebase this branch on v5 and fix CI here; your commits and sign-off stay. Push a new head yourself at any time and the hive yields it back to you.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Shared CI incident: the red mobile-shots check here is the tracker-header-geometry probe failing identically on every open v5 dashboard PR (probe still expects a single-line chevron; v5 now ships a two-line box). It is not caused by this diff and is reporting-only (continue-on-error). Tracked in #16551 β€” no per-PR repair; deferred until the probe is updated.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the tree at afaa0df and built ./cmd/hive ./pkg/review ./pkg/config ./pkg/dashboard on befc8c5 β€” compiles clean, and the routing logic looks correct to me.

What I verified

  • Opt-in really is inert when unset. EffectiveFixerFallbackAfterUndeliverable() (review_dispatch.go:242) returns 0 whenever the agent list is empty, so a threshold configured alone changes nothing; agentAvailability (main_sweeps.go:240) short-circuits to the old paused + cadence contract in that case, and reviewFixerAvailability (:276) returns nil so selectFixerAgent keeps its capability-only path. The three other agentAvailability consumers (merge re-engage hook, red-PR reaper, release sentinel) are therefore unaffected for non-opted-in hives.
  • Threshold semantics are sound. KicksUndeliverable is cleared by resetBusyVisibilityLocked (crux/agent/busy_visibility.go:285) the moment a kick is delivered, so "consecutive refusals" means what the config comment says and a recovered primary is re-selected on the next sweep (covered by TestReEngagementFallsBackAndReturnsToRecoveredPrimary).
  • No spurious human hold. dispatchFix (dispatch.go:1607-1614) re-runs selection with FixerAvailable=nil before writing the "no enabled kick-capable fixer" hold, so all-candidates-busy stays retryable rather than parking the PR on a maintainer β€” the right call for transient backpressure.
  • Candidate filter matches existing precedent. The OnDemand || !UsesGovernorKick() exclusion at main_sweeps.go:244 is the same test main.go:268 already applies to governor kicks.

One behavioural note (info, not a defect): once opted in, agentAvailability also applies the kick-capability and ProviderErrorBackoffUntil checks to the reaper / merge-hook / release-sentinel paths, not only review fixes. That seems intended (the comment at :238-240 says so), but it means enabling fixer_fallback_agents tightens those sweeps too, which an operator reading only the review.* key might not expect. Worth one sentence in review-swarm.md if it isn't already there.

The red mobile-shots check is the pre-existing chevron-height failure #16556 fixes, not this diff.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the tree at v5 plus the callers of everything this touches. The routing itself is correct and the opt-in gating holds; three low-severity notes below, none of which I'd block on.

What I verified

  • selectFixerAgent (src/pkg/review/dispatch.go:1660-1673): fallbacks are appended after the existing FixerAgent β†’ pr.Lane β†’ DefaultFixerAgent chain, and the FixerAvailable filter is only non-nil when EffectiveFixerFallbackAfterUndeliverable() > 0 (main_sweeps.go:275-280), which is itself zero whenever the list is empty (review_dispatch.go:243-245). So an unset list β€” or a threshold set alone β€” is byte-for-byte the old behaviour. resolveRedPRFixer (main_sweeps.go:333) gets the same appended chain.
  • The capability gate added to agentAvailability (main_sweeps.go:243-245) mirrors the one selectFixerAgent already applies at dispatch.go:1668 (Enabled && !Paused && !OnDemand && UsesKick), so no new class of agent becomes eligible.
  • Retry-without-hold (dispatch.go:1604-1614): the human hold is kept only when the capability-only re-selection also returns ""; attempt budget survives because ConfirmDelivered already copies delivered fixes into State.FixAttempts (dispatch.go:1736) and dispatchFix reads that history at :1596.
  • KicksUndeliverable is reset on a delivered/cancelled kick or restart (src/crux/agent/busy_visibility.go:268-290), so "primary resumes after recovery" holds once any kick lands on it.

Notes (low)

  1. dispatch.go:1606 still records FixDispatchExcludedNoEnabledFixer ("no-enabled-fixer") in the retryable branch. auditWithheldReviewFixes (main_sweeps.go:2952-2962) logs that at WARN once per (pr, head, reason), so an operator sees "no enabled fixer" when every fixer is merely busy. A distinct reason (e.g. fixers-undeliverable) would make the log honest and would also let the WARN de-dup key distinguish the two cases.
  2. fixerUndeliverable (main_sweeps.go:265-272): the transcript-silence clause fires on KicksUndeliverable >= 1 plus stale LastTranscriptActivity. Because the counter only resets on a delivered kick (busy_visibility.go:285), a primary that refused one kick mid-turn and then went idle for longer than busy_no_activity_threshold is still routed around until its next cadence kick lands β€” your "silent busy" test case has the same inputs as that idle state. The fleet already treats a leftover counter as busy (eval_cycle_seams.go:192), so this is consistent, just worth knowing: recovery is "next delivered kick", not "turn ended".
  3. agentAvailability is shared by four callers (github_client_config.go:346, main_eval_cycle.go:723,731, and this path). With fallback opted in, the merge re-engage hook and the release sentinel also pick up the capability gate and the GetStatusFast error β†’ unavailable rule; the sentinel's error text at release_sentinel.go:181 ("paused or has no reachable cadence") then under-describes a busy agent. Intentional for the red-PR reaper per your comment; flagging so the others aren't a surprise.

The red mobile-shots check is the pre-existing chevron-height failure that #16556 addresses, not something this diff touches (no dashboard CSS/markup changed here beyond the two new Settings fields).

Body and diff agree. Looks correct to me.

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Evidence bundle `#16541d8a0e71` sha256:d8fce797b55630801e7dbd8b7f3260f75ada61effcb3322043318bbcdbbf4ff3 β€” download from the dashboard or `hivectl review evidence #16541`.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v5 tree plus the callers of agentAvailability (main_eval_cycle.go:723,731, github_client_config.go:346), the undeliverable/transcript counters in crux/agent/busy_visibility.go (reset at :285, so the primary does come back after recovery), and the exclusion consumers in pkg/review/fix_dispatch_status.go and main_sweeps.go:2944-2962. The change does what the body says: unset fixer_fallback_agents leaves every path identical (EffectiveFixerFallbackAfterUndeliverable() returns 0 β†’ agentAvailability returns early, reviewFixerAvailability returns nil), and with the list set the skip conditions match the review-swarm.md paragraph. Looks correct to me.

One low-severity observability gap, not a blocker:

  • src/pkg/review/dispatch.go:1606 β€” when every candidate is excluded only by the new liveness filter, dispatchFix still records FixDispatchExcludedNoEnabledFixer before the capability-only recheck. auditWithheldReviewFixes (src/cmd/hive/main_sweeps.go:2960) then logs WARN review fix dispatch excluded reason=no-enabled-fixer for what is really "all fixers busy, retrying next cycle", and the two new "routed to fallback fixer" info logs never fire in this case. An operator debugging a stalled repair reads "no enabled fixer" and goes looking at config. A distinct reason (e.g. fixer-busy) recorded only when the capability-only pass succeeds would make the log truthful; FixDispatchExclusionText has no case for it today, so it would render the raw reason string, which is still accurate.

Note for the maintainer: because agentAvailability is shared, opting in also applies the liveness gate to the merge re-engage hook and the release sentinel, not just the review-fix and stuck-red-PR paths. That reads as intended (same "don't kick a refusing agent" rule) but the docs only name the first two.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit b12d7a6 into hivecommons:v5 Nov 4, 2026
71 of 73 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Merged by hive contributor-lane auto-merge: CI green for β‰₯10m, reviewer verdict approve, head d8a0e711d40a adopted by the hive (stage/074-adopted-by-hive) with @thatguywob's commits β€” merged under the hive's own authority.

clubanderson added a commit that referenced this pull request Nov 4, 2026
src/cmd/hive/fixer_fallback_test.go arrived from v5 (#16541) and drives
planReviewDispatch with actionableWithPR("hive-bot"). v6 gates every fix
kick on fixPushAuthorOK (#13129): only a "[bot]"-suffixed
login may be pushed to, so on v6 the fix was withheld and FixKicks was
empty in every subtest, including the no-fallback baseline. Git merged the
file cleanly because the two sides never touched the same lines.

Use "hive-bot[bot]", matching v6's own review_dispatch_wiring_test.go, so
the fallback routing under test is reached. Assertions are unchanged.

Fixes test (rest 4/4), and thereby the dependent test and coverage jobs,
on the v5 -> v6 top-up.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] Fixer lane: route fix kicks to fallback agent when primary fixer is undeliverable

1 participant