Skip to content

πŸ› fix(contribute): treat an instantly failed relay fix-pr task as undelivered and retry - #14690

Merged
hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-14668
Oct 27, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
v5from
scanner/fix-14668

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #14668

Why

The issue's rationguard#397 trace shows three separate problems:

  1. The nudge was reported as delivered after the relay had already failed it. fix-pr task failed arrived 116 ms after fix-pr task dispatched. Nudge still returned mode=spawn because it reported the mode at send time. That's why the stage log said contributor nudged … mode=spawn and the comment told the contributor their relay had received the task. Now Nudge waits up to 1s for the relay to refuse the dispatch. If the relay does refuse it, the outcome is queued with RelayFailed, and the comment says the relay could not take the task and quotes the relay's reason. The stage log also gets relay_failed=true.
  2. A failed fix was never retried. handleTaskFailed called completeFixTask, so a failure was settled like a success. The PR was dequeued, its owner was forgotten, and nothing re-enqueued it. That matches the 40 silent minutes in the issue. Now a failure requeues the PR. It keeps its place by age and gets 5 minutes of backoff, so a relay that may be mid-turn isn't interrupted again straight away. The relay's next ready still picks it up, because the existing boundary dispatch path ignores the backoff. After 3 failed dispatches the PR leaves the fix queue and falls back to the stage engine's existing re-nudge / adoption path. The failure reason is logged.
  3. The hive:unauthorized mark stayed after a conflict. evaluateContributorLanePR returns conflicting before reaching the step that calls clearUnauthorized, so the approve-workflow-runs mark could never clear on a dirty head. As the issue says, approving those held runs is wasted work because the rebase push creates new ones. The lane now clears the mark when the head conflicts, and the next head is re-marked if its runs are held. The row's blocker is then the conflict, which only the contributor can fix.

Nothing in this issue was already fixed on v5.

Tests (each fails without its change):

  • fixqueue.TestRequeueFailedDispatchBacksOff
  • dashboard.TestFixQueueInstantRelayFailureIsNotDelivered
  • automerge.TestContributorNudgeCommentRelayFailed
  • automerge.TestContributorLaneConflictClearsUnauthorized

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

…ivered and retry

A relay that failed a stage-072 fix-pr task 116 ms after dispatch was
still reported as reached (mode=spawn, "your relay session received this
as a fix-pr task"), and handleTaskFailed settled the dispatch exactly like
task_complete, so the PR silently left the fix queue and was never
retried. Separately, a conflicting contributor PR kept its
hive:unauthorized mark because the lane returned "conflicting" before the
step that clears it, so the row named "approve runs" as the blocker when
only a rebase can move the PR.

- Nudge waits briefly for the relay to refuse the dispatch; a refusal is
  reported as queued with the relay's reason, and the comment says so.
- task_failed on a fix requeues it with backoff (served at the relay's
  next task boundary); after three failures the PR leaves the queue for
  the stage engine's re-nudge / adoption path.
- The contributor lane clears the approve-workflow-runs unauthorized mark
  when the head conflicts; the next head re-marks if its runs are held.

Closes #14668

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 27, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 27, 2026
contributorLaneConflictReason returns "conflicting" before the later
dirty/conflicting branch, so the clearUnauthorized added there never ran and
the label stayed on a conflicting head. Clear it at the early return instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: c0f861c48

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 27, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v5 tree and the callers; this looks correct to me.

What I verified:

  • Refusal wait has no lost-wakeup window. beginFixWait (contribute_fixqueue.go:272) takes the signal channel before dispatchFixes, pulseFixFailed records the reason only while a waiter exists and closes/replaces the channel, and awaitFixTaskRefusal re-checks takeFixFailure before each select β€” so a task_failed racing the dispatch is caught either way. No lock is held across the 1 s wait, and failFixTask takes c.mu then fixQueueMu in disjoint sections, so no deadlock against the WS session goroutine.
  • Requeue keeps backoff across re-nudges. Scheduler.Enqueue now carries Attempts/NotBefore over an existing key (fixqueue.go:232,239), so a stage-engine re-nudge of the same PR does not reset the counter; Dispatch honours NotBefore and DispatchAtBoundary deliberately does not (fixqueue.go:363), which matches the body's "the relay's next ready still picks it up".
  • Conflict clears the mark at the right place. contributorLaneConflictReason returns before the later dirty branch, so the new clearUnauthorized at automerge_contributor_lane.go:1180 is the only point that runs on a conflicting head; clearUnauthorized and unauthorizedStepApproveWorkflowRuns exist at base (automerge_unauthorized.go:51,150).
  • Body matches the diff; each of the four named tests is present.

One design note, not a defect: Dispatch checks NotBefore on queue[0] only (fixqueue.go:392), so while a failed head backs off, other queued fixes for that contributor are also withheld from the interrupt path for the same 5 min. That reads as intended β€” a relay that just refused one fix-pr task is likely to refuse the next β€” but if per-item backoff was the intent, the guard would need to skip past backing-off items instead of returning nil.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit c6d82d0 into v5 Oct 27, 2026
78 of 80 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
Evidence bundle `#14690c0f861c` sha256:34c33c812c4c5fe4a5ca1d470fdf0f527a378ee9333e55f60e24eb30a16f3808 β€” download from the dashboard or `hivectl review evidence #14690`.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-14668 branch October 27, 2026 21:11
@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

0 participants