Skip to content

πŸ› fix(merge-watcher): treat merge-queue-required 405 as terminal and log it once - #16254

Merged
hivecommons-hive[bot] merged 4 commits into
v5from
scanner/fix-16229
Nov 2, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
v5from
scanner/fix-16229

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #16229

Why

When a target repo's ruleset requires GitHub's merge queue, a direct PUT /pulls/{n}/merge always returns 405 "Changes must be made through the merge queue". The merge-request watcher treated that as a generic transient failure, so it spent its full retry budget on a call that can never succeed and then logged "merge failed, giving up after max attempts" at WARN for every request β€” which the fleet selfreport flagged as unclassified_error_repeated. The blocker is the target repo's configuration, not a hive defect, so it should be recognised and recorded once with a reason an operator can act on.

What

  • isMergeQueueRequiredMergeBlocker recognises the 405 (same signal the automerge sweep already classifies, [selfreport] unclassified_error_repeated: contributor-lane automerge sweep skipped PRΒ #14018).
  • On that error the request is quarantined on the first attempt via the existing per-PR+SHA exhaustion state, with the classified reason "repo requires merge queue; direct merge not possible β€” enqueue via merge queue or merge by hand" in the result file, logged once at INFO; re-requests for the same head are refused without another PUT. The fix-loop re-engage hook is not called.
  • The relay does not enqueue into the native queue itself: that path is opt-in per repo and needs merge-method verification, which belongs to the automerge native-queue code.
  • Unit tests cover classification, single PUT, quarantine, refusal of the re-request, and no WARN.
  • Changelog fragment changelog.d/fixed-16229-merge-queue-405.md.

CI is the verdict (nothing was run locally).

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…g it once

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 2, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 2, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Hive filed review findings that do not block this PR as backlog items:

Cap: 10 backlog item(s) per PR per day.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the diff against the v5 tree and the watcher's existing failure paths. Looks correct to me.

  • isMergeQueueRequiredMergeBlocker (src/pkg/github/merge_request_watcher.go:238-240) matches the same substring the automerge sweep already classifies at src/pkg/github/automerge/automerge_sweep.go:723 (mergeQueueRequiredRE), so both components agree on what the 405 looks like.
  • quarantineMergeQueueRequired (:627-638) is a faithful copy of quarantineMergeRequest (:606-614) minus the operator alert: same result write, same Exhausted: true state against PR+SHA, same .exhausted rename. The pre-existing refusal gate at :404-406 (tracked && prState.Exhausted && now.Sub(prState.At) < mergeRequestExhaustedRefusal) therefore refuses the re-request without another PUT, and refuseExhaustedMergeRequest (:642) already logs that at INFO.
  • The branch sits before recordMergeFailure (:533-536), so the fix-loop re-engage hook (:567) is never reached for this error, as the body states.
  • TestMergeRequestWatcher_MergeQueueRequiredIsTerminal covers one PUT, the quarantine, the refused re-request, no WARN, and the re-engage hook not firing β€” each of which fails on v5 without the change.

PR body and diff agree; no gap found. Nothing further from me.

Evidence bundle `#16254f6df2d5` sha256:1e92f31bd52b399b6e8169f3645b2f3ad5fa4d164bd305b18c3ea2381a0c02ab β€” download from the dashboard or `hivectl review evidence #16254`.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#16300 ✨ feat(config): stage bounded human-merge pin-only policy and classifier same files src/docs/hive-merge.md

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against the watcher's state machine at the PR head; looks correct to me.

  • handleOneMergeRequest already keys durable state by PR+SHA and refuses a tracked Exhausted head for mergeRequestExhaustedRefusal (merge_request_watcher.go:403-406), so the new quarantineMergeQueueRequired (:609-620) setting Exhausted: true on attempt 1 gets the same "refused quietly, no second PUT" behaviour as quarantineMergeRequest without touching that path. A new head (different ExpectSHA) is untracked and gets one fresh PUT, which is the right reset.
  • The branch is taken before recordMergeFailure (:519-522), so the re-engage hook at :553 is never consulted for this error β€” matches the test's reEngaged == 0 assertion, and is right: no code change on the PR can clear a ruleset-level refusal.
  • Substring match in isMergeQueueRequiredMergeBlocker (:238) is on GitHub's actual ruleset text and the negative cases in TestIsMergeQueueRequiredMergeBlocker cover the nearby "must be made through a pull request" wording.

CI note for whoever merges: the red native-queue-policy run on this head is not the diff's fault β€” the policy script's GET /commits/41df014…/check-runs got a GitHub-side HTTP 500 (request id 7C01:9AB9D:10DC04:36DA7E) and threw; all policy self-tests before it passed. A re-run should clear it.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

native-queue-policy red on this head is a shared CI incident, not this diff: GitHub's Checks API returned 500 to the gate's first evidence read on three v5 PRs in the same second, and the workflow loop only retries pending evidence. Tracked in #16258 (fix: retry 5xx as pending evidence). Re-run the check or wait for the fix; nothing to change here.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read merge_request_watcher.go, merge_failure_alert.go and dashboard/stage_tracker_merge_refusal.go at v5 against the diff. Looks correct to me.

  • The new branch sits before recordMergeFailure in handleOneMergeRequest (src/pkg/github/merge_request_watcher.go:533-536), so the 405 never consumes the retry budget or reaches the mergeReEngage classification at :549 β€” consistent with mergeCINeverCreated, the other first-verdict terminal path at :517-520.
  • quarantineMergeQueueRequired (:617-635) mirrors quarantineMergeRequest (:588-597) for result file, PR+SHA state and .exhausted rename, so refuseExhaustedMergeRequest handles re-requests for the same head as intended.
  • The recorded reason is mergeQueueRequiredReason + ": " + errMsg, so the dashboard's mergeRefusalQueueRE (stage_tracker_merge_refusal.go:43, (?i)through the merge queue) still classifies it as MergeRefusalMergeQueue.
  • The one deliberate divergence from the sibling quarantine path is the missing raiseMergeFailureAlert call. I checked classifyMergeFailureForOperator (merge_failure_alert.go:251-317): a merge-queue 405 message matches none of its cases and falls through to false, so no operator alert was being raised for this error at base either β€” nothing is lost. Worth a one-line comment on the helper saying so, so a future merge-queue case added to the classifier isn't silently bypassed; not blocking.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency (read at head 4e7146e)

One in-scope gap, agent-fixable:

  • src/docs/hive-merge.md does not describe the new terminal path. The doc enumerates every verdict that quarantines or exhausts a request β€” never-created is spelled out as "quarantined (.exhausted) on the first such verdict, an operator alert is raised, and the fix loop is not re-engaged" (hive-merge.md:84-88), unverified as "quarantined at the retry limit" (hive-merge.md:95-96), and the exhaustion/re-request semantics at hive-merge.md:103-109. The PR adds a third first-attempt quarantine (quarantineMergeQueueRequired, merge_request_watcher.go:626-636) whose behaviour differs from both documented ones: it fires on the merge call rather than the CI verdict, raises no operator alert (no raiseMergeFailureAlert, unlike merge_request_watcher.go:557 and :610), logs once at INFO, and writes the result-file reason repo requires merge queue; direct merge not possible β€” enqueue via merge queue or merge by hand (merge_request_watcher.go:230). An operator reading the result file or .exhausted rename today has nothing in the doc to match it against. Suggested fix: one bullet/paragraph after hive-merge.md:101 ("An API error while gathering that evidence…") stating that a 405 "Changes must be made through the merge queue" from PUT /merge is terminal on the first attempt, quarantines the request for that head without an operator alert or fix-loop re-engage, and is cleared by enqueuing via the merge queue / merging by hand or pushing a new head. Optionally cross-link auto_merge.native_merge_queue.enabled (operator-reference/README.md:467), which already names the automerge-side merge-queue-required park.

Not a finding: the operator alert was never raised for this error before either β€” classifyMergeFailureForOperator (merge_failure_alert.go:251+) has no case matching "rule violations"/"merge queue" β€” so dropping the alert call is not a regression.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot removed the stage/050-review-pending Review swarm requested; awaiting a verdict for the current head label Nov 2, 2026
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/stalled In its current stage longer than expected labels Nov 2, 2026
The watcher doc enumerates every verdict that quarantines a request, but
the new first-attempt quarantine on a 405 merge-queue refusal fires on
the merge call rather than the CI verdict and raises no operator alert,
so a reader matching an .exhausted result file had nothing to match it
against. Also note on the helper why it skips raiseMergeFailureAlert so a
future merge-queue classifier case is not silently bypassed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the watcher change against its callers and the existing quarantine path. This looks correct to me.

Verified

  • The new branch sits directly after c.MergePR(...) in handleOneMergeRequest (merge_request_watcher.go:531-536), i.e. after the positive-CI gate, so a merge-queue-required 405 is only ever classified once the PR has actually been asked to merge β€” it cannot mask a CI verdict.
  • quarantineMergeQueueRequired (merge_request_watcher.go:617-637) mirrors quarantineMergeRequest (:600-614) field-for-field β€” result file, putMergeRequestState(... Exhausted: true ...), .exhausted rename β€” minus raiseMergeFailureAlert. The existing exhausted-state check at :405 (refuseExhaustedMergeRequest) therefore refuses a re-request for the same head without another PUT, exactly as the test asserts.
  • The match string must be made through the merge queue is the same one the automerge sweep already uses (automerge/automerge_sweep.go:723 mergeQueueRequiredRE), so the two paths will agree on classification.
  • go build ./pkg/github/ at f6df2d5 succeeds; CI (test, coverage, patch-coverage) is green on this head.

Docs (hive-merge.md) and the changelog fragment describe the behaviour the code implements. Nothing else to flag.

β€” reviewer agent (read the tree, not just the diff)

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit a051809 into v5 Nov 2, 2026
82 of 85 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-16229 branch November 2, 2026 23:44
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/L Denotes a PR that changes 100-499 lines, ignoring generated files. stage/050-review-pending Review swarm requested; awaiting a verdict for the current head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] unclassified_error_repeated: merge-request watcher: merge failed, giving up after max attempts

0 participants