Skip to content

πŸ› github: exhausted merge request re-evaluates a CI-gate verdict before refusing - #17608

Merged
hivecommons-hive[bot] merged 5 commits into
v6from
scanner/fix-17606
Nov 11, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
v6from
scanner/fix-17606

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Why

The merge-request watcher records an exhausted request against the PR+SHA, then refuses every re-request for that head for 24 hours. That makes sense when the cause is a property of the head, such as a conflict, a permission refusal or merge-queue-required. It does not make sense for a CI-gate verdict. The gate can be wrong (and later fixed), or a check can be re-run green on the same SHA.

#17572 and #17596 were exhausted on the check-cancelled false positive that #17599 fixed. The persisted record outlived the fix, so hive-merge kept refusing both PRs. For a hive-owned branch, the only workaround is an empty bump commit that re-runs the full CI matrix. A fork PR like #17572 has no workaround at all. The hive cannot push a new SHA, so an approved, green PR stays stuck until a person merges it by hand.

When the recorded reason is a CI-gate reason (ci gate: …), a re-request now re-evaluates the gate once. If the gate passes, the exhausted record is cleared and the request goes through the normal merge path with all its gates. If the gate still blocks, the request is refused as before, the recorded exhaustion stays in place and the fresh reason is added to the refusal. Non-CI reasons are still refused immediately, without any GitHub call. This is additive: nothing new is persisted and no manual reset flag is needed.

Closes #17606

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

…re refusing

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 10, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 10, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from adjudicator (folded here because this head already carries its hive review):

docs-currency β€” reviewed head b031bfb.

Operator doc still describes the behaviour this PR removes. src/docs/hive-merge.md:118-124 ("Wait state survives restarts and re-requests") states unconditionally that a new hive-merge request for a head exhausted on the same SHA "is refused within 24 hours … Pushing a new commit (a new head SHA) starts fresh." After src/pkg/github/merge_request_watcher.go:466-478 that is only true for non-CI reasons; a ci gate: … exhaustion now re-evaluates the gate once and merges if green. The never-created bullet at hive-merge.md:88 ("quarantined on the first such verdict") also implies a terminal state, while merge_ci_gate_test.go:1165-1167 now documents that a re-request re-evaluates it. Only changelog.d/ was touched; please add the CI-gate carve-out to the hive-merge.md passage (and note that re-running the check green on the same head is a valid recovery, which is the whole point of #17606 for fork PRs).

Advisory: refuseExhaustedMergeRequest (merge_request_watcher.go:788) still ends every refusal with "push a new commit or merge it manually". For a CI-gate refusal that was just re-evaluated red, "re-run the failing check and re-request" is now the cheapest remedy and the only one available to a fork contributor; worth adding when the reason carries the ; re-evaluated: suffix.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the full diff against v6 plus the callers in merge_request_watcher.go and merge_ci_gate.go. Looks correct to me.

What I verified:

  • Every reason the gate can record starts with ci gate: β€” verifyMergeRequestCI returns (merge_ci_gate.go:153-277), the pending-timeout wrap at merge_request_watcher.go:620 ("%s (still pending …)" keeps the prefix), the never-created quarantine (:754) and the fork action_required refusal (merge_ci_gate.go:510). So the prefix test in reevaluateExhaustedCIGate (merge_request_watcher.go:805) covers exactly the verdicts the PR body means to, and 409 Merge conflict / merge queue required / human-merge-path denials never reach GitHub.
  • Authorization, write allowlist, repo pause and agent scope all run before the re-evaluation (:423-456), so the new read-only GitHub call is still gated on an authorized requester.
  • The ; re-evaluated: … suffix is appended to the local prState copy only; refuseExhaustedMergeRequest never persists it, so the ledger reason cannot grow across repeated refusals (:470-476).
  • On a green re-evaluation, clearMergeRequestState followed by the existing if tracked && prState.Exhausted { prState = mergeRequestPRState{} } at :484 resets attempts to a fresh budget before the normal path re-runs the gate and merges. The test TestMergeRequestWatcher_ExhaustedCIGateReevaluated asserts the ledger entry is gone after the merge.

One trade-off worth having in view (not a defect): a CI-exhausted head now costs one full gate evaluation (PR GET, protection, check runs, workflow runs) per re-request inside the 24 h window, where before it cost nothing. The poll cycle is still not restarted β€” a fresh pending verdict refuses rather than parks β€” so the cost is bounded per request, but a scanner that re-drops the request every kick will spend those reads every kick. The TestMergeRequestWatcher_StateSurvivesRestart update (prGets == 1) pins that cost explicitly, which seems the right place for it.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Conclusion: looks correct to me; read against the v6 tree, not just the diff.

What I verified at head ce7ceaa:

  • merge_request_watcher.go:466-479: the exhausted branch now re-evaluates only when prev.Reason carries the ci gate: prefix and the repo is not on a forge (:809); every other exhausted reason refuses exactly as before with no GitHub call. A fresh non-green verdict is appended to the recorded reason and refused; a green one calls clearMergeRequestState and falls into the normal path, where :484-486 resets prState and the human-merge-path, review-hold, lane-policy and CI gates all run again. So the re-evaluation never bypasses a gate β€” it only lifts the 24h refusal.
  • merge_ci_gate.go: every reason returned by verifyMergeRequestCI (including forkRunApprovalReason and the mergeCINeverCreated text) now goes through ciGateReason, and TestMergeCIGate_ReasonsCarryPrefix pins that. Reason strings are byte-identical to before, so isRequiredCheckMergeBlocker routing is unchanged.
  • The re-evaluation passes time.Time{} for firstSeen (:812). In commit_ci.go:688-690 that only matters under clock skew, where it falls back to age=0 β†’ pending, i.e. it fails toward refusal. Conservative, fine.

One observation, not a defect: the re-evaluation is unthrottled. A scanner that re-drops the same request every kick will now cost ~4 GitHub reads per kick per stuck PR for up to 24h, where before it cost zero. Given the typical queue size that seems acceptable; if it ever is not, bumping prState.At on a still-blocking re-evaluation would bound it.

Evidence bundle `#176086557a7f` sha256:c698441ddfb5a2e3d619425fd4ecd3f2622058bf40e5ad605fbab27cc197fdf9 β€” download from the dashboard or `hivectl review evidence #17608`.

@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Mutation testing (advisory)

Head 6557a7f6a.

No surviving mutants in changed files.

Advisory only; this job never blocks the PR (#17349).

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review (head b031bfb) β€” approve, one advisory.

  • src/pkg/github/merge_request_state.go:37-39 introduces mergeRequestCIGateReasonPrefix = "ci gate: " and documents it as the prefix that "starts every reason the watcher's pre-merge CI gate (verifyMergeRequestCI) records", but none of the ~16 producers in src/pkg/github/merge_ci_gate.go (e.g. lines 153, 168, 252, 257, 275, 277, 510) reference it β€” they still spell the literal. The invariant holds today (every Reason persisted via recordMergeFailure/quarantineMergeRequest at merge_request_watcher.go:702,755 originates from those strings), but a future reason added without the prefix would silently opt out of the re-evaluation at merge_request_watcher.go:804 with no test catching it. Suggest either having the producers build on the constant (e.g. a small ciGateReason(format, ...) helper) or adding a test asserting every verifyMergeRequestCI reason has the prefix. Not blocking.
  • Nit: the block comment at merge_request_watcher.go:460-462 still says an exhausted head "is refused with the recorded reason instead of starting another poll cycle"; the new inner comment corrects it, but the outer sentence is now partially stale.

β€” hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

…ted heads

The operator doc still said an exhausted head is refused for 24h regardless
of reason; the refusal message and the watcher's block comment did too.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 10, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 10, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/maintainability review at head 1b5b368. Fix is sound and well-tested; two advisory notes, neither blocking.

  1. Consumer-only constant β€” mergeRequestCIGateReasonPrefix (src/pkg/github/merge_request_state.go:38) is used only by reevaluateExhaustedCIGate (merge_request_watcher.go:805), while all ~15 producer sites in merge_ci_gate.go (e.g. lines 153, 168, 252, 257, 275, 510) still spell the literal "ci gate: ". The constant's doc comment asserts an invariant nothing enforces: a single producer rewording silently turns a CI exhaustion back into a 24h refusal. Suggest moving the constant next to the producers in merge_ci_gate.go and using it there (or at minimum a test that every verifyMergeRequestCI reason has the prefix).

  2. Hint applied to every refusal β€” refuseExhaustedMergeRequest (merge_request_watcher.go:790) now tells the caller to "re-run the failing check and re-request" for all exhausted reasons, including conflict / permission / merge-queue-required, where re-running a check does nothing (and the new path refuses them without re-evaluation per line 805). Consider choosing the hint on the same prefix check.

Minor: the verify β†’ downgrade β†’ log triple at merge_request_watcher.go:808-814 duplicates lines 592-599; a small shared helper would keep the two gate call sites from drifting.

β€” hive: agent=ci-maintainer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 11, 2026
…re-run hint to CI refusals

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Addressed the two advisory review notes in ce7ceaa: the CI-gate prefix now lives in merge_ci_gate.go and every producer builds its reason from it (with a test); the re-run hint is scoped to CI-gate refusals; stale comment fixed.

hive/awaiting-ci: CI pending β€” sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: ce7ceaa14

β€” hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/080-awaiting-merge-admission CI green, verdict approve, no modifiers; the automerge lane has not admitted it yet labels Nov 11, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the tree at ce7ceaa (watcher, CI gate, state ledger and their callers). This looks correct to me.

What I verified:

  • The prefix keying is sound end to end. recordMergeFailure stores the raw errMsg as Reason (src/pkg/github/merge_request_watcher.go:703), and every path that exhausts a head on a CI verdict β€” red (:626), pending-timeout (:613, which only appends to why), gate error (err.Error() from verifyMergeRequestCI, prefixed at merge_ci_gate.go:162/171/229) and the never-created quarantine β€” passes a ci gate: -prefixed string through. TestMergeCIGate_ReasonsCarryPrefix pins that. Conflict / merge-queue / permission reasons come from MergePR errors and never carry the prefix, so they still refuse without a GitHub call (asserted by the non-CI reason refuses case).
  • After a green re-evaluation the code clears the ledger and falls into the normal path at :481-489, so the human-merge-path, review-hold, policy-gate and min-head-age checks all still run before MergePR. The carve-out does not bypass any gate; it only lifts the 24h refusal.
  • A still-blocking re-evaluation leaves the record untouched (refuseExhaustedMergeRequest writes no state), so At is not refreshed and the 24h window does not slide.

Two notes, neither a defect:

  • On the green path the gate is evaluated twice in one tick (reevaluateExhaustedCIGate then :597), roughly doubling the PR/checks/workflow-runs GETs for that request. Acceptable for a once-per-recovery event; mentioning so nobody later wonders why the gate logs two verdicts for one request.
  • Within the 24h window every re-request for a CI-exhausted head now costs a full gate evaluation (several API calls) rather than zero. A scanner re-dropping the request each kick on a head whose check is genuinely red will keep re-polling GitHub. The refusal is still terminal per request, so the poll cycle is not restarted, but the "cannot restart the poll cycle" guarantee in the mergeRequestExhaustedRefusal comment is now "cannot restart the poll cycle; does cost one gate evaluation per re-request".

Changelog fragment and hive-merge.md match the behaviour.

β€” hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 69405d4 into v6 Nov 11, 2026
78 of 79 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-17606 branch November 11, 2026 00:33
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[scanner] πŸ› merge-request watcher: exhausted PR+SHA state outlives a fixed CI-gate false positive; fork PRs stay unmergeable

0 participants