Skip to content

🐛 fix(automerge): log ok-to-test approval warning once per PR head - #16201

Merged
hivecommons-hive[bot] merged 4 commits into
v5from
scanner/fix-16199
Nov 2, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
v5from
scanner/fix-16199

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Closes #16199

Why

The sweep re-evaluates held PRs every tick, and the post-grace 'workflow approval still required' WARN had no dedupe, so a stuck dependabot PR tripped unclassified_error_repeated. It now logs once per PR head via the existing markTerminalHoldLogged helper; the returned hold reason is unchanged.

CI is the verdict (nothing was run locally).

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 2, 2026
@kubestellar-prow kubestellar-prow Bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Nov 2, 2026
clubanderson pushed a commit that referenced this pull request Nov 2, 2026
Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@hivecommons-hive

hivecommons-hive Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

hive/awaiting-ci: changelog fragment converted to an entry bullet (the earlier coverage red was cancelled shards from a superseded head). CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 08cbd84

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the tree at b73ee19, not just the diff. This looks correct to me.

  • automerge_held_runs.go:231 dedupes under workflow-approval-grace-warned:<context>, while the sweep loop at automerge_sweep_loop.go:325 dedupes its own "automerge hold" line under the bare reason workflow-approval-required:missing-workflow:<context>. markTerminalHoldLogged (automerge_held_runs.go:678) keys on head@reason, so the two never collide and the hold line still fires once per head — the new assertion at automerge_held_runs_test.go:430 pins exactly that.
  • The key includes the head SHA, so a new push re-warns once; the returned hold reason is unchanged, so staging/isTerminalRequiredCheckHold behaviour is untouched.
  • The added third tick in the test would log the WARN twice without the guard, so the test fails without the fix.

go build ./pkg/github/automerge/ is clean (go test is disabled on this host). The red native-queue-policy check is the queue policy refusing a PR carrying needs-human ("head/base moved, draft, closed, or held"), not a defect in this change.

Review findings from reviewer (folded here because this head already carries its hive review):

Looks correct to me. Read the dedupe helper and its callers at head b9042e6:

  • src/pkg/github/automerge/automerge_held_runs.go:678 — markTerminalHoldLogged keys on <repo>#<pr>@<sha>@<reason>, so the new workflow-approval-grace-warned:<ctx> key is per-head and distinct from the returned reason workflow-approval-required:missing-workflow:<ctx>.
  • src/pkg/github/automerge/automerge_sweep_loop.go:325 — the sweep loop's once-per-head "automerge hold" line still claims the bare reason, so it is not pre-empted by the grace warning. The test's last assertion (automerge_held_runs_test.go:430) pins exactly that.
  • c.warn writes through c.logger (automerge_sweep_queued.go:657), so swapping c.logger for a buffer in the test is the right capture point.

Return value and the within-grace info line are unchanged; the diff only gates the WARN. Scope matches #16199.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Reviewer adjudication: DE-ESCALATE — evidence: the only red check is native-queue-policy, which failed at assertEligible with 'head/base moved, draft, closed, or held' because BLOCKED_LABEL (.github/scripts/native-queue-policy.cjs:22) matches the PR's own needs-human label — a circular, environmental hold, not a defect in the branch. Every build/test/coverage/lint/DCO job on head b73ee19 is SUCCESS; v5 still lacks workflowApprovalGraceWarnKey so the fix is still wanted (#16199 open); diff is additive (+24/-2, only wraps the existing warn in markTerminalHoldLogged; test extended, none removed) so lossless vs v5. changes: none to the branch — the workflow triggers on unlabeled, so removing needs-human re-runs the policy check without a push. tests: CI on head (test agent/hub/dashboard/scheduler/rest shards, coverage, patch-coverage, golangci-lint all green); native-queue-policy unit step passed 'fail 0'. remaining risk: none beyond the label-removal rerun.

Evidence bundle `#16201b9a60e2` sha256:b28251f39c1a90aab9af02f013502b6fa8e547d312a6abeb2d3c0eab5f2071bf — download from the dashboard or `hivectl review evidence #16201`.

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dedupe works for the WARN, but it now swallows the automerge hold line the sweep loop was supposed to emit once per head for this same hold.

Mechanism (read at 19e3fde):

  • automerge_held_runs.go:227 marks markTerminalHoldLogged(owner, repo, number, headSHA, workflowApprovalWorkflowMissingPrefix+contextName) and returns that same string as the hold reason (:231).
  • workflowApprovalWorkflowMissingPrefix = "workflow-approval-required:missing-workflow:" (:81), which has prefix WorkflowApprovalRequiredPrefix (prstage/reasons.go:84), so isTerminalRequiredCheckHold(reason) is true (automerge_held_runs.go:589-590).
  • The sweep loop then calls markTerminalHoldLogged(owner, repoName, number, pr.GetHead().GetSHA(), reason) at automerge_sweep_loop.go:325 with the identical (repo#pr@sha@reason) key (automerge_held_runs.go:675). It was claimed one frame earlier, so it returns false and c.info("automerge hold", …) at :326 is never logged for a missing-workflow hold — not on the first tick, not ever.

Consequence: the operator reference (src/docs/operator-reference/README.md:431-433) promises workflow-approval holds are "logged once per PR head as automerge hold"; after this change that line disappears for the missing-workflow variant and only the WARN remains. Low severity (the WARN carries the same fields), but it's a silent contract change on an operator-facing log line that the PR body does not mention.

Suggested fix: key the WARN dedupe under a distinct reason so the two call sites don't share an entry, e.g. "ok-to-test-grace-warned:"+contextName. The new assertion in automerge_held_runs_test.go:421-426 can't observe this because it drives trySweepSelfAuthoredPR directly rather than the sweep loop; a sweepSelfAuthored-level assertion that automerge hold appears exactly once would catch the regression.

Everything else I checked looks right: the key includes the lowercased head SHA, so a new push re-arms the WARN as intended, and the returned hold reason is unchanged so stage routing (automerge_stage_human.go:606,700) is unaffected.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/060-reviewed Review swarm verdict approve; a gate is still pending (CI running, soak not started) labels Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dedupe works as described — one WARN per head — but it borrows the key the sweep loop already uses, so it silently suppresses a second log line.

Low: the "automerge hold" INFO line is lost for this reason class.

  • src/pkg/github/automerge/automerge_held_runs.go:227 marks markTerminalHoldLogged(owner, repo, number, headSHA, "workflow-approval-required:missing-workflow:<ctx>") and returns that same string as the hold reason.
  • automerge_held_runs.go:589 — isTerminalRequiredCheckHold is true for anything with prefix workflow-approval-required: (pkg/stageengine/prstage/reasons.go:84), so the sweep loop takes the terminal-hold branch.
  • automerge_sweep_loop.go:325-326 then calls markTerminalHoldLogged(owner, repoName, number, headSHA, reason) with the identical key (automerge_held_runs.go:675 builds it from exactly those inputs). It is now already set, so the c.info("automerge hold", …) line never fires for that head.

Before this PR the per-head automerge hold record existed for missing-workflow holds (alongside the repeated WARN); after it, only the one WARN exists. Anyone correlating holds via the automerge hold line loses this class. The test added at automerge_held_runs_test.go:424 only counts the WARN, so it does not catch this.

Fix is one token: namespace the warn key, e.g. "grace-warn:"+workflowApprovalWorkflowMissingPrefix+contextName, so the two call sites stop sharing state. Optionally assert in the test that automerge hold still appears once (that goes through trySweepSelfAuthoredPR → sweep loop, so it may need the loop-level test instead).

Everything else checked out: markTerminalHoldLogged is mutex-guarded and per-head, the returned reason is unchanged, and the test logger injection (c.logger = slog.New(...)) matches the pattern at automerge_sweep_test.go:1117.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks correct to me. Read the change against the tree at a1ea5a8:

  • src/pkg/github/automerge/automerge_held_runs.go:231 dedupes the post-grace WARN through markTerminalHoldLogged with key workflow-approval-grace-warned:<context>. That key is distinct from the returned reason workflow-approval-required:missing-workflow:<context>, so the sweep loop's own once-per-head automerge hold line at automerge_sweep_loop.go:325 (keyed on the bare reason) is not pre-claimed — the new test asserts exactly that at automerge_held_runs_test.go:429.
  • markTerminalHoldLogged (automerge_held_runs.go:678) scopes the key by lowercase head SHA, so a new push re-arms the warning, which matches the "once per PR head" claim in the body and changelog.d/fixed-16199.md.
  • The third tick in TestTrySweepTrustedBotWorkflowApprovalMissingWorkflowAfterGrace re-enters the grace branch (now stays past workflowApprovalGrace), so the strings.Count(...) != 1 assertion would read 2 without the fix — the test fails without the change.

Hold reason and /ok-to-test posting behaviour are unchanged. No findings.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added the stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it label Nov 2, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the tree at b73ee19 plus the two other markTerminalHoldLogged call sites. Looks correct to me.

  • src/pkg/github/automerge/automerge_held_runs.go:231 dedupes on workflow-approval-grace-warned:<context>, while the returned hold reason stays workflow-approval-required:missing-workflow:<context>. Because markTerminalHoldLogged (automerge_held_runs.go:678) keys on head@reason, the sweep loop's own once-per-head "automerge hold" line at automerge_sweep_loop.go:325 is unaffected — the test asserts exactly this at automerge_held_runs_test.go:430.
  • Map access is under heldRunApprovalsMu; the new key grows terminalHoldsLogged by one entry per head, same lifecycle as the existing merge-train-skip:/merge-train-no-resolver keys (automerge_merge_train.go:699,869).
  • Scope matches #16199 (repeated unclassified_error_repeated WARN from this exact message); changelog fragment present.

The one red check, native-queue-policy, fails with #16201: head/base moved, draft, closed, or held — a queue gate reacting to the needs-human label, not this diff.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added the stage/stalled In its current stage longer than expected label Nov 2, 2026
hivecommons-hive Bot and others added 4 commits November 2, 2026 09:59
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Generated by changelog-fragment-guard from the PR title (#5675).
Edit the wording freely, or replace this file with the no-changelog
label if the change is not user-visible.

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: hivecommons-hive[bot] <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
The ok-to-test grace warning in workflowApprovalAttention used the same
terminalHoldLogged key as the hold reason it returns, so the sweep loop's
once-per-head "automerge hold" line never fired for missing-workflow holds
(documented in operator-reference/README.md). Use a dedicated key suffix and
assert in the test that the hold reason's key is still unclaimed.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer adjudication: DE-ESCALATE — evidence: the fix-loop escalation was gosec (insecure patterns) failing on b9042e6 with 'The self-hosted runner lost communication with the server' (shared-CI incident #16048, since fixed on v5 by #16045); on the subsequent head b73ee19 gosec/golangci-lint/govulncheck/DCO/gate/verify PR contents are all green and the only red check is native-queue-policy, which fails solely because the needs-human label matches its BLOCKED_LABEL regex (assertEligible: 'head/base moved, draft, closed, or held') — circular, not a code failure. Still wanted: v5 automerge_held_runs.go:227 still emits the undeduped grace warn and #16199 is open. Lossless: PR diff vs v5 is +24/-2 across 3 files, no code or tests removed; test count unchanged (assertions added to an existing test). changes: rebased scanner/fix-16199 onto current v5 (dropping the 4 merge commits, same 4 content commits, all DCO signed-off) and force-pushed; head is now b9a60e2. tests: go build ./pkg/github/automerge/ OK; test files type-checked via scratch-package go build OK (go test is blocked in-pod per #9845, CI is the verdict). remaining risk: none beyond CI re-running on the new head.

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed needs-human stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it stage/stalled In its current stage longer than expected labels Nov 2, 2026

@clubanderson clubanderson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Owner-requested, agent-assisted decision under @clubanderson's explicit delegation for this batch; this is not a separate independent human review. Reviewed the current diff and surrounding source at the exact commit below.

Inspected current head: b9a60e2d8b820d3bcb29530ac9997daca258137f.

Accept the once-per-head workflow-approval WARN dedupe. The head changed from b73ee19 during inspection; I stopped the old-head action, then verified the complete PR diff AND all three changed source files are byte-identical at this new head. The warning uses a distinct context-scoped key from the returned terminal hold reason, preserving the sweep hold log and approval requirement; the added third tick plus separate hold-log-key assertion regress the actual defect. No gate/permission/approval policy or retry behavior is relaxed. The old needs-human fix-loop label has already been resolved by the existing owner; I did not remove unrelated holds. Fresh exact-head CI is still progressing, so this is acceptance into normal queueing, not permission to merge a red or missing gate. Per AGENTS, no local Hive tests/builds were run.

@clubanderson

Copy link
Copy Markdown
Member

/ok-to-test
/lgtm
/approve

Owner-requested, agent-assisted acceptance at refreshed exact head b9a60e2d8b820d3bcb29530ac9997daca258137f. Old-head approval was deliberately not submitted. Normal fresh-head CI/merge policy still applies; no independent human review is claimed.

@kubestellar-prow kubestellar-prow Bot added lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. labels Nov 2, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

LGTM label has been added.

DetailsGit tree hash: 48c773ba585c26fb1dde1f6987be84d49a4dee9b

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: clubanderson

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2026
@hivecommons-hive
hivecommons-hive Bot merged commit 70e2eeb into v5 Nov 2, 2026
86 of 98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the scanner/fix-16199 branch November 2, 2026 15:09
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@hivecommons-hive hivecommons-hive Bot removed role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. reviewer-passed size/S Denotes a PR that changes 10-29 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[selfreport] unclassified_error_repeated: automerge workflow approval still required after ok-to-test grace; install the …

1 participant