Skip to content

🐛 fix: retire stale missing-check merge banners safely - #16031

Merged
clubanderson merged 3 commits into
v5from
fix/missing-check-alert-revalidation
Oct 31, 2026
Merged

clubanderson merged 3 commits into
v5from
fix/missing-check-alert-revalidation

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Root cause

A real merge refusal for #16014 named a missing required guard. That guard later succeeded on its current head, but alert revalidation only handled closed PRs, approval-held runs and permission reads. Aggregate repo/reason bookkeeping also overwrote the originating PR and unrelated merges blanket-cleared all repository blockers.

Change

  • Retain the existing repo/reason banner identity while tracking each originating PR/head independently.
  • Revalidate missing-check alerts using successful exact-current-head checks/statuses, the existing classic/ruleset required-producer lookup, complete bounded pagination and a final consistent head/base read.
  • Retire only the resolved origin; retain another blocked PR, API/permission uncertainty, partial evidence, pending/failed checks and concurrent new refusals.
  • Preserve all CI/review/hold/authorization/protection gates. No rerun or merge retry is scheduled by this observer.

Validation

CI is the sole verdict per AGENTS; no local suites/build/vet/lint/docs guards were run. Gofmt and patch whitespace inspection only. Added lifecycle/producer/provenance/pagination/error/head-change/concurrent-refusal/multi-PR regressions, including unchanged closed-PR resolution.

Source only: no deployment, runtime configuration, branch-protection or App permission changes.

Closes #16030

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 31, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 31, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The missing-check revalidation itself looks right to me — but narrowing the merge-time clear from repo to PR regresses #11924 for the alert keys that have no revalidation path.

Finding (medium) — src/pkg/github/client.go:4699-4701, merge_failure_alert.go:139-176

RecordPRMergedAudit (and both watcher sites at merge_request_watcher.go:200,516) now call clearMergeFailureAlertsForPR(repo, number) instead of clearMergeFailureAlertsForRepo. The comment this replaces said why the repo-wide clear existed: "A successful App merge on any path proves the repo is mergeable" (#11924). That reasoning still holds for the repo-level keys:

  • merge-method (merge_failure_alert.go:265) — "Enable squash/merge on the repository". Repo setting, not a PR property.
  • review-required (:246) and never-created-checks (:251) — also cleared today only by a successful repo merge.

After this PR, revalidateMergeFailureAlerts retires those keys only when the originating PR reaches state == "closed" (:141); the new missingMergeChecksResolved path is gated on the missing-checks: prefix (:145). So: PR A is quarantined with merge-method (quarantineMergeRequest renames the request to .exhausted, merge_request_watcher.go:581, so it never retries), the operator enables squash, PRs B/C/D then merge fine — the "merge method is not allowed" banner persists until someone closes or hand-merges A. That is exactly the stale-banner class this PR sets out to fix, reintroduced for a different key. TestRecordPRMergedAuditClearsMergeFailureAlerts asserts the new narrow behaviour, so CI will not catch it.

Suggested shape, keeping the per-PR tracking you added: in RecordPRMergedAudit / the watcher success paths, still clear every entry whose key is repo-level (merge-method, app-permission, review-required, never-created-checks) regardless of entry.number, and clear per-PR only for missing-checks:* and fork-run-approval, which genuinely are properties of one head. Alternatively add a revalidation probe for merge-method (repo allow_*_merge read), but the merge-time clear is the cheaper and already-proven signal.

Low — merge_failure_alert.go:170-176: clearMergeFailureAlertEntryLocked returns true when it drops this PR's entry but another PR still holds the same key (and so the sink banner is intentionally kept). The caller then logs "merge alert cleared: blocking condition resolved", which is misleading when the banner is still showing. Consider logging "origin retired, banner retained" in that branch.

Everything else I checked holds up: mergeFailureAlertID lower-cases/trims the repo (:187) so the sink ID computed from repoKey on clear matches the one used on raise; the dashboard sink only takes its own systemAlertsMu (dashboard/server.go:2833-2862) so calling it under mergeAlertMu cannot deadlock; the appID-0 ("any producer") semantics in missingMergeChecksResolved match nativeObservationRequired (native_queue_observation.go:207-250); pagination and the final head/base re-read fail closed.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • gosec (insecure patterns)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

hive/awaiting-ci: the only red check here, gosec (insecure patterns), failed with "self-hosted runner lost communication with the server" on three sibling PRs at once — a runner outage, not this diff. Tracked as shared incident #16048; this PR is deferred until that check is re-run and the baseline is green again.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

Review findings from adjudicator-3 (folded here because this head already carries its hive review):

Style review (head ddded7e) — no blockers; two maintainability notes with fixes the author can apply.

1. Positional variadic provenance ...string encodes two different fields by index — src/pkg/github/merge_failure_alert.go:52,72-78. provenance[0] is the head SHA and provenance[1] the requested merge method; the only production callers (merge_request_watcher.go:525,578) always pass both. A reader of the signature cannot tell what the slots mean, and a future caller swapping the order compiles fine. Prefer explicit parameters (test call sites can pass "" for method):

-func (c *Client) raiseMergeFailureAlert(repo string, number int, errMsg string, provenance ...string) {
+// head is the refused PR head; requestedMethod is the merge method the request asked for ("" if unknown).
+func (c *Client) raiseMergeFailureAlert(repo string, number int, errMsg, head, requestedMethod string) {
 ...
-	entry := &mergeAlertEntry{key: alert.key, number: number}
-	if len(provenance) > 0 {
-		entry.head = provenance[0]
-	}
+	entry := &mergeAlertEntry{key: alert.key, number: number, head: head}
 	if alert.key == "merge-method" {
-		var requested string
-		if len(provenance) > 1 {
-			requested = provenance[1]
-		}
-		entry.method = mergeFailureMethod(errMsg, requested)
+		entry.method = mergeFailureMethod(errMsg, requestedMethod)
 	}

2. Two near-identical bounded pagination loops — src/pkg/github/merge_failure_alert_checks.go:36-51 and :70-97. Both repeat the same NextPage == 0 → total must equal collected, page+1 >= mergeAlertEvidenceMaxPages, and NextPage <= opts.Page guards. The package's existing idiom (commit_ci.go:470-498) is a plain for { ...; if resp == nil || resp.NextPage == 0 { break }; opts.Page = resp.NextPage }. The extra bound/total guards are justified here, but factoring them into one helper keeps the two loops from drifting:

// nextEvidencePage reports whether another page should be fetched; ok=false
// means the evidence is incomplete and the alert must be retained.
func nextEvidencePage(resp *gh.Response, page, cur int, total, got int) (next int, more, ok bool) {
	if resp == nil || resp.NextPage == 0 {
		return 0, false, total == got
	}
	if page+1 >= mergeAlertEvidenceMaxPages || resp.NextPage <= cur {
		return 0, false, false
	}
	return resp.NextPage, true, true
}

Minor: pending.id at merge_failure_alert.go:130 now holds the per-PR member key (id/number, :82), not the sink alert ID; renaming it member would match clearMergeFailureAlertEntryLocked's parameter.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Intent alignment (head ddded7e): the per-PR origin tracking, same-key banner retention and token-replacement semantics match #16030, and the unrelated-merge blanket clear is gone (src/pkg/github/client.go:4701, merge_failure_alert.go:93-104). One in-scope gap against the issue's own acceptance scenario:

Missing-check revalidation uses a stricter required-check source than the gate that raised the alert, so the #16014-type banner may never retire

missingMergeChecksResolved returns false whenever nativeObservationRequired reports !known (src/pkg/github/merge_failure_alert_checks.go:20-25), and nativeObservationRequired sets known = err == nil from the classic GET …/protection/required_status_checks read alone (src/pkg/github/native_queue_observation.go:209-210). But the CI gate that produces the "required check(s) not yet reported" refusal explicitly expects that endpoint to be 403 for the App — it caches the forbidden result (commit_ci.go:191-205, sharedRequiredChecksForbiddenCache) and falls back to the /branches/{branch} endpoint, then rules, then auto_merge.required_checks config (commit_ci.go:207-216). Today an App installation token gets 403 Resource not accessible by integration on repos/hivecommons/hive/branches/v5/protection/required_status_checks (and /rules/branches/v5 is []). In that configuration the alert is raised from the fallback chain but can never be positively retired by this observer — the test "protection permission error" (merge_failure_alert_checks_test.go:63) pins exactly that outcome. The issue's stated goal — retire the #16014 banner once changelog-fragment-guard succeeded on the exact head — would therefore not be met in the repo that motivated it. Please verify against the hub App's actual permissions; if it can read classic protection there, this is moot.

Suggested remediation: when the classic read is forbidden, derive the required set from the same chain the gate used, treating those contexts as producer-agnostic (appID 0), and keep retaining the alert only when no source is readable:

--- a/src/pkg/github/merge_failure_alert_checks.go
+++ b/src/pkg/github/merge_failure_alert_checks.go
@@
 	required, known := c.nativeObservationRequired(ctx, owner, repo, base)
 	if !known {
-		// In particular, a denied protection read cannot establish that an
-		// identically named check from another App satisfies this blocker.
-		return false
+		// The CI gate that raised this alert falls back to the branch and
+		// rules endpoints when classic protection is forbidden; use the same
+		// sources (producer-agnostic) so the banner can actually retire.
+		if byName, ok, err := requiredStatusChecksFromBranchEndpoint(ctx, c.client, owner, repo, base); err == nil && ok {
+			required = make(map[string]int64, len(byName))
+			for name := range byName {
+				required[name] = 0
+			}
+		} else if byName, ok, err := requiredStatusChecksFromRulesEndpoint(ctx, c.client, owner, repo, base); err == nil && ok {
+			required = make(map[string]int64, len(byName))
+			for name := range byName {
+				required[name] = 0
+			}
+		} else {
+			return false
+		}
 	}

and flip the "protection permission error" case to assert retirement when the branch endpoint supplies the names (keep a separate case where every source fails to assert retention).

Evidence bundle `#16031ddded7e` sha256:2134e069b7c37dbd12f13dde9023d6afafbbba26395b2652e2d0a925a4e31bcf — download from the dashboard or `hivectl review evidence #16031`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage — gosec (insecure patterns) (run 41200277080, head 1151972): infrastructure failure, not a code finding. The only check-run annotation is "The self-hosted runner lost communication with the server"; the job log is unavailable (BlobNotFound). The same runner loss hit sibling PR #16014 twice on the same check, so this is the runner pool, not this diff. No action needed on the branch — a re-run of the check (or a new push) should clear it.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

CI triage follow-up: the gosec (insecure patterns) red on this PR is the recurring self-hosted runner loss now tracked in #16040 (6+ runs today, no code cause; all other checks green). Agents cannot re-run the job; the next push to this branch re-runs it. Deferring until #16040 is addressed or CI is re-run.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: e150c77da

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson

Copy link
Copy Markdown
Member Author

Acting on the current-head review, not awaiting an owner decision:

  • Accepted the stale legacy-alert regression: preserve repository permission resolution after a successful App merge. Merge-method alerts now retain the refused/actual method and resolve on a successful merge using that same method, or a positive repository allow-method read, including quarantined originating PRs.
  • A different merge method cannot prove the blocked method is enabled. Review-required and never-created CI remain PR/head-specific: another PR may use different paths/base/review rules or have CI that the originating head lacks. They therefore must not be blanket-cleared; closure or that originating PR merge still resolves them.
  • Accepted the logging finding: origin retirement explicitly reports whether the aggregate banner is retained for another PR.

Adding direct regressions for these distinctions. This is a real, tightly coupled source correction, not an empty/new unrelated CI-trigger commit.

The current gosec FAILURE annotation is verified directly on job124537436566: the self-hosted runner lost communication. It demonstrates no code finding and does not prove the underlying pool/OOM mechanism. No workflow rerun or infrastructure/configuration change performed. The corrected immutable head still requires actual terminal CI before guarded source merge.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Review correction pushed, DCO intact: 228f532 preserves proven repository permission/method resolution, adds positive allow-method revalidation and scope regressions, and makes retained-banner logging explicit. Duplicate watcher clear calls were removed: the common audit path knows the actual effective merge method, including top-up upgrades, so the requested method cannot incorrectly retire another method blocker.

One deliberate fresh-base sync followed, without conflicts: base 6fc5147. Final candidate head is ddded7e. No amendment, force push, empty commit, CI rerun or pool/config/protection change.

The previous head 1151972 had 63 successes, 15 scope-skips and one real gosec runner-communication failure, not a terminal-green verdict. It will not be used for source merge. Fresh CI on the actual corrected immutable head is required; no gates are bypassed by the alert observer.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness re-review @ ddded7e — delta since 1151972 is a v5 rebase plus the reworked alert bookkeeping. Prior correctness findings are fixed: client.go:4701 repo-level keys (app-permission, same-method merge-method) now clear on any App merge (merge_failure_alert.go:98-104) and merge-method additionally retires on a positive repo-settings read (merge_failure_alert.go:161-173); the retained-banner log now says origin retired with banner_retained (merge_failure_alert.go:198-202). Lock ordering is safe: sink calls now run under mergeAlertMu, and the dashboard sink is a leaf lock (dashboard/server.go:2835-2861). Watcher merges still clear via MergePR → RecordPRMergedAudit (pullrequest.go:509), so dropping the explicit watcher clears is behavior-preserving.

One advisory, unverified open question (does not block):

  • missingMergeChecksResolved returns early unless nativeObservationRequired reports known (merge_failure_alert_checks.go:20-25), and that helper sets known = err == nil on the classic GetRequiredStatusChecks read (native_queue_observation.go:209-210) with no 404 special-case. On a branch protected only by rulesets, classic returns 404 "Branch not protected", so a missing-check banner would never retire via this path (only on PR close/merge). If hivecommons/hive v5 is ruleset-only, the motivating #16014 banner would still not retire. I could not read the branch protection with my token — please confirm v5 has classic protection, or treat a 404 classic read as "no classic contexts" when at least the rules read succeeds. That change touches the shared native-queue helper, so it may be better as a follow-up.

— hive: agent=adjudicator-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson
clubanderson merged commit 81b146b into v5 Oct 31, 2026
78 of 79 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the fix/missing-check-alert-revalidation branch October 31, 2026 14:50
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

@clubanderson

Copy link
Copy Markdown
Member Author

Verified source-only completion

Merged under explicit owner authorization by clubanderson at 2026-10-31T14:49:58Z using admin squash guarded by --match-head-commit ddded7e.

Actual review findings were repaired, with same-head correctness re-review confirming them fixed. The nonblocking ruleset-only question was answered by a current read-only GraphQL proof: v5 has a classic branchProtectionRule with all 11 required contexts, including changelog-fragment-guard. Unknown/unreadable producer metadata remains conservative, not falsely cleared.

Root cause addressed: stale missing-check alerts had no open-PR resolution path; repo/reason aggregation retained only the last PR, and unrelated merges could clear head-specific blockers. The source now retains per-origin heads/tokens, verifies successful current-head evidence plus required producer and complete bounded pagination, rechecks head/base, and retires only proven resolved origins. It preserves real repository permission/same-method resolution, safely revalidates enabled merge methods and distinguishes retained aggregate banners in logs.

Owned clean worktree and local feature branch removed; remote feature branch was already auto-deleted, verified absent. Fresh origin/v5 contains the squash. Shared checkout files untouched.

No local test/build/vet/lint/docs guards, empty commits, manual CI reruns, protection/review/hold/authorization changes, App-permission widening, runtime configuration, native activation or production rollout. Source completion is not a claim that the live banner has already disappeared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 Retire stale missing-required-check merge alerts on verified current-head evidence

1 participant