Repository navigation
✨ React to failed CI checks via webhook instead of waiting for the next sweep - #15306
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply. |
|
@clubanderson this PR is held at Why:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
…xt sweep check_run/check_suite/status failure webhooks now resolve to open PRs in tracked repos, record a CI-failure signal (dedup per head SHA + check), and immediately kick the owning agent through the FIX-BEFORE-NEW red-CI path or nudge the contributor lane for a targeted automerge pass. The sweep still merges the signal store as fallback. Guarded by governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s, repo pause, agent pause, and governor eligibility. Fixes #15291 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
fb3802a to
e715931
Compare
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #15306 supersedes the hive-authored #15305 (same issue #15291, same new files, same config keys) so #15305 can be closed, or whether #15305 lands first and this PR is rebased onto it.
Conclusion: this PR and #15305 are competing implementations of #15291 and cannot both merge. Neither is wrong; one must be closed. I have not closed anything.
Overlap I verified (read both diffs in full):
- Both create
src/cmd/hive/ci_failure_kick.go+_test.go,changelog.d/added-ci-failure-webhook-kick.md, andsrc/pkg/dashboard/api_github_webhook_ci_failure_test.go— same paths, different content, so a merge of either makes the otherdirty. - Both add the same
GovernorConfigfields with the same YAML keys and defaults:ci_failure_webhook_kick(*bool, default on) andci_failure_kick_cooldown_s(DefaultCIFailureKickCooldownS = 600) —src/pkg/config/config.go:1571-1577in each. - Both parse the same three events (
check_run/check_suitecompleted withfailure|timed_out,statusfailure|error) and resolve the head SHA via the existing PR-detail cache only.
Where they differ (why this is a conflict, not a pure duplicate):
- Hook name:
Dependencies.CIFailure func(CIFailureSignal)(#15305) vsDependencies.CIFailureKick func(CIFailureSignal) CIFailureKickOutcome(this PR,src/pkg/dashboard/deps.go:304-309). - Status surface:
webhooks.ci_failures {count,last}(#15305) vsci_failure_signals {count,last,kicks,unmatched}(this PR). The operator docs in each describe only their own key. - Scope: #15305's body explicitly defers "folding pending CI failure signals into regular (cadence) kicks" as a follow-up. This PR implements that piece (
src/pkg/scheduler/policy_overlays.go—formatRedPRFixDataWithSignalsmerges the signal store into the FIX-BEFORE-NEW block, skipping checks already listed inBaseRedChecks) and also drains the sweep trigger in the legacy loop. So this PR is a superset; #15305 is alreadychanges_requestedat its current head.
Recommendation, if you agree: close #15305 in favour of this one, since this one covers the deferred part and the two share every operator-visible name.
One correctness note on this PR (low): src/cmd/hive/ci_failure_kick.go:139-141 claims the cooldown slot via AllowKick before the kick is delivered, and the spawned delivery at :156-160 does not give it back on error. If agentMgr is still nil at boot (k.send returns errNoAgentRuntime, :72-77) or the pane send fails, that PR head is silent to the webhook path for the full 600s; the sweep still catches it. #15305 handles this with an explicit release(key) after a failed Kick. Worth mirroring, but not blocking.
Routing (HiveAuthored → kicker, else SweepNudge) is decided at api_github_webhook_ci_failure.go:81-84 before the scanner default in the kicker applies, so a contributor PR is never kicked to an agent — the body's claim holds.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether this PR or #15305 is the one implementation of #15291 that lands; both add the same governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s keys and the same changelog.d/added-ci-failure-webhook-kick.md, so they cannot both merge.
Conflicts with #15305 (scanner), which implements the same issue. Both add changelog.d/added-ci-failure-webhook-kick.md, src/cmd/hive/ci_failure_kick.go, and the same CIFailureWebhookKick / CIFailureKickCooldownS fields and accessors in src/pkg/config/config.go; whichever merges second fails to apply. Details of the overlap and the differences (status key ci_failure_signals vs webhooks.ci_failures, 2h vs 24h retention, scanner default owner vs skip, governor eligibility vs fleet-breaker guard) are in my comment on #15305.
Read on its own: reactToCIFailure (src/pkg/dashboard/api_github_webhook_ci_failure.go) is nil-safe on cfg because CIFailureKickCooldown() / CIFailureWebhookKick() handle a nil receiver (config.go additions), and the agent kick is spawned off the handler goroutine with a 2-minute send timeout (ci_failure_kick.go, spawn/ciFailureKickSendTimeout), so the webhook response is not held. I found no defect. This one is the broader of the two (it also folds the signal into policy_overlays.go FIX-BEFORE-NEW and the sweep), which is why I suggested it as the survivor on #15305 — but that is a maintainer call.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Needs a decision, not a merge:
Head: verdict requires_human @e715931, 61/64 checks green, 3 pending, DCO yes, soak 0s of 30m0s. How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #15291
Mechanism
/api/webhook/githubnow parsescheck_run(completed; conclusionfailure/timed_out),check_suite(completed;failure/timed_out) andstatus(failure/error) events. The head SHA is resolved to an open PR via the existing PR-detail cache (one cached lookup, no newGET /pullsfan-out; an unknown PR is kept by number and left to the sweep). Each failure is recorded in an in-memoryCIFailureSignalStore{repo, number, headSHA, checkName, detailsURL, at}, deduplicated per head SHA + check, pruned after 2h.policy_overlays.go; the prompt carries the check name and log pointer. Contributor/human PR →SweepNudgeso the automerge/contributor lane (stage 072 / clanker rework) runs a targeted pass now instead of at the next ticker (legacy loop now drains the sweep trigger; forgestate mode already did). Base-red checks are excluded when merging signals into the work list; the sweep also merges the signal store so nothing is lost if a kick is skipped.governor.ci_failure_webhook_kick(default on), repo pause (project.paused_repos), agent disabled/paused/breaker, governor kick eligibility (AgentEligibleForCELKick+RecordKickForRepo), and a per-PR-head cooldowngovernor.ci_failure_kick_cooldown_s(default 600). Contributor nudges go through the same cooldown; the existing per-head comment idempotency inautomerge_contributor_nudge.gois unchanged, so a contributor PR is never commented more than once per head SHA by this path.Principle
Agents never watch or poll CI themselves. Detection is out-of-band — via webhook (this PR) or the governor sweep as fallback. The reviewer-lane prompt keeps its "Do Not Wait for CI" instruction, now asserted by
TestReviewerLanePromptKeepsDoNotWaitForCI.Surface
/api/statusgainsci_failure_signals{count, last, kicks, unmatched}.CI failed on #N (check) — kicked agent X/nudged contributor lane.check_run/check_suite/statusto hosted spokes (pkg/hub/webhook.go) — verified, no change needed.changelog.d/added-ci-failure-webhook-kick.md.Tests
Handler unit tests for parse/match/dedup/pause/flag/cooldown/unmatched, kicker gate tests, signal-merge tests, reviewer-prompt assertion. No
time.Sleep.go build ./... && go vetclean; touched packages pass with-race.