Skip to content

✨ React to failed CI checks via webhook instead of waiting for the next sweep - #15306

Merged
clubanderson merged 1 commit into
v5from
ci-webhook-kick
Oct 29, 2026
Merged

clubanderson merged 1 commit into
v5from
ci-webhook-kick

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #15291

Mechanism

  1. Webhook → signal. /api/webhook/github now parses check_run (completed; conclusion failure/timed_out), check_suite (completed; failure/timed_out) and status (failure/error) events. The head SHA is resolved to an open PR via the existing PR-detail cache (one cached lookup, no new GET /pulls fan-out; an unknown PR is kept by number and left to the sweep). Each failure is recorded in an in-memory CIFailureSignalStore {repo, number, headSHA, checkName, detailsURL, at}, deduplicated per head SHA + check, pruned after 2h.
  2. Signal → reaction. Hive-authored PR → immediate targeted kick of the owning agent through the existing FIX-BEFORE-NEW red-CI work-list path in policy_overlays.go; the prompt carries the check name and log pointer. Contributor/human PR → SweepNudge so the automerge/contributor lane (stage 072 / clanker rework) runs a targeted pass now instead of at the next ticker (legacy loop now drains the sweep trigger; forgestate mode already did). Base-red checks are excluded when merging signals into the work list; the sweep also merges the signal store so nothing is lost if a kick is skipped.
  3. Guards. governor.ci_failure_webhook_kick (default on), repo pause (project.paused_repos), agent disabled/paused/breaker, governor kick eligibility (AgentEligibleForCELKick + RecordKickForRepo), and a per-PR-head cooldown governor.ci_failure_kick_cooldown_s (default 600). Contributor nudges go through the same cooldown; the existing per-head comment idempotency in automerge_contributor_nudge.go is unchanged, so a contributor PR is never commented more than once per head SHA by this path.

Principle

Agents never watch or poll CI themselves. Detection is out-of-band — via webhook (this PR) or the governor sweep as fallback. The reviewer-lane prompt keeps its "Do Not Wait for CI" instruction, now asserted by TestReviewerLanePromptKeepsDoNotWaitForCI.

Surface

  • /api/status gains ci_failure_signals {count, last, kicks, unmatched}.
  • Activity log line: CI failed on #N (check) — kicked agent X / nudged contributor lane.
  • Hub relay already forwards check_run/check_suite/status to hosted spokes (pkg/hub/webhook.go) — verified, no change needed.
  • Docs: api-reference, operator-reference, webhooks; changelog fragment changelog.d/added-ci-failure-webhook-kick.md.

Tests

Handler unit tests for parse/match/dedup/pause/flag/cooldown/unmatched, kicker gate tests, signal-merge tests, reviewer-prompt assertion. No time.Sleep. go build ./... && go vet clean; touched packages pass with -race.

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 29, 2026
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15305 ✨ feat(webhook): react to failed CI checks within seconds of the delivery CONFLICT changelog.d/added-ci-failure-webhook-kick.md
src/cmd/hive/ci_failure_kick.go
src/cmd/hive/ci_failure_kick_test.go
src/docs/webhooks.md
src/pkg/config/config.go
src/pkg/dashboard/api_github_webhook_ci_failure_test.go
src/pkg/github/webhook_cache.go
#15309 🐛 fix: clear blocker ledger on ordinary issue closure same files src/pkg/github/webhook_cache.go
#15294 🐛 fix(writeback): gate adoption close-first writes instead of baselining them same files src/docs/operator-reference.md
#15301 🐛 Let hive decide: close already-fixed PRs instead of adopting them same files src/docs/api-reference.md
#15303 🌱 refactor: split API reference into area files same files src/docs/api-reference.md
#15304 🌱 docs: split api-reference into per-area files CONFLICT src/docs/api-reference.md
#15308 ✨ feature: enqueue eligible PRs into GitHub native merge queues same files src/pkg/github/automerge/automerge_sweep.go

@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson is auto-promoted for contributor-lane merging on this PR: the last 5 merged PRs by this author in this hive's repositories have no revert or sentinel-alert evidence. CI, reviewer verdict, diff fences, soak, holds and branch protection still apply.

@hivecommons-hive hivecommons-hive Bot added author/human A person pushed this PR themselves dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Why:

  • the branch conflicts with the base and needs a rebase

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

…xt sweep

check_run/check_suite/status failure webhooks now resolve to open PRs in
tracked repos, record a CI-failure signal (dedup per head SHA + check),
and immediately kick the owning agent through the FIX-BEFORE-NEW red-CI
path or nudge the contributor lane for a targeted automerge pass. The
sweep still merges the signal store as fallback. Guarded by
governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s,
repo pause, agent pause, and governor eligibility.

Fixes #15291

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/071-needs-author Human-authored PR: CI red, changes requested or conflicting; diagnosis posted, the author must act labels Oct 29, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #15306 supersedes the hive-authored #15305 (same issue #15291, same new files, same config keys) so #15305 can be closed, or whether #15305 lands first and this PR is rebased onto it.

Conclusion: this PR and #15305 are competing implementations of #15291 and cannot both merge. Neither is wrong; one must be closed. I have not closed anything.

Overlap I verified (read both diffs in full):

  • Both create src/cmd/hive/ci_failure_kick.go + _test.go, changelog.d/added-ci-failure-webhook-kick.md, and src/pkg/dashboard/api_github_webhook_ci_failure_test.go — same paths, different content, so a merge of either makes the other dirty.
  • Both add the same GovernorConfig fields with the same YAML keys and defaults: ci_failure_webhook_kick (*bool, default on) and ci_failure_kick_cooldown_s (DefaultCIFailureKickCooldownS = 600) — src/pkg/config/config.go:1571-1577 in each.
  • Both parse the same three events (check_run/check_suite completed with failure|timed_out, status failure|error) and resolve the head SHA via the existing PR-detail cache only.

Where they differ (why this is a conflict, not a pure duplicate):

  • Hook name: Dependencies.CIFailure func(CIFailureSignal) (#15305) vs Dependencies.CIFailureKick func(CIFailureSignal) CIFailureKickOutcome (this PR, src/pkg/dashboard/deps.go:304-309).
  • Status surface: webhooks.ci_failures {count,last} (#15305) vs ci_failure_signals {count,last,kicks,unmatched} (this PR). The operator docs in each describe only their own key.
  • Scope: #15305's body explicitly defers "folding pending CI failure signals into regular (cadence) kicks" as a follow-up. This PR implements that piece (src/pkg/scheduler/policy_overlays.go — formatRedPRFixDataWithSignals merges the signal store into the FIX-BEFORE-NEW block, skipping checks already listed in BaseRedChecks) and also drains the sweep trigger in the legacy loop. So this PR is a superset; #15305 is already changes_requested at its current head.

Recommendation, if you agree: close #15305 in favour of this one, since this one covers the deferred part and the two share every operator-visible name.

One correctness note on this PR (low): src/cmd/hive/ci_failure_kick.go:139-141 claims the cooldown slot via AllowKick before the kick is delivered, and the spawned delivery at :156-160 does not give it back on error. If agentMgr is still nil at boot (k.send returns errNoAgentRuntime, :72-77) or the pane send fails, that PR head is silent to the webhook path for the full 600s; the sweep still catches it. #15305 handles this with an explicit release(key) after a failed Kick. Worth mirroring, but not blocking.

Routing (HiveAuthored → kicker, else SweepNudge) is decided at api_github_webhook_ci_failure.go:81-84 before the scanner default in the kicker applies, so a contributor PR is never kicked to an agent — the body's claim holds.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether this PR or #15305 is the one implementation of #15291 that lands; both add the same governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s keys and the same changelog.d/added-ci-failure-webhook-kick.md, so they cannot both merge.

Conflicts with #15305 (scanner), which implements the same issue. Both add changelog.d/added-ci-failure-webhook-kick.md, src/cmd/hive/ci_failure_kick.go, and the same CIFailureWebhookKick / CIFailureKickCooldownS fields and accessors in src/pkg/config/config.go; whichever merges second fails to apply. Details of the overlap and the differences (status key ci_failure_signals vs webhooks.ci_failures, 2h vs 24h retention, scanner default owner vs skip, governor eligibility vs fleet-breaker guard) are in my comment on #15305.

Read on its own: reactToCIFailure (src/pkg/dashboard/api_github_webhook_ci_failure.go) is nil-safe on cfg because CIFailureKickCooldown() / CIFailureWebhookKick() handle a nil receiver (config.go additions), and the agent kick is spawned off the handler goroutine with a 2-minute send timeout (ci_failure_kick.go, spawn/ciFailureKickSendTimeout), so the webhook response is not held. I found no defect. This one is the broader of the two (it also folds the signal into policy_overlays.go FIX-BEFORE-NEW and the sweep), which is why I suggested it as the survivor on #15305 — but that is a maintainer call.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Needs a decision, not a merge:

Head: verdict requires_human @e715931, 61/64 checks green, 3 pending, DCO yes, soak 0s of 30m0s.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed. An ordinary contributor cannot release their own PR.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@clubanderson
clubanderson merged commit 7419e7b into v5 Oct 29, 2026
82 of 85 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the ci-webhook-kick branch October 29, 2026 05:24
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves conflicts-with-open-pr Open PR has a merge conflict with another open PR dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

React to failed CI checks via webhook instead of waiting for the next sweep

1 participant