Repository navigation
✨ feat(webhook): react to failed CI checks within seconds of the delivery - #15305
hivecommons-hive[bot] wants to merge 2 commits into
Conversation
…very A failed check_run, check_suite or status webhook only invalidated the PR cache and scheduled a generic wake, so a red PR waited for the sweep and the next kick's ci-failing.json block - up to the 15 minute webhook-mode tick - before anyone reacted. The webhook now records a CI failure signal per open PR the head resolves to (PR detail cache only, deduplicated per head + check, served in /api/status webhooks.ci_failures) and the governor reacts at once: a targeted FIX-BEFORE-NEW kick to the owning agent of a hive-authored PR, an automerge pass for any other PR. governor.ci_failure_webhook_kick and governor.ci_failure_kick_cooldown_s control it; paused repos, the fleet breaker and paused agents are respected. Refs #15291 Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
hive/awaiting-ci: CI pending — sweep will merge when green. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
Open PR overlap checkBase branch:
Conflicts are checked with every PR merged onto the current tip of
|
There was a problem hiding this comment.
CI is red for a reason in this diff: TestSettingsDialogReferencesEveryHiveYAMLLeaf (src/pkg/dashboard/settings_parity_test.go:33) fails on test (dashboard 4/4) and test (dashboard shuffle 3/3) with
hive.yaml settings are not referenced by the Settings UI and are not allowlisted:
governor.ci_failure_kick_cooldown_s
governor.ci_failure_webhook_kick
The two new Governor leaves (src/pkg/config/config.go, CIFailureWebhookKick / CIFailureKickCooldownS) are in hive.yaml.example but neither surfaced in the Settings dialog nor listed in src/pkg/dashboard/settings_parity_allowlist.txt (read at settings_parity_test.go:18). Either add them to the allowlist or wire them into the governor settings tab. coverage / test are downstream of that failure; the earlier Sign-off survives the squash reds are from a superseded run and pass on the current head.
The rest I read checks out:
src/pkg/dashboard/api_github_webhook.gocallss.deps.CIFailure(sig)per new signal;ciFailureFuncwrapsReactingo, so the handler goroutine is not held whileauditPRAgentswalks the audit log.ciFailureReactor.react(src/cmd/hive/ci_failure_kick.go): the cooldown key isrepo#pr@head, so several distinct failing checks on one head produce one kick per 600s, and the reservation is released on breaker / unavailable / kick-failed paths so a transient failure does not burn the window. Disabled and paused-repo checks come first.webhookCIFailure(src/pkg/github/webhook_ci_failure.go) only acceptsfailure/timed_outconclusions andfailure/errorstatuses;cancelled/neutralcorrectly do not kick.
One small thing, not blocking: ciFailureStore.pruneLocked deletes expired signals but never clears s.last, so after 24h /api/status webhooks.ci_failures can report count: 0 with a stale last. Harmless for routing; worth a one-line reset if you touch the file again.
Body says Refs #15291 with the cadence-kick fold-in explicitly left for a follow-up — that matches the diff.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
5a7fde8
ℹ️ Sentinel notice — informational; author is trusted (hive agent), merge is not blockedHive flagged this PR (author @hivecommons-hive[bot], head
Hive did not add the |
…n prune Add governor.ci_failure_webhook_kick and governor.ci_failure_kick_cooldown_s to settings_parity_allowlist.txt so TestSettingsDialogReferencesEveryHiveYAMLLeaf passes, and clear ciFailureStore.last once it ages past the retention so /api/status webhooks.ci_failures never reports count 0 with a stale last. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether #15305 (scanner) or #15306 (yours) is the one implementation of #15291 that lands; both add the same governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s keys and the same changelog.d/added-ci-failure-webhook-kick.md, so they cannot both merge.
This appears to duplicate #15306, which implements the same issue (#15291) with a wider scope. I read both diffs in full.
What is the same (verified, not by file set):
- Same signal:
check_runcompletedfailure|timed_out,check_suitefailure|timed_out,statusfailure|error, head SHA resolved via the PR detail cache only —src/pkg/github/webhook_ci_failure.go(here) vssrc/pkg/github/ci_failure_signal.go+src/pkg/dashboard/api_github_webhook_ci_failure.go(#15306). - Same config keys in
src/pkg/config/config.go(CIFailureWebhookKick *bool,CIFailureKickCooldownS int, default 600) and the same accessor namesCIFailureWebhookKick()/CIFailureKickCooldown(). - Same routing: hive-authored PR → targeted FIX-BEFORE-NEW kick to the owning agent; other PRs → automerge/contributor sweep nudge; per-PR-head cooldown.
- Identical new path
changelog.d/added-ci-failure-webhook-kick.mdand a newsrc/cmd/hive/ci_failure_kick.goin both — a textual conflict on merge of whichever goes second.
Where they differ (so these conflict, not merely overlap):
/api/statuskey:webhooks.ci_failures {count,last}here vsci_failure_signals {count,last,kicks,unmatched}in #15306.- Retention 24h here (
ciFailureRetention) vs 2h in #15306. - This PR skips a hive PR with no audit attribution; #15306 falls back to
ciFailureKickDefaultAgent = "scanner". - This PR gates on the fleet breaker (
ciFailureReactor.breaker) but not on governor kick eligibility; #15306 goes throughAgentEligibleForCELKick+RecordKickForRepo(breaker covered via runtime pause). - This PR's body says folding the signal into regular cadence kicks via
addRedPRFixFirstis left for a follow-up; #15306 does that insrc/pkg/scheduler/policy_overlays.goand merges the store inautomerge_sweep.go, and also updatesapi-reference.md/operator-reference.md, which this PR does not.
I found no defect in this PR's code on its own. The reason to prefer #15306 is purely scope: it closes #15291 (Fixes) where this one Refs it and defers the scheduler fold. If #15306 is the survivor, this one could be closed; if the narrower one is preferred, #15306 needs to drop its changelog/config duplicates first. I have not closed anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Needs a decision, not a merge:
Head: verdict requires_human @644ecec, 62/63 checks green, 1 pending, DCO yes, soak 0s of 30m0s. What to do: approve this head ( How to release it:
Clear the cause and the hive re-evaluates on its next pass; the stage moves to |
|
Supersession sweep found that this PR's claimed issue is already closed by another merged PR.
Closing this hive-authored PR as superseded by #15306: every issue it closes is already closed and that merged PR closed #15291. Head 644ecec conflicts with This comment is edited in place by the supersession sweep; it is not duplicated. |
Refs #15291 — folding pending CI failure signals into regular (cadence) kicks until
ci-failing.jsoncatches up is left for a follow-up: it needs the scheduler's FIX-BEFORE-NEW block (addRedPRFixFirst) to merge a second source, and the immediate targeted kick already delivers the same block. Everything else in the issue is implemented here.Why
A failed
check_run/check_suite/statuswebhook only invalidated the PR cache and scheduled a generic debounced wake. A red PR then waited for the automerge/stage sweep and the next kick'sci-failing.jsonblock, which can take most of the 15-minute webhook-mode tick.What
HandleWebhookEventspots a failed check (check_runcompleted withfailure/timed_out,check_suitefailure/timed_out,statusfailure/error). It resolves the head SHA to open PRs from the payload'spull_requestsand the existing PR detail cache only, with no new GitHub call; an unknown head is left to the sweep's dirty mark. It records{repo, number, head_sha, check, details_url, at}in a small in-memory store, deduplicated per PR + head + check, with 24h retention and capped./api/statuswebhooks.ci_failuresserves{count, last}.WebhookEventResult.ci_failuresis additive.cmd/hive/ci_failure_kick.go, wired via the newdashboard.Dependencies.CIFailurehook, run off the handler goroutine):agent_pr_createdattribution): an immediate targeted FIX-BEFORE-NEW kick to the owning agent. It reusesredPRFixKick, so it carries the check name, head and log pointer plus the "do not watch/poll CI" instruction.governor.ci_failure_webhook_kick,project.paused_repos, the fleet breaker, per-agent pause and every-mode-paused cadence (agentAvailability). One reaction per PR head per cooldown. A skipped or undeliverable reaction does not use up the cooldown.CI failed on #N (check) - kicked agent X.Config keys added
governor.ci_failure_webhook_kicktrue(unset = on)governor.ci_failure_kick_cooldown_s600Tests
pkg/github: failed check deliveries → signals from the cache (fork PR with emptypull_requests), dedupe, non-failures and unknown heads ignored, status block, retention.pkg/dashboard: webhook → hook called once (redelivery deduped) →/api/statuswebhooks.ci_failures.cmd/hive: owner kick content and activity line, cooldown per head (default and configured), contributor automerge pass, guards (disabled, paused repo, breaker, paused agent, undeliverable kick).pkg/policies: reviewer-lane keeps "Do Not Wait for CI".— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson