Skip to content

✨ feat(webhook): react to failed CI checks within seconds of the delivery - #15305

Closed
hivecommons-hive[bot] wants to merge 2 commits into
v5from
scanner/fix-15291
Closed

hivecommons-hive[bot] wants to merge 2 commits into
v5from
scanner/fix-15291

Conversation

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Refs #15291 — folding pending CI failure signals into regular (cadence) kicks until ci-failing.json catches up is left for a follow-up: it needs the scheduler's FIX-BEFORE-NEW block (addRedPRFixFirst) to merge a second source, and the immediate targeted kick already delivers the same block. Everything else in the issue is implemented here.

Why

A failed check_run / check_suite / status webhook only invalidated the PR cache and scheduled a generic debounced wake. A red PR then waited for the automerge/stage sweep and the next kick's ci-failing.json block, which can take most of the 15-minute webhook-mode tick.

What

  • Signal: HandleWebhookEvent spots a failed check (check_run completed with failure/timed_out, check_suite failure/timed_out, status failure/error). It resolves the head SHA to open PRs from the payload's pull_requests and the existing PR detail cache only, with no new GitHub call; an unknown head is left to the sweep's dirty mark. It records {repo, number, head_sha, check, details_url, at} in a small in-memory store, deduplicated per PR + head + check, with 24h retention and capped. /api/status webhooks.ci_failures serves {count, last}. WebhookEventResult.ci_failures is additive.
  • Reaction (cmd/hive/ci_failure_kick.go, wired via the new dashboard.Dependencies.CIFailure hook, run off the handler goroutine):
    • Hive-authored PR (audit agent_pr_created attribution): an immediate targeted FIX-BEFORE-NEW kick to the owning agent. It reuses redPRFixKick, so it carries the check name, head and log pointer plus the "do not watch/poll CI" instruction.
    • Contributor/human PR: a targeted automerge pass now via the existing sweep nudge (ForgeState trigger, or a governor wake without the cache). That is the pass that runs the contributor-lane / stage 072 / rework routing.
    • Guards: governor.ci_failure_webhook_kick, project.paused_repos, the fleet breaker, per-agent pause and every-mode-paused cadence (agentAvailability). One reaction per PR head per cooldown. A skipped or undeliverable reaction does not use up the cooldown.
    • Activity (audit) line: CI failed on #N (check) - kicked agent X.
  • The reviewer-lane "Do Not Wait for CI" section is kept, and a test asserts its heading.

Config keys added

Key Default
governor.ci_failure_webhook_kick true (unset = on)
governor.ci_failure_kick_cooldown_s 600

Tests

  • pkg/github: failed check deliveries → signals from the cache (fork PR with empty pull_requests), dedupe, non-failures and unknown heads ignored, status block, retention.
  • pkg/dashboard: webhook → hook called once (redelivery deduped) → /api/status webhooks.ci_failures.
  • cmd/hive: owner kick content and activity line, cooldown per head (default and configured), contributor automerge pass, guards (disabled, paused repo, breaker, paused agent, undeliverable kick).
  • pkg/policies: reviewer-lane keeps "Do Not Wait for CI".

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 requested_by=@clubanderson

…very

A failed check_run, check_suite or status webhook only invalidated the PR
cache and scheduled a generic wake, so a red PR waited for the sweep and
the next kick's ci-failing.json block - up to the 15 minute webhook-mode
tick - before anyone reacted.

The webhook now records a CI failure signal per open PR the head resolves
to (PR detail cache only, deduplicated per head + check, served in
/api/status webhooks.ci_failures) and the governor reacts at once: a
targeted FIX-BEFORE-NEW kick to the owning agent of a hive-authored PR,
an automerge pass for any other PR. governor.ci_failure_webhook_kick and
governor.ci_failure_kick_cooldown_s control it; paused repos, the fleet
breaker and paused agents are respected.

Refs #15291

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: clubanderson <407614+clubanderson@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 29, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

hive/awaiting-ci: CI pending — sweep will merge when green.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

This PR conflicts with v5 itself. Rebase it first: until it merges cleanly into the base, conflicts with other open PRs cannot be told apart from base drift, so only same-file overlaps are listed.

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15309 🐛 fix: clear blocker ledger on ordinary issue closure same files src/pkg/github/webhook_cache.go
#15312 🐛 fix: clear blocker ledger entry when an issue closes same files src/pkg/github/webhook_cache.go

@hivecommons-hive hivecommons-hive Bot added author/hive This repository's own hive (the App or one of its agent lanes) authored this PR dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI is red for a reason in this diff: TestSettingsDialogReferencesEveryHiveYAMLLeaf (src/pkg/dashboard/settings_parity_test.go:33) fails on test (dashboard 4/4) and test (dashboard shuffle 3/3) with

hive.yaml settings are not referenced by the Settings UI and are not allowlisted:
        governor.ci_failure_kick_cooldown_s
        governor.ci_failure_webhook_kick

The two new Governor leaves (src/pkg/config/config.go, CIFailureWebhookKick / CIFailureKickCooldownS) are in hive.yaml.example but neither surfaced in the Settings dialog nor listed in src/pkg/dashboard/settings_parity_allowlist.txt (read at settings_parity_test.go:18). Either add them to the allowlist or wire them into the governor settings tab. coverage / test are downstream of that failure; the earlier Sign-off survives the squash reds are from a superseded run and pass on the current head.

The rest I read checks out:

  • src/pkg/dashboard/api_github_webhook.go calls s.deps.CIFailure(sig) per new signal; ciFailureFunc wraps React in go, so the handler goroutine is not held while auditPRAgents walks the audit log.
  • ciFailureReactor.react (src/cmd/hive/ci_failure_kick.go): the cooldown key is repo#pr@head, so several distinct failing checks on one head produce one kick per 600s, and the reservation is released on breaker / unavailable / kick-failed paths so a transient failure does not burn the window. Disabled and paused-repo checks come first.
  • webhookCIFailure (src/pkg/github/webhook_ci_failure.go) only accepts failure/timed_out conclusions and failure/error statuses; cancelled/neutral correctly do not kick.

One small thing, not blocking: ciFailureStore.pruneLocked deletes expired signals but never clears s.last, so after 24h /api/status webhooks.ci_failures can report count: 0 with a stale last. Harmless for routing; worth a one-line reset if you touch the file again.

Body says Refs #15291 with the cadence-kick fold-in explicitly left for a follow-up — that matches the diff.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author
5a7fde8

ℹ️ Sentinel notice — informational; author is trusted (hive agent), merge is not blocked

Hive flagged this PR (author @hivecommons-hive[bot], head 5a7fde8b2aec) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • src/pkg/policies/reviewer_lane_ci_poll_test.go

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

…n prune

Add governor.ci_failure_webhook_kick and governor.ci_failure_kick_cooldown_s
to settings_parity_allowlist.txt so TestSettingsDialogReferencesEveryHiveYAMLLeaf
passes, and clear ciFailureStore.last once it ages past the retention so
/api/status webhooks.ci_failures never reports count 0 with a stale last.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
@github-actions github-actions Bot added the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot added stage/050-review-pending Review swarm requested; awaiting a verdict for the current head and removed stage/070-fixing Hive-authored PR: CI red, changes requested or needs rebase; a hive agent is fixing it labels Oct 29, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether #15305 (scanner) or #15306 (yours) is the one implementation of #15291 that lands; both add the same governor.ci_failure_webhook_kick / governor.ci_failure_kick_cooldown_s keys and the same changelog.d/added-ci-failure-webhook-kick.md, so they cannot both merge.

This appears to duplicate #15306, which implements the same issue (#15291) with a wider scope. I read both diffs in full.

What is the same (verified, not by file set):

  • Same signal: check_run completed failure|timed_out, check_suite failure|timed_out, status failure|error, head SHA resolved via the PR detail cache only — src/pkg/github/webhook_ci_failure.go (here) vs src/pkg/github/ci_failure_signal.go + src/pkg/dashboard/api_github_webhook_ci_failure.go (#15306).
  • Same config keys in src/pkg/config/config.go (CIFailureWebhookKick *bool, CIFailureKickCooldownS int, default 600) and the same accessor names CIFailureWebhookKick() / CIFailureKickCooldown().
  • Same routing: hive-authored PR → targeted FIX-BEFORE-NEW kick to the owning agent; other PRs → automerge/contributor sweep nudge; per-PR-head cooldown.
  • Identical new path changelog.d/added-ci-failure-webhook-kick.md and a new src/cmd/hive/ci_failure_kick.go in both — a textual conflict on merge of whichever goes second.

Where they differ (so these conflict, not merely overlap):

  • /api/status key: webhooks.ci_failures {count,last} here vs ci_failure_signals {count,last,kicks,unmatched} in #15306.
  • Retention 24h here (ciFailureRetention) vs 2h in #15306.
  • This PR skips a hive PR with no audit attribution; #15306 falls back to ciFailureKickDefaultAgent = "scanner".
  • This PR gates on the fleet breaker (ciFailureReactor.breaker) but not on governor kick eligibility; #15306 goes through AgentEligibleForCELKick + RecordKickForRepo (breaker covered via runtime pause).
  • This PR's body says folding the signal into regular cadence kicks via addRedPRFixFirst is left for a follow-up; #15306 does that in src/pkg/scheduler/policy_overlays.go and merges the store in automerge_sweep.go, and also updates api-reference.md / operator-reference.md, which this PR does not.

I found no defect in this PR's code on its own. The reason to prefer #15306 is purely scope: it closes #15291 (Fixes) where this one Refs it and defers the scheduler fold. If #15306 is the survivor, this one could be closed; if the narrower one is preferred, #15306 needs to drop its changelog/config duplicates first. I have not closed anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot added stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Oct 29, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Needs a decision, not a merge:

Head: verdict requires_human @644ecec, 62/63 checks green, 1 pending, DCO yes, soak 0s of 30m0s.

What to do: approve this head (644ecec) to merge as-is, or leave a changes requested review (or reply /hive decision <what to change>) and the hive will push the fix.

How to release it:

  • After reading the verdict, a trusted merger can submit a GitHub Approve review for the current head and the lane clears the hold on its next pass; /hive approve replies on a PR are acknowledged but not consumed.

Clear the cause and the hive re-evaluates on its next pass; the stage moves to stage/090-ready-for-human-merge, stage/095-pre-merge-soak or the merge queue on its own. Approval does not bypass CI, hold labels or other merge gates.

@hivecommons-hive

Copy link
Copy Markdown
Contributor Author

Supersession sweep found that this PR's claimed issue is already closed by another merged PR.

Closing this hive-authored PR as superseded by #15306: every issue it closes is already closed and that merged PR closed #15291. Head 644ecec conflicts with v5, so there is nothing of its own left to rebase and measure. Reopen if that's wrong.

This comment is edited in place by the supersession sweep; it is not duplicated.

@hivecommons-hive hivecommons-hive Bot added the hive/superseded Closed by Hive: a different merged PR already landed this change label Oct 29, 2026
@hivecommons-hive hivecommons-hive Bot closed this Oct 29, 2026
@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/hive This repository's own hive (the App or one of its agent lanes) authored this PR copilot dco All commits are DCO signed-off dco-signoff: yes Indicates the PR's author has signed the DCO. hive/superseded Closed by Hive: a different merged PR already landed this change role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. stage/085-needs-decision A human decision is owed: sentinel alert, hold/needs-decision label, human-only rule or parked issue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants