Repository navigation
✨ celtrigger: ci.failed event for default-branch CI failures - #17572
Conversation
Add the forge-neutral CEL event kind ci.failed. The existing red-main sweep (every 10 minutes, every active repo's default branch tip) now reports each check whose newest completed run concluded failure or timed_out, once per (repo, check, head SHA), and the event kicks the matching agents through celTriggerKickAgents with the usual gates. - Every check counts, not only required ones; cancelled, skipped, neutral, action_required and stale do not; a pull request's run on the same commit is excluded by its check suite's head branch. - Only rules whose expression names ci.failed are evaluated for it (Engine.ForKind), so existing rules keep their meaning, and with no such rule the sweep makes no extra reads. - Dedupe is in memory; a restart may re-report a still-red SHA once. - Event triggers editor: Default-branch CI failed preset and help; the triggers API accepts ci.failed as a one-repository preset kind. - Docs: cel-triggers.md, v6-controls.md, API reference. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
A failing pull request run on the default-branch tip was skipped but never recorded, so every red-main pass re-read the check suites for that SHA (TestSweepRedMainCIFailedOncePerSHA caught it). Record every decided run, keyed by head SHA and check suite, so a default-branch run of the same check on that SHA is still reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
There was a problem hiding this comment.
Read the sweep and its callers at the PR head, not just the diff. This looks correct to me.
What I verified
- No-op without a rule.
src/pkg/github/red_main.go:145resolvesciFailedEmitonce per pass;ciFailedEmitter()(red_main_ci_failed.go:76-87) returns nil unless both hooks are set andwant()is true. With nil, the!rules.Known && ciFailedEmit == nilguard atred_main.go:164is the pre-existing skip, andobserveCIFailedreturns onemit == nilatred_main_ci_failed.go:115before any read. The sweep's control flow for existing hives is unchanged. - The finpilot gap is actually closed. For a repo with
rules.Known == falseand a rule present, the tip's runs are now read (red_main.go:168), handed toobserveCIFailed, and only then does the sweepcontinue(red_main.go:175-179) — the breaker still never evaluates an unknown-rules repo. - Dedupe ordering. The seen map is written only after
ciFailedSuiteBranchessucceeds (red_main_ci_failed.go:148-153); on a suite-read error the function returns before recording, so the next pass retries, as the body says. BecauseredCheckRunsForRefkeeps the newest run per name (red_main.go:225), a default-branch run landing after a PR run on the same SHA has a different suite ID and is still reported — the (SHA, suite) value atred_main_ci_failed.go:100is what makes that work. - Lock discipline.
redMainMuis released beforeemitand before the suite-list API call;emitruns synchronously without client locks held, matching theSetCIFailedHookcontract. - Wiring.
ForKindandLenare both nil-safe (celtrigger.go:285-299, 303), socelEngineFor(...).ForKind(...).Len()inciFailedTriggerWantedis safe when no triggers are configured. The hook is installed inapplyGitHubClientConfigHooks(github_client_config.go:179) alongside the other config-reading hooks, so a rebuilt client keeps it. - The body's claims match the diff; the ten tests it names exist.
One thing blocking merge: the only non-green check is PR Issue (in progress). The body has no Closes #N / Refs #N line — if there is no tracking issue, one may need to be opened and referenced for that gate to pass.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
/lgtm |
|
@Danathar: you cannot LGTM your own PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Danathar The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@Danathar this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (3 waiting): fixes are handed out before new work as soon as you have capacity. |
There was a problem hiding this comment.
Read the sweep (src/pkg/github/red_main.go:146-180), observeCIFailed (red_main_ci_failed.go), latestCheckRunsByNameAndApp (commit_ci.go:1107-1132), the boot wiring (celwire_ci_failed.go, github_client_config.go:179) and the tests. The hook is consulted once per pass, observeCIFailed runs before the !rules.Known skip so repos without required checks are covered only when a rule wants the kind, the lock is released before emit, and a failed suite read records nothing. ForKind keeps issue/PR-only rules from firing on CI. Looks correct to me; one narrow gap worth knowing about:
Low — same check name in a PR suite and a push suite on one SHA collapses to one run. redCheckRunsForRef keys red[name] on the winner of latestCheckRunsByNameAndApp (red_main.go:243,254; identity is name+app+head, commit_ci.go:1101-1105). So when a rebase/fast-forward merge puts the PR head SHA on the default branch and a workflow runs on both pull_request and push under the same check name, runs holds only the newer of the two. If that is the PR-suite run, observeCIFailed records (sha, prSuite) and skips (red_main_ci_failed.go:146-157), and the default-branch failure of that check on that SHA is never reported — the body's "a default-branch run of the same check on that SHA is still reported" holds only when the push run is the newer one. TestSweepRedMainCIFailedOncePerSHA uses a distinct name (pr-check) for the PR suite, so this path is untested. Squash merges (a fresh SHA) are unaffected, and the next push re-arms it, so this is a bounded miss rather than a false fire. A one-line note in cel-triggers.md next to the restart caveat, or filtering by suite before the per-name collapse, would close it; either is fine to defer.
Everything else — docs, changelog fragment, preset allowlist, static/index.html parity — matches the diff.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. stage 080-awaiting-merge-admission → 072-returned-to-contributor: check-cancelled Review findings from Read the diff against
One design note, not a defect: Checks: everything green except Review findings from Read the diff against What I verified in the tree, not just the diff:
One non-blocking note (info): CI is green apart from |
|
Triage of the
Concrete fix: push any new head (e.g. 🐝 Hive Agent: — hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
…artial (#17604) * 🌱 ci: mutation report keeps its own time budget and says when it is partial A 20-minute runner kill of the advisory gremlins step wrote no comment.md, so XXL PRs kept the <!-- hive-mutation --> comment from an earlier head (#17596) or got none (#17572), and the text could not tell a clean run from a timeout. The run phase now bounds each package by the remaining budget, always writes the report, names the head SHA, and lists the packages it did not finish. test-mutation-report.sh pins the three shapes with a stubbed gremlins; TRUSTED_SCRIPTS_TREE re-pinned. Refs #17602 Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com> * 🌱 ci: mutation report review fixes — cite #17602, drop redundant total counter Fix review findings on #17604: the test header rationale now points at #17602 (the mutation-advisory timeout issue) rather than the unrelated #17601, and the package loop reuses `i` instead of a duplicate `total` counter that was always equal to it. Hive-Fix-Attempt: 1/6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com> * ci: re-pin TRUSTED_SCRIPTS_TREE to the v6-merged src/scripts tree The runtime-tests contract compares the pin against the merge ref's src/scripts tree, so the pin must follow v6, not the branch tip alone. Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> --------- Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com> Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com> Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
There was a problem hiding this comment.
Read the diff against the v6 merge-base (38fb7204) and the sweep it hooks into. Looks correct to me; the body's claims match the code.
What I checked:
- No-op without a rule, verified at both ends.
ciFailedEmitter()(red_main_ci_failed.go:74-82) returns nil unlesswant()is true, andwantisciFailedTriggerWanted→celEngineFor(...).ForKind("ci.failed").Len() > 0.celEngineForis signature-cached (celwire.go:78-81), so the once-per-sweep call does not recompile rules. With nil emit,red_main.go:164keeps the original!rules.Known → skipbehaviour andobserveCIFailedreturns at its first guard, so pre-existing sweeps make exactly the reads they did before. - Check-suite read only for new failures. The dedupe pass (
red_main_ci_failed.go:114-125) runs underredMainMubefore any network call;ciFailedSuiteBranchesis hit only for SHAs with a not-yet-seen failing run, cached per SHA within the pass, and a failed read records nothing so the next pass retries (:135-139). Theseenentry is written before the branch test (:142-147) so a PR run left on the tip does not re-read suites every pass — as the comment says. ForKindsemantics are safe. A rule likeevent.kind == "pr.opened"lacks the substring and is never evaluated for CI failures; a rule that namesci.failedbut then guards onevent.number > 0simply evaluates false.celTriggerKickAgents(celwire.go:282) has noNumber-based logic, soNumber: 0is harmless.- Breaker unchanged.
observeCIFailedis called before the second!rules.Knownskip (red_main.go:175-180), and the required-check loop below is untouched. - Preset gate in
api_event_triggers.go:25-31plus the JS/index.htmloption, docs (cel-triggers.md,v6-controls.md,configuration.md) and the changelog fragment are all present.
One narrow edge, info only, not a defect: redCheckRunsForRef keeps one run per check name/app via latestCheckRunsByNameAndApp (commit_ci.go:1131-1153). If a repository fast-forwards or rebase-merges so main's tip equals a PR head, and that PR's run of the same check is newer than the default-branch run (e.g. a re-run from the PR), the default-branch failing run is never in runs and cannot be reported. That matches the documented "a pull request's run on the same commit is not one" and the normal case (push-to-main run created after the PR run) is fine; mentioning it so the limitation is known.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the new code against the v6 tree (red_main.go sweep body, redCheckRunsForRef, latestCheckRunsByNameAndApp, celEngineFor, celTriggerKickAgents) rather than just the diff. This looks correct to me; the one thing worth a maintainer's glance is below.
What I verified
SweepRedMainordering (src/pkg/github/red_main.go:160-180): when no rule namesci.failed,ciFailedEmit == niland the!rules.Knownearly-continue is unchanged, so hives without such a rule make exactly the reads they made before. With a rule, the no-required-checks repo is read once, handed toobserveCIFailed, then still counted asSkipped— breaker behaviour is untouched. The test atred_main_ci_failed_test.go:152asserts that.- Locking (
red_main_ci_failed.go:120-131, 153-158):redMainMuis released beforeciFailedSuiteBranchesand beforeemit, andciFailedEmitterreleases it before callingwant()(which takesglobalCELEngine.muviacelEngineFor). No lock-order issue. - Dedupe key is (SHA, check-suite ID) (
red_main_ci_failed.go:98-100), so a rerun of the same check on the same SHA does not re-fire (same suite), while a default-branch suite on a SHA whose PR suite was skipped still does. Matches the body's claim. ForKind(celtrigger.go:285-299) is a substring test; a rule likeevent.kind != "ci.failed"would be included but then evaluates false on the event, so it is harmless. A rule written asevent.kind.startsWith("ci.")would not be picked up — the doc comment says "plain substring test", so this is disclosed, just noting it for the record.- No other kind allowlist exists in config/compile paths (grepped
KindStageCompletedusages), so nothing else needed registering. The dashboard preset allowlist (api_event_triggers.go:25-31) is the only one and is updated.
One observation (low)
ciFailedConclusion (src/pkg/github/red_main_ci_failed.go:87-89) accepts only failure and timed_out. checkConclusionIsRed (src/pkg/github/commit_ci.go:779-785) also treats startup_failure as red, and the new docs list the excluded conclusions as cancelled, skipped, neutral, action_required, stale — startup_failure is not mentioned either way. A workflow file with a syntax error merged to main produces startup_failure on every push, which is exactly the "main went red and nobody reacted" case this PR targets, and it will not fire ci.failed. Whether that is intended is a product call; if it is, the docs should say so, and if not, adding it to ciFailedConclusion is a one-line change plus a test case next to TestSweepRedMainCIFailedOncePerSHA.
Not a blocker. CI is green across all test shards; I have not approved or changed anything.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
There was a problem hiding this comment.
Read the sweep hook, the CEL wiring and their callers against the v6 tree (520d441) before the body; the code does what the body says and I found no defect. Looks correct to me.
What I verified:
- Breaker behaviour unchanged without a rule.
ciFailedEmitter()returns nil unlesswant()is true (src/pkg/github/red_main_ci_failed.go:72-83), and the only branch that depends on it isif !rules.Known && ciFailedEmit == nil(red_main.go:164), which is the pre-existing skip. With the hook nil the loop reads exactly what it read before — no extra GitHub calls. - With a rule, the breaker is still unchanged. The
!rules.Knownskip is re-applied right afterobserveCIFailed(red_main.go:176-180), sotracker.ObservePendingis never reached for a repo the breaker previously skipped. The ci.failed read is additive. - Conclusion filter is right for the stated purpose.
redCheckRunsForRefreturns everythingcheckConclusionIsRedaccepts, includingcancelled/action_required/stale(commit_ci.go:779-785);ciFailedConclusionnarrows tofailure/timed_out(red_main_ci_failed.go:88-90). Matches the body. - Dedupe and the suite-branch check. Seen key is
(repo, check) → sha\x00suiteID, recorded underredMainMuwhether or not the run turns out to be a default-branch suite, so a PR run left on the tip costs oneListCheckSuitesForRefper SHA, not one per pass. A failed suite read returns before recording, so it retries. Map reset at 5000 entries may re-report once, as the comment says. Theemitcall happens with no lock held, asSetCIFailedHookpromises. - Wiring.
SetCIFailedHook(b.ciFailedTriggerWanted, b.emitCIFailedTrigger)inapplyGitHubClientConfigHooks(github_client_config.go:182); both readb.cfg/b.agentMgr/b.govlazily, andcelEngineForis cached by triggers signature (celwire.go:68-80), so the once-per-sweepwant()is cheap.celTriggerKickAgentssignature matches;Engine.Len()is nil-safe, soForKind(...)returning nil is fine. - Dashboard preset gate widened via
eventTriggerPresetKindwith the same one-repo validation as the other two kinds; docs andchangelog.dfragment present.
One informational note, not a defect: Engine.ForKind is a substring test on the rule's expression text (celtrigger.go:285-298), as its doc comment says. A rule that selects the kind indirectly — event.kind.startsWith("ci."), or a kind list held in a CEL variable — will not be considered a ci.failed rule and the sweep stays a no-op for it. Given the goal (never fire an issue/PR rule on a CI event), that is the conservative side to err on, and src/docs/cel-triggers.md documents the literal-name requirement, so I'd leave it.
Checks are green apart from the advisory mutation job; tide is waiting on lgtm. I have not approved or merged anything.
— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88
Summary
Adds a CEL trigger event kind,
ci.failed, so an operator can write "kickci-maintainerwhen main goes red" (expr: event.kind == "ci.failed"). Right now an agent like ci-maintainer only notices a failure on a default branch at its next cadence slot (6 to 24h), and no CEL kind covers CI. The event comes from the red-main sweep the hive already runs every 10 minutes. It fires once per repository, check and head SHA, and only rules that nameci.failedare evaluated for it. Hives without such a rule see no change and make no extra GitHub reads.Why
Any hive whose repositories build or publish after merge has this gap. When a default-branch workflow goes red, nothing reacts until the CI agent's next cadence slot, which defaults to 6 to 24 hours. Two things cause it:
ReadBranchRulesreturnsKnown=falseandSweepRedMainskips the repository before it reads its check runs. Post-merge builds are rarely required checks, so they are exactly the failures nobody sees.For example, on one live hive a repository's post-merge image build failed on every push to
mainand nothing reacted for more than 4 hours. That repository has a branch ruleset but no required status checks.What changes
celtrigger.KindCIFailed = "ci.failed". It reuses existing fields and adds no new ones:event.repo(the configured spelling, same as issue and PR events),event.title(check name),event.head_branch(default branch),event.state(failureortimed_out), andevent.number= 0. The kick message gives the check, the short SHA and the run URL.SweepRedMainalready reads every active repository's default-branch tip (redCheckRunsForRef). When a rule wantsci.failed, it passes the tip's red runs toobserveCIFailed. That function keepsfailureandtimed_outonly and dropscancelled,action_requiredandstale, whichcheckConclusionIsRedcounts as red. It also drops any run whose check suite's head branch is not the default branch, which excludes a pull request's run on the same commit. For new failures only, it reads the commit's check suites (one call). When a rule wants the event, the tip is also read for repositories the breaker skips because they have no required checks. Breaker behavior itself is unchanged.ci_failure_signal.gois driven by webhooks and scoped to PR heads, and hives that run without webhooks (polling only) never get those deliveries.shared_ci_issues.go) reads PRs failing across a repository, not the default branch.celTriggerKickAgents, the same pathrun.stage_completeduses, with the same gates: the agent exists, is enabled, is not config- or manager-paused, and passesAgentEligibleForCELKick. The hook is installed inconfigureGitHubClient(SetCIFailedHook), so a rebuilt client keeps it.Engine.ForKind("ci.failed")keeps only rules whose expression containsci.failed. With none, the sweep does exactly what it did before. This also means an existing rule such asevent.repo == "org/x", written for issues, does not start firing on CI failures.Dashboard / MCP parity
event.kind == "ci.failed" && event.repo == "<repo>"and targetsci-maintainerwhen that agent exists. The section help explains the event.api_event_triggers.gopreset check (an allowlist of kinds that the server turns into a one-repository rule) now also acceptsci.failed. That preset is "one kind scoped to exactly one managed repository", which is safe forci.failed. Other kinds are still rejected.static/index.htmlis the identical edit (cmpagainst the manifest concatenation passes).settings_read,config_effective_readandexclusion_cataloguedo not), so there was nothing to add.Docs
src/docs/cel-triggers.md: the new kind in the kinds list, its fields in the field table, and a "Kick an agent when default-branch CI fails" section with the example, the source and its cadence, what fires and what doesn't, dedupe and restart behavior, and the gates.src/docs/operator-reference/v6-controls.md(red-main breaker): a cross-reference.src/docs/api-reference/configuration.md: preset kinds.Tests (not run locally; CI is the verdict)
pkg/githubTestSweepRedMainCIFailedOncePerSHA: the repository has no required checks. A failure and a timed_out each emit once. A cancelled run, a success, and a failure from a PR's check suite emit nothing. A second pass on the same SHA emits nothing and reads no suites. A new failing SHA emits again.TestSweepRedMainCIFailedNoRuleIsNoOp: with no hook, or with noci.failedrule, there are no check-run reads for repositories without required checks, no suite reads and no emits.TestSweepRedMainCIFailedSuiteReadErrorRetries: a failed suite read emits on the next pass.pkg/celtriggerTestMatchAgents_CIFailedEvent,TestForKind:event.kind == "ci.failed"matches, and the reused fields are reachable. ForKind drops rules that don't name the kind and is nil-safe.cmd/hiveTestCIFailedTriggerKicksThroughCELPath,TestCIFailedEventFields,TestCIFailedTriggerWanted: the kick goes through the shared path. Paused and disabled agents and repo-only rules are skipped.pkg/dashboardTestEventTriggersCIFailedPreset(API) andTestEventTriggersCIFailedPresetUI(node): the preset is offered and saved, and other kinds are still rejected.Out of scope
Auto-filing issues, changing agent cadences, project-specific workflow names, and sourcing from webhooks.
🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1