Skip to content

✨ celtrigger: ci.failed event for default-branch CI failures - #17572

Merged
hivecommons-hive[bot] merged 7 commits into
hivecommons:v6from
Danathar:feat/cel-ci-failed
Nov 11, 2026
Merged

hivecommons-hive[bot] merged 7 commits into
hivecommons:v6from
Danathar:feat/cel-ci-failed

Conversation

@Danathar

@Danathar Danathar commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a CEL trigger event kind, ci.failed, so an operator can write "kick ci-maintainer when main goes red" (expr: event.kind == "ci.failed"). Right now an agent like ci-maintainer only notices a failure on a default branch at its next cadence slot (6 to 24h), and no CEL kind covers CI. The event comes from the red-main sweep the hive already runs every 10 minutes. It fires once per repository, check and head SHA, and only rules that name ci.failed are evaluated for it. Hives without such a rule see no change and make no extra GitHub reads.

Why

Any hive whose repositories build or publish after merge has this gap. When a default-branch workflow goes red, nothing reacts until the CI agent's next cadence slot, which defaults to 6 to 24 hours. Two things cause it:

  • No CEL kind exists for CI, so an operator cannot route a default-branch failure to an agent.
  • The red-main breaker only watches required checks. When a branch has no required status checks, ReadBranchRules returns Known=false and SweepRedMain skips the repository before it reads its check runs. Post-merge builds are rarely required checks, so they are exactly the failures nobody sees.

For example, on one live hive a repository's post-merge image build failed on every push to main and nothing reacted for more than 4 hours. That repository has a branch ruleset but no required status checks.

What changes

  • New kind celtrigger.KindCIFailed = "ci.failed". It reuses existing fields and adds no new ones: event.repo (the configured spelling, same as issue and PR events), event.title (check name), event.head_branch (default branch), event.state (failure or timed_out), and event.number = 0. The kick message gives the check, the short SHA and the run URL.
  • Source: the red-main sweep. SweepRedMain already reads every active repository's default-branch tip (redCheckRunsForRef). When a rule wants ci.failed, it passes the tip's red runs to observeCIFailed. That function keeps failure and timed_out only and drops cancelled, action_required and stale, which checkConclusionIsRed counts as red. It also drops any run whose check suite's head branch is not the default branch, which excludes a pull request's run on the same commit. For new failures only, it reads the commit's check suites (one call). When a rule wants the event, the tip is also read for repositories the breaker skips because they have no required checks. Breaker behavior itself is unchanged.
  • Why this hook and not the others:
    • ci_failure_signal.go is driven by webhooks and scoped to PR heads, and hives that run without webhooks (polling only) never get those deliveries.
    • The shared-CI tracker (shared_ci_issues.go) reads PRs failing across a repository, not the default branch.
    • The red-main sweep is the only existing place that reads default-branch tips. It needs no new loop.
  • No opt-in gate: the red-main sweep runs on every hive with no setting. Like the other unstick sweeps, it is shed only while the GitHub API budget is critical. The docs say so.
  • Delivery: through celTriggerKickAgents, the same path run.stage_completed uses, with the same gates: the agent exists, is enabled, is not config- or manager-paused, and passes AgentEligibleForCELKick. The hook is installed in configureGitHubClient (SetCIFailedHook), so a rebuilt client keeps it.
  • No-op without a rule: Engine.ForKind("ci.failed") keeps only rules whose expression contains ci.failed. With none, the sweep does exactly what it did before. This also means an existing rule such as event.repo == "org/x", written for issues, does not start firing on CI failures.
  • Dedupe: in memory, keyed by (repo, check) and storing the head SHA and check suite of the last decided run. A skipped pull request run is recorded too, so it is not re-read every pass, while a default-branch run of the same check on that SHA is still reported. The red-main tracker is not persisted, so there was no cheap store to reuse. A restart may report a still-red SHA once more, and the docs say so. A failed check-suite read records nothing, so the next pass retries.

Dashboard / MCP parity

  • Settings → Features → Event triggers: the preset picker gains a Default-branch CI failed option. It writes event.kind == "ci.failed" && event.repo == "<repo>" and targets ci-maintainer when that agent exists. The section help explains the event.
  • The api_event_triggers.go preset check (an allowlist of kinds that the server turns into a one-repository rule) now also accepts ci.failed. That preset is "one kind scoped to exactly one managed repository", which is safe for ci.failed. Other kinds are still rejected. static/index.html is the identical edit (cmp against the manifest concatenation passes).
  • MCP: no admin MCP read lists trigger kinds (settings_read, config_effective_read and exclusion_catalogue do not), so there was nothing to add.

Docs

  • src/docs/cel-triggers.md: the new kind in the kinds list, its fields in the field table, and a "Kick an agent when default-branch CI fails" section with the example, the source and its cadence, what fires and what doesn't, dedupe and restart behavior, and the gates.
  • src/docs/operator-reference/v6-controls.md (red-main breaker): a cross-reference.
  • src/docs/api-reference/configuration.md: preset kinds.

Tests (not run locally; CI is the verdict)

  • pkg/github TestSweepRedMainCIFailedOncePerSHA: the repository has no required checks. A failure and a timed_out each emit once. A cancelled run, a success, and a failure from a PR's check suite emit nothing. A second pass on the same SHA emits nothing and reads no suites. A new failing SHA emits again.
  • TestSweepRedMainCIFailedNoRuleIsNoOp: with no hook, or with no ci.failed rule, there are no check-run reads for repositories without required checks, no suite reads and no emits.
  • TestSweepRedMainCIFailedSuiteReadErrorRetries: a failed suite read emits on the next pass.
  • pkg/celtrigger TestMatchAgents_CIFailedEvent, TestForKind: event.kind == "ci.failed" matches, and the reused fields are reachable. ForKind drops rules that don't name the kind and is nil-safe.
  • cmd/hive TestCIFailedTriggerKicksThroughCELPath, TestCIFailedEventFields, TestCIFailedTriggerWanted: the kick goes through the shared path. Paused and disabled agents and repo-only rules are skipped.
  • pkg/dashboard TestEventTriggersCIFailedPreset (API) and TestEventTriggersCIFailedPresetUI (node): the preset is offered and saved, and other kinds are still rejected.

Out of scope

Auto-filing issues, changing agent cadences, project-specific workflow names, and sourcing from webhooks.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

Add the forge-neutral CEL event kind ci.failed. The existing red-main
sweep (every 10 minutes, every active repo's default branch tip) now
reports each check whose newest completed run concluded failure or
timed_out, once per (repo, check, head SHA), and the event kicks the
matching agents through celTriggerKickAgents with the usual gates.

- Every check counts, not only required ones; cancelled, skipped,
  neutral, action_required and stale do not; a pull request's run on
  the same commit is excluded by its check suite's head branch.
- Only rules whose expression names ci.failed are evaluated for it
  (Engine.ForKind), so existing rules keep their meaning, and with no
  such rule the sweep makes no extra reads.
- Dedupe is in memory; a restart may re-report a still-red SHA once.
- Event triggers editor: Default-branch CI failed preset and help;
  the triggers API accepts ci.failed as a one-repository preset kind.
- Docs: cel-triggers.md, v6-controls.md, API reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 10, 2026
A failing pull request run on the default-branch tip was skipped but never
recorded, so every red-main pass re-read the check suites for that SHA
(TestSweepRedMainCIFailedOncePerSHA caught it). Record every decided run,
keyed by head SHA and check suite, so a default-branch run of the same
check on that SHA is still reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the sweep and its callers at the PR head, not just the diff. This looks correct to me.

What I verified

  • No-op without a rule. src/pkg/github/red_main.go:145 resolves ciFailedEmit once per pass; ciFailedEmitter() (red_main_ci_failed.go:76-87) returns nil unless both hooks are set and want() is true. With nil, the !rules.Known && ciFailedEmit == nil guard at red_main.go:164 is the pre-existing skip, and observeCIFailed returns on emit == nil at red_main_ci_failed.go:115 before any read. The sweep's control flow for existing hives is unchanged.
  • The finpilot gap is actually closed. For a repo with rules.Known == false and a rule present, the tip's runs are now read (red_main.go:168), handed to observeCIFailed, and only then does the sweep continue (red_main.go:175-179) — the breaker still never evaluates an unknown-rules repo.
  • Dedupe ordering. The seen map is written only after ciFailedSuiteBranches succeeds (red_main_ci_failed.go:148-153); on a suite-read error the function returns before recording, so the next pass retries, as the body says. Because redCheckRunsForRef keeps the newest run per name (red_main.go:225), a default-branch run landing after a PR run on the same SHA has a different suite ID and is still reported — the (SHA, suite) value at red_main_ci_failed.go:100 is what makes that work.
  • Lock discipline. redMainMu is released before emit and before the suite-list API call; emit runs synchronously without client locks held, matching the SetCIFailedHook contract.
  • Wiring. ForKind and Len are both nil-safe (celtrigger.go:285-299, 303), so celEngineFor(...).ForKind(...).Len() in ciFailedTriggerWanted is safe when no triggers are configured. The hook is installed in applyGitHubClientConfigHooks (github_client_config.go:179) alongside the other config-reading hooks, so a rebuilt client keeps it.
  • The body's claims match the diff; the ten tests it names exist.

One thing blocking merge: the only non-green check is PR Issue (in progress). The body has no Closes #N / Refs #N line — if there is no tracking issue, one may need to be opened and referenced for that gate to pass.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17596 ✨ knowledge: edit an existing git source (PUT /api/knowledge/git-sources) same files src/docs/api-reference/configuration.md
src/pkg/dashboard/static/index.html

@Danathar

Copy link
Copy Markdown
Contributor Author

/lgtm
/approve

@kubestellar-prow

Copy link
Copy Markdown
Contributor

@Danathar: you cannot LGTM your own PR.

Details

In response to this:

/lgtm
/approve

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Danathar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 10, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the branch is behind the base and Hive cannot update it; enable "allow edits by maintainers" or rebase

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (3 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the sweep (src/pkg/github/red_main.go:146-180), observeCIFailed (red_main_ci_failed.go), latestCheckRunsByNameAndApp (commit_ci.go:1107-1132), the boot wiring (celwire_ci_failed.go, github_client_config.go:179) and the tests. The hook is consulted once per pass, observeCIFailed runs before the !rules.Known skip so repos without required checks are covered only when a rule wants the kind, the lock is released before emit, and a failed suite read records nothing. ForKind keeps issue/PR-only rules from firing on CI. Looks correct to me; one narrow gap worth knowing about:

Low — same check name in a PR suite and a push suite on one SHA collapses to one run. redCheckRunsForRef keys red[name] on the winner of latestCheckRunsByNameAndApp (red_main.go:243,254; identity is name+app+head, commit_ci.go:1101-1105). So when a rebase/fast-forward merge puts the PR head SHA on the default branch and a workflow runs on both pull_request and push under the same check name, runs holds only the newer of the two. If that is the PR-suite run, observeCIFailed records (sha, prSuite) and skips (red_main_ci_failed.go:146-157), and the default-branch failure of that check on that SHA is never reported — the body's "a default-branch run of the same check on that SHA is still reported" holds only when the push run is the newer one. TestSweepRedMainCIFailedOncePerSHA uses a distinct name (pr-check) for the PR suite, so this path is untested. Squash merges (a fresh SHA) are unaffected, and the next push re-arms it, so this is a bounded miss rather than a false fire. A one-line note in cel-triggers.md next to the restart caveat, or filtering by suite before the per-name collapse, would close it; either is fine to defer.

Everything else — docs, changelog fragment, preset allowlist, static/index.html parity — matches the diff.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

stage 080-awaiting-merge-admission → 072-returned-to-contributor: check-cancelled

Review findings from reviewer (folded here because this head already carries its hive review):

Read the diff against v6 together with pkg/github/red_main.go (sweep loop and redCheckRunsForRef), cmd/hive/celwire.go (celEngineFor, celTriggerKickAgents) and dashboard/api_event_triggers.go. The core logic looks correct to me; the specific things I checked, in case they save a maintainer the trace:

  • Dedupe cannot swallow events through Number == 0. celTriggerKickAgents (celwire.go:282-330) has no per-item first-sight check; the issue:%s#%d / pr:%s#%d first-sight keys live only in the issue/PR poll path (celwire.go:232,238). So a ci.failed event with Number: 0 is kicked on every emit, and the only dedupe is the sweep's own (repo, check) → (sha, suite) map in red_main_ci_failed.go:121-171.
  • No behaviour change without a rule. ciFailedEmitter() returns nil unless want() is true, and the only new control-flow in SweepRedMain is if !rules.Known && ciFailedEmit == nil (red_main.go:164) plus the observeCIFailed call, which returns immediately on a nil emit. With a rule present the Skipped++ for unknown rules still happens, just after the tip read, so sweep counters are unchanged.
  • ciFailedTriggerWanted is cheap. celEngineFor caches by rules signature (celwire.go:78-81), so the per-sweep ForKind call only allocates a slice; it does not recompile CEL.
  • Non-default-branch runs on the tip are recorded before they are filtered (red_main_ci_failed.go:154-164), so a PR run left on the same SHA costs one ListCheckSuitesForRef once, not every 10 minutes. A failed suite read returns before recording, so it retries. Matches the body.
  • Preset path. The server-side preset builds event.kind == %q && event.repo == %q generically from p.Kind (api_event_triggers.go:108), and the JS already passes d.preset through as the kind, so widening the allowlist via eventTriggerPresetKind is the only change needed. static/index.html carries the same two edits.

One design note, not a defect: Engine.ForKind is a substring test on the expression text, so a rule like event.kind in ["pr.opened", "ci.failed"] qualifies but a rule that tests event.kind.startsWith("ci.") would never see this kind. The doc comment says so; just flagging it for whoever writes the operator docs later.

Checks: everything green except mutation (still pending — this is the XXL 20-minute timeout case #17602 describes, advisory) and patch-coverage, which has no conclusion yet.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the diff against v6 at a6b3f27 plus the callers it hooks into; this looks correct to me.

What I verified in the tree, not just the diff:

  • No-op without a rule holds. SweepRedMain (src/pkg/github/red_main.go:161-165 in this PR) only relaxes the !rules.Known skip when ciFailedEmitter() is non-nil, and ciFailedEmitter returns nil unless want() is true. With no ci.failed rule the sweep's read/skip sequence is byte-for-byte the base behaviour, so the "no extra GitHub reads" claim in the body is accurate.
  • Conclusion narrowing is needed and present. redCheckRunsForRef hands back everything checkConclusionIsRed accepts (src/pkg/github/commit_ci.go:779-785: failure, timed_out, cancelled, action_required, startup_failure, stale). ciFailedConclusion in red_main_ci_failed.go keeps only failure/timed_out, so a cancelled or superseded run does not kick an agent.
  • event.repo spelling is consistent. observeCIFailed emits repo straight from activeRepos() (configured spelling), the same thing issueToEvent/prToEvent in celwire.go:118,138 carry, so the preset's event.repo == "<repo>" matches for all three kinds.
  • ForKind is cheap on the hot path. celEngineFor is signature-cached (celwire.go:68-104), so the once-per-sweep ForKind call is a slice filter over already-compiled rules, no recompile.
  • Dedupe + lock discipline. seen is only touched under redMainMu, and emit runs with no client lock held, so a kick that blocks cannot stall other redMain readers. Recording a skipped PR-suite run so it is not re-read each pass is the right call.

One non-blocking note (info): Engine.ForKind is a plain substring test on Expr, so a rule such as event.kind != "ci.failed" is selected for evaluation; it then evaluates false on a ci.failed event, so the outcome is still correct — just mentioning it because the doc comment says "names" the kind and a reader might expect an exact-match parse.

CI is green apart from mutation still pending at 22095e5. Nothing here needs a decision from a maintainer; leaving it to the existing approval.

Evidence bundle `#1757243c06c7` sha256:77b5146b3364c727572981270b1e71590971949555868c1e104eae0d3e45aa58 — download from the dashboard or `hivectl review evidence #17572`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Triage of the check-cancelled block on head b6d1217 (fork head — hive cannot push here):

  • The only non-green check on this head is mutation from Mutation (advisory) (run 42892862418). It started 20:11:53Z and was cancelled 20:32:10Z — exactly its timeout-minutes: 20 — with gremlins still running on this size/XXL diff. It is continue-on-error: true and by design never blocks a PR.
  • Every other check (build-gate gate, all test shards, lint, security, docker, DCO) is green. Baseline check: mutation does not run on v6 itself and the other cancelled runs in its history are cancel-in-progress supersedes, so this is PR-local to the diff size, not a shared incident.
  • The merge request was refused because the hive's merge-request CI gate treats a cancelled check as red when it cannot read the branch's required-check set; I am filing that as a hive bug separately.

Concrete fix: push any new head (e.g. git rebase origin/v6 + git push --force-with-lease, same sign-off). That re-runs CI, and the merge request can be re-issued. Alternatively a maintainer can merge manually — nothing in the PR itself is failing.


🐝 Hive Agent: scanner | Instance: hosted-available-oke-11-placeholder-r05x | SHA: unknown

— hive: agent=scanner backend=copilot model=claude-fable-5.1 copilot=1.0.88

hivecommons-hive Bot added a commit that referenced this pull request Nov 10, 2026
…artial (#17604)

* 🌱 ci: mutation report keeps its own time budget and says when it is partial

A 20-minute runner kill of the advisory gremlins step wrote no comment.md,
so XXL PRs kept the <!-- hive-mutation --> comment from an earlier head
(#17596) or got none (#17572), and the text could not tell a clean run
from a timeout. The run phase now bounds each package by the remaining
budget, always writes the report, names the head SHA, and lists the
packages it did not finish. test-mutation-report.sh pins the three
shapes with a stubbed gremlins; TRUSTED_SCRIPTS_TREE re-pinned.

Refs #17602

Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* 🌱 ci: mutation report review fixes — cite #17602, drop redundant total counter

Fix review findings on #17604: the test header rationale now points at
#17602 (the mutation-advisory timeout issue) rather than the unrelated
#17601, and the package loop reuses `i` instead of a duplicate `total`
counter that was always equal to it.

Hive-Fix-Attempt: 1/6
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>

* ci: re-pin TRUSTED_SCRIPTS_TREE to the v6-merged src/scripts tree

The runtime-tests contract compares the pin against the merge ref's
src/scripts tree, so the pin must follow v6, not the branch tip alone.

Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>

---------

Signed-off-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Signed-off-by: ci-maintainer <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Signed-off-by: scanner <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: quality <280983584+hivecommons-hive[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against the v6 merge-base (38fb7204) and the sweep it hooks into. Looks correct to me; the body's claims match the code.

What I checked:

  • No-op without a rule, verified at both ends. ciFailedEmitter() (red_main_ci_failed.go:74-82) returns nil unless want() is true, and want is ciFailedTriggerWanted → celEngineFor(...).ForKind("ci.failed").Len() > 0. celEngineFor is signature-cached (celwire.go:78-81), so the once-per-sweep call does not recompile rules. With nil emit, red_main.go:164 keeps the original !rules.Known → skip behaviour and observeCIFailed returns at its first guard, so pre-existing sweeps make exactly the reads they did before.
  • Check-suite read only for new failures. The dedupe pass (red_main_ci_failed.go:114-125) runs under redMainMu before any network call; ciFailedSuiteBranches is hit only for SHAs with a not-yet-seen failing run, cached per SHA within the pass, and a failed read records nothing so the next pass retries (:135-139). The seen entry is written before the branch test (:142-147) so a PR run left on the tip does not re-read suites every pass — as the comment says.
  • ForKind semantics are safe. A rule like event.kind == "pr.opened" lacks the substring and is never evaluated for CI failures; a rule that names ci.failed but then guards on event.number > 0 simply evaluates false. celTriggerKickAgents (celwire.go:282) has no Number-based logic, so Number: 0 is harmless.
  • Breaker unchanged. observeCIFailed is called before the second !rules.Known skip (red_main.go:175-180), and the required-check loop below is untouched.
  • Preset gate in api_event_triggers.go:25-31 plus the JS/index.html option, docs (cel-triggers.md, v6-controls.md, configuration.md) and the changelog fragment are all present.

One narrow edge, info only, not a defect: redCheckRunsForRef keeps one run per check name/app via latestCheckRunsByNameAndApp (commit_ci.go:1131-1153). If a repository fast-forwards or rebase-merges so main's tip equals a PR head, and that PR's run of the same check is newer than the default-branch run (e.g. a re-run from the PR), the default-branch failing run is never in runs and cannot be reported. That matches the documented "a pull request's run on the same commit is not one" and the normal case (push-to-main run created after the PR run) is fine; mentioning it so the limitation is known.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the new code against the v6 tree (red_main.go sweep body, redCheckRunsForRef, latestCheckRunsByNameAndApp, celEngineFor, celTriggerKickAgents) rather than just the diff. This looks correct to me; the one thing worth a maintainer's glance is below.

What I verified

  • SweepRedMain ordering (src/pkg/github/red_main.go:160-180): when no rule names ci.failed, ciFailedEmit == nil and the !rules.Known early-continue is unchanged, so hives without such a rule make exactly the reads they made before. With a rule, the no-required-checks repo is read once, handed to observeCIFailed, then still counted as Skipped — breaker behaviour is untouched. The test at red_main_ci_failed_test.go:152 asserts that.
  • Locking (red_main_ci_failed.go:120-131, 153-158): redMainMu is released before ciFailedSuiteBranches and before emit, and ciFailedEmitter releases it before calling want() (which takes globalCELEngine.mu via celEngineFor). No lock-order issue.
  • Dedupe key is (SHA, check-suite ID) (red_main_ci_failed.go:98-100), so a rerun of the same check on the same SHA does not re-fire (same suite), while a default-branch suite on a SHA whose PR suite was skipped still does. Matches the body's claim.
  • ForKind (celtrigger.go:285-299) is a substring test; a rule like event.kind != "ci.failed" would be included but then evaluates false on the event, so it is harmless. A rule written as event.kind.startsWith("ci.") would not be picked up — the doc comment says "plain substring test", so this is disclosed, just noting it for the record.
  • No other kind allowlist exists in config/compile paths (grepped KindStageCompleted usages), so nothing else needed registering. The dashboard preset allowlist (api_event_triggers.go:25-31) is the only one and is updated.

One observation (low)

ciFailedConclusion (src/pkg/github/red_main_ci_failed.go:87-89) accepts only failure and timed_out. checkConclusionIsRed (src/pkg/github/commit_ci.go:779-785) also treats startup_failure as red, and the new docs list the excluded conclusions as cancelled, skipped, neutral, action_required, stale — startup_failure is not mentioned either way. A workflow file with a syntax error merged to main produces startup_failure on every push, which is exactly the "main went red and nobody reacted" case this PR targets, and it will not fire ci.failed. Whether that is intended is a product call; if it is, the docs should say so, and if not, adding it to ciFailedConclusion is a one-line change plus a test case next to TestSweepRedMainCIFailedOncePerSHA.

Not a blocker. CI is green across all test shards; I have not approved or changed anything.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the sweep hook, the CEL wiring and their callers against the v6 tree (520d441) before the body; the code does what the body says and I found no defect. Looks correct to me.

What I verified:

  • Breaker behaviour unchanged without a rule. ciFailedEmitter() returns nil unless want() is true (src/pkg/github/red_main_ci_failed.go:72-83), and the only branch that depends on it is if !rules.Known && ciFailedEmit == nil (red_main.go:164), which is the pre-existing skip. With the hook nil the loop reads exactly what it read before — no extra GitHub calls.
  • With a rule, the breaker is still unchanged. The !rules.Known skip is re-applied right after observeCIFailed (red_main.go:176-180), so tracker.ObservePending is never reached for a repo the breaker previously skipped. The ci.failed read is additive.
  • Conclusion filter is right for the stated purpose. redCheckRunsForRef returns everything checkConclusionIsRed accepts, including cancelled/action_required/stale (commit_ci.go:779-785); ciFailedConclusion narrows to failure/timed_out (red_main_ci_failed.go:88-90). Matches the body.
  • Dedupe and the suite-branch check. Seen key is (repo, check) → sha\x00suiteID, recorded under redMainMu whether or not the run turns out to be a default-branch suite, so a PR run left on the tip costs one ListCheckSuitesForRef per SHA, not one per pass. A failed suite read returns before recording, so it retries. Map reset at 5000 entries may re-report once, as the comment says. The emit call happens with no lock held, as SetCIFailedHook promises.
  • Wiring. SetCIFailedHook(b.ciFailedTriggerWanted, b.emitCIFailedTrigger) in applyGitHubClientConfigHooks (github_client_config.go:182); both read b.cfg/b.agentMgr/b.gov lazily, and celEngineFor is cached by triggers signature (celwire.go:68-80), so the once-per-sweep want() is cheap. celTriggerKickAgents signature matches; Engine.Len() is nil-safe, so ForKind(...) returning nil is fine.
  • Dashboard preset gate widened via eventTriggerPresetKind with the same one-repo validation as the other two kinds; docs and changelog.d fragment present.

One informational note, not a defect: Engine.ForKind is a substring test on the rule's expression text (celtrigger.go:285-298), as its doc comment says. A rule that selects the kind indirectly — event.kind.startsWith("ci."), or a kind list held in a CEL variable — will not be considered a ci.failed rule and the sweep stays a no-op for it. Given the goal (never fire an issue/PR rule on a CI event), that is the conservative side to err on, and src/docs/cel-triggers.md documents the literal-name requirement, so I'd leave it.

Checks are green apart from the advisory mutation job; tide is waiting on lgtm. I have not approved or merged anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit fe9cba9 into hivecommons:v6 Nov 11, 2026
71 of 72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant