Repository navigation
✨ feat(github): file one [shared-ci] tracking issue when a check fails across a repo - #17546
Conversation
…s across a repo Only an agent triaging a red PR ever filed the [shared-ci] incident the red-main breaker, the PR-request refusal, the kick builders and the rerun sweep key on. When no agent picked one of the red PRs up, nothing was filed: on the projectbluefin hive zizmor sat red on ten server PRs and validate on nine finpilot PRs for over a day each, and the red-main breaker never saw either check. A new shared_ci_issues sweep reads the governor's snapshot every 15 minutes. When one check fails on shared_ci_issues.min_prs (default 3) open PRs in a repo for 30 minutes, or trips the red-main breaker, it files one "[shared-ci] <check> failing across <owner/repo>" issue (agent lane ci-maintainer) naming the check, the PRs, the base-branch state and a log excerpt, and keeps the body of its own issue up to date. An incident an agent or person filed is left alone; nothing is posted on the PRs. The red-main tracking lookup now shares the open-incident listing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Read the diff against v6 at merge-base 0b22971 and the code it reuses. This looks correct to me; the body describes what the code does.
What I verified in the tree:
- Open/closed incident lookups key consistently.
SharedCITrackingCheck(red_main.go:338) lowercases the check, soopenSharedCIIncidentIssueskeys, theopen[strings.ToLower(cl.Check)]lookup inSweepSharedCIIssues, and thegot != wantcomparison insharedCIIssueRecentlyClosedall compare lowercase to lowercase. A mixed-case check name will not slip past dedupe and refile. - The red-main refactor is behaviour-preserving plus the stated widening. The old loop in
openSharedCITrackingIssues(page cap 3, lowest number wins) is whatopenSharedCIIncidentIssuesdoes; the only addition issharedCIIncidentCheckalso accepting an agent-prefixed title ([scanner] [shared-ci] …) viaissueshape.AgentPrefix, which the body calls out. - Ownership before edit.
sharedCIIssueOwnedrequires both the<!-- hive-shared-ci-tracker -->marker (whichRenderSharedCIIssueBodyemits at:227) and, when the client knows its own accounts, a hive author (isHiveAccount,label_approval.go:123). Agent- or human-filed incidents are left alone. - Write path is gated and bounded.
WritebackAllowed(KindIssueFile)beforeCreateIssueand beforeIssues.Edit,maxWritesdefault 5 per pass,AlreadyExisted/Consolidated/RejectedTwinfromCreateIssueshort-circuit the event, canary scan +logscrubon edit bodies, mentions neutralised. - Every helper referenced exists at merge-base:
redMainRepoKey,activeRepos,knowsOwnAccounts,noteWritebackOutcome,issueLabelNames,cacheableSince,ForkGuardChecksFor,redMainTracker().SetTrackingIssue,config.AgentLaneLabel.
Two notes, neither a defect:
SharedCIIssueTracker.settledreturnstruewhen the tracker is nil, so a caller that passes no tracker gets no 30-minute settle and files on first sight.cmd/hive/sweep_workers.goalways constructs one, so this only matters for future callers; a comment onsettledor a nil-guard that returnsfalsewould make the default the safe one.firstSeenis process memory and is dropped whenever a cluster falls belowmin_prsfor one pass, so the 30-minute settle restarts after a hive restart or a PR count flicker. That is the conservative direction; just flagging it so nobody reads "30 minutes" as a hard SLA.
— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@Danathar this PR is back in your court ( Review asks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity. |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. stage 080-awaiting-merge-admission → 072-returned-to-contributor: conflicting Evidence bundle `#17546ae0bbcd` sha256:589aed04c3ada7b427aa880b6938d36c2e9950e7b3beb3d5cd5dc6bfcc4540e9 — download from the dashboard or `hivectl review evidence #17546`. |
Keep both new docs sections: the shared-CI tracking issues sweep and the stale-red base refresh (hivecommons#17544). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Review follow-ups (below
|
There was a problem hiding this comment.
Security perspective (read at head ae0bbcd, a merge of v6 into 29aecfd; content reviewed from the feature commit).
No blocker. The create path goes through createIssue (canary scan + logscrub on title and body), the body is passed through advisory.NeutralizeMentions (src/pkg/github/shared_ci_issues.go:432), filing is gated by WritebackAllowed (:463), and updateSharedCIIssue only edits an issue that carries the marker and was authored by a hive account (sharedCIIssueOwned, :496-501). Good.
One advisory hardening, worth doing because this sweep moves filing from agent judgment to a deterministic path that fork PRs can drive (three fork PRs with a failing job of a chosen name: meet the default min_prs, and ForkGuardChecksFor at :135 only drops checks that refuse forks by design):
cl.Checkis written raw inside backtick spans in the body (RenderSharedCIIssueBody,:234,:241,:247) and into the title (SharedCIIssueTitle,:83), while excerpt lines are escaped bysharedCIIssueExcerptLine(:208-212). A check-run name containing`,<or>breaks out of the code span and lets fork-controlled markdown/HTML (links, images,<!-- … -->comments) land in a hive-authored issue labelledagent/ci-maintainer+kind/bug(:78). Low severity — mentions are neutralized and secrets scrubbed — but the check name should get the same treatment as the excerpt:
// sharedCIIssueCheckName makes a check-run name safe for a backtick span
// and the title: no code-span breakout, no HTML/placeholder-looking spans.
func sharedCIIssueCheckName(s string) string {
return sharedCIIssueExcerptLine(strings.ReplaceAll(strings.TrimSpace(s), "`", "'"))
}
and use sharedCIIssueCheckName(cl.Check) at :234, :241, :247 (and strings.TrimSpace(check) → sharedCIIssueCheckName(check) at :83, provided SharedCITrackingCheck round-trips the escaped form — it lower-cases and trims only, so it does).
— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88
|
Quality pass at head
Happy to open the test-only follow-up once this merges. 🐝 Hive Agent: — hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88 |
…18946) * fix(github): re-run shared-CI PRs once the base recovers, tracker open The shared-CI rerun sweep started only from [shared-ci] incidents a person had closed, and re-ran only PRs a fix lane had stamped with the incident's marker. The hive never closes a tracker (#17546), so an outside outage that ended on its own (projectbluefin/utah#1336, codeberg.org DNS) left all 18 listed PRs red; only one carried the marker. - The base-red gate's base-tip read now also records when each check last completed green there (baseRedMemory.noteGreen); no new listing. - An open incident counts as over for a PR once its check completed green on that PR's base tip after the incident was filed. - PRs the tracker body lists (the hive tracker's PR line, or an agent's "Affected PRs:") qualify as well as marker-stamped ones, for closed incidents too. - The rerun marker lifts the deferral while the incident stays open. - The tracker is never closed; once-per-PR-per-incident dedupe, the pass cap and the not_planned exclusion are unchanged. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 * fix(github): key the shared-CI rerun memo on the PR's own base The "nothing to do" memo was keyed on the repo-wide recovered set, so a PR seen while its base was still red was skipped after that base went green (review on #18946). Key it on the incidents over for the PR's base. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 * fix(github): trust only the hive's own [shared-ci] trackers' PR lists Anyone can open an issue under the [shared-ci] title; a stranger's list must not pick which PRs the hive re-runs (review on #18946). Also refresh the seen-memo doc comment for the per-base key. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Summary
When one CI check fails on several open PRs in a repository, the hive now files one
[shared-ci] <check> failing across <owner/repo>tracking issue by itself, naming the check, the PRs and a log excerpt, so the failure reaches the triage queue once. Until now only an agent that happened to triage one of the red PRs filed that issue, and on the projectbluefin hive nobody did: one check sat red on ten PRs in one repo and nine in another for more than a day each.What went wrong
The shared-CI "detector" was an instruction in the fix-lane policy: an agent repairing a red PR runs
hive-baseline-check.shand, on a shared verdict, files the[shared-ci]issue. Everything else (the red-main breaker, the PR-request refusal, the kick builders, the rerun sweep) only reads that issue. No Go code ever filed it. So it fired only when an agent picked up one of the red PRs.On the projectbluefin hive (build ffdb67f, nothing relevant changed on v6 since):
zizmorred on ten PRs from 2026-11-08 22:24Z, aftertaiki-e/install-action'sv2tag moved. The scanner's 09:22Z kick listed them (most as fork PRs it may only comment on) and it repaired nothing and filed nothing. The red-main breaker could not see it: main's newestzizmorrun (2026-11-07 23:21Z) is green and predates the tag move.validatecheck red on nine PRs from 2026-11-08 22:51Z to 2026-11-10 11:36Z (run 42699613369).validateonly runs on pull requests, so the breaker never sees it on main. No incident was filed.No knuckle setting explains it: there is no setting that makes the hive file the issue, and
red_main_thresholdonly governs the breaker.What changes
shared_ci_issues(every 15 minutes, unstick class) reads the governor's last snapshot, so it costs nothing per PR. It groups open PRs, held ones included, by repo and failing check.shared_ci_issues.min_prsPRs (default 3, the same threshold the baseline helper uses) for 30 minutes, or a check that has tripped the red-main breaker, gets one issue titled exactly like the agents' incidents. It is labelledagent/ci-maintainer+kind/bug, origingovernor. The body names the check, the PRs, whether the base branch is red, and up to ten distinct annotation lines.[scanner] [shared-ci] …) count too. The sweep edits only the body of an issue it filed itself (marker<!-- hive-shared-ci-tracker -->), and only when the evidence changed. Edits do not notify.issue_file, audited asshared-ci-issue-filed/shared-ci-issue-updated.shared_ci_issues.enabled(default on) andshared_ci_issues.min_prs. They are YAML-only for now and on the settings-parity allowlist.This does not touch the files #17502 / #17506 change in the rerun sweep and the deferral relay (
shared_ci_rerun.go,pr_deferral.go). The doc rows sit away from the lines #17506 edits.Files
src/pkg/github/shared_ci_issues.go: clustering, body, sweep (new)src/cmd/hive/shared_ci_issues.go,src/cmd/hive/sweep_workers.go: wiring and auditsrc/pkg/config/review_config.go,src/pkg/config/config.go,src/pkg/dashboard/settings_parity_allowlist.txt: configsrc/pkg/github/red_main.go: shared lookupoperator-reference/sweeps-and-operations.md(new section and write-surface row),operator-reference/v6-controls.md,operator-reference.md,github-write-surface.md; changelog fragmentTests
src/pkg/github/shared_ci_issues_test.go(none of these run locally; CI is the verdict):TestSharedCIClustersFindsCheckFailingAcrossPRs: zizmor red on four PRs clusters, a one-PR check does not, a fork-publication guard is never counted, and a red-main fact clusters with no PR. Also covers the title round trip, the body, and that the body passes the issue template guard.TestSweepSharedCIIssuesFilesOnceThenUpdatesOwnIssue: nothing is filed on first sight. One issue is filed after 30 minutes. An unchanged failure costs no edit, a new red PR costs one edit, and a foreign incident is never edited or duplicated.TestSweepSharedCIIssuesRespectsRecentCloseAndFilesRedMain: an incident closed 2h ago holds back a refile. A red-main check is filed at once and becomes the breaker's tracking issue.On the v6 tip all three fail because no code files the issue (
SweepSharedCIIssuesandSharedCIClustersdo not exist).🤖 Generated with Claude Code
https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1