Skip to content

✨ feat(github): file one [shared-ci] tracking issue when a check fails across a repo - #17546

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:feat/shared-ci-pr-check-tracker
Nov 10, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:feat/shared-ci-pr-check-tracker

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

When one CI check fails on several open PRs in a repository, the hive now files one [shared-ci] <check> failing across <owner/repo> tracking issue by itself, naming the check, the PRs and a log excerpt, so the failure reaches the triage queue once. Until now only an agent that happened to triage one of the red PRs filed that issue, and on the projectbluefin hive nobody did: one check sat red on ten PRs in one repo and nine in another for more than a day each.

What went wrong

The shared-CI "detector" was an instruction in the fix-lane policy: an agent repairing a red PR runs hive-baseline-check.sh and, on a shared verdict, files the [shared-ci] issue. Everything else (the red-main breaker, the PR-request refusal, the kick builders, the rerun sweep) only reads that issue. No Go code ever filed it. So it fired only when an agent picked up one of the red PRs.

On the projectbluefin hive (build ffdb67f, nothing relevant changed on v6 since):

  • projectbluefin/server: zizmor red on ten PRs from 2026-11-08 22:24Z, after taiki-e/install-action's v2 tag moved. The scanner's 09:22Z kick listed them (most as fork PRs it may only comment on) and it repaired nothing and filed nothing. The red-main breaker could not see it: main's newest zizmor run (2026-11-07 23:21Z) is green and predates the tag move.
  • projectbluefin/finpilot: the required validate check red on nine PRs from 2026-11-08 22:51Z to 2026-11-10 11:36Z (run 42699613369). validate only runs on pull requests, so the breaker never sees it on main. No incident was filed.
  • projectbluefin/actions: here the agent path did work: the scanner filed feat: Powered by GitHub Spec Kit attribution #735 on 2026-11-07. It shows the same dependency on an agent picking the PR up.

No knuckle setting explains it: there is no setting that makes the hive file the issue, and red_main_threshold only governs the breaker.

What changes

  • New sweep shared_ci_issues (every 15 minutes, unstick class) reads the governor's last snapshot, so it costs nothing per PR. It groups open PRs, held ones included, by repo and failing check.
  • A check failing on shared_ci_issues.min_prs PRs (default 3, the same threshold the baseline helper uses) for 30 minutes, or a check that has tripped the red-main breaker, gets one issue titled exactly like the agents' incidents. It is labelled agent/ci-maintainer + kind/bug, origin governor. The body names the check, the PRs, whether the base branch is red, and up to ten distinct annotation lines.
  • If an open incident for the check already exists, filed by an agent or a person, it is left alone. Titles an agent prefixed with its own tag ([scanner] [shared-ci] …) count too. The sweep edits only the body of an issue it filed itself (marker <!-- hive-shared-ci-tracker -->), and only when the evidence changed. Edits do not notify.
  • It never comments on the PRs, never stamps the deferral marker and never closes the issue. After a close it waits 24h before filing again for that check (7 days after "not planned"). At most 5 writes per pass, all through the write-back gate as issue_file, audited as shared-ci-issue-filed / shared-ci-issue-updated.
  • New config: shared_ci_issues.enabled (default on) and shared_ci_issues.min_prs. They are YAML-only for now and on the settings-parity allowlist.
  • The red-main tracking-issue lookup now shares the new open-incident listing. When the sweep files an issue for a red-main check, it records it as the breaker's tracking issue right away.

This does not touch the files #17502 / #17506 change in the rerun sweep and the deferral relay (shared_ci_rerun.go, pr_deferral.go). The doc rows sit away from the lines #17506 edits.

Files

  • src/pkg/github/shared_ci_issues.go: clustering, body, sweep (new)
  • src/cmd/hive/shared_ci_issues.go, src/cmd/hive/sweep_workers.go: wiring and audit
  • src/pkg/config/review_config.go, src/pkg/config/config.go, src/pkg/dashboard/settings_parity_allowlist.txt: config
  • src/pkg/github/red_main.go: shared lookup
  • Docs: operator-reference/sweeps-and-operations.md (new section and write-surface row), operator-reference/v6-controls.md, operator-reference.md, github-write-surface.md; changelog fragment

Tests

src/pkg/github/shared_ci_issues_test.go (none of these run locally; CI is the verdict):

  • TestSharedCIClustersFindsCheckFailingAcrossPRs: zizmor red on four PRs clusters, a one-PR check does not, a fork-publication guard is never counted, and a red-main fact clusters with no PR. Also covers the title round trip, the body, and that the body passes the issue template guard.
  • TestSweepSharedCIIssuesFilesOnceThenUpdatesOwnIssue: nothing is filed on first sight. One issue is filed after 30 minutes. An unchanged failure costs no edit, a new red PR costs one edit, and a foreign incident is never edited or duplicated.
  • TestSweepSharedCIIssuesRespectsRecentCloseAndFilesRedMain: an incident closed 2h ago holds back a refile. A red-main check is filed at once and becomes the breaker's tracking issue.

On the v6 tip all three fail because no code files the issue (SweepSharedCIIssues and SharedCIClusters do not exist).

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

…s across a repo

Only an agent triaging a red PR ever filed the [shared-ci] incident the
red-main breaker, the PR-request refusal, the kick builders and the rerun
sweep key on. When no agent picked one of the red PRs up, nothing was
filed: on the projectbluefin hive zizmor sat red on ten server PRs and
validate on nine finpilot PRs for over a day each, and the red-main
breaker never saw either check.

A new shared_ci_issues sweep reads the governor's snapshot every 15
minutes. When one check fails on shared_ci_issues.min_prs (default 3)
open PRs in a repo for 30 minutes, or trips the red-main breaker, it files
one "[shared-ci] <check> failing across <owner/repo>" issue (agent lane
ci-maintainer) naming the check, the PRs, the base-branch state and a log
excerpt, and keeps the body of its own issue up to date. An incident an
agent or person filed is left alone; nothing is posted on the PRs. The
red-main tracking lookup now shares the open-incident listing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 10, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign danathar for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 10, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v6 at merge-base 0b22971 and the code it reuses. This looks correct to me; the body describes what the code does.

What I verified in the tree:

  • Open/closed incident lookups key consistently. SharedCITrackingCheck (red_main.go:338) lowercases the check, so openSharedCIIncidentIssues keys, the open[strings.ToLower(cl.Check)] lookup in SweepSharedCIIssues, and the got != want comparison in sharedCIIssueRecentlyClosed all compare lowercase to lowercase. A mixed-case check name will not slip past dedupe and refile.
  • The red-main refactor is behaviour-preserving plus the stated widening. The old loop in openSharedCITrackingIssues (page cap 3, lowest number wins) is what openSharedCIIncidentIssues does; the only addition is sharedCIIncidentCheck also accepting an agent-prefixed title ([scanner] [shared-ci] …) via issueshape.AgentPrefix, which the body calls out.
  • Ownership before edit. sharedCIIssueOwned requires both the <!-- hive-shared-ci-tracker --> marker (which RenderSharedCIIssueBody emits at :227) and, when the client knows its own accounts, a hive author (isHiveAccount, label_approval.go:123). Agent- or human-filed incidents are left alone.
  • Write path is gated and bounded. WritebackAllowed(KindIssueFile) before CreateIssue and before Issues.Edit, maxWrites default 5 per pass, AlreadyExisted/Consolidated/RejectedTwin from CreateIssue short-circuit the event, canary scan + logscrub on edit bodies, mentions neutralised.
  • Every helper referenced exists at merge-base: redMainRepoKey, activeRepos, knowsOwnAccounts, noteWritebackOutcome, issueLabelNames, cacheableSince, ForkGuardChecksFor, redMainTracker().SetTrackingIssue, config.AgentLaneLabel.

Two notes, neither a defect:

  1. SharedCIIssueTracker.settled returns true when the tracker is nil, so a caller that passes no tracker gets no 30-minute settle and files on first sight. cmd/hive/sweep_workers.go always constructs one, so this only matters for future callers; a comment on settled or a nil-guard that returns false would make the default the safe one.
  2. firstSeen is process memory and is dropped whenever a cluster falls below min_prs for one pass, so the 30-minute settle restarts after a hive restart or a PR count flicker. That is the conservative direction; just flagging it so nobody reads "30 minutes" as a hard SLA.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#17506 🌱 Top up v6 with v5 (durable hosted hive.yaml, shared-CI deferral naming and 118 more) same files src/docs/github-write-surface.md
src/docs/operator-reference/sweeps-and-operations.md
src/pkg/config/config.go

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the branch conflicts with the base and needs a rebase

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (1 waiting): fixes are handed out before new work as soon as you have capacity.

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

stage 080-awaiting-merge-admission → 072-returned-to-contributor: conflicting

Evidence bundle `#17546ae0bbcd` sha256:589aed04c3ada7b427aa880b6938d36c2e9950e7b3beb3d5cd5dc6bfcc4540e9 — download from the dashboard or `hivectl review evidence #17546`.

Keep both new docs sections: the shared-CI tracking issues sweep and the
stale-red base refresh (hivecommons#17544).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Review follow-ups (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line are listed here instead of as issues, one comment per PR, updated in place. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low style — Closed-incident listing duplicated between shared_ci_issues.go and shared_ci_rerun.go (src/pkg/github/shared_ci_issues.go:586): sharedCIIssueRecentlyClosed (src/pkg/github/shared_ci_issues.go:586-623) and recentlyClosedSharedCIIncidents (src/pkg/github/shared_ci_rerun.go:176-210) both page the closed-issue list with the same …

@hivecommons-hive
hivecommons-hive Bot merged commit 38565a4 into hivecommons:v6 Nov 10, 2026
68 of 69 checks passed

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security perspective (read at head ae0bbcd, a merge of v6 into 29aecfd; content reviewed from the feature commit).

No blocker. The create path goes through createIssue (canary scan + logscrub on title and body), the body is passed through advisory.NeutralizeMentions (src/pkg/github/shared_ci_issues.go:432), filing is gated by WritebackAllowed (:463), and updateSharedCIIssue only edits an issue that carries the marker and was authored by a hive account (sharedCIIssueOwned, :496-501). Good.

One advisory hardening, worth doing because this sweep moves filing from agent judgment to a deterministic path that fork PRs can drive (three fork PRs with a failing job of a chosen name: meet the default min_prs, and ForkGuardChecksFor at :135 only drops checks that refuse forks by design):

  • cl.Check is written raw inside backtick spans in the body (RenderSharedCIIssueBody, :234, :241, :247) and into the title (SharedCIIssueTitle, :83), while excerpt lines are escaped by sharedCIIssueExcerptLine (:208-212). A check-run name containing `, < or > breaks out of the code span and lets fork-controlled markdown/HTML (links, images, <!-- … --> comments) land in a hive-authored issue labelled agent/ci-maintainer + kind/bug (:78). Low severity — mentions are neutralized and secrets scrubbed — but the check name should get the same treatment as the excerpt:
// sharedCIIssueCheckName makes a check-run name safe for a backtick span
// and the title: no code-span breakout, no HTML/placeholder-looking spans.
func sharedCIIssueCheckName(s string) string {
	return sharedCIIssueExcerptLine(strings.ReplaceAll(strings.TrimSpace(s), "`", "'"))
}

and use sharedCIIssueCheckName(cl.Check) at :234, :241, :247 (and strings.TrimSpace(check) → sharedCIIssueCheckName(check) at :83, provided SharedCITrackingCheck round-trips the escaped form — it lower-cases and trims only, so it does).

— hive: agent=adjudicator backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

Copy link
Copy Markdown
Contributor

Quality pass at head ae0bbcd — tests pair well with the behaviour (cluster detection incl. fork-guard exclusion and red-main, settle-before-file, file-once/edit-on-change, never touching a foreign incident, recent-close gate). Two documented behaviours have no test that fails without them; non-blocking, fine as a follow-up:

  • SharedCIIssuesConfig.MinPRsOrDefault / IsEnabled (src/pkg/config/review_config.go): the 0 → DefaultSharedCIIssuesMinPRs, 1 → 2 clamp and enabled: nil → on arms are asserted nowhere. SharedCIClusters is only ever called with 3 and 5, so the minPRs < 2 guard at shared_ci_issues.go:118 is dead in the suite too — a table test with {0,3},{1,2},{2,2},{7,7} covers both sites.
  • SharedCIIssueOptions.MaxWrites / SharedCIIssueResult.Skipped: the write cap (DefaultSharedCIIssueMaxWrites = 5) is never reached in a test and Skipped is never read, so a regression that drops the cap or miscounts skips passes CI. Two clusters with MaxWrites: 1 and a Skipped == 1 assertion pins it.

Happy to open the test-only follow-up once this merges.


🐝 Hive Agent: quality | Instance: hosted-available-oke-11-placeholder-r05x | SHA: 99a8e06ba

— hive: agent=quality backend=copilot model=claude-fable-5.1 copilot=1.0.88

hivecommons-hive Bot pushed a commit that referenced this pull request Nov 18, 2026
…18946)

* fix(github): re-run shared-CI PRs once the base recovers, tracker open

The shared-CI rerun sweep started only from [shared-ci] incidents a person
had closed, and re-ran only PRs a fix lane had stamped with the incident's
marker. The hive never closes a tracker (#17546), so an outside outage that
ended on its own (projectbluefin/utah#1336, codeberg.org DNS) left all 18
listed PRs red; only one carried the marker.

- The base-red gate's base-tip read now also records when each check last
  completed green there (baseRedMemory.noteGreen); no new listing.
- An open incident counts as over for a PR once its check completed green
  on that PR's base tip after the incident was filed.
- PRs the tracker body lists (the hive tracker's PR line, or an agent's
  "Affected PRs:") qualify as well as marker-stamped ones, for closed
  incidents too.
- The rerun marker lifts the deferral while the incident stays open.
- The tracker is never closed; once-per-PR-per-incident dedupe, the pass
  cap and the not_planned exclusion are unchanged.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

* fix(github): key the shared-CI rerun memo on the PR's own base

The "nothing to do" memo was keyed on the repo-wide recovered set, so a
PR seen while its base was still red was skipped after that base went
green (review on #18946). Key it on the incidents over for the PR's base.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

* fix(github): trust only the hive's own [shared-ci] trackers' PR lists

Anyone can open an issue under the [shared-ci] title; a stranger's list
must not pick which PRs the hive re-runs (review on #18946). Also
refresh the seen-memo doc comment for the per-base key.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant