Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-shared-ci-rerun-on-recovery.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The shared-CI rerun sweep no longer waits for a person to close the `[shared-ci]` tracker: once the incident's check has completed green on a PR's base tip after the tracker was filed, the PRs the tracker lists (Renovate, Dependabot and 072 PRs included) and the PRs deferred to it get their failed jobs re-run once each, and the rerun lifts their deferral. The tracker stays open for a person to close. PRs a closed tracker lists are now re-run too, not only the ones a fix lane stamped.
36 changes: 28 additions & 8 deletions src/docs/operator-reference/sweeps-and-operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -306,19 +306,37 @@ someone re-runs it. The shared-CI rerun sweep
(`src/pkg/github/shared_ci_rerun.go`) closes that gap every ~15 minutes:

- It lists `[shared-ci]` issues closed in the last 72h in each watched repo,
ignoring ones closed as "not planned". With none, it makes no other calls.
ignoring ones closed as "not planned".
- It also treats an **open** incident as over once the incident's check has
completed green on a base branch tip after the incident was filed. The hive
never closes a tracker itself, and an outside outage that ends on its own
(projectbluefin/utah#1336: codeberg.org stopped resolving) otherwise left
every PR red until a person closed the issue. The green times come from the
base-red gate's own reads of the base tip (no extra listing), so the open
incidents are listed only for a repo where such a green run was seen. A PR
counts the open incident as over only when the check passed on **its own**
base. The tracker stays open; closing it is still a person's step.
- With no incident over, it makes no other calls.
- For each open PR whose newest shared-CI marker names one of those incidents,
it re-runs the failed jobs (`POST .../actions/runs/{id}/rerun-failed-jobs`)
or that the incident's tracker body lists (the hive tracker's "Open pull
requests failing it" line, or an agent's "Affected PRs:" line; only a
tracker one of the hive's own accounts wrote, since anyone can open an
issue under that title), it re-runs
the failed jobs (`POST .../actions/runs/{id}/rerun-failed-jobs`)
of the newest run per workflow on the PR head when that run concluded
`failure` or `timed_out`, using the hive's App/daemon credential. A newest
run that concluded `cancelled` — how a job cut off by its `timeout-minutes`
is recorded, the shape of the `mutation` incident #17463 — is red too and
is re-run whole (`POST .../actions/runs/{id}/rerun`), since it has no
"failed" job for the other endpoint (hivecommons/hive#17500). It pushes
nothing to the branch, so it applies to every author; PRs with a hold or
`do-not-merge` label are skipped.
nothing to the branch, so it applies to every author, Renovate, Dependabot
and contributor branches at `stage/072-returned-to-contributor` included;
PRs with a hold or `do-not-merge` label are skipped. A PR whose newest
marker names an incident that is not over is still deferred there and is
left alone.
- It then posts one comment, *Shared-CI incident #N closed — re-running
`<check>`*, carrying `<!-- hive-shared-ci-rerun-<n> -->` and
`<check>`* (for an open incident: *#N (still open; `<check>` has passed on
the base branch since it was filed) is over*), carrying `<!-- hive-shared-ci-rerun-<n> -->` and
`<!-- hive-shared-ci-lifted-<n> -->`; either marker keeps the PR to one
re-run per incident. A PR with nothing red on its
head gets no comment. At most 10 PRs are re-run per pass; each re-run is
Expand All @@ -329,13 +347,15 @@ someone re-runs it. The shared-CI rerun sweep
failed jobs have not run again yet, so the stage pass does not hand it
back to its author (072) and the kick builders do not list it for repair
in that window. A "not planned" close gets no re-run and releases the PR
at once.
at once. Once the rerun comment is on the PR its deferral is lifted even
while the incident stays open: the re-run happened after the base
recovered, so a red that survives it is the PR's own.
- It is an unstick step (`shared_ci_rerun` in `api_budget.unstick_steps`): it
keeps running in `conserve` and is shed only in `critical`.

| Key | Default | Meaning |
|---|---|---|
| `shared_ci_rerun.enabled` | `true` (unset) | Set `false` to stop re-running PRs when a `[shared-ci]` incident closes. |
| `shared_ci_rerun.enabled` | `true` (unset) | Set `false` to stop re-running PRs when a `[shared-ci]` incident closes or its check passes on the base again. |

## Shared-CI tracking issues (`shared_ci_issues`)

Expand Down Expand Up @@ -634,7 +654,7 @@ that level, the power the write-back exists to govern?
| Issue filing (`pkg/upstreamwatch/github_filer.go`, `advisory.go`, scanner/deps/regression agents, `dedupe_advisory_close.go`) | agents, advisory digest, upstream watch | **new** issues hive authors, pinned advisory issue + comments | `project.repos` | `issue_file` → L3, no label (this is what listing a repo opts in to; `per_repo.mode: issues-only` keeps only this) |
| Code scanning sweep (`pkg/github/code_scanning.go`, `pkg/codescanning`) | hourly sweep (`code_scanning`, opt-in per repo) | **new** grouped issues from code scanning alerts, alert-table body edits, one closing comment + close when every alert is fixed or dismissed | the hive's own `hive/code-scanning` issues in repos with `code_scanning` on | `issue_file` → L3, no label (own items); see [code-scanning.md](../code-scanning.md) |
| Shared-CI tracking issues (`shared_ci_issues.go`) | 15-min sweep (`shared_ci_issues`) | **new** `[shared-ci] <check> failing across <owner/repo>` issue, body edits of its own issue | repos where one check fails on `shared_ci_issues.min_prs`+ open PRs or trips the red-main breaker | `issue_file` → L3, no label (own items) |
| Shared-CI rerun (`shared_ci_rerun.go`) | CI failure watcher | `rerun-failed-jobs`, `hive-shared-ci-rerun-<n>` comment | PRs with failed shared workflows | `ci_rerun` → L6 + label |
| Shared-CI rerun (`shared_ci_rerun.go`) | CI failure watcher | `rerun-failed-jobs`, `hive-shared-ci-rerun-<n>` comment | PRs deferred to or listed by a `[shared-ci]` incident that closed or whose check passed on their base again | `ci_rerun` → L6 + label |
| Transient-CI rerun (`transient_ci_rerun.go`) | eval cycle, before the stuck-PR reaper | `rerun-failed-jobs`, once per head, no comment | hive-authored and `auto_merge.trusted_bot_authors` red PRs whose every failing run carries `hive-ci-transient` | `ci_rerun` (own items: allowed unless the repo is paused, off or issues-only; trusted-bot PRs: L6 + label, or `writeback.kinds.ci_rerun.unlabeled_min_level`; a refused re-run hands the bot PR to a person with the reason) |
| Stale-red base refresh (`cmd/hive/stale_red_base_refresh.go`, `pkg/github/stale_red_base_refresh.go`) | eval cycle, before the stuck-PR reaper, at most every 5 min | **branch push** (`update-branch` merge of the base), once per head and base tip, no comment | red PRs behind their base whose every failing check is green on the base tip; not drafts, held, non-editable forks, dependency-bot branches, 072 or 074 | `base_refresh` → hive-authored PRs, else L6 + label |
| Missing-check base refresh (`cmd/hive/missing_check_base_refresh.go`, `pkg/github/missing_check_base_refresh.go`) | eval cycle, after the stale-red base refresh, at most every 5 min | **branch push** (`update-branch` merge of the base), once per head and base tip and once per missing check, no comment | PRs behind their base, otherwise green, whose head has no run at all for a required check that runs on the base tip or another open PR; not drafts, held, parked, non-editable forks, dependency-bot branches, 072 or 074 | `base_refresh` → hive-authored PRs, else L6 + label |
Expand Down
6 changes: 4 additions & 2 deletions src/pkg/config/review_config.go
Original file line number Diff line number Diff line change
Expand Up @@ -843,8 +843,10 @@ type SupersessionSweepConfig struct {

// SharedCIRerunConfig gates the shared-CI rerun sweep
// (github.SweepSharedCIReruns): when a `[shared-ci]` incident issue closes,
// Hive re-runs the failed jobs of every open PR stamped with that incident's
// `<!-- hive-shared-ci-<n> -->` marker, once per PR per incident.
// or its check passes on the PR's base tip again while it stays open, Hive
// re-runs the failed jobs of every open PR stamped with that incident's
// `<!-- hive-shared-ci-<n> -->` marker or listed in its tracker, once per PR
// per incident.
type SharedCIRerunConfig struct {
// Enabled is *bool so unset means on; set `shared_ci_rerun.enabled: false`
// to disable.
Expand Down
16 changes: 16 additions & 0 deletions src/pkg/github/base_branch_red.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"slices"
"sort"
"strings"
"time"

gh "github.com/google/go-github/v72/github"
)
Expand Down Expand Up @@ -124,6 +125,7 @@ func (c *Client) baseCheckStatesForRef(ctx context.Context, owner, repo, ref str
opts.Page = resp.NextPage
}
states := map[string]baseCheckState{}
completed := map[string]time.Time{}
for _, cr := range latestCheckRunsByNameAndApp(all) {
name := cr.GetName()
if name == "" || isMetaCheck(name) {
Expand All @@ -141,6 +143,20 @@ func (c *Client) baseCheckStatesForRef(ctx context.Context, owner, repo, ref str
if prev, seen := states[name]; !seen || baseCheckRank(st) > baseCheckRank(prev) {
states[name] = st
}
if at := cr.GetCompletedAt().Time; st == baseCheckGreen && at.After(completed[name]) {
completed[name] = at
}
}
// When each check last went green on the tip, for the shared-CI rerun
// sweep's recovery test; a name another app still runs or failed is
// not green.
for name := range completed {
if states[name] != baseCheckGreen {
delete(completed, name)
}
}
if len(completed) > 0 {
c.baseRed.noteGreen(baseRedMemoryKey(owner, repo, ref), completed)
}
return states, nil
}
Expand Down
83 changes: 82 additions & 1 deletion src/pkg/github/base_branch_red_memory.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,22 @@ const (
baseCheckGreen
)

// baseGreenMemoryTTL bounds the green record: a check's green completion
// older than the shared-CI rerun lookback can no longer show that an open
// incident recovered.
const baseGreenMemoryTTL = DefaultSharedCIRerunLookback

// baseRedMemory is the per-base red set. The zero value is ready to use.
type baseRedMemory struct {
mu sync.Mutex
// bases maps baseRedMemoryKey → check name → when it was last seen red.
bases map[string]map[string]time.Time
now func() time.Time
// greens maps baseRedMemoryKey → lower-cased check name → when the
// tip's newest run of it completed green, from the same reads. The
// shared-CI rerun sweep reads it to tell that an open incident's check
// passed on the base again (sharedCIIncidentRecovered).
greens map[string]map[string]time.Time
now func() time.Time
}

func baseRedMemoryKey(owner, repo, ref string) string {
Expand Down Expand Up @@ -79,6 +89,77 @@ func (m *baseRedMemory) observe(key string, states map[string]baseCheckState) ma
return m.liveLocked(key, now)
}

// noteGreen records when each check in completed last completed green on
// the base tip key, keeping the newest time and dropping times past
// baseGreenMemoryTTL.
func (m *baseRedMemory) noteGreen(key string, completed map[string]time.Time) {
m.mu.Lock()
defer m.mu.Unlock()
now := m.clock()
if m.greens == nil {
m.greens = map[string]map[string]time.Time{}
}
checks := m.greens[key]
if checks == nil {
checks = map[string]time.Time{}
}
for name, at := range completed {
name = strings.ToLower(strings.TrimSpace(name))
if name == "" || at.IsZero() {
continue
}
if prev, ok := checks[name]; !ok || at.After(prev) {
checks[name] = at
}
}
for name, at := range checks {
if now.Sub(at) >= baseGreenMemoryTTL {
delete(checks, name)
}
}
if len(checks) == 0 {
delete(m.greens, key)
return
}
m.greens[key] = checks
}

// anyGreen reports whether any check is recorded green on a base of
// owner/repo, so a sweep with nothing to test skips its issue listing.
func (m *baseRedMemory) anyGreen(owner, repo string) bool {
m.mu.Lock()
defer m.mu.Unlock()
prefix := baseRedMemoryKey(owner, repo, "")
for key, checks := range m.greens {
if strings.HasPrefix(key, prefix) && len(checks) > 0 {
return true
}
}
return false
}

// greenAfter reports whether check completed green on a tip of owner/repo
// after since: on the base ref when ref is set, on any base of the repo
// otherwise.
func (m *baseRedMemory) greenAfter(owner, repo, ref, check string, since time.Time) bool {
m.mu.Lock()
defer m.mu.Unlock()
check = strings.ToLower(strings.TrimSpace(check))
prefix := baseRedMemoryKey(owner, repo, "")
for key, checks := range m.greens {
if ref != "" && key != baseRedMemoryKey(owner, repo, ref) {
continue
}
if ref == "" && !strings.HasPrefix(key, prefix) {
continue
}
if at, ok := checks[check]; ok && at.After(since) {
return true
}
}
return false
}

// remembered is the base's red set for a pass whose read failed, or nil.
func (m *baseRedMemory) remembered(key string) map[string]bool {
m.mu.Lock()
Expand Down
57 changes: 57 additions & 0 deletions src/pkg/github/base_branch_red_memory_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,60 @@ func TestBaseCheckStatesForRefPrecedence(t *testing.T) {
t.Fatalf("states = %v, want %v (red > running > green across apps)", got, want)
}
}

// The green record behind the shared-CI rerun sweep's recovery test: the
// newest green completion per base and check, case-insensitive, scoped to
// the base when one is named, and dropped past its TTL.
func TestBaseRedMemoryGreen(t *testing.T) {
now := time.Date(2026, 11, 18, 12, 0, 0, 0, time.UTC)
m := &baseRedMemory{now: func() time.Time { return now }}
main := baseRedMemoryKey("Acme", "Widget", "main")
opened := now.Add(-6 * time.Hour)
if m.anyGreen("acme", "widget") || m.greenAfter("acme", "widget", "", "contract", opened) {
t.Fatal("nothing recorded yet")
}
m.noteGreen(main, map[string]time.Time{"Contract": now.Add(-8 * time.Hour)})
if !m.anyGreen("acme", "widget") || m.greenAfter("acme", "widget", "main", "contract", opened) {
t.Fatal("a green run from before the incident does not show recovery")
}
m.noteGreen(main, map[string]time.Time{"contract": now.Add(-time.Hour)})
m.noteGreen(main, map[string]time.Time{"contract": now.Add(-7 * time.Hour)})
if !m.greenAfter("acme", "widget", "main", "CONTRACT", opened) || !m.greenAfter("acme", "widget", "", "contract", opened) {
t.Fatal("the newest green completion must be kept, matched case-insensitively")
}
if m.greenAfter("acme", "widget", "v5", "contract", opened) || m.greenAfter("acme", "other", "", "contract", opened) {
t.Fatal("green on main must not show recovery on another base or repo")
}
now = now.Add(baseGreenMemoryTTL)
m.noteGreen(main, nil)
if m.anyGreen("acme", "widget") {
t.Fatal("green record kept past its TTL")
}
}

// The base-red gate's own base-tip read records when each check last went
// green there; a name another app still runs is not green.
func TestBaseCheckStatesForRefRecordsGreen(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/repos/acme/widget/commits/main/check-runs", func(w http.ResponseWriter, r *http.Request) {
_ = json.NewEncoder(w).Encode(map[string]any{"total_count": 3, "check_runs": []map[string]any{
{"id": 1, "name": "contract", "status": "completed", "conclusion": "success", "completed_at": "2026-11-18T11:00:00Z", "app": map[string]any{"id": 1}},
{"id": 2, "name": "build", "status": "completed", "conclusion": "success", "completed_at": "2026-11-18T11:00:00Z", "app": map[string]any{"id": 1}},
{"id": 3, "name": "build", "status": "in_progress", "app": map[string]any{"id": 2}},
}})
})
srv := httptest.NewServer(mux)
defer srv.Close()
c := newTestClient(t, srv, "acme", []string{"widget"})
c.baseRed.now = func() time.Time { return time.Date(2026, 11, 18, 12, 0, 0, 0, time.UTC) }
if _, err := c.baseCheckStatesForRef(context.Background(), "acme", "widget", "main"); err != nil {
t.Fatalf("read: %v", err)
}
opened := time.Date(2026, 11, 18, 5, 44, 0, 0, time.UTC)
if !c.baseRed.greenAfter("acme", "widget", "main", "contract", opened) {
t.Fatal("contract's green run on the tip was not recorded")
}
if c.baseRed.greenAfter("acme", "widget", "main", "build", opened) {
t.Fatal("build still runs under another app; it is not green")
}
}
10 changes: 7 additions & 3 deletions src/pkg/github/shared_ci_incident.go
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,11 @@ type sharedCIIncidentState struct {
// defers the PR until the rerun sweep has stamped its
// `<!-- hive-shared-ci-rerun-<n> -->` marker: the red is the incident's
// until the failed jobs have run again on the repaired base. A
// not-planned close gets no re-run, so it releases the PR at once.
// not-planned close gets no re-run, so it releases the PR at once. Once the
// sweep has stamped that marker the deferral is lifted even while the
// incident stays open: the sweep re-runs a PR for an open incident only
// after its check passed on the base again, so a red that survives the
// re-run is the PR's own.
//
// Every lookup failure returns 0: listing a deferred PR costs one redundant
// triage, hiding a PR that is not deferred stalls it, so the error side is
Expand All @@ -101,7 +105,7 @@ type sharedCIIncidentState struct {
// issue GET.
func (c *Client) openSharedCIIncident(ctx context.Context, pr *PullRequest, texts []string, states map[string]sharedCIIncidentState) int {
incident := newestSharedCIMarker(texts)
if incident == 0 {
if incident == 0 || sharedCIRerunStamped(texts, incident) {
return 0
}
owner, repo := c.splitRepo(pr.Repo)
Expand All @@ -123,7 +127,7 @@ func (c *Client) openSharedCIIncident(ctx context.Context, pr *PullRequest, text
if state.open {
return incident
}
if state.awaitingRerun && !sharedCIRerunStamped(texts, incident) {
if state.awaitingRerun {
return incident
}
return 0
Expand Down
17 changes: 17 additions & 0 deletions src/pkg/github/shared_ci_incident_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -217,3 +217,20 @@ func TestEnrichCIStatus_GreenPRSkipsSharedCILookup(t *testing.T) {
t.Errorf("green PR: comment calls = %d, SharedCIIncident = %d; want 0, 0", commentCalls, prs[0].SharedCIIncident)
}
}

// The rerun sweep re-runs a PR for an open incident only after the
// incident's check passed on the base again, so its rerun marker lifts the
// deferral while the tracker stays open: a red that survives the re-run is
// the PR's own. A PR not yet re-run stays deferred.
func TestEnrichCIStatus_SharedCIRerunMarkerLiftsOpenIncident(t *testing.T) {
srv, _, _ := sharedCIServer(t, map[int][]string{
1: {"<!-- hive-shared-ci-1336 -->", "<!-- hive-shared-ci-rerun-1336 -->\n<!-- hive-shared-ci-lifted-1336 -->"},
2: {"<!-- hive-shared-ci-1336 -->"},
}, map[int]string{1336: "open"})
c := newTestClient(t, srv, "org", []string{"repo"})
prs := []PullRequest{{Repo: "org/repo", Number: 1, HeadSHA: "a1"}, {Repo: "org/repo", Number: 2, HeadSHA: "a2"}}
c.EnrichCIStatus(context.Background(), prs)
if prs[0].SharedCIIncident != 0 || prs[1].SharedCIIncident != 1336 {
t.Fatalf("SharedCIIncident = %d, %d; want 0 (re-run after recovery) and 1336 (still deferred)", prs[0].SharedCIIncident, prs[1].SharedCIIncident)
}
}
Loading
Loading