diff --git a/changelog.d/fixed-shared-ci-rerun-on-recovery.md b/changelog.d/fixed-shared-ci-rerun-on-recovery.md new file mode 100644 index 0000000000..2f99075dfe --- /dev/null +++ b/changelog.d/fixed-shared-ci-rerun-on-recovery.md @@ -0,0 +1 @@ +- The shared-CI rerun sweep no longer waits for a person to close the `[shared-ci]` tracker: once the incident's check has completed green on a PR's base tip after the tracker was filed, the PRs the tracker lists (Renovate, Dependabot and 072 PRs included) and the PRs deferred to it get their failed jobs re-run once each, and the rerun lifts their deferral. The tracker stays open for a person to close. PRs a closed tracker lists are now re-run too, not only the ones a fix lane stamped. diff --git a/src/docs/operator-reference/sweeps-and-operations.md b/src/docs/operator-reference/sweeps-and-operations.md index 81c5a88e47..a8e9f1a106 100644 --- a/src/docs/operator-reference/sweeps-and-operations.md +++ b/src/docs/operator-reference/sweeps-and-operations.md @@ -306,19 +306,37 @@ someone re-runs it. The shared-CI rerun sweep (`src/pkg/github/shared_ci_rerun.go`) closes that gap every ~15 minutes: - It lists `[shared-ci]` issues closed in the last 72h in each watched repo, - ignoring ones closed as "not planned". With none, it makes no other calls. + ignoring ones closed as "not planned". +- It also treats an **open** incident as over once the incident's check has + completed green on a base branch tip after the incident was filed. The hive + never closes a tracker itself, and an outside outage that ends on its own + (projectbluefin/utah#1336: codeberg.org stopped resolving) otherwise left + every PR red until a person closed the issue. The green times come from the + base-red gate's own reads of the base tip (no extra listing), so the open + incidents are listed only for a repo where such a green run was seen. A PR + counts the open incident as over only when the check passed on **its own** + base. The tracker stays open; closing it is still a person's step. +- With no incident over, it makes no other calls. - For each open PR whose newest shared-CI marker names one of those incidents, - it re-runs the failed jobs (`POST .../actions/runs/{id}/rerun-failed-jobs`) + or that the incident's tracker body lists (the hive tracker's "Open pull + requests failing it" line, or an agent's "Affected PRs:" line; only a + tracker one of the hive's own accounts wrote, since anyone can open an + issue under that title), it re-runs + the failed jobs (`POST .../actions/runs/{id}/rerun-failed-jobs`) of the newest run per workflow on the PR head when that run concluded `failure` or `timed_out`, using the hive's App/daemon credential. A newest run that concluded `cancelled` — how a job cut off by its `timeout-minutes` is recorded, the shape of the `mutation` incident #17463 — is red too and is re-run whole (`POST .../actions/runs/{id}/rerun`), since it has no "failed" job for the other endpoint (hivecommons/hive#17500). It pushes - nothing to the branch, so it applies to every author; PRs with a hold or - `do-not-merge` label are skipped. + nothing to the branch, so it applies to every author, Renovate, Dependabot + and contributor branches at `stage/072-returned-to-contributor` included; + PRs with a hold or `do-not-merge` label are skipped. A PR whose newest + marker names an incident that is not over is still deferred there and is + left alone. - It then posts one comment, *Shared-CI incident #N closed — re-running - ``*, carrying `` and + ``* (for an open incident: *#N (still open; `` has passed on + the base branch since it was filed) is over*), carrying `` and ``; either marker keeps the PR to one re-run per incident. A PR with nothing red on its head gets no comment. At most 10 PRs are re-run per pass; each re-run is @@ -329,13 +347,15 @@ someone re-runs it. The shared-CI rerun sweep failed jobs have not run again yet, so the stage pass does not hand it back to its author (072) and the kick builders do not list it for repair in that window. A "not planned" close gets no re-run and releases the PR - at once. + at once. Once the rerun comment is on the PR its deferral is lifted even + while the incident stays open: the re-run happened after the base + recovered, so a red that survives it is the PR's own. - It is an unstick step (`shared_ci_rerun` in `api_budget.unstick_steps`): it keeps running in `conserve` and is shed only in `critical`. | Key | Default | Meaning | |---|---|---| -| `shared_ci_rerun.enabled` | `true` (unset) | Set `false` to stop re-running PRs when a `[shared-ci]` incident closes. | +| `shared_ci_rerun.enabled` | `true` (unset) | Set `false` to stop re-running PRs when a `[shared-ci]` incident closes or its check passes on the base again. | ## Shared-CI tracking issues (`shared_ci_issues`) @@ -634,7 +654,7 @@ that level, the power the write-back exists to govern? | Issue filing (`pkg/upstreamwatch/github_filer.go`, `advisory.go`, scanner/deps/regression agents, `dedupe_advisory_close.go`) | agents, advisory digest, upstream watch | **new** issues hive authors, pinned advisory issue + comments | `project.repos` | `issue_file` → L3, no label (this is what listing a repo opts in to; `per_repo.mode: issues-only` keeps only this) | | Code scanning sweep (`pkg/github/code_scanning.go`, `pkg/codescanning`) | hourly sweep (`code_scanning`, opt-in per repo) | **new** grouped issues from code scanning alerts, alert-table body edits, one closing comment + close when every alert is fixed or dismissed | the hive's own `hive/code-scanning` issues in repos with `code_scanning` on | `issue_file` → L3, no label (own items); see [code-scanning.md](../code-scanning.md) | | Shared-CI tracking issues (`shared_ci_issues.go`) | 15-min sweep (`shared_ci_issues`) | **new** `[shared-ci] failing across ` issue, body edits of its own issue | repos where one check fails on `shared_ci_issues.min_prs`+ open PRs or trips the red-main breaker | `issue_file` → L3, no label (own items) | -| Shared-CI rerun (`shared_ci_rerun.go`) | CI failure watcher | `rerun-failed-jobs`, `hive-shared-ci-rerun-` comment | PRs with failed shared workflows | `ci_rerun` → L6 + label | +| Shared-CI rerun (`shared_ci_rerun.go`) | CI failure watcher | `rerun-failed-jobs`, `hive-shared-ci-rerun-` comment | PRs deferred to or listed by a `[shared-ci]` incident that closed or whose check passed on their base again | `ci_rerun` → L6 + label | | Transient-CI rerun (`transient_ci_rerun.go`) | eval cycle, before the stuck-PR reaper | `rerun-failed-jobs`, once per head, no comment | hive-authored and `auto_merge.trusted_bot_authors` red PRs whose every failing run carries `hive-ci-transient` | `ci_rerun` (own items: allowed unless the repo is paused, off or issues-only; trusted-bot PRs: L6 + label, or `writeback.kinds.ci_rerun.unlabeled_min_level`; a refused re-run hands the bot PR to a person with the reason) | | Stale-red base refresh (`cmd/hive/stale_red_base_refresh.go`, `pkg/github/stale_red_base_refresh.go`) | eval cycle, before the stuck-PR reaper, at most every 5 min | **branch push** (`update-branch` merge of the base), once per head and base tip, no comment | red PRs behind their base whose every failing check is green on the base tip; not drafts, held, non-editable forks, dependency-bot branches, 072 or 074 | `base_refresh` → hive-authored PRs, else L6 + label | | Missing-check base refresh (`cmd/hive/missing_check_base_refresh.go`, `pkg/github/missing_check_base_refresh.go`) | eval cycle, after the stale-red base refresh, at most every 5 min | **branch push** (`update-branch` merge of the base), once per head and base tip and once per missing check, no comment | PRs behind their base, otherwise green, whose head has no run at all for a required check that runs on the base tip or another open PR; not drafts, held, parked, non-editable forks, dependency-bot branches, 072 or 074 | `base_refresh` → hive-authored PRs, else L6 + label | diff --git a/src/pkg/config/review_config.go b/src/pkg/config/review_config.go index acefccc680..a663575d8e 100644 --- a/src/pkg/config/review_config.go +++ b/src/pkg/config/review_config.go @@ -843,8 +843,10 @@ type SupersessionSweepConfig struct { // SharedCIRerunConfig gates the shared-CI rerun sweep // (github.SweepSharedCIReruns): when a `[shared-ci]` incident issue closes, -// Hive re-runs the failed jobs of every open PR stamped with that incident's -// `` marker, once per PR per incident. +// or its check passes on the PR's base tip again while it stays open, Hive +// re-runs the failed jobs of every open PR stamped with that incident's +// `` marker or listed in its tracker, once per PR +// per incident. type SharedCIRerunConfig struct { // Enabled is *bool so unset means on; set `shared_ci_rerun.enabled: false` // to disable. diff --git a/src/pkg/github/base_branch_red.go b/src/pkg/github/base_branch_red.go index a85bd3835d..118d675a03 100644 --- a/src/pkg/github/base_branch_red.go +++ b/src/pkg/github/base_branch_red.go @@ -6,6 +6,7 @@ import ( "slices" "sort" "strings" + "time" gh "github.com/google/go-github/v72/github" ) @@ -124,6 +125,7 @@ func (c *Client) baseCheckStatesForRef(ctx context.Context, owner, repo, ref str opts.Page = resp.NextPage } states := map[string]baseCheckState{} + completed := map[string]time.Time{} for _, cr := range latestCheckRunsByNameAndApp(all) { name := cr.GetName() if name == "" || isMetaCheck(name) { @@ -141,6 +143,20 @@ func (c *Client) baseCheckStatesForRef(ctx context.Context, owner, repo, ref str if prev, seen := states[name]; !seen || baseCheckRank(st) > baseCheckRank(prev) { states[name] = st } + if at := cr.GetCompletedAt().Time; st == baseCheckGreen && at.After(completed[name]) { + completed[name] = at + } + } + // When each check last went green on the tip, for the shared-CI rerun + // sweep's recovery test; a name another app still runs or failed is + // not green. + for name := range completed { + if states[name] != baseCheckGreen { + delete(completed, name) + } + } + if len(completed) > 0 { + c.baseRed.noteGreen(baseRedMemoryKey(owner, repo, ref), completed) } return states, nil } diff --git a/src/pkg/github/base_branch_red_memory.go b/src/pkg/github/base_branch_red_memory.go index 6c8d27f152..6b62c8ff81 100644 --- a/src/pkg/github/base_branch_red_memory.go +++ b/src/pkg/github/base_branch_red_memory.go @@ -34,12 +34,22 @@ const ( baseCheckGreen ) +// baseGreenMemoryTTL bounds the green record: a check's green completion +// older than the shared-CI rerun lookback can no longer show that an open +// incident recovered. +const baseGreenMemoryTTL = DefaultSharedCIRerunLookback + // baseRedMemory is the per-base red set. The zero value is ready to use. type baseRedMemory struct { mu sync.Mutex // bases maps baseRedMemoryKey → check name → when it was last seen red. bases map[string]map[string]time.Time - now func() time.Time + // greens maps baseRedMemoryKey → lower-cased check name → when the + // tip's newest run of it completed green, from the same reads. The + // shared-CI rerun sweep reads it to tell that an open incident's check + // passed on the base again (sharedCIIncidentRecovered). + greens map[string]map[string]time.Time + now func() time.Time } func baseRedMemoryKey(owner, repo, ref string) string { @@ -79,6 +89,77 @@ func (m *baseRedMemory) observe(key string, states map[string]baseCheckState) ma return m.liveLocked(key, now) } +// noteGreen records when each check in completed last completed green on +// the base tip key, keeping the newest time and dropping times past +// baseGreenMemoryTTL. +func (m *baseRedMemory) noteGreen(key string, completed map[string]time.Time) { + m.mu.Lock() + defer m.mu.Unlock() + now := m.clock() + if m.greens == nil { + m.greens = map[string]map[string]time.Time{} + } + checks := m.greens[key] + if checks == nil { + checks = map[string]time.Time{} + } + for name, at := range completed { + name = strings.ToLower(strings.TrimSpace(name)) + if name == "" || at.IsZero() { + continue + } + if prev, ok := checks[name]; !ok || at.After(prev) { + checks[name] = at + } + } + for name, at := range checks { + if now.Sub(at) >= baseGreenMemoryTTL { + delete(checks, name) + } + } + if len(checks) == 0 { + delete(m.greens, key) + return + } + m.greens[key] = checks +} + +// anyGreen reports whether any check is recorded green on a base of +// owner/repo, so a sweep with nothing to test skips its issue listing. +func (m *baseRedMemory) anyGreen(owner, repo string) bool { + m.mu.Lock() + defer m.mu.Unlock() + prefix := baseRedMemoryKey(owner, repo, "") + for key, checks := range m.greens { + if strings.HasPrefix(key, prefix) && len(checks) > 0 { + return true + } + } + return false +} + +// greenAfter reports whether check completed green on a tip of owner/repo +// after since: on the base ref when ref is set, on any base of the repo +// otherwise. +func (m *baseRedMemory) greenAfter(owner, repo, ref, check string, since time.Time) bool { + m.mu.Lock() + defer m.mu.Unlock() + check = strings.ToLower(strings.TrimSpace(check)) + prefix := baseRedMemoryKey(owner, repo, "") + for key, checks := range m.greens { + if ref != "" && key != baseRedMemoryKey(owner, repo, ref) { + continue + } + if ref == "" && !strings.HasPrefix(key, prefix) { + continue + } + if at, ok := checks[check]; ok && at.After(since) { + return true + } + } + return false +} + // remembered is the base's red set for a pass whose read failed, or nil. func (m *baseRedMemory) remembered(key string) map[string]bool { m.mu.Lock() diff --git a/src/pkg/github/base_branch_red_memory_test.go b/src/pkg/github/base_branch_red_memory_test.go index bc5c4dfecf..d04b207c7e 100644 --- a/src/pkg/github/base_branch_red_memory_test.go +++ b/src/pkg/github/base_branch_red_memory_test.go @@ -134,3 +134,60 @@ func TestBaseCheckStatesForRefPrecedence(t *testing.T) { t.Fatalf("states = %v, want %v (red > running > green across apps)", got, want) } } + +// The green record behind the shared-CI rerun sweep's recovery test: the +// newest green completion per base and check, case-insensitive, scoped to +// the base when one is named, and dropped past its TTL. +func TestBaseRedMemoryGreen(t *testing.T) { + now := time.Date(2026, 11, 18, 12, 0, 0, 0, time.UTC) + m := &baseRedMemory{now: func() time.Time { return now }} + main := baseRedMemoryKey("Acme", "Widget", "main") + opened := now.Add(-6 * time.Hour) + if m.anyGreen("acme", "widget") || m.greenAfter("acme", "widget", "", "contract", opened) { + t.Fatal("nothing recorded yet") + } + m.noteGreen(main, map[string]time.Time{"Contract": now.Add(-8 * time.Hour)}) + if !m.anyGreen("acme", "widget") || m.greenAfter("acme", "widget", "main", "contract", opened) { + t.Fatal("a green run from before the incident does not show recovery") + } + m.noteGreen(main, map[string]time.Time{"contract": now.Add(-time.Hour)}) + m.noteGreen(main, map[string]time.Time{"contract": now.Add(-7 * time.Hour)}) + if !m.greenAfter("acme", "widget", "main", "CONTRACT", opened) || !m.greenAfter("acme", "widget", "", "contract", opened) { + t.Fatal("the newest green completion must be kept, matched case-insensitively") + } + if m.greenAfter("acme", "widget", "v5", "contract", opened) || m.greenAfter("acme", "other", "", "contract", opened) { + t.Fatal("green on main must not show recovery on another base or repo") + } + now = now.Add(baseGreenMemoryTTL) + m.noteGreen(main, nil) + if m.anyGreen("acme", "widget") { + t.Fatal("green record kept past its TTL") + } +} + +// The base-red gate's own base-tip read records when each check last went +// green there; a name another app still runs is not green. +func TestBaseCheckStatesForRefRecordsGreen(t *testing.T) { + mux := http.NewServeMux() + mux.HandleFunc("/repos/acme/widget/commits/main/check-runs", func(w http.ResponseWriter, r *http.Request) { + _ = json.NewEncoder(w).Encode(map[string]any{"total_count": 3, "check_runs": []map[string]any{ + {"id": 1, "name": "contract", "status": "completed", "conclusion": "success", "completed_at": "2026-11-18T11:00:00Z", "app": map[string]any{"id": 1}}, + {"id": 2, "name": "build", "status": "completed", "conclusion": "success", "completed_at": "2026-11-18T11:00:00Z", "app": map[string]any{"id": 1}}, + {"id": 3, "name": "build", "status": "in_progress", "app": map[string]any{"id": 2}}, + }}) + }) + srv := httptest.NewServer(mux) + defer srv.Close() + c := newTestClient(t, srv, "acme", []string{"widget"}) + c.baseRed.now = func() time.Time { return time.Date(2026, 11, 18, 12, 0, 0, 0, time.UTC) } + if _, err := c.baseCheckStatesForRef(context.Background(), "acme", "widget", "main"); err != nil { + t.Fatalf("read: %v", err) + } + opened := time.Date(2026, 11, 18, 5, 44, 0, 0, time.UTC) + if !c.baseRed.greenAfter("acme", "widget", "main", "contract", opened) { + t.Fatal("contract's green run on the tip was not recorded") + } + if c.baseRed.greenAfter("acme", "widget", "main", "build", opened) { + t.Fatal("build still runs under another app; it is not green") + } +} diff --git a/src/pkg/github/shared_ci_incident.go b/src/pkg/github/shared_ci_incident.go index e48b835419..7063bed01f 100644 --- a/src/pkg/github/shared_ci_incident.go +++ b/src/pkg/github/shared_ci_incident.go @@ -92,7 +92,11 @@ type sharedCIIncidentState struct { // defers the PR until the rerun sweep has stamped its // `` marker: the red is the incident's // until the failed jobs have run again on the repaired base. A -// not-planned close gets no re-run, so it releases the PR at once. +// not-planned close gets no re-run, so it releases the PR at once. Once the +// sweep has stamped that marker the deferral is lifted even while the +// incident stays open: the sweep re-runs a PR for an open incident only +// after its check passed on the base again, so a red that survives the +// re-run is the PR's own. // // Every lookup failure returns 0: listing a deferred PR costs one redundant // triage, hiding a PR that is not deferred stalls it, so the error side is @@ -101,7 +105,7 @@ type sharedCIIncidentState struct { // issue GET. func (c *Client) openSharedCIIncident(ctx context.Context, pr *PullRequest, texts []string, states map[string]sharedCIIncidentState) int { incident := newestSharedCIMarker(texts) - if incident == 0 { + if incident == 0 || sharedCIRerunStamped(texts, incident) { return 0 } owner, repo := c.splitRepo(pr.Repo) @@ -123,7 +127,7 @@ func (c *Client) openSharedCIIncident(ctx context.Context, pr *PullRequest, text if state.open { return incident } - if state.awaitingRerun && !sharedCIRerunStamped(texts, incident) { + if state.awaitingRerun { return incident } return 0 diff --git a/src/pkg/github/shared_ci_incident_test.go b/src/pkg/github/shared_ci_incident_test.go index c335931eb9..5cf8f4587d 100644 --- a/src/pkg/github/shared_ci_incident_test.go +++ b/src/pkg/github/shared_ci_incident_test.go @@ -217,3 +217,20 @@ func TestEnrichCIStatus_GreenPRSkipsSharedCILookup(t *testing.T) { t.Errorf("green PR: comment calls = %d, SharedCIIncident = %d; want 0, 0", commentCalls, prs[0].SharedCIIncident) } } + +// The rerun sweep re-runs a PR for an open incident only after the +// incident's check passed on the base again, so its rerun marker lifts the +// deferral while the tracker stays open: a red that survives the re-run is +// the PR's own. A PR not yet re-run stays deferred. +func TestEnrichCIStatus_SharedCIRerunMarkerLiftsOpenIncident(t *testing.T) { + srv, _, _ := sharedCIServer(t, map[int][]string{ + 1: {"", "\n"}, + 2: {""}, + }, map[int]string{1336: "open"}) + c := newTestClient(t, srv, "org", []string{"repo"}) + prs := []PullRequest{{Repo: "org/repo", Number: 1, HeadSHA: "a1"}, {Repo: "org/repo", Number: 2, HeadSHA: "a2"}} + c.EnrichCIStatus(context.Background(), prs) + if prs[0].SharedCIIncident != 0 || prs[1].SharedCIIncident != 1336 { + t.Fatalf("SharedCIIncident = %d, %d; want 0 (re-run after recovery) and 1336 (still deferred)", prs[0].SharedCIIncident, prs[1].SharedCIIncident) + } +} diff --git a/src/pkg/github/shared_ci_rerun.go b/src/pkg/github/shared_ci_rerun.go index 5242fdc7a2..f719b7f44e 100644 --- a/src/pkg/github/shared_ci_rerun.go +++ b/src/pkg/github/shared_ci_rerun.go @@ -76,7 +76,8 @@ type SharedCIRerunResult struct { } // sharedCIRerunSeen memoizes a PR that needed no action, so later passes skip -// re-listing its comments until the PR or the closed-incident set changes. +// re-listing its comments until the PR or the set of incidents over for its +// base changes. type sharedCIRerunSeen struct { updatedAt time.Time incidents string @@ -111,12 +112,18 @@ func (c *Client) hasSharedCIRerunBlocks(owner, repo string) bool { } // SweepSharedCIReruns re-runs the failed jobs on open PRs that a fix lane -// deferred to a [shared-ci] incident (``) once that -// incident is closed, so a PR left red by the broken baseline gets a fresh -// verdict without a human pushing or clicking re-run (hivecommons/hive#12253). -// It pushes nothing to the branch, so it applies to every author. Each PR is -// re-run at most once per incident: the `` -// comment it leaves afterwards is the dedupe record. +// deferred to a [shared-ci] incident (``), or that +// the incident's tracker lists, once that incident is over, so a PR left red +// by the broken baseline gets a fresh verdict without a human pushing or +// clicking re-run (hivecommons/hive#12253). An incident is over when it is +// closed, or, while it stays open, once its check has completed green on the +// PR's base tip after the incident was filed: the hive never closes a +// tracker itself (#17546), and an outside outage that ends on its own +// (projectbluefin/utah#1336, codeberg.org DNS) left every PR red until a +// person closed it. It pushes nothing to the branch, so it applies to every +// author, bot branches included. Each PR is re-run at most once per +// incident: the `` comment it leaves +// afterwards is the dedupe record. func (c *Client) SweepSharedCIReruns(ctx context.Context, opts SharedCIRerunOptions) (*SharedCIRerunResult, error) { if c == nil { return nil, ErrNoGitHubClient @@ -141,15 +148,19 @@ func (c *Client) SweepSharedCIReruns(ctx context.Context, opts SharedCIRerunOpti break } owner, repoName := c.splitRepo(repo) - closed, err := c.recentlyClosedSharedCIIncidents(ctx, owner, repoName, now.Add(-lookback)) + incs := &sharedCIRerunIncidents{listed: map[int]map[int]bool{}} + closed, err := c.recentlyClosedSharedCIIncidents(ctx, owner, repoName, now.Add(-lookback), incs.listed) if err != nil { return result, err } - if len(closed) == 0 && !c.hasSharedCIRerunBlocks(owner, repoName) { + incs.closed = closed + if incs.recovered, err = c.recoveredOpenSharedCIIncidents(ctx, owner, repoName, incs.listed); err != nil { + return result, err + } + if len(closed) == 0 && len(incs.recovered) == 0 && !c.hasSharedCIRerunBlocks(owner, repoName) { continue } - result.Incidents += len(closed) - incidentKey := sharedCIIncidentSetKey(closed) + result.Incidents += len(closed) + len(incs.recovered) prs, err := c.listOpenPRsForSupersessionSweep(ctx, owner, repoName) if err != nil { return result, err @@ -162,7 +173,7 @@ func (c *Client) SweepSharedCIReruns(ctx context.Context, opts SharedCIRerunOpti continue } result.Seen++ - event, ok, err := c.trySharedCIRerun(ctx, repo, owner, repoName, pr, closed, incidentKey) + event, ok, err := c.trySharedCIRerun(ctx, repo, owner, repoName, pr, incs) if err != nil { return result, err } @@ -179,10 +190,127 @@ func (c *Client) SweepSharedCIReruns(ctx context.Context, opts SharedCIRerunOpti return result, nil } +// sharedCIRerunIncidents is what one repository's [shared-ci] incidents +// say for one sweep pass. +type sharedCIRerunIncidents struct { + // closed are the incidents closed as completed within the lookback. + closed map[int]bool + // recovered are the open incidents whose check has completed green on a + // base tip of the repository since the incident was filed. A PR is due + // a rerun for one only when the check passed on its own base. + recovered map[int]sharedCIOpenIncident + // listed maps an incident to the PRs its tracker body names. + listed map[int]map[int]bool +} + +// sharedCIOpenIncident is an open incident's check and filing time. +type sharedCIOpenIncident struct { + check string + opened time.Time +} + +// eligibleFor is the incidents over for a PR on baseRef: every closed one, +// and each open one whose check passed on that base after it was filed (on +// any base of the repo when the PR's base is unknown). +func (s *sharedCIRerunIncidents) eligibleFor(c *Client, owner, repo, baseRef string) map[int]bool { + out := maps.Clone(s.closed) + if out == nil { + out = map[int]bool{} + } + for n, inc := range s.recovered { + if c.baseRed.greenAfter(owner, repo, baseRef, inc.check, inc.opened) { + out[n] = true + } + } + return out +} + +// listedFor is the incidents whose tracker lists PR number, ascending. +func (s *sharedCIRerunIncidents) listedFor(number int) []int { + var out []int + for n, prs := range s.listed { + if prs[number] { + out = append(out, n) + } + } + sort.Ints(out) + return out +} + +// sharedCITrackerPRLineRE matches the line of a [shared-ci] tracker body +// that lists the PRs failing its check: the hive's own tracker ("- Open +// pull requests failing it (18): #1, #2") or an agent's ("Affected PRs: +// #1274, #1295"). +var sharedCITrackerPRLineRE = regexp.MustCompile(`(?im)^[ \t>*-]*(?:open pull requests failing it(?: \(\d+\))?|affected (?:prs|pull requests))\**:\**(.*)$`) + +var sharedCITrackerPRRefRE = regexp.MustCompile(`#([1-9][0-9]*)\b`) + +// sharedCITrackerListedPRs returns the PR numbers a tracker body lists. +func sharedCITrackerListedPRs(body string) map[int]bool { + var out map[int]bool + for _, line := range sharedCITrackerPRLineRE.FindAllStringSubmatch(body, -1) { + for _, m := range sharedCITrackerPRRefRE.FindAllStringSubmatch(line[1], -1) { + if n, err := strconv.Atoi(m[1]); err == nil { + if out == nil { + out = map[int]bool{} + } + out[n] = true + } + } + } + return out +} + +// trustedSharedCITrackerListedPRs is sharedCITrackerListedPRs for a tracker +// one of the hive's own accounts wrote (its sweep's tracker, or an agent's +// filed for help): anyone can open an issue under the `[shared-ci]` title, and +// a stranger's list must not pick which PRs the hive re-runs (#18946 review). +// A hive that cannot name its own accounts trusts every author, as the other +// hive-authorship checks do. Marker-stamped PRs are unaffected. +func (c *Client) trustedSharedCITrackerListedPRs(issue *gh.Issue) map[int]bool { + prs := sharedCITrackerListedPRs(issue.GetBody()) + if len(prs) == 0 || c.isHiveAccount(issue.GetUser()) || !c.knowsOwnAccounts() { + return prs + } + return nil +} + +// recoveredOpenSharedCIIncidents returns the open [shared-ci] incidents in +// owner/repo whose check has completed green on a base tip since the +// incident was filed, and records the PRs each one's tracker lists. The +// green times come from the base-red gate's own base-tip reads +// (baseRedMemory.noteGreen); with none recorded for the repo it lists +// nothing. +func (c *Client) recoveredOpenSharedCIIncidents(ctx context.Context, owner, repo string, listed map[int]map[int]bool) (map[int]sharedCIOpenIncident, error) { + if !c.baseRed.anyGreen(owner, repo) { + return nil, nil + } + open, err := c.openSharedCIIncidentIssues(ctx, owner, repo) + if err != nil { + return nil, err + } + var out map[int]sharedCIOpenIncident + for check, issue := range open { + opened := issue.GetCreatedAt().Time + if opened.IsZero() || !c.baseRed.greenAfter(owner, repo, "", check, opened) { + continue + } + if out == nil { + out = map[int]sharedCIOpenIncident{} + } + out[issue.GetNumber()] = sharedCIOpenIncident{check: check, opened: opened} + if prs := c.trustedSharedCITrackerListedPRs(issue); len(prs) > 0 { + listed[issue.GetNumber()] = prs + } + } + return out, nil +} + // recentlyClosedSharedCIIncidents returns the [shared-ci] incident issues in -// owner/repo closed at or after since. Incidents closed as not planned are -// excluded, matching .github/scripts/pr-auto-update.sh: nothing was fixed. -func (c *Client) recentlyClosedSharedCIIncidents(ctx context.Context, owner, repo string, since time.Time) (map[int]bool, error) { +// owner/repo closed at or after since, and records in listed the PRs each +// one's tracker body names. Incidents closed as not planned are excluded, +// matching .github/scripts/pr-auto-update.sh: nothing was fixed. +func (c *Client) recentlyClosedSharedCIIncidents(ctx context.Context, owner, repo string, since time.Time, listed map[int]map[int]bool) (map[int]bool, error) { opts := &gh.IssueListByRepoOptions{ State: "closed", Sort: "updated", @@ -210,6 +338,9 @@ func (c *Client) recentlyClosedSharedCIIncidents(ctx context.Context, owner, rep continue } out[issue.GetNumber()] = true + if prs := c.trustedSharedCITrackerListedPRs(issue); len(prs) > 0 && listed != nil { + listed[issue.GetNumber()] = prs + } } if resp == nil || resp.NextPage == 0 { break @@ -232,15 +363,20 @@ func sharedCIIncidentSetKey(incidents map[int]bool) string { return strings.Join(parts, ",") } -// pendingSharedCIReruns returns the closed incidents a PR's comments (oldest -// first) defer it to that have not been re-run yet. A PR whose newest marker -// names an incident outside closed is still deferred elsewhere and gets none. -func pendingSharedCIReruns(texts []string, closed map[int]bool) []int { +// pendingSharedCIReruns returns the incidents in over (closed, or open and +// recovered) that a PR's comments (oldest first) defer it to, or whose +// tracker lists it (listed), and that have not been re-run yet. A PR whose +// newest marker names an incident outside over is still deferred elsewhere +// and gets none. +func pendingSharedCIReruns(texts []string, over map[int]bool, listed ...int) []int { newest := newestSharedCIMarker(texts) - if newest == 0 || !closed[newest] { + if newest != 0 && !over[newest] { return nil } stamped := map[int]bool{} + for _, n := range listed { + stamped[n] = true + } done := map[int]bool{} for _, text := range texts { for _, m := range sharedCIMarkerRE.FindAllStringSubmatch(text, -1) { @@ -261,7 +397,7 @@ func pendingSharedCIReruns(texts []string, closed map[int]bool) []int { } var out []int for n := range stamped { - if closed[n] && !done[n] { + if over[n] && !done[n] { out = append(out, n) } } @@ -269,10 +405,15 @@ func pendingSharedCIReruns(texts []string, closed map[int]bool) []int { return out } -func (c *Client) trySharedCIRerun(ctx context.Context, repo, owner, repoName string, pr *gh.PullRequest, closed map[int]bool, incidentKey string) (SharedCIRerunEvent, bool, error) { +func (c *Client) trySharedCIRerun(ctx context.Context, repo, owner, repoName string, pr *gh.PullRequest, incs *sharedCIRerunIncidents) (SharedCIRerunEvent, bool, error) { if labels := labelNames(pr.Labels); c.isHeld(labels) || c.isExempt(labels) { return SharedCIRerunEvent{}, false, nil } + // The memo is keyed on the incidents over for this PR's own base, not + // the repo's: a PR seen while its base was still red must be looked at + // again once that base goes green, though nothing else changed. + over := incs.eligibleFor(c, owner, repoName, pr.GetBase().GetRef()) + incidentKey := sharedCIIncidentSetKey(over) seenKey := fmt.Sprintf("%s/%s#%d", owner, repoName, pr.GetNumber()) updatedAt := pr.GetUpdatedAt().Time c.sharedCIRerunMu.Lock() @@ -312,18 +453,14 @@ func (c *Client) trySharedCIRerun(ctx context.Context, repo, owner, repoName str if prev.block.RunID != 0 && prev.block.HeadSHA == pr.GetHead().GetSHA() { // Discovery's lookback must not strand an already-observed refusal. // Keep its incident eligible without broadening discovery for other PRs. - closed = maps.Clone(closed) - if closed == nil { - closed = map[int]bool{} - } for _, raw := range strings.Split(prev.incidents, ",") { if n, err := strconv.Atoi(raw); err == nil && n > 0 { - closed[n] = true + over[n] = true } } - incidentKey = sharedCIIncidentSetKey(closed) + incidentKey = sharedCIIncidentSetKey(over) } - incidents := pendingSharedCIReruns(texts, closed) + incidents := pendingSharedCIReruns(texts, over, incs.listedFor(pr.GetNumber())...) if len(incidents) == 0 { markSeen() return SharedCIRerunEvent{}, false, nil @@ -381,7 +518,13 @@ func (c *Client) trySharedCIRerun(ctx context.Context, repo, owner, repoName str return event, true, nil } - body := renderSharedCIRerunComment(incidents, sha, rerun) + recovered := map[int]string{} + for _, n := range incidents { + if inc, ok := incs.recovered[n]; ok && !incs.closed[n] { + recovered[n] = inc.check + } + } + body := renderSharedCIRerunComment(incidents, recovered, sha, rerun) if _, _, err := c.client.Issues.CreateComment(ctx, owner, repoName, pr.GetNumber(), &gh.IssueComment{Body: &body}); err != nil { return event, false, fmt.Errorf("stamping shared-CI rerun marker on %s#%d: %w", repo, pr.GetNumber(), err) } @@ -479,13 +622,19 @@ func (c *Client) latestFailedWorkflowRuns(ctx context.Context, owner, repo, sha return out, nil } -func renderSharedCIRerunComment(incidents []int, sha string, runs []*gh.WorkflowRun) string { +// renderSharedCIRerunComment is the rerun comment for incidents; recovered +// names the still-open ones (incident → check) whose check passed on the +// base again. +func renderSharedCIRerunComment(incidents []int, recovered map[int]string, sha string, runs []*gh.WorkflowRun) string { var b strings.Builder refs := make([]string, len(incidents)) for i, n := range incidents { fmt.Fprintln(&b, sharedCIRerunMarker(n)) fmt.Fprintln(&b, sharedCILiftedMarker(n)) refs[i] = "#" + strconv.Itoa(n) + if check, ok := recovered[n]; ok { + refs[i] += " (still open; `" + check + "` has passed on the base branch since it was filed)" + } } short := sha if len(short) > 7 { @@ -499,7 +648,11 @@ func renderSharedCIRerunComment(incidents []int, sha string, runs []*gh.Workflow } names = append(names, "`"+name+"`") } - fmt.Fprintf(&b, "Shared-CI incident %s closed — re-running %s on `%s`:\n\n", strings.Join(refs, ", "), strings.Join(names, ", "), short) + over := "closed" + if len(recovered) > 0 { + over = "is over" + } + fmt.Fprintf(&b, "Shared-CI incident %s %s — re-running %s on `%s`:\n\n", strings.Join(refs, ", "), over, strings.Join(names, ", "), short) for _, run := range runs { name := run.GetName() if name == "" { diff --git a/src/pkg/github/shared_ci_rerun_test.go b/src/pkg/github/shared_ci_rerun_test.go index 8c2506201a..69fa1eeafd 100644 --- a/src/pkg/github/shared_ci_rerun_test.go +++ b/src/pkg/github/shared_ci_rerun_test.go @@ -7,6 +7,7 @@ import ( "net/http/httptest" "reflect" "regexp" + "sort" "strconv" "strings" "sync" @@ -20,6 +21,20 @@ type sharedCIRerunIssue struct { StateReason string ClosedAt time.Time IsPR bool + // Open serves the issue from the open listing (CreatedAt, Body) instead + // of the closed one. + Open bool + CreatedAt time.Time + Body string + // Author is the issue's author; empty means the hive's App, hive[bot]. + Author string +} + +func (is sharedCIRerunIssue) author() string { + if is.Author != "" { + return is.Author + } + return "hive[bot]" } type sharedCIRerunRun struct { @@ -34,6 +49,8 @@ type sharedCIRerunRun struct { type sharedCIRerunPR struct { Number int SHA string + Base string + Author string Labels []string Comments []string Runs []sharedCIRerunRun @@ -49,6 +66,7 @@ type sharedCIRerunFake struct { commentLists map[int]int forbidRerun bool forbidAfter int + openLists int } func newSharedCIRerunFake(t *testing.T, issues []sharedCIRerunIssue, prs []*sharedCIRerunPR) (*httptest.Server, *sharedCIRerunFake) { @@ -66,12 +84,26 @@ func newSharedCIRerunFake(t *testing.T, issues []sharedCIRerunIssue, prs []*shar defer f.mu.Unlock() switch { case r.URL.Path == "/repos/org/repo/issues" && r.Method == http.MethodGet: + if r.URL.Query().Get("state") == "open" { + f.openLists++ + out := []map[string]any{} + for _, is := range f.issues { + if is.Open { + out = append(out, map[string]any{"number": is.Number, "title": is.Title, "state": "open", "created_at": is.CreatedAt.Format(time.RFC3339), "body": is.Body, "user": map[string]any{"login": is.author()}}) + } + } + _ = json.NewEncoder(w).Encode(out) + return + } if r.URL.Query().Get("state") != "closed" || r.URL.Query().Get("since") == "" { t.Errorf("issue list query = %q, want state=closed with since", r.URL.RawQuery) } out := []map[string]any{} for _, is := range f.issues { - item := map[string]any{"number": is.Number, "title": is.Title, "state": "closed", "closed_at": is.ClosedAt.Format(time.RFC3339)} + if is.Open { + continue + } + item := map[string]any{"number": is.Number, "title": is.Title, "state": "closed", "closed_at": is.ClosedAt.Format(time.RFC3339), "body": is.Body, "user": map[string]any{"login": is.author()}} if is.StateReason != "" { item["state_reason"] = is.StateReason } @@ -92,7 +124,15 @@ func newSharedCIRerunFake(t *testing.T, issues []sharedCIRerunIssue, prs []*shar for _, l := range pr.Labels { labels = append(labels, map[string]any{"name": l}) } - out = append(out, map[string]any{"number": pr.Number, "state": "open", "updated_at": updated, "labels": labels, "head": map[string]any{"sha": pr.SHA}}) + base := pr.Base + if base == "" { + base = "main" + } + author := pr.Author + if author == "" { + author = "contributor" + } + out = append(out, map[string]any{"number": pr.Number, "state": "open", "updated_at": updated, "labels": labels, "head": map[string]any{"sha": pr.SHA}, "base": map[string]any{"ref": base}, "user": map[string]any{"login": author}}) } _ = json.NewEncoder(w).Encode(out) case commentsRE.MatchString(r.URL.Path): @@ -453,3 +493,226 @@ func TestSharedCIRerunPartialPermissionFailureDoesNotFinishIncident(t *testing.T t.Fatal("partial recovery hid outstanding permission block") } } + +// openOutageIncident is projectbluefin/utah#1336 in miniature: the scanner's +// open tracker for `contract`, filed while codeberg.org did not resolve, +// listing the PRs it found red. +func openOutageIncident(now time.Time, body string) sharedCIRerunIssue { + return sharedCIRerunIssue{Number: 1336, Title: "[shared-ci] contract failing across org/repo", Open: true, CreatedAt: now.Add(-6 * time.Hour), Body: body} +} + +// With the tracker still open, once `contract` has completed green on the +// base tip after the tracker was filed, the PRs it lists (no marker needed: +// a Renovate PR and a 072 PR included) and the PRs deferred to it get their +// failed jobs re-run once each. An unlisted, unmarked PR is left alone, and +// the tracker is never closed. +func TestSweepSharedCIReruns_OpenIncidentRecoveredOnBase(t *testing.T) { + now := sharedCIRerunNow() + issues := []sharedCIRerunIssue{openOutageIncident(now, "**Summary:** codeberg.org does not resolve.\n\nAffected PRs: #2, #3, #9.\n")} + prs := []*sharedCIRerunPR{ + {Number: 1, SHA: "aaaaaaaaaa", Comments: []string{"deferring\n"}, Runs: []sharedCIRerunRun{failedRun(11, 1)}}, + {Number: 2, SHA: "bbbbbbbbbb", Author: "renovate[bot]", Runs: []sharedCIRerunRun{failedRun(21, 1)}}, + {Number: 3, SHA: "cccccccccc", Labels: []string{"stage/072-returned-to-contributor"}, Runs: []sharedCIRerunRun{failedRun(31, 1)}}, + {Number: 4, SHA: "dddddddddd", Runs: []sharedCIRerunRun{failedRun(41, 1)}}, + } + srv, fake := newSharedCIRerunFake(t, issues, prs) + c := newTestClient(t, srv, "org", []string{"org/repo"}) + c.SetAppBotLogin("hive[bot]") + c.baseRed.now = sharedCIRerunNow + c.baseRed.noteGreen(baseRedMemoryKey("org", "repo", "main"), map[string]time.Time{"contract": now.Add(-time.Hour)}) + + opts := SharedCIRerunOptions{Now: sharedCIRerunNow} + res, err := c.SweepSharedCIReruns(context.Background(), opts) + if err != nil { + t.Fatalf("sweep: %v", err) + } + reruns, posted, _ := fake.snap() + if !reflect.DeepEqual(reruns, []int64{11, 21, 31}) { + t.Fatalf("reruns = %v, want PRs 1 (marker), 2 (listed, Renovate) and 3 (listed, 072)", reruns) + } + if res.Incidents != 1 || len(res.Rerun) != 3 { + t.Fatalf("result = %+v", res) + } + for _, n := range []int{1, 2, 3} { + if len(posted[n]) != 1 || !strings.Contains(posted[n][0], "") || !strings.Contains(posted[n][0], "still open; `contract` has passed on the base branch") { + t.Fatalf("PR %d rerun comment = %q", n, posted[n]) + } + } + if len(posted[4]) != 0 { + t.Fatalf("unlisted, unmarked PR 4 got %q", posted[4]) + } + + // Once per PR per incident: the next pass, with the checks still red + // for a reason of their own, re-runs nothing. + if res, err = c.SweepSharedCIReruns(context.Background(), opts); err != nil || len(res.Rerun) != 0 { + t.Fatalf("second pass: result=%+v err=%v", res, err) + } + if reruns, _, _ = fake.snap(); len(reruns) != 3 { + t.Fatalf("second pass re-ran again: %v", reruns) + } +} + +// No recovery, no rerun: the base tip's check has not passed since the +// tracker was filed (its last green run predates the outage), or it passed +// only on another base; with no green recorded at all the sweep does not +// even list the open incidents. +func TestSweepSharedCIReruns_OpenIncidentNotRecovered(t *testing.T) { + now := sharedCIRerunNow() + body := "Affected PRs: #1, #2.\n" + for _, tc := range []struct { + name string + greenKey string + greenAt time.Time + base string + wantLists int + }{ + {name: "no green recorded", wantLists: 0}, + {name: "green before the tracker was filed", greenKey: "main", greenAt: now.Add(-8 * time.Hour), wantLists: 1}, + {name: "green on another base", greenKey: "main", greenAt: now.Add(-time.Hour), base: "v5", wantLists: 1}, + } { + t.Run(tc.name, func(t *testing.T) { + prs := []*sharedCIRerunPR{ + {Number: 1, SHA: "aaaaaaaaaa", Base: tc.base, Comments: []string{""}, Runs: []sharedCIRerunRun{failedRun(11, 1)}}, + {Number: 2, SHA: "bbbbbbbbbb", Base: tc.base, Runs: []sharedCIRerunRun{failedRun(21, 1)}}, + } + srv, fake := newSharedCIRerunFake(t, []sharedCIRerunIssue{openOutageIncident(now, body)}, prs) + c := newTestClient(t, srv, "org", []string{"org/repo"}) + c.SetAppBotLogin("hive[bot]") + c.baseRed.now = sharedCIRerunNow + if tc.greenKey != "" { + c.baseRed.noteGreen(baseRedMemoryKey("org", "repo", tc.greenKey), map[string]time.Time{"contract": tc.greenAt}) + } + if _, err := c.SweepSharedCIReruns(context.Background(), SharedCIRerunOptions{Now: sharedCIRerunNow}); err != nil { + t.Fatalf("sweep: %v", err) + } + reruns, posted, _ := fake.snap() + if len(reruns) != 0 || len(posted) != 0 { + t.Fatalf("reruns=%v posted=%v, want nothing before the base recovers", reruns, posted) + } + fake.mu.Lock() + lists := fake.openLists + fake.mu.Unlock() + if lists != tc.wantLists { + t.Fatalf("open incident listings = %d, want %d", lists, tc.wantLists) + } + }) + } +} + +// A closed incident's tracker list counts too: a listed PR no fix lane ever +// stamped is re-run after the close, like a stamped one. +func TestSweepSharedCIReruns_ClosedIncidentRerunsListedPRs(t *testing.T) { + now := sharedCIRerunNow() + issues := []sharedCIRerunIssue{{Number: 500, Title: "[shared-ci] lint failing across org/repo", ClosedAt: now.Add(-time.Hour), + Body: RenderSharedCIIssueBody(SharedCICluster{Repo: "org/repo", Check: "lint", PRs: []int{1}})}} + prs := []*sharedCIRerunPR{ + {Number: 1, SHA: "aaaaaaaaaa", Author: "dependabot[bot]", Runs: []sharedCIRerunRun{failedRun(11, 1)}}, + {Number: 2, SHA: "bbbbbbbbbb", Runs: []sharedCIRerunRun{failedRun(21, 1)}}, + } + srv, fake := newSharedCIRerunFake(t, issues, prs) + c := newTestClient(t, srv, "org", []string{"org/repo"}) + c.SetAppBotLogin("hive[bot]") + if _, err := c.SweepSharedCIReruns(context.Background(), SharedCIRerunOptions{Now: sharedCIRerunNow}); err != nil { + t.Fatalf("sweep: %v", err) + } + reruns, posted, _ := fake.snap() + if !reflect.DeepEqual(reruns, []int64{11}) || len(posted[1]) != 1 || !strings.Contains(posted[1][0], "Shared-CI incident #500 closed") { + t.Fatalf("reruns=%v posted=%v, want listed PR 1 only", reruns, posted) + } +} + +func TestSharedCITrackerListedPRs(t *testing.T) { + hive := RenderSharedCIIssueBody(SharedCICluster{Repo: "o/r", Check: "contract", PRs: []int{1274, 1295}}) + scanner := "Check: `contract` on projectbluefin/utah.\n\nAffected PRs: #1274, #1295, #1296.\n\nRelated: open PR #1191 (retry transient Codeberg submodule fetch failures)." + bold := "**Affected PRs:** #7, #8" + for _, tc := range []struct { + name, body string + want []int + }{ + {name: "hive tracker", body: hive, want: []int{1274, 1295}}, + {name: "scanner tracker", body: scanner, want: []int{1274, 1295, 1296}}, + {name: "bold label", body: bold, want: []int{7, 8}}, + {name: "no list", body: "Related: open PR #1191", want: nil}, + } { + got := sharedCITrackerListedPRs(tc.body) + var nums []int + for n := range got { + nums = append(nums, n) + } + sort.Ints(nums) + if !reflect.DeepEqual(nums, tc.want) { + t.Errorf("%s: listed = %v, want %v", tc.name, nums, tc.want) + } + } +} + +func TestPendingSharedCIRerunsListed(t *testing.T) { + over := map[int]bool{1336: true} + if got := pendingSharedCIReruns(nil, over, 1336); !reflect.DeepEqual(got, []int{1336}) { + t.Fatalf("listed, unmarked = %v", got) + } + if got := pendingSharedCIReruns([]string{""}, over, 1336); got != nil { + t.Fatalf("listed, already re-run = %v", got) + } + if got := pendingSharedCIReruns([]string{""}, over, 1336); got != nil { + t.Fatalf("listed but deferred to another incident still open = %v", got) + } + if got := pendingSharedCIReruns(nil, map[int]bool{}, 1336); got != nil { + t.Fatalf("listed by an incident that is not over = %v", got) + } +} + +// A PR on a base that has not recovered yet is looked at again once that +// base goes green, though neither the PR nor the repo's incident set changed +// (#18946 review): the "nothing to do" memo is keyed on the incidents over +// for the PR's own base. +func TestSweepSharedCIReruns_SecondBaseRecoversLater(t *testing.T) { + now := sharedCIRerunNow() + prs := []*sharedCIRerunPR{ + {Number: 1, SHA: "aaaaaaaaaa", Base: "main", Runs: []sharedCIRerunRun{failedRun(11, 1)}}, + {Number: 2, SHA: "bbbbbbbbbb", Base: "v5", Runs: []sharedCIRerunRun{failedRun(21, 1)}}, + } + srv, fake := newSharedCIRerunFake(t, []sharedCIRerunIssue{openOutageIncident(now, "Affected PRs: #1, #2.\n")}, prs) + c := newTestClient(t, srv, "org", []string{"org/repo"}) + c.SetAppBotLogin("hive[bot]") + c.baseRed.now = sharedCIRerunNow + c.baseRed.noteGreen(baseRedMemoryKey("org", "repo", "main"), map[string]time.Time{"contract": now.Add(-time.Hour)}) + opts := SharedCIRerunOptions{Now: sharedCIRerunNow} + if _, err := c.SweepSharedCIReruns(context.Background(), opts); err != nil { + t.Fatalf("first sweep: %v", err) + } + if reruns, _, _ := fake.snap(); !reflect.DeepEqual(reruns, []int64{11}) { + t.Fatalf("first sweep reruns = %v, want only PR 1 (main recovered, v5 not yet)", reruns) + } + c.baseRed.noteGreen(baseRedMemoryKey("org", "repo", "v5"), map[string]time.Time{"contract": now.Add(-time.Minute)}) + if _, err := c.SweepSharedCIReruns(context.Background(), opts); err != nil { + t.Fatalf("second sweep: %v", err) + } + if reruns, _, _ := fake.snap(); !reflect.DeepEqual(reruns, []int64{11, 21}) { + t.Fatalf("second sweep reruns = %v, want PR 2 re-run once v5 recovered", reruns) + } +} + +// Anyone can open an issue under the [shared-ci] title: a tracker the hive's +// own accounts did not write does not pick PRs to re-run (#18946 review), +// though a PR deferred to it by marker still is. +func TestSweepSharedCIReruns_StrangerTrackerListIgnored(t *testing.T) { + now := sharedCIRerunNow() + issue := openOutageIncident(now, "Affected PRs: #1, #2.\n") + issue.Author = "stranger" + prs := []*sharedCIRerunPR{ + {Number: 1, SHA: "aaaaaaaaaa", Comments: []string{""}, Runs: []sharedCIRerunRun{failedRun(11, 1)}}, + {Number: 2, SHA: "bbbbbbbbbb", Runs: []sharedCIRerunRun{failedRun(21, 1)}}, + } + srv, fake := newSharedCIRerunFake(t, []sharedCIRerunIssue{issue}, prs) + c := newTestClient(t, srv, "org", []string{"org/repo"}) + c.SetAppBotLogin("hive[bot]") + c.baseRed.now = sharedCIRerunNow + c.baseRed.noteGreen(baseRedMemoryKey("org", "repo", "main"), map[string]time.Time{"contract": now.Add(-time.Hour)}) + if _, err := c.SweepSharedCIReruns(context.Background(), SharedCIRerunOptions{Now: sharedCIRerunNow}); err != nil { + t.Fatalf("sweep: %v", err) + } + if reruns, _, _ := fake.snap(); !reflect.DeepEqual(reruns, []int64{11}) { + t.Fatalf("reruns = %v, want only the marker-stamped PR 1", reruns) + } +}