Skip to content

🐛 shared-ci: re-run PRs once the base recovers, tracker still open - #18946

Merged
hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/shared-ci-rerun-on-recovery
Nov 18, 2026
Merged

hivecommons-hive[bot] merged 3 commits into
hivecommons:v6from
Danathar:fix/shared-ci-rerun-on-recovery

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

When an outside outage turns a check red on many PRs, the hive files a [shared-ci] tracker. Until now it re-ran the failed checks only after a person closed that tracker, and then only on PRs that a fix lane had stamped with the tracker's marker. The hive never closes a tracker itself (#17546). So when an outage ended on its own, most of the PRs it hit stayed red.

This PR re-runs those PRs as soon as the base shows the check passing again, while the tracker stays open. It also includes every PR the tracker lists, not only the stamped ones. Re-running failed jobs pushes nothing, so Renovate, Dependabot and 072 PRs are included. The tracker is never closed by this.

What happened

The fix

  • Recovery signal, no new listing. When the base-red gate reads a base tip (baseCheckStatesForRef), it now also records when each check last completed green there (baseRedMemory.noteGreen). Times are kept for the 72 h rerun lookback. A check name that another app still runs, or that is red, is not counted as green.
  • Open incidents. An open incident counts as over for a PR once its check completed green on that PR's base tip after the incident was filed.
    • It must be the PR's own base: green on main doesn't count for a v5 PR.
    • Open trackers are listed only for a repo where such a green run was recorded, so quiet repos pay nothing extra.
    • A green run from before the outage doesn't count. Without this, a base tip that wasn't re-run during the outage would look recovered.
  • Which PRs. A PR qualifies if it carries the marker (today's rule), or if the tracker body lists it. That covers the hive tracker's "Open pull requests failing it (N): …" line and an agent's "Affected PRs: …" line. This applies to closed trackers too.
    • A PR whose newest marker names an incident that isn't over is still left alone.
  • Deferral. Once the rerun comment is on a PR, its deferral to the incident is lifted even while the tracker stays open. The re-run happened after the base recovered, so a red that survives it is the PR's own. Without this, a PR that stays red for its own reason would wait on the tracker until a person closes it.
  • Unchanged:
    • one rerun per PR per incident (the hive-shared-ci-rerun-<n> marker);
    • the 10-PR pass cap;
    • the not_planned exclusion;
    • the closed-tracker path.
  • The comment for an open incident reads "#N (still open; <check> has passed on the base branch since it was filed) is over — re-running …".

Left out

  • "Latest run of the check failed on the current head" as a third way to qualify. It would need a check-run listing for every open PR in the repo on every pass. The tracker list covers the reported case.
    • A caveat: the hive tracker names at most 30 PRs, so PRs beyond that are only reached through the marker.
  • Repos without a tracker (projectbluefin/common's codeberg failures). Without an incident there is nothing to attach the once-per-incident dedupe to. That needs its own design.
  • A base tip that never re-runs the check after the outage (no merges, or a check that runs only on PRs). That base gives no recovery signal, so its PRs wait for the close, as they do today.
  • Closing trackers. That stays with ✨ feat(github): file one [shared-ci] tracking issue when a check fails across a repo #17546 and the held auto-close decision.

Existing issues

None cover this. I searched for "SweepSharedCIReruns", "shared-ci rerun" and "hive-shared-ci-rerun".

None of them decided against triggering on recovery.

Tests

  • shared_ci_rerun_test.go:
    • an open incident with a green base tip re-runs the marked PR plus the listed Renovate and 072 PRs once each, leaves an unlisted PR alone, and a second pass does nothing;
    • no green recorded, green from before the filing, or green only on another base re-runs nothing, and with no green recorded the open trackers aren't even listed;
    • a closed tracker re-runs a listed, unmarked Dependabot PR;
    • tracker-body parsing for the hive, scanner and bold-label forms;
    • pendingSharedCIReruns with listed incidents.
  • base_branch_red_memory_test.go:
    • the green record: newest time kept, case-insensitive, base-scoped, dropped after the TTL;
    • baseCheckStatesForRef records a green completion and skips a name another app still runs.
  • shared_ci_incident_test.go: the rerun marker lifts the deferral on an open incident, and an unmarked PR stays deferred.

Locally I ran only gofmt -l (AGENTS.md bans local go test/go build/go vet) and check-docs-citations.py, which verified 2163 references. CI is the test run.

Appendix: code refs

  • src/pkg/github/shared_ci_rerun.go: sharedCIRerunIncidents, recoveredOpenSharedCIIncidents, sharedCITrackerListedPRs, pendingSharedCIReruns, renderSharedCIRerunComment
  • src/pkg/github/base_branch_red.go (baseCheckStatesForRef), base_branch_red_memory.go (noteGreen, greenAfter, anyGreen)
  • src/pkg/github/shared_ci_incident.go (openSharedCIIncident)
  • src/pkg/config/review_config.go (SharedCIRerunConfig doc)
  • Docs: src/docs/operator-reference/sweeps-and-operations.md

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

The shared-CI rerun sweep started only from [shared-ci] incidents a person
had closed, and re-ran only PRs a fix lane had stamped with the incident's
marker. The hive never closes a tracker (hivecommons#17546), so an outside outage that
ended on its own (projectbluefin/utah#1336, codeberg.org DNS) left all 18
listed PRs red; only one carried the marker.

- The base-red gate's base-tip read now also records when each check last
  completed green there (baseRedMemory.noteGreen); no new listing.
- An open incident counts as over for a PR once its check completed green
  on that PR's base tip after the incident was filed.
- PRs the tracker body lists (the hive tracker's PR line, or an agent's
  "Affected PRs:") qualify as well as marker-stamped ones, for closed
  incidents too.
- The rerun marker lifts the deferral while the incident stays open.
- The tracker is never closed; once-per-PR-per-incident dedupe, the pass
  cap and the not_planned exclusion are unchanged.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 18, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 18, 2026
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Review asks:

  • the review swarm requested changes

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (6 waiting): fixes are handed out before new work as soon as you have capacity.

@Danathar Danathar changed the title fix(github): re-run shared-CI PRs once the base recovers, tracker open 🐛 shared-ci: re-run PRs once the base recovers, tracker still open Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v6

Conflicts are checked with every PR merged onto the current tip of v6, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18945 🐛 automerge: name held workflow runs and stop base-refreshing them same files src/docs/operator-reference/sweeps-and-operations.md

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness review at head 929b688: one in-scope defect. The sweep's per-PR "nothing to do" memo is keyed on the repo-wide recovered-incident set, not on whether the PR's own base is green. A PR on a not-yet-green base is memoized and then skipped after its base recovers. Details and a suggested fix are in the inline thread. The rest of the diff (green-time recording, TTL, deferral lift) reads consistent.

— hive: agent=adjudicator backend=claude model=claude-sonnet-5-5 claude=2.1.327

Comment thread src/pkg/github/shared_ci_rerun.go Resolved
@hivecommons-hive

hivecommons-hive Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from ci-maintainer (folded here because this head already carries its hive review):

Security review: changes requested

Tracker body is trusted without checking who wrote the tracker (medium). sharedCITrackerListedPRs(issue.GetBody()) now decides which PRs get re-run. Both listers select trackers by title prefix and state only:

  • openSharedCIIncidentIssues: shared_ci_issues.go:549-580
  • recentlyClosedSharedCIIncidents: shared_ci_rerun.go, the closed-tracker lister the diff touches

Neither checks sharedCIIssueOwned or isHiveAccount (shared_ci_issues.go:496-500). Previously the hive-stamped <!-- hive-shared-ci-N --> comment marker was the trust anchor.

Now anyone who can open (or open and close) an issue titled [shared-ci] <check> failing across <repo> with an Affected PRs: #N line makes the hive re-run failed jobs and comment on arbitrary PRs. That includes Renovate, Dependabot and 072 PRs, and PRs held back by the base-red gate. The blast radius is bounded: once per PR per incident, a 10-PR cap, no new code runs, and held/exempt labels are respected. It is still a trust-boundary regression. The title-derived check name is also interpolated into backticks in the rerun comment.

Requested fix: skip trackers where !c.sharedCIIssueOwned(issue) before parsing the body list in both listers. Reject check names containing backticks or newlines before rendering them.

Review findings from adjudicator-2 (folded here because this head already carries its hive review):

Delta re-review at 75aa89f.

  • correctness (seen-memo key ignoring per-base eligibility): fixed — shared_ci_rerun.go:400-401 keys the memo on eligibleFor for the PR's own base; new test covers a later-recovering base.
  • security (tracker body from any issue author selects PRs to re-run): still present — see inline thread at :316. The delta does not change it.

Findings not on a changed line

  • src/pkg/github/shared_ci_rerun.go:316: Still present from the previous head: incidents are trusted by title prefix only, and the tracker body drives which PRs get rerun. Restrict to trusted authors:
    			if !strings.HasPrefix(strings.TrimSpace(issue.GetTitle()), sharedCIIncidentTitlePrefix) || !sharedCITrustedAuthor(issue) {
    
    with sharedCITrustedAuthor returning true only for author_association OWNER/MEMBER/COLLABORATOR (or the hive bot login); apply the same filter in openSharedCIIncidentIssues.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the sweep, the base-red memory and the openSharedCIIncident change against the tree at head. The logic holds up: eligibleFor keys recovery to the PR's own base ref, the memo key now follows the per-PR incident set, and the deferral lift only fires once the sweep's own hive-shared-ci-rerun-<n> marker exists, which is only stamped after recovery. Tests cover the green record, the per-base scoping and the lift. Looks correct to me; one low-severity note inline.

Not a defect, just a behaviour change worth knowing: PRs listed in a closed tracker (not only fix-lane-stamped ones) are now re-run too, as the changelog says.

Findings not on a changed line

  • src/pkg/github/shared_ci_rerun.go:287: Low: the "Affected PRs:" / "Open pull requests failing it" lines are parsed from any open issue whose title matches [shared-ci] <check> failing across <repo> (openSharedCIIncidentIssues in shared_ci_issues.go filters on title only, not author). Anyone who can file an issue can therefore name PR numbers and get their failed jobs re-run once, provided the check really went green on the base after the issue was filed. Impact is bounded (10 PRs per pass, once per incident, no push), but if you want the list to be trusted input, restrict listing to issues authored by the hive App/known agents before reading the body.
Evidence bundle `#18946e6f53be` sha256:604a50cc194f12efac141f1e3685f32982dbd464d6abad21bd4d34eaab3af378 — download from the dashboard or `hivectl review evidence #18946`.

@clubanderson clubanderson added the author/human A person pushed this PR themselves label Nov 18, 2026
The "nothing to do" memo was keyed on the repo-wide recovered set, so a
PR seen while its base was still red was skipped after that base went
green (review on hivecommons#18946). Key it on the incidents over for the PR's base.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style review, delta 929b688..75aa89f (read at head 75aa89f).

Prior findings:

  • [correctness] seen-memo key ignores per-base eligibility: fixed. trySharedCIRerun now computes over := incs.eligibleFor(...) for the PR's own base before the memo check and derives incidentKey from it (shared_ci_rerun.go:400-401). The repo-wide key() helper and the extra parameter are removed, so there is one source of truth. over is a fresh maps.Clone, so the later mutation at :443 is safe. TestSweepSharedCIReruns_SecondBaseRecoversLater covers the regression.
  • [security] tracker body from any author selects PRs: still present. The delta does not touch it; sharedCITrackerListedPRs(issue.GetBody()) is still read without an author check (:287, :326). Out of this style perspective's scope; I am not re-filing it.

New, advisory only (not blocking): the doc comment on sharedCIRerunSeen (shared_ci_rerun.go:78-79) still says the memo is invalidated when "the closed-incident set changes". The key is now the per-base set of incidents over (closed plus recovered), so the wording is stale. Suggested rewording: "until the PR or the set of incidents over for its base changes."

Style verdict: the delta is a small, idiomatic simplification and the test follows the file's fixtures. Approve from this perspective.

— hive: agent=ci-maintainer backend=claude model=claude-sonnet-5-5 claude=2.1.327

Anyone can open an issue under the [shared-ci] title; a stranger's list
must not pick which PRs the hive re-runs (review on hivecommons#18946). Also
refresh the seen-memo doc comment for the per-base key.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
@Danathar

Copy link
Copy Markdown
Contributor Author

Re the ci-maintainer review at 75aa89f:

  • [security] tracker body from any author selects PRs: taken in e6f53be. Anyone can open an issue under the [shared-ci] title, so a tracker's PR list now counts only when one of the hive's own accounts wrote the issue (trustedSharedCITrackerListedPRs, via isHiveAccount). As with the other hive-authorship checks, a hive that cannot name its own accounts still trusts every author. Both real cases are App-authored: the sweep's own tracker, and the scanner's [shared-ci] contract failing across projectbluefin/utah projectbluefin/utah#1336. Marker-stamped PRs are unaffected. New test: TestSweepSharedCIReruns_StrangerTrackerListIgnored. The sweeps doc says so too.
  • Stale sharedCIRerunSeen doc comment: reworded in the same commit to "until the PR or the set of incidents over for its base changes".

@hivecommons-hive
hivecommons-hive Bot merged commit ea7c8c0 into hivecommons:v6 Nov 18, 2026
71 of 72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/human A person pushed this PR themselves dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants