Skip to content

✨ feat(github): merge the base into a red PR whose failing checks are now green on the base tip - #17544

Merged
hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:feat/update-behind-red-prs
Nov 10, 2026
Merged

hivecommons-hive[bot] merged 2 commits into
hivecommons:v6from
Danathar:feat/update-behind-red-prs

Conversation

@Danathar

@Danathar Danathar commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary: When a PR goes red on a check its base branch also broke, fixing the base does not fix the PR: a re-run reuses the old merge ref, so the PR stays red until someone presses "Update branch". The hive already does that for PRs it adopted, but for nothing else, and never in a repository with auto-merge off. This adds a small, capped eval-cycle step that merges the base into a red PR that is behind its base when every failing check has completed green on the current base tip. It goes through a new base_refresh write-back kind, so on someone else's PR it needs L6 plus the opt-in label unless the operator widens it.

What went wrong

  • projectbluefin/actions#544: head 01f4643 (2026-11-09 23:06Z) was red on pytest (test_zizmor_baseline_pins), inherited from main at 460aab7, where pytest was also red. Main repaired it with fix(pinned-fetches): skip only the gate checkout at the scan root projectbluefin/actions#734 (09d8fba, 05:00Z on 2026-11-10, pytest green). The PR stayed red and behind until a maintainer pressed "Update branch" at 11:33Z, and pytest then passed on 0cc822c.
  • Nothing updated it: projectbluefin/actions has auto-merge off (every automerge lane logs repo-auto-merge-disabled), so the adoption recovery (automerge_adoption_recovery.go) never saw it. The current-base gate and the "behind" update only run for PRs that are otherwise ready to merge. The hold guard and the PR follow-up base sync only act on behind/dirty mergeable states, and GitHub reports neither when up-to-date branches are not required. The shared-CI rerun only re-runs jobs, which reuses the old merge ref.
  • The other PRs in the report were not this gap. utah#1180 was green, held, and on a Renovate branch, which the hive leaves to the bot by design (🐛 fix: leave dependency-bot branches to the bot on every base-sync path #16862). fsdk-containers#505 was green: the current-base gate tried to update it, and GitHub refused (fork-workflow-update). actions#731 stayed red after its manual update, so the failure was its own.

What changes

  • New eval-cycle step, refreshStaleRedBehindPRs, runs at most every 5 minutes just before the stuck-PR reaper. It is an API-budget unstick step, so it is shed only in critical mode and on cached-snapshot cycles.
  • A PR is a candidate when it is red on named checks, its CI has settled, and it has no open shared-CI incident and no base-red check. The step skips drafts, held and exempt PRs, forks without maintainer edits, dependency-bot branches, stage 072 and stage 074 (the adoption recovery owns 074), and repos in auto_merge.require_current_base.
  • The signal is simple: every failing check has a completed, green latest run on the current base tip, and the head is behind that tip. A check the base never runs, or has not finished yet, does not count. Nothing tries to guess which files a check depends on.
  • The update is update-branch pinned to the evaluated head. It runs once per (head, base tip), with at most 2 updates per repo and 6 per pass, and posts no comment. Each update is logged and audited as stale-red-base-refresh, and the reaper does not kick a fixer for that PR in the same cycle.
  • New write-back kind base_refresh, default {min_level: 6, require_label: true}. It is always allowed on hive-authored PRs. On other PRs it needs the opt-in label, or writeback.kinds.base_refresh.unlabeled_min_level: 6.
  • Deliberate exception, for the reviewer to confirm: PRs labelled needs-human (and the other gated-band labels) are not skipped. The dashboard's Gated band says "automation does not continue" on them, but this step only brings the branch up to date (the same as GitHub's "Update branch" button, pinned to the head it checked); it never merges, approves or decides. The one real case behind this PR, ci(reusable-sync-branches): make force-reset opt-in, never on the default branch projectbluefin/actions#544, carries needs-human, and a maintainer did exactly this update by hand. If owners prefer the strict reading, skipping them is a one-line change in staleRedRefreshSkip.

Files

  • src/pkg/github/stale_red_base_refresh.go: reads the base tip state, gives the verdict, and does the gated compare + update-branch.
  • src/cmd/hive/stale_red_base_refresh.go: candidate filter, caps, interval and memo. main_eval_cycle.go wires it in. apibudget.go lists it as an unstick step.
  • src/pkg/writeback/kinds.go, dedupe.go: the base_refresh kind.
  • Docs: operator-reference/sweeps-and-operations.md (new section, write-surface table row, budget list), operator-reference.md index, github-write-surface.md, acmm-policy-matrix.md. design/agent-turn-model.md gets two line citations of main_eval_cycle.go that shifted.
  • changelog.d/added-stale-red-base-refresh.md.

Tests

None were run locally (repo AGENTS.md). CI is the verdict.

  • pkg/github: TestStaleRedBaseVerdictFor, TestStaleRedBaseStateFor (newest run wins, running and meta checks ignored) and TestRefreshStaleRedPR. The last one covers a behind PR updated and pinned to its head, a current PR, a gate refusal making no call, a foreign PR without the label refused, a repo owned by automerge, a conflict that settles, and a head that moved staying open.
  • cmd/hive: TestRefreshStaleRedBehindPRs_OncePerHeadAndBaseTip, _Interval, _BaseMustClearEveryFailingCheck, _PerRepoCap, _ErrorRetriesNextPass and TestStaleRedRefreshSkip.
  • pkg/writeback: TestKindSets now lists base_refresh among the kinds that are not content-deduped.

Refs projectbluefin/actions#544

🤖 Generated with Claude Code

https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

… green on the base tip

A PR that went red on a check its base also broke stays red after the
base is fixed: a re-run reuses the old merge ref, so only a new head picks
the fix up. projectbluefin/actions#544 sat red on pytest for six hours
after main repaired the baseline (projectbluefin/actions#734), until a person pressed "Update
branch". The automerge lanes only do this for PRs the hive adopted, and
never in repositories with auto-merge off.

The eval cycle now runs a stale-red base refresh (at most every 5 minutes,
an unstick step): for an open PR that is red on named checks, behind its
base, with every failing check completed green on the current base tip,
it calls update-branch pinned to the evaluated head. It skips drafts, held
and exempt PRs, open shared-CI incidents, base-red checks, forks without
maintainer edits, dependency-bot branches, stage 072 and 074, and repos
whose base refresh belongs to the automerge preparation front. At most 2
updates per repo and 6 per pass, once per (head, base tip). Each update
goes through a new base_refresh write-back kind (hive-authored PRs, else
L6 with the opt-in label), and the reaper holds its fix kick for a PR it
refreshed in the same cycle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 10, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign clubanderson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 10, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff against v6 at merge-base 292d11d and the code it leans on. This looks correct to me, and the body matches what the code does.

What I verified in the tree:

  • Every helper the new code calls exists with the signature used: latestCheckRunsByNameAndApp (commit_ci.go:1079), checkConclusionIsRed (:749), isMetaCheck (client.go:2975), UpdateBranchExpectedHead / IsUpdateBranchStaleHead (pullrequest.go:635-660), ClassifyBranchUpdateError, baseRefreshScheduled, BranchHead, WritebackItemForPR/WritebackAllowed, and in cmd/hive hasLabel, fullRepoName, reaperSkipAfterRerun, budgetDecision.SkipUnstick, dashSrv.AuditLogRecord.
  • CompareCommits(base.SHA, pr.HeadSHA) → BehindBy is the right direction (commits on base the head lacks).
  • The update is pinned to the enumerated head and the memo key is (head, base tip), so a head that moves under the request is head-moved, unsettled, and re-evaluated next pass rather than merged onto unseen commits — same discipline as the automerge current-base gate.
  • Verdict logic is conservative in the right direction: a base-red check wins over unknown, and a check with no completed run on the base tip (PR-only check, or still running) blocks the refresh rather than being assumed fixed.
  • KindBaseRefresh {MinLevel: 6, RequireLabel: true} matches KindCIRerun, which is the closest existing unsolicited-write kind, and the acmm matrix / write-surface docs were updated in step.
  • The skip list in staleRedRefreshSkip matches the body's list (draft, held/exempt, not-red, CI running, shared-CI incident, base-red, fork without maintainer edits, dependency bot, stage 072/074).

One open question, not a finding: pr.FailingChecks can presumably include failures reported as commit statuses (Prow-style contexts), while StaleRedBaseStateFor reads only check runs on the base tip. If so, a PR whose only red item is a status context will always get base-unknown and never be refreshed. That is the safe failure mode, so it only matters if the projectbluefin repos in the report rely on statuses; worth a line in the sweeps doc if it does.

— hive: agent=reviewer backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive

hivecommons-hive Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

Read the diff against the v6 tree plus the helpers it calls. The code looks correct to me; the one thing I cannot settle is the policy question the body itself raises, so I'm routing that rather than guessing.

Decision needed (from the PR body, "Deliberate exception"): staleRedRefreshSkip (src/cmd/hive/stale_red_base_refresh.go:177-201) does not skip needs-human or the other Gated-band labels, so this step will push an update-branch merge commit onto a PR the dashboard says "automation does not continue" on. The author argues it is the same as the "Update branch" button and the motivating case (projectbluefin/actions#544) carried needs-human. That is a trust/side-effect call for an approver, not for me. It is bounded by the write-back gate (src/pkg/github/stale_red_base_refresh.go:155-157: KindBaseRefresh defaults to {MinLevel: 6, RequireLabel: true}, src/pkg/writeback/kinds.go:150), so on a non-hive PR it only fires at L6 with the opt-in label unless the operator widens it.

What I verified as correct:

  • Same check-name space on both sides. FailingChecks is built from check-run names (src/pkg/github/client.go:2912-2937, meta checks excluded), and StaleRedBaseStateFor reads check runs on the base tip through the same latestCheckRunsByNameAndApp / isMetaCheck filters (stale_red_base_refresh.go:98-109). A PR-only check that never runs on the base lands in StaleRedBaseUnknown and blocks the refresh (:66-77) — conservative, as the comment says.
  • Pinned update, no race onto an unseen head. RefreshStaleRedPR calls UpdateBranchExpectedHead(…, pr.HeadSHA, base.SHA) (:166) and maps IsUpdateBranchStaleHead to an unsettled head-moved (:167-170), so the memo in cmd/hive (stale_red_base_refresh.go:140-152) does not record it and the next pass re-evaluates the new head.
  • Memo only on Settled, and CompareCommits/API errors return err without settling (stale_red_base_refresh.go:159-162 / cmd :135-141), so transient failures retry next pass.
  • Automerge ownership respected: baseRefreshScheduled(repo) short-circuits before any write (:151-153), matching the auto_merge.require_current_base claim.
  • Reaper interplay: reaperSkipAfterRerun(reapSkip, refreshed) (main_eval_cycle.go:830) unions the refreshed escalation keys into the skip set, so no fixer is kicked for a head the merge replaces in the same cycle.
  • Fields/helpers referenced (BaseRedChecks, SharedCIIncident, MaintainerCanModify, DependencyBotBranch, escalation.Key, AuditLogRecord) all exist on v6.

Non-blocking observation: StaleRedBaseStateFor reads only check runs, not commit statuses; a red commit status (e.g. a Prow context) on the head would not be in FailingChecks either, so the two sides stay consistent — just noting that status-only repos are outside this step's signal.

Evidence bundle `#1754423b230a` sha256:25ffb933821edd51981d424ba7521cdade0fa98390c6075f0e4cf5fc81bfc200 — download from the dashboard or `hivectl review evidence #17544`.

@hivecommons-hive
hivecommons-hive Bot merged commit 52cef81 into hivecommons:v6 Nov 10, 2026
69 of 70 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @Danathar holds role owner and GitHub permission maintain on repo; CI green at 23b230a.

Danathar added a commit to Danathar/hive that referenced this pull request Nov 10, 2026
Keep both new docs sections: the shared-CI tracking issues sweep and the
stale-red base refresh (hivecommons#17544).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant