Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-base-red-after-restart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- After a restart, a PR red only on its base branch's own check is no longer sent back to its author, or adopted and superseded, before the first CI enumeration finishes. The base-red memory and index live in the process and stay empty until the first CI enumeration finishes (3 to 27 minutes on knuckle on 2026-11-18), so every base read as not red: projectbluefin/common#1689, at 072 past its grace, was adopted at once for main's red E2E check, losing a maintainer's approval, and its copy stayed red for the same reason ([projectbluefin/common#1689](https://github.com/projectbluefin/common/pull/1689)). Until the memory is published, the stage pass reads the base tip's checks itself, at most once per base every 10 minutes, and automatic adoption waits; an operator's Adopt now or adopt-supersede request does not.
1 change: 1 addition & 0 deletions src/cmd/hive/merge_eligibility.go
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,7 @@ func stageLabelPolicyFunc(cfg *config.Config) func() automerge.StageLabelPolicy
SharedCIIncident: stageSharedCIIncident,
BaseBranchRed: stageBaseBranchRed,
BaseRedRemembered: stageBaseRedRemembered,
BaseRedRememberedReady: stageBaseRedRememberedReady,
ForkGuard: stageForkGuard,
HeldWorkflowRuns: stageHeldWorkflowRuns,
DCORequired: stage.DCO.RequiredFor,
Expand Down
8 changes: 8 additions & 0 deletions src/cmd/hive/stage_shared_ci.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,14 @@ func stageBaseRedRemembered(repo, baseRef string) []string {
return append([]string(nil), (*m)[github.BaseRedRememberedKey(repo, baseRef)]...)
}

// stageBaseRedRememberedReady answers
// StageLabelPolicy.BaseRedRememberedReady: whether a CI-enriched
// enumeration has published the memory since the hive started. Until then
// the stage reconciler reads base tips itself (projectbluefin/common#1689).
func stageBaseRedRememberedReady() bool {
return stageBaseRedMemory.Load() != nil
}

// stageSharedCIIncident answers StageLabelPolicy.SharedCIIncident. repo is
// "owner/name"; the snapshot may key a PR by either form. A head pushed
// since the snapshot is not covered: the deferral was decided for the old
Expand Down
6 changes: 6 additions & 0 deletions src/cmd/hive/stage_shared_ci_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ func TestStageBaseRedRemembered(t *testing.T) {
if got := stageBaseRedRemembered("acme/widget", "main"); got != nil {
t.Fatalf("nothing published: %v", got)
}
if stageBaseRedRememberedReady() {
t.Fatal("ready before any publish")
}
publishStageBaseRedMemory(map[string][]string{github.BaseRedRememberedKey("acme/widget", "main"): {"Tests Passed", "Unit Tests"}})
if got := stageBaseRedRemembered("Acme/Widget", "main"); len(got) != 2 || got[0] != "Tests Passed" || got[1] != "Unit Tests" {
t.Fatalf("remembered = %v", got)
Expand All @@ -85,6 +88,9 @@ func TestStageBaseRedRemembered(t *testing.T) {
if got := stageBaseRedRemembered("acme/widget", "main"); len(got) != 0 {
t.Fatalf("after an empty publish: %v", got)
}
if !stageBaseRedRememberedReady() {
t.Fatal("an empty publish is still a publish")
}
}

func TestStageHeldWorkflowRunsIndex(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion src/docs/labels-and-control-signals/stage-labels.md
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ used to admit the PR:
| `contributor-lane:soaking(...)` β€” and no red check on the head | `095-pre-merge-soak` |
| `head-changed-since-eval`, `merge-failed`, `merge-queue-required` (the base branch's ruleset only accepts merges through GitHub's merge queue and the transport is a static token that cannot enqueue, so the direct REST merge is refused; with App auth on a repo opted in through `auto_merge.native_merge_queue` these bases are enqueued instead, see [native-merge-queue.md](../native-merge-queue.md); logged as a skip rather than a merge error, [#14018](https://github.com/hivecommons/hive/issues/14018); the self-authored sweep then skips the direct merge on that base for 30 minutes and logs the skip once per PR head, [#14068](https://github.com/hivecommons/hive/issues/14068); the trusted-author, queued and contributor-lane sweeps skip it the same way, sharing one memory of such bases; while a base is remembered the dashboard shows its otherwise-green PRs as a person's to merge, not Merge-eligible; with `auto_merge.require_current_base` on, a head behind such a base is not updated for 24 hours after the last refusal and holds here instead, so the base moving does not reset its review), `required-checks-expected`, `behind`, `updated-branch`, `behind-update-budget-exhausted` (a behind self-authored PR the sweep did not update only because this tick's branch-update budget was spent; retried next tick, waits on the hive, [#12607](https://github.com/hivecommons/hive/issues/12607)), `release-hold*`, `native-merge-queued: queued (position N, STATE)` (the PR was enqueued into GitHub's native merge queue) and the failures `native-merge-queue-read`, `native-merge-queue-enqueue`, `native-merge-queue-method`, `native-merge-queue-rejected` (the merge queue removed the hive's own entry for this head without merging it; see [native-merge-queue.md](../native-merge-queue.md#current-base-gate)) β€” and no red check on the head | `100-merge-queue` |
| CI red (required or optional check) while the PR is deferred to an open `[shared-ci]` incident (`<!-- hive-shared-ci-<n> -->`, the governor's `deferred_incident`) for the same head | `050-review-pending` / `060-reviewed`, reason `shared-ci-incident:<n>`: the base or a shared check is what is red, so the PR is never sent to 070/071/072, never nudged and never stalled. A 072 nudge already posted for the head (the nudge goes out on entering 072, and the incident marker can land minutes later) is edited in place to say it was withdrawn, and the head's nudge count is reset. Once the incident closes the PR stays deferred until the shared-CI rerun sweep has re-run its failed jobs (at most 30 minutes); a head still red after that is handed back as usual, with a fresh nudge. Conflicts and requested changes stay author-facing ([#12606](https://github.com/hivecommons/hive/issues/12606)) |
| CI red (required or optional) while **every** failing check on the head is also red on the base branch's tip β€” the governor's `base_red_checks` / `base_red`, set during CI enrichment when a same-named latest check run on the base ref (meta checks excluded) concluded failure / timed_out / cancelled / action_required / startup_failure / stale **and** the PR's changed files include none of `.github/workflows/*`, `Dockerfile` / `Dockerfile.*` / `*.Dockerfile`, `go.mod` / `go.sum` (a PR touching those plausibly owns the check and stays author-facing; a diff beyond the file-list page budget also stays author-facing). The governor remembers each base's red checks across passes: a check stays red on the base until a read sees the tip's run of it complete green, so a tip whose run is still queued or in progress, or a base listing that fails (logged at Warn), keeps the base's last red set; a check not seen red for 2 h is dropped. A new head the CI enrichment has not read yet is classified from that memory at the stage pass: every check red on the head (its check digest) is remembered red on the base and the head touches no CI-owning path (an unreadable file list stays author-facing) | `050-review-pending` / `060-reviewed`, reason `base-branch-red:<check>,<check>` ("waiting for the base branch's CI to recover (…) β€” not this PR's fault"): never 070/071/072, never nudged (a 072 nudge already posted for the head is edited in place to say it was withdrawn), never stalled, still not merged (the lane's skip reason is the same token). The fix loop counts no attempt, kicks no fixer, and posts one `<!-- hive-base-branch-red -->` hold comment per head; an escalation the hive itself raised (ledger `label_applied`) over such a head has its `needs-human` removed with the hold wording. The next evaluation after the base goes green re-evaluates the head: a still-red PR is handed back as usual. Conflicts and requested changes stay author-facing (seen on [#13270](https://github.com/hivecommons/hive/pull/13270) and [#13258](https://github.com/hivecommons/hive/pull/13258), 2026-10-20) |
| CI red (required or optional) while **every** failing check on the head is also red on the base branch's tip β€” the governor's `base_red_checks` / `base_red`, set during CI enrichment when a same-named latest check run on the base ref (meta checks excluded) concluded failure / timed_out / cancelled / action_required / startup_failure / stale **and** the PR's changed files include none of `.github/workflows/*`, `Dockerfile` / `Dockerfile.*` / `*.Dockerfile`, `go.mod` / `go.sum` (a PR touching those plausibly owns the check and stays author-facing; a diff beyond the file-list page budget also stays author-facing). The governor remembers each base's red checks across passes: a check stays red on the base until a read sees the tip's run of it complete green, so a tip whose run is still queued or in progress, or a base listing that fails (logged at Warn), keeps the base's last red set; a check not seen red for 2 h is dropped. A new head the CI enrichment has not read yet is classified from that memory at the stage pass: every check red on the head (its check digest) is remembered red on the base and the head touches no CI-owning path (an unreadable file list stays author-facing). The memory lives in the process and is first published when the first CI enrichment after a start finishes; until then the stage pass reads the base tip itself, at most once per base every 10 minutes (a failed read is retried after a minute), and a head whose every red check is red there is base red the same way. That read is the head digest's own (commit statuses as well as check runs, latest run per name and app, ignorable checks skipped), not the enrichment's check-runs-only read; a check still running on the tip, or with no run there, is not red, and an unreadable tip stays author-facing. Because that read can miss the base's red, an automatic adoption also waits until the memory is published (logged once per head as `adoption deferred until the base-red memory is published`); an operator's Adopt now or `adoption/supersede` request does not wait. Before this, every base read as not red for that window, and a PR already past its 072 grace was adopted for the base's red ([projectbluefin/common#1689](https://github.com/projectbluefin/common/pull/1689), 2026-11-18) | `050-review-pending` / `060-reviewed`, reason `base-branch-red:<check>,<check>` ("waiting for the base branch's CI to recover (…) β€” not this PR's fault"): never 070/071/072, never nudged (a 072 nudge already posted for the head is edited in place to say it was withdrawn), never stalled, still not merged (the lane's skip reason is the same token). The fix loop counts no attempt, kicks no fixer, and posts one `<!-- hive-base-branch-red -->` hold comment per head; an escalation the hive itself raised (ledger `label_applied`) over such a head has its `needs-human` removed with the hold wording. The next evaluation after the base goes green re-evaluates the head: a still-red PR is handed back as usual. Conflicts and requested changes stay author-facing (seen on [#13270](https://github.com/hivecommons/hive/pull/13270) and [#13258](https://github.com/hivecommons/hive/pull/13258), 2026-10-20) |
| CI red (required or optional) on a **fork** PR while **every** failing check's annotation says the check refuses forks (case-insensitive `fork pull request(s) cannot`, `maintainer must push this branch`, or `cannot publish` together with `fork` β€” a fork-publication / fork-secrets guard) | `085-needs-decision`, reason `fork-guard:<check>,<check>`: never 070/071/072, never nudged, never adopted (074) β€” neither the author nor the hive can turn the check green by pushing to the fork. The entry comment names the check and says a maintainer must push the branch to the repository (or run or approve it for head `<sha>` as the repository requires). A new head on the same fork PR (a merge of the base, or the author's push) keeps the guard while the lane's red check is one of the recorded `fork-guard:` checks, so the PR does not drop to 072 and nobody is nudged before the annotation for that head is read. A fork PR head that nothing has classified yet (its first red head, or a PR that came to 072 for another reason first) is classified from its own check annotations with the same rule, once per head, before it can be nudged: a head red only on fork-guarded checks goes straight to 085 and the guard is recorded for later heads (projectbluefin/utah-packages#972, where the author was nudged and sent relay rework for a fork-gated Canary five minutes before the annotation was read). The hive also does not update the branch of a fork head that waits on a fork-guarded check: the run a maintainer dispatches proves one exact head, and a merge of the base would discard it. Once the check is green the guard drops and updates resume before merge. While the PR stays parked across new heads, the entry comment is rewritten in place for each one, so it always names the current head, and it says that a run which passed on an earlier head does not cover this one even when the patch is the same, because a rebase or a merge of the base changes the code the check runs (projectbluefin/utah-packages#917). A same-repo PR with the same message, or a fork PR with any other red check, is handed back as usual ([#14273](https://github.com/hivecommons/hive/issues/14273), projectbluefin/utah-packages#862 and #869) |
| GitHub holds workflow runs on the head behind its approve-workflows gate (`action_required`), and the hive's approval sweep decided it will not approve them: its trust policy does not cover the head (an untrusted author or a fork head), `governor.ci.auto_approve_workflows` is off, or GitHub refused the approval (no `actions: write`). Whatever the lane said next (`contributor-lane:author-association`, `contributor-lane:no-linked-issue`, `contributor-lane:no-verdict`, a lane error) | `085-needs-decision`, reason `workflow-approval-required:held-runs:<run id>,<run id>`: no required check can report until a maintainer clicks **Approve and run** (or approves the head on the dashboard, which the sweep honours), so this wins over the lane's own reason. A trusted head the sweep has not decided on yet is not parked. Closed, merged, draft, held and sentinel-flagged PRs keep their own stage, and a lane that already reports `workflow-approval-required:<ctx>` (after posting `/ok-to-test`) or `unauthorized:<step>` keeps its own, more specific reason. The reason clears on the next enumeration after the runs start or the head changes. The missing-check base refresh never updates such a PR: a new head's runs would be held again (projectbluefin/server#686). |
| any **non-required** check on the current head is red, whatever the lane's reason (the lane gates on required checks only, so it still soaks and merges; a reader of the rollup sees βœ—). Parking modifiers (`hold`, `needs-human`, admission reasons) still win | `070-fixing` / `071-needs-author` / `072-returned-to-contributor` (by author kind), reason `optional-check-failing:<check>` (the nudge lists every red check); never 080 or 100. A red *required* check is the lane's own `required-check-failing:*` β†’ 070/071/072 ([#12538](https://github.com/hivecommons/hive/issues/12538), seen on [#12457](https://github.com/hivecommons/hive/pull/12457)) |
Expand Down
Loading
Loading