From e20290d810d3d43c75654d524e0c4194e9273ced Mon Sep 17 00:00:00 2001 From: Danathar Date: Wed, 28 Oct 2026 14:16:43 -0400 Subject: [PATCH] =?UTF-8?q?=F0=9F=90=9B=20fix(stage):=20retry=20a=20failed?= =?UTF-8?q?=20085=20escalation=20note,=20once=20per=20head=20and=20reason?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The adoption escalation marked the head noted before posting its note, so when GitHub answered the adoption-stalled note with a 500 the PR sat at stage/085-needs-decision with no comment on that head ever (projectbluefin/utah-packages#650 and #652, 2026-10-26). The same mark-before-post pattern was in the three workflow-path refusal paths and in escalateUnadoptableFork. - stagelabels: MarkAdoptionEscalated records the escalation only; the new MarkAdoptionEscalationNoted records (head, reason) after the post succeeds. AdoptionEscalationNoted(head, reason) checks both, so a new blocker on the same head gets its own note (mirrors EntryCommentExplains, hivecommons/hive#13139). A record from before reasons were kept (noted head, no reasons) reads as noted for every reason on that head, so the upgrade reposts nothing. - automerge: all five call sites post through postAdoptionEscalationNote. The held-escalation branch of escapeStalledAdoption retries an unposted workflow-path refusal note, and hands an unposted or changed stall reason to the stall path instead of holding silently. - Hold an active escalation against a weak opinion: an errored lane on an approved head reads 080 at role rank, which the adoption pin does not claim, so the pass dropped the escalation for a sweep and projectbluefin/utah#919 flipped 085 -> 080 -> 085 on one head (10-26, 10-28). stageengine.HoldsEscalation is shared by the live pass and passDecision so enforce mode derives the same stage. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1 Signed-off-by: Danathar --- .../fixed-085-escalation-note-retry.md | 1 + src/docs/hive-merge.md | 3 +- src/docs/labels-and-control-signals.md | 15 +- .../automerge_adoption_escape_test.go | 206 +++++++++++++++++- .../automerge_adoption_rebase_test.go | 1 + .../automerge/automerge_contributor_adopt.go | 107 +++++---- .../automerge_contributor_adopt_test.go | 103 +++++++++ .../automerge/automerge_merge_commit_test.go | 3 +- .../automerge/automerge_stage_labels.go | 15 ++ .../automerge/automerge_stage_labels_test.go | 11 + src/pkg/github/automerge/stage_facts_test.go | 63 ++++++ src/pkg/stageengine/derive_pr.go | 23 ++ src/pkg/stageengine/facts.go | 6 +- src/pkg/stagelabels/stagelabels_test.go | 67 +++++- src/pkg/stagelabels/store.go | 65 ++++-- 15 files changed, 622 insertions(+), 67 deletions(-) create mode 100644 changelog.d/fixed-085-escalation-note-retry.md diff --git a/changelog.d/fixed-085-escalation-note-retry.md b/changelog.d/fixed-085-escalation-note-retry.md new file mode 100644 index 0000000000..29aa148ec9 --- /dev/null +++ b/changelog.d/fixed-085-escalation-note-retry.md @@ -0,0 +1 @@ +- **A PR parked at `stage/085-needs-decision` for a stalled adoption is no longer left without its note.** The escalation marked the head noted before posting, so when GitHub answered the `` (or workflow-path refusal, or unadoptable-fork) note with a 500 the PR sat at 085 with no comment on that head ever (projectbluefin/utah-packages#650, #652). The note is now recorded only once posted and is retried on later sweeps; it posts once per head and reason, so a different blocker on the same head (a check failure, then the hive's own sign-off attribution) gets its own note while the same one flapping back stays silent. A lane that errored on an approved head (080 at role rank) no longer drops an active escalation for a pass, which flipped projectbluefin/utah#919 085 → 080 → 085 on one head. Stage records from the previous build are read as already noted for their head, so the upgrade reposts nothing. diff --git a/src/docs/hive-merge.md b/src/docs/hive-merge.md index 3138235f94..3fc5537f76 100644 --- a/src/docs/hive-merge.md +++ b/src/docs/hive-merge.md @@ -774,7 +774,8 @@ at most once per (head, base tip) and three times per adoption, logged as (hivecommons/hive#14500). An adoption with nothing the hive can push past the dwell — after that recovery was tried and the head is still red — goes to `stage/085-needs-decision` as -`adoption-stalled:` with one comment naming the blocker, so the +`adoption-stalled:` with one comment naming the blocker (once +per head and reason, retried until GitHub accepts it), so the dashboard shows a person's item, not a silent 074; "waiting on maintainer lgtm" is only ever the state at 080+ (hivecommons/hive#12858). The dwell runs from the head's last hive push, a red optional check alone is not a blocker, diff --git a/src/docs/labels-and-control-signals.md b/src/docs/labels-and-control-signals.md index 23249cc7f6..38c8d23c5d 100644 --- a/src/docs/labels-and-control-signals.md +++ b/src/docs/labels-and-control-signals.md @@ -933,7 +933,8 @@ When a clanker-authored contributor PR goes red the hive does not just relabel i queued `fix-pr` task, and after the `072` dwell (24h) since the nudge on that head it parks at `085` as `adoption-stalled:fork-rehome-refused` with one note asking a maintainer to review it and bring it in or close - it. A new head from the author re-arms the normal path. An explicit + it (once per head and reason, retried on later sweeps until GitHub + accepts it; an errored lane's weak 080 does not lift it). A new head from the author re-arms the normal path. An explicit owner-delegated adoption request is still recorded and parks at once. - `reason=dco-rehome` (always `mode=supersede`) — not a fresh adoption but a second takeover: an in-place-adopted head stayed @@ -1079,9 +1080,15 @@ When a clanker-authored contributor PR goes red the hive does not just relabel i person to decide and the PR stays 074. The escalation holds only while its reason holds — the sweep re-evaluates it every pass and when the blocker clears (CI re-runs, the check goes green, a verdict lands) the - PR returns to 074 and the lane carries on; the comment is posted once - per head even if the blocker flaps, and a hive push to a new head - starts over. A pending `tide` or missing `lgtm` is never the reason — + PR returns to 074 and the lane carries on. A weak opinion is not such + a release: a lane that errored on an approved head (080 at role rank) + leaves the escalation at 085 for that pass instead of flipping the PR + 085 → 080 → 085. The comment is posted once per (head, reason): the + same blocker flapping back stays silent, a different blocker on the + same head (a check failure, then the hive's own sign-off attribution) + gets its own note, and a post GitHub refuses is not counted, so the + next sweep retries it until it lands. A hive push to a new head starts + over. A pending `tide` or missing `lgtm` is never the reason — the lane has not admitted the head; the comment says so. - **supersede** (`mode=supersede`) — the fork forbids maintainer edits, `in_place: false`, or the adopted head has unsigned commits (`no-dco`) diff --git a/src/pkg/github/automerge/automerge_adoption_escape_test.go b/src/pkg/github/automerge/automerge_adoption_escape_test.go index 19deb54110..31b802e283 100644 --- a/src/pkg/github/automerge/automerge_adoption_escape_test.go +++ b/src/pkg/github/automerge/automerge_adoption_escape_test.go @@ -3,12 +3,15 @@ package automerge import ( "context" "encoding/json" + "errors" "os" "path/filepath" "strings" "testing" "time" + gh "github.com/google/go-github/v72/github" + "github.com/hivecommons/hive/pkg/stagelabels" ) @@ -533,7 +536,7 @@ func TestAdoptedEscalationReleasesWhenReasonClears(t *testing.T) { if added := h.addedLabels(); len(added) != 1 || added[0] != stagelabels.AdoptedByHive.Label() || len(h.api.removed) != 1 || h.api.removed[0] != stagelabels.NeedsDecision.Label() { t.Fatalf("085 → 074: added=%v removed=%v comments=%v", h.api.added, h.api.removed, h.api.comments) } - if rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("aaa") || !rec.AdoptionEscalationNoted("aaa") || !rec.AdoptedInPlace() || rec.AdoptedHead != "aaa" { + if rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("aaa") || !rec.AdoptionEscalationNoted("aaa", stagelabels.LaneReasonAdoptionStalledPrefix+"check-failing:unit") || !rec.AdoptedInPlace() || rec.AdoptedHead != "aaa" { t.Fatalf("escalation not released: %+v", rec) } if len(stalledComments(h.api)) != 0 { @@ -560,7 +563,7 @@ func TestAdoptedEscalationReleasesWhenReasonClears(t *testing.T) { h.api.reset() h.clock.advance(time.Minute) h.observe(adoptPR(7, "bbb", escalated...), "check-failing:unit") - if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("bbb") || rec.AdoptionEscalationNoted("bbb") { + if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("bbb") || rec.AdoptionEscalationNoted("bbb", stagelabels.LaneReasonAdoptionStalledPrefix+"check-failing:unit") { t.Fatalf("new head: %+v", rec) } h.clock.advance(adoptedDwell + time.Minute) @@ -636,3 +639,202 @@ func TestAdoptedSignoffStallAskAgreesWithWorkflowFence(t *testing.T) { }) } } + +// observeErr is observe for a pass whose error the test expects (a comment +// POST the fake API refuses): the pass still runs to the end. +func (h *adoptHarness) observeErr(pr *gh.PullRequest, reason string) (StageLabelResult, error) { + h.t.Helper() + h.adopt.mu.Lock() + h.adopt.heads[pr.GetNumber()] = pr.GetHead().GetSHA() + h.adopt.mu.Unlock() + h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, reason, false, nil) + h.e.noteStageRole("acme", "widget", pr.GetNumber(), stagelabels.RoleContributor) + return h.e.ApplyStageLabels(context.Background()) +} + +// projectbluefin/utah-packages#650 and #652: GitHub answered the +// adoption-stalled note with a 500, and the escalation had already marked +// the head noted, so the PR sat at 085 with no comment on that head ever. +// The note is recorded only once posted: the next sweep posts it, and the +// one after does not repeat it. +func TestAdoptionStalledNoteRetriedAfterFailedPost(t *testing.T) { + p := adoptPolicy() + p.AdoptInPlace = true + h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p) + defer done() + key := adoptInPlaceAt(h, "aaa", "check-failing:unit") + reason := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:unit" + parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"} + + h.clock.advance(adoptedDwell + time.Minute) + h.api.failComments, h.api.failCommentsOn = 1, AdoptionStalledMarker + if _, err := h.observeErr(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit"); err == nil { + t.Fatal("the refused note was not reported") + } + rec, _ := h.e.stageStoreOrShared().Get(key) + if rec.Stage != stagelabels.NeedsDecision || !rec.AdoptionEscalated("aaa") || rec.AdoptionEscalationNoted("aaa", reason) || len(stalledComments(h.api)) != 0 { + t.Fatalf("failed post: %+v comments=%v", rec, h.api.comments) + } + + h.api.reset() + h.clock.advance(5 * time.Minute) + res := h.observe(adoptPR(7, "aaa", parked...), "check-failing:unit") + if notes := stalledComments(h.api); len(notes) != 1 || !strings.Contains(notes[0], "reason="+reason) || res.Comments != 1 { + t.Fatalf("note not retried: res=%+v comments=%v", res, h.api.comments) + } + if len(h.api.added) != 0 || len(h.api.removed) != 0 { + t.Fatalf("retry churned labels: added=%v removed=%v", h.api.added, h.api.removed) + } + if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision || !rec.AdoptionEscalationNoted("aaa", reason) { + t.Fatalf("retried note not recorded: %+v", rec) + } + + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(adoptPR(7, "aaa", parked...), "check-failing:unit") + if len(h.api.comments) != 0 || len(h.api.added) != 0 || len(h.api.removed) != 0 { + t.Fatalf("same head and reason repeated: comments=%v added=%v removed=%v", h.api.comments, h.api.added, h.api.removed) + } +} + +// A different blocker on the same head is a different note: the maintainer +// who read the first must learn what holds the PR now. Each reason is +// explained once per head, so the blocker flapping back stays silent. +func TestAdoptionStalledNotePerReasonOnSameHead(t *testing.T) { + p := adoptPolicy() + p.AdoptInPlace = true + h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p) + defer done() + key := adoptInPlaceAt(h, "aaa", "check-failing:unit") + parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"} + + h.clock.advance(adoptedDwell + time.Minute) + h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit") + if notes := stalledComments(h.api); len(notes) != 1 { + t.Fatalf("first note: %v", h.api.comments) + } + + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(adoptPR(7, "aaa", parked...), "check-failing:lint") + lint := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:lint" + if notes := stalledComments(h.api); len(notes) != 1 || !strings.Contains(notes[0], "reason="+lint) { + t.Fatalf("changed reason not noted: %v", h.api.comments) + } + rec, _ := h.e.stageStoreOrShared().Get(key) + if rec.Stage != stagelabels.NeedsDecision || rec.Reason != lint || rec.AdoptEscalatedReason != lint { + t.Fatalf("escalation kept the stale reason: %+v", rec) + } + if len(h.api.added) != 0 || len(h.api.removed) != 0 { + t.Fatalf("reason change churned labels: added=%v removed=%v", h.api.added, h.api.removed) + } + + for _, lane := range []string{"check-failing:unit", "check-failing:lint", "check-failing:lint"} { + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(adoptPR(7, "aaa", parked...), lane) + if len(h.api.comments) != 0 { + t.Fatalf("%s: reason already noted on this head posted again: %v", lane, h.api.comments) + } + // The record follows the blocker, so a later hold (a weak opinion) + // names the current one. + if rec, _ := h.e.stageStoreOrShared().Get(key); rec.AdoptEscalatedReason != stagelabels.LaneReasonAdoptionStalledPrefix+lane || rec.Reason != rec.AdoptEscalatedReason { + t.Fatalf("%s: escalation reason lags the blocker: %+v", lane, rec) + } + } +} + +// A record from before the noted reasons were kept: escalated and noted on +// the head, no reasons. It reads as noted for every reason on that head, so +// an upgrade does not repost on every parked PR. +func TestAdoptionStalledLegacyNotedHeadNoRepost(t *testing.T) { + p := adoptPolicy() + p.AdoptInPlace = true + h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p) + defer done() + key := adoptInPlaceAt(h, "aaa", "check-failing:unit") + parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"} + h.clock.advance(adoptedDwell + time.Minute) + h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit") + if notes := stalledComments(h.api); len(notes) != 1 { + t.Fatalf("first note: %v", h.api.comments) + } + + // Rewrite the record as the previous binary left it. + path := filepath.Join(t.TempDir(), "legacy-stage.json") + store := stagelabels.NewStore(path, h.clock.now) + src := h.e.stageStoreOrShared() + raw, err := json.Marshal(map[string]stagelabels.Record{key: func() stagelabels.Record { r, _ := src.Get(key); return r }()}) + if err != nil { + t.Fatal(err) + } + var items map[string]map[string]any + if err := json.Unmarshal(raw, &items); err != nil { + t.Fatal(err) + } + if _, ok := items[key]["adopt_escalation_noted_reasons"]; !ok { + t.Fatalf("fixture: no noted reasons to strip: %s", raw) + } + delete(items[key], "adopt_escalation_noted_reasons") + raw, _ = json.Marshal(items) + if err := os.WriteFile(path, raw, 0o644); err != nil { + t.Fatal(err) + } + if err := store.Load(); err != nil { + t.Fatal(err) + } + if rec, _ := store.Get(key); rec.AdoptEscalationNotedHead != "aaa" || len(rec.AdoptEscalationNotedReasons) != 0 { + t.Fatalf("legacy fixture: %+v", rec) + } + h.e.stageStore = store + + for _, lane := range []string{"check-failing:unit", "check-failing:lint"} { + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(adoptPR(7, "aaa", parked...), lane) + if len(h.api.comments) != 0 { + t.Fatalf("%s: legacy noted head reposted: %v", lane, h.api.comments) + } + if rec, _ := store.Get(key); rec.Stage != stagelabels.NeedsDecision { + t.Fatalf("%s: legacy escalation dropped: %+v", lane, rec) + } + } +} + +// projectbluefin/utah#919 flipped 085 → 080 → 085 within two minutes on +// one head: a lane that errored on an approved head reads it author-side +// done (080 at role rank), which is not a stage the adoption pin claims, +// so the pass dropped the active escalation for that sweep. A weak opinion +// does not release it; the labels (written from the engine's derivation) +// stay put. +func TestAdoptionEscalationHoldsAgainstErroredLane(t *testing.T) { + p := adoptPolicy() + p.AdoptInPlace = true + h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p) + defer done() + key := adoptInPlaceAt(h, "aaa", "check-failing:unit") + reason := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:unit" + parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"} + h.clock.advance(adoptedDwell + time.Minute) + h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit") + if rec, _ := h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision || len(stalledComments(h.api)) != 1 { + t.Fatalf("not escalated: %+v comments=%v", rec, h.api.comments) + } + + h.api.reset() + h.clock.advance(2 * time.Minute) + pr := adoptPR(7, "aaa", parked...) + h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, "", false, errors.New("lane tripped")) + h.e.noteStageRole("acme", "widget", 7, stagelabels.RoleContributor) + h.e.noteStageVerdict("acme", "widget", 7, ContributorLaneVerdict{Configured: true, Approved: true, Verdict: "approve"}) + if _, err := h.e.ApplyStageLabels(context.Background()); err != nil { + t.Fatal(err) + } + rec, _ := h.e.stageStoreOrShared().Get(key) + if rec.Stage != stagelabels.NeedsDecision || rec.Reason != reason || !rec.AdoptionEscalated("aaa") { + t.Fatalf("errored lane dropped the escalation: %+v", rec) + } + if len(h.api.added) != 0 || len(h.api.removed) != 0 || len(h.api.comments) != 0 { + t.Fatalf("escalation churned: added=%v removed=%v comments=%v", h.api.added, h.api.removed, h.api.comments) + } +} diff --git a/src/pkg/github/automerge/automerge_adoption_rebase_test.go b/src/pkg/github/automerge/automerge_adoption_rebase_test.go index 7408bf1a75..8f97ff7cc7 100644 --- a/src/pkg/github/automerge/automerge_adoption_rebase_test.go +++ b/src/pkg/github/automerge/automerge_adoption_rebase_test.go @@ -28,6 +28,7 @@ func TestAdoptedNeedsRebaseSupersedesInsteadOfStalling(t *testing.T) { store := h.e.stageStoreOrShared() reason := stagelabels.LaneReasonAdoptionStalledPrefix + "label:needs-rebase" store.MarkAdoptionEscalated(key, "aaa", reason) + store.MarkAdoptionEscalationNoted(key, "aaa", reason) store.Observe(key, stagelabels.NeedsDecision, reason) labels[0] = stagelabels.NeedsDecision.Label() } diff --git a/src/pkg/github/automerge/automerge_contributor_adopt.go b/src/pkg/github/automerge/automerge_contributor_adopt.go index 60e2fa3f8f..def6b66138 100644 --- a/src/pkg/github/automerge/automerge_contributor_adopt.go +++ b/src/pkg/github/automerge/automerge_contributor_adopt.go @@ -764,18 +764,16 @@ func (c *Engine) adoptContributorPR(ctx context.Context, store *stagelabels.Stor // delegated request; the head goes to a person (085) with one note. store.NoteAdoptReason(key, reason) // A note for another reason (a stalled adoption) does not explain - // the refusal of a maintainer's explicit request, so it still posts. - noted := rec.AdoptionEscalationNoted(obs.HeadSHA) && !(requested && rec.AdoptEscalatedReason != adoptionWorkflowRefusedReason) + // the refusal of a maintainer's explicit request, so it still posts; + // the explicit check covers records from before notes were kept per + // reason, which read as noted for every reason on the head. + noted := rec.AdoptionEscalationNoted(obs.HeadSHA, adoptionWorkflowRefusedReason) && !(requested && rec.AdoptEscalatedReason != adoptionWorkflowRefusedReason) store.MarkAdoptionEscalated(key, obs.HeadSHA, adoptionWorkflowRefusedReason) c.warn("stage labels: refusing workflow-path supersession; escalating to needs-decision", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "reason", reason, "noted", noted) if noted { return 0, nil } - if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(AdoptionWorkflowRefusedComment(obs))}); err != nil { - c.warn("stage labels: workflow-path refusal note failed", "repo", obs.Display, "pr", obs.Number, "error", err) - return 0, err - } - return 1, nil + return c.postAdoptionEscalationNote(ctx, store, key, obs, adoptionWorkflowRefusedReason, AdoptionWorkflowRefusedComment(obs), "stage labels: workflow-path refusal note failed") } if appLogin == "" { c.warn("stage labels: cannot adopt contributor PR, App login unknown", "repo", obs.Display, "pr", obs.Number) @@ -1008,17 +1006,13 @@ func (c *Engine) settleAdoption(ctx context.Context, store *stagelabels.Store, k // Terminal for this head: refuse publication, keep the // in-place adoption and hand the re-home to a person (085). store.NoteAdoptDiffFence(key, fence) - noted := rec.AdoptionEscalationNoted(obs.HeadSHA) + noted := rec.AdoptionEscalationNoted(obs.HeadSHA, adoptionWorkflowRefusedReason) store.MarkAdoptionEscalated(key, obs.HeadSHA, adoptionWorkflowRefusedReason) c.warn("stage labels: refusing workflow-path DCO re-home; escalating to needs-decision", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "noted", noted) if noted { return 0, nil } - if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(AdoptionWorkflowRefusedComment(obs))}); err != nil { - c.warn("stage labels: workflow-path refusal note failed", "repo", obs.Display, "pr", obs.Number, "error", err) - return 0, err - } - return 1, nil + return c.postAdoptionEscalationNote(ctx, store, key, obs, adoptionWorkflowRefusedReason, AdoptionWorkflowRefusedComment(obs), "stage labels: workflow-path refusal note failed") } if fence != "" { // The fence travels into the superseding PR's body; the @@ -1466,6 +1460,21 @@ type adoptionEscape struct { Reason string } +// postAdoptionEscalationNote posts body as the 085 note for reason on the +// head and records it as noted only once GitHub accepted it: marking first +// left a post that failed (a 500) noted, so the PR sat at 085 with no +// comment on that head ever. Callers check AdoptionEscalationNoted(head, +// reason) first; failed is the warning logged on an error. Returns the +// comments posted. +func (c *Engine) postAdoptionEscalationNote(ctx context.Context, store *stagelabels.Store, key string, obs stageObservation, reason, body, failed string) (int, error) { + if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(body)}); err != nil { + c.warn(failed, "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "reason", reason, "error", err) + return 0, err + } + store.MarkAdoptionEscalationNoted(key, obs.HeadSHA, reason) + return 1, nil +} + // adoptionCIPending reports whether the lane saw CI still running on the // head: a fix may be in flight, so the head is not stale yet. func adoptionCIPending(obs stageObservation) bool { @@ -1564,8 +1573,9 @@ func adoptionStalledReason(obs stageObservation, rec stagelabels.Record, policy // escapeStalledAdoption runs before the 074 pin for a PR the hive owns whose // lane verdict is still below the merge gate. It converts a dead-end // in-place adoption to a supersession, or escalates an adoption nobody is -// working to 085 with a comment naming the lane reason; the re-home and the -// note happen once per head. An escalation holds only while its reason +// working to 085 with a comment naming the lane reason; the re-home happens +// once per head and the note once per (head, reason), retried until GitHub +// accepts it. An escalation holds only while its reason // does: the sweep re-evaluates it and releases the PR to 074 when the // blocker clears (hivecommons/hive#13040). Returns the decision and the // number of comments posted. @@ -1599,12 +1609,32 @@ func (c *Engine) escapeStalledAdoption(ctx context.Context, store *stagelabels.S if rec.AdoptionEscalated(obs.HeadSHA) { // A workflow-path refusal is a verdict on the head itself: it holds // until the head moves, whatever the re-home or stall reasons say. - if rec.AdoptEscalatedReason == adoptionWorkflowRefusedReason || (rehomeReason == "" && adoptionStalledReason(obs, rec, policy, now) != "") { - return adoptionEscape{Escalated: true, Reason: rec.AdoptEscalatedReason}, 0, nil + // Its note is retried here if the post that recorded it failed: the + // path that refused the head does not run again for it. + if rec.AdoptEscalatedReason == adoptionWorkflowRefusedReason { + esc := adoptionEscape{Escalated: true, Reason: adoptionWorkflowRefusedReason} + if rec.AdoptionEscalationNoted(obs.HeadSHA, adoptionWorkflowRefusedReason) { + return esc, 0, nil + } + n, err := c.postAdoptionEscalationNote(ctx, store, key, obs, adoptionWorkflowRefusedReason, AdoptionWorkflowRefusedComment(obs), "stage labels: workflow-path refusal note failed") + return esc, n, err } - store.ClearAdoptionEscalation(key) - c.info("stage labels: adoption escalation reason cleared; back to adopted-by-hive", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "was", rec.AdoptEscalatedReason, "lane_reason", nonEmpty(obs.Reason, obs.LaneReason), "merge_commit", obs.Detail.MergeCommit) - rec, _ = store.Get(key) + stalled := "" + if rehomeReason == "" { + stalled = adoptionStalledReason(obs, rec, policy, now) + } + if stalled != "" && stalled == rec.AdoptEscalatedReason && rec.AdoptionEscalationNoted(obs.HeadSHA, stalled) { + return adoptionEscape{Escalated: true, Reason: stalled}, 0, nil + } + if stalled == "" { + store.ClearAdoptionEscalation(key) + c.info("stage labels: adoption escalation reason cleared; back to adopted-by-hive", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "was", rec.AdoptEscalatedReason, "lane_reason", nonEmpty(obs.Reason, obs.LaneReason), "merge_commit", obs.Detail.MergeCommit) + rec, _ = store.Get(key) + } + // Otherwise the escalation holds but its note never posted, or the + // blocker changed on this head (a check failure, then the hive's + // own sign-off attribution): the stall path below records the + // current reason and posts the note it is owed. } comments := 0 var firstErr error @@ -1626,18 +1656,15 @@ func (c *Engine) escapeStalledAdoption(ctx context.Context, store *stagelabels.S // Terminal for this head: refuse publication, keep the in-place // adoption and hand the re-home to a person (085) with one note. store.NoteAdoptDiffFence(key, fence) - noted := rec.AdoptionEscalationNoted(obs.HeadSHA) + noted := rec.AdoptionEscalationNoted(obs.HeadSHA, adoptionWorkflowRefusedReason) store.MarkAdoptionEscalated(key, obs.HeadSHA, adoptionWorkflowRefusedReason) c.warn("stage labels: refusing workflow-path re-home; escalating to needs-decision", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "reason", reason, "noted", noted) esc := adoptionEscape{Escalated: true, Reason: adoptionWorkflowRefusedReason} if noted { return esc, 0, nil } - if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(AdoptionWorkflowRefusedComment(obs))}); err != nil { - c.warn("stage labels: workflow-path refusal note failed", "repo", obs.Display, "pr", obs.Number, "error", err) - return esc, 0, err - } - return esc, 1, nil + n, err := c.postAdoptionEscalationNote(ctx, store, key, obs, adoptionWorkflowRefusedReason, AdoptionWorkflowRefusedComment(obs), "stage labels: workflow-path refusal note failed") + return esc, n, err } if fence != "" { store.NoteAdoptDiffFence(key, fence) @@ -1659,11 +1686,12 @@ func (c *Engine) escapeStalledAdoption(ctx context.Context, store *stagelabels.S if reason == "" { return adoptionEscape{}, comments, firstErr } - noted := rec.AdoptionEscalationNoted(obs.HeadSHA) + noted := rec.AdoptionEscalationNoted(obs.HeadSHA, reason) store.MarkAdoptionEscalated(key, obs.HeadSHA, reason) c.info("stage labels: adopted PR sits with nothing the hive can push; escalating to needs-decision", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "reason", reason, "head_since", rec.AdoptedHeadSince(), "noted", noted) if noted { - // The blocker flapped on a head already noted: re-escalate silently. + // The blocker flapped back to a reason already noted on this head: + // re-escalate silently. return adoptionEscape{Escalated: true, Reason: reason}, comments, firstErr } if adoptedHeadSignoffUnfixable(obs, rec) { @@ -1671,13 +1699,10 @@ func (c *Engine) escapeStalledAdoption(ctx context.Context, store *stagelabels.S obs = c.withMergeAskFacts(ctx, key, obs) } body := AdoptionStalledComment(obs, rec, reason, now.Sub(rec.AdoptedHeadSince())) - if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(body)}); err != nil { - c.warn("stage labels: adoption escalation note failed", "repo", obs.Display, "pr", obs.Number, "error", err) - if firstErr == nil { - firstErr = err - } - } else { - comments++ + n, err := c.postAdoptionEscalationNote(ctx, store, key, obs, reason, body, "stage labels: adoption escalation note failed") + comments += n + if err != nil && firstErr == nil { + firstErr = err } return adoptionEscape{Escalated: true, Reason: reason}, comments, firstErr } @@ -1687,7 +1712,8 @@ func (c *Engine) escapeStalledAdoption(ctx context.Context, store *stagelabels.S // stage dwell since the nudge: the same 085 park and maintainer ask an // adopted fork PR gets, but after the author's relay had its fix-pr window // rather than minutes after it went offline. A zero dwell (never stalled) -// never escalates. The note posts once per head. Returns the decision and +// never escalates. The note posts once per (head, reason), retried until +// GitHub accepts it. Returns the decision and // the number of comments posted. func (c *Engine) escalateUnadoptableFork(ctx context.Context, store *stagelabels.Store, key string, obs stageObservation, policy StageLabelPolicy, now time.Time) (adoptionEscape, int, error) { rec, ok := store.Get(key) @@ -1701,15 +1727,12 @@ func (c *Engine) escalateUnadoptableFork(ctx context.Context, store *stagelabels } reason := stagelabels.LaneReasonAdoptionStalledPrefix + adoptStallForkRehomeRefused esc := adoptionEscape{Escalated: true, Reason: reason} - noted := rec.AdoptionEscalationNoted(obs.HeadSHA) + noted := rec.AdoptionEscalationNoted(obs.HeadSHA, reason) store.MarkAdoptionEscalated(key, obs.HeadSHA, reason) if noted { return esc, 0, nil } c.info("stage labels: fork PR the hive cannot adopt waited past the 072 dwell; escalating to needs-decision", "repo", obs.Display, "pr", obs.Number, "head", obs.HeadSHA, "head_repo", obs.HeadRepo, "nudged_at", start, "lane_reason", nonEmpty(obs.LaneReason, obs.Reason)) - if _, _, err := c.gh.Issues.CreateComment(ctx, obs.Owner, obs.Repo, obs.Number, &gh.IssueComment{Body: gh.Ptr(ForkNotAdoptedComment(obs, reason, now.Sub(start)))}); err != nil { - c.warn("stage labels: fork escalation note failed", "repo", obs.Display, "pr", obs.Number, "error", err) - return esc, 0, err - } - return esc, 1, nil + n, err := c.postAdoptionEscalationNote(ctx, store, key, obs, reason, ForkNotAdoptedComment(obs, reason, now.Sub(start)), "stage labels: fork escalation note failed") + return esc, n, err } diff --git a/src/pkg/github/automerge/automerge_contributor_adopt_test.go b/src/pkg/github/automerge/automerge_contributor_adopt_test.go index fe4f64d890..c99eeaa694 100644 --- a/src/pkg/github/automerge/automerge_contributor_adopt_test.go +++ b/src/pkg/github/automerge/automerge_contributor_adopt_test.go @@ -3,6 +3,7 @@ package automerge import ( "context" "encoding/json" + "errors" "fmt" "net/http" "slices" @@ -1627,6 +1628,70 @@ func TestAdoptionForkWithoutPushStaysWithAuthorUntilDwell(t *testing.T) { stageIs(stagelabels.ReturnedToContributor, "new head") } +// The fork escalation note (escalateUnadoptableFork) is recorded only once +// GitHub accepted it: a 500 on the post is retried on the next sweep, and +// a weak opinion from a lane that errored on an approved head does not +// drop the parked PR to 080 for a pass (projectbluefin/utah#919, 10-28). +func TestForkEscalationNoteRetriedAndHeld(t *testing.T) { + p := adoptPolicy() + p.AdoptInPlace = true + h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p) + defer done() + h.sink.presence = &ContributorPresence{Registered: true, Online: false} + h.adopt.allowMaintainerEdits(7, false) + key := stageKey("acme", "widget", 7) + returned := []string{"stage/072-returned-to-contributor", "role/contributor", "author/clanker"} + parked := []string{"stage/085-needs-decision", "role/contributor", "author/clanker"} + want := stagelabels.LaneReasonAdoptionStalledPrefix + adoptStallForkRehomeRefused + + h.observe(forkPR(7, "aaa"), "conflicting") + h.clock.advance(30 * time.Minute) + h.observe(forkPR(7, "aaa", returned...), "conflicting") + h.clock.advance(24 * time.Hour) + h.api.reset() + h.api.failComments, h.api.failCommentsOn = 1, AdoptionStalledMarker + if _, err := h.observeErr(forkPR(7, "aaa", returned...), "conflicting"); err == nil { + t.Fatal("the refused note was not reported") + } + rec, _ := h.e.stageStoreOrShared().Get(key) + if rec.Stage != stagelabels.NeedsDecision || rec.Reason != want || rec.AdoptionEscalationNoted("aaa", want) || len(stalledComments(h.api)) != 0 { + t.Fatalf("failed post: %+v comments=%v", rec, h.api.comments) + } + + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(forkPR(7, "aaa", parked...), "conflicting") + if notes := stalledComments(h.api); len(notes) != 1 || !strings.Contains(notes[0], "reason="+want) { + t.Fatalf("fork note not retried: %v", h.api.comments) + } + if len(h.api.added) != 0 || len(h.api.removed) != 0 { + t.Fatalf("retry churned labels: added=%v removed=%v", h.api.added, h.api.removed) + } + + h.api.reset() + h.clock.advance(5 * time.Minute) + h.observe(forkPR(7, "aaa", parked...), "conflicting") + if len(h.api.comments) != 0 { + t.Fatalf("fork note repeated: %v", h.api.comments) + } + + h.api.reset() + h.clock.advance(2 * time.Minute) + pr := forkPR(7, "aaa", parked...) + h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, "", false, errors.New("lane tripped")) + h.e.noteStageRole("acme", "widget", 7, stagelabels.RoleContributor) + h.e.noteStageVerdict("acme", "widget", 7, ContributorLaneVerdict{Configured: true, Approved: true, Verdict: "approve"}) + if _, err := h.e.ApplyStageLabels(context.Background()); err != nil { + t.Fatal(err) + } + if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision || rec.Reason != want { + t.Fatalf("errored lane dropped the fork escalation: %+v", rec) + } + if len(h.api.added) != 0 || len(h.api.removed) != 0 || len(h.api.comments) != 0 { + t.Fatalf("fork escalation churned: added=%v removed=%v comments=%v", h.api.added, h.api.removed, h.api.comments) + } +} + // An explicit owner-delegated adoption still goes through the supersede // path, which refuses a fork and parks it for a maintainer: the request is // one attempt, so it must be recorded, not declined like the timed path. @@ -2137,6 +2202,7 @@ func TestDelegatedSupersedeWorkflowPathNotesOverStalledNote(t *testing.T) { h.e.noteStageRole("acme", "widget", 7, stagelabels.RoleContributor) key := stageKey("acme", "widget", 7) h.e.stageStoreOrShared().MarkAdoptionEscalated(key, "aaa", stagelabels.LaneReasonAdoptionStalledPrefix+"stalled") + h.e.stageStoreOrShared().MarkAdoptionEscalationNoted(key, "aaa", stagelabels.LaneReasonAdoptionStalledPrefix+"stalled") if _, err := h.e.ApplyStageLabels(context.Background()); err != nil { t.Fatal(err) } @@ -2340,3 +2406,40 @@ func TestAdoptionClosesCarriesRefs(t *testing.T) { t.Fatalf("adoptionCloses: %v", got) } } + +// A workflow-path refusal note GitHub answered with a 500 is posted on the +// next sweep: the refusing path does not run again for the head, so the +// held escalation retries it, once. +func TestWorkflowPathRefusalNoteRetriedAfterFailedPost(t *testing.T) { + h, done := newAdoptHarness(t, ContributorNudgeOutcome{}, adoptPolicy()) + defer done() + h.adopt.prFiles = []map[string]any{{"filename": ".github/workflows/ci.yml", "status": "modified"}} + pr := adoptPR(7, "aaa", "stage/050-review-pending", delegatedecision.LabelAdoptSupersede) + h.adopt.mu.Lock() + h.adopt.heads[7] = "aaa" + h.adopt.mu.Unlock() + key := stageKey("acme", "widget", 7) + h.api.failComments, h.api.failCommentsOn = 1, AdoptionWorkflowRefusedMarker + h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, contributorLaneReasonNoReviewer, false, nil) + h.e.noteStageRole("acme", "widget", 7, stagelabels.RoleContributor) + if _, err := h.e.ApplyStageLabels(context.Background()); err == nil { + t.Fatal("the refused note was not reported") + } + rec, _ := h.e.stageStoreOrShared().Get(key) + if rec.AdoptEscalatedReason != adoptionWorkflowRefusedReason || rec.AdoptionEscalationNoted("aaa", adoptionWorkflowRefusedReason) || len(workflowRefusedComments(h.api)) != 0 { + t.Fatalf("failed post: %+v comments=%v", rec, h.api.comments) + } + for i, want := range []int{1, 0} { + h.api.reset() + h.clock.advance(5 * time.Minute) + // A lane opinion the adoption pin claims (review pending) is what + // re-runs the escape for the adopted head. + h.observe(adoptPR(7, "aaa", "stage/085-needs-decision", "role/contributor", "author/clanker"), contributorLaneReasonNoVerdict) + if got := len(workflowRefusedComments(h.api)); got != want { + t.Fatalf("sweep %d: refusal notes = %d, want %d: %v", i+1, got, want, h.api.comments) + } + if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision { + t.Fatalf("sweep %d: stage %s", i+1, rec.Stage) + } + } +} diff --git a/src/pkg/github/automerge/automerge_merge_commit_test.go b/src/pkg/github/automerge/automerge_merge_commit_test.go index 7d1a011dab..efc284792d 100644 --- a/src/pkg/github/automerge/automerge_merge_commit_test.go +++ b/src/pkg/github/automerge/automerge_merge_commit_test.go @@ -238,6 +238,7 @@ func TestParkedSignoffAttributionEscalationReleasesOnMergeCommitElection(t *test // The previous build's record: escalated and noted on this head, the // adopted-head clock long past the dwell, stage 085 written. store.MarkAdoptionEscalated(key, "a1899cc", stalled) + store.MarkAdoptionEscalationNoted(key, "a1899cc", stalled) store.Observe(key, stagelabels.NeedsDecision, stalled) h.clock.advance(adoptedDwell + time.Hour) if rec, _ := store.Get(key); !rec.AdoptionEscalated("a1899cc") || rec.Stage != stagelabels.NeedsDecision { @@ -250,7 +251,7 @@ func TestParkedSignoffAttributionEscalationReleasesOnMergeCommitElection(t *test if rec.AdoptionEscalated("a1899cc") || rec.Stage != stagelabels.AdoptedByHive || rec.Reason == stalled { t.Fatalf("escalation not released: %+v", rec) } - if !rec.AdoptionEscalationNoted("a1899cc") { + if !rec.AdoptionEscalationNoted("a1899cc", stalled) { t.Fatalf("noted head forgotten: %+v", rec) } if added := h.addedLabels(); !containsLabel(added, stagelabels.MergeCommitLabel) || !containsLabel(added, stagelabels.AdoptedByHive.Label()) { diff --git a/src/pkg/github/automerge/automerge_stage_labels.go b/src/pkg/github/automerge/automerge_stage_labels.go index 87e8d2e39e..f1c4337d6d 100644 --- a/src/pkg/github/automerge/automerge_stage_labels.go +++ b/src/pkg/github/automerge/automerge_stage_labels.go @@ -984,6 +984,21 @@ func (c *Engine) ApplyStageLabels(ctx context.Context) (StageLabelResult, error) pass.ForkEscalationReason = esc.Reason } } + if hadPrev && stageengine.HoldsEscalation(verdict) && prevRec.AdoptionEscalated(obs.HeadSHA) && (prevRec.AdoptionLive(obs.HeadSHA) || prevRec.ForkRehomeRefused(obs.HeadSHA)) { + // The escalation is the hive's own verdict on this head; a + // weak opinion (an errored lane reading an approved head as + // 080) does not release it — only the escape above, on an + // owned opinion, does. Without the hold the PR flipped + // 085 → 080 → 085 on one head. + verdict.Stage = stagelabels.NeedsDecision + verdict.Rank = stageRankOwned + obs.Reason = prevRec.AdoptEscalatedReason + if prevRec.AdoptionLive(obs.HeadSHA) { + pass.AdoptionLive, pass.EscalationReason = true, prevRec.AdoptEscalatedReason + } else { + pass.ForkEscalationReason = prevRec.AdoptEscalatedReason + } + } tr = store.Observe(key, verdict.Stage, obs.Reason) // hivecommons/hive#12502: remember the author so the dashboard can // list "PRs returned to this contributor" from the store alone. diff --git a/src/pkg/github/automerge/automerge_stage_labels_test.go b/src/pkg/github/automerge/automerge_stage_labels_test.go index 2d4d30c2cc..d73427a8b1 100644 --- a/src/pkg/github/automerge/automerge_stage_labels_test.go +++ b/src/pkg/github/automerge/automerge_stage_labels_test.go @@ -48,6 +48,12 @@ type stageAPI struct { // lane's reason alone states it. prFiles map[int][]string fileReads int + // failComments answers that many comment POSTs with a 500 before + // accepting any, as GitHub did for the adoption-stalled note on + // projectbluefin/utah-packages#650; failCommentsOn, when set, limits + // that to bodies starting with it. + failComments int + failCommentsOn string // extra handles routes beyond labels/comments (returns true when it // served the request); the adoption tests use it for refs and pulls. extra func(w http.ResponseWriter, r *http.Request) bool @@ -154,6 +160,11 @@ func newStageAPI(t *testing.T, api *stageAPI) *httptest.Server { case r.Method == http.MethodPost && strings.HasSuffix(path, "/comments"): var body map[string]string _ = json.NewDecoder(r.Body).Decode(&body) + if api.failComments > 0 && strings.HasPrefix(body["body"], api.failCommentsOn) { + api.failComments-- + w.WriteHeader(http.StatusInternalServerError) + return + } api.comments = append(api.comments, body["body"]) w.WriteHeader(http.StatusCreated) _ = json.NewEncoder(w).Encode(map[string]any{"id": 1}) diff --git a/src/pkg/github/automerge/stage_facts_test.go b/src/pkg/github/automerge/stage_facts_test.go index f42036c8f7..aa06cfe9f7 100644 --- a/src/pkg/github/automerge/stage_facts_test.go +++ b/src/pkg/github/automerge/stage_facts_test.go @@ -279,6 +279,69 @@ func TestPRDeriverPassCarriesForkEscalation(t *testing.T) { } } +// TestPRDeriverPassHoldsEscalationAgainstWeakOpinion: a lane that errored +// on an approved head reads it author-side done, 080 at role rank. With an +// active escalation on the head the pass keeps 085 (projectbluefin/utah#919 +// flipped 085 → 080 → 085 on one head); the engine must derive the same +// from PRPassFacts, and 080 without an escalation. +func TestPRDeriverPassHoldsEscalationAgainstWeakOpinion(t *testing.T) { + approved := stageDetail{Role: stagelabels.RoleContributor, VerdictKnown: true, Verdict: ContributorLaneVerdict{Configured: true, Approved: true, Verdict: "approve"}} + list := []stageObservation{{Owner: "acme", Repo: "widgets", Number: 9, Author: "relay", HeadSHA: "sha", Lane: stageLaneContributor, Err: true, Detail: approved}} + if v, _, _ := resolveStage(list); v.Stage != stagelabels.AwaitingMergeAdmission || v.Rank != stageRankRole { + t.Fatalf("fixture: errored approved lane resolved %+v, want weak 080", v) + } + stalled := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failure" + fork := stagelabels.LaneReasonAdoptionStalledPrefix + adoptStallForkRehomeRefused + for _, tc := range []struct { + name string + pass stageengine.PRPassFacts + stage stagelabels.Stage + reason string + }{ + {"adoption escalated", stageengine.PRPassFacts{AuthorKind: authorkind.Clanker, AdoptionLive: true, EscalationReason: stalled}, stagelabels.NeedsDecision, stalled}, + {"fork escalated", stageengine.PRPassFacts{AuthorKind: authorkind.Clanker, ForkEscalationReason: fork}, stagelabels.NeedsDecision, fork}, + {"adopted, not escalated", stageengine.PRPassFacts{AuthorKind: authorkind.Clanker, AdoptionLive: true}, stagelabels.AwaitingMergeAdmission, ""}, + {"nothing", stageengine.PRPassFacts{AuthorKind: authorkind.Clanker}, stagelabels.AwaitingMergeAdmission, ""}, + } { + f := factsFrom(list) + pass := tc.pass + f.PR.Pass = &pass + got := stageengine.PRDeriver().Derive(f) + if got.Abstain || got.Stage != tc.stage || (tc.reason != "" && got.Reason != tc.reason) { + t.Errorf("%s: derived %+v, want %s %q", tc.name, got, tc.stage, tc.reason) + } + } + + // Only a weak opinion is held: an owned one past the merge gate (the + // lane admitted the head and is soaking it) is the lane's to state, and + // a weak 085 (a requires_human verdict on an author no lane vouches + // for, hivecommons/hive#13115) keeps its own reason. + soaking := []stageObservation{{Owner: "acme", Repo: "widgets", Number: 9, Author: "relay", HeadSHA: "sha", Lane: stageLaneContributor, Reason: contributorLaneReasonSoaking, Detail: approved}} + weakHuman := []stageObservation{{Owner: "acme", Repo: "widgets", Number: 9, Author: "relay", HeadSHA: "sha", Lane: stageLaneContributor, Reason: "untrusted-author-role", Detail: stageDetail{VerdictKnown: true, Verdict: ContributorLaneVerdict{Configured: true, Verdict: "requires_human"}}}} + for _, tc := range []struct { + name string + list []stageObservation + rank int + }{ + {"owned soak", soaking, stageRankOwned}, + {"weak 085", weakHuman, stageRankRole}, + } { + v, _, _ := resolveStage(tc.list) + owned := tc.rank == stageRankOwned + if (owned && (v.Rank != stageRankOwned || v.Stage.YieldsToAdoption() || v.Stage == stagelabels.NeedsDecision)) || (!owned && (v.Rank >= stageRankOwned || v.Stage != stagelabels.NeedsDecision)) { + t.Fatalf("%s fixture: resolved %+v", tc.name, v) + } + plain := factsFrom(tc.list) + plain.PR.Pass = &stageengine.PRPassFacts{AuthorKind: authorkind.Clanker, AdoptionLive: true} + want := stageengine.PRDeriver().Derive(plain) + f := factsFrom(tc.list) + f.PR.Pass = &stageengine.PRPassFacts{AuthorKind: authorkind.Clanker, AdoptionLive: true, EscalationReason: stalled} + if got := stageengine.PRDeriver().Derive(f); got.Stage != want.Stage || got.Reason != want.Reason || got.Reason == stalled { + t.Errorf("%s: escalation overrode the opinion: derived %+v, want %+v", tc.name, got, want) + } + } +} + // TestVerdictStateFold pins the ReviewFacts.State fold ADR-0025 defines // (review-state fold, resolved in plan PR-F): the one verdict fact the PR // deriver routes on. diff --git a/src/pkg/stageengine/derive_pr.go b/src/pkg/stageengine/derive_pr.go index 3a18cb8a33..d4802665f9 100644 --- a/src/pkg/stageengine/derive_pr.go +++ b/src/pkg/stageengine/derive_pr.go @@ -287,6 +287,17 @@ func passDecision(verdict stageVerdict, role stagelabels.Role, obs stageObservat verdict.Rank = stageRankOwned obs.Reason = pass.ForkEscalationReason } + if HoldsEscalation(verdict) { + reason := pass.ForkEscalationReason + if pass.AdoptionLive { + reason = pass.EscalationReason + } + if reason != "" { + verdict.Stage = stagelabels.NeedsDecision + verdict.Rank = stageRankOwned + obs.Reason = reason + } + } d := Decision{ Stage: verdict.Stage, Role: role, @@ -311,6 +322,18 @@ func passDecision(verdict stageVerdict, role stagelabels.Role, obs stageObservat return d } +// HoldsEscalation reports whether an active 085 escalation on the head (a +// stalled adoption or an unadoptable fork PR) outranks the pass's opinion: +// a weak one at or past the merge gate that neither yields to adoption nor +// is already 085. Such an opinion is not evidence the blocker cleared — an +// errored lane whose approved head looks author-side done reads 080 at role +// rank — and dropping the escalation on it flipped a parked PR 085 → 080 → +// 085 within one sweep pair on an unchanged head. The escalation holds for +// the pass; the next owned opinion is what re-evaluates it. +func HoldsEscalation(v stageVerdict) bool { + return !v.Terminal && v.Rank != stageRankNone && v.Rank < stageRankOwned && v.Stage != stagelabels.None && v.Stage != stagelabels.NeedsDecision && !v.Stage.YieldsToAdoption() +} + // decisionFrom renders ResolveStage's answer as a Decision. The recorded // reason is the verdict's own when it set one, else the owning lane's. func decisionFrom(v stageVerdict, role stagelabels.Role, obs stageObservation, f Facts) Decision { diff --git a/src/pkg/stageengine/facts.go b/src/pkg/stageengine/facts.go index 04e577cc73..f8158f26a6 100644 --- a/src/pkg/stageengine/facts.go +++ b/src/pkg/stageengine/facts.go @@ -247,11 +247,13 @@ type PRPassFacts struct { SignoffHiveCaused bool // AdoptionLive is prev.AdoptionLive(head): the hive owns the PR, so a // stage that yields to adoption reads 074. EscalationReason is set when - // the stalled-adoption escape moved it to 085 instead. + // the stalled-adoption escape moved it to 085 instead, or when the pass + // held that escalation against a weak opinion (HoldsEscalation). AdoptionLive bool EscalationReason string // ForkEscalationReason is set when a fork PR the hive declined to adopt - // waited at 072 past the stage dwell and the pass moved it to 085. + // waited at 072 past the stage dwell and the pass moved it to 085, or + // held that escalation against a weak opinion. ForkEscalationReason string // Stalled is the pass's dwell verdict (stagelabels.StalledWith over the // store record after this pass's Observe); KindLabels are the diff --git a/src/pkg/stagelabels/stagelabels_test.go b/src/pkg/stagelabels/stagelabels_test.go index 4861693928..741eb8143e 100644 --- a/src/pkg/stagelabels/stagelabels_test.go +++ b/src/pkg/stagelabels/stagelabels_test.go @@ -1211,11 +1211,21 @@ func TestStoreAdoptionEscapeBookkeeping(t *testing.T) { } s.MarkAdoptionEscalated(key, "f72fa44", "adoption-stalled:check-failing:unit") r, _ = s.Get(key) - if !r.AdoptionEscalated("f72fa44") || r.AdoptionEscalated("5665cda") || r.AdoptEscalatedReason != "adoption-stalled:check-failing:unit" || !r.AdoptionEscalationNoted("f72fa44") { + if !r.AdoptionEscalated("f72fa44") || r.AdoptionEscalated("5665cda") || r.AdoptEscalatedReason != "adoption-stalled:check-failing:unit" { t.Fatalf("escalation not recorded: %+v", r) } + // The escalation alone is not a posted note: a post GitHub refused + // must be retried, not read back as noted. + if r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:check-failing:unit") || r.AdoptEscalationNotedHead != "" { + t.Fatalf("escalating marked the note posted: %+v", r) + } + s.MarkAdoptionEscalationNoted(key, "f72fa44", "adoption-stalled:check-failing:unit") + if r, _ = s.Get(key); !r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:check-failing:unit") || r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:hive-signoff-attribution") { + t.Fatalf("note not recorded per reason: %+v", r) + } s.dirty = false s.MarkAdoptionEscalated(key, "f72fa44", "adoption-stalled:check-failing:unit") + s.MarkAdoptionEscalationNoted(key, "f72fa44", "adoption-stalled:check-failing:unit") if s.dirty { t.Fatal("re-marking the same escalation dirtied the store") } @@ -1223,9 +1233,14 @@ func TestStoreAdoptionEscapeBookkeeping(t *testing.T) { // clears, but the head stays noted so a flapping blocker is noted once. s.ClearAdoptionEscalation(key) r, _ = s.Get(key) - if r.AdoptionEscalated("f72fa44") || r.AdoptEscalatedReason != "" || !r.AdoptionEscalationNoted("f72fa44") || r.AdoptionEscalationNoted("5665cda") { + if r.AdoptionEscalated("f72fa44") || r.AdoptEscalatedReason != "" || !r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:check-failing:unit") || r.AdoptionEscalationNoted("5665cda", "adoption-stalled:check-failing:unit") { t.Fatalf("escalation not released: %+v", r) } + // A second reason on the same head is noted beside the first. + s.MarkAdoptionEscalationNoted(key, "f72fa44", "adoption-stalled:hive-signoff-attribution") + if r, _ = s.Get(key); !r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:check-failing:unit") || !r.AdoptionEscalationNoted("f72fa44", "adoption-stalled:hive-signoff-attribution") { + t.Fatalf("second reason lost the first: %+v", r) + } s.dirty = false s.ClearAdoptionEscalation(key) s.ClearAdoptionEscalation("missing") @@ -1261,6 +1276,54 @@ func TestStoreAdoptionEscapeBookkeeping(t *testing.T) { } } +// A note marked for one head starts over on the next, and a record written +// before the noted reasons were kept reads as noted for every reason on its +// head (no mass repost after an upgrade) but for nothing on another head. +func TestAdoptionEscalationNotedPerHeadAndLegacy(t *testing.T) { + s := &Store{items: map[string]Record{}} + const key = "o/r#650" + s.items[key] = Record{AdoptEscalatedHead: "aaa", AdoptEscalatedReason: "adoption-stalled:check-failure", AdoptEscalationNotedHead: "aaa"} + r, _ := s.Get(key) + if !r.AdoptionEscalationNoted("aaa", "adoption-stalled:check-failure") || !r.AdoptionEscalationNoted("aaa", "adoption-stalled:hive-signoff-attribution") { + t.Fatalf("legacy noted head not read as noted for every reason: %+v", r) + } + if r.AdoptionEscalationNoted("bbb", "adoption-stalled:check-failure") || r.AdoptionEscalationNoted("", "") { + t.Fatalf("legacy noted head leaked to another head: %+v", r) + } + s.dirty = false + s.MarkAdoptionEscalationNoted(key, "aaa", "adoption-stalled:hive-signoff-attribution") + if r, _ = s.Get(key); s.dirty || len(r.AdoptEscalationNotedReasons) != 0 { + t.Fatalf("marking a legacy noted head narrowed it to one reason: %+v", r) + } + s.MarkAdoptionEscalationNoted(key, "bbb", "adoption-stalled:check-failure") + r, _ = s.Get(key) + if r.AdoptEscalationNotedHead != "bbb" || !r.AdoptionEscalationNoted("bbb", "adoption-stalled:check-failure") || r.AdoptionEscalationNoted("bbb", "adoption-stalled:x") || r.AdoptionEscalationNoted("aaa", "adoption-stalled:check-failure") { + t.Fatalf("new head did not start the reasons over: %+v", r) + } + s.MarkAdoptionEscalationNoted(key, "ccc", "adoption-stalled:hive-signoff-attribution") + if r, _ = s.Get(key); r.AdoptionEscalationNoted("ccc", "adoption-stalled:check-failure") || !r.AdoptionEscalationNoted("ccc", "adoption-stalled:hive-signoff-attribution") { + t.Fatalf("a reason noted on the previous head carried over: %+v", r) + } + s.MarkAdoptionEscalationNoted(key, "", "adoption-stalled:x") + s.MarkAdoptionEscalationNoted(key, "ccc", "") + s.MarkAdoptionEscalationNoted("missing", "bbb", "adoption-stalled:x") + if r, _ = s.Get(key); len(r.AdoptEscalationNotedReasons) != 1 || r.AdoptEscalationNotedHead != "ccc" { + t.Fatalf("empty head/reason or missing key was recorded: %+v", r) + } + if _, ok := s.Get("missing"); ok { + t.Fatal("MarkAdoptionEscalationNoted created a record") + } + // The reasons survive the JSON round trip the store file takes. + b, err := json.Marshal(r) + if err != nil { + t.Fatal(err) + } + var back Record + if err := json.Unmarshal(b, &back); err != nil || !back.AdoptionEscalationNoted("ccc", "adoption-stalled:hive-signoff-attribution") || back.AdoptionEscalationNoted("ccc", "adoption-stalled:x") { + t.Fatalf("reasons lost in JSON (%v): %s", err, b) + } +} + // hivecommons/hive#13040: a record written before AdoptedHeadAt existed // carries a head the hive pushed but no clock for it. The reconciler stamps // the clock on first sight instead of reading the dwell from AdoptedAt. diff --git a/src/pkg/stagelabels/store.go b/src/pkg/stagelabels/store.go index 2b9d89d00c..be08be66c2 100644 --- a/src/pkg/stagelabels/store.go +++ b/src/pkg/stagelabels/store.go @@ -170,10 +170,15 @@ type Record struct { // when the blocker clears or the head moves. AdoptEscalatedHead string `json:"adopt_escalated_head,omitempty"` AdoptEscalatedReason string `json:"adopt_escalated_reason,omitempty"` - // AdoptEscalationNotedHead is the head the escalation note was posted - // for; it survives ClearAdoptionEscalation so a blocker that flaps on - // one head is noted once. - AdoptEscalationNotedHead string `json:"adopt_escalation_noted_head,omitempty"` + // AdoptEscalationNotedHead / AdoptEscalationNotedReasons are the head + // the escalation note was posted for and every escalation reason a note + // explained on it. They are written only once GitHub accepted the post, + // so a failed one is retried on a later sweep, and they survive + // ClearAdoptionEscalation so a blocker that flaps on one head is noted + // once per reason. A head with no reasons is a record from before the + // reasons were kept; AdoptionEscalationNoted reads it as noted for any. + AdoptEscalationNotedHead string `json:"adopt_escalation_noted_head,omitempty"` + AdoptEscalationNotedReasons []string `json:"adopt_escalation_noted_reasons,omitempty"` // AdoptRecoveryAttempt is the "@" pair the hive last // tried to recover a stale-red in-place adoption on (update-branch or a // re-run of the failed jobs, hivecommons/hive#14500); one attempt per @@ -923,10 +928,20 @@ func (r Record) AdoptionEscalated(head string) bool { return head != "" && r.AdoptEscalatedHead == head } -// AdoptionEscalationNoted reports whether the escalation note was already -// posted for head, whether or not the escalation still holds. -func (r Record) AdoptionEscalationNoted(head string) bool { - return head != "" && r.AdoptEscalationNotedHead == head +// AdoptionEscalationNoted reports whether an escalation note for reason +// was already posted on head, whether or not the escalation still holds. A +// new reason on the same head is not noted: the maintainer who read the +// first note must learn what holds the PR now, as the 085 entry comment +// does (hivecommons/hive#13139). Records written before reasons were kept +// (a noted head, no reasons) are taken as noted for every reason on that +// head: the reason they posted is unknown — a cleared escalation keeps +// none — and a repost on every parked PR after an upgrade is worse than +// one missed refresh. +func (r Record) AdoptionEscalationNoted(head, reason string) bool { + if head == "" || r.AdoptEscalationNotedHead != head { + return false + } + return len(r.AdoptEscalationNotedReasons) == 0 || slices.Contains(r.AdoptEscalationNotedReasons, reason) } // ForkRehomeRefused reports whether the hive declined to re-home head, a @@ -1066,6 +1081,7 @@ func (s *Store) MarkYielded(key, head string) { r.AdoptedHead, r.AdoptedBy, r.AdoptedAt, r.AdoptCheckedAt = "", 0, time.Time{}, time.Time{} r.AdoptMode, r.AdoptReason, r.AdoptDiffFence = "", "", "" r.AdoptedHeadAt, r.AdoptEscalatedHead, r.AdoptEscalatedReason, r.AdoptEscalationNotedHead = time.Time{}, "", "", "" + r.AdoptEscalationNotedReasons = nil r.AdoptRecoveryAttempt, r.AdoptRecoveryCount = "", 0 r.YieldedHead = head s.items[key] = r @@ -1115,16 +1131,37 @@ func (s *Store) MarkAdoptChecked(key string, now time.Time) { } // MarkAdoptionEscalated records that the adoption at head is a person's -// for reason (stage 085) and that head has been noted; AdoptionEscalated -// (head) then holds until ClearAdoptionEscalation or the head changes. +// for reason (stage 085); AdoptionEscalated(head) then holds until +// ClearAdoptionEscalation or the head changes. It does not mark the note: +// MarkAdoptionEscalationNoted does, after the post succeeds, so a note +// GitHub refused is not taken as posted. func (s *Store) MarkAdoptionEscalated(key, head, reason string) { s.mu.Lock() defer s.mu.Unlock() r, ok := s.items[key] - if !ok || head == "" || (r.AdoptEscalatedHead == head && r.AdoptEscalatedReason == reason && r.AdoptEscalationNotedHead == head) { + if !ok || head == "" || (r.AdoptEscalatedHead == head && r.AdoptEscalatedReason == reason) { + return + } + r.AdoptEscalatedHead, r.AdoptEscalatedReason = head, reason + s.items[key] = r + s.dirty = true +} + +// MarkAdoptionEscalationNoted records that the escalation note for reason +// was posted on head. A new head starts the reason list over; a record from +// before reasons were kept already reads as noted for every reason on its +// head and is left alone. +func (s *Store) MarkAdoptionEscalationNoted(key, head, reason string) { + s.mu.Lock() + defer s.mu.Unlock() + r, ok := s.items[key] + if !ok || head == "" || reason == "" || r.AdoptionEscalationNoted(head, reason) { return } - r.AdoptEscalatedHead, r.AdoptEscalatedReason, r.AdoptEscalationNotedHead = head, reason, head + if r.AdoptEscalationNotedHead != head { + r.AdoptEscalationNotedHead, r.AdoptEscalationNotedReasons = head, nil + } + r.AdoptEscalationNotedReasons = append(slices.Clone(r.AdoptEscalationNotedReasons), reason) s.items[key] = r s.dirty = true } @@ -1178,7 +1215,8 @@ func (s *Store) MarkSignoffAttributionNoted(key, head string) { // ClearAdoptionEscalation releases a stage-085 escalation whose reason no // longer holds (the blocker cleared), returning the adoption to 074 flow. -// The noted head is kept so the same head is not noted twice. +// The noted head and reasons are kept so the same head is not noted twice +// for one reason. func (s *Store) ClearAdoptionEscalation(key string) { s.mu.Lock() defer s.mu.Unlock() @@ -1202,6 +1240,7 @@ func (s *Store) ClearAdoption(key string) { r.AdoptedHead, r.AdoptedBy, r.AdoptedAt, r.AdoptCheckedAt = "", 0, time.Time{}, time.Time{} r.AdoptMode, r.AdoptReason, r.AdoptDiffFence = "", "", "" r.AdoptedHeadAt, r.AdoptEscalatedHead, r.AdoptEscalatedReason, r.AdoptEscalationNotedHead = time.Time{}, "", "", "" + r.AdoptEscalationNotedReasons = nil r.AdoptRecoveryAttempt, r.AdoptRecoveryCount = "", 0 r.SupersededBy, r.SupersededAt, r.SupersededOpen, r.SupersederMergedNotedHead = 0, time.Time{}, false, "" s.items[key] = r