Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-085-escalation-note-retry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **A PR parked at `stage/085-needs-decision` for a stalled adoption is no longer left without its note.** The escalation marked the head noted before posting, so when GitHub answered the `<!-- hive:adoption-stalled … -->` (or workflow-path refusal, or unadoptable-fork) note with a 500 the PR sat at 085 with no comment on that head ever (projectbluefin/utah-packages#650, #652). The note is now recorded only once posted and is retried on later sweeps; it posts once per head and reason, so a different blocker on the same head (a check failure, then the hive's own sign-off attribution) gets its own note while the same one flapping back stays silent. A lane that errored on an approved head (080 at role rank) no longer drops an active escalation for a pass, which flipped projectbluefin/utah#919 085 β†’ 080 β†’ 085 on one head. Stage records from the previous build are read as already noted for their head, so the upgrade reposts nothing.
3 changes: 2 additions & 1 deletion src/docs/hive-merge.md
Original file line number Diff line number Diff line change
Expand Up @@ -774,7 +774,8 @@ at most once per (head, base tip) and three times per adoption, logged as
(hivecommons/hive#14500). An adoption with nothing the hive can
push past the dwell β€” after that recovery was tried and the head is still
red β€” goes to `stage/085-needs-decision` as
`adoption-stalled:<lane reason>` with one comment naming the blocker, so the
`adoption-stalled:<lane reason>` with one comment naming the blocker (once
per head and reason, retried until GitHub accepts it), so the
dashboard shows a person's item, not a silent 074; "waiting on maintainer
lgtm" is only ever the state at 080+ (hivecommons/hive#12858). The dwell runs
from the head's last hive push, a red optional check alone is not a blocker,
Expand Down
15 changes: 11 additions & 4 deletions src/docs/labels-and-control-signals.md
Original file line number Diff line number Diff line change
Expand Up @@ -933,7 +933,8 @@ When a clanker-authored contributor PR goes red the hive does not just relabel i
queued `fix-pr` task, and after the `072` dwell (24h) since the nudge on
that head it parks at `085` as `adoption-stalled:fork-rehome-refused`
with one note asking a maintainer to review it and bring it in or close
it. A new head from the author re-arms the normal path. An explicit
it (once per head and reason, retried on later sweeps until GitHub
accepts it; an errored lane's weak 080 does not lift it). A new head from the author re-arms the normal path. An explicit
owner-delegated adoption request is still recorded and parks at once.
- `reason=dco-rehome` (always `mode=supersede`) β€” not a fresh adoption
but a second takeover: an in-place-adopted head stayed
Expand Down Expand Up @@ -1079,9 +1080,15 @@ When a clanker-authored contributor PR goes red the hive does not just relabel i
person to decide and the PR stays 074. The escalation holds only while
its reason holds β€” the sweep re-evaluates it every pass and when the
blocker clears (CI re-runs, the check goes green, a verdict lands) the
PR returns to 074 and the lane carries on; the comment is posted once
per head even if the blocker flaps, and a hive push to a new head
starts over. A pending `tide` or missing `lgtm` is never the reason β€”
PR returns to 074 and the lane carries on. A weak opinion is not such
a release: a lane that errored on an approved head (080 at role rank)
leaves the escalation at 085 for that pass instead of flipping the PR
085 β†’ 080 β†’ 085. The comment is posted once per (head, reason): the
same blocker flapping back stays silent, a different blocker on the
same head (a check failure, then the hive's own sign-off attribution)
gets its own note, and a post GitHub refuses is not counted, so the
next sweep retries it until it lands. A hive push to a new head starts
over. A pending `tide` or missing `lgtm` is never the reason β€”
the lane has not admitted the head; the comment says so.
- **supersede** (`mode=supersede`) β€” the fork forbids maintainer edits,
`in_place: false`, or the adopted head has unsigned commits (`no-dco`)
Expand Down
206 changes: 204 additions & 2 deletions src/pkg/github/automerge/automerge_adoption_escape_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,15 @@ package automerge
import (
"context"
"encoding/json"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"

gh "github.com/google/go-github/v72/github"

"github.com/hivecommons/hive/pkg/stagelabels"
)

Expand Down Expand Up @@ -533,7 +536,7 @@ func TestAdoptedEscalationReleasesWhenReasonClears(t *testing.T) {
if added := h.addedLabels(); len(added) != 1 || added[0] != stagelabels.AdoptedByHive.Label() || len(h.api.removed) != 1 || h.api.removed[0] != stagelabels.NeedsDecision.Label() {
t.Fatalf("085 β†’ 074: added=%v removed=%v comments=%v", h.api.added, h.api.removed, h.api.comments)
}
if rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("aaa") || !rec.AdoptionEscalationNoted("aaa") || !rec.AdoptedInPlace() || rec.AdoptedHead != "aaa" {
if rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("aaa") || !rec.AdoptionEscalationNoted("aaa", stagelabels.LaneReasonAdoptionStalledPrefix+"check-failing:unit") || !rec.AdoptedInPlace() || rec.AdoptedHead != "aaa" {
t.Fatalf("escalation not released: %+v", rec)
}
if len(stalledComments(h.api)) != 0 {
Expand All @@ -560,7 +563,7 @@ func TestAdoptedEscalationReleasesWhenReasonClears(t *testing.T) {
h.api.reset()
h.clock.advance(time.Minute)
h.observe(adoptPR(7, "bbb", escalated...), "check-failing:unit")
if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("bbb") || rec.AdoptionEscalationNoted("bbb") {
if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.AdoptedByHive || rec.AdoptionEscalated("bbb") || rec.AdoptionEscalationNoted("bbb", stagelabels.LaneReasonAdoptionStalledPrefix+"check-failing:unit") {
t.Fatalf("new head: %+v", rec)
}
h.clock.advance(adoptedDwell + time.Minute)
Expand Down Expand Up @@ -636,3 +639,202 @@ func TestAdoptedSignoffStallAskAgreesWithWorkflowFence(t *testing.T) {
})
}
}

// observeErr is observe for a pass whose error the test expects (a comment
// POST the fake API refuses): the pass still runs to the end.
func (h *adoptHarness) observeErr(pr *gh.PullRequest, reason string) (StageLabelResult, error) {
h.t.Helper()
h.adopt.mu.Lock()
h.adopt.heads[pr.GetNumber()] = pr.GetHead().GetSHA()
h.adopt.mu.Unlock()
h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, reason, false, nil)
h.e.noteStageRole("acme", "widget", pr.GetNumber(), stagelabels.RoleContributor)
return h.e.ApplyStageLabels(context.Background())
}

// projectbluefin/utah-packages#650 and #652: GitHub answered the
// adoption-stalled note with a 500, and the escalation had already marked
// the head noted, so the PR sat at 085 with no comment on that head ever.
// The note is recorded only once posted: the next sweep posts it, and the
// one after does not repeat it.
func TestAdoptionStalledNoteRetriedAfterFailedPost(t *testing.T) {
p := adoptPolicy()
p.AdoptInPlace = true
h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p)
defer done()
key := adoptInPlaceAt(h, "aaa", "check-failing:unit")
reason := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:unit"
parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"}

h.clock.advance(adoptedDwell + time.Minute)
h.api.failComments, h.api.failCommentsOn = 1, AdoptionStalledMarker
if _, err := h.observeErr(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit"); err == nil {
t.Fatal("the refused note was not reported")
}
rec, _ := h.e.stageStoreOrShared().Get(key)
if rec.Stage != stagelabels.NeedsDecision || !rec.AdoptionEscalated("aaa") || rec.AdoptionEscalationNoted("aaa", reason) || len(stalledComments(h.api)) != 0 {
t.Fatalf("failed post: %+v comments=%v", rec, h.api.comments)
}

h.api.reset()
h.clock.advance(5 * time.Minute)
res := h.observe(adoptPR(7, "aaa", parked...), "check-failing:unit")
if notes := stalledComments(h.api); len(notes) != 1 || !strings.Contains(notes[0], "reason="+reason) || res.Comments != 1 {
t.Fatalf("note not retried: res=%+v comments=%v", res, h.api.comments)
}
if len(h.api.added) != 0 || len(h.api.removed) != 0 {
t.Fatalf("retry churned labels: added=%v removed=%v", h.api.added, h.api.removed)
}
if rec, _ = h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision || !rec.AdoptionEscalationNoted("aaa", reason) {
t.Fatalf("retried note not recorded: %+v", rec)
}

h.api.reset()
h.clock.advance(5 * time.Minute)
h.observe(adoptPR(7, "aaa", parked...), "check-failing:unit")
if len(h.api.comments) != 0 || len(h.api.added) != 0 || len(h.api.removed) != 0 {
t.Fatalf("same head and reason repeated: comments=%v added=%v removed=%v", h.api.comments, h.api.added, h.api.removed)
}
}

// A different blocker on the same head is a different note: the maintainer
// who read the first must learn what holds the PR now. Each reason is
// explained once per head, so the blocker flapping back stays silent.
func TestAdoptionStalledNotePerReasonOnSameHead(t *testing.T) {
p := adoptPolicy()
p.AdoptInPlace = true
h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p)
defer done()
key := adoptInPlaceAt(h, "aaa", "check-failing:unit")
parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"}

h.clock.advance(adoptedDwell + time.Minute)
h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit")
if notes := stalledComments(h.api); len(notes) != 1 {
t.Fatalf("first note: %v", h.api.comments)
}

h.api.reset()
h.clock.advance(5 * time.Minute)
h.observe(adoptPR(7, "aaa", parked...), "check-failing:lint")
lint := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:lint"
if notes := stalledComments(h.api); len(notes) != 1 || !strings.Contains(notes[0], "reason="+lint) {
t.Fatalf("changed reason not noted: %v", h.api.comments)
}
rec, _ := h.e.stageStoreOrShared().Get(key)
if rec.Stage != stagelabels.NeedsDecision || rec.Reason != lint || rec.AdoptEscalatedReason != lint {
t.Fatalf("escalation kept the stale reason: %+v", rec)
}
if len(h.api.added) != 0 || len(h.api.removed) != 0 {
t.Fatalf("reason change churned labels: added=%v removed=%v", h.api.added, h.api.removed)
}

for _, lane := range []string{"check-failing:unit", "check-failing:lint", "check-failing:lint"} {
h.api.reset()
h.clock.advance(5 * time.Minute)
h.observe(adoptPR(7, "aaa", parked...), lane)
if len(h.api.comments) != 0 {
t.Fatalf("%s: reason already noted on this head posted again: %v", lane, h.api.comments)
}
// The record follows the blocker, so a later hold (a weak opinion)
// names the current one.
if rec, _ := h.e.stageStoreOrShared().Get(key); rec.AdoptEscalatedReason != stagelabels.LaneReasonAdoptionStalledPrefix+lane || rec.Reason != rec.AdoptEscalatedReason {
t.Fatalf("%s: escalation reason lags the blocker: %+v", lane, rec)
}
}
}

// A record from before the noted reasons were kept: escalated and noted on
// the head, no reasons. It reads as noted for every reason on that head, so
// an upgrade does not repost on every parked PR.
func TestAdoptionStalledLegacyNotedHeadNoRepost(t *testing.T) {
p := adoptPolicy()
p.AdoptInPlace = true
h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p)
defer done()
key := adoptInPlaceAt(h, "aaa", "check-failing:unit")
parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"}
h.clock.advance(adoptedDwell + time.Minute)
h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit")
if notes := stalledComments(h.api); len(notes) != 1 {
t.Fatalf("first note: %v", h.api.comments)
}

// Rewrite the record as the previous binary left it.
path := filepath.Join(t.TempDir(), "legacy-stage.json")
store := stagelabels.NewStore(path, h.clock.now)
src := h.e.stageStoreOrShared()
raw, err := json.Marshal(map[string]stagelabels.Record{key: func() stagelabels.Record { r, _ := src.Get(key); return r }()})
if err != nil {
t.Fatal(err)
}
var items map[string]map[string]any
if err := json.Unmarshal(raw, &items); err != nil {
t.Fatal(err)
}
if _, ok := items[key]["adopt_escalation_noted_reasons"]; !ok {
t.Fatalf("fixture: no noted reasons to strip: %s", raw)
}
delete(items[key], "adopt_escalation_noted_reasons")
raw, _ = json.Marshal(items)
if err := os.WriteFile(path, raw, 0o644); err != nil {
t.Fatal(err)
}
if err := store.Load(); err != nil {
t.Fatal(err)
}
if rec, _ := store.Get(key); rec.AdoptEscalationNotedHead != "aaa" || len(rec.AdoptEscalationNotedReasons) != 0 {
t.Fatalf("legacy fixture: %+v", rec)
}
h.e.stageStore = store

for _, lane := range []string{"check-failing:unit", "check-failing:lint"} {
h.api.reset()
h.clock.advance(5 * time.Minute)
h.observe(adoptPR(7, "aaa", parked...), lane)
if len(h.api.comments) != 0 {
t.Fatalf("%s: legacy noted head reposted: %v", lane, h.api.comments)
}
if rec, _ := store.Get(key); rec.Stage != stagelabels.NeedsDecision {
t.Fatalf("%s: legacy escalation dropped: %+v", lane, rec)
}
}
}

// projectbluefin/utah#919 flipped 085 β†’ 080 β†’ 085 within two minutes on
// one head: a lane that errored on an approved head reads it author-side
// done (080 at role rank), which is not a stage the adoption pin claims,
// so the pass dropped the active escalation for that sweep. A weak opinion
// does not release it; the labels (written from the engine's derivation)
// stay put.
func TestAdoptionEscalationHoldsAgainstErroredLane(t *testing.T) {
p := adoptPolicy()
p.AdoptInPlace = true
h, done := newAdoptHarness(t, ContributorNudgeOutcome{Mode: "comment"}, p)
defer done()
key := adoptInPlaceAt(h, "aaa", "check-failing:unit")
reason := stagelabels.LaneReasonAdoptionStalledPrefix + "check-failing:unit"
parked := []string{stagelabels.NeedsDecision.Label(), "role/contributor", "author/clanker"}
h.clock.advance(adoptedDwell + time.Minute)
h.observe(adoptPR(7, "aaa", adoptedLabels...), "check-failing:unit")
if rec, _ := h.e.stageStoreOrShared().Get(key); rec.Stage != stagelabels.NeedsDecision || len(stalledComments(h.api)) != 1 {
t.Fatalf("not escalated: %+v comments=%v", rec, h.api.comments)
}

h.api.reset()
h.clock.advance(2 * time.Minute)
pr := adoptPR(7, "aaa", parked...)
h.e.recordStage(stageLaneContributor, "acme/widget", "acme", "widget", pr, "", false, errors.New("lane tripped"))
h.e.noteStageRole("acme", "widget", 7, stagelabels.RoleContributor)
h.e.noteStageVerdict("acme", "widget", 7, ContributorLaneVerdict{Configured: true, Approved: true, Verdict: "approve"})
if _, err := h.e.ApplyStageLabels(context.Background()); err != nil {
t.Fatal(err)
}
rec, _ := h.e.stageStoreOrShared().Get(key)
if rec.Stage != stagelabels.NeedsDecision || rec.Reason != reason || !rec.AdoptionEscalated("aaa") {
t.Fatalf("errored lane dropped the escalation: %+v", rec)
}
if len(h.api.added) != 0 || len(h.api.removed) != 0 || len(h.api.comments) != 0 {
t.Fatalf("escalation churned: added=%v removed=%v comments=%v", h.api.added, h.api.removed, h.api.comments)
}
}
1 change: 1 addition & 0 deletions src/pkg/github/automerge/automerge_adoption_rebase_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ func TestAdoptedNeedsRebaseSupersedesInsteadOfStalling(t *testing.T) {
store := h.e.stageStoreOrShared()
reason := stagelabels.LaneReasonAdoptionStalledPrefix + "label:needs-rebase"
store.MarkAdoptionEscalated(key, "aaa", reason)
store.MarkAdoptionEscalationNoted(key, "aaa", reason)
store.Observe(key, stagelabels.NeedsDecision, reason)
labels[0] = stagelabels.NeedsDecision.Label()
}
Expand Down
Loading
Loading