diff --git a/changelog.d/fixed-merge-train-verdicts-shared.md b/changelog.d/fixed-merge-train-verdicts-shared.md new file mode 100644 index 0000000000..47e50fa5b6 --- /dev/null +++ b/changelog.d/fixed-merge-train-verdicts-shared.md @@ -0,0 +1 @@ +- automerge: a PR the lane leaves to a person (for example `contributor-lane:no-linked-issue`) now stays out of the merge train on every sweep, so its stage no longer flips between 080 and 090. The lane's per-head merge-train verdict lived on the engine that reached it. The contributor lane runs both in the long-lived loop and every minute on a fresh engine, and the fresh engine never saw the loop's verdict: it read the PR's 090 stage record as ready and seated the PR behind another train member at 080 (`merge-train:queued-behind#…`), and the next pass put it back at 090. projectbluefin `utah-packages#1222`, a github-actions bot PR with no linked issue, did this 51 times in 3.5 hours, each one suppressed by the stage dampener. The verdicts now live in the sweep memory shared by every lane engine over the same GitHub client, next to the once-per-head marks. A read-only integration-eligibility check keeps its own memory, as before. diff --git a/src/docs/dashboard-sections/activity-monitor.md b/src/docs/dashboard-sections/activity-monitor.md index eaeeebf440..e25a706286 100644 --- a/src/docs/dashboard-sections/activity-monitor.md +++ b/src/docs/dashboard-sections/activity-monitor.md @@ -336,7 +336,7 @@ The tooltip lists the PRs ahead and the overlapping files. Waiting siblings are **The header counts the rows under it.** A member drawn as a sub-row of its linked issue (`nested_under`) still sits in the train run — the issue row rides its open PR's train, in that PR's merge order, and wears the rail — so a train of three is three rows under one header whether or not its PRs have issues. When the page does not carry every member, the header says where the rest are and names them — `merge train · 1 of 3 PRs here · 1st: #15425 · on another page: #15529 · hidden by the filter: #15530` (`not in this payload: #N` for one the server no longer lists) — never a bare "not in this view" (owner: *what does 'not in this view' mean?*) and never `3 PRs` over one row (hive's v5#15425 did, with #15529 and #15530 drawn under their issues). `size` / `members` count only the PRs in the merge queue in this payload, never the planner's wider grouping (hive's own header once read `2 of 9 PRs here · not in this view: 7 more` where the seven were PRs still at CI / Review). The header's hover says the mechanics: the members touch the same files, so they merge one at a time in order, and each is rebased onto the base only after the PR ahead merges; a member's own hover adds its slot, *Position: 3rd of 4 · after #15085 merges*. No member on the page, no header. -**Who rides.** A PR the lane leaves to a person is not a train member and never its head: an author below the lane's trust (`untrusted-author-role`, `contributor-lane:no-linked-issue`, …), the title/label human-merge rule, a human-merge path, a merge GitHub refused for review, and every other `human_merge_reason` code except the catch-all `policy`. The lane's verdict is remembered per lane and head (`noteMergeTrainOutcome`); a new head starts clean. A head the lane last saw unable to merge on its own (red required check, terminal hold, permanent refusal) stays a member but never fronts. +**Who rides.** A PR the lane leaves to a person is not a train member and never its head: an author below the lane's trust (`untrusted-author-role`, `contributor-lane:no-linked-issue`, …), the title/label human-merge rule, a human-merge path, a merge GitHub refused for review, and every other `human_merge_reason` code except the catch-all `policy`. The lane's verdict is remembered per lane and head (`noteMergeTrainOutcome`) in the sweep memory every lane engine over the GitHub client shares, so the long-lived loop and the per-minute one-shot sweep plan the same train (projectbluefin utah-packages#1222 flipped between 080 and 090 while only one of them knew the verdict); a new head starts clean. A head the lane last saw unable to merge on its own (red required check, terminal hold, permanent refusal) stays a member but never fronts. **PRs under repair leave the train.** A PR somebody is fixing is not a train member either: the planner (`mergeTrainUnderRepair`) skips a PR at 070 fixing, 071 needs-author, 072 returned-to-contributor or 085 needs-decision (label or stage store), and the tracker (`stageTrackerTrainUnderRepair`) drops one whose stage says so (074 adopted-by-hive included) or whose `reason` is a red required check (`required-check-failing:` / `check-failing:` / `ci-red`) — hive's own #15405 once led a header as `merge train · 2 PRs · 1st: #15405` while its own chevron read *Fixing* (a maintainer had just requested changes). The dropped row carries `left_train: {why, note}` and no `merge_train`: no ordinal, no rail, and its hover ends with the one muted line *Merge train: left the merge train while being fixed; rejoins when green*. The survivors close ranks — `size`, `position`, `front`, `ahead`, `members` are recomputed, and the next in line fronts when the front itself left — and a train left with one member is no train: no header, no ordinal, the 080 row back to *Green and approved; waiting for the automerge lane to admit it*. The exception is the `rebase-exhausted` tail at 085: the train put it there, and it stays so the person who rebases it sees where it merges. Provisional members (050 / 060 with nothing red, [#15580](https://github.com/hivecommons/hive/issues/15580)) keep their slot. diff --git a/src/pkg/github/automerge/automerge_merge_train.go b/src/pkg/github/automerge/automerge_merge_train.go index 6361f1a430..5a2c6af8b2 100644 --- a/src/pkg/github/automerge/automerge_merge_train.go +++ b/src/pkg/github/automerge/automerge_merge_train.go @@ -1011,9 +1011,10 @@ type mergeTrainState struct { mergeTrainAttemptCount map[string]int mergeTrainMerged map[string]int mergeTrainLogged map[string]string - // mergeTrainVerdicts remembers, per lane and PR, the last head whose - // verdict keeps it from fronting (or riding) a train. - mergeTrainVerdicts map[string]mergeTrainVerdict + // The lanes' verdicts per head live in the sweep memory + // (sweepMemory.mergeTrainVerdicts), shared by every engine over the + // same client: a train planned on the per-minute one-shot engine must + // see the verdict the long-lived loop reached, and the reverse. // mergeTrainSeats remembers, per lane and PR, the head that holds a // train seat and since when (hivecommons/hive#15825): the seat orders // the train (earliest first) and its loss is the ejection the log and @@ -1156,7 +1157,7 @@ func (c *Engine) mergeTrainEjectWhyLocked(lane, owner, repo string, number int, case mergeTrainDirty(pr.GetMergeableState()): return mergeTrainEjectConflicting } - if v, ok := c.mergeTrainVerdicts[mergeTrainVerdictKey(lane, owner, repo, number)]; ok && v.sha == sha { + if v, ok := c.mergeTrainVerdictFor(lane, owner, repo, number, sha); ok { switch { case v.humanMerge: return mergeTrainEjectHumanMerge @@ -1260,25 +1261,31 @@ func (c *Engine) noteMergeTrainOutcome(lane, owner, repo string, number int, sha } key := mergeTrainVerdictKey(lane, owner, repo, number) v := mergeTrainVerdict{sha: sha, blocked: mergeTrainBlockingReason(reason), humanMerge: mergeTrainHumanMergeReason(reason), readiness: mergeTrainReadinessOf(reason), reason: reason} - c.mergeTrainMu.Lock() - defer c.mergeTrainMu.Unlock() + m := c.memory() + m.mergeTrainVerdictsMu.Lock() + defer m.mergeTrainVerdictsMu.Unlock() if (!v.humanMerge && !v.blocked && v.readiness == mergeTrainReadinessUnknown) || strings.TrimSpace(sha) == "" { - delete(c.mergeTrainVerdicts, key) + delete(m.mergeTrainVerdicts, key) return } - if c.mergeTrainVerdicts == nil { - c.mergeTrainVerdicts = map[string]mergeTrainVerdict{} + if m.mergeTrainVerdicts == nil { + m.mergeTrainVerdicts = map[string]mergeTrainVerdict{} } - c.mergeTrainVerdicts[key] = v + m.mergeTrainVerdicts[key] = v } +// mergeTrainVerdictFor is the lane's remembered verdict on this exact +// head, from the sweep memory every engine over the client shares. It +// takes only the memory's leaf lock, so a caller holding mergeTrainMu +// (mergeTrainEjectWhyLocked) may use it. func (c *Engine) mergeTrainVerdictFor(lane, owner, repo string, number int, sha string) (mergeTrainVerdict, bool) { if c == nil || sha == "" { return mergeTrainVerdict{}, false } - c.mergeTrainMu.Lock() - defer c.mergeTrainMu.Unlock() - v, ok := c.mergeTrainVerdicts[mergeTrainVerdictKey(lane, owner, repo, number)] + m := c.memory() + m.mergeTrainVerdictsMu.Lock() + defer m.mergeTrainVerdictsMu.Unlock() + v, ok := m.mergeTrainVerdicts[mergeTrainVerdictKey(lane, owner, repo, number)] return v, ok && v.sha == sha } diff --git a/src/pkg/github/automerge/automerge_sweep_memory.go b/src/pkg/github/automerge/automerge_sweep_memory.go index 1fde33c776..3fbf9355e5 100644 --- a/src/pkg/github/automerge/automerge_sweep_memory.go +++ b/src/pkg/github/automerge/automerge_sweep_memory.go @@ -8,7 +8,8 @@ import ( // sweepMemory is what the lanes remember across sweeps about a PR head: // the merge-forward tried on it, an update-branch or merge-forward GitHub // refused for lack of the workflows permission, a merge GitHub refused -// (automerge_merge_refusal.go), and the holds already logged. +// (automerge_merge_refusal.go), the holds already logged, and each lane's +// last verdict on the head for the merge train (noteMergeTrainOutcome). // // It is shared by every engine built over the same GitHub client, not kept // on one engine. The hive runs the queued, trusted-author and contributor @@ -17,7 +18,11 @@ import ( // the long-lived self-authored loop, so a mark on the engine was empty // again on the next tick: projectbluefin fsdk-containers#508 sent the same // refused update-branch to GitHub every ~75 s, and the "logged once per -// head" holds were logged every sweep. +// head" holds were logged every sweep. The merge-train verdicts had the +// same gap: the one-shot contributor lane never saw the long-lived loop's +// `contributor-lane:no-linked-issue` on projectbluefin utah-packages#1222, +// so it seated the PR in a train (080, queued-behind) on the passes the +// other one parked it at 090, and the stage dampener absorbed the flip. // // An engine over a transport that does not share state (the sweep's test // fakes, a nil transport) keeps its own memory, as before. A read-only @@ -31,6 +36,13 @@ type sweepMemory struct { // mergeRefusalsMu guards mergeRefusals (automerge_merge_refusal.go). mergeRefusalsMu sync.Mutex mergeRefusals map[string]mergeRefusal + // mergeTrainVerdictsMu guards mergeTrainVerdicts + // (automerge_merge_train.go). It is a leaf lock: it may be taken while + // an engine holds its mergeTrainMu, never the other way round. + mergeTrainVerdictsMu sync.Mutex + // mergeTrainVerdicts remembers, per lane and PR, the last head whose + // verdict keeps it from fronting (or riding) a train. + mergeTrainVerdicts map[string]mergeTrainVerdict } // sweepMemoryStateName is the name the memory is kept under on the client diff --git a/src/pkg/github/automerge/automerge_sweep_memory_test.go b/src/pkg/github/automerge/automerge_sweep_memory_test.go index f9565051c3..20e05ee582 100644 --- a/src/pkg/github/automerge/automerge_sweep_memory_test.go +++ b/src/pkg/github/automerge/automerge_sweep_memory_test.go @@ -21,6 +21,7 @@ import ( gh "github.com/google/go-github/v72/github" hgithub "github.com/hivecommons/hive/pkg/github" + "github.com/hivecommons/hive/pkg/stageengine/prstage" "github.com/hivecommons/hive/pkg/stagelabels" ) @@ -227,6 +228,57 @@ func TestMergeTrainSkipLoggedOnceAcrossSweeps(t *testing.T) { } } +// A lane's human-merge verdict on a head keeps the PR out of the train on +// every engine over the client, not only on the one that reached it. +// projectbluefin utah-packages#1222 (a github-actions bot PR, no linked +// issue) sat at 090 after the long-lived loop's contributor lane said +// `contributor-lane:no-linked-issue`; the per-minute one-shot engine had +// no verdict, read the 090 stage record as ready, and parked the PR as +// `merge-train:queued-behind#1217` at 080. The dampener suppressed the +// 080/090 flip 51 times in 3.5 h. +func TestMergeTrainHumanMergeVerdictSharedAcrossSweeps(t *testing.T) { + api := &mergeTrainAPI{ + files: map[int][]string{1217: {"config/upstream-sources.json"}, 1222: {"config/upstream-sources.json"}}, + state: map[int]string{1217: "clean", 1222: "clean"}, + } + srv := newMergeTrainServer(t, api) + defer srv.Close() + store := stagelabels.NewStore("", nil) + tick := sweepTickEngines(srv.URL, Options{ + StageStore: store, + MergeTrain: func() MergeTrainPolicy { return DefaultMergeTrainPolicy }, + }) + t0 := time.Date(2026, 11, 19, 13, 30, 0, 0, time.UTC) + front, bot := mtPR(1217, t0, "clean"), mtPR(1222, t0.Add(time.Hour), "clean") + prs := []*gh.PullRequest{front, bot} + plan := func(e *Engine) *mergeTrainPlan { + return e.planMergeTrains(context.Background(), stageLaneContributor, "acme/widget", "acme", "widget", prs, func(*gh.PullRequest) string { return "" }) + } + + // The long-lived loop's contributor lane: #1217 ready, #1222 a + // person's merge, recorded at 090. + loop := tick() + mtReady(loop, stageLaneContributor, front) + loop.noteMergeTrainOutcome(stageLaneContributor, "acme", "widget", 1222, bot.GetHead().GetSHA(), prstage.ContributorLaneReasonNoLinkedIssue) + store.Observe(stagelabels.Key("acme/widget", 1222), stagelabels.ReadyForHumanMerge, prstage.ContributorLaneReasonNoLinkedIssue) + + for i := 1; i <= 3; i++ { + p := plan(tick()) + if p.isFront(1222) || p.isSibling(1222) || p.isCandidate(1222) { + t.Fatalf("one-shot sweep %d put the human-merge PR in a train: front=%v sibling=%v candidate=%v", i, p.isFront(1222), p.isSibling(1222), p.isCandidate(1222)) + } + } + if p := plan(loop); p.isSibling(1222) || p.isFront(1222) { + t.Fatal("the long-lived engine put the human-merge PR in a train") + } + + // A new head starts clean: the old head's verdict does not hold it out. + bot.Head.SHA = gh.Ptr("sha1222-2") + if got := tick().mergeTrainHeadHumanMerge(stageLaneContributor, "acme", "widget", 1222, bot.GetHead().GetSHA()); got { + t.Fatal("a verdict on the old head held the new head out of the train") + } +} + // Engines over one client share the memory; a read-only integration probe // and an engine over a transport that keeps no state each get their own. func TestSweepMemorySharedPerClient(t *testing.T) { @@ -242,6 +294,13 @@ func TestSweepMemorySharedPerClient(t *testing.T) { if !first.markMergeForwardOnce("k") || second.markMergeForwardOnce("k") || !probe.markMergeForwardOnce("k") { t.Fatal("once marks do not follow the shared memory") } + first.noteMergeTrainOutcome(stageLaneContributor, "acme", "widget", 7, "sha7", prstage.ContributorLaneReasonNoLinkedIssue) + if !second.mergeTrainHeadHumanMerge(stageLaneContributor, "acme", "widget", 7, "sha7") { + t.Fatal("a merge-train verdict does not follow the shared memory") + } + if probe.mergeTrainHeadHumanMerge(stageLaneContributor, "acme", "widget", 7, "sha7") { + t.Fatal("the read-only integration probe reads the sweeps' merge-train verdicts") + } other := sweepTickEngines("http://127.0.0.1:0", Options{})() if other.memory() == first.memory() { t.Fatal("engines over different clients share a memory")