diff --git a/changelog.d/fixed-14934-held-lane-front.md b/changelog.d/fixed-14934-held-lane-front.md new file mode 100644 index 0000000000..935625de29 --- /dev/null +++ b/changelog.d/fixed-14934-held-lane-front.md @@ -0,0 +1 @@ +- Serialized merge lanes now release a held front PR on the next queued or self-authored sweep and promote the next waiting PR, instead of waiting for the front timeout ([#14934](https://github.com/hivecommons/hive/issues/14934)). diff --git a/src/docs/labels-and-control-signals.md b/src/docs/labels-and-control-signals.md index ea7acbc4e2..12527ec155 100644 --- a/src/docs/labels-and-control-signals.md +++ b/src/docs/labels-and-control-signals.md @@ -8,10 +8,8 @@ Most hard gates run before an agent sees work. GitHub issue enumeration reads la Some sweeps bypass that enumeration and list items themselves, but every one of them gates on the same hold predicate as enumeration (`Client.IsHeldLabels` / `Client.isHeld`, `src/pkg/github/client.go:1969-1977`, `src/pkg/github/client.go:2679-2681`), so the generic hold substrings and the exact dashboard `hive-pause/` label hold everywhere: -- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; both the queued (`lgtm`) and the self-authored sweep skip a held PR before fetching it (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`). -- The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:797-815`, `src/cmd/hive/claimescalation.go`). -- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam; the queued (`lgtm`) sweep skips a held PR before fetching it, and the self-authored sweep skips held PRs plus `needs-human` before merging (`src/pkg/github/automerge/automerge_sweep.go:23-38`, `src/pkg/github/automerge/automerge_sweep.go:980-985`, `src/pkg/github/automerge/automerge_sweep.go:987-1025`, `src/pkg/github/automerge/automerge_sweep.go:1050-1067`). -- The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:848-854`). +- Auto-merge sweeps reach it through the `Transport.IsHeldLabels` seam and skip held PRs; the self-authored sweep also skips `needs-human` before merging. On `hive-serialized` repos, a rejected PR is still handed to the lane with `FrontOnly: true`; the lane's own label policy (`defaultBlocked` in `src/pkg/mergelane/lane.go`: the hold family, `hive-pause/`, `do-not-merge*`) then lets a held front leave so the next waiting PR is promoted. A front rejected for another label (for example PR `needs-human` or `needs-rebase`) is not refused by that policy and still waits for the front timeout; other held PRs are never admitted. The queued sweep fetches the base branch only for serialized repos; the self-authored sweep reuses its list response (`rejectPrefilteredBeforeLane` and `rejectBeforeLane` in `src/pkg/github/automerge/automerge_sweep.go`). +- The task-list sweep skips a held issue before its exempt / issue `needs-human` gate, so a held issue is never commented on, relabelled, or closed by it (`src/pkg/github/task_list_sweep.go:893-900`). - The SHA-hold sweep lists primary-repo `kind/bug` issues itself, adds literal `hold` plus a marker notice when no SHA is present (for reporters in `issue_filter.reporter_trust` trusted associations it adds the soft `issues.sha_hold_trusted_label`, default `needs-sha`, instead, #12002), and removes `hold` only when the current hold is its own: the newest `hold` label event is a `labeled` by the App bot and an App-authored SHA-hold notice was posted at or after it (`EnforceSHAHold` / `shaHoldIsOwn` in `src/pkg/github/client.go`). - PR-request watching applies server-side PR holds after a PR is opened; the agent's `--label hold` flag is deliberately discarded by the wrapper (`src/pkg/github/pr_request_watcher.go:452-524`, `bin/hive-open-pr.sh:142-150`). diff --git a/src/docs/running-at-level-6.md b/src/docs/running-at-level-6.md index d9f1b375a3..f43aaffd9d 100644 --- a/src/docs/running-at-level-6.md +++ b/src/docs/running-at-level-6.md @@ -250,7 +250,7 @@ Verify a routine claim/review no longer floods unrelated maintainers, while a te First canary choice: in image repositories every lane merge to `main` publishes an image, so pick the first canary repo with that cost in mind. -Front timeout and stuck lanes (`v6`): a front PR whose required checks never go green (missing context, path-filtered workflow, or red) holds the lane until the front timeout (`merge_front_timeout`, 60 minutes by default; see [Front timeout](operator-reference.md#front-timeout-merge_front_timeout)) runs out (a front whose required checks are still queued or in progress is extended by one more timeout, at most three times; that limit is fixed, not configurable); the lane panel shows it at the front and later PRs show `deferred: not at the front of the lane`. A held or red front PR can block the lane, and the current red-front behaviour is tracked in [#11023](https://github.com/hivecommons/hive/issues/11023). +Front timeout and stuck lanes (`v6`): a front PR whose required checks never go green (missing context, path-filtered workflow, or red) holds the lane until the front timeout (`merge_front_timeout`, 60 minutes by default; see [Front timeout](operator-reference.md#front-timeout-merge_front_timeout)) runs out (a front whose required checks are still queued or in progress is extended by one more timeout, at most three times; that limit is fixed, not configurable); the lane panel shows it at the front and later PRs show `deferred: not at the front of the lane`. A red front PR holds the lane until that timeout. A held front PR does not: the next queued or self-authored sweep hands it to the lane front-only, the lane's own label policy refuses it, and it leaves so the next waiting PR is promoted ([#14934](https://github.com/hivecommons/hive/issues/14934)). That policy refuses only the hold family (`hold`, `on-hold`, `hold/review`, any label containing `hold`), the dashboard's `hive-pause/` label and `do-not-merge*` (`defaultBlocked` in [mergelane](../pkg/mergelane/lane.go)). A green front carrying only `needs-human` or `needs-rebase` is not refused by that policy; the front-only hand-off supplies no merge authorization, so it stays at the front and still waits for the front timeout. Add `hold` if such a PR must leave the lane now (see [label reference](labels-and-control-signals.md)). **R3. Confirm intent after every switch, restart and upgrade.** On the dashboard, check three things: the ACMM level badge in the top bar shows the level you intended; each agent's card shows the mode, cadence and paused state you expect; and each repository card shows the Auto-merge and paused state you set. Then open **Audit Log** and confirm that merges are appearing, or have stopped, as you intended. If you keep settings in the config file, compare it with what the dashboard shows. Use a harmless held test PR to confirm it stays held, then a qualified canary to confirm the intended path works. Without read-back, a saved level alone can hide reconciliation or persistence failure. Hive reports state and errors; this verification discipline is **JUDGMENT CALL**. Sources: [pack API](../pkg/dashboard/api_packs.go), [sweep startup](../pkg/github/automerge/automerge_sweep.go), [config layering](config-layering.md). diff --git a/src/pkg/github/automerge/automerge_lane_test.go b/src/pkg/github/automerge/automerge_lane_test.go index eb158601d2..72761c79fc 100644 --- a/src/pkg/github/automerge/automerge_lane_test.go +++ b/src/pkg/github/automerge/automerge_lane_test.go @@ -3,6 +3,8 @@ package automerge import ( "context" "errors" + "net/http" + "net/http/httptest" "strings" "testing" "time" @@ -167,7 +169,7 @@ func TestLaneSweep_RefusalAndErrorAreSkips(t *testing.T) { } } -// AC4: auto-merge off or a held PR never reaches the lane. +// Auto-merge off never consults the lane. func TestLaneSweep_GatesAheadOfTheLane(t *testing.T) { var merged []int held := greenLanePR(7) @@ -180,10 +182,6 @@ func TestLaneSweep_GatesAheadOfTheLane(t *testing.T) { if result, err := off.SweepQueuedAutoMerges(context.Background(), AutoMergeSweepOptions{}); err != nil || len(result.Merged) != 0 { t.Fatalf("auto-merge off: result = %+v, err = %v", result, err) } - on := newLaneSweepEngine(api.URL, config.MergeStrategyHiveSerialized, g.gate, Options{}) - if result, err := on.SweepQueuedAutoMerges(context.Background(), AutoMergeSweepOptions{}); err != nil || len(result.Merged) != 0 { - t.Fatalf("held PR: result = %+v, err = %v", result, err) - } if len(g.reqs) != 0 || len(merged) != 0 { t.Fatalf("lane requests = %d, merges = %v; want none", len(g.reqs), merged) } @@ -481,3 +479,76 @@ func TestLaneSweep_ContributorLanePrefilterRejectsFrontOnly(t *testing.T) { t.Fatalf("lane request = %+v", g.reqs[0]) } } + +// #14934: list-response label rejections must still release a serialized +// front. FrontOnly also ensures a held non-front PR cannot join waiting. +func TestLaneSweep_HeldQueuedFrontLeavesTheLane(t *testing.T) { + for _, label := range []string{"hold", "needs-human", "needs-rebase", "do-not-merge/review"} { + for _, self := range []bool{false, true} { + for _, strategy := range []string{config.MergeStrategyHiveSerialized, config.MergeStrategyDirect} { + for _, outcome := range []string{hgithub.LaneOutcomeLeft, hgithub.LaneOutcomeDeferred} { + name := strategy + "/" + outcome + "/queued" + if self { + name = strategy + "/" + outcome + "/self-authored" + } + t.Run(label+"/"+name, func(t *testing.T) { + var merged []int + var api *httptest.Server + if self { + pr := greenSelfPR(7) + pr.extraLabels = []string{label} + api = newSelfAuthoredAutoMergeAPI(t, []selfAuthoredPR{pr}, &merged) + } else { + pr := greenLanePR(7) + pr.extraLabels = []string{label} + pr.baseRef = "main" + api = newAutoMergeSweepAPI(t, hgithub.AutoMergeQueuedLabel, []sweepPR{pr}, &merged) + } + defer api.Close() + gets := 0 + counted := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet && r.URL.Path == "/repos/acme/widget/pulls/7" { + gets++ + } + api.Config.Handler.ServeHTTP(w, r) + })) + defer counted.Close() + g := &laneGate{answer: laneAnswer(outcome, "a hold label is present")} + c := newLaneSweepEngine(counted.URL, strategy, g.gate, Options{}) + var result *AutoMergeSweepResult + var err error + if self { + result, err = c.SweepSelfAuthoredAutoMerges(context.Background(), AutoMergeSweepOptions{}) + } else { + result, err = c.SweepQueuedAutoMerges(context.Background(), AutoMergeSweepOptions{}) + } + if err != nil || result.Seen != 1 || result.Skipped != 1 || len(result.Merged) != 0 || len(merged) != 0 { + t.Fatalf("result = %+v, err = %v, merges = %v; want one skip", result, err, merged) + } + wantGets := 0 + if strategy == config.MergeStrategyHiveSerialized { + if !self { + wantGets = 1 + } + if len(g.reqs) != 1 { + t.Fatalf("lane requests = %+v; want one", g.reqs) + } + path := hgithub.PRAuditPathQueue + if self { + path = hgithub.PRAuditPathSweep + } + if req := g.reqs[0]; !req.FrontOnly || req.Authorize != nil || req.Repo != "acme/widget" || req.Branch != "main" || req.Number != 7 || req.Path != path { + t.Fatalf("lane request = %+v; want front-only without authorization", req) + } + } else if len(g.reqs) != 0 { + t.Fatalf("direct lane requests = %+v; want none", g.reqs) + } + if gets != wantGets { + t.Fatalf("PR gets = %d; want %d", gets, wantGets) + } + }) + } + } + } + } +} diff --git a/src/pkg/github/automerge/automerge_sweep.go b/src/pkg/github/automerge/automerge_sweep.go index 99e7a1654a..856e230ea3 100644 --- a/src/pkg/github/automerge/automerge_sweep.go +++ b/src/pkg/github/automerge/automerge_sweep.go @@ -972,7 +972,10 @@ func (c *Engine) SweepQueuedAutoMerges(ctx context.Context, opts AutoMergeSweepO } result.Seen++ if reason := c.prefilterQueuedIssue(issue, label); reason != "" { - if reason == "held" || reason == "exempt-label" { + c.rejectPrefilteredBeforeLane(ctx, repo, owner, repoName, issue.GetNumber(), nil, hgithub.PRAuditPathQueue, reason) + // prefilterQueuedIssue reports holds as labelBlockReason's + // "label:hold", never the bare "held" used by the PR path. + if reason == "held" || reason == "label:hold" || reason == "exempt-label" { c.debug("automerge sweep skipped PR", "repo", repo, "pr", issue.GetNumber(), "reason", reason) } else { c.info("automerge sweep skipped PR", "repo", repo, "pr", issue.GetNumber(), "reason", reason) @@ -1146,6 +1149,9 @@ func (c *Engine) SweepSelfAuthoredAutoMerges(ctx context.Context, opts AutoMerge continue } else if reason != "" { if reason == "hold" { + // Same reason string as labelBlockReason so the lane + // rejection log reads alike on every hold path. + c.rejectPrefilteredBeforeLane(ctx, repo, owner, repoName, number, pr, hgithub.PRAuditPathSweep, "label:hold") reason = "held" } result.Skipped++ @@ -1168,6 +1174,7 @@ func (c *Engine) SweepSelfAuthoredAutoMerges(ctx context.Context, opts AutoMerge continue } if reason := c.prefilterSelfAuthoredPR(pr); reason != "" { + c.rejectPrefilteredBeforeLane(ctx, repo, owner, repoName, number, pr, hgithub.PRAuditPathSweep, reason) c.info("automerge skip", "repo", repo, "pr", number, "reason", reason) result.Skipped++ repoSkipped++ @@ -1986,6 +1993,28 @@ func (c *Engine) serializedLane(repo string) (bool, hgithub.SerializedLaneGate) return hgithub.SerializedStrategy(strategy, repo), gate } +// rejectPrefilteredBeforeLane keeps cheap sweep rejections cheap for direct +// repos, while letting a serialized front leave immediately. Queued issue list +// entries lack the base branch, so only that path needs an extra PR fetch: +// on a hive-serialized repo every prefiltered queued PR (label-removed, +// exempt-label, sentinel or label block) costs one PullRequests.Get per sweep, +// because the sweep cannot tell from the issue entry whether it is the front. +func (c *Engine) rejectPrefilteredBeforeLane(ctx context.Context, displayRepo, owner, repo string, number int, pr *gh.PullRequest, path, reason string) { + serialized, gate := c.serializedLane(displayRepo) + if !serialized { + return + } + if pr == nil { + var err error + pr, _, err = c.gh.PullRequests.Get(hgithub.WithRESTCaller(ctx, "hive:automerge_sweep"), owner, repo, number) + if err != nil { + c.warn("automerge sweep: could not fetch a prefiltered PR for the serialized lane", "repo", owner+"/"+repo, "pr", number, "reason", reason, "error", err) + return + } + } + c.rejectBeforeLane(ctx, serialized, gate, owner, repo, number, pr, path, reason) +} + // rejectBeforeLane returns a sweep path's own rejection of pr. In a // hive-serialized repo the PR is first handed to the lane front-only // (#11023): when it is the current front, the lane re-evaluates it and lets diff --git a/src/pkg/github/automerge/automerge_sweep_test.go b/src/pkg/github/automerge/automerge_sweep_test.go index 4bf7aceea0..bee0d425b0 100644 --- a/src/pkg/github/automerge/automerge_sweep_test.go +++ b/src/pkg/github/automerge/automerge_sweep_test.go @@ -36,6 +36,7 @@ type sweepPR struct { checkStatus string checkConclusion string headRepo string + baseRef string } func issueLabels(primary string, extra []string) []map[string]string { @@ -1822,6 +1823,7 @@ func newSelfAuthoredAutoMergeAPI(t *testing.T, prs []selfAuthoredPR, merged *[]i "draft": pr.draft, "user": map[string]string{"login": pr.author}, "head": map[string]string{"sha": "sha" + strconv.Itoa(pr.number)}, + "base": map[string]string{"ref": "main"}, "labels": issueLabels("", pr.extraLabels), }) } @@ -2968,6 +2970,13 @@ func newAutoMergeSweepAPI(t *testing.T, expectedLabel string, prs []sweepPR, mer if headRepo == "" { headRepo = "acme/widget" } + // The base ref is omitted unless a test opts in, so existing + // callers keep the empty-base-branch path and need no + // protection or base-status stubs. + base := map[string]any{"repo": map[string]string{"full_name": "acme/widget"}} + if pr.baseRef != "" { + base["ref"] = pr.baseRef + } json.NewEncoder(w).Encode(map[string]any{ "number": pr.number, "state": "open", @@ -2976,7 +2985,7 @@ func newAutoMergeSweepAPI(t *testing.T, expectedLabel string, prs []sweepPR, mer "mergeable": pr.mergeableState == "clean", "user": map[string]string{"login": pr.author}, "head": map[string]any{"sha": headSHA, "repo": map[string]string{"full_name": headRepo}}, - "base": map[string]any{"repo": map[string]string{"full_name": "acme/widget"}}, + "base": base, "labels": prLabels, }) case r.Method == http.MethodGet && strings.HasPrefix(r.URL.Path, "/repos/acme/widget/commits/") && strings.HasSuffix(r.URL.Path, "/status") && hasShaNumber(r.URL.Path):