Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-15682-hotspot-issue-cap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- The merge-hotspot advisor now caps the modularize issues it keeps open per repo (`governor.hotspot_max_open`, default 2; issues with a PR in flight still count) and files at most `governor.hotspot_max_per_pass` (default 1) per pass, filing the next-ranked hotspot only when a slot frees instead of opening five refactor issues in one pass ([#15682](https://github.com/hivecommons/hive/issues/15682)).
1 change: 1 addition & 0 deletions src/cmd/hive/hotspot_advisor.go
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ func runHotspotAdvisor(ctx context.Context, cfg *config.Config, ghClient *github
filer := &hotspot.Filer{
Store: store, Gate: ghAdapter, Creator: ghAdapter, Stater: ghAdapter,
Opts: opts, Refile: refile,
MaxOpen: g.HotspotMaxOpenOrDefault(), MaxPerPass: g.HotspotMaxPerPassOrDefault(),
Logf: func(format string, args ...any) { logger.Info(fmt.Sprintf(format, args...)) },
}
for _, repo := range store.Repos() {
Expand Down
4 changes: 3 additions & 1 deletion src/docs/operator-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -1439,7 +1439,7 @@ The hive already predicts PR file overlap for the merge train. The advisor reuse
- A file is a **hotspot** when, over the rolling window, at least `hotspot_min_prs` distinct PRs touched it and at least `hotspot_min_overlaps` pairs of them were open at the same time. Score = concurrent pairs × a size weight (1 / 1.25 / 1.5 / 2 at 800 / 1500 / 4000 lines) + 2 × attributed conflicts. Hunk-level "same region" weighting is not available from the train's records and is skipped.
- **Don't splinter.** A split is only recommended for code when the file has at least `hotspot_min_lines` lines **and** two or more top-level symbol groups (Go: funcs and types via `go/ast`, methods grouped under their receiver; JS/TS/Python: top-level declarations; Markdown: `##` headings). The proposal targets at most 4 files of at least 150 lines, no behaviour change, exported API unchanged, tests move with the code.
- **By kind.** Docs (`*.md`, `*.rst`, `*.txt`) are reported but only recommended for a by-section split above 1500 lines (the `docs/api-reference/<area>.md` pattern). Generated files (`// Code generated … DO NOT EDIT`, `*.pb.go`, the dashboard SPA under `pkg/dashboard/static/` → `static-src/`) point at their source — unless that source tree is already split into more than 4 files, in which case they are report-only (regeneration makes the output touch every source change). Lock and data files are report-only.
- **One issue per file.** Title `refactor: <path> is a merge hotspot — split into <N> units`, labels `agent/ci-maintainer` + `kind/cleanup`, origin `governor`, body marker `<!-- hive:hotspot file=<path> -->`, evidence table, the top-level units, a concrete split proposal and the don't-splinter rule. While that issue is open no second one is filed; after it closes the file waits `hotspot_refile_days` before it can be filed again. Issues go through the writeback gate as `issue_file` (a paused repo or `writeback.per_repo[<repo>].mode: off` stops them) and are skipped when the GitHub budget governor is skipping optional sweeps.
- **One issue per file.** Title `refactor: <path> is a merge hotspot — split into <N> units`, labels `agent/ci-maintainer` + `kind/cleanup`, origin `governor`, body marker `<!-- hive:hotspot file=<path> -->`, evidence table, the top-level units, a concrete split proposal and the don't-splinter rule. While that issue is open no second one is filed; after it closes the file waits `hotspot_refile_days` before it can be filed again. **Capped.** At most `hotspot_max_open` modularize issues are open per repo at once (an issue that already has a PR still counts) and at most `hotspot_max_per_pass` are filed per advisor pass; the highest-ranked uncapped hotspot is filed when a slot frees, and the advisor logs one INFO line per pass when the cap holds it back. Issues go through the writeback gate as `issue_file` (a paused repo or `writeback.per_repo[<repo>].mode: off` stops them) and are skipped when the GitHub budget governor is skipping optional sweeps.
- **Where to look.** `GET /api/hotspots` (or `hivectl governor hotspots [--repo owner/name]`) lists every hotspot with its recommendation and issue; `/api/status` carries `hotspots: {count, actionable, filed}`. State lives in `/data/hotspots.json`; the analysis runs at most hourly in the eval cycle, the intake on every merge-train sweep. A PR the train has not seen for two hours has its interval closed at its last sighting.

| Key | Default | Meaning |
Expand All @@ -1450,6 +1450,8 @@ The hive already predicts PR file overlap for the merge train. The advisor reuse
| `governor.hotspot_min_overlaps` | `3` | Concurrent PR pairs on the file before it is reported. |
| `governor.hotspot_min_lines` | `800` | Smallest code file a split is recommended for. |
| `governor.hotspot_refile_days` | `30` | Wait after a modularize issue closes before the same file may be filed again. |
| `governor.hotspot_max_open` | `2` | Most modularize issues open at once per repo; no new one is filed until one closes. |
| `governor.hotspot_max_per_pass` | `1` | Most modularize issues filed in one advisor pass. |

## Delivery SLA

Expand Down
17 changes: 17 additions & 0 deletions src/pkg/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -1719,6 +1719,11 @@ type GovernorConfig struct {
HotspotMinOverlaps int `yaml:"hotspot_min_overlaps,omitempty" json:"hotspot_min_overlaps,omitempty"`
HotspotMinLines int `yaml:"hotspot_min_lines,omitempty" json:"hotspot_min_lines,omitempty"`
HotspotRefileDays int `yaml:"hotspot_refile_days,omitempty" json:"hotspot_refile_days,omitempty"`
// HotspotMaxOpen caps the modularize issues open at once per repo; the
// advisor files the next-ranked hotspot only when one closes.
// HotspotMaxPerPass caps new issues per advisor pass.
HotspotMaxOpen int `yaml:"hotspot_max_open,omitempty" json:"hotspot_max_open,omitempty"`
HotspotMaxPerPass int `yaml:"hotspot_max_per_pass,omitempty" json:"hotspot_max_per_pass,omitempty"`

// ProjectObservability configures what the telemetry and operations agents
// recommend for the MANAGED PROJECT. It is deliberately separate from
Expand Down Expand Up @@ -2226,6 +2231,8 @@ const (
DefaultHotspotMinOverlaps = 3
DefaultHotspotMinLines = 800
DefaultHotspotRefileDays = 30
DefaultHotspotMaxOpen = 2
DefaultHotspotMaxPerPass = 1
)

// HotspotIssuesEnabled resolves GovernorConfig.HotspotEnabled (default on).
Expand Down Expand Up @@ -2265,6 +2272,16 @@ func (g GovernorConfig) HotspotRefileDaysOrDefault() int {
return hotspotOrDefault(g.HotspotRefileDays, DefaultHotspotRefileDays)
}

// HotspotMaxOpenOrDefault resolves governor.hotspot_max_open.
func (g GovernorConfig) HotspotMaxOpenOrDefault() int {
return hotspotOrDefault(g.HotspotMaxOpen, DefaultHotspotMaxOpen)
}

// HotspotMaxPerPassOrDefault resolves governor.hotspot_max_per_pass.
func (g GovernorConfig) HotspotMaxPerPassOrDefault() int {
return hotspotOrDefault(g.HotspotMaxPerPass, DefaultHotspotMaxPerPass)
}

// Bounds for AdvisoryConfig.UpdateIntervalS. Exported because the dashboard
// PUT validates against them and pkg/hub pins the invariant that its
// advisory-staleness threshold exceeds the maximum allowed posting cadence
Expand Down
32 changes: 32 additions & 0 deletions src/pkg/config/governor_hotspot_caps_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package config

import "testing"

// TestGovernorHotspotCapsOrDefault pins the resolution of the merge-hotspot
// issue caps (governor.hotspot_max_open / governor.hotspot_max_per_pass):
// zero and negative values fall back to the defaults, positive values win.
func TestGovernorHotspotCapsOrDefault(t *testing.T) {
var zero GovernorConfig
if got := zero.HotspotMaxOpenOrDefault(); got != DefaultHotspotMaxOpen {
t.Fatalf("HotspotMaxOpenOrDefault() zero = %d, want %d", got, DefaultHotspotMaxOpen)
}
if got := zero.HotspotMaxPerPassOrDefault(); got != DefaultHotspotMaxPerPass {
t.Fatalf("HotspotMaxPerPassOrDefault() zero = %d, want %d", got, DefaultHotspotMaxPerPass)
}

neg := GovernorConfig{HotspotMaxOpen: -1, HotspotMaxPerPass: -3}
if got := neg.HotspotMaxOpenOrDefault(); got != DefaultHotspotMaxOpen {
t.Fatalf("HotspotMaxOpenOrDefault() negative = %d, want %d", got, DefaultHotspotMaxOpen)
}
if got := neg.HotspotMaxPerPassOrDefault(); got != DefaultHotspotMaxPerPass {
t.Fatalf("HotspotMaxPerPassOrDefault() negative = %d, want %d", got, DefaultHotspotMaxPerPass)
}

set := GovernorConfig{HotspotMaxOpen: 5, HotspotMaxPerPass: 3}
if got := set.HotspotMaxOpenOrDefault(); got != 5 {
t.Fatalf("HotspotMaxOpenOrDefault() set = %d, want 5", got)
}
if got := set.HotspotMaxPerPassOrDefault(); got != 3 {
t.Fatalf("HotspotMaxPerPassOrDefault() set = %d, want 3", got)
}
}
2 changes: 2 additions & 0 deletions src/pkg/dashboard/settings_parity_allowlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -523,3 +523,5 @@ governor.hotspot_min_prs # merge-hotspot advisor; yaml-only
governor.hotspot_min_overlaps # merge-hotspot advisor; yaml-only
governor.hotspot_min_lines # merge-hotspot advisor; yaml-only
governor.hotspot_refile_days # merge-hotspot advisor; yaml-only
governor.hotspot_max_open # merge-hotspot advisor; yaml-only
governor.hotspot_max_per_pass # merge-hotspot advisor; yaml-only
79 changes: 79 additions & 0 deletions src/pkg/hotspot/hotspot_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package hotspot
import (
"context"
"errors"
"fmt"
"path/filepath"
"strings"
"testing"
Expand Down Expand Up @@ -471,3 +472,81 @@ func TestGeneratedWithSplitSourceIsReportOnly(t *testing.T) {
}
}
}

// TestFilerOpenAndPerPassCaps pins the flood guard: with three ranked
// hotspots and the default caps (2 open, 1 per pass) the filer opens one
// issue per pass, stops at two open, and fills the slot with the next
// highest-ranked hotspot only after one closes — logging once per pass.
func TestFilerOpenAndPerPassCaps(t *testing.T) {
store, err := NewStore("")
if err != nil {
t.Fatal(err)
}
clock := day(0)
now := func() time.Time { return clock }
store.SetClock(now)
cr := &fakeCreator{}
st := &fakeStater{open: true}
var logs []string
f := &Filer{Store: store, Creator: cr, Stater: st, Now: now, Refile: 30 * 24 * time.Hour,
Logf: func(format string, args ...any) { logs = append(logs, fmt.Sprintf(format, args...)) }}
ctx := context.Background()
a, b, c := hot(), hot(), hot()
b.File, c.File = "pkg/b.go", "pkg/c.go"
hs := []Hotspot{a, b, c}

actions := func(out []Outcome) string {
var s []string
for _, o := range out {
s = append(s, o.Action)
}
return strings.Join(s, ",")
}
capLogs := func() int {
n := 0
for _, l := range logs {
if strings.Contains(l, "not filing") {
n++
}
}
return n
}

// Pass 1: per-pass cap → only the top-ranked file is filed.
if got := actions(f.File(ctx, "o/r", hs)); got != "filed,capped,capped" || cr.calls != 1 {
t.Fatalf("pass 1: %s calls=%d", got, cr.calls)
}
if capLogs() != 1 {
t.Fatalf("pass 1: want one cap log line, got %q", logs)
}
// Pass 2: a is open, one slot left → b filed, c capped.
if got := actions(f.File(ctx, "o/r", hs)); got != "open,filed,capped" || cr.calls != 2 {
t.Fatalf("pass 2: %s calls=%d", got, cr.calls)
}
// Pass 3: two open → c stays capped, nothing filed.
if got := actions(f.File(ctx, "o/r", hs)); got != "open,open,capped" || cr.calls != 2 {
t.Fatalf("pass 3: %s calls=%d", got, cr.calls)
}
if capLogs() != 3 {
t.Fatalf("want exactly one cap log per pass, got %d in %q", capLogs(), logs)
}
if n := store.OpenIssues("o/r"); n != 2 {
t.Fatalf("OpenIssues=%d", n)
}

// a's issue closes → slot frees → c (next ranked) is filed; a waits refile.
st.open, st.closedAt = false, day(1)
clock = day(2)
if got := actions(f.File(ctx, "o/r", hs)); got != "refile-wait,refile-wait,filed" || cr.calls != 3 {
t.Fatalf("pass 4: %s calls=%d", got, cr.calls)
}

// Raised caps: fresh store, both remaining files land in one pass.
store2, _ := NewStore("")
store2.SetClock(now)
cr2 := &fakeCreator{}
f2 := &Filer{Store: store2, Creator: cr2, Stater: &fakeStater{open: true}, Now: now, MaxOpen: 3, MaxPerPass: 2}
if got := actions(f2.File(ctx, "o/r", hs)); got != "filed,filed,capped" || cr2.calls != 2 {
t.Fatalf("raised caps: %s calls=%d", got, cr2.calls)
}
}
55 changes: 45 additions & 10 deletions src/pkg/hotspot/issue.go
Original file line number Diff line number Diff line change
Expand Up @@ -205,30 +205,38 @@ type IssueStater interface {
HotspotIssueState(ctx context.Context, repo string, number int) (open bool, closedAt time.Time, err error)
}

// Filer turns actionable hotspots into issues with the dedupe and refile
// rules applied.
// Filer turns actionable hotspots into issues with the dedupe, refile and
// open-issue cap rules applied.
type Filer struct {
Store *Store
Gate Gate
Creator IssueCreator
Stater IssueStater
Opts Options
Refile time.Duration
Now func() time.Time
Logf func(format string, args ...any)
// MaxOpen caps the modularize issues that may be open for a repo at
// once; no new one is filed while that many are open (0 = default).
MaxOpen int
// MaxPerPass caps the issues one File call may open (0 = default).
MaxPerPass int
Now func() time.Time
Logf func(format string, args ...any)
}

// Outcome is what File decided for one hotspot.
type Outcome struct {
File string
Action string // "filed", "open", "refile-wait", "gated", "skip", "error"
Action string // "filed", "open", "refile-wait", "capped", "gated", "skip", "error"
Number int
Err error
}

// File walks hotspots and files at most one new issue per actionable file.
// It is safe to call every eval cycle: existing issues are checked, closed
// ones wait Refile before a new one may be opened.
// File walks hotspots (highest score first) and files at most one new
// issue per actionable file, at most MaxPerPass per call, and none while
// MaxOpen issues it filed earlier are still open. It is safe to call every
// eval cycle: existing issues are checked, closed ones wait Refile before a
// new one may be opened, and the next-ranked hotspot is filed when a slot
// frees.
func (f *Filer) File(ctx context.Context, repo string, hs []Hotspot) []Outcome {
now := time.Now
if f.Now != nil {
Expand All @@ -238,10 +246,20 @@ func (f *Filer) File(ctx context.Context, repo string, hs []Hotspot) []Outcome {
if refile <= 0 {
refile = time.Duration(DefaultHotspotRefileDays) * 24 * time.Hour
}
maxOpen := f.MaxOpen
if maxOpen <= 0 {
maxOpen = DefaultHotspotMaxOpen
}
maxPerPass := f.MaxPerPass
if maxPerPass <= 0 {
maxPerPass = DefaultHotspotMaxPerPass
}
logf := f.Logf
if logf == nil {
logf = func(string, ...any) {}
}
filedThisPass := 0
cappedLogged := false
var out []Outcome
for _, h := range hs {
if !h.Actionable() {
Expand Down Expand Up @@ -272,6 +290,14 @@ func (f *Filer) File(ctx context.Context, repo string, hs []Hotspot) []Outcome {
continue
}
}
if open := f.Store.OpenIssues(repo); open >= maxOpen || filedThisPass >= maxPerPass {
if !cappedLogged {
logf("hotspot: %s: not filing for %s — %d modularize issue(s) open (cap %d), %d filed this pass (cap %d)", repo, h.File, open, maxOpen, filedThisPass, maxPerPass)
cappedLogged = true
}
out = append(out, Outcome{File: h.File, Action: "capped"})
continue
}
if f.Gate != nil && !f.Gate.IssueFileAllowed(repo, h.File) {
out = append(out, Outcome{File: h.File, Action: "gated"})
continue
Expand All @@ -288,11 +314,20 @@ func (f *Filer) File(ctx context.Context, repo string, hs []Hotspot) []Outcome {
continue
}
f.Store.RecordIssue(IssueRecord{Repo: repo, File: h.File, Number: num, URL: url, FiledAt: now()})
filedThisPass++
logf("hotspot: filed %s#%d for %s", repo, num, h.File)
out = append(out, Outcome{File: h.File, Action: "filed", Number: num})
}
return out
}

// DefaultHotspotRefileDays mirrors config.DefaultHotspotRefileDays.
const DefaultHotspotRefileDays = 30
// Defaults mirroring config.DefaultHotspot*.
const (
DefaultHotspotRefileDays = 30
// DefaultHotspotMaxOpen caps open modularize issues per repo so the
// advisor cannot flood the work queue (hivecommons/hive: five filed in
// one pass while three older ones still had PRs in flight).
DefaultHotspotMaxOpen = 2
// DefaultHotspotMaxPerPass caps new issues per File call.
DefaultHotspotMaxPerPass = 1
)
14 changes: 14 additions & 0 deletions src/pkg/hotspot/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,20 @@ func (s *Store) Issue(repo, file string) (IssueRecord, bool) {
return IssueRecord{}, false
}

// OpenIssues counts the modularize issues filed for repo that are not yet
// recorded as closed. The Filer compares it against its open-issue cap.
func (s *Store) OpenIssues(repo string) int {
s.mu.Lock()
defer s.mu.Unlock()
n := 0
for _, r := range s.state.Issues {
if r.Repo == repo && r.ClosedAt.IsZero() {
n++
}
}
return n
}

// Prune drops PRs closed and conflicts older than window, and issue
// records closed longer ago than refile. It keeps the store bounded.
func (s *Store) Prune(window, refile time.Duration) {
Expand Down
Loading