Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/added-18719-pr-models-close-reasons.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- **Governor PRs by model says why unmerged PRs were closed (#18719).** Each `/api/governor/pr-models` bucket gains `closed_unmerged_reasons` (`superseded`, `duplicate`, `other`), which always sum to `closed_unmerged`. A PR counts as superseded when it carries `hive/superseded` or another PR in the same repository says `Supersedes #N` for it; duplicate when it has a `duplicate` label; anything else is other. Hover a row to see the split after the closed-unmerged count. Closed attributed PR records now keep their `Supersedes #N` number too, read from the body the scan already fetches, so no new GitHub calls. Existing fields, the bar, merge rate and rank are unchanged.
1 change: 1 addition & 0 deletions src/docs/dashboard-sections/overview-and-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ Which AI models wrote the pull requests Hive opened, how well each model's work
- Each row is one model. Model ids are normalised so one model gets one row: a provider prefix (`openai-codex/gpt-6.1-sol`), a `:variant` suffix (`gpt-5.6-sol:high`) and a dash between version digits (`claude-opus-5-5`) are folded into `gpt-6.1-sol`, `gpt-5.6-sol` and `claude-opus-5.5`. The backends that drove the model appear as chips under its name; hover the row to see the raw ids and how many pull requests each contributed.
- The row's tooltip also splits the model by backend and by effort, the reasoning effort the PR's `— hive:` signature line records (`effort=high`; **unknown** when the line has none). Each backend line and each effort line shows its pull requests (merged, open, closed without merging), how many merged first pass, and the average review rounds and fix attempts of its merged pull requests, so you can see whether one harness or effort level needed more fixing. The API returns these as the extra fields on `sources[]` and as `efforts[]` on each model.
- The bar splits a model's pull requests into merged, still open, and closed without merging.
- Hover a row to see why its closed pull requests were closed: **superseded** (it carries the `hive/superseded` label, or another pull request in the same repository says `Supersedes #N` for it), **duplicate** (it has a `duplicate` label), or **other** (abandoned or rejected). Only **other** says something about the model. The three always add up to the closed count. The API returns them as `closed_unmerged_reasons` on each bucket of `/api/governor/pr-models`.
- **Merged** is the number of merged pull requests in the window.
- **1st pass** is the share of merged pull requests that merged without a review-requested fix round.
- **PR runs** is the share of recorded task runs (from the contributor run log) that ended with a verified pull request.
Expand Down
5 changes: 5 additions & 0 deletions src/pkg/dashboard/api_governor_pr_models.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,10 @@ type governorPRModelBucket struct {
// Efforts splits the bucket by the trailer's effort= value ("unknown"
// when absent), sorted by PRs descending (hivecommons/hive#18715).
Efforts []governorPRModelEffort `json:"efforts,omitempty"`

// ClosedUnmergedReasons splits ClosedUnmerged by why the PR was closed
// (#18719); the three counts always sum to ClosedUnmerged.
ClosedUnmergedReasons governorPRModelCloseReasons `json:"closed_unmerged_reasons"`
// Repos splits the bucket by canonical owner/repo (effectivePRRepo), PRs
// descending. Weeks splits it by ISO week (Monday, UTC) of created_at,
// oldest first, and is omitted for window=7d. Rework figures in both
Expand Down Expand Up @@ -254,6 +258,7 @@ func (s *Server) handleGovernorPRModels(w http.ResponseWriter, r *http.Request)
minMergedPRs := effectiveModelsMinMergedPRs()
actionable := s.lastActionableForPRModels()
resp := aggregateGovernorPRModels(actionable.PRs.Attributed, window, time.Now())
applyGovernorPRModelCloseReasons(&resp, actionable.PRs.Attributed, governorPRModelsSupersededKeys(actionable.PRs.Attributed, actionable.PRs.Items), time.Now())
resp.FixerQueue = s.governorFixerQueue()
now := time.Now()
runs, rotated, _ := readEffectiveModelRuns(s.effectiveModelTaskRunLogPath(), governorPRModelsWindowDuration(window))
Expand Down
136 changes: 136 additions & 0 deletions src/pkg/dashboard/api_governor_pr_models_close_reasons.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
package dashboard

import (
"strings"
"time"

ghpkg "github.com/hivecommons/hive/pkg/github"
"github.com/hivecommons/hive/pkg/modelfamily"
)

// Close reasons for the PRs-by-model card (hivecommons/hive#18719). A PR the
// hive closed because another PR landed the same change, or closed as a
// duplicate, says nothing about the model that wrote it; only "other"
// (abandoned or rejected) does. The split is computed from labels and the
// "Supersedes #N" body line the hive already records, with no extra GitHub
// calls. Telling a human's close from an author giving up would need the
// timeline API, so those stay in "other".
const (
governorPRCloseReasonSuperseded = "superseded"
governorPRCloseReasonDuplicate = "duplicate"
governorPRCloseReasonOther = "other"

governorPRDuplicateLabel = "duplicate"
)

// governorPRModelCloseReasons splits a bucket's closed_unmerged count. The
// three fields always sum to closed_unmerged.
type governorPRModelCloseReasons struct {
Superseded int `json:"superseded"`
Duplicate int `json:"duplicate"`
Other int `json:"other"`
}

// governorPRModelsSupersededKeys returns the repo#number keys of PRs that
// another PR declares it supersedes ("Supersedes #N" in its body, recorded
// as PullRequest.Supersedes). The referenced PR is in the superseder's own
// repository.
func governorPRModelsSupersededKeys(lists ...[]ghpkg.PullRequest) map[string]bool {
keys := map[string]bool{}
for _, prs := range lists {
for _, pr := range prs {
if pr.Supersedes <= 0 || pr.Supersedes == pr.Number {
continue
}
keys[effectivePRKey(effectivePRRepo(pr), pr.Supersedes)] = true
}
}
return keys
}

// governorPRModelCloseReason classifies a closed-unmerged PR. Superseded
// wins over duplicate when both apply.
func governorPRModelCloseReason(pr ghpkg.PullRequest, superseded map[string]bool) string {
duplicate := false
for _, l := range pr.Labels {
switch {
case strings.EqualFold(strings.TrimSpace(l), ghpkg.SupersededLabel):
return governorPRCloseReasonSuperseded
case strings.EqualFold(strings.TrimSpace(l), governorPRDuplicateLabel):
duplicate = true
}
}
if superseded[effectivePRKey(effectivePRRepo(pr), pr.Number)] {
return governorPRCloseReasonSuperseded
}
if duplicate {
return governorPRCloseReasonDuplicate
}
return governorPRCloseReasonOther
}

func (r *governorPRModelCloseReasons) add(reason string) {
switch reason {
case governorPRCloseReasonSuperseded:
r.Superseded++
case governorPRCloseReasonDuplicate:
r.Duplicate++
default:
r.Other++
}
}

// governorPRModelsCutoff is the created-at cutoff for a normalized window;
// zero for "all". It mirrors the cutoff in aggregateGovernorPRModels.
func governorPRModelsCutoff(window string, now time.Time) time.Time {
switch normalizeGovernorPRModelsWindow(window) {
case governorPRModelsWindow7d:
return now.AddDate(0, 0, -governorPRModelsDays7)
case governorPRModelsWindow30d:
return now.AddDate(0, 0, -governorPRModelsDays30)
}
return time.Time{}
}

// applyGovernorPRModelCloseReasons fills ClosedUnmergedReasons on each bucket
// of an aggregateGovernorPRModels response built from the same prs, window
// and now. It walks the PRs with the same filter and bucket key.
func applyGovernorPRModelCloseReasons(resp *governorPRModelsResponse, prs []ghpkg.PullRequest, superseded map[string]bool, now time.Time) {
if resp == nil || len(resp.Buckets) == 0 {
return
}
cutoff := governorPRModelsCutoff(resp.Window, now)
reasons := map[string]*governorPRModelCloseReasons{}
for _, pr := range prs {
if !pr.HiveAttributed {
continue
}
if !cutoff.IsZero() && pr.CreatedAt.Before(cutoff) {
continue
}
if governorPRModelOutcome(pr) != "closed_unmerged" {
continue
}
model := modelfamily.Canonical(pr.HiveModel)
r := reasons[model]
if r == nil {
r = &governorPRModelCloseReasons{}
reasons[model] = r
}
r.add(governorPRModelCloseReason(pr, superseded))
}
for i := range resp.Buckets {
b := &resp.Buckets[i]
r := governorPRModelCloseReasons{}
if got := reasons[b.Model]; got != nil {
r = *got
}
// Other is the remainder, so the sum matches closed_unmerged even if
// a PR sits on the window edge between the two time.Now() calls.
r.Other = b.ClosedUnmerged - r.Superseded - r.Duplicate
if r.Other < 0 {
r.Other = 0
}
b.ClosedUnmergedReasons = r
}
}
125 changes: 125 additions & 0 deletions src/pkg/dashboard/api_governor_pr_models_close_reasons_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
package dashboard

import (
"encoding/json"
"net/http"
"net/http/httptest"
"strconv"
"testing"
"time"

ghpkg "github.com/hivecommons/hive/pkg/github"
"github.com/hivecommons/hive/pkg/modelfamily"
)

func closedReasonsFor(t *testing.T, resp governorPRModelsResponse, model string) governorPRModelBucket {
t.Helper()
for _, b := range resp.Buckets {
if b.Model == modelfamily.Canonical(model) {
return b
}
}
t.Fatalf("no bucket %q in %+v", model, resp.Buckets)
return governorPRModelBucket{}
}

func assertCloseReasonsSum(t *testing.T, resp governorPRModelsResponse) {
t.Helper()
for _, b := range resp.Buckets {
r := b.ClosedUnmergedReasons
if r.Superseded+r.Duplicate+r.Other != b.ClosedUnmerged {
t.Fatalf("bucket %s: reasons %+v do not sum to closed_unmerged %d", b.Model, r, b.ClosedUnmerged)
}
}
}

func TestGovernorPRModelsCloseReasons(t *testing.T) {
now := time.Date(2026, 11, 17, 12, 0, 0, 0, time.UTC)
closed := func(number int, model string, labels ...string) ghpkg.PullRequest {
return ghpkg.PullRequest{
Repo: "org/repo", Number: number, URL: "https://github.com/org/repo/pull/" + strconv.Itoa(number),
HiveAttributed: true, HiveModel: model, HiveBackend: "claude",
CreatedAt: now.AddDate(0, 0, -1), State: "closed", ClosedAt: now, Labels: labels,
}
}
attributed := []ghpkg.PullRequest{
closed(1, "opus", ghpkg.SupersededLabel), // superseded by label
closed(2, "opus"), // superseded by another PR (#10 below)
closed(3, "opus", "Duplicate"), // duplicate (case-insensitive)
closed(4, "opus"), // other
closed(5, "opus", "duplicate", "hive/superseded"), // both: superseded wins
closed(6, "sonnet"), // other; #20 in another repo supersedes a #6 there
{Repo: "org/repo", Number: 10, URL: "https://github.com/org/repo/pull/10", HiveAttributed: true, HiveModel: "opus", CreatedAt: now.AddDate(0, 0, -1), MergedAt: now, Supersedes: 2},
{Repo: "org/repo", Number: 11, URL: "https://github.com/org/repo/pull/11", HiveAttributed: true, HiveModel: "opus", CreatedAt: now.AddDate(0, 0, -1), State: "open"},
// Outside the 7d window: not counted anywhere.
{Repo: "org/repo", Number: 12, HiveAttributed: true, HiveModel: "opus", CreatedAt: now.AddDate(0, 0, -20), State: "closed", ClosedAt: now, Labels: []string{"duplicate"}},
}
open := []ghpkg.PullRequest{
{Repo: "org/other", Number: 20, URL: "https://github.com/org/other/pull/20", State: "open", Supersedes: 6},
}
superseded := governorPRModelsSupersededKeys(attributed, open)

resp := aggregateGovernorPRModels(attributed, "7d", now)
applyGovernorPRModelCloseReasons(&resp, attributed, superseded, now)
assertCloseReasonsSum(t, resp)

opus := closedReasonsFor(t, resp, "opus")
if opus.ClosedUnmerged != 5 || opus.Merged != 1 || opus.Open != 1 {
t.Fatalf("opus outcomes changed: %+v", opus)
}
if want := (governorPRModelCloseReasons{Superseded: 3, Duplicate: 1, Other: 1}); opus.ClosedUnmergedReasons != want {
t.Fatalf("opus reasons = %+v, want %+v", opus.ClosedUnmergedReasons, want)
}
sonnet := closedReasonsFor(t, resp, "sonnet")
if want := (governorPRModelCloseReasons{Other: 1}); sonnet.ClosedUnmergedReasons != want {
t.Fatalf("sonnet reasons = %+v, want %+v (a Supersedes in another repo must not match)", sonnet.ClosedUnmergedReasons, want)
}

all := aggregateGovernorPRModels(attributed, "all", now)
applyGovernorPRModelCloseReasons(&all, attributed, superseded, now)
assertCloseReasonsSum(t, all)
if got := closedReasonsFor(t, all, "opus").ClosedUnmergedReasons.Duplicate; got != 2 {
t.Fatalf("all-window opus duplicate = %d, want 2", got)
}
}

func TestHandleGovernorPRModelsCloseReasonsFromOpenSuperseder(t *testing.T) {
s := covApiServer(t)
now := time.Now().UTC()
s.deps.Scheduler = metricsSchedulerStub{actionable: &ghpkg.ActionableResult{
PRs: ghpkg.PRResult{
Attributed: []ghpkg.PullRequest{
{Repo: "org/repo", Number: 1, URL: "https://github.com/org/repo/pull/1", HiveAttributed: true, HiveModel: "opus", CreatedAt: now, State: "closed", ClosedAt: now},
{Repo: "org/repo", Number: 2, URL: "https://github.com/org/repo/pull/2", HiveAttributed: true, HiveModel: "opus", CreatedAt: now, State: "closed", ClosedAt: now},
},
Items: []ghpkg.PullRequest{
{Repo: "org/repo", Number: 3, URL: "https://github.com/org/repo/pull/3", State: "open", Supersedes: 1},
},
},
}}
req := httptest.NewRequest(http.MethodGet, "/api/governor/pr-models?window=7d", nil)
rec := httptest.NewRecorder()
s.handleGovernorPRModels(rec, req)
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
var raw struct {
Buckets []map[string]json.RawMessage `json:"buckets"`
}
if err := json.Unmarshal(rec.Body.Bytes(), &raw); err != nil {
t.Fatalf("decode: %v", err)
}
if len(raw.Buckets) != 1 {
t.Fatalf("buckets = %d, want 1", len(raw.Buckets))
}
var reasons governorPRModelCloseReasons
if err := json.Unmarshal(raw.Buckets[0]["closed_unmerged_reasons"], &reasons); err != nil {
t.Fatalf("closed_unmerged_reasons missing or malformed: %v", err)
}
if want := (governorPRModelCloseReasons{Superseded: 1, Other: 1}); reasons != want {
t.Fatalf("closed_unmerged_reasons = %+v, want %+v", reasons, want)
}
if string(raw.Buckets[0]["closed_unmerged"]) != "2" {
t.Fatalf("closed_unmerged = %s, want 2 (unchanged)", raw.Buckets[0]["closed_unmerged"])
}
}
56 changes: 56 additions & 0 deletions src/pkg/dashboard/pr_models_close_reasons.test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
// "PRs by model" row tooltip splits closed-unmerged PRs by why they were
// closed (hivecommons/hive#18719): superseded, duplicate, other. A server
// that predates closed_unmerged_reasons, or a row with nothing closed,
// keeps the old line unchanged.
const test = require('node:test');
const assert = require('node:assert');
const fs = require('node:fs');
const path = require('node:path');

const html = fs.readFileSync(path.join(__dirname, 'static', 'index.html'), 'utf8');

function pick(re, what) {
const m = html.match(re);
assert.ok(m, what + ' not found in index.html');
return m[0];
}

const src = [
'let govPRModelsSort = "effectiveness"; let govPRModelsWindow = "7d"; let govPRModelsData = null;',
'const escapeHtml = s => String(s).replace(/[&<>"\']/g, c => ({"&":"&amp;","<":"&lt;",">":"&gt;",\'"\':"&quot;","\'":"&#39;"}[c]));',
'const dashboardSectionHelpMark = () => "";',
'const formatFixTime = () => "";',
pick(/function governorPRModelUnattributed\(b\) \{[\s\S]*?\n \}\n/, 'governorPRModelUnattributed'),
pick(/function governorPRModelSliceDetail\(b\) \{[\s\S]*?\n \}\n/, 'governorPRModelSliceDetail'),
pick(/const GOV_PR_MODEL_COLUMNS = \[[\s\S]*?\n \];\n/, 'GOV_PR_MODEL_COLUMNS'),
pick(/function governorPRModelRows\(data\) \{[\s\S]*?\n \}\n/, 'governorPRModelRows'),
// Per-model PR list (#18722): rows render a PRs toggle and, when open, the list.
'let govPRModelPRsOpen = ""; const govPRModelPRsCache = {}; const loadGovernorPRModelPRs = () => {};',
pick(/function normalizeGovernorPRModelsWindowName\(windowName\) \{[\s\S]*?\n \}\n/, 'normalizeGovernorPRModelsWindowName'),
pick(/function governorPRModelPRsKey\(windowName, model\) \{[\s\S]*?\n \}\n/, 'governorPRModelPRsKey'),
pick(/function governorPRModelPRsToggleHtml\(model\) \{[\s\S]*?\n \}\n/, 'governorPRModelPRsToggleHtml'),
pick(/function governorPRModelPRListHtml\(model\) \{[\s\S]*?\n \}\n/, 'governorPRModelPRListHtml'),
pick(/function renderGovernorPRModelsTile\(\) \{[\s\S]*?\n \}\n\n function renderGovernor\(/, 'renderGovernorPRModelsTile').replace(/\n\n function renderGovernor\($/, ''),
].join('\n');
const lifted = new Function(src + '; return { render: data => { govPRModelsData = data; return renderGovernorPRModelsTile(); } };')();

const bucket = (model, closed, reasons) => ({
model, backend: 'claude', backends: ['claude'], prs: 10, merged: 10 - closed, open: 0, closed_unmerged: closed,
...(reasons ? { closed_unmerged_reasons: reasons } : {}),
rework: {}, effectiveness: {}, sources: [],
});

const render = b => lifted.render({ window: '7d', total: b.prs, min_merged_prs: 5, buckets: [b] });

test('row tooltip splits closed-unmerged PRs by close reason', () => {
const out = render(bucket('claude-opus-5.5', 6, { superseded: 3, duplicate: 1, other: 2 }));
assert.ok(out.includes('6 closed-unmerged (3 superseded · 1 duplicate · 2 other)'), 'split rendered after the closed-unmerged count');
});

test('no split when the server predates the field or nothing was closed', () => {
const old = render(bucket('claude-opus-5.5', 2, null));
assert.ok(old.includes('2 closed-unmerged\n'), 'old line kept');
assert.ok(!old.includes('superseded'), 'no split without the field');
const none = render(bucket('claude-opus-5.5', 0, { superseded: 0, duplicate: 0, other: 0 }));
assert.ok(!none.includes('superseded'), 'no split when nothing was closed');
});
Loading
Loading