Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/fixed-dashboard-pr-models-rank-score.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- dashboard: "PRs by model" rank now uses a confidence-weighted score (Wilson 95% lower bound of first-pass merge rate × (1 − fail) × (1 − no ship)) instead of raw first-pass %, folds model-id spellings (provider prefix, `:variant`, `5-5`→`5.5`) into one row with the raw ids in the tooltip, lists unattributed rows unranked at the bottom, renders the metric column labels once in a sticky header with `?` tooltips, and explains the formula and each column in the help text (#15618).
22 changes: 14 additions & 8 deletions src/docs/dashboard-sections.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,21 +127,27 @@ How busy Hive thinks your projects are right now, and how often it wakes its wor

## PRs by model

Which AI models wrote the pull requests Hive opened, and how often each model's work was merged without extra fixes.
Which AI models wrote the pull requests Hive opened, how well each model's work landed, and how the models rank against each other.

**What it tells you.** This part of the Governor section compares the AI models your workers use. It shows how many pull requests each model wrote and how they ended.
**What it tells you.** This part of the Governor section compares the AI models your workers use. It shows how many pull requests each model wrote, how they ended, and a rank that favours models with a long, clean record over models with a short, lucky one.

**How the numbers are worked out.**

- Each row is one model. The bar splits its pull requests into merged, still open, and closed without merging.
- **First-pass rate** is the share of merged pull requests that needed no follow-up fixes.
- **Failure rate** and **nothing to ship** count runs that failed or finished without a pull request.
- A model gets a **rank** only after it has at least 5 merged pull requests in the window. With fewer, the sample is too small to compare.
- Each row is one model. Model ids are normalised so one model gets one row: a provider prefix (`openai-codex/gpt-6.1-sol`), a `:variant` suffix (`gpt-5.6-sol:high`) and a dash between version digits (`claude-opus-5-5`) are folded into `gpt-6.1-sol`, `gpt-5.6-sol` and `claude-opus-5.5`. The backends that drove the model appear as chips under its name; hover the row to see the raw ids and how many pull requests each contributed.
- The bar splits a model's pull requests into merged, still open, and closed without merging.
- **Merged** is the number of merged pull requests in the window.
- **1st pass** is the share of merged pull requests that merged without a review-requested fix round.
- **PR runs** is the share of recorded task runs (from the contributor run log) that ended with a verified pull request.
- **Fail** is the share of recorded task runs that failed: crashed, timed out, or exited non-zero.
- **No ship** is the share of recorded task runs that completed but opened no pull request.
- **Rank** orders models by a score that combines those columns and penalises small samples: `score = Wilson 95% lower bound(1st-pass merged ÷ merged) × (1 − Fail) × (1 − No ship)`. The Wilson lower bound is the pessimistic end of the confidence interval for the first-pass rate, so 5 of 5 (lower bound 0.57) scores below 21 of 22 (0.78) and well below 203 of 313 (0.60 after the bound, but with no failures). A model with no recorded runs keeps its bound as the score.
- A model gets a rank only after it has at least 5 merged pull requests in the window. With fewer, the sample is too small to compare and the row is listed without a rank.
- Rows whose pull requests carry no model id are shown last, muted, as **unattributed**. They are counted but never ranked.
- You can show the last 7 days, the last 30 days, or all time. You can sort by rank or by count.

**What it is good for.** Use it when choosing which model a worker should use. A model with a low first-pass rate costs more review time.
**What it is good for.** Use it when choosing which model a worker should use. A model with a low first-pass rate or a high failure rate costs more review and retry time; the rank already weighs both.

**Example.** Model A has 40 pull requests and a 70% first-pass rate. Model B has 6 pull requests and a 30% rate. Model A is the safer default.
**Example.** Model A has 40 merged pull requests, 28 of them first pass, and 2% failed runs. Model B has 5 merged pull requests, all first pass, and 51% failed runs. A scores about 0.54 and B about 0.28, so A ranks higher even though B shows 100% first pass.

**When it appears.** Inside the Governor section, so at autonomy level L2 and above.

Expand Down
1 change: 1 addition & 0 deletions src/internal/testutil/dashboard_import_allowlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ github.com/hivecommons/hive/pkg/hubbackup
github.com/hivecommons/hive/pkg/inferencehealth
github.com/hivecommons/hive/pkg/ioscan
github.com/hivecommons/hive/pkg/knowledge
github.com/hivecommons/hive/pkg/modelfamily
github.com/hivecommons/hive/pkg/notify
github.com/hivecommons/hive/pkg/nous
github.com/hivecommons/hive/pkg/planning
Expand Down
85 changes: 73 additions & 12 deletions src/pkg/dashboard/api_governor_pr_models.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
package dashboard

import (
Expand All @@ -8,6 +8,7 @@
"time"

ghpkg "github.com/hivecommons/hive/pkg/github"
"github.com/hivecommons/hive/pkg/modelfamily"
)

const (
Expand All @@ -28,9 +29,16 @@
MostReworked []governorPRMostReworkedItem `json:"most_reworked,omitempty"`
}

// governorPRModelBucket is one canonical model (modelfamily.Canonical folds
// "openai-codex/gpt-6.1-sol", "gpt-6.1-sol" and "gpt-6.1-sol:high" into one
// row). Backend lists the distinct backends that drove it, comma-joined;
// Sources keeps the raw model/backend spellings and their counts so the
// dashboard can show them in a tooltip.
type governorPRModelBucket struct {
Model string `json:"model"`
Backend string `json:"backend"`
Backends []string `json:"backends,omitempty"`
Sources []governorPRModelSource `json:"sources,omitempty"`
PRs int `json:"prs"`
Merged int `json:"merged"`
ClosedUnmerged int `json:"closed_unmerged"`
Expand All @@ -40,6 +48,14 @@
Agents []governorPRModelAgentBucket `json:"agents,omitempty"`
}

// governorPRModelSource is one raw attribution spelling folded into a bucket.
type governorPRModelSource struct {
Model string `json:"model"`
Backend string `json:"backend"`
PRs int `json:"prs"`
Merged int `json:"merged"`
}

type governorPRModelRework struct {
SamplePRs int `json:"sample_prs"`
FirstPassMerged int `json:"first_pass_merged"`
Expand All @@ -56,7 +72,10 @@
type governorPRModelEffectiveness struct {
Rank int `json:"rank,omitempty"`
Eligible bool `json:"eligible"`
Unattributed bool `json:"unattributed,omitempty"`
Score float64 `json:"score"`
MergedPRs int `json:"merged_prs"`
FirstPassMerged int `json:"first_pass_merged"`
FirstPassMergeRate float64 `json:"first_pass_merge_rate"`
RunCount int `json:"runs"`
VerifiedPRRunRate float64 `json:"verified_pr_run_rate"`
Expand Down Expand Up @@ -132,8 +151,10 @@

type bucketState struct {
governorPRModelBucket
agents map[string]*governorPRModelAgentBucket
rework reworkAccumulator
agents map[string]*governorPRModelAgentBucket
sources map[string]*governorPRModelSource
backends map[string]bool
rework reworkAccumulator
}
buckets := map[string]*bucketState{}
resp := governorPRModelsResponse{Window: window, MinMergedPRs: effectiveModelsMinMergedPRs()}
Expand All @@ -144,23 +165,36 @@
if !cutoff.IsZero() && pr.CreatedAt.Before(cutoff) {
continue
}
model := ghpkg.NormalizeAttributionModel(pr.HiveModel)
rawModel := ghpkg.NormalizeAttributionModel(pr.HiveModel)
model := modelfamily.Canonical(pr.HiveModel)
backend := ghpkg.NormalizeAttributionValue(pr.HiveBackend)
agent := ghpkg.NormalizeAttributionValue(pr.HiveAgent)
if agent == "unknown" {
agent = ""
}
key := model + "\x00" + backend
b := buckets[key]
b := buckets[model]
if b == nil {
b = &bucketState{
governorPRModelBucket: governorPRModelBucket{Model: model, Backend: backend},
governorPRModelBucket: governorPRModelBucket{Model: model},
agents: map[string]*governorPRModelAgentBucket{},
sources: map[string]*governorPRModelSource{},
backends: map[string]bool{},
}
buckets[key] = b
buckets[model] = b
}
b.backends[backend] = true
outcome := governorPRModelOutcome(pr)
addOutcome(&b.PRs, &b.Merged, &b.ClosedUnmerged, &b.Open, outcome)
srcKey := rawModel + "\x00" + backend
src := b.sources[srcKey]
if src == nil {
src = &governorPRModelSource{Model: rawModel, Backend: backend}
b.sources[srcKey] = src
}
src.PRs++
if outcome == "merged" {
src.Merged++
}
if outcome == "merged" {
b.rework.add(pr.Rework)
}
Expand All @@ -187,14 +221,27 @@
}
addOutcome(&ab.PRs, &ab.Merged, &ab.ClosedUnmerged, &ab.Open, outcome)
resp.Total++
if model == "unknown" {
if model == modelfamily.Unknown {
resp.Unknown++
}
}

resp.Buckets = make([]governorPRModelBucket, 0, len(buckets))
for _, b := range buckets {
b.Rework = b.rework.finish()
b.Backends, b.Backend = joinedBackends(b.backends)
for _, src := range b.sources {
b.Sources = append(b.Sources, *src)
}
sort.Slice(b.Sources, func(i, j int) bool {
if b.Sources[i].PRs != b.Sources[j].PRs {
return b.Sources[i].PRs > b.Sources[j].PRs
}
if b.Sources[i].Model != b.Sources[j].Model {
return b.Sources[i].Model < b.Sources[j].Model
}
return b.Sources[i].Backend < b.Sources[j].Backend
})
for _, ab := range b.agents {
b.Agents = append(b.Agents, *ab)
}
Expand Down Expand Up @@ -232,27 +279,41 @@
return resp
}

// joinedBackends returns the sorted distinct backends and their comma-joined
// display form ("codex, pi").
func joinedBackends(set map[string]bool) ([]string, string) {
out := make([]string, 0, len(set))
for b := range set {
out = append(out, b)
}
sort.Strings(out)
return out, strings.Join(out, ", ")
}

func applyGovernorPRModelEffectiveness(resp *governorPRModelsResponse, effective contributeEffectiveModelsResponse, minMergedPRs int) {
if resp == nil {
return
}
resp.MinMergedPRs = minMergedPRs
rows := map[string]contributeEffectiveModelRow{}
for _, row := range effective.Ranked {
rows[effectiveModelKey(row.Model, row.Backend)] = row
rows[row.Model] = row
}
for _, row := range effective.Insufficient {
rows[effectiveModelKey(row.Model, row.Backend)] = row
rows[row.Model] = row
}
for i := range resp.Buckets {
row, ok := rows[effectiveModelKey(resp.Buckets[i].Model, resp.Buckets[i].Backend)]
row, ok := rows[resp.Buckets[i].Model]
if !ok {
continue
}
resp.Buckets[i].Effectiveness = governorPRModelEffectiveness{
Rank: row.EffectivenessRank,
Eligible: row.MergedPRs >= minMergedPRs,
Eligible: row.EffectivenessRank > 0,
Unattributed: row.Unattributed,
Score: row.Score,
MergedPRs: row.MergedPRs,
FirstPassMerged: row.FirstPassMerged,
FirstPassMergeRate: row.FirstPassMergeRate,
RunCount: row.RunCount,
VerifiedPRRunRate: row.VerifiedPRRunRate,
Expand Down
Loading
Loading