Repository navigation
Description
Target: v6.
Target branch:
v6. Open the PR with--base v6. Citations arepath:lineonv6@e912f7c(release 6.52.4), plus the function name, so the pointers still work after the lines drift.
Executive summary
The PRs by model card (Governor) and the Most effective models panel (Operations) rank AI models by how well their PRs do. Both offer "30d" and "all" windows, but the hive only remembers closed PRs that were updated in the last 14 days. Anything older silently falls out.
That does more than shorten the history. It skews the ranking, because each model loses a different share of its PRs depending on when it was busy. On projectbluefin, a model used mostly in the last two weeks keeps nearly all of its PRs. A model used heavily three weeks ago loses about 90% of them. The card then compares those models as if they had been measured over the same period.
#18739 added a coverage block that admits the data is partial. That was the right first step, but the ranking itself is still wrong. The fix is to keep attributed PR records past 14 days, so a 30-day window really covers 30 days.
Problem
Closed attributed PRs come from one live scan per repo, fetchAttributedClosedPRs. It lists closed PRs sorted by updated and stops at a cutoff of now minus HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK, which defaults to 14 days. The result is kept only in an in-memory cache. The forge-state snapshot holds only open PRs. Nothing durable keeps closed-PR attribution, so the "30d" and "all" windows can never see a closed PR last touched more than 14 days ago.
Evidence
Every PR in the 20 projectbluefin repos created since 2026-10-20 that carries a — hive: trailer: 707 PRs. I applied two rules to them: drop closed PRs not updated since 2026-11-06 (the 14-day cutoff), and drop PRs whose trailer is not the last line (sibling issue linked below). What's left matches /api/governor/pr-models?window=30d on that hive almost exactly:
| Model | PRs on GitHub | Lost to the 14-day cutoff | Predicted | Dashboard |
|---|---|---|---|---|
| claude-opus-5.5 | 267 | 146 (55%) | 96 | 95 |
| Ornith 1.5 35B | 133 | 56 (42%) | 70 | 69 |
| gpt-6.1-terra | 95 | 11 (12%) | 83 | 83 |
| gpt-6.1-sol | 79 | 69 (87%) | 9 | 8 |
| claude-fable-5.1 | 42 | 38 (90%) | 4 | 4 |
| gpt-6-astra | 10 | 9 (90%) | 1 | 1 |
| gemini-3.8-flash | 9 | 9 (100%) | 0 | not listed |
| All | 707 | 368 (52%) | 303 | 306 |
The card reported coverage.closed_pr_lookback_days: 14, closed_prs_partial: true. For contrast, the Throughput card on the same hive (which reads forge state, not attribution) shows 831 PRs opened and 4,144 merged over 104 days of recorded history.
Consequence: Terra keeps 87% of its PRs and Sol keeps 11%. Their first-pass merge rates, rework and cost per PR are computed on very different samples, and models that were busy earlier drop below the min_merged_prs cutoff entirely.
Root cause
src/pkg/github/client.go:1702-1707:attributedClosedPRLookbackEnv,attributedClosedPRDefaultLookback = 14 * 24 * time.Hour,attributedClosedPRMaxPages = 5.src/pkg/github/client.go:2667fetchAttributedClosedPRs: the updated-time cutoff, with results kept only inattributedClosedPRCache(storeAttributedClosedPRs).src/pkg/github/forgestate_reader.go:342fetchAttributedClosedPRsForEnumeration: reuses that cache. Forge state carries open PRs only.src/pkg/dashboard/contribute_effective_models.go:190effectiveModelsCoverageandsrc/pkg/dashboard/api_governor_pr_models.go:272: report the gap but don't close it.
Wanted behavior
- Attributed PR records (repo, number, model, backend, agent, effort, state, created/merged/closed times, and the rework fields the cards use) are persisted when the hive first sees a PR and updated when it closes. The pr-models and effective-models endpoints read their windows from that store.
- The first deploy backfills the store once, back to at least 30 days. The backfill is bounded and spread over ticks.
coverage.closed_prs_partialbecomesfalsefor a window the store fully covers. The cards keep showing coverage honestly for anything older than the store's first record.
Invariants (do not break)
- 🐛 github: bound per-tick closed-PR scan and never let it fail enumeration #9253 / 🐛 github: bound per-tick closed-PR scan and never let it fail enumeration (v6) #9255 bounded the per-tick closed-PR scan so it never burns the GitHub API budget or fails enumeration. Keep that: the steady-state scan stays bounded, and any backfill is one-time, paged and rate-limit aware.
- A closed-PR scan error must still never fail enumeration (
warnAttributedClosedPRScanpath). - The store must hold no PR bodies, only the parsed attribution and outcome fields.
Acceptance criteria
- After the backfill,
/api/governor/pr-models?window=30dcounts every attributed PR created in the last 30 days, including closed PRs not updated in the last 14. -
/api/contribute/effective-models?window=30duses the same records. - A hive restart does not lose history: the store survives it.
-
coveragereports the store's real start date, andclosed_prs_partialisfalsewhen the window is inside it. - The per-tick GitHub call count does not grow after the backfill finishes.
Test plan
- Extend
src/pkg/github/attributed_closed_pr_test.go: a PR closed 20 days ago stays in the store after a scan that no longer lists it. - Add a dashboard test next to
api_governor_pr_modelstests: a 30d window with records 3, 16 and 29 days old returns all three, and coverage is not partial. - A test that the backfill stops at its page budget and resumes on the next tick.
- Run in CI. Per AGENTS.md, do not run local
go test/go build;gofmt -lonly.
Out of scope
- PRs dropped because a line follows the attribution trailer (sibling issue linked below).
- Changing the scoring formula or
min_merged_prs.
Sibling issue (trailer must be the last line): #19364
Activity
hivecommons-hive commentedon Nov 20, 2026
🔓 Claim by @hanthor released (covering PR merged or closed).
Claim history
hanthor(contributor on hive hosted-available-oke-11-placeholder-r05x)hanthor(contributor; covering PR merged or closed)Danathar commentedon Nov 20, 2026
Fixed by #19371 (#19371), merged into v6 at 2d427cb. The PR body lacked a closing keyword, so closing by hand.