Skip to content

🐛 [v6] PRs by model: closed PRs older than 14 days drop out, so 30d/all windows rank models on different spans #19365

Closed

Description

@Danathar

Target: v6.

Target branch: v6. Open the PR with --base v6. Citations are path:line on v6 @ e912f7c (release 6.52.4), plus the function name, so the pointers still work after the lines drift.

Executive summary

The PRs by model card (Governor) and the Most effective models panel (Operations) rank AI models by how well their PRs do. Both offer "30d" and "all" windows, but the hive only remembers closed PRs that were updated in the last 14 days. Anything older silently falls out.

That does more than shorten the history. It skews the ranking, because each model loses a different share of its PRs depending on when it was busy. On projectbluefin, a model used mostly in the last two weeks keeps nearly all of its PRs. A model used heavily three weeks ago loses about 90% of them. The card then compares those models as if they had been measured over the same period.

#18739 added a coverage block that admits the data is partial. That was the right first step, but the ranking itself is still wrong. The fix is to keep attributed PR records past 14 days, so a 30-day window really covers 30 days.

Problem

Closed attributed PRs come from one live scan per repo, fetchAttributedClosedPRs. It lists closed PRs sorted by updated and stops at a cutoff of now minus HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK, which defaults to 14 days. The result is kept only in an in-memory cache. The forge-state snapshot holds only open PRs. Nothing durable keeps closed-PR attribution, so the "30d" and "all" windows can never see a closed PR last touched more than 14 days ago.

Evidence

Every PR in the 20 projectbluefin repos created since 2026-10-20 that carries a — hive: trailer: 707 PRs. I applied two rules to them: drop closed PRs not updated since 2026-11-06 (the 14-day cutoff), and drop PRs whose trailer is not the last line (sibling issue linked below). What's left matches /api/governor/pr-models?window=30d on that hive almost exactly:

Model PRs on GitHub Lost to the 14-day cutoff Predicted Dashboard
claude-opus-5.5 267 146 (55%) 96 95
Ornith 1.5 35B 133 56 (42%) 70 69
gpt-6.1-terra 95 11 (12%) 83 83
gpt-6.1-sol 79 69 (87%) 9 8
claude-fable-5.1 42 38 (90%) 4 4
gpt-6-astra 10 9 (90%) 1 1
gemini-3.8-flash 9 9 (100%) 0 not listed
All 707 368 (52%) 303 306

The card reported coverage.closed_pr_lookback_days: 14, closed_prs_partial: true. For contrast, the Throughput card on the same hive (which reads forge state, not attribution) shows 831 PRs opened and 4,144 merged over 104 days of recorded history.

Consequence: Terra keeps 87% of its PRs and Sol keeps 11%. Their first-pass merge rates, rework and cost per PR are computed on very different samples, and models that were busy earlier drop below the min_merged_prs cutoff entirely.

Root cause

  • src/pkg/github/client.go:1702-1707: attributedClosedPRLookbackEnv, attributedClosedPRDefaultLookback = 14 * 24 * time.Hour, attributedClosedPRMaxPages = 5.
  • src/pkg/github/client.go:2667 fetchAttributedClosedPRs: the updated-time cutoff, with results kept only in attributedClosedPRCache (storeAttributedClosedPRs).
  • src/pkg/github/forgestate_reader.go:342 fetchAttributedClosedPRsForEnumeration: reuses that cache. Forge state carries open PRs only.
  • src/pkg/dashboard/contribute_effective_models.go:190 effectiveModelsCoverage and src/pkg/dashboard/api_governor_pr_models.go:272: report the gap but don't close it.

Wanted behavior

  • Attributed PR records (repo, number, model, backend, agent, effort, state, created/merged/closed times, and the rework fields the cards use) are persisted when the hive first sees a PR and updated when it closes. The pr-models and effective-models endpoints read their windows from that store.
  • The first deploy backfills the store once, back to at least 30 days. The backfill is bounded and spread over ticks.
  • coverage.closed_prs_partial becomes false for a window the store fully covers. The cards keep showing coverage honestly for anything older than the store's first record.

Invariants (do not break)

Acceptance criteria

  • After the backfill, /api/governor/pr-models?window=30d counts every attributed PR created in the last 30 days, including closed PRs not updated in the last 14.
  • /api/contribute/effective-models?window=30d uses the same records.
  • A hive restart does not lose history: the store survives it.
  • coverage reports the store's real start date, and closed_prs_partial is false when the window is inside it.
  • The per-tick GitHub call count does not grow after the backfill finishes.

Test plan

  • Extend src/pkg/github/attributed_closed_pr_test.go: a PR closed 20 days ago stays in the store after a scan that no longer lists it.
  • Add a dashboard test next to api_governor_pr_models tests: a 30d window with records 3, 16 and 29 days old returns all three, and coverage is not partial.
  • A test that the backfill stops at its page budget and resumes on the next tick.
  • Run in CI. Per AGENTS.md, do not run local go test/go build; gofmt -l only.

Out of scope

  • PRs dropped because a line follows the attribution trailer (sibling issue linked below).
  • Changing the scoring formula or min_merged_prs.

Sibling issue (trailer must be the last line): #19364

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 20, 2026
  2. hivecommons-hive commented on Nov 20, 2026

    @hivecommons-hive
    Contributor

    🔓 Claim by @hanthor released (covering PR merged or closed).

    Claim history
    • 2026-11-20 04:18 UTC — claimed by hanthor (contributor on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-20 10:31 UTC — released by hanthor (contributor; covering PR merged or closed)
  3. Danathar commented on Nov 20, 2026

    @Danathar
    ContributorAuthor

    Fixed by #19371 (#19371), merged into v6 at 2d427cb. The PR body lacked a closing keyword, so closing by hand.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions