Skip to content

πŸ› [v5] PRs by model: Governor card shows 0% for unmeasured runs, and gives no history coverage or confidence rangeΒ #18713

Closed

Description

@Danathar

Target: v5.

Target branch: v5. Open the PR with --base v5. The code paths are the same on v6 (line numbers can differ), and the regular v5 β†’ v6 top-up carries it there, so do not open a second PR for v6. Citations are path:line on v5 @ 8881a15, plus the function name, so the pointers still work after the lines drift.

In the PR body write Refs #<this issue>. Do not use a closing keyword: a PR based on v5 cannot close an issue that way, so a maintainer closes this one after the merge.

Executive summary

The Governor's PRs by model card ranks the AI models that wrote the hive's pull requests. In three ways it looks more certain than its data is:

  1. A model with no runs in the window shows 0% failure, which reads as "never fails" when nothing was measured.
  2. The card does not say how much history it covers. "All" really means "what the hive still remembers": open PRs plus the closed PRs a bounded scan still holds.
  3. Rates on six PRs look as solid as rates on ninety.

The Operations page's Most effective models panel already fixed the first two for itself (#10689, #10688). This issue brings the same honesty to the Governor card and its endpoint. It also adds a confidence range to each rate.

The ranking does not change.

Problem

  1. Unmeasured shown as 0%. When effectiveness.runs is 0, the card renders PR runs, Fail and No ship as 0%. The score tooltip still states the full formula, Γ— (1 βˆ’ fail) Γ— (1 βˆ’ no ship), although those factors were skipped. (The score is a Wilson lower bound of first-pass merges Γ· merged PRs, times the two run factors. A Wilson lower bound is a conservative success-rate estimate that penalises small samples. By design from πŸ› dashboard: PRs-by-model rank uses a confidence-weighted score and normalised model idsΒ #15619, a row with no runs keeps the bound as-is.)

  2. No coverage. Two things limit what the card can see:

    • Closed PRs come from a rolling scan: 14 days by default, set by HIVE_ATTRIBUTED_CLOSED_PR_LOOKBACK, and at most five 100-PR pages per repo.
    • Runs come from the live task-run log plus one rotated file.

    πŸ› bug: Most effective models 30d/All silently use bounded recent historyΒ #10688 added a coverage block to /api/contribute/effective-models and a coverage note to the Operations panel. /api/governor/pr-models has neither.

  3. No uncertainty on rates. first_pass_merge_rate, verified_pr_run_rate, failure_rate and nothing_to_ship_rate are point values. The Wilson bound exists only inside score.

Evidence

1. Zero runs shown as 0%. In window=30d, claude-fable-5.1 ranks #1: score 0.436, 6 merged, 5/6 first pass, runs: 0. The card shows it with 0% fail and 0% no-ship, above claude-opus-5.5 (score 0.408, 94 merged, 61/94 first pass, 186 runs, 0.5% fail, 25.3% no-ship). With window=all, where 26 runs fall inside the window, the same six Fable PRs score 0.235 and rank #2.

2. "All" is not all history.

window total attributed PRs
7d 125
30d 308
all 309

Only one more PR in all than in 30d. Yet GET /api/cost on the same hive reports merged_prs: 513 for PRs authored by hivecommons-hive[bot] since 2026-09-15. That count uses a different population: bot-authored only, counted by a separate cached search. It still shows that the PR history exists and the card cannot see it.

3. Small samples. claude-fable-5.1 shows first pass 83% on 6 merged PRs. gpt-6.1-terra shows 60% on 73. Nothing on the card says the first figure could plausibly sit anywhere from roughly 44% to 97% (Wilson 95%).

Root cause

  • src/pkg/dashboard/static-src/js/governor-view.js:157-159@8881a15 (renderGovernorPRModelsTile) calls pct(...) on the three run rates with no check on eff.runs. The tooltip string at :166 always states the full formula.
  • src/pkg/dashboard/api_governor_pr_models.go:24@8881a15: governorPRModelsResponse has no coverage field. handleGovernorPRModels (:111) reads readEffectiveModelRuns but discards its rotated result (:117).
  • src/pkg/dashboard/contribute_effective_models.go:190@8881a15 effectiveModelsCoverage computes exactly the needed block. It is called only from handleContributeEffectiveModels (:105).
  • src/pkg/dashboard/contribute_effective_models.go:345@8881a15 wilsonLowerBound exists. There is no upper bound, and no interval on the response rows.

Wanted behavior

  • Unmeasured run columns on the Governor card show "β€”" when runs is 0. Use the same wording as the Operations panel: "no runs recorded, not a measured 0%". The score tooltip says the score is the first-pass bound only when there were no runs.
  • Coverage. /api/governor/pr-models returns a coverage object. It has the same fields as the one on /api/contribute/effective-models (closed_pr_lookback_days, closed_prs_partial, runs_since, runs_partial), plus earliest_pr_created_at, the oldest created_at among the PRs counted. The card shows a short coverage note when either *_partial is true.
  • Intervals. Each bucket's rework gains first_pass_merge_rate_ci: [low, high]. Each effectiveness gains verified_pr_run_rate_ci, failure_rate_ci and nothing_to_ship_rate_ci. All are Wilson 95% intervals with the same z as wilsonLowerBound. They are omitted, or null, when the denominator is 0. The card shows them in the cell tooltip. Do not widen the table.

Acceptance criteria

  • With runs == 0, the card shows "β€”" in the three run columns. A row with runs > 0 and a measured 0% still shows 0%.
  • coverage is present on every /api/governor/pr-models response. It matches what /api/contribute/effective-models returns for the same window.
  • earliest_pr_created_at equals the minimum created_at of the counted PRs. It is omitted when there are none.
  • The interval fields exist where the denominator is greater than 0, and satisfy 0 ≀ low ≀ rate ≀ high ≀ 1. For 5/6 they are β‰ˆ [0.44, 0.97].
  • score, rank, eligible and every existing field are unchanged for the same input.

Test plan (CI decides)

  • src/pkg/dashboard/effective_models_rank_score_test.go:
    • next to TestWilsonLowerBoundPenalisesSmallSamples, add TestWilsonIntervalBracketsRate, covering 0/0, 0/n, n/n and 5/6;
    • extend TestGovernorPRModelsEffectivenessCarriesScoreAndUnattributed to assert the score is unchanged.
  • src/pkg/dashboard/api_governor_pr_models_test.go:
    • extend TestHandleGovernorPRModelsIncludesEffectiveMetrics to check coverage and the interval keys;
    • model a coverage test on TestHandleContributeEffectiveModelsReportsCoverage (in contribute_effective_models_test.go).
  • dashboard/test/governor-pr-models-ui.test.js:
    • runs: 0 renders "β€”" in the three columns, and runs: 5, failure_rate: 0 renders 0%;
    • the coverage note appears when closed_prs_partial is true.
  • src/pkg/dashboard/governor_pr_models_ui_test.go: add the new strings to the embedded-HTML check.
  • Model the UI test on TestContributeOperationsEffectiveModelsUnmeasuredRunRates, which is how πŸ› bug: Most effective models displays 0% run rates when no runs were recordedΒ #10689 tested the Operations panel.

Suggested implementation

  1. Add wilsonInterval(successes, n) (low, high float64, ok bool) next to wilsonLowerBound, and make wilsonLowerBound call it.
  2. Add the *_ci fields with omitempty pointer or slice types, so they are absent when undefined.
  3. In handleGovernorPRModels, keep rotated and call effectiveModelsCoverage(governorPRModelsWindowDuration(window), ghpkg.AttributedClosedPRLookback(), runs, rotated, time.Now()). Compute earliest_pr_created_at inside aggregateGovernorPRModels.
  4. governor-view.js: a helper that returns "β€”" for unmeasured run cells, the CI in tooltips, and the coverage note. Match the rest of the Governor tab for muted text and notes.
  5. Document the new fields where the PRs-by-model columns are described in src/docs/.

Rejected options

Out of scope / do not change

  • The score formula, min_merged_prs (5), the ranking and the scan's lookback.
  • The Operations page panel, which already does this.

Dependencies

None. Other issues in this set also edit api_governor_pr_models.go and governor-view.js. Land them one at a time and rebase.

Rules that apply to every issue in this set

  • Backward compatible. Existing response fields keep their names, types and meaning. Add new fields; never rename or repurpose one. Old dashboards and the admin MCP read these endpoints.
  • No new GitHub API calls without a cache. Everything here is computed from data the hive already holds: the scheduler's PR snapshot, the contributor task-run log and the token collector.
  • Tests. AGENTS.md forbids running go test, go build or go vet locally. Write the tests, run gofmt -l locally, and let CI decide. Node UI tests under dashboard/test/ run in CI too.
  • Changelog. Add a changelog.d/ fragment, as other dashboard PRs do.
  • Numbers. The evidence was pulled on 2026-11-17 between 9:10 and 9:20 AM EST, from the projectbluefin hive running a v6 edge build, 3b50113. It came through the same endpoints the dashboard reads: GET /api/governor/pr-models, GET /api/contribute/run-stats/models and GET /api/cost. The code paths quoted are identical on v5.

Activity

  1. added
    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.
    on Nov 17, 2026
  2. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    πŸ”“ Claim by agent scanner released (holding agent relaunched).

    Claim history
    • 2026-11-17 15:22 UTC β€” claimed by scanner (agent on hive hosted-available-oke-11-placeholder-r05x)
    • 2026-11-17 15:25 UTC β€” released by scanner (agent; holding agent relaunched)
  3. hivecommons-hive commented on Nov 17, 2026

    @hivecommons-hive
    Contributor

    Hive status β€” updated in place; edits do not notify. Full history is in the Activity Monitor.

    PR #18739 covers this issue: #18739

    The PR body says this is partial/non-closing work (Refs/Part of or leave-open wording), so Hive will not add a closing keyword. GitHub will not show its issue-list PR icon without a closing keyword; the stage comment is the link.

  4. added
    hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionable
    stage/010-actionableTriaged; eligible for the hive to pick up
    and removed on Nov 17, 2026
  5. clubanderson commented on Nov 17, 2026

    @clubanderson
    Member

    Fixed by #18739 (merged to v5): run columns show a dash when a model has no runs, the card drops the skipped run factors from the score tooltip, /api/governor/pr-models returns coverage and earliest_pr_created_at, and rates carry Wilson 95% intervals. Closing by hand because a v5-based PR cannot close the issue itself.

  6. github-actions commented on Nov 17, 2026

    @github-actions
    Contributor

    @Danathar this issue was closed by @clubanderson as completed.

    If this was closed by a merged PR, the fix only takes effect once it ships in a tagged release and your deployment updates to it β€” check your running version against the fix before concluding the fix failed.

    If you are still seeing the problem, comment /reopen and it will be reopened for you β€” you do not need write access. Please add what you are still observing (running version, logs, steps) so it is not closed the same way again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.hive/partially-addressedA merged PR addressed only part of this issue; remainder still actionable

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions