Skip to content

πŸ› dashboard: PRs-by-model rank uses a confidence-weighted score and normalised model ids - #15619

Merged
clubanderson merged 2 commits into
v5from
fix/model-rank
Oct 30, 2026
Merged

clubanderson merged 2 commits into
v5from
fix/model-rank

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Fixes #15618

Problem

Rank mode on the dashboard "PRs by model" panel sorted models by raw first-pass merge rate with a 5-merged floor, so gpt-6-astra (5/5 first pass on six PRs, 51 % run-failure rate) ranked #1 above claude-opus-5.5 (21/22, no failures). unknown models were ranked (#4, #7). One model was split across spellings (openai-codex/gpt-6.1-sol vs gpt-6.1-sol, claude-opus-5-5 vs claude-opus-5.5, …:high). The ? help did not define PR runs / Fail / No ship or state the formula. And every row repeated the five column labels as eyebrows β€” the chip-stat pattern leaking into a table.

Change

  • Score = Wilson 95 % lower bound of (first-pass merged Γ· merged) Γ— (1 βˆ’ fail rate) Γ— (1 βˆ’ no-ship rate). 5-merged floor kept; ranked rows sort by score, then merged. Live numbers: opus 21/22 β†’ 0.78, fable 203/313 β†’ 0.60, astra 5/5Β·0.49Β·0.79 β†’ 0.22. Score and first_pass_merged are exposed on the API rows.
  • modelfamily.Canonical folds provider prefix, :variant suffix and dash-between-digits (5-5 β†’ 5.5). PR buckets and run stats key on it; backend becomes the comma-joined distinct backends (backends[] too), and sources[] keeps the raw model/backend spellings with counts β€” shown in the row tooltip. Runtime (hosted/local/mixed) is derived from the raw sources so the Pi prefix still classifies.
  • Unattributed: unknown model β†’ unattributed: true, never ranked, sorted last, rendered muted with an "unattributed" chip instead of a rank.
  • One header row: sticky .gov-pr-model-head with MODEL / OUTCOME / MERGED / 1ST PASS / PR RUNS / FAIL / NO SHIP / PRSΒ·SHARE and a ? tooltip per metric column; cells are plain right-aligned numbers. Labels come back only at the @container (max-width:42rem) / @media (max-width:620px) breakpoints via attr(data-label) where the header is hidden.
  • Help text (? on the section) states the normalisation, the formula and defines each column; row tooltip shows the score decomposition.
  • Backends rendered as chips under the model name.

Tests

  • Go: TestCanonicalFoldsSpellings; TestWilsonLowerBoundPenalisesSmallSamples; TestEffectiveModelsRankUsesConfidenceWeightedScoreOnLiveRows builds the live rows and asserts gpt-6-astra does not out-rank claude-opus-5.5 (it is last of the ranked), unknown is never ranked and folds into one unattributed row last, gpt-6.1-sol folds both spellings with codex, pi, raw ids survive in sources; TestGovernorPRModelsEffectivenessCarriesScoreAndUnattributed.
  • Node (pr_models_header_once.test.js): each column label is rendered exactly once as header text with its tooltip; no <span>label</span> eyebrows inside metric cells (ratchet); data-label on every data row; unattributed last, muted, unranked; raw ids and formula in the tooltip.

Note: folding claude-opus-5-5/claude (6/15 first pass, 45 % no-ship) into claude-opus-5.5 is what the normalisation asks for, so the combined row lands #4 behind fable/sonnet/gpt-6.1-sol β€” still well above gpt-6-astra.

Before / after (live rows, 7d)

Before β€” rank = raw 1st-pass %, unknown ranked, labels repeated per row:

before: PRs by model with gpt-6-astra ranked #1 and per-row eyebrow labels

After β€” Wilson composite score, normalised ids with backend chips, unattributed last, single sticky header row:

after: PRs by model with a single header row, claude-fable-5.1 ranked #1, unattributed row muted at the bottom

static/index.html regenerated via go generate ./pkg/dashboard/webstatic/; changelog fragment changelog.d/fixed-dashboard-pr-models-rank-score.md.

…ormalised model ids

Rank mode ordered models by raw first-pass merge rate with a 5-merged
floor, so 5/5 on six PRs with a 51% run-failure rate out-ranked 21/22
with none; unknown models were ranked; one model split across spellings
(openai-codex/gpt-6.1-sol vs gpt-6.1-sol, claude-opus-5-5 vs
claude-opus-5.5, :high suffixes); and every row repeated the column
eyebrow labels.

- Score = Wilson 95% lower bound(first-pass merged / merged) Γ— (1 βˆ’ fail)
  Γ— (1 βˆ’ no ship); 5-merged floor kept; ranked rows sorted by score.
- modelfamily.Canonical folds provider prefix, :variant suffix and
  dash-between-digits; buckets key on it, Backend lists the distinct
  backends and Sources keeps the raw ids for the tooltip.
- unknown model β†’ unattributed: listed last, muted, never ranked.
- One sticky header row carries the column names and ? tooltips; cells
  are plain right-aligned numbers; data-label restores eyebrows only at
  the narrow breakpoints.
- Help text states the formula and defines each column.
- Tests: Canonical cases, Wilson bound, live-row ordering (gpt-6-astra
  must not out-rank claude-opus-5.5), node test that labels appear once.

Fixes #15618

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Oct 30, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#15530 πŸ› fix(dashboard): issue-row linked PR passes null when the lineage chip is empty same files src/pkg/dashboard/static/index.html
#15526 🌱 dashboard: split stage_tracker.go into 4 files same files src/docs/dashboard-sections.md
#15620 ✨ dashboard: running-cost tile rotates hour / today / week / month same files src/docs/dashboard-sections.md
src/pkg/dashboard/static-src/css/governor.css
src/pkg/dashboard/static/index.html
#15616 πŸ› tracker: PRs under repair leave the merge train same files src/docs/dashboard-sections.md
src/pkg/dashboard/static/index.html
#15574 🌱 dashboard: split stage_tracker.go into 4 files same files src/docs/dashboard-sections.md
#15566 🌱 refactor: split stage tracker UI tests same files src/pkg/dashboard/static/index.html
#15623 πŸ› dashboard: kiosk mode renders dark by default same files src/pkg/dashboard/static/index.html

@hivecommons-hive

Copy link
Copy Markdown
Contributor

@clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (dashboard 1/4)
  • test (rest 2/4)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

stage 080-awaiting-merge-admission β†’ 071-needs-author: optional-check-failing:test (dashboard 1/4)

…hange

- Keep the ? short description to one ≀25-word sentence that matches
  the docs page; move the formula and per-column definitions into
  docs/dashboard-sections.md (TestDashboardSectionHelpCoverage).
- Allow pkg/dashboard β†’ pkg/modelfamily in the import ratchet: the
  panel now keys buckets on modelfamily.Canonical.
- Use tokens for the header ? mark and backend chips (style ratchet);
  lower the hub inline-style baseline to the current 925 as the test
  asks.

Signed-off-by: Andrew Anderson <andy@clubanderson.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

dashboard: PRs-by-model rank is raw first-pass % with a 5-merged floor, ranks unknown models, splits one model across spellings

1 participant