Repository navigation
π dashboard: PRs-by-model rank uses a confidence-weighted score and normalised model ids - #15619
Conversation
β¦ormalised model ids Rank mode ordered models by raw first-pass merge rate with a 5-merged floor, so 5/5 on six PRs with a 51% run-failure rate out-ranked 21/22 with none; unknown models were ranked; one model split across spellings (openai-codex/gpt-6.1-sol vs gpt-6.1-sol, claude-opus-5-5 vs claude-opus-5.5, :high suffixes); and every row repeated the column eyebrow labels. - Score = Wilson 95% lower bound(first-pass merged / merged) Γ (1 β fail) Γ (1 β no ship); 5-merged floor kept; ranked rows sorted by score. - modelfamily.Canonical folds provider prefix, :variant suffix and dash-between-digits; buckets key on it, Backend lists the distinct backends and Sources keeps the raw ids for the tooltip. - unknown model β unattributed: listed last, muted, never ranked. - One sticky header row carries the column names and ? tooltips; cells are plain right-aligned numbers; data-label restores eyebrows only at the narrow breakpoints. - Help text states the formula and defines each column. - Tests: Canonical cases, Wilson bound, live-row ordering (gpt-6-astra must not out-rank claude-opus-5.5), node test that labels appear once. Fixes #15618 Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@clubanderson this PR is held at Failing checks:
Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path. |
|
stage 080-awaiting-merge-admission β 071-needs-author: optional-check-failing:test (dashboard 1/4) |
β¦hange - Keep the ? short description to one β€25-word sentence that matches the docs page; move the formula and per-column definitions into docs/dashboard-sections.md (TestDashboardSectionHelpCoverage). - Allow pkg/dashboard β pkg/modelfamily in the import ratchet: the panel now keys buckets on modelfamily.Canonical. - Use tokens for the header ? mark and backend chips (style ratchet); lower the hub inline-style baseline to the current 925 as the test asks. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Fixes #15618
Problem
Rank mode on the dashboard "PRs by model" panel sorted models by raw first-pass merge rate with a 5-merged floor, so
gpt-6-astra(5/5 first pass on six PRs, 51 % run-failure rate) ranked #1 aboveclaude-opus-5.5(21/22, no failures).unknownmodels were ranked (#4, #7). One model was split across spellings (openai-codex/gpt-6.1-solvsgpt-6.1-sol,claude-opus-5-5vsclaude-opus-5.5,β¦:high). The?help did not define PR runs / Fail / No ship or state the formula. And every row repeated the five column labels as eyebrows β the chip-stat pattern leaking into a table.Change
first_pass_mergedare exposed on the API rows.modelfamily.Canonicalfolds provider prefix,:variantsuffix and dash-between-digits (5-5β5.5). PR buckets and run stats key on it;backendbecomes the comma-joined distinct backends (backends[]too), andsources[]keeps the raw model/backend spellings with counts β shown in the row tooltip. Runtime (hosted/local/mixed) is derived from the raw sources so the Pi prefix still classifies.unknownmodel βunattributed: true, never ranked, sorted last, rendered muted with an "unattributed" chip instead of a rank..gov-pr-model-headwith MODEL / OUTCOME / MERGED / 1ST PASS / PR RUNS / FAIL / NO SHIP / PRSΒ·SHARE and a?tooltip per metric column; cells are plain right-aligned numbers. Labels come back only at the@container (max-width:42rem)/@media (max-width:620px)breakpoints viaattr(data-label)where the header is hidden.?on the section) states the normalisation, the formula and defines each column; row tooltip shows the score decomposition.Tests
TestCanonicalFoldsSpellings;TestWilsonLowerBoundPenalisesSmallSamples;TestEffectiveModelsRankUsesConfidenceWeightedScoreOnLiveRowsbuilds the live rows and asserts gpt-6-astra does not out-rank claude-opus-5.5 (it is last of the ranked), unknown is never ranked and folds into one unattributed row last,gpt-6.1-solfolds both spellings withcodex, pi, raw ids survive insources;TestGovernorPRModelsEffectivenessCarriesScoreAndUnattributed.pr_models_header_once.test.js): each column label is rendered exactly once as header text with its tooltip; no<span>label</span>eyebrows inside metric cells (ratchet);data-labelon every data row; unattributed last, muted, unranked; raw ids and formula in the tooltip.Note: folding
claude-opus-5-5/claude(6/15 first pass, 45 % no-ship) intoclaude-opus-5.5is what the normalisation asks for, so the combined row lands #4 behind fable/sonnet/gpt-6.1-sol β still well above gpt-6-astra.Before / after (live rows, 7d)
Before β rank = raw 1st-pass %, unknown ranked, labels repeated per row:
After β Wilson composite score, normalised ids with backend chips, unattributed last, single sticky header row:
static/index.htmlregenerated viago generate ./pkg/dashboard/webstatic/; changelog fragmentchangelog.d/fixed-dashboard-pr-models-rank-score.md.