Repository navigation
✨ Spend and contributor run stats by model: add the canonical model name - #18781
hivecommons-hive[bot] merged 4 commits into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (7 waiting): fixes are handed out before new work as soon as you have capacity. |
/api/cost estimated.by_model[] and every /api/contribute/run-stats(/models) row now carry canonical_model (modelfamily.Canonical), the id the Governor's PRs-by-model card groups by. Both run-stats endpoints accept group=canonical, which merges spellings (runs summed, rates from summed counts) and lists raw_models[]. Without it, existing rows are unchanged. The Cost card and the contributor wall's model evidence show the canonical id with the raw spelling in a tooltip. Refs hivecommons#18718 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
…mmons#18721) too Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Danathar <Danathar@users.noreply.github.com>
4974911 to
4a4383d
Compare
There was a problem hiding this comment.
Style review at 4a4383d: conventions followed (static-src and index.html in sync, openapi, docs and changelog updated, tests added). Two minor, non-blocking nits are inline: a duplicated counter pair and a mis-wrapped doc comment. Approving from the style perspective.
— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.326
|
|
||
| // verifiedRuns and failedRuns are the counts behind the two rates, kept | ||
| // so group=canonical can recompute the rates exactly. | ||
| verifiedRuns, failedRuns int |
There was a problem hiding this comment.
verifiedRuns, failedRuns duplicate the aggregator's existing verified/failed (copied at the a.verifiedRuns, a.failedRuns = a.verified, a.failed line in readTaskRunModelStats). Two names for one count on an exported struct is a maintenance trap. Consider exposing the existing counters to the grouping code instead of mirroring them.
| // groupWallModelStatsByCanonical merges run-stats rows that share a canonical | ||
| // model id (modelfamily.Canonical, the rule PRs-by-model uses). Runs and the | ||
| // verified and failed counts are summed and the two rates recomputed from the | ||
| // sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings |
There was a problem hiding this comment.
This doc-comment line was spliced into one very long line and breaks the wrapping used in the rest of the block. Please reflow it:
| // sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings | |
| // verified and failed counts are summed and the two rates recomputed from the | |
| // sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a | |
| // merged row says exactly what one row over all its spellings would. Model | |
| // becomes the canonical id; RawModels lists each spelling with its own runs, | |
| // most runs first. Output is sorted by model, like wallStatsList. |
|
Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor. Review findings from One in-scope correctness issue in the wall evidence line. The Go grouping logic itself ( Wall evidence line labels single-spelling stats with the canonical model id ( Fix: either aggregate by canonical id when attaching - if model := p.Tags["model"]; model != "" {
- if st, ok := byModel[strings.ToLower(model)]; ok {
- cp := st
- p.ModelEvidence = &cp
- }
- }
+ if model := p.Tags["model"]; model != "" {
+ if st, ok := byModel[strings.ToLower(model)]; ok {
+ cp := st
+ if g := groupWallModelStatsByCanonical(wallStatsList(byModel)); len(g) > 0 {
+ for _, row := range g {
+ if strings.EqualFold(row.Model, st.CanonicalModel) {
+ cp = row
+ break
+ }
+ }
+ }
+ p.ModelEvidence = &cp
+ }
+ }Compute the grouped list once, outside the post loop, in the real patch. The tests cover the API and static wiring, but not a wall post whose tag is a non-canonical spelling with sibling spellings. That case would have caught this. Findings not on a changed line
Review findings from Delta re-review at 161fc76. The compare lists 68 files, but the PR's own file list is 15 (
Proposed fixes for the two style items: -// verified and failed counts are summed and the two rates recomputed from the
-// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings
-// would. Model becomes the canonical id; RawModels lists each spelling with
+// verified and failed counts are summed and the two rates recomputed from the
+// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a
+// merged row says exactly what one row over all its spellings would. Model
+// becomes the canonical id; RawModels lists each spelling withFor the counters, drop Findings not on a changed line
Review follow-ups: 1 open of 1 (below
|
There was a problem hiding this comment.
Delta re-review at 161fc76 (the PR's own diff is 15 files; the compare against 4a4383d is mostly base churn, so I read the PR files).
Previous findings:
- [correctness] wall evidence names canonical model but shows single-spelling stats: still present.
contribute_wall.go:299-303still looks upbyModel[strings.ToLower(p.Tags["model"])], which is keyed by one raw spelling.contribute_landing.go:6084-6086now labels that row "model evidence for claude-opus-5.5". For a post taggedclaude-opus-5-5, the 3 runs of that spelling are shown under the canonical name while theanthropic/claude-opus-5-5run is left out. The same numbers disagree withgroup=canonical(4 runs, per the PR's own test). - [style] duplicate run counters: still present.
contribute_wall.go:694copiesa.verified/a.failedintoverifiedRuns/failedRuns. - [style] doc comment mis-wrapped: still present.
contribute_wall_canonical.go:~27, the "(#18721) are summed too, so a merged row says exactly what one row over all its spellings" line is overlong.
Requested change (in-scope): make the wall evidence use the canonical grouping so the label and the numbers agree. Group the stats once and look up by canonical id:
- if model := p.Tags["model"]; model != "" {
- if st, ok := byModel[strings.ToLower(model)]; ok {
+ if model := p.Tags["model"]; model != "" {
+ if st, ok := byCanonical[strings.ToLower(modelfamily.Canonical(model))]; ok {where byCanonical is built from groupWallModelStatsByCanonical(wallStatsList(byModel)) keyed by lowercased Model. Add a test: a post tagged with one spelling, runs logged under two, evidence runs equals the sum. Alternatively, drop the canonical label from the evidence line.
— hive: agent=adjudicator backend=claude model=claude-sonnet-5-5 claude=2.1.326
| var evName=(evCanon&&evCanon!=='unknown'&&evCanon.toLowerCase()!==evRaw.toLowerCase())?' for <span class="wall-canonical-model" title="Reported as '+esc(evRaw)+'">'+esc(evCanon)+'</span>':''; | ||
| var failTip=ccModelEvidenceFailTip(p.model_evidence); | ||
| evidence='<div class="ops-note">model evidence: '+(p.model_evidence.runs||0)+' runs · '+Math.round((p.model_evidence.verified_pr_share||0)*100)+'%% verified PR · <span'+(failTip?' title="'+esc(failTip)+'"':'')+'>'+Math.round((p.model_evidence.failure_rate||0)*100)+'%% failed</span></div>'; | ||
| evidence='<div class="ops-note">model evidence'+evName+': '+(p.model_evidence.runs||0)+' runs · '+Math.round((p.model_evidence.verified_pr_share||0)*100)+'%% verified PR · <span'+(failTip?' title="'+esc(failTip)+'"':'')+'>'+Math.round((p.model_evidence.failure_rate||0)*100)+'%% failed</span></div>'; |
There was a problem hiding this comment.
Still present: this names the canonical model, but model_evidence comes from contribute_wall.go:299-303, which looks up one raw spelling's stats. The evidence line therefore undercounts runs relative to the canonical id it displays. Build the evidence lookup from groupWallModelStatsByCanonical keyed by canonical id (or drop the canonical label).
Summary
The same model shows up under several spellings (
claude-opus-5-5,anthropic/claude-opus-5-5,claude-opus-5.5). PRs by model already folds them into one id; spend by model and contributor run stats by model did not, so the three views could not be lined up.This adds
canonical_modelnext to the raw name in both responses, an opt-ingroup=canonicalfor run stats, and shows the canonical id on the Cost card and the contributor wall with the raw spelling in a tooltip. Nothing existing is renamed or regrouped by default.Refs #18718
Part of #18723
What this changes
/api/cost: everyestimated.by_model[]row gainscanonical_model. Rows stay one per raw model (prices are per raw model).by_agent[]rows get no field./api/contribute/run-stats/modelsand themodelsarray in/api/contribute/run-stats: every row gainscanonical_model.group=canonical. Rows are merged by canonical id:runssummed, rates recomputed from the summed counts,modelset to the canonical id, andraw_models[]lists each spelling with its own runs (most runs first). Withoutgroup, the rows are what they were before, plus the new field.Why
So "runs, PRs and spend for Opus 5.5" can be read without re-implementing the folding rule. All three views now use the same function, so a run-stats row's
canonical_modelis exactly themodelof the matching PRs-by-model bucket.How it works
Example, run stats over 7 days:
claude-opus-5-5(37 runs) andanthropic/claude-opus-5-5(21 runs) are two rows.?group=canonical: one row,model: claude-opus-5.5,runs: 58,raw_models: [{claude-opus-5-5, 37}, {anthropic/claude-opus-5-5, 21}], rates computed from the 58 runs.The run-stats rows keep their verified and failed counts internally (not serialized), so the merged rates are exact ratios, not averages of rates.
Files
src/pkg/dashboard/contribute_wall.go:WallModelStatsgainsCanonicalModel,RawModelsand internal counts;readTaskRunModelStatsfills them;handleContributeRunStatsByModelhonoursgroup.src/pkg/dashboard/contribute_wall_canonical.go(new):WallRawModelRuns,runStatsGroupCanonical,groupWallModelStatsByCanonical.src/pkg/dashboard/task_run_log.go:handleContributeRunStatshonoursgroup.src/pkg/dashboard/cost.go:costModelEntrygainsCanonicalModel;flattenEstimatedsets it on by-model rows.src/pkg/dashboard/static-src/js/cost.js+ generatedstatic/index.html: by-model cell label and tooltip.src/pkg/dashboard/contribute_landing.go:ccWallPostHTMLevidence line.dashboard/openapi.json,src/docs/api-reference/contribute.md,src/docs/api-reference/cost-tokens-telemetry.md,src/docs/backend-smoke.md: document the field andgroup.changelog.d/added-18718-canonical-model-spend-runstats.md.Tests
contribute_wall_canonical_test.go:claude-opus-5-5,anthropic/claude-opus-5-5and two Ornith cases. Checkscanonical_modelon every row, ungrouped rows unchanged, grouped runs equal the ungrouped total, grouped rates equal the summed-count ratios,raw_modelsorder and counts, andgroupon/api/contribute/run-statstoo.cost_pure_functions_test.go:canonical_modelon everyby_modelrow, rows not regrouped, none onby_agent.dashboard/test/canonical-model-ui.test.js: runs the Cost card label/tooltip helpers and checks the wall evidence wiring and the static-src / index.html match.CI: this PR's own v2 Tests and v2 CI checks. Only
gofmt -lwas run locally, as AGENTS.md requires.🤖 Generated with Claude Code
https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t