Skip to content

✨ Spend and contributor run stats by model: add the canonical model name - #18781

Merged
hivecommons-hive[bot] merged 4 commits into
hivecommons:v5from
Danathar:feat/canonical-model-spend-runstats
Nov 17, 2026
Merged

hivecommons-hive[bot] merged 4 commits into
hivecommons:v5from
Danathar:feat/canonical-model-spend-runstats

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

Summary

The same model shows up under several spellings (claude-opus-5-5, anthropic/claude-opus-5-5, claude-opus-5.5). PRs by model already folds them into one id; spend by model and contributor run stats by model did not, so the three views could not be lined up.

This adds canonical_model next to the raw name in both responses, an opt-in group=canonical for run stats, and shows the canonical id on the Cost card and the contributor wall with the raw spelling in a tooltip. Nothing existing is renamed or regrouped by default.

Refs #18718
Part of #18723

What this changes

  • /api/cost: every estimated.by_model[] row gains canonical_model. Rows stay one per raw model (prices are per raw model). by_agent[] rows get no field.
  • /api/contribute/run-stats/models and the models array in /api/contribute/run-stats: every row gains canonical_model.
  • Both run-stats endpoints accept group=canonical. Rows are merged by canonical id: runs summed, rates recomputed from the summed counts, model set to the canonical id, and raw_models[] lists each spelling with its own runs (most runs first). Without group, the rows are what they were before, plus the new field.
  • Cost card, by-model table: shows the canonical id; hovering shows "Reported as ". When the canonical id equals the raw name, the cell is unchanged.
  • Contributor wall, model evidence line: names the canonical id when it differs from the post's model tag, raw spelling in the tooltip.

Why

So "runs, PRs and spend for Opus 5.5" can be read without re-implementing the folding rule. All three views now use the same function, so a run-stats row's canonical_model is exactly the model of the matching PRs-by-model bucket.

How it works

Example, run stats over 7 days:

  • Before: claude-opus-5-5 (37 runs) and anthropic/claude-opus-5-5 (21 runs) are two rows.
  • With ?group=canonical: one row, model: claude-opus-5.5, runs: 58, raw_models: [{claude-opus-5-5, 37}, {anthropic/claude-opus-5-5, 21}], rates computed from the 58 runs.

The run-stats rows keep their verified and failed counts internally (not serialized), so the merged rates are exact ratios, not averages of rates.

Files

  • src/pkg/dashboard/contribute_wall.go: WallModelStats gains CanonicalModel, RawModels and internal counts; readTaskRunModelStats fills them; handleContributeRunStatsByModel honours group.
  • src/pkg/dashboard/contribute_wall_canonical.go (new): WallRawModelRuns, runStatsGroupCanonical, groupWallModelStatsByCanonical.
  • src/pkg/dashboard/task_run_log.go: handleContributeRunStats honours group.
  • src/pkg/dashboard/cost.go: costModelEntry gains CanonicalModel; flattenEstimated sets it on by-model rows.
  • src/pkg/dashboard/static-src/js/cost.js + generated static/index.html: by-model cell label and tooltip.
  • src/pkg/dashboard/contribute_landing.go: ccWallPostHTML evidence line.
  • dashboard/openapi.json, src/docs/api-reference/contribute.md, src/docs/api-reference/cost-tokens-telemetry.md, src/docs/backend-smoke.md: document the field and group.
  • changelog.d/added-18718-canonical-model-spend-runstats.md.

Tests

  • contribute_wall_canonical_test.go: claude-opus-5-5, anthropic/claude-opus-5-5 and two Ornith cases. Checks canonical_model on every row, ungrouped rows unchanged, grouped runs equal the ungrouped total, grouped rates equal the summed-count ratios, raw_models order and counts, and group on /api/contribute/run-stats too.
  • cost_pure_functions_test.go: canonical_model on every by_model row, rows not regrouped, none on by_agent.
  • dashboard/test/canonical-model-ui.test.js: runs the Cost card label/tooltip helpers and checks the wall evidence wiring and the static-src / index.html match.

CI: this PR's own v2 Tests and v2 CI checks. Only gofmt -l was run locally, as AGENTS.md requires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t

@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18782 ✨ Governor PRs by model: split closed-unmerged PRs into superseded, duplicate and other same files src/pkg/dashboard/static/index.html
#18780 ✨ Governor PRs by model: what each model's PRs cost the hive to review same files src/pkg/dashboard/static/index.html
#18783 ✨ Evidence summary: change failures (reverts, reopened issues) per model same files dashboard/openapi.json
src/pkg/dashboard/static/index.html

@hivecommons-hive

hivecommons-hive Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • server-tests

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (7 waiting): fixes are handed out before new work as soon as you have capacity.

@github-actions github-actions Bot removed the conflicts-with-open-pr Open PR has a merge conflict with another open PR label Nov 17, 2026
Danathar and others added 2 commits November 17, 2026 20:32
/api/cost estimated.by_model[] and every /api/contribute/run-stats(/models)
row now carry canonical_model (modelfamily.Canonical), the id the Governor's
PRs-by-model card groups by. Both run-stats endpoints accept group=canonical,
which merges spellings (runs summed, rates from summed counts) and lists
raw_models[]. Without it, existing rows are unchanged. The Cost card and the
contributor wall's model evidence show the canonical id with the raw
spelling in a tooltip.

Refs hivecommons#18718

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
…mmons#18721) too

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@Danathar
Danathar force-pushed the feat/canonical-model-spend-runstats branch from 4974911 to 4a4383d Compare November 17, 2026 20:33
@kubestellar-prow kubestellar-prow Bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 17, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style review at 4a4383d: conventions followed (static-src and index.html in sync, openapi, docs and changelog updated, tests added). Two minor, non-blocking nits are inline: a duplicated counter pair and a mis-wrapped doc comment. Approving from the style perspective.

— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.326


// verifiedRuns and failedRuns are the counts behind the two rates, kept
// so group=canonical can recompute the rates exactly.
verifiedRuns, failedRuns int

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

verifiedRuns, failedRuns duplicate the aggregator's existing verified/failed (copied at the a.verifiedRuns, a.failedRuns = a.verified, a.failed line in readTaskRunModelStats). Two names for one count on an exported struct is a maintenance trap. Consider exposing the existing counters to the grouping code instead of mirroring them.

// groupWallModelStatsByCanonical merges run-stats rows that share a canonical
// model id (modelfamily.Canonical, the rule PRs-by-model uses). Runs and the
// verified and failed counts are summed and the two rates recomputed from the
// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doc-comment line was spliced into one very long line and breaks the wrapping used in the rest of the block. Please reflow it:

Suggested change
// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings
// verified and failed counts are summed and the two rates recomputed from the
// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a
// merged row says exactly what one row over all its spellings would. Model
// becomes the canonical id; RawModels lists each spelling with its own runs,
// most runs first. Output is sorted by model, like wallStatsList.

@hivecommons-hive

hivecommons-hive Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Review findings from adjudicator (folded here because this head already carries its hive review):

One in-scope correctness issue in the wall evidence line. The Go grouping logic itself (contribute_wall_canonical.go) reads correctly: counts are summed, rates are recomputed from the sums, and raw_models order is deterministic.

Wall evidence line labels single-spelling stats with the canonical model id (contribute_landing.go:6084, fed by contribute_wall.go:300-303).
model_evidence is looked up by the post's raw tag (byModel[strings.ToLower(model)]). It therefore holds only that one spelling's runs and rates. The PR now renders it as model evidence for claude-opus-5.5: 21 runs .... In the PR's own example, claude-opus-5.5 has 58 runs, so a post tagged anthropic/claude-opus-5-5 would show 21 runs under the canonical name. The label implies the canonical total, the same figure group=canonical and the PRs-by-model card report, so the three views would disagree by exactly the amount this PR set out to reconcile.

Fix: either aggregate by canonical id when attaching model_evidence, or drop the "for " wording and keep the canonical id only in the tooltip. The first option is the smaller change:

-		if model := p.Tags["model"]; model != "" {
-			if st, ok := byModel[strings.ToLower(model)]; ok {
-				cp := st
-				p.ModelEvidence = &cp
-			}
-		}
+		if model := p.Tags["model"]; model != "" {
+			if st, ok := byModel[strings.ToLower(model)]; ok {
+				cp := st
+				if g := groupWallModelStatsByCanonical(wallStatsList(byModel)); len(g) > 0 {
+					for _, row := range g {
+						if strings.EqualFold(row.Model, st.CanonicalModel) {
+							cp = row
+							break
+						}
+					}
+				}
+				p.ModelEvidence = &cp
+			}
+		}

Compute the grouped list once, outside the post loop, in the real patch.

The tests cover the API and static wiring, but not a wall post whose tag is a non-canonical spelling with sibling spellings. That case would have caught this.

Findings not on a changed line

  • src/pkg/dashboard/contribute_landing.go:6084: model_evidence here holds stats for the post's single raw spelling only (looked up by raw tag in contribute_wall.go:300-303), but this line now labels it with the canonical id. A post tagged anthropic/claude-opus-5-5 would read "model evidence for claude-opus-5.5: 21 runs" while group=canonical reports 58. Aggregate by canonical id when attaching model_evidence, or drop the "for " wording.

Review findings from adjudicator-4 (folded here because this head already carries its hive review):

Delta re-review at 161fc76. The compare lists 68 files, but the PR's own file list is 15 (gh pr view). The rest is base drift, so I judged only the PR's files. None of the three earlier notes was addressed:

  • correctness, contribute_landing.go:6084 (wall evidence names the canonical model but shows stats for one spelling): still present. The evidence is looked up by the raw tag, byModel[strings.ToLower(model)] at contribute_wall.go:299. It is labelled with the canonical name, so a reader assumes the numbers cover every spelling of that model.
  • style, contribute_wall.go:97 (duplicate run counters): still present. agg still carries verified, failed (contribute_wall.go:654) next to the new verifiedRuns, failedRuns on WallModelStats (line 97). They are copied across at line 694.
  • style, contribute_wall_canonical.go:27 (doc comment mis-wrapped): still present. Line 27 is one overlong line.

Proposed fixes for the two style items:

-// verified and failed counts are summed and the two rates recomputed from the
-// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a merged row says exactly what one row over all its spellings
-// would. Model becomes the canonical id; RawModels lists each spelling with
+// verified and failed counts are summed and the two rates recomputed from the
+// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a
+// merged row says exactly what one row over all its spellings would. Model
+// becomes the canonical id; RawModels lists each spelling with

For the counters, drop verified, failed from agg and increment a.verifiedRuns and a.failedRuns directly in the loop. Then delete the copy at line 694 and compute the rates from those two fields.

Findings not on a changed line

  • src/pkg/dashboard/contribute_wall_canonical.go:27: Rewrap this comment line (it is far longer than its neighbours).
  • src/pkg/dashboard/contribute_wall.go:97: Duplicates agg.verified/agg.failed (line 654); keep one pair and drop the copy at line 694.
Review follow-ups: 1 open of 1 (below review.backlog.min_severity: P2)

Non-blocking findings below the backlog filing line, listed here instead of as issues. Tick them off as they are fixed; set review.backlog.min_severity: P3 to file every finding.

  • P3 low security — Seen-log edit accepts any author's marker comment when the App login is unknown (src/pkg/hub/selfreport_relay.go:737): findHubSelfReportSeenLog only filters by author when botLogin != "". If mintHubFeedbackAppToken returns an empty login, a third party's comment containing would be s…
Evidence bundle `#1878198bd408` sha256:7dd71e1460df0a1760ab467c98db2ca760986682a2b6d855a5f01de6a4227061 — download from the dashboard or `hivectl review evidence #18781`.

merged by Hive trusted-author auto-merge: author Danathar holds role owner and GitHub permission maintain on repo; CI green at 98bd408.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta re-review at 161fc76 (the PR's own diff is 15 files; the compare against 4a4383d is mostly base churn, so I read the PR files).

Previous findings:

  • [correctness] wall evidence names canonical model but shows single-spelling stats: still present. contribute_wall.go:299-303 still looks up byModel[strings.ToLower(p.Tags["model"])], which is keyed by one raw spelling. contribute_landing.go:6084-6086 now labels that row "model evidence for claude-opus-5.5". For a post tagged claude-opus-5-5, the 3 runs of that spelling are shown under the canonical name while the anthropic/claude-opus-5-5 run is left out. The same numbers disagree with group=canonical (4 runs, per the PR's own test).
  • [style] duplicate run counters: still present. contribute_wall.go:694 copies a.verified/a.failed into verifiedRuns/failedRuns.
  • [style] doc comment mis-wrapped: still present. contribute_wall_canonical.go:~27, the "(#18721) are summed too, so a merged row says exactly what one row over all its spellings" line is overlong.

Requested change (in-scope): make the wall evidence use the canonical grouping so the label and the numbers agree. Group the stats once and look up by canonical id:

-		if model := p.Tags["model"]; model != "" {
-			if st, ok := byModel[strings.ToLower(model)]; ok {
+		if model := p.Tags["model"]; model != "" {
+			if st, ok := byCanonical[strings.ToLower(modelfamily.Canonical(model))]; ok {

where byCanonical is built from groupWallModelStatsByCanonical(wallStatsList(byModel)) keyed by lowercased Model. Add a test: a post tagged with one spelling, runs logged under two, evidence runs equals the sum. Alternatively, drop the canonical label from the evidence line.

— hive: agent=adjudicator backend=claude model=claude-sonnet-5-5 claude=2.1.326

var evName=(evCanon&&evCanon!=='unknown'&&evCanon.toLowerCase()!==evRaw.toLowerCase())?' for <span class="wall-canonical-model" title="Reported as '+esc(evRaw)+'">'+esc(evCanon)+'</span>':'';
var failTip=ccModelEvidenceFailTip(p.model_evidence);
evidence='<div class="ops-note">model evidence: '+(p.model_evidence.runs||0)+' runs · '+Math.round((p.model_evidence.verified_pr_share||0)*100)+'%% verified PR · <span'+(failTip?' title="'+esc(failTip)+'"':'')+'>'+Math.round((p.model_evidence.failure_rate||0)*100)+'%% failed</span></div>';
evidence='<div class="ops-note">model evidence'+evName+': '+(p.model_evidence.runs||0)+' runs · '+Math.round((p.model_evidence.verified_pr_share||0)*100)+'%% verified PR · <span'+(failTip?' title="'+esc(failTip)+'"':'')+'>'+Math.round((p.model_evidence.failure_rate||0)*100)+'%% failed</span></div>';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still present: this names the canonical model, but model_evidence comes from contribute_wall.go:299-303, which looks up one raw spelling's stats. The evidence line therefore undercounts runs relative to the canonical id it displays. Build the evidence lookup from groupWallModelStatsByCanonical keyed by canonical id (or drop the canonical label).

@hivecommons-hive
hivecommons-hive Bot merged commit 74f40b3 into hivecommons:v5 Nov 17, 2026
73 of 75 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant