Repository navigation
✨ Evidence summary: change failures (reverts, reopened issues) per model - #18783
hivecommons-hive[bot] merged 5 commits into
Conversation
GET /api/evidence/summary adds outcomes_by_author_model: the same EvidenceOutcomes as outcomes_by_author_kind, keyed by modelfamily.Canonical(Author.Model) (the ids /api/governor/pr-models uses), with human / unknown bucketed as evidenceAuthorFamily does. It is filled in the same loop as outcomes_by_author_family and finished the same way; the revert and re-open detection is reused unchanged. The Evidence card's Outcome by author kind panel renders it as a compact table (model, n, merged, reverted, re-opened, change-failure rate). Refs hivecommons#18720 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
…ed field
TestBuildEvidenceSummaryAuthorModelBuckets assumed the family map folds
provider-prefixed ids ("anthropic/claude-opus-5-5") into "claude". It does
not: modelfamily.Family takes the text before the first dash without
stripping a provider prefix, so those PRs are under "other" in
outcomes_by_author_family. That is pre-existing behaviour this PR must
leave unchanged, so the test now pins it; the per-model map (canonical
ids) was already correct. Also removes the unused `failures` field
golangci-lint flagged.
Refs hivecommons#18720
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Family took the text before the first dash of the normalized id without dropping a provider or gateway path, so "anthropic/claude-opus-5-5" read as family "anthropic/claude" (bucket "other") and "openai-codex/gpt-6.1-terra:high" as "openai". Canonical already strips that path and the ":variant" suffix; Family now reads the family from the Canonical id. SameFamily, Bucket and review_models.exclude_author_family follow, so a prefixed and a bare spelling of the same vendor are one family. Found while working #18720 (#18783). Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
|
@Danathar this PR is back in your court ( Failing checks:
Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path. Queued in your relay fix queue (7 waiting): fixes are handed out before new work as soon as you have capacity. |
…vecommons#18803) v5's hivecommons#18803 derives modelfamily.Family from the canonical id, so the prefixed Claude PR joins the claude family and the openai-codex PR is gpt; update the family expectations and their comment to match. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Picks up the governor-pr-models-ui test fix so server-tests runs against a base whose test file compiles. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
There was a problem hiding this comment.
Delta re-review at cf77aaf (since ecb7a40): no prior findings to re-check. Style-wise the delta is clean and follows repo idioms (changelog fragments, #NNNN refs in comments, openapi + docs updated alongside the new group= and model= params). Two low-severity maintainability notes, neither blocking:
governorPRModelPRList(src/pkg/dashboard/api_governor_pr_models_list.go:56-63, 70-74) copies the window-cutoff switch and theHiveAttributed/CreatedAtfilter fromaggregateGovernorPRModels(api_governor_pr_models.go:293-317 at head). Its doc comment says the list length must match the bucket'sprscount, so the two copies have to stay in lockstep. A small shared helper (e.g.governorPRModelsCutoff(window, now)plus anincluded(pr)predicate) would keep that invariant in one place.- src/pkg/dashboard/contribute_wall_canonical.go:27 is a run-on comment line mixing two sentences mid-paragraph; re-wrap it.
Findings not on a changed line
src/pkg/dashboard/api_governor_pr_models_list.go:56: This cutoff switch and the attribution/created_at filter below duplicateaggregateGovernorPRModels. The doc comment promises the list length matches the bucket's prs count, so extract a shared helper to keep them from drifting.src/pkg/dashboard/contribute_wall_canonical.go:27: Run-on comment line; re-wrap so theabandoned, scenarios and failure_kindssentence sits on normal-width lines.// sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a // merged row says exactly what one row over all its spellings would. Model // becomes the canonical id; RawModels lists each spelling with
— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.326
The Evidence summary now breaks change failures down by individual model, not just by model family.
GET /api/evidence/summarygainsoutcomes_by_author_model, and the Evidence card's Outcome by author kind panel shows it as a small table. Until nowclaude-opus-5.5andclaude-sonnet-5.5shared oneclaudebucket, so a model whose merged PRs kept getting reverted looked just like one whose work stuck. Nothing new is detected: the existing revert and re-open detection (#13241) is reused, and existing fields are unchanged.Refs #18720
Part of #18723
What this changes
outcomes_by_author_modelin/api/evidence/summary. It has the same per-bucket fields asoutcomes_by_author_kind(n,merged,closed,open,unknown,reverted_14d,issue_reopened_14d,change_failures,change_failure_rate).n, merged, reverted, re-opened and change-failure rate. The models with the most PRs come first, andhuman/unknowncome last. A note saysncovers only PRs with an evidence record, so it can differ from the PRs-by-model card.Why
The PRs-by-model card ranks models without any signal about whether their merged work later got undone. The Evidence pipeline already finds those undos, but before this change it only grouped them by author kind and by model family.
How it works
/api/governor/pr-modelsuses, so different spellings of one model share a key:claude-opus-5-5,anthropic/claude-opus-5-5andclaude-opus-5.5all count asclaude-opus-5.5.human, and an agent PR with no recorded model goes underunknown, the same way the family map buckets them. Every PR lands in exactly one model key and one family key, so the two maps add up to the same totals.outcomes_by_author_familyand its rates are finished the same way.Example (one row):
claude-opus-5.5 · n=3 · merged 2 · reverted 1 · re-opened 0 · 50%.Files
src/pkg/dashboard/api_evidence_summary.go:EvidenceSummary.OutcomesByAuthorModel,evidenceAuthorModel, and the fill and finish steps inbuildEvidenceSummaryWindowHints.src/pkg/dashboard/static-src/js/evidence.js+ regeneratedstatic/index.html:evOutcomesByModelHtml, called fromrenderEvidenceSummary.src/pkg/dashboard/api_evidence_family_test.go:TestBuildEvidenceSummaryAuthorModelBuckets,TestEvidenceAuthorModel.src/pkg/dashboard/evidence_model_outcomes.test.js: the new table (rows, order, escaping, empty or absent map).src/pkg/dashboard/evidence_card_ui_test.go: pins the new call and adds the function to the render harness.dashboard/openapi.json,src/docs/review-evidence.md,src/docs/api-reference/agents-and-controls.md: the new field documented.changelog.d/added-18720-evidence-change-failures-per-model.md.Tests
node --test src/pkg/dashboard/*.test.jsin v2 CI): rows, ordering, the dash for no merges, HTML escaping, and empty output for an older server.gofmt -lwas run locally, as AGENTS.md requires.🤖 Generated with Claude Code
https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t