Skip to content

✨ Evidence summary: change failures (reverts, reopened issues) per model - #18783

Merged
hivecommons-hive[bot] merged 5 commits into
hivecommons:v5from
Danathar:feat/evidence-change-failures-per-model
Nov 17, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
hivecommons:v5from
Danathar:feat/evidence-change-failures-per-model

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

The Evidence summary now breaks change failures down by individual model, not just by model family. GET /api/evidence/summary gains outcomes_by_author_model, and the Evidence card's Outcome by author kind panel shows it as a small table. Until now claude-opus-5.5 and claude-sonnet-5.5 shared one claude bucket, so a model whose merged PRs kept getting reverted looked just like one whose work stuck. Nothing new is detected: the existing revert and re-open detection (#13241) is reused, and existing fields are unchanged.

Refs #18720
Part of #18723

What this changes

  • API: new outcomes_by_author_model in /api/evidence/summary. It has the same per-bucket fields as outcomes_by_author_kind (n, merged, closed, open, unknown, reverted_14d, issue_reopened_14d, change_failures, change_failure_rate).
  • Dashboard: under the author-kind bars, a table with model, n, merged, reverted, re-opened and change-failure rate. The models with the most PRs come first, and human / unknown come last. A note says n covers only PRs with an evidence record, so it can differ from the PRs-by-model card.

Why

The PRs-by-model card ranks models without any signal about whether their merged work later got undone. The Evidence pipeline already finds those undos, but before this change it only grouped them by author kind and by model family.

How it works

  • Keys are the canonical model ids that /api/governor/pr-models uses, so different spellings of one model share a key: claude-opus-5-5, anthropic/claude-opus-5-5 and claude-opus-5.5 all count as claude-opus-5.5.
  • A person's PR goes under human, and an agent PR with no recorded model goes under unknown, the same way the family map buckets them. Every PR lands in exactly one model key and one family key, so the two maps add up to the same totals.
  • The map is filled in the same loop as outcomes_by_author_family and its rates are finished the same way.

Example (one row): claude-opus-5.5 · n=3 · merged 2 · reverted 1 · re-opened 0 · 50%.

Files

  • src/pkg/dashboard/api_evidence_summary.go: EvidenceSummary.OutcomesByAuthorModel, evidenceAuthorModel, and the fill and finish steps in buildEvidenceSummaryWindowHints.
  • src/pkg/dashboard/static-src/js/evidence.js + regenerated static/index.html: evOutcomesByModelHtml, called from renderEvidenceSummary.
  • src/pkg/dashboard/api_evidence_family_test.go: TestBuildEvidenceSummaryAuthorModelBuckets, TestEvidenceAuthorModel.
  • src/pkg/dashboard/evidence_model_outcomes.test.js: the new table (rows, order, escaping, empty or absent map).
  • src/pkg/dashboard/evidence_card_ui_test.go: pins the new call and adds the function to the render harness.
  • dashboard/openapi.json, src/docs/review-evidence.md, src/docs/api-reference/agents-and-controls.md: the new field documented.
  • changelog.d/added-18720-evidence-change-failures-per-model.md.

Tests

  • Go: two models from the same family, one reverted within 14 days and one with a re-opened issue, plus a GPT model, a human PR and an agent PR with no model. The test checks per-model counts, field-by-field sums against the family map, canonical folding of the raw spellings, and that the family and author-kind maps are unchanged.
  • JS (runs under node --test src/pkg/dashboard/*.test.js in v2 CI): rows, ordering, the dash for no merges, HTML escaping, and empty output for an older server.
  • CI: this PR's own v2 Tests and v2 CI checks. Only gofmt -l was run locally, as AGENTS.md requires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t

GET /api/evidence/summary adds outcomes_by_author_model: the same
EvidenceOutcomes as outcomes_by_author_kind, keyed by
modelfamily.Canonical(Author.Model) (the ids /api/governor/pr-models
uses), with human / unknown bucketed as evidenceAuthorFamily does. It is
filled in the same loop as outcomes_by_author_family and finished the
same way; the revert and re-open detection is reused unchanged.

The Evidence card's Outcome by author kind panel renders it as a compact
table (model, n, merged, reverted, re-opened, change-failure rate).

Refs hivecommons#18720

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nicholasjackson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18780 ✨ Governor PRs by model: what each model's PRs cost the hive to review same files src/docs/api-reference/agents-and-controls.md
src/pkg/dashboard/static/index.html
#18782 ✨ Governor PRs by model: split closed-unmerged PRs into superseded, duplicate and other same files src/pkg/dashboard/static/index.html
#18829 ✨ dashboard: merge banner shows when it merged; Time format option same files src/pkg/dashboard/static/index.html

…ed field

TestBuildEvidenceSummaryAuthorModelBuckets assumed the family map folds
provider-prefixed ids ("anthropic/claude-opus-5-5") into "claude". It does
not: modelfamily.Family takes the text before the first dash without
stripping a provider prefix, so those PRs are under "other" in
outcomes_by_author_family. That is pre-existing behaviour this PR must
leave unchanged, so the test now pins it; the per-model map (canonical
ids) was already correct. Also removes the unused `failures` field
golangci-lint flagged.

Refs hivecommons#18720

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
hivecommons-hive Bot pushed a commit that referenced this pull request Nov 17, 2026
Family took the text before the first dash of the normalized id without
dropping a provider or gateway path, so "anthropic/claude-opus-5-5" read
as family "anthropic/claude" (bucket "other") and
"openai-codex/gpt-6.1-terra:high" as "openai". Canonical already strips
that path and the ":variant" suffix; Family now reads the family from the
Canonical id. SameFamily, Bucket and review_models.exclude_author_family
follow, so a prefixed and a bare spelling of the same vendor are one
family.

Found while working #18720 (#18783).

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
@hivecommons-hive

hivecommons-hive Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • server-tests

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Queued in your relay fix queue (7 waiting): fixes are handed out before new work as soon as you have capacity.

…vecommons#18803)

v5's hivecommons#18803 derives modelfamily.Family from the canonical id, so the
prefixed Claude PR joins the claude family and the openai-codex PR is
gpt; update the family expectations and their comment to match.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Picks up the governor-pr-models-ui test fix so server-tests runs
against a base whose test file compiles.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delta re-review at cf77aaf (since ecb7a40): no prior findings to re-check. Style-wise the delta is clean and follows repo idioms (changelog fragments, #NNNN refs in comments, openapi + docs updated alongside the new group= and model= params). Two low-severity maintainability notes, neither blocking:

  1. governorPRModelPRList (src/pkg/dashboard/api_governor_pr_models_list.go:56-63, 70-74) copies the window-cutoff switch and the HiveAttributed/CreatedAt filter from aggregateGovernorPRModels (api_governor_pr_models.go:293-317 at head). Its doc comment says the list length must match the bucket's prs count, so the two copies have to stay in lockstep. A small shared helper (e.g. governorPRModelsCutoff(window, now) plus an included(pr) predicate) would keep that invariant in one place.
  2. src/pkg/dashboard/contribute_wall_canonical.go:27 is a run-on comment line mixing two sentences mid-paragraph; re-wrap it.

Findings not on a changed line

  • src/pkg/dashboard/api_governor_pr_models_list.go:56: This cutoff switch and the attribution/created_at filter below duplicate aggregateGovernorPRModels. The doc comment promises the list length matches the bucket's prs count, so extract a shared helper to keep them from drifting.
  • src/pkg/dashboard/contribute_wall_canonical.go:27: Run-on comment line; re-wrap so the abandoned, scenarios and failure_kinds sentence sits on normal-width lines.
    // sums; abandoned, scenarios and failure_kinds (#18721) are summed too, so a
    // merged row says exactly what one row over all its spellings would. Model
    // becomes the canonical id; RawModels lists each spelling with
    

— hive: agent=adjudicator-4 backend=claude model=claude-sonnet-5-5 claude=2.1.326

@hivecommons-hive
hivecommons-hive Bot merged commit e198726 into hivecommons:v5 Nov 17, 2026
75 of 76 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#18783cf77aaf` sha256:432f7d815866fb48060a8e5123b2118076b749534d0120d8eba213bc33df06d8 — download from the dashboard or `hivectl review evidence #18783`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant