Repository navigation
🐛 modelfamily: read the vendor family from the provider-stripped model id - #18803
Merged
hivecommons-hive[bot] merged 1 commit intoNov 17, 2026
Conversation
Family took the text before the first dash of the normalized id without dropping a provider or gateway path, so "anthropic/claude-opus-5-5" read as family "anthropic/claude" (bucket "other") and "openai-codex/gpt-6.1-terra:high" as "openai". Canonical already strips that path and the ":variant" suffix; Family now reads the family from the Canonical id. SameFamily, Bucket and review_models.exclude_author_family follow, so a prefixed and a bare spelling of the same vendor are one family. Found while working hivecommons#18720 (hivecommons#18783). Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Contributor
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
Contributor
Danathar
added a commit
to Danathar/hive
that referenced
this pull request
Nov 17, 2026
…vecommons#18803) v5's hivecommons#18803 derives modelfamily.Family from the canonical id, so the prefixed Claude PR joins the claude family and the openai-codex PR is gpt; update the family expectations and their comment to match. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
hivecommons-hive Bot
pushed a commit
that referenced
this pull request
Nov 17, 2026
…del (#18783) * ✨ Evidence summary: change failures per author model GET /api/evidence/summary adds outcomes_by_author_model: the same EvidenceOutcomes as outcomes_by_author_kind, keyed by modelfamily.Canonical(Author.Model) (the ids /api/governor/pr-models uses), with human / unknown bucketed as evidenceAuthorFamily does. It is filled in the same loop as outcomes_by_author_family and finished the same way; the revert and re-open detection is reused unchanged. The Evidence card's Outcome by author kind panel renders it as a compact table (model, n, merged, reverted, re-opened, change-failure rate). Refs #18720 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> * 🐛 Evidence model test: match the existing family bucketing; drop unused field TestBuildEvidenceSummaryAuthorModelBuckets assumed the family map folds provider-prefixed ids ("anthropic/claude-opus-5-5") into "claude". It does not: modelfamily.Family takes the text before the first dash without stripping a provider prefix, so those PRs are under "other" in outcomes_by_author_family. That is pre-existing behaviour this PR must leave unchanged, so the test now pins it; the per-model map (canonical ids) was already correct. Also removes the unused `failures` field golangci-lint flagged. Refs #18720 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> * test: evidence family buckets follow the provider-stripped Family (#18803) v5's #18803 derives modelfamily.Family from the canonical id, so the prefixed Claude PR joins the claude family and the openai-codex PR is gpt; update the family expectations and their comment to match. Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A model id that carries a provider or gateway path gets the wrong vendor family today.
anthropic/claude-opus-5-5reads as familyanthropic/claude(so it is counted under "other"), andopenai-codex/gpt-6.1-terra:highreads asopenai. This PR reads the family from the same cleaned-up id the per-model stats already use.What changes for operators: per-family counts on the Evidence page and the reviewer-accuracy card move from "other" to the right vendor. Cross-family reviewer routing (
review_models.exclude_author_family) now treats a prefixed and a bare spelling of the same vendor as the same family, so a "cross-family" review can no longer go to a model from the same vendor as the author.Found while working #18720 (#18783). There is no issue for this bug; it is described here.
What this changes
anthropic/claude-opus-5-5→ familyclaude, bucketclaude(wasanthropic/claude, bucketother).openai-codex/gpt-6.1-terra:high→ familygpt(wasopenai, bucketother).claude-opus-5-5orgpt-6.1-solare unchanged.Example: the author is
anthropic/claude-opus-5-5, the review pool is[claude-sonnet-5, gpt-5.6-terra], andexclude_author_family: true. Before, the families differed (anthropic/claudevsclaude), so the Claude reviewer was picked as "cross-family". Now it is skipped andgpt-5.6-terrareviews.Why
The family check took the text before the first dash of the normalized id. Normalizing lowercases and trims, but it does not drop a provider path. The canonical-id function next to it already drops the path and a
:variantsuffix, so per-model stats folded these spellings while per-family stats did not.How it works
The family is now the text before the first dash of the canonical id. That id has the provider path and
:variantsuffix removed. Its other folds (digits joined with dots, date stamps dropped) only touch later segments, so they never change the family. A value that is empty after cleanup (for exampleanthropic/or-) reads as unknown, like an absent model.Effects on each place that uses the family:
exclude_author_family): prefixed same-vendor pairs are now excluded. With the option off, nothing changes.same_familystamped on new evidence verdicts: prefixed same-vendor pairs are now stamped "same". Verdicts stored before this change keep their old stamp, and the summary reads the stored stamp first, so older records are not reclassified.claude/gptinstead ofanthropic/claude/openai, so the hub's family-mix reading gets more accurate and needs fewer distinct values.No Prometheus metric uses the family.
Files
src/pkg/modelfamily/modelfamily.go:Familyreads fromCanonical.src/pkg/modelfamily/modelfamily_test.go: new table tests.src/docs/agent-configuration.md:exclude_author_familydescription mentions the stripped path.changelog.d/fixed-modelfamily-family-provider-prefix.mdTests
TestFamilyIgnoresProviderPrefixAndVariant: prefixed and bare spellings of claude, gpt, gemini and other vendors give the same family and bucket. Values that are empty after cleanup read as unknown.TestSameFamilyAcrossProviderSpellings: a prefixed and a bare same-vendor pair is the same family; a prefixed Claude/GPT pair is still cross-family.CI: this PR's own checks. Only gofmt -l was run locally, as AGENTS.md requires.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t