Skip to content

🐛 modelfamily: read the vendor family from the provider-stripped model id - #18803

Merged
hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
Danathar:fix/modelfamily-family-provider-prefix
Nov 17, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
hivecommons:v5from
Danathar:fix/modelfamily-family-provider-prefix

Conversation

@Danathar

Copy link
Copy Markdown
Contributor

A model id that carries a provider or gateway path gets the wrong vendor family today. anthropic/claude-opus-5-5 reads as family anthropic/claude (so it is counted under "other"), and openai-codex/gpt-6.1-terra:high reads as openai. This PR reads the family from the same cleaned-up id the per-model stats already use.

What changes for operators: per-family counts on the Evidence page and the reviewer-accuracy card move from "other" to the right vendor. Cross-family reviewer routing (review_models.exclude_author_family) now treats a prefixed and a bare spelling of the same vendor as the same family, so a "cross-family" review can no longer go to a model from the same vendor as the author.

Found while working #18720 (#18783). There is no issue for this bug; it is described here.

What this changes

  • anthropic/claude-opus-5-5 → family claude, bucket claude (was anthropic/claude, bucket other).
  • openai-codex/gpt-6.1-terra:high → family gpt (was openai, bucket other).
  • Bare ids such as claude-opus-5-5 or gpt-6.1-sol are unchanged.

Example: the author is anthropic/claude-opus-5-5, the review pool is [claude-sonnet-5, gpt-5.6-terra], and exclude_author_family: true. Before, the families differed (anthropic/claude vs claude), so the Claude reviewer was picked as "cross-family". Now it is skipped and gpt-5.6-terra reviews.

Why

The family check took the text before the first dash of the normalized id. Normalizing lowercases and trims, but it does not drop a provider path. The canonical-id function next to it already drops the path and a :variant suffix, so per-model stats folded these spellings while per-family stats did not.

How it works

The family is now the text before the first dash of the canonical id. That id has the provider path and :variant suffix removed. Its other folds (digits joined with dots, date stamps dropped) only touch later segments, so they never change the family. A value that is empty after cleanup (for example anthropic/ or -) reads as unknown, like an absent model.

Effects on each place that uses the family:

  • Review-model routing (exclude_author_family): prefixed same-vendor pairs are now excluded. With the option off, nothing changes.
  • Evidence summary, by-family outcomes and per-model by-author-family: authors with prefixed ids move out of "other". The set of bucket names is unchanged.
  • Reviewer-accuracy card, author families and the same/cross split: same as above.
  • same_family stamped on new evidence verdicts: prefixed same-vendor pairs are now stamped "same". Verdicts stored before this change keep their old stamp, and the summary reads the stored stamp first, so older records are not reclassified.
  • Hub heartbeat family lists: prefixed ids now report claude / gpt instead of anthropic/claude / openai, so the hub's family-mix reading gets more accurate and needs fewer distinct values.

No Prometheus metric uses the family.

Files

  • src/pkg/modelfamily/modelfamily.go: Family reads from Canonical.
  • src/pkg/modelfamily/modelfamily_test.go: new table tests.
  • src/docs/agent-configuration.md: exclude_author_family description mentions the stripped path.
  • changelog.d/fixed-modelfamily-family-provider-prefix.md

Tests

  • TestFamilyIgnoresProviderPrefixAndVariant: prefixed and bare spellings of claude, gpt, gemini and other vendors give the same family and bucket. Values that are empty after cleanup read as unknown.
  • TestSameFamilyAcrossProviderSpellings: a prefixed and a bare same-vendor pair is the same family; a prefixed Claude/GPT pair is still cross-family.
  • No existing test pinned the old values. The current Family/Bucket/SameFamily tests only use bare ids and still pass unchanged.

CI: this PR's own checks. Only gofmt -l was run locally, as AGENTS.md requires.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t

Family took the text before the first dash of the normalized id without
dropping a provider or gateway path, so "anthropic/claude-opus-5-5" read
as family "anthropic/claude" (bucket "other") and
"openai-codex/gpt-6.1-terra:high" as "openai". Canonical already strips
that path and the ":variant" suffix; Family now reads the family from the
Canonical id. SameFamily, Bucket and review_models.exclude_author_family
follow, so a prefixed and a bare spelling of the same vendor are one
family.

Found while working hivecommons#18720 (hivecommons#18783).

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 17, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign kellyaa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

PR Title Relation Files
#18805 🐛 config: adopt review and agent_sandbox from the dashboard overlay on reload same files src/docs/agent-configuration.md

@hivecommons-hive
hivecommons-hive Bot merged commit d1f3b21 into hivecommons:v5 Nov 17, 2026
82 of 83 checks passed
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#18803302d731` sha256:725697c09e9c195940398f72fa21f3cfb4cec572a2c844c7a3897d6ce38bc0bc — download from the dashboard or `hivectl review evidence #18803`.

Danathar added a commit to Danathar/hive that referenced this pull request Nov 17, 2026
…vecommons#18803)

v5's hivecommons#18803 derives modelfamily.Family from the canonical id, so the
prefixed Claude PR joins the claude family and the openai-codex PR is
gpt; update the family expectations and their comment to match.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
hivecommons-hive Bot pushed a commit that referenced this pull request Nov 17, 2026
…del (#18783)

* ✨ Evidence summary: change failures per author model

GET /api/evidence/summary adds outcomes_by_author_model: the same
EvidenceOutcomes as outcomes_by_author_kind, keyed by
modelfamily.Canonical(Author.Model) (the ids /api/governor/pr-models
uses), with human / unknown bucketed as evidenceAuthorFamily does. It is
filled in the same loop as outcomes_by_author_family and finished the
same way; the revert and re-open detection is reused unchanged.

The Evidence card's Outcome by author kind panel renders it as a compact
table (model, n, merged, reverted, re-opened, change-failure rate).

Refs #18720

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>

* 🐛 Evidence model test: match the existing family bucketing; drop unused field

TestBuildEvidenceSummaryAuthorModelBuckets assumed the family map folds
provider-prefixed ids ("anthropic/claude-opus-5-5") into "claude". It does
not: modelfamily.Family takes the text before the first dash without
stripping a provider prefix, so those PRs are under "other" in
outcomes_by_author_family. That is pre-existing behaviour this PR must
leave unchanged, so the test now pins it; the per-model map (canonical
ids) was already correct. Also removes the unused `failures` field
golangci-lint flagged.

Refs #18720

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qFmFJp26r9LPdmi3t
Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>

* test: evidence family buckets follow the provider-stripped Family (#18803)

v5's #18803 derives modelfamily.Family from the canonical id, so the
prefixed Claude PR joins the claude family and the openai-codex PR is
gpt; update the family expectations and their comment to match.

Signed-off-by: Danathar <6772335+Danathar@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BQYnPGFJx81c8JXR3o1sg1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant