Skip to content

🌱 Weekly static model list freshness check and list-shape test - #18474

Merged
hivecommons-hive[bot] merged 1 commit into
v5from
feat/static-model-freshness
Nov 16, 2026
Merged

hivecommons-hive[bot] merged 1 commit into
v5from
feat/static-model-freshness

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Summary

Item 4 of the gateway model-discovery work: a freshness check for static model lists. Follows #18449 and #18463.

What has a static list on v5

cli_models*.go has no aider, opencode, kilo or openhands lists on v5. The static lists are:

  • Fallbacks for probed CLIs: claudeStaticModels / claudePinnedCLIModels, copilotStaticModels / copilotPinnedCLIModels, codexStaticModels, geminiStaticModels / geminiPinnedCLIModels, agyStaticModels, ompStaticModels.
  • Lists with no live probe: piKiroStaticModels (pi), bobStaticModels (auto only), gooseProviderStaticModels, inferenceStaticModelAliases.
  • The curated OpenRouter SuggestedModels / DefaultModel in crux/openrouter.

Weekly workflow (.github/workflows/static-model-freshness.yml)

Runs Mondays at 07:17 UTC, or on demand (dry_run prints the report only). .github/scripts/check-static-model-freshness.py uses only public, keyless sources:

  • OpenRouter GET /api/v1/models:
    • The curated OpenRouter ids and default are OpenRouter ids, so a missing id is drift.
    • Claude, Gemini and OpenAI/Codex/Copilot ids are mapped onto anthropic/, google/ and openai/ and normalized (dots and dashes unified, date suffix dropped). The report lists newer models of the same line (same family and suffix, higher version) and ids with no upstream counterpart (often CLI-only aliases, so treat these as hints).
  • npm registry latest versions of @anthropic-ai/claude-code, @github/copilot, @openai/codex and @google/gemini-cli, as context.

It opens one tracking issue (label static-model-freshness) on the first drift and edits that same issue on later runs. With no drift it updates an open issue to say so and never opens a new one. A failed catalog fetch is reported, not treated as drift. No secrets: only the workflow's own github.token with issues: write. Checkout uses persist-credentials: false.

Not covered (needs auth, or has no machine-readable list): the Claude CLI's /v1/models (OAuth), the Copilot SDK catalog, the Codex app server, Kiro (pi), agy, omp, goose providers and bob. Those CLIs are probed live at runtime, and the report says they were not checked.

The current run against the live catalog would already flag, for example, claude-haiku-5-5 and claude-fable-5-1 / 5-2 missing from the Claude fallback, and gemini-3.x-flash missing from the Gemini fallback.

Unit test (TestStaticModelListsShape)

  • Every static list above, including each goose provider list, must be non-empty with no empty or duplicate ids.
  • First-paint copies in static/index.html must match the Go lists exactly, order included: CLAUDE, GEMINI, CODEX, AGY, OMP and BOB. The Copilot first-paint list is a curated subset, so each of its ids must be in copilotPinnedCLIModels. GOOSE is checked for shape.

Docs

src/docs/agent-configuration.md now says that gateway-backed agents, and any CLI pointed at a gateway, inherit the gateway's live /v1/models list. Only CLI-native backends ever fall back to a static list. It also describes both checks.

🤖 Generated with Claude Code

- .github/workflows/static-model-freshness.yml + script: compare static
  fallbacks and OpenRouter suggestions with OpenRouter /models and npm
  release metadata; keep one tracking issue; no secrets
- TestStaticModelListsShape: non-empty, no duplicates, Go/JS in sync
- docs: gateway-backed agents inherit the gateway's live model list

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added the dco-signoff: yes Indicates the PR's author has signed the DCO. label Nov 16, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubestellar-prow kubestellar-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 16, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor
ab3aeae

ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blocked

Hive flagged this PR (author @clubanderson, head ab3aeae2aedb) because it matches behaviors that can override security controls, escalate privileges or damage the codebase. This is a heuristic, not an accusation — a maintainer should confirm the change is intended before it merges.

  • sensitive_path — changes 1 sensitive path(s) (Touches a sensitive path (OWNERS, workflows, policies, hive config, security docs, …))
    • .github/workflows/static-model-freshness.yml
  • ci_gate_weakening — CI configuration is deleted, softened or has gate checks removed (Disables or softens CI gates (continue-on-error, || true, removed checks))
    • .github/workflows/static-model-freshness.yml

Hive did not add the sentinel-alert label because this author is trusted. The finding remains recorded for audit and the dashboard Security tab; set sentinel.trusted_authors_block: true to require the blocking label for trusted authors too.

@hivecommons-hive
hivecommons-hive Bot merged commit c47c678 into v5 Nov 16, 2026
94 of 98 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the feat/static-model-freshness branch November 16, 2026 19:39
@hivecommons-hive

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

Evidence bundle `#18474ab3aeae` sha256:88c28f27d3ecb3ada3b7915e6d7f9bb5ea3a8522510c8dba254e034bb67c1d34 — download from the dashboard or `hivectl review evidence #18474`.

@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at ab3aeae.

@clubanderson clubanderson added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Nov 16, 2026

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision needed: @clubanderson should decide whether this new weekly workflow (issues: write, opens/edits a tracking issue) should be added to .github/workflows.

Routing to a human: this adds a scheduled workflow with issues: write, a CI-permission surface.

What I verified at ab3aeae:

  • .github/workflows/static-model-freshness.yml:24-26: permissions is contents: read + issues: write; the only token is the workflow's own GITHUB_TOKEN, no repository secrets.
  • Checkout is SHA-pinned with persist-credentials: false; ref: v5 is the repo's default branch. No untrusted input is interpolated into run: blocks (only a boolean dry_run input).
  • The script reads src/pkg/dashboard/cli_models.go and src/crux/openrouter/openrouter.go (both exist) and only fetches public OpenRouter/npm endpoints; it catches fetch failures and exits 0.

I found no code defect in the script or the shape test. The open point is purely whether an unattended weekly job that creates/edits an issue should land.

— hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.325

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has signed the DCO. lgtm Indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant