Repository navigation
🌱 Weekly static model list freshness check and list-shape test - #18474
Conversation
- .github/workflows/static-model-freshness.yml + script: compare static fallbacks and OpenRouter suggestions with OpenRouter /models and npm release metadata; keep one tracking issue; no secrets - TestStaticModelListsShape: non-empty, no duplicates, Go/JS in sync - docs: gateway-backed agents inherit the gateway's live model list Signed-off-by: Andy Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ab3aeae
ℹ️ Sentinel notice — informational; author is trusted (owner), merge is not blockedHive flagged this PR (author @clubanderson, head
Hive did not add the |
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at ab3aeae. |
There was a problem hiding this comment.
Decision needed: @clubanderson should decide whether this new weekly workflow (issues: write, opens/edits a tracking issue) should be added to .github/workflows.
Routing to a human: this adds a scheduled workflow with issues: write, a CI-permission surface.
What I verified at ab3aeae:
.github/workflows/static-model-freshness.yml:24-26:permissionsiscontents: read+issues: write; the only token is the workflow's ownGITHUB_TOKEN, no repository secrets.- Checkout is SHA-pinned with
persist-credentials: false;ref: v5is the repo's default branch. No untrusted input is interpolated intorun:blocks (only a booleandry_runinput). - The script reads
src/pkg/dashboard/cli_models.goandsrc/crux/openrouter/openrouter.go(both exist) and only fetches public OpenRouter/npm endpoints; it catches fetch failures and exits 0.
I found no code defect in the script or the shape test. The open point is purely whether an unattended weekly job that creates/edits an issue should land.
— hive: agent=reviewer backend=claude model=claude-sonnet-5-5 claude=2.1.325
Summary
Item 4 of the gateway model-discovery work: a freshness check for static model lists. Follows #18449 and #18463.
What has a static list on v5
cli_models*.gohas no aider, opencode, kilo or openhands lists on v5. The static lists are:claudeStaticModels/claudePinnedCLIModels,copilotStaticModels/copilotPinnedCLIModels,codexStaticModels,geminiStaticModels/geminiPinnedCLIModels,agyStaticModels,ompStaticModels.piKiroStaticModels(pi),bobStaticModels(auto only),gooseProviderStaticModels,inferenceStaticModelAliases.SuggestedModels/DefaultModelincrux/openrouter.Weekly workflow (
.github/workflows/static-model-freshness.yml)Runs Mondays at 07:17 UTC, or on demand (
dry_runprints the report only)..github/scripts/check-static-model-freshness.pyuses only public, keyless sources:GET /api/v1/models:anthropic/,google/andopenai/and normalized (dots and dashes unified, date suffix dropped). The report lists newer models of the same line (same family and suffix, higher version) and ids with no upstream counterpart (often CLI-only aliases, so treat these as hints).@anthropic-ai/claude-code,@github/copilot,@openai/codexand@google/gemini-cli, as context.It opens one tracking issue (label
static-model-freshness) on the first drift and edits that same issue on later runs. With no drift it updates an open issue to say so and never opens a new one. A failed catalog fetch is reported, not treated as drift. No secrets: only the workflow's owngithub.tokenwithissues: write. Checkout usespersist-credentials: false.Not covered (needs auth, or has no machine-readable list): the Claude CLI's
/v1/models(OAuth), the Copilot SDK catalog, the Codex app server, Kiro (pi), agy, omp, goose providers and bob. Those CLIs are probed live at runtime, and the report says they were not checked.The current run against the live catalog would already flag, for example,
claude-haiku-5-5andclaude-fable-5-1/5-2missing from the Claude fallback, andgemini-3.x-flashmissing from the Gemini fallback.Unit test (
TestStaticModelListsShape)static/index.htmlmust match the Go lists exactly, order included: CLAUDE, GEMINI, CODEX, AGY, OMP and BOB. The Copilot first-paint list is a curated subset, so each of its ids must be incopilotPinnedCLIModels. GOOSE is checked for shape.Docs
src/docs/agent-configuration.mdnow says that gateway-backed agents, and any CLI pointed at a gateway, inherit the gateway's live/v1/modelslist. Only CLI-native backends ever fall back to a static list. It also describes both checks.🤖 Generated with Claude Code