Repository navigation
π Fall back to the gateway default_model when the per-agent model is not entitled - #12160
Conversation
β¦not entitled (Refs #9945) An agent entry's own model: (typically left over from an earlier copilot/ claude backend, e.g. claude-sonnet-4-6) always beat the litellm gateway's default_model, so FMA agents sent every request for aws/claude-sonnet-4-6 and 403'd with 'team not allowed to access model' while the gateway card showed the entitled default rits/zai-org/glm-5-3. - InferenceRoute carries DefaultModel (gateway default_model or the legacy block's default) beside Model for litellm, named-gateway and watsonx routes. - routeWithEntitledModel: when the stored model is neither entitled nor a drift of an entitled id, but the DefaultModel is, forward the default and warn with the agent name and remedy. Stored route is never mutated. - Gateway form Test (discover) now returns the same endpoint / default model / key SHA / route diagnostic on success as the saved-card Test, and flags an endpoint that is not saved yet; the UI renders it instead of a bare 'OK β N models'. - Tests: proxy fallback repro with the live FMA 403 body, boot-level route resolution carrying DefaultModel, dashboard form-Test success parity. Signed-off-by: Andrew Anderson <andy@clubanderson.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Open PR overlap checkBase branch: Conflicts are checked with every PR merged onto the current tip of
|
|
merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 3b90a4c. |
|
Thank you for your contribution! Your PR has been merged. We'd love to hear how your experience was: share feedback |
Refs #9945 β cc @MikeSpreitzer
Root cause
After #11794 the endpoint and key were right, and Mike's new error showed the remaining dimension: the model.
The route resolver honours a per-agent
model:over the gatewaydefault_model(correct in general). The FMA agents still carrymodel: claude-sonnet-4-6from their original copilot backend (the pack template writes it for every agent). When the backend was switched tolitellm, that value came along; the #4400 entitlement rewrite then mapped it to the key-info-listedaws/claude-sonnet-4-6, and LiteLLM's team scope refused it. The gateway default β the only thing Mike could see, and the one model the team can use β was never considered. The "Please run /login" prefix is just the Claude CLI's rendering of any 403; the request itself was built by Hive's translator with the route's model.Fix
proxy.InferenceRoute.DefaultModelcarries the backend's default (gatewaydefault_model, or the legacy block's) besideModel, for litellm / named-gateway / watsonx routes.routeWithEntitledModel: once the gateway has told us the entitled set (key-info probe or the 403 itself), a per-agent model that is neither entitled nor a spelling drift of an entitled id is replaced per request by the gateway default (when that is entitled). A warning names the agent, the configured model, the default sent and the remedy. The stored route is never mutated; an entitled explicit model still wins; no default / un-entitled default β unchanged, so the gateway's own 403 still reaches the operator. Self-healing: first 403 teaches the set, second request succeeds./gateways/discover) returns the sameEndpoint / Default model / Using key / SHA256 / Connection OK / Agents on backend "litellm" send inference to POST β¦ (model β¦)text as the saved-card Test, and flags when the endpoint typed is not saved yet. The UI renders it instead of a bare "OK β N models available".Tests
pkg/proxyTestRouteWithEntitledModel_FallsBackToGatewayDefaultβ reproduces with the live FMA 403 body: no per-agent model β default; staleclaude-sonnet-4-6β default after the 403; outbound OpenAI body never carries the CLI's own model; entitled explicit model wins; drifted default resolves; un-entitled default / no default pass through.cmd/hiveTestBootProxyWith_LiteLLMGatewayDefaultModelRidesAlongβ gateway default X rides on the route for agents with and without a model, named gateways and legacy-only litellm.pkg/dashboardTestGatewayFormTestSuccessEchoesWhatWasTested(+ static wiring) β form and saved Test success diagnostics are byte-identical, include the route line, never leak the key, and warn on an unsaved endpoint.go vet ./pkg/... ./cmd/...,go build ./...,go test ./pkg/agent/... ./pkg/dashboard/... ./pkg/config/... ./pkg/proxy/... ./pkg/inference/... ./cmd/hive/...pass (twoTestApplySessionEnv_*tmux tests fail locally on macOS with "socket path too long" β pre-existing, unrelated).