diff --git a/CHANGELOG.md b/CHANGELOG.md index bb892fd25b1..a500739343b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,11 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Added +- Provider-scoped client endpoints under `/provider/` with filtered OpenAI, Anthropic and + Gemini model catalogs, provider-pinned text execution and isolated session affinity. Named + OpenAI-compatible upstreams such as `mistral` are supported. Antigravity CLI setup now uses + its enterprise model gateway environment variables. + - Mistral monthly API and Vibe Code budget history for API-key upstreams on `api.mistral.ai`, with cached subscription plans and optional `budget` and `currency` window fields. These usage-only windows use account weight 1. @@ -62,6 +67,9 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixed +- Two Mistral keys of one customer (a console key and a Vibe key) are recorded as one account, + so the monthly API and Vibe Code budgets are not counted twice. + - `soonest-reset-first` no longer sends new work to an account it knows is spent while the account with allowance left is only on a short hold. Seen on a live hub: one "overloaded" answer put the usable Codex account on a five-minute hold and the next conversations were diff --git a/docs/providers.md b/docs/providers.md index a99ea24918c..5160056b569 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -13,6 +13,36 @@ Two hard rules apply throughout this page. Follow [setup](setup.md) for installation and keys. [Environments](environments.md) covers machines that run agents, and [box setup](box-setup.md) gives an ordered checklist. +## Pin a client to one provider + +Use `https:///provider/` as the client base URL. Provider names match registry keys +and are case-insensitive, including configured `api-keys.openai-compatibility` names such as +`mistral`. For example, set an OpenAI-compatible client's base URL to +`https:///provider/mistral/v1`, or a Gemini client's base URL to +`https:///provider/antigravity` when it appends `/v1beta` itself. Authenticate with a hub +access key. + +The prefix supports these routes. + +- `GET /provider//v1/models` and `/v1/models/` use the existing OpenAI or Anthropic + catalog shape selected by the client headers. +- `GET /provider//v1beta/models` and `/v1beta/models/` return Gemini model metadata. +- `POST /provider//v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages` + and `/v1/messages/count_tokens` use the existing HTTP handlers, including streaming. +- `POST /provider//v1beta/models/:generateContent`, `:streamGenerateContent` and + `:countTokens` use the Gemini handlers. + +Catalogs contain that provider's registered models. Requests select credentials from that +provider's accounts, including when another provider serves the same model ID. A model outside +the scope gets the usual `model_not_found` error. An unknown provider returns HTTP 404. + +Configured public model aliases, credential prefixes and thinking suffixes keep their usual +meaning within the selected provider. Keep any credential prefix in the model ID, such as +`team/model-name`. The `auto` model selects from the scoped catalog. Session affinity bindings +are isolated by provider scope. Scoped routes resolve through the registry and bypass plugin +model routers. Payload rules and request interceptors still run through the existing execution +pipeline. + ## Capability table The table describes this checkout. **Text APIs** means the hub's translated Chat Completions, diff --git a/docs/t3code.md b/docs/t3code.md index fcd0f40664f..1455649b879 100644 --- a/docs/t3code.md +++ b/docs/t3code.md @@ -41,34 +41,49 @@ signs in directly and does not go through the hub. ## Antigravity (Gemini) -T3 Code's Antigravity provider runs the Antigravity CLI (`agy`). With its **Gemini API key** sign-in -the CLI sends Gemini API requests, and it reads the endpoint from `GOOGLE_GEMINI_BASE_URL`. That -lets the stock provider use the hub's Antigravity account. It needs an `agy` release that knows -this variable; 1.5.9 does. +T3 Code's Antigravity provider runs the Antigravity CLI (`agy`). Its enterprise model gateway +mode discovers models from the hub and sends requests through the hub's Antigravity accounts. +This needs a recent `agy` release with gateway support. 1. Open **Settings → Providers → Antigravity**. -2. Set the sign-in method to **Gemini API key** and enter a hub access key, not a Google key. -3. Add the environment variable `GOOGLE_GEMINI_BASE_URL` with the hub's root URL, such as - `https://..ts.net:8317`. +2. Set the sign-in method to **Gemini API key** and enter a hub access key. +3. Add these provider environment variables. Mark the access key as sensitive. -The model list comes from the CLI, not from the hub, so it also shows Gemini models your -Antigravity account does not serve. Those fail with `unknown provider for model`. Hide them in -the model picker and keep the ones `GET /v1beta/models` on the hub returns. The Google account -sign-in method does not go through the hub. + ```text + AGY_LLM_GATEWAY_URL=https:///provider/antigravity + AGY_LLM_GATEWAY_API_KEY= + ``` + +The model list comes from the hub and contains the models served by its Antigravity accounts. +The Google account sign-in method connects directly to Google. ## Mistral (OpenCode and Vibe) Mistral has two monthly budgets on one account: API usage and Vibe Code. Which one a request -draws on is decided by Mistral from the client that sent it. - -- **OpenCode** pointed at the hub uses the hub's Mistral upstream (an `api-keys.openai-compatibility` - entry for `https://api.mistral.ai/v1`) and draws on the API budget. -- **Mistral Vibe**, added in T3 Code as an ACP agent and signed in with Mistral, talks to Mistral - directly and draws on the Vibe Code budget. It does not pass through the hub. - -The hub reads both budgets with its Mistral key and shows them on the Overview page as money -spent of the monthly amount, with the reset date, whichever client did the spending. See -[quota history](quota-history.md). +draws on depends on the key that sends it: a key made in the Mistral console draws on the API +budget, and the key Mistral Vibe gets when you sign in with Mistral draws on the Vibe Code budget. +To run both through the hub, give the hub both keys as two upstreams. + +1. Keep the console key as the `mistral` upstream (`api-keys.openai-compatibility`, base URL + `https://api.mistral.ai/v1`). OpenCode and other clients use it as ``. +2. Add the Vibe key as a second upstream with the same base URL and models and `prefix: vibe`, and + set `force-model-prefix: true`. Its models are then reachable only as `vibe/`, so nothing + else spends the Vibe budget by accident. On macOS, Vibe keeps its key in the keychain under the + service `ai.mistral.vibe`. +3. Declare the reasoning levels Mistral accepts on models that have them, for example + `thinking: {levels: [none, high], zero-allowed: true}` on `mistral-vibe-cli-latest`. Without it + the hub turns "thinking off" into `reasoning_effort: low`, which Mistral rejects. +4. Point Vibe at the hub in `~/.vibe/config.toml`: a `mistral` provider with + `api_base = "https://..ts.net:8317/v1"`, `backend = "mistral"`, + `preserve_api_base = true`, and `api_key_env_var = "MISTRAL_API_KEY"`, with the hub access key + in `~/.vibe/.env`. Add each model as `[[models]]` with `name = "vibe/"` and the alias Vibe + shows, such as `mistral-medium-3.5` for `vibe/mistral-vibe-cli-latest`. + +T3 Code's Mistral Vibe agent (ACP) reads the same files, so it follows. + +The hub reads both budgets and shows them on the Overview page as money spent of the monthly +amount, with the reset date. Two keys of one Mistral customer are recorded as one account, so +neither allowance is counted twice. See [quota history](quota-history.md). ## Other upstream providers diff --git a/internal/api/handlers/management/quota_history_mistral_test.go b/internal/api/handlers/management/quota_history_mistral_test.go index 2339f077a7a..7ab22486a99 100644 --- a/internal/api/handlers/management/quota_history_mistral_test.go +++ b/internal/api/handlers/management/quota_history_mistral_test.go @@ -20,7 +20,7 @@ import ( func TestQuotaHistoryMistralAPIKeyEndpoint(t *testing.T) { budget := `{"vibe_budget":{"usage_percentage":0.0,"initial_budget":150.0,"currency":"USD","reset_at":"2026-12-01T00:00:00Z","payg_enabled":false},"api_budget":{"usage_percentage":2.59892596,"initial_budget":15.0,"currency":"USD","reset_at":"2026-12-01T00:00:00Z","payg_enabled":false},"usage_percentage":2.59892596,"initial_budget":15.0,"currency":"USD","reset_at":"2026-12-01T00:00:00Z"}` upstream := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { - if req.Method != "GET" || req.Host != "api.mistral.ai" || req.Header.Get("Authorization") != "Bearer fixture-mistral-key" { + if req.Method != "GET" || (req.Host != "api.mistral.ai" && req.Host != "console.mistral.ai") || req.Header.Get("Authorization") != "Bearer fixture-mistral-key" { t.Error("billing request must use the upstream's own Bearer key and host") w.WriteHeader(http.StatusBadRequest) return @@ -31,6 +31,8 @@ func TestQuotaHistoryMistralAPIKeyEndpoint(t *testing.T) { _, _ = w.Write([]byte(budget)) case "/v1/billing/subscription": _, _ = w.Write([]byte(`{"plan_name":"EDU","payg_enabled":false,"vibe_payg_enabled":false,"enterprise":false}`)) + case "/api/vibe/whoami": + _, _ = w.Write([]byte(`{"customer_id":"fixture-customer"}`)) default: t.Errorf("unexpected billing path %q", req.URL.Path) w.WriteHeader(http.StatusNotFound) diff --git a/internal/api/provider_routes.go b/internal/api/provider_routes.go new file mode 100644 index 00000000000..10506f185d2 --- /dev/null +++ b/internal/api/provider_routes.go @@ -0,0 +1,58 @@ +package api + +import ( + "errors" + "fmt" + "net/http" + "strings" + + "github.com/gin-gonic/gin" + "github.com/rune-oai/t3code-proxy/internal/interfaces" + "github.com/rune-oai/t3code-proxy/internal/registry" + "github.com/rune-oai/t3code-proxy/internal/util" + "github.com/rune-oai/t3code-proxy/sdk/api/handlers" + "github.com/rune-oai/t3code-proxy/sdk/api/handlers/claude" + "github.com/rune-oai/t3code-proxy/sdk/api/handlers/gemini" + "github.com/rune-oai/t3code-proxy/sdk/api/handlers/openai" + "github.com/rune-oai/t3code-proxy/sdk/cliproxy/auth" +) + +func (s *Server) setupProviderRoutes(openaiHandler *openai.OpenAIAPIHandler, claudeHandler *claude.ClaudeCodeAPIHandler, responsesHandler *openai.OpenAIResponsesAPIHandler, geminiHandler *gemini.GeminiAPIHandler) { + group := s.engine.Group("/provider/:provider", AuthMiddleware(s.accessManager)) + group.Use(func(c *gin.Context) { + provider := strings.ToLower(strings.TrimSpace(c.Param("provider"))) + modelRegistry := registry.GetGlobalRegistry() + if !modelRegistry.HasProvider(provider) { + provider = util.OpenAICompatibleProviderKey(provider) + } + if !modelRegistry.HasProvider(provider) { + message := fmt.Sprintf("unknown provider %q", c.Param("provider")) + errMsg := &interfaces.ErrorMessage{StatusCode: http.StatusNotFound, Error: errors.New(message)} + switch { + case strings.Contains(c.FullPath(), "/v1beta/"): + c.JSON(http.StatusNotFound, gin.H{"error": gin.H{"code": http.StatusNotFound, "message": message, "status": "NOT_FOUND"}}) + case strings.Contains(c.FullPath(), "/messages") || (strings.Contains(c.FullPath(), "/v1/models") && isAnthropicModelsRequest(c)): + claudeHandler.WriteErrorResponse(c, errMsg) + default: + s.handlers.WriteErrorResponse(c, errMsg) + } + c.Abort() + return + } + c.Request = c.Request.WithContext(auth.WithProviderScope(c.Request.Context(), provider)) + c.Next() + }) + group.GET("/v1/models", s.unifiedModelsHandler(openaiHandler, claudeHandler)) + group.GET("/v1/models/*model", func(c *gin.Context) { + c.Set(handlers.ModelDetailIDContextKey, strings.TrimPrefix(c.Param("model"), "/")) + s.unifiedModelsHandler(openaiHandler, claudeHandler)(c) + }) + group.POST("/v1/chat/completions", openaiHandler.ChatCompletions) + group.POST("/v1/completions", openaiHandler.Completions) + group.POST("/v1/responses", responsesHandler.Responses) + group.POST("/v1/messages", claudeHandler.ClaudeMessages) + group.POST("/v1/messages/count_tokens", claudeHandler.ClaudeCountTokens) + group.GET("/v1beta/models", s.geminiModelsHandler(geminiHandler)) + group.GET("/v1beta/models/*action", s.geminiGetHandler(geminiHandler)) + group.POST("/v1beta/models/*action", geminiHandler.GeminiHandler) +} diff --git a/internal/api/provider_routes_test.go b/internal/api/provider_routes_test.go new file mode 100644 index 00000000000..6cef4e1a977 --- /dev/null +++ b/internal/api/provider_routes_test.go @@ -0,0 +1,282 @@ +package api + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + "github.com/MoeForge/moego/ojson" + "github.com/rune-oai/t3code-proxy/internal/config" + "github.com/rune-oai/t3code-proxy/internal/registry" + "github.com/rune-oai/t3code-proxy/internal/util" + "github.com/rune-oai/t3code-proxy/sdk/cliproxy/auth" + coreexecutor "github.com/rune-oai/t3code-proxy/sdk/cliproxy/executor" +) + +const scopedSharedModel = "claude-provider-scope-shared" +const scopedOutsideModel = "claude-provider-scope-outside" + +type providerScopeExecutor struct { + provider string + authIDs []string + models []string + fail bool +} + +func (e *providerScopeExecutor) Identifier() string { return e.provider } + +func (e *providerScopeExecutor) Execute(_ context.Context, a *auth.Auth, req coreexecutor.Request, _ coreexecutor.Options) (coreexecutor.Response, error) { + e.authIDs = append(e.authIDs, a.ID) + e.models = append(e.models, req.Model) + if e.fail { + return coreexecutor.Response{}, errors.New("scoped upstream failed") + } + return coreexecutor.Response{Payload: []byte(`{"id":"scope-response","object":"chat.completion","choices":[{"index":0,"message":{"role":"assistant","content":"ok"},"finish_reason":"stop"}],"content":[{"type":"text","text":"ok"}],"candidates":[{"content":{"parts":[{"text":"ok"}]}}],"input_tokens":1,"totalTokens":1}`)}, nil +} + +func (e *providerScopeExecutor) ExecuteStream(ctx context.Context, a *auth.Auth, req coreexecutor.Request, opts coreexecutor.Options) (*coreexecutor.StreamResult, error) { + response, errExecute := e.Execute(ctx, a, req, opts) + if errExecute != nil { + return nil, errExecute + } + chunk := response.Payload + switch opts.SourceFormat.String() { + case "openai": + chunk = []byte("data: {\"id\":\"scope-response\",\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"},\"finish_reason\":\"stop\"}]}\n\ndata: [DONE]\n\n") + case "claude": + chunk = []byte("event: message_start\ndata: {\"type\":\"message_start\",\"message\":{\"id\":\"scope-response\",\"type\":\"message\",\"role\":\"assistant\",\"content\":[]}}\n\nevent: message_stop\ndata: {\"type\":\"message_stop\"}\n\n") + case "openai-response": + chunk = []byte("event: response.completed\ndata: {\"type\":\"response.completed\",\"response\":{\"id\":\"scope-response\",\"status\":\"completed\",\"output\":[]}}\n\n") + } + chunks := make(chan coreexecutor.StreamChunk, 1) + chunks <- coreexecutor.StreamChunk{Payload: chunk} + close(chunks) + return &coreexecutor.StreamResult{Chunks: chunks}, nil +} + +func (e *providerScopeExecutor) CountTokens(ctx context.Context, a *auth.Auth, req coreexecutor.Request, opts coreexecutor.Options) (coreexecutor.Response, error) { + return e.Execute(ctx, a, req, opts) +} + +func (e *providerScopeExecutor) Refresh(_ context.Context, a *auth.Auth) (*auth.Auth, error) { + return a, nil +} + +func (e *providerScopeExecutor) HttpRequest(context.Context, *auth.Auth, *http.Request) (*http.Response, error) { + return nil, errors.New("unexpected HTTP passthrough") +} + +func newProviderScopeServer(t *testing.T) (*Server, map[string]*providerScopeExecutor) { + t.Helper() + server := newTestServer(t) + executors := make(map[string]*providerScopeExecutor) + for _, provider := range []string{"antigravity", "claude", "mistral"} { + providerKey := provider + if provider == "mistral" { + providerKey = util.OpenAICompatibleProviderKey(provider) + } + executor := &providerScopeExecutor{provider: providerKey} + executors[provider] = executor + server.handlers.AuthManager.RegisterExecutor(executor) + a := &auth.Auth{ID: "provider-scope-" + provider, Provider: provider, Status: auth.StatusActive} + if provider == "mistral" { + a.Provider = "openai-compatibility" + a.Label = "mistral" + } + if _, errRegister := server.handlers.AuthManager.Register(context.Background(), a); errRegister != nil { + t.Fatal(errRegister) + } + models := []*registry.ModelInfo{{ID: scopedSharedModel, OwnedBy: provider, DisplayName: provider + " shared"}} + if provider == "claude" { + models = append(models, ®istry.ModelInfo{ID: scopedOutsideModel, OwnedBy: provider}) + } + registry.GetGlobalRegistry().RegisterClient(a.ID, providerKey, models) + t.Cleanup(func() { registry.GetGlobalRegistry().UnregisterClient(a.ID) }) + } + return server, executors +} + +func scopedRequest(server *Server, method, path, body string, headers map[string]string) *httptest.ResponseRecorder { + req := httptest.NewRequest(method, path, strings.NewReader(body)) + req.Header.Set("Authorization", "Bearer test-key") + req.Header.Set("Content-Type", "application/json") + for key, value := range headers { + req.Header.Set(key, value) + } + recorder := httptest.NewRecorder() + server.engine.ServeHTTP(recorder, req) + return recorder +} + +func TestProviderScopedModels(t *testing.T) { + server, _ := newProviderScopeServer(t) + for _, test := range []struct { + path string + headers map[string]string + field string + }{ + {path: "/v1/models", field: "data"}, + {path: "/v1/models", headers: map[string]string{"Anthropic-Version": "2023-06-01"}, field: "data"}, + {path: "/v1/models?client_version=cpa", field: "models"}, + {path: "/v1beta/models", field: "models"}, + } { + t.Run(test.path+test.field, func(t *testing.T) { + response := scopedRequest(server, http.MethodGet, "/provider/AnTiGrAvItY"+test.path, "", test.headers) + if response.Code != http.StatusOK { + t.Fatalf("status=%d body=%s", response.Code, response.Body.String()) + } + models := ojson.Get(response.Body.String(), test.field).Array() + if len(models) != 1 || !strings.Contains(models[0].Raw, scopedSharedModel) || strings.Contains(response.Body.String(), scopedOutsideModel) { + t.Fatalf("unexpected scoped catalog: %s", response.Body.String()) + } + if test.field == "data" && !strings.Contains(response.Body.String(), `"owned_by":"antigravity"`) { + t.Fatalf("shared model metadata came from another provider: %s", response.Body.String()) + } + unscoped := scopedRequest(server, http.MethodGet, test.path, "", test.headers) + if unscoped.Code != http.StatusOK || !strings.Contains(unscoped.Body.String(), scopedOutsideModel) { + t.Fatalf("unexpected unscoped catalog: %d %s", unscoped.Code, unscoped.Body.String()) + } + }) + } + for _, path := range []string{"/v1/models/", "/v1beta/models/"} { + response := scopedRequest(server, http.MethodGet, "/provider/antigravity"+path+scopedSharedModel, "", nil) + if response.Code != http.StatusOK || !strings.Contains(response.Body.String(), scopedSharedModel) { + t.Fatalf("model detail: %d %s", response.Code, response.Body.String()) + } + response = scopedRequest(server, http.MethodGet, "/provider/antigravity"+path+scopedOutsideModel, "", nil) + if response.Code != http.StatusNotFound { + t.Fatalf("outside model detail: %d %s", response.Code, response.Body.String()) + } + } +} + +func TestProviderScopedExecution(t *testing.T) { + for _, test := range []struct { + path string + body string + }{ + {path: "/v1/chat/completions", body: `{"model":"MODEL","messages":[{"role":"user","content":"hi"}]}`}, + {path: "/v1/chat/completions", body: `{"model":"MODEL","messages":[{"role":"user","content":"hi"}],"stream":true}`}, + {path: "/v1/completions", body: `{"model":"MODEL","prompt":"hi"}`}, + {path: "/v1/responses", body: `{"model":"MODEL","input":"hi"}`}, + {path: "/v1/responses", body: `{"model":"MODEL","input":"hi","stream":true}`}, + {path: "/v1/messages", body: `{"model":"MODEL","max_tokens":32,"messages":[{"role":"user","content":"hi"}]}`}, + {path: "/v1/messages", body: `{"model":"MODEL","max_tokens":32,"messages":[{"role":"user","content":"hi"}],"stream":true}`}, + {path: "/v1/messages/count_tokens", body: `{"model":"MODEL","messages":[{"role":"user","content":"hi"}]}`}, + {path: "/v1beta/models/MODEL:generateContent", body: `{"contents":[{"parts":[{"text":"hi"}]}]}`}, + {path: "/v1beta/models/MODEL:streamGenerateContent", body: `{"contents":[{"parts":[{"text":"hi"}]}]}`}, + {path: "/v1beta/models/MODEL:countTokens", body: `{"contents":[{"parts":[{"text":"hi"}]}]}`}, + } { + t.Run(test.path+test.body, func(t *testing.T) { + server, executors := newProviderScopeServer(t) + for _, provider := range []string{"antigravity", "mistral"} { + body := strings.ReplaceAll(test.body, "MODEL", scopedSharedModel) + path := strings.ReplaceAll(test.path, "MODEL", scopedSharedModel) + response := scopedRequest(server, http.MethodPost, "/provider/"+provider+path, body, nil) + if response.Code != http.StatusOK || len(executors[provider].authIDs) != 1 || executors[provider].authIDs[0] != "provider-scope-"+provider { + t.Fatalf("provider=%s status=%d body=%s auths=%v", provider, response.Code, response.Body.String(), executors[provider].authIDs) + } + response = scopedRequest(server, http.MethodPost, "/provider/"+provider+strings.ReplaceAll(test.path, "MODEL", scopedOutsideModel), strings.ReplaceAll(test.body, "MODEL", scopedOutsideModel), nil) + if response.Code != http.StatusBadRequest || !strings.Contains(response.Body.String(), "unknown provider for model") { + t.Fatalf("outside scope: status=%d body=%s", response.Code, response.Body.String()) + } + if len(executors[provider].authIDs) != 1 || len(executors["claude"].authIDs) != 0 { + t.Fatal("request reached credentials outside the provider scope") + } + } + }) + } +} + +func TestProviderScopedErrorsAndAuth(t *testing.T) { + server, _ := newProviderScopeServer(t) + for _, test := range []struct { + method string + path string + field string + }{ + {http.MethodGet, "/v1/models", "error.message"}, + {http.MethodPost, "/v1/chat/completions", "error.message"}, + {http.MethodPost, "/v1/messages", "error.message"}, + {http.MethodGet, "/v1beta/models", "error.message"}, + {http.MethodPost, "/v1beta/models/x:generateContent", "error.message"}, + } { + response := scopedRequest(server, test.method, "/provider/no-such-provider"+test.path, `{}`, nil) + if response.Code != http.StatusNotFound || !strings.Contains(ojson.Get(response.Body.String(), test.field).String(), "unknown provider") { + t.Fatalf("unknown provider: %d %s", response.Code, response.Body.String()) + } + if strings.Contains(test.path, "/v1beta/") && ojson.Get(response.Body.String(), "error.status").String() != "NOT_FOUND" { + t.Fatalf("Gemini error shape: %s", response.Body.String()) + } + if test.path == "/v1/messages" && ojson.Get(response.Body.String(), "type").String() != "error" { + t.Fatalf("Claude error shape: %s", response.Body.String()) + } + } + response := scopedRequest(server, http.MethodGet, "/provider/antigravity/v1/models", "", map[string]string{"Authorization": "Bearer invalid"}) + if response.Code != http.StatusUnauthorized { + t.Fatalf("authentication: %d %s", response.Code, response.Body.String()) + } + response = scopedRequest(server, http.MethodPost, "/v1/chat/completions", `{"model":"`+scopedOutsideModel+`","messages":[{"role":"user","content":"hi"}]}`, nil) + if response.Code != http.StatusOK { + t.Fatalf("unscoped execution: %d %s", response.Code, response.Body.String()) + } +} + +func TestProviderScopedAffinityAndFailure(t *testing.T) { + server, executors := newProviderScopeServer(t) + selector := auth.NewSessionAffinitySelectorWithConfig(auth.SessionAffinityConfig{ + Fallback: &auth.RoundRobinSelector{}, RebindOn: auth.SessionAffinityRebindOnNever, + }) + t.Cleanup(selector.Stop) + server.handlers.AuthManager.SetSelector(selector) + body := `{"model":"` + scopedSharedModel + `","messages":[{"role":"user","content":"hi"}]}` + headers := map[string]string{"X-Session-ID": "scope-shared-session"} + for _, prefix := range []string{"", "/provider/claude", "/provider/antigravity", "/provider/mistral", "/provider/claude", "/provider/antigravity"} { + response := scopedRequest(server, http.MethodPost, prefix+"/v1/chat/completions", body, headers) + if response.Code != http.StatusOK { + t.Fatalf("affinity %s: %d %s", prefix, response.Code, response.Body.String()) + } + } + claudeCalls := len(executors["claude"].authIDs) + mistralCalls := len(executors["mistral"].authIDs) + executors["antigravity"].fail = true + response := scopedRequest(server, http.MethodPost, "/provider/antigravity/v1/chat/completions", body, headers) + if response.Code == http.StatusOK || len(executors["claude"].authIDs) != claudeCalls || len(executors["mistral"].authIDs) != mistralCalls { + t.Fatalf("failure fell through to another provider: %d %s", response.Code, response.Body.String()) + } +} + +func TestProviderScopedAliasesPrefixesAndAuto(t *testing.T) { + server, executors := newProviderScopeServer(t) + manager := server.handlers.AuthManager + manager.SetOAuthModelAlias(map[string][]config.OAuthModelAlias{ + "antigravity": {{Name: "antigravity-upstream", Alias: "claude-public-scope"}}, + "claude": {{Name: "claude-upstream", Alias: "claude-public-scope"}}, + }) + for _, provider := range []string{"antigravity", "claude"} { + a, _ := manager.GetByID("provider-scope-" + provider) + a.Prefix = "team" + if _, errUpdate := manager.Update(context.Background(), a); errUpdate != nil { + t.Fatal(errUpdate) + } + registry.GetGlobalRegistry().RegisterClient(a.ID, provider, []*registry.ModelInfo{{ID: "team/claude-public-scope", OwnedBy: provider}}) + } + for _, model := range []string{"team/claude-public-scope(8192)", "auto(8192)"} { + body := `{"model":"` + model + `","messages":[{"role":"user","content":"hi"}]}` + response := scopedRequest(server, http.MethodPost, "/provider/antigravity/v1/chat/completions", body, nil) + if response.Code != http.StatusOK { + t.Fatalf("model=%s status=%d body=%s", model, response.Code, response.Body.String()) + } + models := executors["antigravity"].models + if models[len(models)-1] != "antigravity-upstream(8192)" || len(executors["claude"].authIDs) != 0 { + t.Fatalf("model=%s upstream models=%v claude auths=%v", model, models, executors["claude"].authIDs) + } + } + response := scopedRequest(server, http.MethodPost, "/provider/antigravity/v1/chat/completions", `{"model":"claude-public-scope","messages":[]}`, nil) + if response.Code != http.StatusBadRequest || ojson.Get(response.Body.String(), "error.code").String() != "model_not_found" { + t.Fatalf("missing credential prefix: %d %s", response.Code, response.Body.String()) + } +} diff --git a/internal/api/server_middleware.go b/internal/api/server_middleware.go index 841a0d384e0..7b7e647bf06 100644 --- a/internal/api/server_middleware.go +++ b/internal/api/server_middleware.go @@ -410,6 +410,8 @@ func (s *Server) serveExampleAPIKeyWarningPage(c *gin.Context) { func isExampleAPIKeySafeModeProxyPath(path string) bool { switch { + case strings.HasPrefix(path, "/provider/"): + return true case path == "/v1" || strings.HasPrefix(path, "/v1/"): return true case path == "/v1beta" || strings.HasPrefix(path, "/v1beta/"): diff --git a/internal/api/server_routes.go b/internal/api/server_routes.go index 158d98ec849..cba66f96226 100644 --- a/internal/api/server_routes.go +++ b/internal/api/server_routes.go @@ -155,6 +155,8 @@ func (s *Server) setupRoutes() { v1beta.GET("/models/*action", s.geminiGetHandler(geminiHandlers)) } + s.setupProviderRoutes(openaiHandlers, claudeCodeHandlers, openaiResponsesHandlers, geminiHandlers) + // Root endpoint s.engine.GET("/", func(c *gin.Context) { c.JSON(http.StatusOK, gin.H{ @@ -659,7 +661,7 @@ func (s *Server) unifiedModelsHandler(openaiHandler *openai.OpenAIAPIHandler, cl cfg := s.getConfig() if _, ok := c.Request.URL.Query()["client_version"]; ok { clientVersion := c.Query("client_version") - if cfg != nil && cfg.Home.Enabled { + if cfg != nil && cfg.Home.Enabled && auth.ProviderScopeFromContext(c.Request.Context()) == "" { s.handleHomeCodexClientModels(c, clientVersion) return } @@ -667,7 +669,7 @@ func (s *Server) unifiedModelsHandler(openaiHandler *openai.OpenAIAPIHandler, cl return } - if cfg != nil && cfg.Home.Enabled { + if cfg != nil && cfg.Home.Enabled && auth.ProviderScopeFromContext(c.Request.Context()) == "" { s.handleHomeModels(c) return } @@ -714,7 +716,9 @@ func grokModelsFromRegistryInfos(infos []*registry.ModelInfo) []grokbuild.ModelI func (s *Server) handleGrokModels(c *gin.Context) { var models []grokbuild.ModelInfo - if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled { + if scope := auth.ProviderScopeFromContext(c.Request.Context()); scope != "" { + models = grokModelsFromRegistryInfos(registry.GetGlobalRegistry().GetAvailableModelsByProvider(scope)) + } else if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled { entries, ok := s.loadHomeModelEntries(c) if !ok { return @@ -1025,7 +1029,7 @@ func (s *Server) loadMuseCodeHomeModels(c *gin.Context) ([]musecode.ModelInput, func (s *Server) geminiModelsHandler(geminiHandler *gemini.GeminiAPIHandler) gin.HandlerFunc { return func(c *gin.Context) { - if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled { + if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled && auth.ProviderScopeFromContext(c.Request.Context()) == "" { s.handleHomeGeminiModels(c) return } @@ -1036,7 +1040,7 @@ func (s *Server) geminiModelsHandler(geminiHandler *gemini.GeminiAPIHandler) gin func (s *Server) geminiGetHandler(geminiHandler *gemini.GeminiAPIHandler) gin.HandlerFunc { return func(c *gin.Context) { - if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled { + if cfg := s.getConfig(); cfg != nil && cfg.Home.Enabled && auth.ProviderScopeFromContext(c.Request.Context()) == "" { s.handleHomeGeminiModel(c) return } diff --git a/internal/config/config_load.go b/internal/config/config_load.go index 7a6889006b5..70214776779 100644 --- a/internal/config/config_load.go +++ b/internal/config/config_load.go @@ -151,7 +151,7 @@ func managementSecretKeyPath(data []byte) []string { return []string{"remote-management", "secret-key"} } -// LoadConfigOptional reads YAML from configFile. +// LoadConfigOptional reads YAML from configFile. A required file must be nonempty. // If optional is true and the file is missing, it returns an empty Config. // If optional is true and the file is empty or invalid, it returns an empty Config. func LoadConfigOptional(configFile string, optional bool) (*Config, error) { @@ -170,8 +170,12 @@ func LoadConfigOptional(configFile string, optional bool) (*Config, error) { return nil, fmt.Errorf("failed to read config file: %w", err) } - // In cloud deploy mode (optional=true), if file is empty or contains only whitespace, return empty config. - if optional && len(bytes.TrimSpace(data)) == 0 { + // A concurrent save can leave the file empty between truncation and writing. + // Required loads must fail so hot reload keeps the current runtime config. + if len(bytes.TrimSpace(data)) == 0 { + if !optional { + return nil, fmt.Errorf("config file is empty: %s", configFile) + } cfg := &Config{CredentialInFlight: DefaultCredentialInFlightConfig()} cfg.NormalizePluginsConfig() cfg.NormalizeClaudeConfig() diff --git a/internal/logging/gin_logger.go b/internal/logging/gin_logger.go index 94fe7c91f9a..b957d9f55b0 100644 --- a/internal/logging/gin_logger.go +++ b/internal/logging/gin_logger.go @@ -18,6 +18,7 @@ import ( // aiAPIPrefixes defines path prefixes for AI API requests that should have request ID tracking. var aiAPIPrefixes = []string{ + "/provider", "/v1", "/v1beta", "/openai/v1", diff --git a/internal/quotahistory/mistral_test.go b/internal/quotahistory/mistral_test.go index 494ca487868..5713490e424 100644 --- a/internal/quotahistory/mistral_test.go +++ b/internal/quotahistory/mistral_test.go @@ -128,6 +128,8 @@ func TestRecorderMistralAPIKeyAccounts(t *testing.T) { return 200, []byte(mistralBudgetFixture), nil case MistralSubscriptionURL: return planStatus, []byte(mistralSubscriptionFixture), nil + case MistralWhoamiURL: + return 200, []byte(`{"customer_id":"fixture-customer"}`), nil } t.Errorf("unexpected billing URL %q", url) return 404, nil, nil @@ -289,3 +291,80 @@ func TestStoreMistralBudgetsRoundTrip(t *testing.T) { t.Fatalf("legacy window JSON changed: %s", got) } } + +// Two keys of one Mistral customer read the same budgets: an API key and the key +// Mistral Vibe signs in with. They are recorded as one account, the one without +// a model prefix, so neither allowance is counted twice. +func TestRecorderMergesMistralKeysOfOneCustomer(t *testing.T) { + now := time.Date(2026, 11, 15, 0, 0, 0, 0, time.UTC) + cfg := &config.Config{OpenAICompatibility: []config.OpenAICompatibility{ + {Name: "mistral-vibe", Prefix: "vibe", BaseURL: "https://api.mistral.ai/v1", APIKeyEntries: []config.OpenAICompatibilityAPIKey{{APIKey: "fixture-vibe-key"}}}, + {Name: "mistral", BaseURL: "https://api.mistral.ai/v1", APIKeyEntries: []config.OpenAICompatibilityAPIKey{{APIKey: "fixture-api-key"}}}, + {Name: "mistral-other", BaseURL: "https://api.mistral.ai/v1", APIKeyEntries: []config.OpenAICompatibilityAPIKey{{APIKey: "fixture-other-key"}}}, + }} + auths, err := synthesizer.NewConfigSynthesizer().Synthesize(&synthesizer.SynthesisContext{Config: cfg, Now: now, IDGenerator: synthesizer.NewStableIDGenerator()}) + if err != nil || len(auths) != 3 { + t.Fatalf("synthesis = %d accounts, %v", len(auths), err) + } + customerOf := map[string]string{"fixture-vibe-key": "customer-a", "fixture-api-key": "customer-a", "fixture-other-key": "customer-b"} + whoamiStatus := 200 + r := NewRecorder(Deps{ + ListAuths: func() []*coreauth.Auth { return auths }, + DoRequest: func(_ context.Context, auth *coreauth.Auth, _, url string, _ map[string]string, _ []byte) (int, []byte, error) { + switch url { + case MistralBudgetURL: + return 200, []byte(mistralBudgetFixture), nil + case MistralSubscriptionURL: + return 200, []byte(mistralSubscriptionFixture), nil + case MistralWhoamiURL: + return whoamiStatus, []byte(`{"customer_id":"` + customerOf[auth.Attributes["api_key"]] + `"}`), nil + } + return 404, nil, nil + }, + }) + r.now = func() time.Time { return now } + r.startupDelay = time.Hour + if err := r.Apply(enabledSettings(t)); err != nil { + t.Fatal(err) + } + t.Cleanup(r.Close) + // The first round learns the customers as it goes, so a key polled before its + // sibling may still be stored once; history hides it. + if n := r.poll(context.Background(), r.store, 0, nil); n < 2 || n > 3 { + t.Fatalf("first round stored %d samples, want 2 or 3", n) + } + now = now.Add(time.Minute) + if n := r.poll(context.Background(), r.store, 0, nil); n != 2 { + t.Fatalf("stored %d samples, want 2 (one per customer)", n) + } + history, err := r.History(0, 0) + if err != nil { + t.Fatal(err) + } + labels := map[string]bool{} + for _, account := range history.Accounts { + labels[account.Label] = true + } + if len(history.Accounts) != 2 || !labels["mistral"] || !labels["mistral-other"] || labels["mistral-vibe"] { + t.Fatalf("accounts = %+v, want mistral and mistral-other", history.Accounts) + } + listed := map[string]bool{} + for _, account := range history.Accounts { + listed[account.AuthIndex] = true + } + for _, sample := range history.Samples { + if !listed[sample.AuthIndex] { + t.Fatalf("history serves a sample of an unlisted account: %+v", sample) + } + } + if len(history.Samples) != 4 { + t.Fatalf("samples = %d, want 4 (two rounds, two customers)", len(history.Samples)) + } + + // A customer that cannot be read keeps the last one known. + whoamiStatus = 503 + now = now.Add(profileTTL) + if n := r.poll(context.Background(), r.store, 0, nil); n != 2 { + t.Fatalf("stored %d samples after a failed customer read, want 2", n) + } +} diff --git a/internal/quotahistory/providers.go b/internal/quotahistory/providers.go index a44dc6ab91f..806f09d1ca4 100644 --- a/internal/quotahistory/providers.go +++ b/internal/quotahistory/providers.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "sort" "strings" "github.com/MoeForge/moego/ojson" @@ -24,6 +25,9 @@ const ( DevinUsageURL = "https://server.codeium.com/exa.seat_management_pb.SeatManagementService/GetUserStatus" MistralBudgetURL = "https://api.mistral.ai/v1/billing/budget" MistralSubscriptionURL = "https://api.mistral.ai/v1/billing/subscription" + // MistralWhoamiURL names the customer behind a Mistral key, so two keys of + // one customer (an API key and a Vibe key) are recorded as one account. + MistralWhoamiURL = "https://console.mistral.ai/api/vibe/whoami" ) var antigravityQuotaURLs = []string{ @@ -87,9 +91,70 @@ func (r *Recorder) fetchMistral(ctx context.Context, auth *coreauth.Auth) (map[s } return parseMistralPlan(body) }) + r.refreshMistralCustomer(ctx, auth) return windows, plan, nil } +// refreshMistralCustomer learns the customer behind a Mistral key at most once +// per profileTTL. A failed request keeps what was known. +func (r *Recorder) refreshMistralCustomer(ctx context.Context, auth *coreauth.Auth) { + r.plansMu.Lock() + last := r.customerAttempts[auth.Index] + due := last.IsZero() || r.now().Sub(last) >= profileTTL + if due { + r.customerAttempts[auth.Index] = r.now() + } + r.plansMu.Unlock() + if !due { + return + } + body, err := r.get(ctx, auth, MistralWhoamiURL, bearerHeaders()) + if err != nil { + return + } + if id := firstString(ojson.ParseBytes(body).Get("customer_id")); id != "" { + r.plansMu.Lock() + r.customers[auth.Index] = id + r.plansMu.Unlock() + } +} + +// mistralDuplicates returns the Mistral keys that another key of the same +// customer already represents. Both keys read the same monthly budgets, so +// recording both would count each allowance twice. The key without a model +// prefix represents the customer, then the lowest index. +func (r *Recorder) mistralDuplicates(auths []*coreauth.Auth) map[string]bool { + r.plansMu.Lock() + customers := make(map[string]string, len(r.customers)) + for index, id := range r.customers { + customers[index] = id + } + r.plansMu.Unlock() + groups := make(map[string][]*coreauth.Auth) + for _, auth := range auths { + if providerOf(auth) != "mistral" { + continue + } + if id := customers[auth.Index]; id != "" { + groups[id] = append(groups[id], auth) + } + } + duplicates := make(map[string]bool) + for _, group := range groups { + sort.Slice(group, func(i, j int) bool { + pi, pj := strings.TrimSpace(group[i].Prefix) != "", strings.TrimSpace(group[j].Prefix) != "" + if pi != pj { + return !pi + } + return group[i].Index < group[j].Index + }) + for _, auth := range group[1:] { + duplicates[auth.Index] = true + } + } + return duplicates +} + func bearerHeaders() map[string]string { return map[string]string{"Authorization": "Bearer $TOKEN$", "Content-Type": "application/json", "Accept": "application/json"} } diff --git a/internal/quotahistory/recorder.go b/internal/quotahistory/recorder.go index 9f8a9b80aba..0c501ce939d 100644 --- a/internal/quotahistory/recorder.go +++ b/internal/quotahistory/recorder.go @@ -136,19 +136,24 @@ type Recorder struct { plansMu sync.Mutex plans map[string]string profileAttempts map[string]time.Time + // customers maps a Mistral key's auth index to the customer behind it. + customers map[string]string + customerAttempts map[string]time.Time } // NewRecorder creates a stopped recorder. Call Apply to start it. func NewRecorder(deps Deps) *Recorder { return &Recorder{ - deps: deps, - now: time.Now, - scheduledGap: defaultScheduledGap, - manualGap: defaultManualGap, - startupDelay: defaultStartupDelay, - settings: Settings{Interval: DefaultInterval, Retention: DefaultRetention}, - plans: make(map[string]string), - profileAttempts: make(map[string]time.Time), + deps: deps, + now: time.Now, + scheduledGap: defaultScheduledGap, + manualGap: defaultManualGap, + startupDelay: defaultStartupDelay, + settings: Settings{Interval: DefaultInterval, Retention: DefaultRetention}, + plans: make(map[string]string), + profileAttempts: make(map[string]time.Time), + customers: make(map[string]string), + customerAttempts: make(map[string]time.Time), } } @@ -353,6 +358,9 @@ func (r *Recorder) poll(ctx context.Context, store *Store, gap time.Duration, co // Shutting down or reconfiguring: do not record a cancellation as an error. return stored } + if providerOf(auth) == "mistral" && r.mistralDuplicates(r.trackable(r.pluginProviders(ctx, false)))[auth.Index] { + continue + } if errPut := store.put(rec); errPut != nil { log.WithError(errPut).Warn("quota-history: store sample") continue @@ -579,7 +587,15 @@ func (r *Recorder) History(since, until int64) (History, error) { since = until - DefaultRange.Milliseconds() } out.Accounts = r.accounts() + listed := make(map[string]bool, len(out.Accounts)) + for _, account := range out.Accounts { + listed[account.AuthIndex] = true + } + duplicates := r.mistralDuplicates(r.trackable(r.pluginProviders(context.Background(), false))) errScan := store.scan(since, until, func(rec record) { + if duplicates[rec.AuthIndex] && !listed[rec.AuthIndex] { + return + } if rec.Windows == nil { rec.Windows = map[string]Window{} } @@ -630,7 +646,12 @@ func downsample(samples []Sample, max int) []Sample { // accounts lists supported subscription credentials, including disabled ones. func (r *Recorder) accounts() []Account { out := []Account{} - for _, auth := range r.trackable(r.pluginProviders(context.Background(), false)) { + auths := r.trackable(r.pluginProviders(context.Background(), false)) + duplicates := r.mistralDuplicates(auths) + for _, auth := range auths { + if duplicates[auth.Index] { + continue + } plan := r.plan(auth.Index) if plan == "" { plan = coreauth.PlanOf(auth) diff --git a/internal/registry/model_registry.go b/internal/registry/model_registry.go index 0cd694b2b22..0363f790464 100644 --- a/internal/registry/model_registry.go +++ b/internal/registry/model_registry.go @@ -1845,6 +1845,29 @@ func (r *ModelRegistry) GetAvailableModelsByProvider(provider string) []*ModelIn return result } +// GetAvailableModelsForProvider returns a provider's catalog in the requested API format. +func (r *ModelRegistry) GetAvailableModelsForProvider(provider, handlerType string) []map[string]any { + infos := r.GetAvailableModelsByProvider(provider) + sort.Slice(infos, func(i, j int) bool { return infos[i].ID < infos[j].ID }) + models := make([]map[string]any, 0, len(infos)) + for _, info := range infos { + models = append(models, r.convertModelToMap(info, handlerType)) + } + return models +} + +// HasProvider reports whether any registered client supplies this provider, including suspended clients. +func (r *ModelRegistry) HasProvider(provider string) bool { + r.mutex.RLock() + defer r.mutex.RUnlock() + for _, registered := range r.clientProviders { + if strings.EqualFold(registered, provider) { + return true + } + } + return false +} + // GetModelCount returns the number of available clients for a specific model // Parameters: // - modelID: The model ID to check diff --git a/internal/runtime/executor/codex_websockets_duplex.go b/internal/runtime/executor/codex_websockets_duplex.go index 3a6a0ef8b56..fc2523cd7ef 100644 --- a/internal/runtime/executor/codex_websockets_duplex.go +++ b/internal/runtime/executor/codex_websockets_duplex.go @@ -1032,7 +1032,7 @@ func (e *CodexWebsocketsExecutor) streamCodexDuplex( credentialErr = streamErr } var status interface{ StatusCode() int } - if errors.As(credentialErr, &status) && (status.StatusCode() == http.StatusUnauthorized || status.StatusCode() == http.StatusForbidden || status.StatusCode() == http.StatusTooManyRequests) { + if errors.As(credentialErr, &status) && (status.StatusCode() == http.StatusUnauthorized || status.StatusCode() == http.StatusForbidden || status.StatusCode() == http.StatusTooManyRequests || isCodexOverloadBootstrapFailure(payload)) { // Account health is independent of which queued request failed. // The conductor records the original classification without replaying // this already-started stream on another credential. diff --git a/internal/watcher/config_reload_ordering_test.go b/internal/watcher/config_reload_ordering_test.go index 659313ac6a1..99ba6bc8983 100644 --- a/internal/watcher/config_reload_ordering_test.go +++ b/internal/watcher/config_reload_ordering_test.go @@ -180,3 +180,49 @@ func TestReloadConfigIfChangedReloadsAfterWriteBetweenHashAndLoad_Issue6631(t *t t.Fatalf("applied request-retry values = %v, want %v", applied, want) } } + +func TestReloadConfigIfChangedKeepsConfigDuringTruncatedWrite_Issue6559(t *testing.T) { + for _, content := range []string{"", " \n"} { + t.Run(fmt.Sprintf("content=%q", content), func(t *testing.T) { + configPath, authDir := newIssue6631ConfigFile(t) + if errWrite := writeIssue6631Config(configPath, authDir, 2); errWrite != nil { + t.Fatal(errWrite) + } + originalLoad := loadConfigFileFunc + t.Cleanup(func() { loadConfigFileFunc = originalLoad }) + loadConfigFileFunc = func(path string) (*config.Config, error) { + // A management save truncates the file after the nonempty hash read, + // before LoadConfig opens it for the actual parse. + if errWrite := os.WriteFile(path, []byte(content), 0o600); errWrite != nil { + return nil, errWrite + } + return originalLoad(path) + } + initial := &config.Config{AuthDir: authDir, RequestRetry: 1} + applied := 0 + w := &Watcher{ + configPath: configPath, + authDir: authDir, + config: initial, + reloadCallback: func(*config.Config) { + applied++ + }, + } + w.ReloadConfigIfChanged() + if w.config != initial || applied != 0 || w.lastConfigHash != "" { + t.Fatalf("truncated write changed runtime config: request-retry=%d, callbacks=%d, hash=%q", w.config.RequestRetry, applied, w.lastConfigHash) + } + + // The completed save must still reload, including when its bytes match + // the first read that preceded truncation. + loadConfigFileFunc = originalLoad + if errWrite := writeIssue6631Config(configPath, authDir, 2); errWrite != nil { + t.Fatal(errWrite) + } + w.ReloadConfigIfChanged() + if w.config.RequestRetry != 2 || applied != 1 { + t.Fatalf("completed save was not applied: request-retry=%d, callbacks=%d", w.config.RequestRetry, applied) + } + }) + } +} diff --git a/sdk/api/handlers/claude/code_handlers.go b/sdk/api/handlers/claude/code_handlers.go index a0c889124ba..170b23496bd 100644 --- a/sdk/api/handlers/claude/code_handlers.go +++ b/sdk/api/handlers/claude/code_handlers.go @@ -160,7 +160,7 @@ func rewriteClaudeDDModelInBody(rawJSON []byte) []byte { func (h *ClaudeCodeAPIHandler) ClaudeModels(c *gin.Context) { cfg := h.CurrentConfig() disableCloaking := cfg != nil && cfg.ClaudeCode.DisableCloakingModelList - h.WriteModelListResponse(c, h.HandlerType(), claudemodels.BuildResponse(h.Models(), disableCloaking)) + h.WriteModelListResponse(c, h.HandlerType(), claudemodels.BuildResponse(h.ModelsForRequest(c, h.HandlerType()), disableCloaking)) } // handleNonStreamingResponse handles non-streaming content generation requests for Claude models. diff --git a/sdk/api/handlers/gemini/gemini_handlers.go b/sdk/api/handlers/gemini/gemini_handlers.go index 1323218e8f6..1ce927534a6 100644 --- a/sdk/api/handlers/gemini/gemini_handlers.go +++ b/sdk/api/handlers/gemini/gemini_handlers.go @@ -49,7 +49,7 @@ func (h *GeminiAPIHandler) Models() []map[string]any { // GeminiModels handles the Gemini models listing endpoint. // It returns a JSON response containing available Gemini models and their specifications. func (h *GeminiAPIHandler) GeminiModels(c *gin.Context) { - rawModels := h.Models() + rawModels := h.ModelsForRequest(c, h.HandlerType()) normalizedModels := make([]map[string]any, 0, len(rawModels)) defaultMethods := []string{"generateContent"} for _, model := range rawModels { @@ -96,7 +96,7 @@ func (h *GeminiAPIHandler) GeminiGetHandler(c *gin.Context) { action := strings.TrimPrefix(request.Action, "/") // Get dynamic models from the global registry and find the matching one - availableModels := h.Models() + availableModels := h.ModelsForRequest(c, h.HandlerType()) var targetModel map[string]any for _, model := range availableModels { diff --git a/sdk/api/handlers/handlers.go b/sdk/api/handlers/handlers.go index b66467507b7..b0e83ca2bba 100644 --- a/sdk/api/handlers/handlers.go +++ b/sdk/api/handlers/handlers.go @@ -303,6 +303,9 @@ func requestExecutionMetadata(ctx context.Context) map[string]any { } meta := make(map[string]any) + if provider := coreauth.ProviderScopeFromContext(ctx); provider != "" { + meta[coreexecutor.ProviderScopeMetadataKey] = provider + } if key != "" { meta[idempotencyKeyMetadataKey] = key } @@ -312,6 +315,9 @@ func requestExecutionMetadata(ctx context.Context) map[string]any { if pinnedAuthID := pinnedAuthIDFromContext(ctx); pinnedAuthID != "" { meta[coreexecutor.PinnedAuthMetadataKey] = pinnedAuthID } + if excludedAuthIDs := excludedAuthIDsFromContext(ctx); len(excludedAuthIDs) > 0 { + meta[coreexecutor.ExcludedAuthIDsMetadataKey] = excludedAuthIDs + } if selectedCallback := selectedAuthIDCallbackFromContext(ctx); selectedCallback != nil { meta[coreexecutor.SelectedAuthCallbackMetadataKey] = selectedCallback } @@ -634,6 +640,9 @@ func (h *BaseAPIHandler) GetContextWithCancel(handler interfaces.APIHandler, c * } if requestCtx != nil { + if provider := coreauth.ProviderScopeFromContext(requestCtx); provider != "" { + parentCtx = coreauth.WithProviderScope(parentCtx, provider) + } parentCtx = httptransport.InheritDecorator(parentCtx, requestCtx) parentCtx = logging.InheritDeferredCapture(parentCtx, requestCtx) } diff --git a/sdk/api/handlers/handlers_context.go b/sdk/api/handlers/handlers_context.go index a986a040456..19c413454b1 100644 --- a/sdk/api/handlers/handlers_context.go +++ b/sdk/api/handlers/handlers_context.go @@ -12,6 +12,8 @@ import ( type pinnedAuthContextKey struct{} +type excludedAuthIDsContextKey struct{} + type selectedAuthCallbackContextKey struct{} type finalizedResponseFormatCallbackContextKey struct{} @@ -81,6 +83,33 @@ func WithPinnedAuthID(ctx context.Context, authID string) context.Context { return context.WithValue(ctx, pinnedAuthContextKey{}, authID) } +// WithExcludedAuthIDs returns a child context that excludes specific auth IDs from selection. +func WithExcludedAuthIDs(ctx context.Context, authIDs []string) context.Context { + if len(authIDs) == 0 { + return ctx + } + filtered := make([]string, 0, len(authIDs)) + seen := make(map[string]struct{}, len(authIDs)) + for _, authID := range authIDs { + trimmed := strings.TrimSpace(authID) + if trimmed == "" { + continue + } + if _, exists := seen[trimmed]; exists { + continue + } + seen[trimmed] = struct{}{} + filtered = append(filtered, trimmed) + } + if len(filtered) == 0 { + return ctx + } + if ctx == nil { + ctx = context.Background() + } + return context.WithValue(ctx, excludedAuthIDsContextKey{}, filtered) +} + // WithSelectedAuthIDCallback returns a child context that receives the selected auth ID. func WithSelectedAuthIDCallback(ctx context.Context, callback func(string)) context.Context { if callback == nil { @@ -217,6 +246,18 @@ func pinnedAuthIDFromContext(ctx context.Context) string { } } +func excludedAuthIDsFromContext(ctx context.Context) []string { + if ctx == nil { + return nil + } + raw := ctx.Value(excludedAuthIDsContextKey{}) + ids, ok := raw.([]string) + if !ok || len(ids) == 0 { + return nil + } + return append([]string(nil), ids...) +} + func selectedAuthIDCallbackFromContext(ctx context.Context) func(string) { if ctx == nil { return nil diff --git a/sdk/api/handlers/handlers_execution.go b/sdk/api/handlers/handlers_execution.go index 7bb8234ccd7..faf701a13b3 100644 --- a/sdk/api/handlers/handlers_execution.go +++ b/sdk/api/handlers/handlers_execution.go @@ -56,7 +56,7 @@ func (h *BaseAPIHandler) executeWithAuthManagerFormats(ctx context.Context, entr if routeDecision.ExecutorPluginID != "" { return h.executeWithPluginExecutor(ctx, entryProtocol, responseProtocol, modelName, originalRequestedModel, rawJSON, alt, routeDecision.ExecutorPluginID, execOptions) } - providers, normalizedModel, errMsg := h.providersForExecution(modelName, originalRequestedModel, allowImageModel, routeDecision, execOptions) + providers, normalizedModel, errMsg := h.providersForExecution(ctx, modelName, originalRequestedModel, allowImageModel, routeDecision, execOptions) if errMsg != nil { return nil, nil, errMsg } @@ -136,7 +136,7 @@ func (h *BaseAPIHandler) executeCountWithAuthManager(ctx context.Context, handle if routeDecision.ExecutorPluginID != "" { return h.countWithPluginExecutor(ctx, handlerType, modelName, originalRequestedModel, rawJSON, alt, routeDecision.ExecutorPluginID, execOptions) } - providers, normalizedModel, errMsg := h.providersForExecution(modelName, originalRequestedModel, false, routeDecision, execOptions) + providers, normalizedModel, errMsg := h.providersForExecution(ctx, modelName, originalRequestedModel, false, routeDecision, execOptions) if errMsg != nil { return nil, nil, errMsg } diff --git a/sdk/api/handlers/handlers_model_router_test.go b/sdk/api/handlers/handlers_model_router_test.go index b2ad1f76f6a..76bbb0482f7 100644 --- a/sdk/api/handlers/handlers_model_router_test.go +++ b/sdk/api/handlers/handlers_model_router_test.go @@ -575,7 +575,7 @@ func TestExecuteModelPropagatesRouterSkipPluginID(t *testing.T) { func TestHandlerProvidersForExecutionUsesRouterProvider(t *testing.T) { handler := NewBaseAPIHandlers(&sdkconfig.SDKConfig{}, nil) decision := modelRouteDecision{Provider: "claude", Model: "claude-sonnet-4"} - providers, normalizedModel, errMsg := handler.providersForExecution("ignored-by-router", "original-model", false, decision, modelExecutionOptions{}) + providers, normalizedModel, errMsg := handler.providersForExecution(context.Background(), "ignored-by-router", "original-model", false, decision, modelExecutionOptions{}) if errMsg != nil { t.Fatalf("providersForExecution() error = %+v", errMsg) } @@ -590,7 +590,7 @@ func TestHandlerProvidersForExecutionUsesRouterProvider(t *testing.T) { func TestHandlerProvidersForExecutionFallsBackToOriginalModel(t *testing.T) { handler := NewBaseAPIHandlers(&sdkconfig.SDKConfig{}, nil) decision := modelRouteDecision{Provider: "claude"} - providers, normalizedModel, errMsg := handler.providersForExecution("ignored-by-router", "original-model", false, decision, modelExecutionOptions{}) + providers, normalizedModel, errMsg := handler.providersForExecution(context.Background(), "ignored-by-router", "original-model", false, decision, modelExecutionOptions{}) if errMsg != nil { t.Fatalf("providersForExecution() error = %+v", errMsg) } @@ -682,7 +682,7 @@ func TestHandlerProvidersForExecutionRejectsImageOnlyModelOnProviderRoute(t *tes } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - _, _, errMsg := handler.providersForExecution("ignored", tc.originalModel, false, tc.decision, modelExecutionOptions{}) + _, _, errMsg := handler.providersForExecution(context.Background(), "ignored", tc.originalModel, false, tc.decision, modelExecutionOptions{}) if errMsg == nil || errMsg.StatusCode != http.StatusServiceUnavailable { t.Fatalf("providersForExecution() error = %+v, want image-only service unavailable", errMsg) } diff --git a/sdk/api/handlers/handlers_routing.go b/sdk/api/handlers/handlers_routing.go index 3d6c5daa758..9d36972c0fc 100644 --- a/sdk/api/handlers/handlers_routing.go +++ b/sdk/api/handlers/handlers_routing.go @@ -10,8 +10,10 @@ import ( . "github.com/rune-oai/t3code-proxy/internal/constant" "github.com/rune-oai/t3code-proxy/internal/interfaces" + "github.com/rune-oai/t3code-proxy/internal/registry" "github.com/rune-oai/t3code-proxy/internal/thinking" "github.com/rune-oai/t3code-proxy/internal/util" + coreauth "github.com/rune-oai/t3code-proxy/sdk/cliproxy/auth" coreexecutor "github.com/rune-oai/t3code-proxy/sdk/cliproxy/executor" "github.com/rune-oai/t3code-proxy/sdk/pluginapi" "golang.org/x/net/context" @@ -98,7 +100,7 @@ func excludeExecutionProvider(providers []string, excluded string) []string { } func (h *BaseAPIHandler) getRequestDetails(modelName string) (providers []string, normalizedModel string, err *interfaces.ErrorMessage) { - return h.getRequestDetailsWithOptions(modelName, false, false) + return h.getRequestDetailsWithOptions(context.Background(), modelName, false, false) } func validateNativeInteractionsExecution(entryProtocol string, execOptions modelExecutionOptions, routeDecision modelRouteDecision) *interfaces.ErrorMessage { @@ -125,7 +127,10 @@ func nativeInteractionsExecutionError() *interfaces.ErrorMessage { // providersForExecution resolves the providers and normalized model for a request. When a model // router selected a built-in provider, it skips model->provider resolution and uses the router's // provider (with an optional target model); otherwise it falls back to the registry-based path. -func (h *BaseAPIHandler) providersForExecution(modelName, originalRequestedModel string, allowImageModel bool, routeDecision modelRouteDecision, execOptions modelExecutionOptions) ([]string, string, *interfaces.ErrorMessage) { +func (h *BaseAPIHandler) providersForExecution(ctx context.Context, modelName, originalRequestedModel string, allowImageModel bool, routeDecision modelRouteDecision, execOptions modelExecutionOptions) ([]string, string, *interfaces.ErrorMessage) { + if coreauth.ProviderScopeFromContext(ctx) != "" { + return h.getRequestDetailsWithOptions(ctx, modelName, allowImageModel, execOptions.AllowSpeechModel) + } forcedProvider := strings.ToLower(strings.TrimSpace(execOptions.ForcedProvider)) if forcedProvider != "" { if routeDecision.ExecutorPluginID != "" { @@ -159,13 +164,24 @@ func (h *BaseAPIHandler) providersForExecution(modelName, originalRequestedModel } return []string{routeDecision.Provider}, normalizedModel, nil } - return h.getRequestDetailsWithOptions(modelName, allowImageModel, execOptions.AllowSpeechModel) + return h.getRequestDetailsWithOptions(ctx, modelName, allowImageModel, execOptions.AllowSpeechModel) } -func (h *BaseAPIHandler) getRequestDetailsWithOptions(modelName string, allowImageModel, allowSpeechModel bool) (providers []string, normalizedModel string, err *interfaces.ErrorMessage) { +func (h *BaseAPIHandler) getRequestDetailsWithOptions(ctx context.Context, modelName string, allowImageModel, allowSpeechModel bool) (providers []string, normalizedModel string, err *interfaces.ErrorMessage) { resolvedModelName := modelName initialSuffix := thinking.ParseSuffix(modelName) - if initialSuffix.ModelName == "auto" { + scope := coreauth.ProviderScopeFromContext(ctx) + if scope != "" { + if initialSuffix.ModelName == "auto" { + models := registry.GetGlobalRegistry().GetAvailableModelsForProvider(scope, "openai") + if len(models) > 0 { + resolvedModelName, _ = models[0]["id"].(string) + if initialSuffix.HasSuffix { + resolvedModelName += "(" + initialSuffix.RawSuffix + ")" + } + } + } + } else if initialSuffix.ModelName == "auto" { if h != nil && h.AuthManager != nil && h.AuthManager.HomeEnabled() { resolvedModelName = modelName } else { @@ -187,17 +203,6 @@ func (h *BaseAPIHandler) getRequestDetailsWithOptions(modelName string, allowIma parsed := thinking.ParseSuffix(resolvedModelName) baseModel := strings.TrimSpace(parsed.ModelName) - if errMsg := h.validateImageOnlyModel(baseModel, allowImageModel); errMsg != nil { - return nil, "", errMsg - } - if errMsg := h.validateSpeechOnlyModel(baseModel, allowSpeechModel); errMsg != nil { - return nil, "", errMsg - } - - if h != nil && h.AuthManager != nil && h.AuthManager.HomeEnabled() { - return []string{"home"}, resolvedModelName, nil - } - providers = util.GetProviderName(baseModel) // Fallback: if baseModel has no provider but differs from resolvedModelName, // try using the full model name. This handles edge cases where custom models @@ -208,24 +213,30 @@ func (h *BaseAPIHandler) getRequestDetailsWithOptions(modelName string, allowIma providers = util.GetProviderName(resolvedModelName) } - if len(providers) == 0 { - // The client asked for a model this proxy cannot route. Report it as a request - // error so streaming clients receive an actionable message instead of a - // gateway failure they would keep retrying. 400 is used rather than 404 to keep - // it distinguishable from an unregistered HTTP route. - // The model name is client supplied, so it is inserted through sjson rather - // than formatted into the JSON literal: an unescaped quote would otherwise - // corrupt the body or let the caller overwrite the error code. - body := `{"error":{"message":"","type":"invalid_request_error","code":"model_not_found","param":"model"}}` - body, errSet := ojson.Set(body, "error.message", "unknown provider for model "+modelName) - if errSet != nil { - body = `{"error":{"message":"unknown provider for model","type":"invalid_request_error","code":"model_not_found","param":"model"}}` - } - return nil, "", &interfaces.ErrorMessage{ - StatusCode: http.StatusBadRequest, - Error: errors.New(body), + if scope != "" { + providers = scopedModelProviders(providers, scope) + if len(providers) == 0 && baseModel != resolvedModelName { + providers = scopedModelProviders(util.GetProviderName(resolvedModelName), scope) } } + if scope != "" && len(providers) == 0 { + return nil, "", modelNotFoundError(modelName) + } + + if errMsg := h.validateImageOnlyModel(baseModel, allowImageModel); errMsg != nil { + return nil, "", errMsg + } + if errMsg := h.validateSpeechOnlyModel(baseModel, allowSpeechModel); errMsg != nil { + return nil, "", errMsg + } + + if h != nil && h.AuthManager != nil && h.AuthManager.HomeEnabled() && scope == "" { + return []string{"home"}, resolvedModelName, nil + } + + if len(providers) == 0 { + return nil, "", modelNotFoundError(modelName) + } // The thinking suffix is preserved in the model name itself, so no // metadata-based configuration passing is needed. @@ -361,6 +372,10 @@ func modelRoutersEnabled(host PluginModelRouterHost, skipPluginID string) bool { func (h *BaseAPIHandler) applyModelRouter(ctx context.Context, handlerType, modelName string, rawJSON []byte, stream bool, execOptions modelExecutionOptions) modelRouteDecision { var decision modelRouteDecision + // A provider scope uses the provider's registered public models and credentials. + if coreauth.ProviderScopeFromContext(ctx) != "" { + return decision + } host := h.modelRouterHost() if host == nil || !modelRoutersEnabled(host, execOptions.SkipRouterPluginID) { return decision @@ -390,3 +405,18 @@ func (h *BaseAPIHandler) applyModelRouter(ctx context.Context, handlerType, mode } return decision } + +func scopedModelProviders(providers []string, scope string) []string { + for _, provider := range providers { + if strings.EqualFold(provider, scope) { + return []string{scope} + } + } + return nil +} + +func modelNotFoundError(modelName string) *interfaces.ErrorMessage { + doc := ojson.NewDocString(`{"error":{"message":"unknown provider for model","type":"invalid_request_error","code":"model_not_found","param":"model"}}`) + _ = doc.Set("error.message", "unknown provider for model "+modelName) + return &interfaces.ErrorMessage{StatusCode: http.StatusBadRequest, Error: errors.New(doc.String())} +} diff --git a/sdk/api/handlers/handlers_speech_only_test.go b/sdk/api/handlers/handlers_speech_only_test.go index baa84e234cf..21273ace3cc 100644 --- a/sdk/api/handlers/handlers_speech_only_test.go +++ b/sdk/api/handlers/handlers_speech_only_test.go @@ -185,7 +185,7 @@ func TestHandlerProvidersForExecutionRejectsSpeechOnlyModelOnProviderRoute(t *te } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { - _, _, errMsg := handler.providersForExecution(tc.modelName, tc.originalModel, false, tc.decision, tc.execOptions) + _, _, errMsg := handler.providersForExecution(context.Background(), tc.modelName, tc.originalModel, false, tc.decision, tc.execOptions) requireSpeechOnlyError(t, errMsg) }) } diff --git a/sdk/api/handlers/handlers_stream.go b/sdk/api/handlers/handlers_stream.go index e328adc5560..2c65de2c5db 100644 --- a/sdk/api/handlers/handlers_stream.go +++ b/sdk/api/handlers/handlers_stream.go @@ -26,8 +26,15 @@ func (h *BaseAPIHandler) ExecuteStreamWithAuthManager(ctx context.Context, handl // HasAlternativeExecutionTarget reports whether an alternative credential other // than excludedAuthID can serve a replayed request for modelName. func (h *BaseAPIHandler) HasAlternativeExecutionTarget(ctx context.Context, handlerType, modelName string, rawJSON []byte, excludedAuthID string) bool { + available, _ := h.CheckAlternativeExecutionTarget(ctx, handlerType, modelName, rawJSON, excludedAuthID) + return available +} + +// CheckAlternativeExecutionTarget also returns the routing error that prevents +// replay, allowing the caller to expose the hub's cooldown to the client. +func (h *BaseAPIHandler) CheckAlternativeExecutionTarget(ctx context.Context, handlerType, modelName string, rawJSON []byte, excludedAuthID string) (bool, *interfaces.ErrorMessage) { if h == nil || h.AuthManager == nil { - return false + return false, nil } originalRequestedModel := modelName execOptions := modelExecutionOptions{ @@ -38,15 +45,16 @@ func (h *BaseAPIHandler) HasAlternativeExecutionTarget(ctx context.Context, hand routeDecision = h.applyModelRouter(ctx, handlerType, modelName, rawJSON, true, execOptions) } if routeDecision.ExecutorPluginID != "" { - return false + return false, nil } - providers, normalizedModel, errMsg := h.providersForExecution(modelName, originalRequestedModel, false, routeDecision, execOptions) + providers, normalizedModel, errMsg := h.providersForExecution(ctx, modelName, originalRequestedModel, false, routeDecision, execOptions) if errMsg != nil { - return false + return false, nil } providers = adjustExecutionProvidersForEntryProtocol(handlerType, providers) reqMeta := requestExecutionMetadata(ctx) delete(reqMeta, coreexecutor.PinnedAuthMetadataKey) + delete(reqMeta, coreexecutor.ExcludedAuthIDsMetadataKey) delete(reqMeta, coreexecutor.ExecutionSessionMetadataKey) delete(reqMeta, coreexecutor.SelectedAuthCallbackMetadataKey) delete(reqMeta, coreexecutor.FinalizedResponseFormatCallbackMetadataKey) @@ -70,7 +78,11 @@ func (h *BaseAPIHandler) HasAlternativeExecutionTarget(ctx context.Context, hand Metadata: reqMeta, } ctx = enrichContextWithSessionHierarchy(ctx, opts.Headers, rawJSON, opts.Metadata) - return h.AuthManager.HasAlternativeTargetForModel(ctx, providers, normalizedModel, opts, excludedAuthID) + available, err := h.AuthManager.CheckAlternativeTargetForModel(ctx, providers, normalizedModel, opts, excludedAuthID) + if err != nil { + return false, ExecutionErrorMessage(err) + } + return available, nil } // ExecuteImageStreamWithAuthManager executes a streaming OpenAI-compatible image endpoint request. @@ -351,7 +363,7 @@ func (h *BaseAPIHandler) executeStreamWithAuthManagerFormats(ctx context.Context if routeDecision.ExecutorPluginID != "" { return h.streamWithPluginExecutor(ctx, entryProtocol, responseProtocol, modelName, originalRequestedModel, rawJSON, alt, routeDecision.ExecutorPluginID, execOptions) } - providers, normalizedModel, errMsg := h.providersForExecution(modelName, originalRequestedModel, allowImageModel, routeDecision, execOptions) + providers, normalizedModel, errMsg := h.providersForExecution(ctx, modelName, originalRequestedModel, allowImageModel, routeDecision, execOptions) if errMsg != nil { errChan := make(chan *interfaces.ErrorMessage, 1) errChan <- errMsg diff --git a/sdk/api/handlers/model_execution_test.go b/sdk/api/handlers/model_execution_test.go index e0c7784f52f..a7a01bfd2df 100644 --- a/sdk/api/handlers/model_execution_test.go +++ b/sdk/api/handlers/model_execution_test.go @@ -763,7 +763,7 @@ func TestExecuteProtocolStreamWithAuthManagerAgentUsesSelectionModelForAuth(t *t func TestProvidersForExecutionForcedGeminiRejectsRouterProvider(t *testing.T) { handler := NewBaseAPIHandlers(&sdkconfig.SDKConfig{}, nil) decision := modelRouteDecision{Provider: "claude", Model: "claude-sonnet-4"} - _, _, errMsg := handler.providersForExecution("agents/test-agent", "agents/test-agent", false, decision, modelExecutionOptions{ForcedProvider: "gemini"}) + _, _, errMsg := handler.providersForExecution(context.Background(), "agents/test-agent", "agents/test-agent", false, decision, modelExecutionOptions{ForcedProvider: "gemini"}) if errMsg == nil { t.Fatal("providersForExecution() error = nil, want native interactions error") } @@ -777,7 +777,7 @@ func TestProvidersForExecutionForcedGeminiRejectsRouterProvider(t *testing.T) { func TestProvidersForExecutionForcedGeminiUsesGeminiProvider(t *testing.T) { handler := NewBaseAPIHandlers(&sdkconfig.SDKConfig{}, nil) - providers, model, errMsg := handler.providersForExecution("agents/test-agent", "agents/test-agent", false, modelRouteDecision{}, modelExecutionOptions{ForcedProvider: "gemini"}) + providers, model, errMsg := handler.providersForExecution(context.Background(), "agents/test-agent", "agents/test-agent", false, modelRouteDecision{}, modelExecutionOptions{ForcedProvider: "gemini"}) if errMsg != nil { t.Fatalf("providersForExecution() error = %+v", errMsg) } diff --git a/sdk/api/handlers/openai/codex_client_models.go b/sdk/api/handlers/openai/codex_client_models.go index 10ae60c328c..45464e5a5a0 100644 --- a/sdk/api/handlers/openai/codex_client_models.go +++ b/sdk/api/handlers/openai/codex_client_models.go @@ -7,6 +7,10 @@ import ( ) func (h *OpenAIAPIHandler) codexClientModelsResponse(clientVersion ...string) map[string]any { + return h.codexClientModelsResponseForModels(h.Models(), clientVersion...) +} + +func (h *OpenAIAPIHandler) codexClientModelsResponseForModels(models []map[string]any, clientVersion ...string) map[string]any { version := "" if len(clientVersion) > 0 { version = clientVersion[0] @@ -26,7 +30,7 @@ func (h *OpenAIAPIHandler) codexClientModelsResponse(clientVersion ...string) ma toolSearchCapabilityForModel = h.SupportsNativeToolSearchModel } modelRegistry := registry.GetGlobalRegistry() - return codexmodels.BuildResponseForClientWithToolCapabilities(h.Models(), modelRegistry.GetModelProviders, modelRegistry.GetResponsesWebSearchCapability, applyPatchCapabilityForModel, toolSearchCapabilityForModel, optimizeMultiAgentV2, version) + return codexmodels.BuildResponseForClientWithToolCapabilities(models, modelRegistry.GetModelProviders, modelRegistry.GetResponsesWebSearchCapability, applyPatchCapabilityForModel, toolSearchCapabilityForModel, optimizeMultiAgentV2, version) } // CodexClientModelsResponse builds a Codex client model response. diff --git a/sdk/api/handlers/openai/openai_handlers.go b/sdk/api/handlers/openai/openai_handlers.go index aa3e92bc397..a4cd1b5c5ba 100644 --- a/sdk/api/handlers/openai/openai_handlers.go +++ b/sdk/api/handlers/openai/openai_handlers.go @@ -65,7 +65,7 @@ func (h *OpenAIAPIHandler) Models() []map[string]any { func (h *OpenAIAPIHandler) OpenAIModels(c *gin.Context) { if _, ok := c.Request.URL.Query()["client_version"]; ok { clientVersion := c.Query("client_version") - body, errMarshal := codexmodels.MarshalCompact(h.codexClientModelsResponse(clientVersion)) + body, errMarshal := codexmodels.MarshalCompact(h.codexClientModelsResponseForModels(h.ModelsForRequest(c, h.HandlerType()), clientVersion)) if errMarshal != nil { c.JSON(http.StatusInternalServerError, handlers.ErrorResponse{ Error: handlers.ErrorDetail{ @@ -80,7 +80,7 @@ func (h *OpenAIAPIHandler) OpenAIModels(c *gin.Context) { } // Get all available models - allModels := h.Models() + allModels := h.ModelsForRequest(c, h.HandlerType()) // Filter to only include the 4 required fields: id, object, created, owned_by filteredModels := make([]map[string]any, len(allModels)) diff --git a/sdk/api/handlers/openai/openai_responses_websocket.go b/sdk/api/handlers/openai/openai_responses_websocket.go index 29a14ac8876..45e81a72785 100644 --- a/sdk/api/handlers/openai/openai_responses_websocket.go +++ b/sdk/api/handlers/openai/openai_responses_websocket.go @@ -190,7 +190,7 @@ func (w *responsesWebsocketWriter) writePing() error { } func (w *responsesWebsocketWriter) keepAlive(ctx context.Context, cancel context.CancelCauseFunc, intervals <-chan time.Duration) { - if errPing := runResponsesWebsocketKeepAlive(ctx, intervals, w.writePing); errPing != nil { + if errPing := runResponsesWebsocketKeepAlive(ctx, intervals, w.writePing); errPing != nil && !errors.Is(errPing, websocket.ErrCloseSent) { cancel(errPing) _, _ = w.closeWithoutError() } @@ -243,13 +243,14 @@ func (w *responsesWebsocketWriter) closeWithPayload(payload []byte) (bool, error if w == nil || w.conn == nil { return false, nil } + // A terminal frame must follow any active data or Ping write. Competing + // terminal callers also wait here, so their deferred connection teardown + // cannot overtake the winning terminal write. + w.writeMu.Lock() + defer w.writeMu.Unlock() if !w.closing.CompareAndSwap(false, true) { return false, nil } - if !w.writeMu.TryLock() { - return false, w.conn.Close() - } - defer w.writeMu.Unlock() errWrite := w.conn.WriteMessage(websocket.TextMessage, payload) errClose := w.conn.Close() @@ -932,62 +933,6 @@ func (h *OpenAIResponsesAPIHandler) ResponsesWebsocket(c *gin.Context) { } modelName := strings.Clone(ojson.GetBytesNoCopy(requestJSON, "model").String()) - lastAttemptedAuthID := pinnedAuthID - attemptedUpstreamMode := responsesWebsocketUpstreamModeUnknown - selectedAuthObserved := false - nativeRequest := util.IsCodexResponsesLiteRequest(payload, c.Request.Header) - var preserveNativeOutput atomic.Bool - codexDuplexStream.Store(false) - pinnedAuthAttempted := false - turnStartEpoch := inputEpoch.Load() - cliCtx, cliCancel := h.GetContextWithCancel(h, c, executionParent) - cliCtx = cliproxyexecutor.WithDownstreamWebsocket(cliCtx) - cliCtx = cliproxyexecutor.WithLocalWebsocketTurnTracker(cliCtx) - cliCtx = cliproxyexecutor.WithWebsocketInputEpoch(cliCtx, &inputEpoch, msgEpoch) - if duplexInput != nil { - cliCtx = cliproxyexecutor.WithWebsocketInput(cliCtx, duplexInput) - cliCtx = cliproxyexecutor.WithWebsocketAuthCheck(cliCtx, func(authID string) bool { - current, ok := sessionAuthByID(authID) - return ok && current != nil && !current.Disabled && current.Status != coreauth.StatusDisabled - }) - boundPrincipal := usage.APIKeyFromContext(cliCtx) - cliCtx = cliproxyexecutor.WithWebsocketTurnAuthorizer(cliCtx, func(turnPayload []byte) ([]byte, error) { - principal, errAuth := h.authenticateResponsesWebsocketTurn(c, upgradeRequest, turnPayload, modelName, false) - if errAuth == nil && principal != "" && principal != boundPrincipal { - errAuth = &interfaces.ErrorMessage{ - StatusCode: http.StatusUnauthorized, - Error: errors.New("client credential changed during the websocket stream; reconnect"), - } - } - if errAuth == nil { - return nil, nil - } - return buildResponsesWebsocketErrorPayload(errAuth) - }) - } - if nativeWebsocketPassthrough && requestRequiresCurrentUpstreamWebsocket { - cliCtx = cliproxyexecutor.WithRequiredUpstreamWebsocket(cliCtx) - } - cliCtx = handlers.WithExecutionSessionID(cliCtx, passthroughSessionID) - cliCtx = handlers.WithSelectedAuthIDCallback(cliCtx, func(authID string) { - preserveNativeOutput.Store(false) - codexDuplexStream.Store(false) - cliproxyexecutor.ResetLocalWebsocketTurn(cliCtx) - authID = strings.TrimSpace(authID) - if authID == "" || h == nil || h.AuthManager == nil { - return - } - lastAttemptedAuthID = authID - selectedAuthObserved = true - pinnedAuthAttempted = pinnedAuthAttempted || (pinnedAuthID != "" && authID == pinnedAuthID) - selectedAuth, ok := sessionAuthByID(authID) - if !ok || selectedAuth == nil { - return - } - attemptedUpstreamMode = upstreamModeForAuth(selectedAuth) - codexDuplexStream.Store(duplexInput != nil && attemptedUpstreamMode == responsesWebsocketUpstreamModeWS && strings.EqualFold(strings.TrimSpace(selectedAuth.Provider), "codex")) - preserveNativeOutput.Store(nativeRequest && strings.EqualFold(strings.TrimSpace(selectedAuth.Provider), "codex")) - }) executionAuthID := "" if !routeOverridesModelResolution { executionAuthID = pinnedAuthID @@ -995,112 +940,249 @@ func (h *OpenAIResponsesAPIHandler) ResponsesWebsocket(c *gin.Context) { if executionAuthID == "" { executionAuthID = observedCompactionReplayAuthID } - if executionAuthID != "" && !isPluginExecutorRoute { - cliCtx = handlers.WithPinnedAuthID(cliCtx, executionAuthID) - } - dataChan, _, errChan := h.ExecuteStreamWithAuthManager(cliCtx, h.HandlerType(), modelName, requestJSON, "") - if !selectedAuthObserved { - // Plugin/alternate routes bypass auth selection. Keep canonical HTTP-mode - // state instead of inheriting the previous pinned websocket mode. - attemptedUpstreamMode = responsesWebsocketUpstreamModeHTTP - } - // A connection-scoped continuation cannot rotate credentials in place. Suppress - // credential errors and make the client replay the full turn on a new socket - // when another eligible target can serve the replayed turn. - var lossWriteErr error - var replayedPinnedAuthFailure bool - replayPinnedAuthFailure := func(errMsg *interfaces.ErrorMessage) bool { - if cliproxyexecutor.IsUpstreamWebsocketTurnError(errMsg.Error) { - // The upstream websocket was lost: tell the client, keep its socket open, and - // fence the input it sent before the loss. A refused continuation is not a loss. - var turnErr *cliproxyexecutor.UpstreamWebsocketTurnError - if errors.As(errMsg.Error, &turnErr) { - if (turnErr.FullContextRequired || turnErr.NotSent || !codexDuplexStream.Load()) && - turnErr.EventID == "" && !turnErr.Idle && len(turnErr.Abandoned) == 0 { - turnErr.EventID = strings.Clone(strings.TrimSpace(ojson.GetBytesNoCopy(payload, "event_id").String())) + + var excludedAuthIDs []string + retriedPinnedFailover := false + var lastAttemptedAuthID string + var attemptedUpstreamMode string + var selectedAuthObserved bool + var preserveNativeOutput atomic.Bool + var pinnedAuthAttempted bool + var cliCtx context.Context + var completedOutput []byte + var completedResponseID string + var completedPendingToolCallIDs []string + var forwardErrMsg *interfaces.ErrorMessage + + for { + lastAttemptedAuthID = pinnedAuthID + attemptedUpstreamMode = responsesWebsocketUpstreamModeUnknown + selectedAuthObserved = false + nativeRequest := util.IsCodexResponsesLiteRequest(payload, c.Request.Header) + preserveNativeOutput.Store(false) + codexDuplexStream.Store(false) + pinnedAuthAttempted = false + turnStartEpoch := inputEpoch.Load() + var cliCancel handlers.APIHandlerCancelFunc + cliCtx, cliCancel = h.GetContextWithCancel(h, c, executionParent) + cliCtx = cliproxyexecutor.WithDownstreamWebsocket(cliCtx) + cliCtx = cliproxyexecutor.WithLocalWebsocketTurnTracker(cliCtx) + cliCtx = cliproxyexecutor.WithWebsocketInputEpoch(cliCtx, &inputEpoch, msgEpoch) + if duplexInput != nil { + cliCtx = cliproxyexecutor.WithWebsocketInput(cliCtx, duplexInput) + cliCtx = cliproxyexecutor.WithWebsocketAuthCheck(cliCtx, func(authID string) bool { + current, ok := sessionAuthByID(authID) + return ok && current != nil && !current.Disabled && current.Status != coreauth.StatusDisabled + }) + boundPrincipal := usage.APIKeyFromContext(cliCtx) + cliCtx = cliproxyexecutor.WithWebsocketTurnAuthorizer(cliCtx, func(turnPayload []byte) ([]byte, error) { + principal, errAuth := h.authenticateResponsesWebsocketTurn(c, upgradeRequest, turnPayload, modelName, false) + if errAuth == nil && principal != "" && principal != boundPrincipal { + errAuth = &interfaces.ErrorMessage{ + StatusCode: http.StatusUnauthorized, + Error: errors.New("client credential changed during the websocket stream; reconnect"), + } } - if !turnErr.FullContextRequired && !turnErr.NotSent && inputEpoch.Load() == turnStartEpoch { - inputEpoch.Add(1) + if errAuth == nil { + return nil, nil } + return buildResponsesWebsocketErrorPayload(errAuth) + }) + } + if nativeWebsocketPassthrough && requestRequiresCurrentUpstreamWebsocket { + cliCtx = cliproxyexecutor.WithRequiredUpstreamWebsocket(cliCtx) + } + cliCtx = handlers.WithExecutionSessionID(cliCtx, passthroughSessionID) + cliCtx = handlers.WithSelectedAuthIDCallback(cliCtx, func(authID string) { + preserveNativeOutput.Store(false) + codexDuplexStream.Store(false) + cliproxyexecutor.ResetLocalWebsocketTurn(cliCtx) + authID = strings.TrimSpace(authID) + if authID == "" || h == nil || h.AuthManager == nil { + return } - lossWriteErr = writeResponsesWebsocketUpstreamLoss(writer, wsTimelineLog, errMsg) - return true + lastAttemptedAuthID = authID + selectedAuthObserved = true + pinnedAuthAttempted = pinnedAuthAttempted || (pinnedAuthID != "" && authID == pinnedAuthID) + selectedAuth, ok := sessionAuthByID(authID) + if !ok || selectedAuth == nil { + return + } + attemptedUpstreamMode = upstreamModeForAuth(selectedAuth) + codexDuplexStream.Store(duplexInput != nil && attemptedUpstreamMode == responsesWebsocketUpstreamModeWS && strings.EqualFold(strings.TrimSpace(selectedAuth.Provider), "codex")) + preserveNativeOutput.Store(nativeRequest && strings.EqualFold(strings.TrimSpace(selectedAuth.Provider), "codex")) + }) + if executionAuthID != "" && !isPluginExecutorRoute { + cliCtx = handlers.WithPinnedAuthID(cliCtx, executionAuthID) } - if !nativeWebsocketPassthrough || !requestRequiresCurrentUpstreamWebsocket || !pinnedAuthAttempted || - !shouldReplayResponsesWebsocketPinnedAuthFailure(errMsg) { - return false + if len(excludedAuthIDs) > 0 { + cliCtx = handlers.WithExcludedAuthIDs(cliCtx, excludedAuthIDs) } - if (responsesWebsocketErrorStatus(errMsg) == http.StatusTooManyRequests || isResponsesWebsocketQuotaOrCooldownError(errMsg.Error)) && - !h.responsesWebsocketHasAlternativeReplayTarget(cliCtx, modelName, requestJSON, lastAttemptedAuthID) { - return false + dataChan, _, errChan := h.ExecuteStreamWithAuthManager(cliCtx, h.HandlerType(), modelName, requestJSON, "") + if !selectedAuthObserved { + // Plugin/alternate routes bypass auth selection. Keep canonical HTTP-mode + // state instead of inheriting the previous pinned websocket mode. + attemptedUpstreamMode = responsesWebsocketUpstreamModeHTTP + } + // A connection-scoped continuation cannot rotate credentials in place. Suppress + // credential errors and make the client replay the full turn on a new socket + // when another eligible target can serve the replayed turn. For a self-contained + // full response.create that fails before committing any downstream output, unpin + // and fail over in place on the existing downstream socket. + var lossWriteErr error + var replayedPinnedAuthFailure bool + var failoverInPlace bool + var downstreamCommitted atomic.Bool + replayPinnedAuthFailure := func(errMsg *interfaces.ErrorMessage) bool { + if cliproxyexecutor.IsUpstreamWebsocketTurnError(errMsg.Error) { + // The upstream websocket was lost: tell the client, keep its socket open, and + // fence the input it sent before the loss. A refused continuation is not a loss. + var turnErr *cliproxyexecutor.UpstreamWebsocketTurnError + if errors.As(errMsg.Error, &turnErr) { + if (turnErr.FullContextRequired || turnErr.NotSent || !codexDuplexStream.Load()) && + turnErr.EventID == "" && !turnErr.Idle && len(turnErr.Abandoned) == 0 { + turnErr.EventID = strings.Clone(strings.TrimSpace(ojson.GetBytesNoCopy(payload, "event_id").String())) + } + if !turnErr.FullContextRequired && !turnErr.NotSent && inputEpoch.Load() == turnStartEpoch { + inputEpoch.Add(1) + } + } + lossWriteErr = writeResponsesWebsocketUpstreamLoss(writer, wsTimelineLog, errMsg) + return true + } + if !pinnedAuthAttempted && lastAttemptedAuthID != "" && (codexDuplexStream.Load() || !downstreamCommitted.Load()) { + // Initial failures and later duplex creates share the dispatch + // selection. Their terminal error has already recorded its hold. + _, routeErr := h.responsesWebsocketAlternativeReplayTarget(cliCtx, modelName, requestJSON, lastAttemptedAuthID) + if routeErr != nil && isResponsesWebsocketQuotaOrCooldownError(routeErr.Error) { + *errMsg = *routeErr + } + return false + } + if !nativeWebsocketPassthrough || !pinnedAuthAttempted { + return false + } + quotaFailure := responsesWebsocketErrorStatus(errMsg) == http.StatusTooManyRequests || isResponsesWebsocketQuotaOrCooldownError(errMsg.Error) + replayableFailure := shouldReplayResponsesWebsocketPinnedAuthFailure(errMsg) || quotaFailure + if !replayableFailure && !shouldReleaseResponsesWebsocketPinnedAuth(errMsg) { + return false + } + if !requestRequiresCurrentUpstreamWebsocket && (retriedPinnedFailover || downstreamCommitted.Load()) { + return false + } + available, routeErr := h.responsesWebsocketAlternativeReplayTarget(cliCtx, modelName, requestJSON, lastAttemptedAuthID) + if routeErr != nil && isResponsesWebsocketQuotaOrCooldownError(routeErr.Error) { + // The same quota-aware selection rules govern replay and ordinary + // retries. Expose the hub's short wait when only spent accounts remain. + *errMsg = *routeErr + return false + } + if !replayableFailure { + return false + } + if !requestRequiresCurrentUpstreamWebsocket { + if !available { + return false + } + failoverInPlace = true + return true + } + if quotaFailure && !available { + return false + } + replayedPinnedAuthFailure = true + return true } - replayedPinnedAuthFailure = true - return true - } - completedOutput, completedResponseID, completedPendingToolCallIDs, forwardErrMsg, errForward := h.forwardResponsesWebsocket( - c, - writer, - cliCancel, - dataChan, - errChan, - wsTimelineLog, - passthroughSessionID, - responsesWebsocketForwardOptions{ - preserveCompletionOutput: preserveNativeOutput.Load, - duplexStream: func() bool { - return codexDuplexStream.Load() && !cliproxyexecutor.LocalWebsocketTurnExecuted(cliCtx) - }, - toolCacheTurn: toolCacheTurn, - suppressError: replayPinnedAuthFailure, - localInterrupt: localInterrupt, - websocketOwned: func() bool { - return attemptedUpstreamMode == responsesWebsocketUpstreamModeWS && !cliproxyexecutor.LocalWebsocketTurnExecuted(cliCtx) + var errForward error + completedOutput, completedResponseID, completedPendingToolCallIDs, forwardErrMsg, errForward = h.forwardResponsesWebsocket( + c, + writer, + cliCancel, + dataChan, + errChan, + wsTimelineLog, + passthroughSessionID, + responsesWebsocketForwardOptions{ + preserveCompletionOutput: preserveNativeOutput.Load, + duplexStream: func() bool { + return codexDuplexStream.Load() && !cliproxyexecutor.LocalWebsocketTurnExecuted(cliCtx) + }, + toolCacheTurn: toolCacheTurn, + suppressError: replayPinnedAuthFailure, + onDownstreamCommit: func() { downstreamCommitted.Store(true) }, + localInterrupt: localInterrupt, + websocketOwned: func() bool { + return attemptedUpstreamMode == responsesWebsocketUpstreamModeWS && !cliproxyexecutor.LocalWebsocketTurnExecuted(cliCtx) + }, }, - }, - ) - if errForward != nil { - wsTerminateErr = errForward - switch { - case errors.Is(errForward, websocket.ErrCloseSent): - case isWebsocketConnectionClosedError(errForward): - // The client hung up while a downstream write was in flight. This is a - // normal shutdown race, not a proxy failure. - log.Debugf("responses websocket: client closed during forward id=%s error=%v", passthroughSessionID, errForward) - default: - log.Warnf("responses websocket: forward failed id=%s error=%v", passthroughSessionID, errForward) - } - return - } - if forwardErrMsg != nil && cliproxyexecutor.IsUpstreamWebsocketTurnError(forwardErrMsg.Error) { - if lossWriteErr != nil { - wsTerminateErr = lossWriteErr + ) + cliCancel(nil) + if errForward != nil { + wsTerminateErr = errForward + switch { + case errors.Is(errForward, websocket.ErrCloseSent): + case isWebsocketConnectionClosedError(errForward): + // The client hung up while a downstream write was in flight. This is a + // normal shutdown race, not a proxy failure. + log.Debugf("responses websocket: client closed during forward id=%s error=%v", passthroughSessionID, errForward) + default: + log.Warnf("responses websocket: forward failed id=%s error=%v", passthroughSessionID, errForward) + } return } - // Keep the credential: the next full create reconnects on it, never on another one. - if (attemptedUpstreamMode == responsesWebsocketUpstreamModeWS || upstreamMode == responsesWebsocketUpstreamModeWS) && lastAttemptedAuthID != "" { - upstreamMode = responsesWebsocketUpstreamModeWS - upstreamWebsocketAuthID = lastAttemptedAuthID - rememberPinnedAuth(lastAttemptedAuthID, modelName) - passthroughModelName = modelName - } - continue - } - if forwardErrMsg != nil { - if pinnedAuthAttempted && shouldReleaseResponsesWebsocketPinnedAuth(forwardErrMsg) { - forgetPinnedAuth() + if forwardErrMsg != nil && cliproxyexecutor.IsUpstreamWebsocketTurnError(forwardErrMsg.Error) { + if lossWriteErr != nil { + wsTerminateErr = lossWriteErr + return + } + // Keep the credential: the next full create reconnects on it, never on another one. + if (attemptedUpstreamMode == responsesWebsocketUpstreamModeWS || upstreamMode == responsesWebsocketUpstreamModeWS) && lastAttemptedAuthID != "" { + upstreamMode = responsesWebsocketUpstreamModeWS + upstreamWebsocketAuthID = lastAttemptedAuthID + rememberPinnedAuth(lastAttemptedAuthID, modelName) + passthroughModelName = modelName + } + break } - if replayedPinnedAuthFailure { - replayErr := responsesWebsocketHTTPReplayRequiredError() - wsTerminateErr = replayErr - matched, errClose := writer.closeForUpstreamError(replayErr) - if !matched { - _ = conn.Close() - } else if errClose != nil && !errors.Is(errClose, websocket.ErrCloseSent) { - log.Debugf("responses websocket: credential replay close failed id=%s error=%v", passthroughSessionID, errClose) + if forwardErrMsg != nil { + if failoverInPlace { + retriedPinnedFailover = true + if lastAttemptedAuthID != "" { + excludedAuthIDs = append(excludedAuthIDs, lastAttemptedAuthID) + } + forgetPinnedAuth() + upstreamWebsocketAuthID = "" + upstreamMode = responsesWebsocketUpstreamModeUnknown + observedCompactionReplayAuthID = "" + executionAuthID = "" + nativeWebsocketPassthrough = false + failoverDoc := ojson.NewDoc(requestJSON) + _ = failoverDoc.Delete("type") + requestJSON = failoverDoc.Bytes() + requestJSON, toolCacheTurn = prepareResponsesWebsocketFallbackTurn(downstreamSessionKey, requestJSON) + nextLastRequest = requestJSON + continue } - return + if pinnedAuthAttempted && shouldReleaseResponsesWebsocketPinnedAuth(forwardErrMsg) { + forgetPinnedAuth() + } + if replayedPinnedAuthFailure { + replayErr := responsesWebsocketHTTPReplayRequiredError() + wsTerminateErr = replayErr + matched, errClose := writer.closeForUpstreamError(replayErr) + if !matched { + _ = conn.Close() + } else if errClose != nil && !errors.Is(errClose, websocket.ErrCloseSent) { + log.Debugf("responses websocket: credential replay close failed id=%s error=%v", passthroughSessionID, errClose) + } + return + } + break } + break + } + if forwardErrMsg != nil { continue } diff --git a/sdk/api/handlers/openai/openai_responses_websocket_forward.go b/sdk/api/handlers/openai/openai_responses_websocket_forward.go index c81aae3c4d1..49007b846f5 100644 --- a/sdk/api/handlers/openai/openai_responses_websocket_forward.go +++ b/sdk/api/handlers/openai/openai_responses_websocket_forward.go @@ -28,6 +28,7 @@ type responsesWebsocketForwardOptions struct { duplexStream func() bool toolCacheTurn *responsesWebsocketToolCacheTurn suppressError func(*interfaces.ErrorMessage) bool + onDownstreamCommit func() localInterrupt *responsesLocalInterrupt // websocketOwned reports whether this turn runs on an upstream // websocket. While true, no interrupt may claim the local HTTP path. @@ -55,6 +56,7 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( // completion bookkeeping. turnEnded := false responseStarted := false + var pendingDuplexError *interfaces.ErrorMessage completedOutput := []byte("[]") completedResponseID := "" outputItemsByIndex := make(map[int64][]byte) @@ -65,6 +67,11 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( downstreamSessionKey = websocketDownstreamSessionKey(c.Request) } + markDownstreamCommitted := func() { + if opts.onDownstreamCommit != nil { + opts.onDownstreamCommit() + } + } if opts.localInterrupt != nil { websocketOwned := opts.websocketOwned != nil && opts.websocketOwned() opts.localInterrupt.begin(websocketOwned) @@ -81,6 +88,7 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( // already gone need no answer. if c.Request.Context().Err() == nil { if claimed := opts.localInterrupt.takeClaimed(); claimed != nil { + markDownstreamCommitted() _, _ = writeResponsesWebsocketError(writer, wsTimelineLog, &interfaces.ErrorMessage{ StatusCode: http.StatusBadRequest, Error: errors.New("no active response to interrupt"), @@ -138,6 +146,7 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( } continue } + markDownstreamCommitted() return completeResponsesWebsocketLocalInterrupt(writer, wsTimelineLog, cancel, interruptPayload, outputItemsByIndex, outputItemsFallback, pendingToolCallIDs) case <-c.Request.Context().Done(): cause := context.Cause(c.Request.Context()) @@ -166,6 +175,9 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( return handleErr(pendingErrMsg) } } + if pendingDuplexError != nil { + return handleErr(pendingDuplexError) + } if duplex { // A duplex stream ends with its socket, not an individual response. // The data channel may close before select observes its final error. @@ -200,10 +212,12 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( if opts.localInterrupt != nil { if interruptPayload := opts.localInterrupt.takeClaimed(); interruptPayload != nil { if !turnEnded { + markDownstreamCommitted() return completeResponsesWebsocketLocalInterrupt(writer, wsTimelineLog, cancel, interruptPayload, outputItemsByIndex, outputItemsFallback, pendingToolCallIDs) } // The response is over: reject instead of fabricating an // incomplete event, then keep forwarding this chunk. + markDownstreamCommitted() if _, errWrite := writeResponsesWebsocketError(writer, wsTimelineLog, &interfaces.ErrorMessage{ StatusCode: http.StatusBadRequest, Error: errors.New("no active response to interrupt"), @@ -223,6 +237,7 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( } if eventType == "response.created" { responseStarted = true + markDownstreamCommitted() completed = false turnEnded = false if opts.localInterrupt != nil { @@ -246,20 +261,34 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( // In Codex duplex mode the executor owns connection termination: // payload errors after response.created are recoverable events; // StreamChunk.Err still arrives through errs and closes the socket. - preserveErrorEvent := responseStarted && opts.duplexStream != nil && opts.duplexStream() - if eventType == wsEventTypeError && preserveErrorEvent { - turnEnded = true - if errMsg := responsesWebsocketErrorMessageFromPayload(payloads[i]); errMsg != nil && h != nil { - h.LoggingAPIResponseError(context.WithValue(context.Background(), "gin", c), errMsg) - } - } else if eventType == wsEventTypeError && !preserveErrorEvent { - payloadErrMsg = responsesWebsocketErrorMessageFromPayload(payloads[i]) - if h != nil { - h.LoggingAPIResponseError(context.WithValue(context.Background(), "gin", c), payloadErrMsg) + // Known credential quota/cooldown errors are not recoverable on the + // same socket and must still consult the replay guard. + if eventType == wsEventTypeError { + errMsg := responsesWebsocketErrorMessageFromPayload(payloads[i]) + if h != nil && h.AuthManager != nil && opts.duplexStream != nil && opts.duplexStream() && errMsg != nil && + (isResponsesWebsocketQuotaOrCooldownError(errMsg.Error) || strings.Contains(errMsg.Error.Error(), "server_is_overloaded")) { + // The executor follows a terminal credential/capacity payload with + // StreamChunk.Err. Let the conductor record its hold before the + // replay guard checks for a cooling sibling. + pendingDuplexError = errMsg + continue } - if opts.suppressError != nil && opts.suppressError(payloadErrMsg) { - cancel(payloadErrMsg.Error) - return completedOutput, completedResponseID, sortedStringSet(pendingToolCallIDs), payloadErrMsg, nil + preserveErrorEvent := responseStarted && opts.duplexStream != nil && opts.duplexStream() && + (errMsg == nil || !isResponsesWebsocketQuotaOrCooldownError(errMsg.Error)) + if preserveErrorEvent { + turnEnded = true + if errMsg != nil && h != nil { + h.LoggingAPIResponseError(context.WithValue(context.Background(), "gin", c), errMsg) + } + } else { + payloadErrMsg = errMsg + if h != nil { + h.LoggingAPIResponseError(context.WithValue(context.Background(), "gin", c), payloadErrMsg) + } + if opts.suppressError != nil && opts.suppressError(payloadErrMsg) { + cancel(payloadErrMsg.Error) + return completedOutput, completedResponseID, sortedStringSet(pendingToolCallIDs), payloadErrMsg, nil + } } } else if isResponsesWebsocketCompletionEvent(eventType) || eventType == "response.incomplete" { // response.incomplete ends the turn too. Codex uses it, with @@ -312,6 +341,7 @@ func (h *OpenAIResponsesAPIHandler) forwardResponsesWebsocket( // reactivated it. opts.localInterrupt.deactivate() } + markDownstreamCommitted() if errWrite := writeResponsesWebsocketPayload(writer, wsTimelineLog, payloads[i], time.Now()); errWrite != nil { log.Warnf( "responses websocket: downstream_out write failed id=%s event=%s error=%v", diff --git a/sdk/api/handlers/openai/openai_responses_websocket_loss_test.go b/sdk/api/handlers/openai/openai_responses_websocket_loss_test.go index bcd1064c2ad..8438cf65005 100644 --- a/sdk/api/handlers/openai/openai_responses_websocket_loss_test.go +++ b/sdk/api/handlers/openai/openai_responses_websocket_loss_test.go @@ -1579,3 +1579,475 @@ func TestBuildResponsesWebsocketErrorPayloadSuppressesStaleRetryAfterWhenTrusted t.Fatalf("headers.X-Request-Id = %q, want req-1 (%s)", got, payload) } } + +func TestResponsesWebsocketSameSessionLaterCreate429FailoverToAlternativeCredential(t *testing.T) { + gin.SetMode(gin.TestMode) + + const quota429Frame = `{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"The usage limit has been reached","plan_type":"team","resets_in_seconds":3600}}` + + for _, buffering := range []bool{false, true} { + t.Run(fmt.Sprintf("buffering=%t", buffering), func(t *testing.T) { + var authACreates, authBCreates atomic.Int32 + var authA429Next atomic.Bool + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + authHeader := strings.TrimSpace(r.Header.Get("Authorization")) + if authHeader != "Bearer key-a" && authHeader != "Bearer key-b" { + t.Errorf("unexpected Authorization header: %q", authHeader) + http.Error(w, "unauthorized", http.StatusUnauthorized) + return + } + c, errUpgrade := (&websocket.Upgrader{}).Upgrade(w, r, nil) + if errUpgrade != nil { + t.Error(errUpgrade) + return + } + defer func() { _ = c.Close() }() + _ = c.SetReadDeadline(time.Now().Add(10 * time.Second)) + + for { + _, payload, errRead := c.ReadMessage() + if errRead != nil { + return + } + if ojson.GetBytesNoCopy(payload, "type").String() != "response.create" { + continue + } + if authHeader == "Bearer key-a" { + turn := authACreates.Add(1) + if authA429Next.Load() { + if buffering { + _ = c.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.created","response":{"id":"resp-a-buffered","status":"in_progress","output":[]}}`)) + _ = c.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.in_progress","response":{"id":"resp-a-buffered","status":"in_progress","output":[]}}`)) + } + _ = c.WriteMessage(websocket.TextMessage, []byte(quota429Frame)) + continue + } + respID := fmt.Sprintf("resp-a-%d", turn) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.created","response":{"id":%q,"status":"in_progress","output":[]}}`, respID))) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.completed","response":{"id":%q,"status":"completed","output":[{"id":"msg-a-%d","type":"message","role":"assistant","content":[{"type":"output_text","text":"from-a"}]}]}}`, respID, turn))) + continue + } + turn := authBCreates.Add(1) + respID := fmt.Sprintf("resp-b-%d", turn) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.created","response":{"id":%q,"status":"in_progress","output":[]}}`, respID))) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.completed","response":{"id":%q,"status":"completed","output":[{"id":"msg-b-%d","type":"message","role":"assistant","content":[{"type":"output_text","text":"from-b"}]}]}}`, respID, turn))) + } + })) + t.Cleanup(upstream.Close) + + cfg := &config.Config{ + DisableCooling: false, + RequestRetry: 3, + MaxRetryCredentials: 0, + Codex: config.CodexConfig{ + StreamBootstrapBuffering: buffering, + }, + } + manager := coreauth.NewManager(nil, &coreauth.FillFirstSelector{}, nil) + manager.SetConfig(cfg) + manager.SetRetryConfig(cfg.RequestRetry, 0, cfg.MaxRetryCredentials) + manager.RegisterExecutor(runtimeexecutor.NewCodexAutoExecutor(cfg)) + + modelName := "codex-same-session-create-429-" + strings.ReplaceAll(t.Name(), "/", "-") + authAID := "auth-a-" + modelName + authBID := "auth-b-" + modelName + for _, spec := range []struct { + id string + key string + priority string + }{ + {id: authAID, key: "key-a", priority: "4"}, + {id: authBID, key: "key-b", priority: "3"}, + } { + if _, errRegister := manager.Register(context.Background(), &coreauth.Auth{ + ID: spec.id, + Provider: "codex", + Status: coreauth.StatusActive, + Attributes: map[string]string{ + "api_key": spec.key, + "base_url": upstream.URL, + "websockets": "true", + "priority": spec.priority, + }, + }); errRegister != nil { + t.Fatal(errRegister) + } + registry.GetGlobalRegistry().RegisterClient(spec.id, "codex", []*registry.ModelInfo{{ID: modelName}}) + id := spec.id + t.Cleanup(func() { registry.GetGlobalRegistry().UnregisterClient(id) }) + } + + base := handlers.NewBaseAPIHandlers(&cfg.SDKConfig, manager) + h := NewOpenAIResponsesAPIHandler(base) + router := gin.New() + router.GET("/v1/responses", h.ResponsesWebsocket) + downstream := httptest.NewServer(router) + t.Cleanup(downstream.Close) + wsURL := "ws" + strings.TrimPrefix(downstream.URL, "http") + "/v1/responses" + + conn, resp, errDial := websocket.DefaultDialer.Dial(wsURL, nil) + if resp != nil && resp.Body != nil { + defer func() { _ = resp.Body.Close() }() + } + if errDial != nil { + t.Fatalf("dial downstream websocket: %v", errDial) + } + defer func() { _ = conn.Close() }() + + // Turn 1: full response.create succeeds on auth-a and pins auth-a for the connection. + req1 := fmt.Sprintf(`{"type":"response.create","model":%q,"input":[{"type":"message","id":"msg-1","role":"user","content":"turn-1"}]}`, modelName) + if errWrite := conn.WriteMessage(websocket.TextMessage, []byte(req1)); errWrite != nil { + t.Fatalf("write turn 1: %v", errWrite) + } + for { + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + _, frame, errRead := conn.ReadMessage() + if errRead != nil { + t.Fatalf("turn 1 read failed: %v", errRead) + } + if ojson.GetBytesNoCopy(frame, "type").String() == "response.completed" { + if got := ojson.GetBytesNoCopy(frame, "response.id").String(); got != "resp-a-1" { + t.Fatalf("turn 1 response.id = %q, want resp-a-1", got) + } + break + } + } + + // Turn 2: on the same downstream websocket, auth-a is now quota-exhausted and returns + // a pre-content 429 usage_limit_reached. Another full response.create must fail over + // to auth-b without closing the downstream websocket or leaking frames from auth-a. + authA429Next.Store(true) + req2 := fmt.Sprintf(`{"type":"response.create","model":%q,"input":[{"type":"message","id":"msg-2","role":"user","content":"turn-2"}]}`, modelName) + if errWrite := conn.WriteMessage(websocket.TextMessage, []byte(req2)); errWrite != nil { + t.Fatalf("write turn 2: %v", errWrite) + } + var turn2Completed []byte + for { + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + _, frame, errRead := conn.ReadMessage() + if errRead != nil { + t.Fatalf("turn 2 failed instead of failing over to auth-b: %v (authACreates=%d authBCreates=%d)", errRead, authACreates.Load(), authBCreates.Load()) + } + if gotID := ojson.GetBytesNoCopy(frame, "response.id").String(); gotID == "resp-a-buffered" { + t.Fatalf("buffered frame from failed credential A leaked to downstream: %s", frame) + } + if ojson.GetBytesNoCopy(frame, "type").String() == "error" { + t.Fatalf("received error frame instead of failing over to auth-b: %s", frame) + } + if ojson.GetBytesNoCopy(frame, "type").String() == "response.completed" { + turn2Completed = frame + break + } + } + if got := ojson.GetBytesNoCopy(turn2Completed, "response.id").String(); got != "resp-b-1" { + t.Fatalf("turn 2 response.id = %q, want resp-b-1", got) + } + if authACreates.Load() != 2 || authBCreates.Load() != 1 { + t.Fatalf("after turn 2: authACreates=%d authBCreates=%d, want 2 and 1", authACreates.Load(), authBCreates.Load()) + } + + authA, okA := manager.GetByID(authAID) + if !okA || authA == nil || !authA.Quota.Exceeded { + t.Fatalf("expected auth-a quota exceeded cooldown to be recorded, got %+v (ok=%t)", authA, okA) + } + + // Turn 3: continuation on the same downstream websocket must stay pinned to auth-b. + req3 := fmt.Sprintf(`{"type":"response.create","model":%q,"previous_response_id":"resp-b-1","input":[{"type":"message","id":"msg-3","role":"user","content":"turn-3"}]}`, modelName) + if errWrite := conn.WriteMessage(websocket.TextMessage, []byte(req3)); errWrite != nil { + t.Fatalf("write turn 3: %v", errWrite) + } + for { + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + _, frame, errRead := conn.ReadMessage() + if errRead != nil { + t.Fatalf("turn 3 read failed: %v", errRead) + } + if ojson.GetBytesNoCopy(frame, "type").String() == "response.completed" { + if got := ojson.GetBytesNoCopy(frame, "response.id").String(); got != "resp-b-2" { + t.Fatalf("turn 3 response.id = %q, want resp-b-2", got) + } + break + } + } + if authACreates.Load() != 2 || authBCreates.Load() != 2 { + t.Fatalf("after turn 3: authACreates=%d authBCreates=%d, want 2 and 2", authACreates.Load(), authBCreates.Load()) + } + }) + } +} + +func TestResponsesWebsocketSameSessionLaterCreate429DoesNotReplayAfterDownstreamCommit(t *testing.T) { + gin.SetMode(gin.TestMode) + + const quota429Frame = `{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"The usage limit has been reached","plan_type":"team","resets_in_seconds":3600}}` + + var authACreates, authBCreates atomic.Int32 + var authA429Next atomic.Bool + downstreamGotCreated := make(chan struct{}) + + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + authHeader := strings.TrimSpace(r.Header.Get("Authorization")) + c, errUpgrade := (&websocket.Upgrader{}).Upgrade(w, r, nil) + if errUpgrade != nil { + t.Error(errUpgrade) + return + } + defer func() { _ = c.Close() }() + _ = c.SetReadDeadline(time.Now().Add(10 * time.Second)) + + for { + _, payload, errRead := c.ReadMessage() + if errRead != nil { + return + } + if ojson.GetBytesNoCopy(payload, "type").String() != "response.create" { + continue + } + if authHeader == "Bearer key-b" { + authBCreates.Add(1) + _ = c.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.completed","response":{"id":"resp-b-1","status":"completed","output":[]}}`)) + continue + } + turn := authACreates.Add(1) + if authA429Next.Load() { + // Emit response.created before the 429 error while buffering is disabled so + // downstream observes a committed response frame before the error arrives. + _ = c.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.created","response":{"id":"resp-a-committed","status":"in_progress","output":[]}}`)) + select { + case <-downstreamGotCreated: + case <-time.After(5 * time.Second): + t.Error("timed out waiting for downstream to receive response.created") + return + } + _ = c.WriteMessage(websocket.TextMessage, []byte(quota429Frame)) + continue + } + respID := fmt.Sprintf("resp-a-%d", turn) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.created","response":{"id":%q,"status":"in_progress","output":[]}}`, respID))) + _ = c.WriteMessage(websocket.TextMessage, []byte(fmt.Sprintf(`{"type":"response.completed","response":{"id":%q,"status":"completed","output":[]}}`, respID))) + } + })) + t.Cleanup(upstream.Close) + + cfg := &config.Config{ + DisableCooling: false, + RequestRetry: 3, + MaxRetryCredentials: 0, + Codex: config.CodexConfig{ + StreamBootstrapBuffering: false, + }, + } + manager := coreauth.NewManager(nil, &coreauth.FillFirstSelector{}, nil) + manager.SetConfig(cfg) + manager.SetRetryConfig(cfg.RequestRetry, 0, cfg.MaxRetryCredentials) + manager.RegisterExecutor(runtimeexecutor.NewCodexAutoExecutor(cfg)) + + modelName := "codex-committed-no-replay-429" + authAID := "auth-a-" + modelName + authBID := "auth-b-" + modelName + for _, spec := range []struct { + id string + key string + priority string + }{ + {id: authAID, key: "key-a", priority: "4"}, + {id: authBID, key: "key-b", priority: "3"}, + } { + if _, errRegister := manager.Register(context.Background(), &coreauth.Auth{ + ID: spec.id, + Provider: "codex", + Status: coreauth.StatusActive, + Attributes: map[string]string{ + "api_key": spec.key, + "base_url": upstream.URL, + "websockets": "true", + "priority": spec.priority, + }, + }); errRegister != nil { + t.Fatal(errRegister) + } + registry.GetGlobalRegistry().RegisterClient(spec.id, "codex", []*registry.ModelInfo{{ID: modelName}}) + id := spec.id + t.Cleanup(func() { registry.GetGlobalRegistry().UnregisterClient(id) }) + } + + base := handlers.NewBaseAPIHandlers(&cfg.SDKConfig, manager) + h := NewOpenAIResponsesAPIHandler(base) + router := gin.New() + router.GET("/v1/responses", h.ResponsesWebsocket) + downstream := httptest.NewServer(router) + t.Cleanup(downstream.Close) + wsURL := "ws" + strings.TrimPrefix(downstream.URL, "http") + "/v1/responses" + + conn, resp, errDial := websocket.DefaultDialer.Dial(wsURL, nil) + if resp != nil && resp.Body != nil { + defer func() { _ = resp.Body.Close() }() + } + if errDial != nil { + t.Fatalf("dial downstream websocket: %v", errDial) + } + defer func() { _ = conn.Close() }() + + req1 := fmt.Sprintf(`{"type":"response.create","model":%q,"input":[{"type":"message","id":"msg-1","role":"user","content":"turn-1"}]}`, modelName) + if errWrite := conn.WriteMessage(websocket.TextMessage, []byte(req1)); errWrite != nil { + t.Fatalf("write turn 1: %v", errWrite) + } + for { + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + _, frame, errRead := conn.ReadMessage() + if errRead != nil { + t.Fatalf("turn 1 failed: %v", errRead) + } + if ojson.GetBytesNoCopy(frame, "type").String() == "response.completed" { + break + } + } + + authA429Next.Store(true) + req2 := fmt.Sprintf(`{"type":"response.create","model":%q,"input":[{"type":"message","id":"msg-2","role":"user","content":"turn-2"}]}`, modelName) + if errWrite := conn.WriteMessage(websocket.TextMessage, []byte(req2)); errWrite != nil { + t.Fatalf("write turn 2: %v", errWrite) + } + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + _, frame1, errRead1 := conn.ReadMessage() + if errRead1 != nil || ojson.GetBytesNoCopy(frame1, "type").String() != "response.created" { + t.Fatalf("turn 2 first frame = %s, err=%v; want committed response.created", frame1, errRead1) + } + close(downstreamGotCreated) + _, frame2, errRead2 := conn.ReadMessage() + if errRead2 != nil || ojson.GetBytesNoCopy(frame2, "type").String() != "error" || int(ojson.GetBytesNoCopy(frame2, "status").Int()) != http.StatusTooManyRequests { + t.Fatalf("turn 2 second frame = %s, err=%v; want terminal 429 error frame without replay", frame2, errRead2) + } + if authBCreates.Load() != 0 { + t.Fatalf("authBCreates = %d, want 0 (must not silently replay after downstream commit)", authBCreates.Load()) + } +} + +func TestResponsesWebsocketFailoverWaitsForCoolingAccountInsteadOfSpentOne(t *testing.T) { + gin.SetMode(gin.TestMode) + for _, failOnFirst := range []bool{true, false} { + for _, refusal := range []string{ + `{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"quota temporarily exhausted","resets_in_seconds":60}}`, + `{"type":"error","status":503,"error":{"code":"server_is_overloaded","message":"server_is_overloaded"}}`, + } { + for _, steering := range []bool{false, true} { + for _, continuation := range []bool{false, true} { + if failOnFirst && continuation { + continue + } + t.Run(fmt.Sprintf("initial=%t/status=%d/steering=%t/continuation=%t", failOnFirst, ojson.Get(refusal, "status").Int(), steering, continuation), func(t *testing.T) { + var createsA, createsB atomic.Int32 + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, errUpgrade := (&websocket.Upgrader{}).Upgrade(w, r, nil) + if errUpgrade != nil { + t.Error(errUpgrade) + return + } + defer func() { _ = conn.Close() }() + for { + _, payload, errRead := conn.ReadMessage() + if errRead != nil { + return + } + if ojson.GetBytesNoCopy(payload, "type").String() != "response.create" { + continue + } + if r.Header.Get("Authorization") == "Bearer key-b" { + createsB.Add(1) + _ = conn.WriteMessage(websocket.TextMessage, []byte(`{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"spent account refusal","resets_in_seconds":172800}}`)) + continue + } + if turn := createsA.Add(1); failOnFirst || turn > 1 { + _ = conn.WriteMessage(websocket.TextMessage, []byte(refusal)) + continue + } + _ = conn.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.created","response":{"id":"resp-allowance","output":[]}}`)) + _ = conn.WriteMessage(websocket.TextMessage, []byte(`{"type":"response.completed","response":{"id":"resp-allowance","status":"completed","output":[]}}`)) + } + })) + t.Cleanup(upstream.Close) + cfg := &config.Config{Codex: config.CodexConfig{ResponseSteering: steering}} + cfg.CodexResponseSteering = steering + manager := coreauth.NewManager(nil, &coreauth.SoonestResetFirstSelector{}, nil) + manager.SetConfig(cfg) + manager.RegisterExecutor(runtimeexecutor.NewCodexAutoExecutor(cfg)) + model := "cooling-replay-" + strings.ReplaceAll(t.Name(), "/", "-") + for _, key := range []string{"key-a", "key-b"} { + id := model + key + _, errRegister := manager.Register(context.Background(), &coreauth.Auth{ + ID: id, Provider: "codex", Status: coreauth.StatusActive, + Attributes: map[string]string{"api_key": key, "base_url": upstream.URL, "websockets": "true"}, + }) + if errRegister != nil { + t.Fatal(errRegister) + } + registry.GetGlobalRegistry().RegisterClient(id, "codex", []*registry.ModelInfo{{ID: model}}) + t.Cleanup(func() { + registry.GetGlobalRegistry().UnregisterClient(id) + coreauth.ForgetQuotaWindows(id) + }) + used := 22.0 + if key == "key-b" { + used = 100 + } + now := time.Now() + coreauth.ObserveQuotaWindows(id, now, []coreauth.QuotaWindow{{Key: "seven_day", UsedPercent: used, ResetAt: now.Add(48 * time.Hour), Period: 7 * 24 * time.Hour}}) + } + base := handlers.NewBaseAPIHandlers(&cfg.SDKConfig, manager) + h := NewOpenAIResponsesAPIHandler(base) + router := gin.New() + router.GET("/v1/responses", h.ResponsesWebsocket) + downstream := httptest.NewServer(router) + t.Cleanup(downstream.Close) + client, resp, errDial := websocket.DefaultDialer.Dial("ws"+strings.TrimPrefix(downstream.URL, "http")+"/v1/responses", nil) + if resp != nil { + defer func() { _ = resp.Body.Close() }() + } + if errDial != nil { + t.Fatal(errDial) + } + defer func() { _ = client.Close() }() + lastTurn := 2 + if failOnFirst { + lastTurn = 1 + } + for turn := 1; turn <= lastTurn; turn++ { + previous := "" + if turn == 2 && continuation { + previous = `,"previous_response_id":"resp-allowance"` + } + payload := fmt.Sprintf(`{"type":"response.create","model":%q,"input":[]%s}`, model, previous) + if errWrite := client.WriteMessage(websocket.TextMessage, []byte(payload)); errWrite != nil { + t.Fatal(errWrite) + } + _ = client.SetReadDeadline(time.Now().Add(5 * time.Second)) + for { + _, frame, errRead := client.ReadMessage() + if errRead != nil { + t.Fatalf("turn %d closed without its result: %v", turn, errRead) + } + if turn < lastTurn && ojson.GetBytesNoCopy(frame, "type").String() == "response.completed" { + break + } + if turn == lastTurn { + if ojson.GetBytesNoCopy(frame, "error.code").String() != "model_cooldown" || ojson.GetBytesNoCopy(frame, "status").Int() != 429 { + t.Fatalf("want the hub cooldown, got %s", frame) + } + wait := ojson.GetBytesNoCopy(frame, "error.reset_seconds").Int() + if wait <= 0 || wait > 300 { + t.Fatalf("cooldown = %ds, want the short account hold: %s", wait, frame) + } + break + } + } + } + if createsA.Load() != int32(lastTurn) || createsB.Load() != 0 { + t.Fatalf("creates A=%d B=%d, want A=%d B=0", createsA.Load(), createsB.Load(), lastTurn) + } + }) + } + } + } + } +} diff --git a/sdk/api/handlers/openai/openai_responses_websocket_session.go b/sdk/api/handlers/openai/openai_responses_websocket_session.go index eb38ea210b0..2001fb2be82 100644 --- a/sdk/api/handlers/openai/openai_responses_websocket_session.go +++ b/sdk/api/handlers/openai/openai_responses_websocket_session.go @@ -7,6 +7,7 @@ import ( "strings" "time" + "github.com/rune-oai/t3code-proxy/internal/interfaces" "github.com/rune-oai/t3code-proxy/internal/registry" "github.com/rune-oai/t3code-proxy/internal/thinking" "github.com/rune-oai/t3code-proxy/internal/util" @@ -88,11 +89,11 @@ func (h *OpenAIResponsesAPIHandler) responsesWebsocketAvailableAuthsForModel(mod return available, modelKey } -func (h *OpenAIResponsesAPIHandler) responsesWebsocketHasAlternativeReplayTarget(ctx context.Context, modelName string, rawJSON []byte, failedAuthID string) bool { +func (h *OpenAIResponsesAPIHandler) responsesWebsocketAlternativeReplayTarget(ctx context.Context, modelName string, rawJSON []byte, failedAuthID string) (bool, *interfaces.ErrorMessage) { if h == nil || h.BaseAPIHandler == nil || h.AuthManager == nil { - return false + return false, nil } - return h.HasAlternativeExecutionTarget(ctx, h.HandlerType(), modelName, rawJSON, failedAuthID) + return h.CheckAlternativeExecutionTarget(ctx, h.HandlerType(), modelName, rawJSON, failedAuthID) } func (h *OpenAIResponsesAPIHandler) responsesWebsocketUsesCodexWebsocketPassthrough(modelName string) bool { diff --git a/sdk/api/handlers/openai/openai_responses_websocket_terminal_order_test.go b/sdk/api/handlers/openai/openai_responses_websocket_terminal_order_test.go new file mode 100644 index 00000000000..f8380e52c41 --- /dev/null +++ b/sdk/api/handlers/openai/openai_responses_websocket_terminal_order_test.go @@ -0,0 +1,173 @@ +package openai + +import ( + "bufio" + "context" + "errors" + "net" + "net/http" + "runtime" + "strings" + "sync/atomic" + "testing" + "testing/synctest" + "time" + + "github.com/MoeForge/moego/ojson" + "github.com/gorilla/websocket" + "github.com/rune-oai/t3code-proxy/internal/interfaces" +) + +// Pause a write before sending it or after delivery while writeMu is still held. +type pausedWebsocketWriteConn struct { + net.Conn + before bool + pause atomic.Bool + entered chan struct{} + release chan struct{} +} + +func (c *pausedWebsocketWriteConn) Write(payload []byte) (int, error) { + pause := c.pause.Swap(false) + if pause && c.before { + close(c.entered) + <-c.release + } + n, err := c.Conn.Write(payload) + if pause && !c.before { + close(c.entered) + <-c.release + } + return n, err +} + +func TestResponsesWebsocketTerminalErrorWaitsForWriter(t *testing.T) { + for _, phase := range []string{"response.created", "ping", "terminal"} { + t.Run(phase, func(t *testing.T) { + clientRaw, serverRaw := net.Pipe() + defer func() { _ = clientRaw.Close() }() + defer func() { _ = serverRaw.Close() }() + paused := &pausedWebsocketWriteConn{Conn: serverRaw, before: phase == "terminal", entered: make(chan struct{}), release: make(chan struct{})} + serverConn := make(chan *websocket.Conn, 1) + go func() { + req, err := http.ReadRequest(bufio.NewReader(serverRaw)) + if err != nil { + t.Error(err) + return + } + conn, err := responsesWebsocketUpgrader.Upgrade(newHijackableWebsocketResponseWriter(paused), req, nil) + if err != nil { + t.Error(err) + return + } + serverConn <- conn + }() + dialer := websocket.Dialer{NetDialContext: func(context.Context, string, string) (net.Conn, error) { return clientRaw, nil }} + client, resp, err := dialer.Dial("ws://example.com/responses", nil) + if err != nil { + t.Fatal(err) + } + defer func() { _ = resp.Body.Close() }() + frames := make(chan []byte, 2) + readDone := make(chan error, 1) + client.SetPingHandler(func(string) error { frames <- nil; return nil }) + go func() { + defer close(frames) + for { + _, payload, errRead := client.ReadMessage() + if errRead != nil { + readDone <- errRead + return + } + frames <- payload + } + }() + writer := newResponsesWebsocketWriter(<-serverConn) + paused.pause.Store(true) + errMsg := &interfaces.ErrorMessage{StatusCode: http.StatusTooManyRequests, Error: websocketPinnedFailoverStatusError{ + status: http.StatusTooManyRequests, msg: `{"error":{"type":"usage_limit_reached","message":"quota exhausted"}}`, + }} + writeDone := make(chan error, 1) + go func() { + switch phase { + case "response.created": + writeDone <- writeResponsesWebsocketPayload(writer, nil, []byte(`{"type":"response.created"}`), time.Now()) + case "ping": + writeDone <- writer.writePing() + case "terminal": + _, _, err := writeResponsesWebsocketTerminalError(writer, nil, errMsg, nil) + writeDone <- err + } + }() + if phase != "terminal" { + <-frames + } + <-paused.entered + contenderDone := make(chan struct{}) + go func() { + _, _, _ = writeResponsesWebsocketTerminalError(writer, nil, errMsg, nil) + // Model the handler's deferred socket teardown after forwarding returns. + _, _ = writer.closeWithoutError() + _ = writer.conn.Close() + close(contenderDone) + }() + waitForTerminalWriterOrClose(t, contenderDone) + select { + case <-contenderDone: + t.Error("terminal contender closed the socket before the active writer returned") + default: + } + close(paused.release) + frame, ok := <-frames + if !ok { + t.Fatalf("terminal frame was lost: %v", <-readDone) + } + if got := ojson.GetBytesNoCopy(frame, "status").Int(); got != http.StatusTooManyRequests { + t.Fatalf("terminal status = %d, frame = %s", got, frame) + } + if err := <-writeDone; err != nil && !errors.Is(err, websocket.ErrCloseSent) { + t.Fatal(err) + } + <-contenderDone + }) + } +} + +// A mutex wait is not durably blocked under testing/synctest. Observe the +// contender at that lock (or its premature return) before releasing the I/O +// barrier, so the regression does not depend on scheduler speed or sleeps. +func waitForTerminalWriterOrClose(t *testing.T, done <-chan struct{}) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + stack := make([]byte, 1<<20) + for time.Now().Before(deadline) { + select { + case <-done: + return + default: + } + n := runtime.Stack(stack, true) + for _, goroutine := range strings.Split(string(stack[:n]), "\n\n") { + if strings.Contains(goroutine, "[sync.Mutex.Lock]") && strings.Contains(goroutine, "(*responsesWebsocketWriter).closeWithPayload") { + return + } + } + runtime.Gosched() + } + t.Fatal("terminal writer did not reach the write lock") +} + +func TestResponsesWebsocketKeepAliveDoesNotCancelTerminalWrite(t *testing.T) { + synctest.Test(t, func(t *testing.T) { + ctx, cancel := context.WithCancelCause(context.Background()) + defer cancel(nil) + writer := newResponsesWebsocketWriter(&websocket.Conn{}) + writer.closing.Store(true) + intervals := make(chan time.Duration, 1) + intervals <- time.Nanosecond + writer.keepAlive(ctx, cancel, intervals) + if ctx.Err() != nil { + t.Fatalf("keepalive canceled the socket during terminal delivery: %v", context.Cause(ctx)) + } + }) +} diff --git a/sdk/api/handlers/openai/openai_responses_websocket_test.go b/sdk/api/handlers/openai/openai_responses_websocket_test.go index 9a2b6b51fe9..c1b6a966c0e 100644 --- a/sdk/api/handlers/openai/openai_responses_websocket_test.go +++ b/sdk/api/handlers/openai/openai_responses_websocket_test.go @@ -120,6 +120,9 @@ func TestResponsesWebsocketHomeSelectedAuthCallbackPinsAndReusesFirstSelection(t manager.PublishHomeDispatch(dispatcher, executionregistry.New(), 1) manager.RegisterExecutor(executor) registry.GetGlobalRegistry().RegisterClient("home-responses-websocket-auth", "codex", []*registry.ModelInfo{{ID: "gpt-5.4"}}) + t.Cleanup(func() { + registry.GetGlobalRegistry().UnregisterClient("home-responses-websocket-auth") + }) base := handlers.NewBaseAPIHandlers(&sdkconfig.SDKConfig{}, manager) h := NewOpenAIResponsesAPIHandler(base) @@ -7759,13 +7762,6 @@ func TestResponsesWebsocketWriterCloseInterruptsBlockedPing(t *testing.T) { return errClose }, }, - { - name: "closeWithPayload", - close: func(w *responsesWebsocketWriter) error { - _, errClose := w.closeWithPayload([]byte(`{"type":"error","status":400}`)) - return errClose - }, - }, } for _, tc := range cases { @@ -8229,3 +8225,318 @@ func TestForwardResponsesWebsocketPrefersBufferedErrorOverClosedData(t *testing. } } } + +func TestForwardResponsesWebsocketDuplexErrorClassificationAfterResponseCreated(t *testing.T) { + gin.SetMode(gin.TestMode) + + const createdFrame = `{"type":"response.created","response":{"id":"r-quota"}}` + + t.Run("quota_and_cooldown_consult_replay_guard_when_suppressed", func(t *testing.T) { + for _, tc := range []struct { + name string + errorFrame string + }{ + { + name: "usage_limit_reached", + errorFrame: `{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"exhausted"}}`, + }, + { + name: "model_cooldown", + errorFrame: `{"type":"error","status":429,"error":{"code":"model_cooldown","message":"cooling down"}}`, + }, + } { + t.Run(tc.name, func(t *testing.T) { + var suppressCalls atomic.Int32 + var cancelCalls atomic.Int32 + serverErrCh := make(chan error, 1) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := responsesWebsocketUpgrader.Upgrade(w, r, nil) + if err != nil { + serverErrCh <- err + return + } + defer func() { _ = conn.Close() }() + + ctx, _ := gin.CreateTestContext(httptest.NewRecorder()) + ctx.Request = r + + data := make(chan []byte, 2) + errCh := make(chan *interfaces.ErrorMessage) + data <- []byte(createdFrame) + data <- []byte(tc.errorFrame) + close(data) + close(errCh) + + _, _, _, errMsg, errForward := (*OpenAIResponsesAPIHandler)(nil).forwardResponsesWebsocket( + ctx, + newResponsesWebsocketWriter(conn), + func(...interface{}) { cancelCalls.Add(1) }, + data, + errCh, + newInMemoryWebsocketTimelineLog(), + "session-duplex-quota", + responsesWebsocketForwardOptions{ + duplexStream: func() bool { return true }, + suppressError: func(msg *interfaces.ErrorMessage) bool { + suppressCalls.Add(1) + return msg != nil && isResponsesWebsocketQuotaOrCooldownError(msg.Error) + }, + }, + ) + if errForward != nil { + serverErrCh <- fmt.Errorf("expected nil errForward when suppressed, got %v", errForward) + return + } + if errMsg == nil { + serverErrCh <- errors.New("expected non-nil errMsg when suppressed") + return + } + serverErrCh <- nil + })) + defer server.Close() + + wsURL := "ws" + strings.TrimPrefix(server.URL, "http") + conn, _, err := websocket.DefaultDialer.Dial(wsURL, nil) + if err != nil { + t.Fatalf("dial websocket: %v", err) + } + defer func() { _ = conn.Close() }() + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + + _, frame1, errRead1 := conn.ReadMessage() + if errRead1 != nil { + t.Fatalf("read response.created: %v", errRead1) + } + if got := ojson.GetBytesNoCopy(frame1, "type").String(); got != "response.created" { + t.Fatalf("frame1 type = %q, want response.created: %s", got, frame1) + } + + if _, frame2, errRead2 := conn.ReadMessage(); errRead2 == nil { + t.Fatalf("expected suppressed quota frame not to be forwarded, got: %s", frame2) + } + + if errServer := <-serverErrCh; errServer != nil { + t.Fatalf("server error: %v", errServer) + } + if got := suppressCalls.Load(); got != 1 { + t.Fatalf("suppressError calls = %d, want 1", got) + } + if got := cancelCalls.Load(); got != 1 { + t.Fatalf("cancel calls = %d, want 1", got) + } + }) + } + }) + + t.Run("quota_and_cooldown_emit_terminal_error_when_not_suppressed", func(t *testing.T) { + for _, tc := range []struct { + name string + errorFrame string + wantField string + wantValue string + }{ + { + name: "usage_limit_reached", + errorFrame: `{"type":"error","status":429,"error":{"type":"usage_limit_reached","message":"exhausted"}}`, + wantField: "error.type", + wantValue: "usage_limit_reached", + }, + { + name: "model_cooldown", + errorFrame: `{"type":"error","status":429,"error":{"code":"model_cooldown","message":"cooling down"}}`, + wantField: "error.code", + wantValue: "model_cooldown", + }, + } { + t.Run(tc.name, func(t *testing.T) { + var suppressCalls atomic.Int32 + var cancelCalls atomic.Int32 + serverErrCh := make(chan error, 1) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := responsesWebsocketUpgrader.Upgrade(w, r, nil) + if err != nil { + serverErrCh <- err + return + } + defer func() { _ = conn.Close() }() + + ctx, _ := gin.CreateTestContext(httptest.NewRecorder()) + ctx.Request = r + + data := make(chan []byte, 2) + errCh := make(chan *interfaces.ErrorMessage) + data <- []byte(createdFrame) + data <- []byte(tc.errorFrame) + close(data) + close(errCh) + + _, _, _, errMsg, errForward := (*OpenAIResponsesAPIHandler)(nil).forwardResponsesWebsocket( + ctx, + newResponsesWebsocketWriter(conn), + func(...interface{}) { cancelCalls.Add(1) }, + data, + errCh, + newInMemoryWebsocketTimelineLog(), + "session-duplex-terminal", + responsesWebsocketForwardOptions{ + duplexStream: func() bool { return true }, + suppressError: func(*interfaces.ErrorMessage) bool { + suppressCalls.Add(1) + return false + }, + }, + ) + if !errors.Is(errForward, websocket.ErrCloseSent) { + serverErrCh <- fmt.Errorf("errForward = %v, want websocket.ErrCloseSent", errForward) + return + } + if errMsg == nil || errMsg.StatusCode != http.StatusTooManyRequests { + serverErrCh <- fmt.Errorf("errMsg = %+v, want status 429", errMsg) + return + } + serverErrCh <- nil + })) + defer server.Close() + + wsURL := "ws" + strings.TrimPrefix(server.URL, "http") + conn, _, err := websocket.DefaultDialer.Dial(wsURL, nil) + if err != nil { + t.Fatalf("dial websocket: %v", err) + } + defer func() { _ = conn.Close() }() + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + + _, frame1, errRead1 := conn.ReadMessage() + if errRead1 != nil { + t.Fatalf("read response.created: %v", errRead1) + } + if got := ojson.GetBytesNoCopy(frame1, "type").String(); got != "response.created" { + t.Fatalf("frame1 type = %q, want response.created: %s", got, frame1) + } + + _, frame2, errRead2 := conn.ReadMessage() + if errRead2 != nil { + t.Fatalf("read terminal error frame: %v", errRead2) + } + if got := ojson.GetBytesNoCopy(frame2, tc.wantField).String(); got != tc.wantValue { + t.Fatalf("frame2 %s = %q, want %q: %s", tc.wantField, got, tc.wantValue, frame2) + } + + if _, extraFrame, errRead3 := conn.ReadMessage(); errRead3 == nil { + t.Fatalf("expected socket close after terminal quota frame, got: %s", extraFrame) + } + + if errServer := <-serverErrCh; errServer != nil { + t.Fatalf("server error: %v", errServer) + } + if got := suppressCalls.Load(); got != 1 { + t.Fatalf("suppressError calls = %d, want 1", got) + } + if got := cancelCalls.Load(); got != 1 { + t.Fatalf("cancel calls = %d, want 1", got) + } + }) + } + }) + + t.Run("request_and_client_rate_limit_errors_stay_on_duplex_path", func(t *testing.T) { + for _, tc := range []struct { + name string + errorFrame string + }{ + { + name: "rate_limit_exceeded", + errorFrame: `{"type":"error","status":429,"error":{"type":"rate_limit_error","code":"rate_limit_exceeded","message":"rate limit exceeded"}}`, + }, + { + name: "invalid_request_error", + errorFrame: `{"type":"error","status":400,"error":{"type":"invalid_request_error","message":"bad request"}}`, + }, + } { + t.Run(tc.name, func(t *testing.T) { + var suppressCalls atomic.Int32 + serverErrCh := make(chan error, 1) + + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + conn, err := responsesWebsocketUpgrader.Upgrade(w, r, nil) + if err != nil { + serverErrCh <- err + return + } + defer func() { _ = conn.Close() }() + + ctx, _ := gin.CreateTestContext(httptest.NewRecorder()) + ctx.Request = r + + data := make(chan []byte, 4) + errCh := make(chan *interfaces.ErrorMessage) + data <- []byte(createdFrame) + data <- []byte(tc.errorFrame) + data <- []byte(`{"type":"response.created","response":{"id":"r-recovered"}}`) + data <- []byte(`{"type":"response.completed","response":{"id":"r-recovered","output":[]}}`) + close(data) + close(errCh) + + _, completedID, _, errMsg, errForward := (*OpenAIResponsesAPIHandler)(nil).forwardResponsesWebsocket( + ctx, + newResponsesWebsocketWriter(conn), + func(...interface{}) {}, + data, + errCh, + newInMemoryWebsocketTimelineLog(), + "session-duplex-preserved", + responsesWebsocketForwardOptions{ + duplexStream: func() bool { return true }, + suppressError: func(*interfaces.ErrorMessage) bool { + suppressCalls.Add(1) + return true + }, + }, + ) + if !errors.Is(errForward, websocket.ErrCloseSent) { + serverErrCh <- fmt.Errorf("errForward = %v, want websocket.ErrCloseSent", errForward) + return + } + if errMsg != nil { + serverErrCh <- fmt.Errorf("errMsg = %v, want nil", errMsg) + return + } + if completedID != "r-recovered" { + serverErrCh <- fmt.Errorf("completedID = %q, want r-recovered", completedID) + return + } + serverErrCh <- nil + })) + defer server.Close() + + wsURL := "ws" + strings.TrimPrefix(server.URL, "http") + conn, _, err := websocket.DefaultDialer.Dial(wsURL, nil) + if err != nil { + t.Fatalf("dial websocket: %v", err) + } + defer func() { _ = conn.Close() }() + _ = conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + + wantTypes := []string{"response.created", "error", "response.created", "response.completed"} + for idx, wantType := range wantTypes { + _, frame, errRead := conn.ReadMessage() + if errRead != nil { + t.Fatalf("read frame %d (%s): %v", idx, wantType, errRead) + } + if got := ojson.GetBytesNoCopy(frame, "type").String(); got != wantType { + t.Fatalf("frame %d type = %q, want %q: %s", idx, got, wantType, frame) + } + } + + if errServer := <-serverErrCh; errServer != nil { + t.Fatalf("server error: %v", errServer) + } + if got := suppressCalls.Load(); got != 0 { + t.Fatalf("suppressError calls = %d, want 0", got) + } + }) + } + }) +} diff --git a/sdk/api/handlers/provider_scope.go b/sdk/api/handlers/provider_scope.go new file mode 100644 index 00000000000..f040c9b92de --- /dev/null +++ b/sdk/api/handlers/provider_scope.go @@ -0,0 +1,18 @@ +package handlers + +import ( + "github.com/gin-gonic/gin" + "github.com/rune-oai/t3code-proxy/internal/registry" + coreauth "github.com/rune-oai/t3code-proxy/sdk/cliproxy/auth" +) + +// ModelsForRequest returns the catalog in the requested format, honoring a provider scope. +func (h *BaseAPIHandler) ModelsForRequest(c *gin.Context, handlerType string) []map[string]any { + r := registry.GetGlobalRegistry() + if c != nil && c.Request != nil { + if scope := coreauth.ProviderScopeFromContext(c.Request.Context()); scope != "" { + return r.GetAvailableModelsForProvider(scope, handlerType) + } + } + return r.GetAvailableModels(handlerType) +} diff --git a/sdk/cliproxy/auth/conductor_execution.go b/sdk/cliproxy/auth/conductor_execution.go index 379b9e2eb96..b514f5f37c5 100644 --- a/sdk/cliproxy/auth/conductor_execution.go +++ b/sdk/cliproxy/auth/conductor_execution.go @@ -207,7 +207,11 @@ func (m *Manager) Execute(ctx context.Context, providers []string, req cliproxye ctx = WithCodexClientRequest(ctx, isOpenAIImageExecutionOptions(opts) || IsCodexClientRequest(ctx, opts.Headers)) ctx = coreusage.CaptureIdentity(ctx) ctx = contextWithInFlightScope(ctx, scope) - normalized := m.normalizeProviders(providers) + if provider := ProviderScopeFromContext(ctx); provider != "" { + opts.EnsureMetadata() + opts.Metadata[cliproxyexecutor.ProviderScopeMetadataKey] = provider + } + normalized := m.normalizeProviders(scopedProviders(ctx, providers)) if len(normalized) == 0 { return cliproxyexecutor.Response{}, &Error{Code: "provider_not_found", Message: "no provider supplied"} } @@ -289,7 +293,11 @@ func (m *Manager) ExecuteCount(ctx context.Context, providers []string, req clip ctx = WithCodexClientRequest(ctx, IsCodexClientRequest(ctx, opts.Headers)) ctx = coreusage.CaptureIdentity(ctx) ctx = contextWithInFlightScope(ctx, scope) - normalized := m.normalizeProviders(providers) + if provider := ProviderScopeFromContext(ctx); provider != "" { + opts.EnsureMetadata() + opts.Metadata[cliproxyexecutor.ProviderScopeMetadataKey] = provider + } + normalized := m.normalizeProviders(scopedProviders(ctx, providers)) if len(normalized) == 0 { return cliproxyexecutor.Response{}, &Error{Code: "provider_not_found", Message: "no provider supplied"} } @@ -368,7 +376,11 @@ func (m *Manager) ExecuteStream(ctx context.Context, providers []string, req cli defer unlockSession() } } - normalized := m.normalizeProviders(providers) + if provider := ProviderScopeFromContext(ctx); provider != "" { + opts.EnsureMetadata() + opts.Metadata[cliproxyexecutor.ProviderScopeMetadataKey] = provider + } + normalized := m.normalizeProviders(scopedProviders(ctx, providers)) if len(normalized) == 0 { return nil, &Error{Code: "provider_not_found", Message: "no provider supplied"} } @@ -1110,12 +1122,16 @@ func (m *Manager) executeStreamMixedOnce(ctx context.Context, providers []string homeMode := m.HomeEnabled() homeAuthCount := 1 tried := make(map[string]struct{}) + homeExcludedAuthIDs := make(map[string]struct{}) + for _, authID := range homeExcludedAuthIDsFromMetadata(opts.Metadata) { + tried[authID] = struct{}{} + homeExcludedAuthIDs[authID] = struct{}{} + } if !homeMode { for authID := range m.requestRetryRoundExclusions(retryRound, defaultRequestRetry) { tried[authID] = struct{}{} } } - homeExcludedAuthIDs := make(map[string]struct{}) homeSameAuthRetries := make(map[string]int) lastHomeAuthID := "" homeSameAuthRetryPending := false diff --git a/sdk/cliproxy/auth/conductor_home.go b/sdk/cliproxy/auth/conductor_home.go index 23ee990acf2..ecb3a2c7da3 100644 --- a/sdk/cliproxy/auth/conductor_home.go +++ b/sdk/cliproxy/auth/conductor_home.go @@ -25,7 +25,7 @@ const ( homeRetryRoundMetadataKey = "request_retry_round" // ExcludedAuthIDsMetadataKey stores credential IDs already attempted in the // current request retry round. - ExcludedAuthIDsMetadataKey = "excluded_auth_ids" + ExcludedAuthIDsMetadataKey = cliproxyexecutor.ExcludedAuthIDsMetadataKey // CloseAllExecutionSessionsID asks an executor to release all active execution sessions. // Executors that do not support this marker may ignore it. CloseAllExecutionSessionsID = "__all_execution_sessions__" @@ -1005,7 +1005,14 @@ func (m *Manager) pickNextViaHome(ctx context.Context, model string, opts clipro return auth, executor, provider, nil } -func (m *Manager) pickHomeDispatchSelection(ctx context.Context, model string, opts cliproxyexecutor.Options) (*HomeDispatchSelection, error) { +func (m *Manager) pickHomeDispatchSelection(ctx context.Context, model string, opts cliproxyexecutor.Options) (selected *HomeDispatchSelection, errSelection error) { + defer func() { + if scope := ProviderScopeFromContext(ctx); scope != "" && selected != nil && !strings.EqualFold(selected.Provider, scope) { + selected.End("provider_scope_mismatch") + selected = nil + errSelection = &Error{Code: "auth_not_found", Message: "home dispatch returned a credential outside the provider scope", HTTPStatus: http.StatusServiceUnavailable} + } + }() if m == nil { return nil, &Error{Code: "auth_not_found", Message: "no auth available"} } diff --git a/sdk/cliproxy/auth/conductor_selection.go b/sdk/cliproxy/auth/conductor_selection.go index 1fca3160023..fb5be968a64 100644 --- a/sdk/cliproxy/auth/conductor_selection.go +++ b/sdk/cliproxy/auth/conductor_selection.go @@ -2734,13 +2734,21 @@ func (m *Manager) pickNext(ctx context.Context, provider, model string, opts cli // HasAlternativeTargetForModel reports whether at least one eligible credential // other than excludedAuthID can serve model across providers. func (m *Manager) HasAlternativeTargetForModel(ctx context.Context, providers []string, model string, opts cliproxyexecutor.Options, excludedAuthID string) bool { + available, _ := m.CheckAlternativeTargetForModel(ctx, providers, model, opts, excludedAuthID) + return available +} + +// CheckAlternativeTargetForModel checks replay availability without selecting an +// account or changing affinity. A cooling sibling retains its allowance even +// when excluded from replay, so a spent account cannot conceal a short cooldown. +func (m *Manager) CheckAlternativeTargetForModel(ctx context.Context, providers []string, model string, opts cliproxyexecutor.Options, excludedAuthID string) (bool, error) { if m == nil { - return false + return false, nil } excludedAuthID = strings.TrimSpace(excludedAuthID) opts = ensureRequestedModelMetadata(opts, model) if m.HomeEnabled() { - return m.hasAlternativeHomeTargetForModel(ctx, model, opts, excludedAuthID) + return m.hasAlternativeHomeTargetForModel(ctx, model, opts, excludedAuthID), nil } normalized := m.normalizeProviders(providers) providerSet := make(map[string]struct{}, len(normalized)) @@ -2755,7 +2763,7 @@ func (m *Manager) HasAlternativeTargetForModel(ctx context.Context, providers [] providerSet[p] = struct{}{} } if len(providerSet) == 0 { - return false + return false, nil } eligibility := authSelectionEligibilityForRequest(ctx, opts) modelKey := canonicalModelKey(model) @@ -2764,12 +2772,13 @@ func (m *Manager) HasAlternativeTargetForModel(ctx context.Context, providers [] m.mu.RLock() defer m.mu.RUnlock() candidates := make([]*Auth, 0, len(m.auths)) + heldPool := make([]*Auth, 0, len(m.auths)) for _, candidate := range m.auths { if candidate == nil || candidate.Disabled { continue } candidateID := strings.TrimSpace(candidate.ID) - if candidateID == "" || candidateID == excludedAuthID { + if candidateID == "" { continue } if !eligibility.allows(candidate) { @@ -2788,13 +2797,34 @@ func (m *Manager) HasAlternativeTargetForModel(ctx context.Context, providers [] if modelKey != "" && !m.authSupportsRouteModel(registryRef, candidate, model) { continue } - candidates = append(candidates, candidate) + heldPool = append(heldPool, candidate) + if candidateID != excludedAuthID { + candidates = append(candidates, candidate) + } } if len(candidates) == 0 { - return false + return false, nil + } + now := time.Now() + _, available, errAvailable := m.availableAuthsForSelector(ctx, m.selector, candidates, "mixed", model, now, false, opts.Headers) + if errAvailable != nil { + return false, errAvailable + } + selector := m.selector + if affinity, ok := selector.(*SessionAffinitySelector); ok { + selector = affinity.fallback + } + if _, ok := selector.(*SoonestResetFirstSelector); ok { + available = preferCodexWebsocketAuths(ctx, "mixed", available) + usable, spent := authsWithGoverningAllowance(available, model, now) + cooling := m.coolingCandidates(ctx, heldPool, model, now, opts.Headers) + if len(usable) == 0 { + if wait, shouldWait := waitForCoolingSibling(withCoolingCandidates(ctx, cooling), spent, model, now); shouldWait { + return false, newModelCooldownError(model, "", wait) + } + } } - available, _, errAvailable := m.availableAuthsForSelector(ctx, m.selector, candidates, "mixed", model, time.Now(), false, opts.Headers) - return errAvailable == nil && len(available) > 0 + return len(available) > 0, nil } func (m *Manager) pickNextMixedLegacy(ctx context.Context, providers []string, model string, opts cliproxyexecutor.Options, tried map[string]struct{}) (*Auth, ProviderExecutor, string, error) { diff --git a/sdk/cliproxy/auth/home_retry_loop_test.go b/sdk/cliproxy/auth/home_retry_loop_test.go index e0ca0052f8e..24040cad2c5 100644 --- a/sdk/cliproxy/auth/home_retry_loop_test.go +++ b/sdk/cliproxy/auth/home_retry_loop_test.go @@ -98,3 +98,39 @@ func TestManagerExecuteHomeStopsWhenDispatchRepeatsTriedAuth(t *testing.T) { t.Fatalf("home dispatch calls = %d, want 2", got) } } + +func TestManagerExecuteStreamHomeRejectsMetadataExcludedAuthWhenRepeated(t *testing.T) { + dispatcher := &repeatedHomeAuthDispatcher{} + oldCurrentHomeDispatcher := currentHomeDispatcher + currentHomeDispatcher = func() homeAuthDispatcher { + return dispatcher + } + t.Cleanup(func() { + currentHomeDispatcher = oldCurrentHomeDispatcher + }) + + executor := &unauthorizedHomeExecutor{} + manager := NewManager(nil, nil, nil) + manager.SetConfig(&internalconfig.Config{Home: internalconfig.HomeConfig{Enabled: true}}) + manager.SetHomeExecutionRegistry(executionregistry.New()) + manager.RegisterExecutor(executor) + + ctx, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + + opts := cliproxyexecutor.Options{ + Metadata: map[string]any{ + ExcludedAuthIDsMetadataKey: []string{"home-auth-1"}, + }, + } + _, err := manager.ExecuteStream(ctx, []string{"home-loop-test"}, cliproxyexecutor.Request{Model: "gemini-3.5-flash-low"}, opts) + if err == nil { + t.Fatal("ExecuteStream error = nil, want repeated auth rejection") + } + if got := executor.calls.Load(); got != 0 { + t.Fatalf("executor calls = %d, want 0 when dispatch returns metadata-excluded auth", got) + } + if got := dispatcher.calls.Load(); got != 1 { + t.Fatalf("home dispatch calls = %d, want 1", got) + } +} diff --git a/sdk/cliproxy/auth/provider_scope.go b/sdk/cliproxy/auth/provider_scope.go new file mode 100644 index 00000000000..60a0fff138c --- /dev/null +++ b/sdk/cliproxy/auth/provider_scope.go @@ -0,0 +1,50 @@ +package auth + +import ( + "context" + "strings" + + cliproxyexecutor "github.com/rune-oai/t3code-proxy/sdk/cliproxy/executor" +) + +type providerScopeContextKey struct{} + +// WithProviderScope restricts a client request to a registry provider key. +func WithProviderScope(ctx context.Context, provider string) context.Context { + if ctx == nil { + ctx = context.Background() + } + return context.WithValue(ctx, providerScopeContextKey{}, strings.ToLower(strings.TrimSpace(provider))) +} + +// ProviderScopeFromContext returns the provider requested by a scoped endpoint. +func ProviderScopeFromContext(ctx context.Context) string { + if ctx == nil { + return "" + } + provider, _ := ctx.Value(providerScopeContextKey{}).(string) + return provider +} + +func scopedProviders(ctx context.Context, providers []string) []string { + scope := ProviderScopeFromContext(ctx) + if scope == "" { + return providers + } + for _, provider := range providers { + if strings.EqualFold(provider, scope) { + return []string{scope} + } + } + return nil +} + +func providerAffinityNamespace(provider string, metadata map[string]any) string { + if provider == "" { + return "" + } + if scope := sessionMetadataString(metadata, cliproxyexecutor.ProviderScopeMetadataKey); scope != "" { + return "provider/" + scope + "/" + provider + } + return provider +} diff --git a/sdk/cliproxy/auth/provider_scope_test.go b/sdk/cliproxy/auth/provider_scope_test.go new file mode 100644 index 00000000000..3ec927c69dd --- /dev/null +++ b/sdk/cliproxy/auth/provider_scope_test.go @@ -0,0 +1,35 @@ +package auth + +import ( + "context" + "strings" + "testing" + + "github.com/rune-oai/t3code-proxy/internal/config" + "github.com/rune-oai/t3code-proxy/sdk/cliproxy/executionregistry" + cliproxyexecutor "github.com/rune-oai/t3code-proxy/sdk/cliproxy/executor" +) + +func TestProviderScopeAffinityNamespaces(t *testing.T) { + metadata := map[string]any{cliproxyexecutor.CallerScopeMetadataKey: "caller"} + unscoped := lcpAffinityNamespace("mixed", "model", metadata) + metadata[cliproxyexecutor.ProviderScopeMetadataKey] = "antigravity" + antigravity := lcpAffinityNamespace("mixed", "model", metadata) + metadata[cliproxyexecutor.ProviderScopeMetadataKey] = "claude" + claude := lcpAffinityNamespace("mixed", "model", metadata) + if unscoped == antigravity || antigravity == claude || unscoped == claude { + t.Fatalf("content affinity namespaces overlap: %q %q %q", unscoped, antigravity, claude) + } +} + +func TestProviderScopeRejectsMismatchedHomeDispatch(t *testing.T) { + manager := NewManager(nil, nil, nil) + manager.SetConfig(&config.Config{Home: config.HomeConfig{Enabled: true}}) + manager.PublishHomeDispatch(homeExecutionDispatcher{}, executionregistry.New(), 1) + manager.RegisterExecutor(&homeExecutionExecutor{}) + ctx := WithProviderScope(context.Background(), "antigravity") + selection, errSelection := manager.pickHomeDispatchSelection(ctx, "model", cliproxyexecutor.Options{}) + if selection != nil || errSelection == nil || !strings.Contains(errSelection.Error(), "outside the provider scope") { + t.Fatalf("selection=%v error=%v", selection, errSelection) + } +} diff --git a/sdk/cliproxy/auth/selector.go b/sdk/cliproxy/auth/selector.go index 0aa4d71c2fd..940e02bdeaf 100644 --- a/sdk/cliproxy/auth/selector.go +++ b/sdk/cliproxy/auth/selector.go @@ -1705,7 +1705,7 @@ func (s *SessionAffinitySelector) Pick(ctx context.Context, provider, model stri fallbackAuths := highestPriorityAuths(available) modelKey := canonicalModelKey(model) - cacheKey := provider + "::" + primaryID + "::" + modelKey + cacheKey := providerAffinityNamespace(provider, opts.Metadata) + "::" + primaryID + "::" + modelKey isFork := false if opts.Metadata != nil { if forkFlag, ok := opts.Metadata[cliproxyexecutor.IsForkMetadataKey].(bool); ok && forkFlag { @@ -1715,7 +1715,7 @@ func (s *SessionAffinitySelector) Pick(ctx context.Context, provider, model stri isSubagent := !isFork && isSubagentSession(primaryID, fallbackID) fallbackKey := "" if fallbackID != "" && fallbackID != primaryID { - fallbackKey = provider + "::" + fallbackID + "::" + modelKey + fallbackKey = providerAffinityNamespace(provider, opts.Metadata) + "::" + fallbackID + "::" + modelKey } bind := func(authID string, recordSelection bool) { keys := []string{cacheKey} @@ -2046,7 +2046,7 @@ func canonicalLCPProvider(provider string) string { } func lcpAffinityNamespace(provider, model string, metadata map[string]any) string { - provider = canonicalLCPProvider(provider) + provider = providerAffinityNamespace(canonicalLCPProvider(provider), metadata) model = canonicalModelKey(model) callerScope := sessionMetadataString(metadata, cliproxyexecutor.CallerScopeMetadataKey) if provider == "" || callerScope == "" { @@ -2366,10 +2366,10 @@ func (s *SessionAffinitySelector) OnResult(res Result) { fallbackID = cliproxysession.BoundSessionIdentity(fallbackID) } - cacheKey := ns + "::" + primaryID + "::" + nsModel + cacheKey := providerAffinityNamespace(ns, res.Options.Metadata) + "::" + primaryID + "::" + nsModel var fallbackKey string if fallbackID != "" && fallbackID != primaryID && !isSubagentSession(primaryID, fallbackID) { - fallbackKey = ns + "::" + fallbackID + "::" + nsModel + fallbackKey = providerAffinityNamespace(ns, res.Options.Metadata) + "::" + fallbackID + "::" + nsModel } if res.Success { s.cache.Touch(cacheKey, res.AuthID) @@ -2439,14 +2439,14 @@ func (s *SessionAffinitySelector) boundAuthForRequest(providers []string, model if primaryID != "" { for _, candProvider := range candidateProviders { - if authID, ok := s.cache.Get(candProvider + "::" + primaryID + "::" + modelKey); ok && authID != "" { + if authID, ok := s.cache.Get(providerAffinityNamespace(candProvider, opts.Metadata) + "::" + primaryID + "::" + modelKey); ok && authID != "" { return authID, true } } } if fallbackID != "" && fallbackID != primaryID && !isSubagent && !isFork { for _, candProvider := range candidateProviders { - if authID, ok := s.cache.Get(candProvider + "::" + fallbackID + "::" + modelKey); ok && authID != "" { + if authID, ok := s.cache.Get(providerAffinityNamespace(candProvider, opts.Metadata) + "::" + fallbackID + "::" + modelKey); ok && authID != "" { return authID, true } } diff --git a/sdk/cliproxy/auth/selector_reset.go b/sdk/cliproxy/auth/selector_reset.go index 0b3963d01f7..956e4815da0 100644 --- a/sdk/cliproxy/auth/selector_reset.go +++ b/sdk/cliproxy/auth/selector_reset.go @@ -53,17 +53,7 @@ func (s *SoonestResetFirstSelector) Pick(ctx context.Context, provider, model st available = preferCodexWebsocketAuths(ctx, provider, available) // Skip accounts whose governing allowance for this model is spent. When every account // is spent, keep them all so ranking picks the one whose cap resets first. - scoped := make(map[*Auth]scopedWeeklyState) - usable := make([]*Auth, 0, len(available)) - for _, auth := range available { - state, ok := governingQuotaWindowFor(auth, model, now) - if ok { - scoped[auth] = state - } - if !ok || !state.exhausted { - usable = append(usable, auth) - } - } + usable, scoped := authsWithGoverningAllowance(available, model, now) if len(usable) > 0 { available = usable } else if wait, ok := waitForCoolingSibling(ctx, scoped, model, now); ok { @@ -136,6 +126,23 @@ type coolingCandidate struct { until time.Time } +// authsWithGoverningAllowance keeps quota filtering consistent between selection +// and read-only replay-target checks. +func authsWithGoverningAllowance(available []*Auth, model string, now time.Time) ([]*Auth, map[*Auth]scopedWeeklyState) { + scoped := make(map[*Auth]scopedWeeklyState) + usable := make([]*Auth, 0, len(available)) + for _, auth := range available { + state, ok := governingQuotaWindowFor(auth, model, now) + if ok { + scoped[auth] = state + } + if !ok || !state.exhausted { + usable = append(usable, auth) + } + } + return usable, scoped +} + type coolingCandidatesKey struct{} // withCoolingCandidates tells the selector which candidates are cooling down. diff --git a/sdk/cliproxy/auth/selector_reset_cooling_test.go b/sdk/cliproxy/auth/selector_reset_cooling_test.go index d2eedf18ac8..c2bf65a0f35 100644 --- a/sdk/cliproxy/auth/selector_reset_cooling_test.go +++ b/sdk/cliproxy/auth/selector_reset_cooling_test.go @@ -131,6 +131,21 @@ func TestManagerWaitsForCoolingAccountInsteadOfSpentOne(t *testing.T) { t.Fatalf("the failed account is not on hold (blocked=%v until %v)", blocked, next) } + // Replay probes must retain the excluded account in their cooling pool. + // They are read-only and honor the configured selector, including affinity. + for _, selector := range []Selector{&SoonestResetFirstSelector{}, NewSessionAffinitySelectorWithConfig(SessionAffinityConfig{Fallback: &SoonestResetFirstSelector{}})} { + manager.SetSelector(selector) + if manager.HasAlternativeTargetForModel(ctx, []string{"codex"}, model, cliproxyexecutor.Options{}, open.ID) { + t.Fatal("spent account counted as a replay target while the account with allowance cools down") + } + available, errAlternative := manager.CheckAlternativeTargetForModel(ctx, []string{"codex"}, model, cliproxyexecutor.Options{}, open.ID) + var cooldown *modelCooldownError + if available || !errors.As(errAlternative, &cooldown) || cooldown.resetIn <= 0 || cooldown.resetIn > 5*time.Minute { + t.Fatalf("alternative = %v, %v; want the short cooldown", available, errAlternative) + } + } + manager.SetSelector(&SoonestResetFirstSelector{}) + got, _, err = manager.pickNext(ctx, "codex", model, cliproxyexecutor.Options{}, nil) var cooldown *modelCooldownError if got != nil || !errors.As(err, &cooldown) { diff --git a/sdk/cliproxy/executor/types.go b/sdk/cliproxy/executor/types.go index d34a17a16aa..2e4925a7671 100644 --- a/sdk/cliproxy/executor/types.go +++ b/sdk/cliproxy/executor/types.go @@ -8,6 +8,9 @@ import ( sdktranslator "github.com/rune-oai/t3code-proxy/sdk/translator" ) +// ProviderScopeMetadataKey isolates affinity bindings for provider-scoped endpoints. +const ProviderScopeMetadataKey = "provider_scope" + // RequestedModelMetadataKey stores the client-requested model name in Options.Metadata. const RequestedModelMetadataKey = "requested_model" @@ -34,6 +37,8 @@ const GenerateMetadataKey = "generate" const ( // PinnedAuthMetadataKey locks execution to a specific auth ID. PinnedAuthMetadataKey = "pinned_auth_id" + // ExcludedAuthIDsMetadataKey excludes specific auth IDs from selection for the current attempt. + ExcludedAuthIDsMetadataKey = "excluded_auth_ids" // SelectionIDMetadataKey stores the per-selection ID generated for a plugin scheduler pick. SelectionIDMetadataKey = "selection_id" // SelectedAuthMetadataKey stores the auth ID selected by the scheduler.