From d46f7eb58affefb41121d61233fbe86a901bb073 Mon Sep 17 00:00:00 2001 From: Erik M Jacobs Date: Fri, 3 Jul 2026 10:45:05 -0400 Subject: [PATCH] OLS-3442 Add reasoning token support spec Update service specs for per-model reasoning_config across all providers. Covers provider-specific reasoning enablement, vLLM ChatVLLMReasoning subclass, streaming reasoning accumulation, Bedrock provider addition, and deprecation of reasoning_effort/reasoning_summary/verbosity fields. Co-Authored-By: Claude Sonnet 4.6 (1M context) --- .ai/spec/how/llm-providers.md | 34 +++++++++++---- .ai/spec/how/query-pipeline.md | 6 ++- .ai/spec/what/config.md | 11 ++--- .ai/spec/what/conversation-history.md | 2 + .ai/spec/what/llm-providers.md | 61 +++++++++++++++++++-------- 5 files changed, 80 insertions(+), 34 deletions(-) diff --git a/.ai/spec/how/llm-providers.md b/.ai/spec/how/llm-providers.md index 43e05d005..d3da0ff58 100644 --- a/.ai/spec/how/llm-providers.md +++ b/.ai/spec/how/llm-providers.md @@ -28,13 +28,15 @@ The LLM provider subsystem translates a (provider name, model name) pair from co | File | Class | Decorator key | LangChain class | Notes | |---|---|---|---|---| -| `openai.py` | `OpenAI` | `"openai"` | `ChatOpenAI` | Reasoning params for o-series/gpt-5 models | -| `azure_openai.py` | `AzureOpenAI` | `"azure_openai"` | `AzureChatOpenAI` | Entra ID token caching; see below | +| `openai.py` | `OpenAI` | `"openai"` | `ChatOpenAI` | Reasoning via `reasoning_config`; [PLANNED: OLS-3442 — replace model-name detection] | +| `azure_openai.py` | `AzureOpenAI` | `"azure_openai"` | `AzureChatOpenAI` | Entra ID token caching; reasoning via `reasoning_config` [PLANNED: OLS-3442] | | `watsonx.py` | `Watsonx` | `"watsonx"` | `ChatWatsonx` | IBM-specific parameter names; see below | -| `rhoai_vllm.py` | `RHOAIVLLM` | `"rhoai_vllm"` | `ChatOpenAI` | OpenAI-compatible, no default URL | -| `rhelai_vllm.py` | `RHELAIVLLM` | `"rhelai_vllm"` | `ChatOpenAI` | OpenAI-compatible, no default URL | -| `google_vertex.py` | `GoogleVertex` | `"google_vertex"` | `ChatGoogleGenerativeAI` | Gemini / publisher models; credentials via `load_vertex_credentials` | -| `google_vertex.py` | `GoogleVertexAnthropic` | `"google_vertex_anthropic"` | `ChatAnthropicVertex` | Claude on Vertex Model Garden; same module | +| `rhoai_vllm.py` | `RHOAIVLLM` | `"rhoai_vllm"` | `ChatOpenAI` or `ChatVLLMReasoning` | OpenAI-compatible, no default URL. [PLANNED: OLS-3442 — `ChatVLLMReasoning` when `reasoning_config.enabled`] | +| `rhelai_vllm.py` | `RHELAIVLLM` | `"rhelai_vllm"` | `ChatOpenAI` or `ChatVLLMReasoning` | OpenAI-compatible, no default URL. [PLANNED: OLS-3442 — `ChatVLLMReasoning` when `reasoning_config.enabled`] | +| `google_vertex.py` | `GoogleVertex` | `"google_vertex"` | `ChatGoogleGenerativeAI` | Gemini / publisher models; credentials via `load_vertex_credentials`. [PLANNED: OLS-3442 — thinking config via `reasoning_config`] | +| `google_vertex.py` | `GoogleVertexAnthropic` | `"google_vertex_anthropic"` | `ChatAnthropicVertex` | Claude on Vertex Model Garden; same module. [PLANNED: OLS-3442 — thinking config via `reasoning_config`] | +| `bedrock.py` | `Bedrock` | `"bedrock"` | `ChatBedrockConverse` or `ChatOpenAI` | AWS Bedrock via Mantle gateway; routes by model prefix (`anthropic.*` → Converse, `openai.*` → Responses API). [PLANNED: OLS-3442 — reasoning config] | +| `vllm_reasoning.py` | `ChatVLLMReasoning` | (not registered) | N/A | [PLANNED: OLS-3442] `BaseChatOpenAI` subclass that captures `reasoning_content`/`reasoning` from vLLM responses. Not a provider — used by vLLM providers when reasoning is enabled | | `fake_provider.py` | `FakeProvider` | `"fake_provider"` | `FakeListLLM` / `FakeStreamingListLLM` | Testing only; monkey-patches `bind_tools` | ### `ols/src/llms/providers/utils.py` -- Shared utilities @@ -126,7 +128,7 @@ Each provider follows the same pattern: read generic fields from `ProviderConfig ### Constants -- Provider type strings: `PROVIDER_OPENAI`, `PROVIDER_AZURE_OPENAI`, `PROVIDER_WATSONX`, `PROVIDER_RHOAI_VLLM`, `PROVIDER_RHELAI_VLLM`, `PROVIDER_FAKE`, `PROVIDER_GOOGLE_VERTEX`, `PROVIDER_GOOGLE_VERTEX_ANTHROPIC` in `ols/constants.py`. +- Provider type strings: `PROVIDER_OPENAI`, `PROVIDER_AZURE_OPENAI`, `PROVIDER_WATSONX`, `PROVIDER_RHOAI_VLLM`, `PROVIDER_RHELAI_VLLM`, `PROVIDER_FAKE`, `PROVIDER_GOOGLE_VERTEX`, `PROVIDER_GOOGLE_VERTEX_ANTHROPIC`, `PROVIDER_BEDROCK` in `ols/constants.py`. - `SUPPORTED_PROVIDER_TYPES` frozenset is checked during config validation (not by the registry). ### TLS and proxy @@ -162,9 +164,23 @@ WatsonX uses IBM's `GenTextParamsMetaNames` constants for parameter keys instead WatsonX also passes `params=self.params` to `ChatWatsonx` (as a nested dict), unlike OpenAI-compatible providers that spread params as `**self.params`. -### OpenAI reasoning model handling +### Reasoning model handling -Both `openai.py` and `azure_openai.py` detect o-series and gpt-5 models by name pattern (`self.model.startswith("o")` or `"gpt-5" in self.model`). For these models, temperature/top_p/frequency_penalty are omitted and reasoning parameters (effort, summary) are set instead. The `ModelParameters` config class provides `reasoning_effort`, `reasoning_summary`, and `verbosity` fields. +[PLANNED: OLS-3442] Currently, `openai.py` and `azure_openai.py` detect o-series and gpt-5 models by name pattern (`self.model.startswith("o")` or `"gpt-5" in self.model`). This model-name detection will be replaced by config-driven enablement via `reasoning_config`. + +When `reasoning_config` is present in a model's `ModelParameters`, the provider reads provider-specific keys from it and passes them to the LangChain adapter. Standard sampling parameters (`temperature`, `top_p`, `frequency_penalty`) are skipped. When `reasoning_config` is absent, the provider applies standard sampling defaults. + +The existing `ModelParameters` fields `reasoning_effort`, `reasoning_summary`, and `verbosity` will be removed and replaced by the freeform `reasoning_config` dict. + +### vLLM reasoning subclass (`ChatVLLMReasoning`) + +[PLANNED: OLS-3442] `ChatVLLMReasoning` in `vllm_reasoning.py` subclasses `BaseChatOpenAI` (same base as `ChatDeepSeek` from `langchain-deepseek`) and overrides two methods: + +1. **`_create_chat_result()`** — non-streaming. Calls `super()`, extracts `reasoning_content` or `reasoning` from the raw `openai.BaseModel` response (which preserves unknown fields via Pydantic `extra="allow"`), stores in `message.additional_kwargs["reasoning_content"]`. + +2. **`_convert_chunk_to_generation_chunk()`** — streaming. Calls `super()`, extracts reasoning from `chunk["choices"][0]["delta"]` dict, stores in `message.additional_kwargs["reasoning_content"]`. + +Both vLLM field names (`reasoning_content` and `reasoning`) are checked to handle the vLLM rename (RFC #27755). Output is normalized to `additional_kwargs["reasoning_content"]` regardless of source field. No new Python dependencies required. ### HTTP client setup diff --git a/.ai/spec/how/query-pipeline.md b/.ai/spec/how/query-pipeline.md index 587a73718..652e99fae 100644 --- a/.ai/spec/how/query-pipeline.md +++ b/.ai/spec/how/query-pipeline.md @@ -217,7 +217,7 @@ All pipeline output flows through `StreamedChunk(type, text, data)`. The `Stream | Type | When emitted | Payload | |---|---|---| | `TEXT` | LLM text tokens | `text` field | -| `REASONING` | Model reasoning tokens (e.g. OpenAI o-series) | `text` field | +| `REASONING` | Model reasoning tokens. Source varies by provider: LangChain `content_blocks` (OpenAI, Gemini, Anthropic) or `additional_kwargs["reasoning_content"]` (vLLM via `ChatVLLMReasoning`). [PLANNED: OLS-3442] | `text` field | | `TOOL_CALL` | Before tool execution | `data`: tool name, args, id, server metadata | | `APPROVAL_REQUIRED` | Tool requires user approval | `data`: approval request details | | `TOOL_RESULT` | After tool execution | `data`: id, name, status, content, round, structured_content | @@ -297,7 +297,9 @@ The budget is calculated once in `DocsSummarizer.__init__` via `TokenBudgetTrack ### How streaming chunks interleave during tool calling -In non-final rounds, `LLMExecutionAgent._invoke_llm()` binds tools to the LLM (no `tool_choice` constraint). `_collect_round_llm_chunks()` accumulates all chunks from one LLM invocation, separating tool-call chunks from text/reasoning chunks. Text chunks are yielded immediately for streaming. After the LLM stream ends, `_process_tool_calls_for_round()` emits `TOOL_CALL` events, executes the tools, then emits `TOOL_RESULT` events. The AI message (including any reasoning content) and all tool messages are appended to the prompt template for the next round. +In non-final rounds, `LLMExecutionAgent._invoke_llm()` binds tools to the LLM (no `tool_choice` constraint). `_collect_round_llm_chunks()` accumulates all chunks from one LLM invocation, separating tool-call chunks from text/reasoning chunks. Text and reasoning chunks are yielded immediately for streaming. [PLANNED: OLS-3442] For vLLM providers, reasoning arrives via `chunk.additional_kwargs["reasoning_content"]` (captured by `ChatVLLMReasoning`) rather than `content_blocks` — a check for this source is added after the existing `match chunk.content` block. After the LLM stream ends, `_process_tool_calls_for_round()` emits `TOOL_CALL` events, executes the tools, then emits `TOOL_RESULT` events. The AI message (including any reasoning content) and all tool messages are appended to the prompt template for the next round. + +[PLANNED: OLS-3442] In `streaming_ols.py`, both `StreamChunkType.TEXT` and `StreamChunkType.REASONING` chunks are accumulated into the `response` string. Previously only TEXT chunks were accumulated, causing reasoning to be lost from the stored conversation history. On the final round (round == max_rounds or no tools available), the LLM is invoked with `tool_choice="none"` and `strict=False` to force a text-only response. The Granite model family requires special handling: the first 6 chunks of a tool call (`"", "<", "tool", "_", "call", ">"`) are suppressed via `skip_special_chunk()` to avoid leaking tool-call markup to the user. diff --git a/.ai/spec/what/config.md b/.ai/spec/what/config.md index dd5d4aad4..6f5f128c4 100644 --- a/.ai/spec/what/config.md +++ b/.ai/spec/what/config.md @@ -109,7 +109,7 @@ Each provider entry under `llm_providers` supports: | Field Path | Type | Default | Purpose | |------------|------|---------|---------| | `llm_providers[].name` | string | (required) | Provider name | -| `llm_providers[].type` | string | =name | Provider type (openai, azure_openai, watsonx, rhoai_vllm, rhelai_vllm, google_vertex, google_vertex_anthropic, fake_provider) | +| `llm_providers[].type` | string | =name | Provider type (openai, azure_openai, watsonx, rhoai_vllm, rhelai_vllm, google_vertex, google_vertex_anthropic, bedrock, fake_provider) | | `llm_providers[].url` | URL | none | Provider endpoint URL | | `llm_providers[].credentials_path` | string | none | Path to credential file or directory | | `llm_providers[].models[]` | list | (required, >= 1) | Model definitions | @@ -117,9 +117,10 @@ Each provider entry under `llm_providers` supports: | `llm_providers[].models[].context_window_size` | int | 128000 | Context window size in tokens | | `llm_providers[].models[].parameters.max_tokens_for_response` | int | 4096 | Tokens reserved for response | | `llm_providers[].models[].parameters.tool_budget_ratio` | float | 0.25 | Fraction of context window for tool outputs (0.1--0.6) | -| `llm_providers[].models[].parameters.reasoning_effort` | enum | low | Reasoning effort level (low, medium, high) | -| `llm_providers[].models[].parameters.reasoning_summary` | enum | concise | Reasoning summary style (auto, concise, detailed) | -| `llm_providers[].models[].parameters.verbosity` | enum | low | General verbosity level (low, medium, high) | +| `llm_providers[].models[].parameters.reasoning_config` | dict | none | [PLANNED: OLS-3442] Freeform dict of provider-specific reasoning/thinking parameters. See `what/llm-providers.md` rule 13 for valid keys per provider | +| `llm_providers[].models[].parameters.reasoning_effort` | enum | low | [DEPRECATED: OLS-3442] Reasoning effort level (low, medium, high). Replaced by `reasoning_config` | +| `llm_providers[].models[].parameters.reasoning_summary` | enum | concise | [DEPRECATED: OLS-3442] Reasoning summary style (auto, concise, detailed). Replaced by `reasoning_config` | +| `llm_providers[].models[].parameters.verbosity` | enum | low | [DEPRECATED: OLS-3442] General verbosity level (low, medium, high). Replaced by `reasoning_config` | | `llm_providers[].models[].options` | dict | none | Arbitrary key-value model options | | `llm_providers[]._config` | object | none | Provider-specific config (at most one per provider) | | `llm_providers[].tlsSecurityProfile` | object | none | TLS security profile for provider connection | @@ -149,7 +150,7 @@ Each provider entry under `llm_providers` supports: 13. `default_model` must name a model that exists within the default provider's model list. 14. `conversation_cache.type` must be either `memory` or `postgres`. The corresponding sub-section (`memory` or `postgres`) must be present when the type is specified. 15. `authentication_config.module` must be one of: `k8s`, `noop`, `noop-with-token`. -16. Provider `type` must be one of the supported provider types: openai, azure_openai, watsonx, rhoai_vllm, rhelai_vllm, google_vertex, google_vertex_anthropic, fake_provider. +16. Provider `type` must be one of the supported provider types: openai, azure_openai, watsonx, rhoai_vllm, rhelai_vllm, google_vertex, google_vertex_anthropic, bedrock, fake_provider. 17. `project_id` is required when provider type is `watsonx`. ### Tool Budget Computation diff --git a/.ai/spec/what/conversation-history.md b/.ai/spec/what/conversation-history.md index 38ac1a2fa..8ab2259e0 100644 --- a/.ai/spec/what/conversation-history.md +++ b/.ai/spec/what/conversation-history.md @@ -8,6 +8,8 @@ The conversation history subsystem preserves prior exchanges within a conversati 2. When a query-response exchange is stored, the system must either append it to an existing conversation (if one exists for the given user_id + conversation_id) or create a new conversation. Each stored exchange contains the user query (HumanMessage), the AI response (AIMessage), any attachments submitted with the query, any tool calls requested by the model, and any tool results returned from tool execution. +3. [PLANNED: OLS-3442] The AI response stored in the cache must include both text and reasoning content accumulated during streaming. Reasoning chunks (`StreamChunkType.REASONING`) must be accumulated into the response string alongside text chunks, so the model has access to its own reasoning within the current conversation turn. The response is stored as a plain-string `AIMessage` — no structured reasoning blocks or provider-specific signatures are preserved. Cache schema changes for structured reasoning storage are deferred until testing and evals show they improve multi-turn quality. + 3. Retrieving history for a user_id + conversation_id must return all stored exchanges ordered oldest to newest. If no conversation exists for the given key, the system must return an empty result. 4. Listing conversations for a user must return metadata for every conversation belonging to that user, ordered by last message timestamp (most recent first). Each entry includes: conversation_id, topic_summary, last_message_timestamp, and message_count. diff --git a/.ai/spec/what/llm-providers.md b/.ai/spec/what/llm-providers.md index d983c6626..2954d1d2c 100644 --- a/.ai/spec/what/llm-providers.md +++ b/.ai/spec/what/llm-providers.md @@ -33,13 +33,20 @@ Every provider must satisfy all of the following: ### Reasoning Model Support -11. Models must be classified as reasoning-capable or standard based on their name. A model is reasoning-capable if its name contains `gpt-5` or starts with `o` (the OpenAI o-series pattern). +11. Reasoning enablement is an explicit per-model configuration decision, not auto-detected from the model name. When a model's `parameters.reasoning_config` is present (a freeform `dict[str, Any]`), the provider must apply provider-specific reasoning/thinking parameters to the LLM invocation and must not set standard sampling parameters (`temperature`, `top_p`, `frequency_penalty`). When `reasoning_config` is absent, the provider must apply standard sampling defaults. -12. For reasoning-capable models, the provider must set reasoning-specific parameters (`reasoning_effort`, `reasoning_summary`, `verbosity`) drawn from the model's `parameters` configuration. Standard sampling parameters (`temperature`, `top_p`, `frequency_penalty`) must not be set. +12. Each provider interprets the keys within `reasoning_config` according to its own backend API. Invalid keys produce a clear error from the provider API — OLS does not validate model-reasoning compatibility. The operator is responsible for correct configuration. [PLANNED: OLS-3442] -13. For standard (non-reasoning) models, the provider must set sampling parameters with sensible defaults (`temperature`, `top_p`, `frequency_penalty`). Reasoning parameters must not be set. +13. The valid `reasoning_config` keys vary by provider and model generation: + - **OpenAI / Azure OpenAI**: `effort` (low/medium/high), `summary` (auto/concise/detailed), `verbosity` (low/medium/high). Passed as the `reasoning` dict and `verbosity` param to `ChatOpenAI` / `AzureChatOpenAI`. + - **Gemini 3 (Vertex)**: `thinking_level` (low/medium/high), `include_thoughts` (bool). Passed as kwargs to `ChatGoogleGenerativeAI`. + - **Gemini 2.5 (Vertex)**: `thinking_budget` (int token count), `include_thoughts` (bool). `thinking_budget` and `thinking_level` are mutually exclusive at the Gemini API level. + - **Anthropic (Vertex)**: `type` (enabled/adaptive), `display` (omitted/summarized), optionally `budget_tokens` (int). Passed as the `thinking` dict to `ChatAnthropicVertex`. + - **Bedrock (Anthropic models)**: `type` (adaptive). Passed as thinking configuration to `ChatBedrockConverse`. + - **Bedrock (OpenAI models)**: `effort` (low/medium/high), `summary` (auto/concise/detailed). Passed as the `reasoning` dict to `ChatOpenAI`. + - **vLLM (RHELAI / RHOAI)**: `enabled` (bool). When `true`, triggers use of the `ChatVLLMReasoning` subclass instead of `ChatOpenAI`. -14. `reasoning_effort` and `verbosity` accept values `low`, `medium`, or `high`. `reasoning_summary` accepts `auto`, `concise`, or `detailed`. +14. Provider APIs return 400 errors if reasoning parameters are sent to non-reasoning models. There is no reliable cross-provider API to auto-detect reasoning capability at startup. [PLANNED: OLS-3442] ### HTTP Client Requirements @@ -57,7 +64,7 @@ The following sections describe only what differs from the standard contract abo ### OpenAI (`openai`) -19. Default URL: `https://api.openai.com/v1`. Uses `ChatOpenAI` from LangChain. No deviations from the standard contract. Uses custom certificate store. +19. Default URL: `https://api.openai.com/v1`. Uses `ChatOpenAI` from LangChain. Uses custom certificate store. When `reasoning_config` is present, passes it as the `reasoning` dict and `verbosity` param; skips temperature/top_p/frequency_penalty. [PLANNED: OLS-3442 — replace current model-name detection with reasoning_config] ### Azure OpenAI (`azure_openai`) @@ -89,27 +96,44 @@ The following sections describe only what differs from the standard contract abo 30. Uses the OpenAI-compatible API via `ChatOpenAI`. No default URL is defined (falls back to `https://api.openai.com/v1` but the admin must configure the actual endpoint). Uses custom certificate store. -31. Sets standard sampling defaults (`temperature`, `top_p`, `frequency_penalty`) regardless of model name -- no reasoning model detection. +31. Sets standard sampling defaults (`temperature`, `top_p`, `frequency_penalty`) when `reasoning_config` is absent. + +32. [PLANNED: OLS-3442] When `reasoning_config` is present with `enabled: true`, uses `ChatVLLMReasoning` (a `BaseChatOpenAI` subclass) instead of `ChatOpenAI`. `ChatVLLMReasoning` captures the `reasoning_content` (or `reasoning`) field from vLLM responses that `ChatOpenAI` drops. LangChain's `ChatOpenAI` explicitly does not preserve non-standard response fields from third-party providers; the LangChain team closed this feature request as "not planned." The subclass follows the same pattern used by `ChatDeepSeek` from `langchain-deepseek`. No new Python dependencies are required. ### RHELAI vLLM (`rhelai_vllm`) -32. Identical to RHOAI vLLM in behavior. Registered under a different type identifier (`rhelai_vllm`) to distinguish the deployment context in configuration. +33. Identical to RHOAI vLLM in behavior. Registered under a different type identifier (`rhelai_vllm`) to distinguish the deployment context in configuration. ### Google Vertex AI - Gemini (`google_vertex`) -33. Requires `credentials` as a JSON string containing a Google service account key. The JSON is parsed and used to create Google OAuth2 credentials scoped to `https://www.googleapis.com/auth/cloud-platform`. +34. Requires `credentials` as a JSON string containing a Google service account key. The JSON is parsed and used to create Google OAuth2 credentials scoped to `https://www.googleapis.com/auth/cloud-platform`. + +35. `project` and `location` are configured via `google_vertex_config`. Default location: `global`. Uses `ChatGoogleGenerativeAI` from LangChain with `vertexai=True`. -34. `project` and `location` are configured via `google_vertex_config`. Default location: `global`. Uses `ChatGoogleGenerativeAI` from LangChain with `vertexai=True`. +36. Does not use httpx clients or custom certificate stores. `max_tokens_for_response` maps to `max_output_tokens`. -35. Does not use httpx clients or custom certificate stores. `max_tokens_for_response` maps to `max_output_tokens`. +37. [PLANNED: OLS-3442] When `reasoning_config` is present, unpacks it as kwargs (`thinking_level`, `thinking_budget`, `include_thoughts`, etc.) into the `ChatGoogleGenerativeAI` constructor. Gemini 2.5 uses `thinking_budget` (int token count); Gemini 3+ uses `thinking_level` (low/medium/high). These are mutually exclusive at the Gemini API level — sending both returns a 400 error. ### Google Vertex AI - Anthropic/Claude (`google_vertex_anthropic`) -36. Same credential handling as Google Vertex (Gemini): requires JSON service account key, same OAuth2 scope. +38. Same credential handling as Google Vertex (Gemini): requires JSON service account key, same OAuth2 scope. + +39. `project` and `location` are configured via `google_vertex_anthropic_config`. Default location: `us-east5`. Uses `ChatAnthropicVertex` from LangChain. + +40. Does not use httpx clients or custom certificate stores. `max_tokens_for_response` maps to `max_output_tokens`. + +41. [PLANNED: OLS-3442] When `reasoning_config` is present, passes it as the `thinking` dict to `ChatAnthropicVertex`. Anthropic extended thinking requires thinking blocks with cryptographic signatures to be round-tripped in subsequent messages. The current cache stores plain-string `AIMessage` objects which do not preserve these signatures. This is acceptable for now — Anthropic's API automatically handles prior-turn thinking blocks, and cache schema changes are deferred until evals show they're needed. + +### AWS Bedrock (`bedrock`) + +42. Requires `url` (Mantle gateway URL, e.g., `https://bedrock-mantle.us-east-1.api.aws`). Region is extracted from the URL hostname. Supports two authentication modes: API key via `credentials_path`, or IAM credentials via `aws_access_key_id` and `aws_secret_access_key` files with optional STS assume-role via `role_arn`. -37. `project` and `location` are configured via `google_vertex_anthropic_config`. Default location: `us-east5`. Uses `ChatAnthropicVertex` from LangChain. +43. Routes to different LangChain classes based on model prefix: + - **Anthropic models** (`anthropic.*`): Uses `ChatBedrockConverse` from `langchain-aws`. Model ID is prefixed with the region (e.g., `us.anthropic.claude-sonnet-4-6-20250514-v1:0`). + - **OpenAI models** (`openai.*`): Uses `ChatOpenAI` with `use_responses_api=True` pointed at the Mantle gateway's `/openai/v1` path. + - **Other models**: Uses `ChatOpenAI` pointed at the Mantle gateway's `/v1` path. -38. Does not use httpx clients or custom certificate stores. `max_tokens_for_response` maps to `max_output_tokens`. +44. [PLANNED: OLS-3442] When `reasoning_config` is present: for Anthropic models, passes as thinking configuration to `ChatBedrockConverse`; for OpenAI models, passes as the `reasoning` dict to `ChatOpenAI`. ### Fake Provider (`fake_provider`) @@ -122,7 +146,7 @@ The following sections describe only what differs from the standard contract abo ## Configuration Surface - `llm_providers[].name` -- Provider instance name (used as the lookup key). -- `llm_providers[].type` -- Provider type string (must match a registered type: `openai`, `azure_openai`, `watsonx`, `rhoai_vllm`, `rhelai_vllm`, `google_vertex`, `google_vertex_anthropic`, `fake_provider`). Defaults to the provider name if not specified. +- `llm_providers[].type` -- Provider type string (must match a registered type: `openai`, `azure_openai`, `watsonx`, `rhoai_vllm`, `rhelai_vllm`, `google_vertex`, `google_vertex_anthropic`, `bedrock`, `fake_provider`). Defaults to the provider name if not specified. - `llm_providers[].url` -- Base URL for the LLM API endpoint. - `llm_providers[].credentials_path` -- Path to file or directory containing the API key/token. - `llm_providers[].project_id` -- Required for WatsonX; project identifier. @@ -133,9 +157,10 @@ The following sections describe only what differs from the standard contract abo - `llm_providers[].models[].context_window_size` -- Token context window (default: 128000). - `llm_providers[].models[].credentials_path` -- Model-level credential override. - `llm_providers[].models[].parameters.max_tokens_for_response` -- Max tokens reserved for the LLM response (default: 4096). -- `llm_providers[].models[].parameters.reasoning_effort` -- Reasoning effort level: `low`, `medium`, or `high` (default: `low`). -- `llm_providers[].models[].parameters.reasoning_summary` -- Reasoning summary mode: `auto`, `concise`, or `detailed` (default: `concise`). -- `llm_providers[].models[].parameters.verbosity` -- Verbosity for reasoning models: `low`, `medium`, or `high` (default: `low`). +- `llm_providers[].models[].parameters.reasoning_config` -- [PLANNED: OLS-3442] Freeform dict of provider-specific reasoning/thinking parameters. Keys vary by provider and model generation (see rule 13). When absent, no reasoning params are sent to the provider. +- `llm_providers[].models[].parameters.reasoning_effort` -- [DEPRECATED: OLS-3442 — replaced by `reasoning_config`] Reasoning effort level: `low`, `medium`, or `high` (default: `low`). +- `llm_providers[].models[].parameters.reasoning_summary` -- [DEPRECATED: OLS-3442 — replaced by `reasoning_config`] Reasoning summary mode: `auto`, `concise`, or `detailed` (default: `concise`). +- `llm_providers[].models[].parameters.verbosity` -- [DEPRECATED: OLS-3442 — replaced by `reasoning_config`] Verbosity for reasoning models: `low`, `medium`, or `high` (default: `low`). - `llm_providers[].models[].parameters.tool_budget_ratio` -- Fraction of context window reserved for tool outputs (default: 0.25, range: 0.1--0.6). - `llm_providers[].models[].options` -- Arbitrary key-value options dict passed through to the model. - `llm_providers[].tlsSecurityProfile` -- TLS security profile with `type`, `minTLSVersion`, and `ciphers`. @@ -174,7 +199,7 @@ The following sections describe only what differs from the standard contract abo ## Planned Changes -- [PLANNED: OLS-1680] Support AWS Bedrock as an LLM provider, enabling models hosted on Amazon's managed inference service. +- [PLANNED: OLS-3442] Reasoning token support: per-model `reasoning_config` for all providers, vLLM `ChatVLLMReasoning` subclass, remove model-name detection from OpenAI/Azure providers. Replaces `reasoning_effort`, `reasoning_summary`, and `verbosity` fields. - [PLANNED: OLS-2776] Support Anthropic as a direct LLM provider (not via Google Vertex), communicating with the Anthropic API natively. Anthropic models are currently available through the Google Vertex Anthropic provider. - [PLANNED: OLS-1320] Support short-lived (rotating) tokens for all providers, replacing static API keys with tokens that are refreshed periodically. - [PLANNED: OLS-1999] Support IBM WatsonX short-lived token authentication, enabling token-based auth that refreshes automatically rather than using a static API key.