Release 3.10.0

CHANGELOG.md

@@ -11,6 +11,119 @@ target.
 
 ## [Unreleased]
 
+## [3.10.0] — 2026-10-25
+
+Upstreams that fail for the moment before any of the answer reached the
+client are now tried again instead of the error going straight back, and
+a model can set the reasoning effort its requests carry, with the
+gateway adapting when an upstream refuses a level. The thinkwatch-core
+crates move from v0.81.0 to v0.83.1.
+
+One new setting (`gateway.retry_wait_secs`), one new column
+(`models.reasoning_effort`, added on startup) and one new metric; no
+route or setting is removed.
+
+### Read before upgrading
+
+- **`models` gains a nullable column, `reasoning_effort`.** The schema
+  applied at startup adds it (`ADD COLUMN IF NOT EXISTS`); existing
+  models keep it empty and their requests go out unchanged.
+- **A reasoning level an upstream refuses is changed and sent again, for
+  every model.** This applies whether or not the model has a reasoning
+  setting and cannot be turned off: when an upstream answers 400 because
+  of the reasoning level a request carries, the request goes out once
+  more with a level the upstream accepts, or without one, and the change
+  is remembered for that provider and model for 7 days. The level sent
+  can differ from the one the caller asked for, in either direction.
+- **A request whose upstreams fail for the moment can take up to two
+  minutes to fail.** When every route of a model failed with an overload
+  or a server error, or could not be reached, before any of the answer
+  reached the client, the gateway now waits and sends the request again
+  instead of returning the error at once, for up to
+  `gateway.retry_wait_secs` (120 seconds by default) after the request
+  arrived. Clients and load balancers with a shorter timeout than that may
+  give up first. Set `gateway.retry_wait_secs` to 0 to keep the earlier
+  behaviour.
+- **A whole answer that runs into the request timeout is now an error
+  with a log row.** A non-streaming request's upstream attempt is cut
+  2 seconds before the gateway's 120-second request timeout (counted from
+  when the request reached the gateway port), and the client gets the
+  last upstream error, or a 504 `ProviderTimeout` when there was none,
+  with its `gateway_logs` row, instead of an empty 408. A retry starts
+  only when its wait and another attempt as long as the last one fit
+  before then.
+- **A stream's opening events arrive with its first content.** While a
+  stream can still be sent again, what the upstream sends before any
+  content (Anthropic `message_start` and `ping`, Responses
+  `response.created` and `response.in_progress`, a Chat chunk with only
+  the role) is held back and goes out together with the first content.
+  While it is held, and while the gateway waits to send again, the client
+  receives an SSE comment (`: keep-alive`) after every 15 seconds without
+  data — except Gemini clients, which never receive one. The status is
+  200 as before.
+
+### Added
+
+- **A model can set its reasoning** (`models.reasoning_effort`, the
+  Reasoning control in the model editor): `off` takes reasoning out of
+  requests that have it on, and a level (`low`, `medium`, `high`,
+  `xhigh`, `max`) changes the effort of requests whose reasoning is on.
+  Requests without reasoning, such as title generation, go out
+  unchanged. The setting is written the way each hop's upstream format
+  writes it: with `off`, OpenAI Chat and Responses upstreams receive
+  `none`, Gemini 2.5 Flash a thinking budget of 0, and Anthropic and
+  Bedrock upstreams no thinking; Gemini models that cannot stop thinking
+  (2.5 Pro, Gemini 3) get no thinking configuration. The response cache
+  keys on the request together with the setting.
+- **A reasoning level the upstream refuses is sent again once with one it
+  accepts.** On a 400 about the reasoning level, the request goes once
+  more to the same route with the level the error names (the lowest one
+  it lists when `none` was refused), or without one when it names none.
+  This is not counted as a retry, though its time falls within the retry
+  window. When that is answered, the choice is remembered in Redis per
+  provider, upstream model and request format for 7 days, so later
+  requests send what worked straight away; a provider's memory is
+  cleared when its URL or credentials change, a route relearns its
+  protocol, or the provider is deleted. A hop whose reasoning level was
+  changed records `"effort": {asked, sent, reason}` on its
+  `gateway_logs` row.
+- **Transient upstream failures are retried.** When every candidate route
+  has been tried and the last one failed for the moment — 500, 502, 503,
+  504 or 529, a connection that could not be made or broke before the
+  answer's head, a stream that broke off or reported an overload or a
+  server error (`overloaded_error`, `api_error`, `server_error`,
+  `server_is_overloaded`, `slow_down`, Gemini `UNAVAILABLE`) before any
+  content — and none of the answer reached the client, the request waits
+  2, 4, 8, then 16 seconds (each give or take 20%), or as long as the
+  upstream said in `retry-after-ms` or `Retry-After` (a zero wait is
+  ignored, and a shorter one than 250 ms is taken as 250 ms), and goes out again
+  to the routes that failed that way. A model with one route, and a
+  stream, goes back to the same route. Each attempt is picked as before:
+  the strategy, the circuit breaker, the route caps, session affinity
+  and the provider holding a conversation's encrypted history all apply,
+  and each failed attempt counts against its route's breaker as it did.
+  Retries stop when the next wait would end past the deadline; an attempt
+  already under way is not cut short. Rate limits and spent quota (429),
+  refused credentials (401, 403), any other 4xx, timeouts, the gateway's
+  own limits, budgets and policies, and a client that left are never
+  retried, and neither is a stream after part of its answer went out.
+- **`gateway.retry_wait_secs`** (Settings → Gateway → Routing strategy):
+  how long after a request arrives it may still be sent again, 0 to 600
+  seconds, default 120; 0 turns retries off.
+- **`gateway_provider_retry_total{provider}`**: attempts sent again after
+  a transient failure, by the provider they went to, next to
+  `gateway_provider_fallback_total`.
+- The `gateway_logs` row of a request that was sent again records
+  `"retries": n` in its detail, whether it then succeeded or failed.
+
+### Changed
+
+- A provider whose base URL cannot be made into a request fails as
+  `ProviderError` instead of `NetworkError`, and is not retried.
+- An error a stream reports as OpenAI's `server_error` is logged as the
+  upstream's 500 (`ProviderHttpError`), as the same error answered with
+  HTTP 500 is, instead of 502 (`ProviderError`).
+
 ## [3.9.1] — 2026-10-23
 
 A conversation whose history a provider encrypted, such as a Codex
@@ -2581,7 +2694,9 @@ unreleased builds should: stop the gateway, run `db/schema.sql`
 against PostgreSQL, restart against this tag. The schema is
 idempotent end-to-end, so the apply is safe to repeat.
 
-[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.9.0...HEAD
+[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.10.0...HEAD
+[3.10.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.10.0
+[3.9.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.9.1
 [3.9.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.9.0
 [3.8.3]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.3
 [3.8.2]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.2

Cargo.toml

@@ -10,7 +10,7 @@ members = [
 ]
 
 [workspace.package]
-version = "3.9.1"
+version = "3.10.0"
 edition = "2024"
 # The MSRV: the newest `rust-version` among the locked dependencies
 # (sqlx 0.9 and the aws-smithy crates under aws-sigv4). Without it,
@@ -70,10 +70,10 @@ opt-level = 3
 # never re-exported through a local shim. And the reverse: something only
 # this side uses (the at-rest crypto, IMDSv2 credentials, the gateway error)
 # lives here, not in core.
-tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.81.0" }
-tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.81.0" }
-tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.81.0" }
-tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.81.0" }
+tw-bedrock = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.83.1" }
+tw-breaker = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.83.1" }
+tw-dialect = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.83.1" }
+tw-guard = { git = "https://github.com/ThinkWatchProject/ThinkWatch-Core.git", tag = "v0.83.1" }
 
 # Web framework
 axum = { version = "0.8", features = ["macros", "ws"] }

crates/common/src/dynamic_config.rs

@@ -216,6 +216,12 @@ macro_rules! dc_getters_u64_from_i64 {
     };
 }
 
+/// `gateway.retry_wait_secs` when the setting is missing.
+pub const RETRY_WAIT_DEFAULT_SECS: i64 = 120;
+/// Upper bound on `gateway.retry_wait_secs`, enforced both at write time
+/// (the settings handler) and at read time.
+pub const RETRY_WAIT_MAX_SECS: i64 = 600;
+
 /// Upper bound on `security.signature_drift_secs`, enforced both at
 /// write time (admin settings validation) and at read time (the
 /// getter below). Anything beyond five minutes weakens replay
@@ -440,6 +446,18 @@ impl DynamicConfig {
         raw.clamp(0, 86400)
     }
 
+    /// How long after it arrived a request may still be sent again to
+    /// upstreams that failed for the moment (overloaded, a server error,
+    /// a broken connection) before any of the answer reached the client.
+    /// Clamped to [0, 600]; 0 never sends a request again.
+    pub async fn retry_wait_secs(&self) -> u64 {
+        let raw = self
+            .get_i64("gateway.retry_wait_secs")
+            .await
+            .unwrap_or(RETRY_WAIT_DEFAULT_SECS);
+        raw.clamp(0, RETRY_WAIT_MAX_SECS) as u64
+    }
+
     /// Exponent for the `latency` strategy: higher = more aggressive
     /// (heavier penalty on slow routes). Default 2.0 (aggressive).
     /// Clamped to [0.5, 8.0] so a typo can't make weights collapse

crates/common/src/models/provider.rs

@@ -65,6 +65,10 @@ pub struct Model {
     /// limit.
     #[serde(default)]
     pub max_output_tokens: Option<i32>,
+    /// The reasoning setting: `off`, or a level (`low` … `max`) for
+    /// requests whose reasoning is on. `None` ⇒ requests go out as sent.
+    #[serde(default)]
+    pub reasoning_effort: Option<String>,
 }
 
 fn default_enabled() -> bool {

crates/gateway/Cargo.toml

@@ -40,3 +40,4 @@ rust_decimal = { workspace = true }
 
 [dev-dependencies]
 http_1x = { package = "http", version = "1" }
+tokio = { workspace = true, features = ["test-util"] }

crates/gateway/src/error.rs

@@ -12,9 +12,17 @@ pub enum GatewayError {
     /// Upstream returned a non-2xx, non-429, non-401 status. The
     /// status is kept structured so error-classifier metrics stay
     /// readable and the gateway can classify retry-eligible 5xx
-    /// versus poison 4xx without parsing the message.
+    /// versus poison 4xx without parsing the message. `retry_after` is
+    /// when the upstream said to try again (`retry-after-ms` or
+    /// `Retry-After`), which an overloaded upstream can send with a 503:
+    /// the gateway waits that long before sending to it again (see
+    /// `proxy::retry`).
     #[error("Provider HTTP {status}: {message}")]
-    ProviderHttpError { status: u16, message: String },
+    ProviderHttpError {
+        status: u16,
+        message: String,
+        retry_after: Option<std::time::Duration>,
+    },
     /// Upstream took longer than the configured timeout. Distinct
     /// from a network drop because the request reached the upstream
     /// — only the response was missing in time.