diff --git a/.env.example b/.env.example index 05f032a..e31923c 100644 --- a/.env.example +++ b/.env.example @@ -1,74 +1,60 @@ -# Copy to .env and set the gateway URL. Docker Compose reads .env automatically. -# After changes, run: docker compose up -d -# Setup: README.md. Full reference: docs/CONFIGURATION.md. -# The native binary reads process environment variables, not this file. +# Copy to .env and set PROXY_UPSTREAM_URL. Docker Compose loads .env automatically. +# Apply changes with: docker compose up -d +# Note: the native binary reads process environment variables directly, not .env. +# Docs: README.md (setup) | docs/CONFIGURATION.md (full reference) -# Required. The proxy appends the client path, such as /v1/messages, to this URL. -# Use http(s)://host[:port][/prefix], usually without a trailing /v1. +# Required. Gateway root URL (http/https, optional path prefix, usually no trailing /v1). +# Incoming paths like /v1/messages are appended directly to this URL. PROXY_UPSTREAM_URL= -# Compose runs as 1000:1000 by default. If your host IDs differ, uncomment these -# and use the results of `id -u` and `id -g` so archive files are owned by you. +# Host user/group IDs for container file ownership (used when writing ./logs). +# Uncomment and set to `id -u` / `id -g` if your host user is not 1000:1000. #UID=1000 #GID=1000 -# Anthropic Messages: buffer for up to 10 minutes, then stream live. -# Keep client timeouts above this window. For Claude Code, use settings.json: +# Anthropic Messages buffering window (10m default). +# Client timeouts must exceed this value. In ~/.claude/settings.json: # CLAUDE_CODE_CONNECT_TIMEOUT_MS=660000 # API_TIMEOUT_MS=720000 # API_FORCE_IDLE_TIMEOUT=0 -# Header waits, retries, backoff, and fallback all share this one window. PROXY_KEEPALIVE_MS=600000 -# Claude Code Workflow agents use a shorter window to avoid their content watchdog. -# Keep this below the script's stallMs. Pings do not count as progress, so a long -# gap in actual upstream content may still need a larger script-side stallMs. -# 0 disables the timed switch to live output for Workflow requests. +# Claude Code Workflow agent window (10s default). Keeps buffering below the +# Workflow runtime's content-stall watchdog (stallMs). 0 disables the timed live switch. PROXY_WORKFLOW_KEEPALIVE_MS=10000 -# Abort an upstream SSE stream or error-body read after 10 minutes without bytes. +# Abort an upstream SSE stream or error read after 10m with no bytes received. PROXY_UPSTREAM_BYTE_IDLE_MS=600000 -# OpenAI Responses: default mode streams as soon as output arrives. This window -# bounds a silent start; the client's stream idle timeout must exceed it. +# OpenAI Responses silent-start window (30s default) when early commit is enabled. PROXY_RESPONSES_KEEPALIVE_MS=30000 -# With early commit disabled, Responses buffers under this independent window. -# The client's stream idle timeout must exceed it. For Codex, use -# stream_idle_timeout_ms=660000 to cover the default. +# OpenAI Responses full-buffering window (10m default) when early commit is disabled. +# Set Codex stream_idle_timeout_ms=660000 to cover this window. PROXY_RESPONSES_BUFFER_MS=600000 -# Set 0 to buffer Responses until completion or the window ends. This protects -# against later stream failures, but delays visible output while buffering. +# Set to 0 to buffer Responses streams until completion (or the buffer window ends). #PROXY_RESPONSES_EARLY_COMMIT=1 -# Native Gemini streamGenerateContent?alt=sse: buffer through clean completion, -# then replay unchanged. Client timeouts must exceed this window. 0 keeps full -# buffering without a timed live switch; upstream idle/request deadlines remain. +# Native Gemini streamGenerateContent?alt=sse buffering window (10m default). +# 0 buffers until completion without a timed live switch. PROXY_GEMINI_BUFFER_MS=600000 -# Extra attempts for recoverable streaming failures before sending a response. -# 0 retains only one limited fast retry for selected connection/HTTP failures. -# Positive values enable broader retries on Messages, Responses, and Gemini SSE. +# Extra in-proxy retries for recoverable streaming failures before commit. +# 0 keeps only a single fast retry (<3s failure, 250ms wait) for connection/HTTP errors. PROXY_TRANSACTIONAL_LOCAL_RETRIES=0 -# Local retries wait for upstream Retry-After plus an exponential extra delay. -# This caps only the extra delay, not the upstream-requested wait. +# Cap on the exponential backoff added on top of upstream Retry-After during local retries. PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS=10000 -# Backstop based on X-Stainless-Retry-Count; it does not raise client retry limits. -# Clients without this header, including Codex, must enforce their own limit. -# 0 disables retry conversion and local retries, but not model fallback. +# Backstop on X-Stainless-Retry-Count (does not raise client retry limits; ignored by +# clients that do not send the header, such as Codex). 0 disables retry conversion. #PROXY_SDK_RETRY_CAP=100 -# Anthropic only: before sending a response, retry a structured refusal or a -# recognized Fable safeguards error with this model. All other fields retain -# their values. The substitution happens at most once per client request. -# Choose a model your gateway supports. off / none / disabled / empty disables it. +# Anthropic only: retry pre-commit structured refusals or recognized Fable safeguard +# errors once with this model. Set to off, none, disabled, or empty to disable. #PROXY_REFUSAL_FALLBACK_MODEL=claude-opus-5 -# Optional full payload archives. Under Compose, ./logs maps to the host's logs. -# Known credential headers and query parameters are redacted, but bodies are -# preserved unencrypted. There is no rotation or retention; disable capture after -# debugging or manage retention yourself. See docs/OPERATIONS.md. +# Optional directory for full JSON request/response archives (mapped to ./logs in Compose). +# Archives are unencrypted (auth headers/params redacted, bodies raw) with no auto-retention. #PROXY_REQUEST_LOG_DIR=./logs diff --git a/README.md b/README.md index ae6556b..920f944 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,34 @@ # steady-proxy -steady-proxy exists so agents can work continuously through common API and gateway -disruptions, with fewer interruptions that need you to restart a turn or type -“continue.” It runs between your **agent or API client** and its gateway, -protecting requests from dropped connections, broken streams, overload, and -temporary errors. - -It provides: - -- **Buffering:** hold responses until they are complete, so a broken reply can be - retried before the agent sees it. -- **Retries and backoff:** turn recoverable failures into automatic client retries, - with optional extra retries inside the proxy. -- **Model fallback:** retry Anthropic refusals with a configured fallback model. -- **Stream validation and repair:** catch malformed Anthropic streams, combine - fragmented tool input, and smooth buffered Responses text replay. -- **Long-turn support:** use configurable buffering windows and keepalives, with - a shorter window for Claude Code Workflow agents. -- **Diagnostics:** log request outcomes and optionally archive full request and - response data for debugging. +`steady-proxy` is a local reverse proxy for AI agents and API clients. It sits between your client and API gateway, buffering streaming responses so dropped connections, truncated streams, rate limits, and gateway 5xx errors can be retried before the client sees a failed turn. ```text -Agents / SDKs / applications → steady-proxy on localhost → your gateway +Agents / SDKs / applications → steady-proxy (127.0.0.1:8789) → your gateway ``` -Stream protection supports **Anthropic Messages** (`/v1/messages`), -**OpenAI Responses** (`/v1/responses`), and native **Gemini GenerateContent** -(`/{version}/models/{model}:streamGenerateContent?alt=sse`). Claude Code and Codex -are supported clients; other agents and SDK-based applications can use these -routes with compatible timeout, retry, and SSE handling. Subagents using the same -API settings are covered without terminal automation. See -[client compatibility](docs/ARCHITECTURE.md#client-compatibility) for requirements -and the behavior of other routes. +### Key features -## What to expect +- **Transactional buffering:** Holds SSE responses until completion (or a configurable time window) so broken streams can be discarded and retried cleanly. +- **Retry normalization:** Converts recoverable HTTP and pre-commit stream failures into standard `503` responses with `Retry-After` (plus optional in-proxy retries). +- **Stream validation and repair:** Validates Anthropic event order and tool JSON, coalesces fragmented `input_json_delta` chunks, and merges bursty Responses text deltas on replay. +- **Refusal fallback:** Optionally re-issues Anthropic refusals against a fallback model before committing the response. +- **Workflow watchdog support:** Uses a short buffering window for Claude Code Workflow agents so their content-stall timer does not trip. +- **Diagnostics:** Logs one-line request outcomes and optionally archives full request/response payloads for debugging. -By default, Messages and Gemini SSE streams are buffered for up to 10 minutes. -A response that finishes within that window appears all at once. Responses -streams go live as soon as output arrives; you can enable full buffering to -protect them from later stream failures. +### Supported routes -Before the proxy sends a response, it can discard a failed attempt and retry. -After it starts streaming to the client, recovery depends on the client's own -stream retry behavior. Invalid requests, exhausted retry budgets, and client -timeouts can still stop a turn. See the [architecture guide](docs/ARCHITECTURE.md) for the -behavior and limits of each mode. +| API | Route | Default behavior | +| --- | --- | --- | +| **Anthropic Messages** | `POST /v1/messages` (`stream: true`) | Buffers up to 10 minutes (`600000` ms) until `message_stop`, then replays. Workflow agents switch to live streaming after 10 seconds once content is ready. | +| **OpenAI Responses** | `POST /v1/responses` (`stream: true`) | Catches start-of-stream failures before commit, then streams live on first output. Set `PROXY_RESPONSES_EARLY_COMMIT=0` for 10-minute full buffering. | +| **Gemini GenerateContent** | `POST /{version}/models/{model}:streamGenerateContent?alt=sse` (including Vertex AI publisher paths) | Buffers up to 10 minutes through clean EOF after all requested candidates finish. | +| **Other routes** | `/v1/chat/completions`, `generateContent`, `countTokens`, non-SSE `streamGenerateContent`, `/v1/models`, non-streaming calls | Forwarded single-shot; HTTP and connection errors still normalize for client retry. | + +See [Architecture](docs/ARCHITECTURE.md#client-compatibility) for client compatibility details. ## Quick start -You need Docker with Compose v2 and a gateway that supports your client's API. -For a native installation, see [building and running from source](docs/DEVELOPMENT.md). +Requires Docker with Compose v2 (or Go 1.23+ to [build from source](docs/DEVELOPMENT.md)). ```bash git clone https://github.com/Xllent-AI/steady-proxy @@ -58,44 +37,38 @@ cp .env.example .env cp docker-compose.yml.example docker-compose.yml ``` -Edit `.env` and set your gateway URL: +Set your gateway URL in `.env`: ```dotenv PROXY_UPSTREAM_URL=https://your-gateway.example.com ``` -Use the gateway root, or its required path prefix. The proxy appends the request -path: `/v1/messages` becomes `https://your-gateway.example.com/v1/messages`. -Do not add another `/v1` unless your gateway requires that extra prefix. +Use the gateway root (or required prefix) without a trailing `/v1`—the proxy appends the incoming request path (`/v1/messages`, `/v1/responses`, etc.) verbatim. -Start the proxy: +Start the proxy and verify it responds: ```bash docker compose up -d --build curl -fsS http://127.0.0.1:8789/__version ``` -The version endpoint confirms that the proxy is running; it does not contact your -gateway. Next, configure the client you use below. Keep your existing API -credentials: the proxy forwards them to the gateway. +`/__version` checks local proxy health only. The proxy forwards your client's existing API keys and auth headers to the gateway. + +### Prebuilt container image -Prebuilt Linux amd64 images are published to GitHub Container Registry: +Linux `amd64` images are published to GitHub Container Registry: ```bash docker pull ghcr.io/xllent-ai/steady-proxy:latest ``` -`latest` follows `main`; release tags such as `v0.1.1` select a release instead. -To use a prebuilt image with the example Compose configuration, change its -`image:` to `ghcr.io/xllent-ai/steady-proxy:latest`, remove the `build:` block, then -run `docker compose pull && docker compose up -d`. See -[container publishing](docs/RELEASING.md#container-images) for tagging and registry -access details. +`latest` tracks `main`; version tags (such as `v0.1.1`) pin a release. To run the prebuilt image with Compose, set `image: ghcr.io/xllent-ai/steady-proxy:latest` in `docker-compose.yml`, remove the `build:` block, and run `docker compose pull && docker compose up -d`. See [Releasing](docs/RELEASING.md#container-images). + +## Client setup ### Claude Code -Merge this `env` block into `~/.claude/settings.json`, keeping your other settings -and authentication values: +Merge this `env` block into `~/.claude/settings.json` alongside your existing settings and credentials: ```json { @@ -108,16 +81,11 @@ and authentication values: } ``` -The proxy sends no response headers while buffering. Both client timeouts must -exceed the default 600,000 ms buffering window, and the idle watchdog must stay -off. If you change that window, adjust the client timers with it; see -[timeouts](docs/CONFIGURATION.md#timeouts). +Because the proxy sends no HTTP headers while buffering, client timeouts must exceed the 600,000 ms buffering window and the idle watchdog must be disabled. If you change `PROXY_KEEPALIVE_MS`, update these client timeouts to match; see [Timeouts](docs/CONFIGURATION.md#timeouts). ### Codex -In `~/.codex/config.toml`, update your **active provider's** section. Replace -`your_provider` below with its existing name and preserve its authentication -settings: +Update your active provider section in `~/.codex/config.toml` (keep `/v1` in `base_url` and keep your auth settings): ```toml [model_providers.your_provider] @@ -128,101 +96,61 @@ request_max_retries = 10 stream_max_retries = 10 ``` -Keep `/v1` in the client URL. The idle timeout above covers both the default -streaming mode and the optional 10-minute buffering mode. - -For full buffering and up to three extra upstream attempts per client request, -add these settings to `.env`, then run `docker compose up -d`: +To enable full 10-minute buffering for Responses and up to 3 in-proxy retries per client attempt, add to `.env` and run `docker compose up -d`: ```dotenv PROXY_RESPONSES_EARLY_COMMIT=0 PROXY_TRANSACTIONAL_LOCAL_RETRIES=3 ``` -Full buffering delays visible output until completion or the buffering window -ends. Extra retries apply to all three supported streaming APIs and must fit within -the active window. See [retry budgets](docs/ARCHITECTURE.md#retry-budgets). +Full buffering delays output until the turn completes (or the window expires). Local retries apply across Messages, Responses, and Gemini SSE within the active buffering window; see [Retry budgets](docs/ARCHITECTURE.md#retry-budgets). ### Gemini -Use a gateway that accepts native Gemini requests, or set -`PROXY_UPSTREAM_URL=https://generativelanguage.googleapis.com` for Google's -Developer API. Point your SDK's HTTP base URL at `http://127.0.0.1:8789`, retaining -its API version and model settings. The proxy forwards `x-goog-api-key`, bearer -authentication, and query parameters unchanged; it does not translate protocols. +Point your Gemini SDK or client base URL at `http://127.0.0.1:8789` and set `PROXY_UPSTREAM_URL` to your Gemini gateway (or `https://generativelanguage.googleapis.com`). The proxy forwards `x-goog-api-key`, Bearer tokens, and query parameters unchanged. -For a direct SSE request, set `GEMINI_MODEL` to an available model and keep your -existing `GEMINI_API_KEY`: +Example streaming call: ```bash curl -N --max-time 720 \ "http://127.0.0.1:8789/v1beta/models/${GEMINI_MODEL}:streamGenerateContent?alt=sse" \ -H "x-goog-api-key: ${GEMINI_API_KEY}" \ - -H 'Content-Type: application/json' \ + -H "Content-Type: application/json" \ -d '{"contents":[{"parts":[{"text":"Say hello."}]}]}' ``` -Gemini selects streaming through the method and `alt=sse`, not a `stream` body -field. `generateContent` and `countTokens` remain single-shot JSON requests with -Gemini-shaped HTTP errors. Standard Vertex AI project/location/publisher model -paths are also recognized; keep the Vertex upstream host and bearer credentials. - -`PROXY_GEMINI_BUFFER_MS` controls the independent 600,000 ms buffering window. -Set client timeouts above it and configure bounded retries for HTTP `503`, honoring -`Retry-After`. After a timed switch to live output, keepalives are SSE comments; -failed streams abort the HTTP transport and require client recovery. +Set client timeouts above `PROXY_GEMINI_BUFFER_MS` (default 600,000 ms) and configure bounded retries on HTTP `503` honoring `Retry-After`. If a Gemini stream fails after committing to live output, the proxy aborts the HTTP transport so the client does not mistake a truncated stream for clean EOF. ### Other API clients -Set your client's base URL so it sends Messages requests to -`http://127.0.0.1:8789/v1/messages` or Responses requests to -`http://127.0.0.1:8789/v1/responses`. Some SDKs append `/v1` themselves; configure -the base URL to produce exactly one `/v1` in the request path. - -Keep authentication in the client and use `stream: true` for stream protection. -Set client timeouts above the active buffering window and configure retries for -HTTP `503`, respecting `Retry-After`. Extra proxy retries can absorb failures -before a response starts, but recovery after streaming begins belongs to the -client. Check the [compatibility requirements](docs/ARCHITECTURE.md#client-compatibility), -especially for Responses clients with strict event parsing. +Point Messages clients at `http://127.0.0.1:8789/v1/messages` and Responses clients at `http://127.0.0.1:8789/v1/responses` (watch whether your SDK appends `/v1` automatically). Enable `stream: true`, set client timeouts above the buffering window, and retry HTTP `503` with `Retry-After`. See [Client compatibility](docs/ARCHITECTURE.md#client-compatibility). -### Check a request +### Verify traffic -Start a new client session and send a short prompt while watching: +Send a test prompt and tail the proxy logs: ```bash docker compose logs -f proxy ``` -`OK` means the request was delivered. `RETRY` means the proxy asked the client to -retry, and `FAIL` means it returned an error without requesting another attempt. -See [reading the logs](docs/OPERATIONS.md#reading-the-logs) for examples and the -other outcomes. - -## Before you use it - -Keep the proxy on **loopback**. It has no authentication of its own; anyone who -can reach its port can send requests through it. Compose publishes the port only -on `127.0.0.1`. +Each request logs a single line starting with `OK` (delivered), `RETRY` (client asked to retry), `WARN` (model fallback), `FAIL` (terminal error), or `DROP` (stream failed after headers were already sent). See [Reading the logs](docs/OPERATIONS.md#reading-the-logs). -Full payload archives are **off by default**. Enabling them saves prompts and -responses unencrypted, without automatic retention. See -[data handling](docs/OPERATIONS.md#data-handling) before enabling them. +## Security and operational notes -Retries repeat upstream work. Server-side or remote tools with side effects need -their own idempotency protection. Buffering does not undo work already performed -by the provider; see [retry limits and side effects](docs/ARCHITECTURE.md#retry-limits-and-side-effects). +- **Bind to loopback only:** The proxy has no authentication of its own; anyone who can reach its port can relay requests through your gateway. Compose binds `127.0.0.1:8789` by default. +- **Payload archives are unencrypted:** `PROXY_REQUEST_LOG_DIR` is off by default. When enabled, it writes full prompts and responses to disk without encryption or automatic retention; see [Data handling](docs/OPERATIONS.md#data-handling). +- **Retries repeat upstream work:** Buffering hides failed attempts from the client, but cannot undo side effects already executed by server-side or remote tools. Ensure remote tools are idempotent; see [Retry limits and side effects](docs/ARCHITECTURE.md#retry-limits-and-side-effects). ## Documentation -| Guide | Use it to… | +| Guide | Contents | | --- | --- | -| [Architecture](docs/ARCHITECTURE.md) | Understand client compatibility, buffering, retries, fallback, and Workflow agents. | -| [Configuration](docs/CONFIGURATION.md) | Tune timeouts, retry budgets, storage limits, and gateway hints. | -| [Operations](docs/OPERATIONS.md) | Read logs, capture payloads, and run the proxy persistently. | -| [Troubleshooting](docs/TROUBLESHOOTING.md) | Match an error or symptom to its cause and next step. | -| [Development](docs/DEVELOPMENT.md) | Build, test, and find the relevant source code. | -| [Releasing](docs/RELEASING.md) | Preview release archives locally and publish through the tag workflow. | +| [Architecture](docs/ARCHITECTURE.md) | Buffering lifecycle, commit points, retry budgets, refusal fallback, and Workflow agents. | +| [Configuration](docs/CONFIGURATION.md) | Environment variables, client timeout matching, and upstream gateway headers. | +| [Operations](docs/OPERATIONS.md) | Compose workflow, log format, payload archiving, and `systemd` service setup. | +| [Troubleshooting](docs/TROUBLESHOOTING.md) | Symptom lookup, error classification rules, and timeout debugging. | +| [Development](docs/DEVELOPMENT.md) | Building from source, running unit and live tests, and code layout. | +| [Releasing](docs/RELEASING.md) | Local release packaging, tag publishing, and GHCR container images. | ## License diff --git a/docker-compose.yml.example b/docker-compose.yml.example index b65c056..65dbb1b 100644 --- a/docker-compose.yml.example +++ b/docker-compose.yml.example @@ -1,7 +1,6 @@ # Copy to docker-compose.yml, set PROXY_UPSTREAM_URL in .env, -# then run `docker compose up -d --build`. -# See README.md for API client setup, including Claude Code and Codex examples. -# Settings and defaults: docs/CONFIGURATION.md. +# and start with: docker compose up -d --build +# Client setup: README.md | Configuration reference: docs/CONFIGURATION.md services: proxy: build: @@ -12,24 +11,23 @@ services: BUILD_DATE: "${PROXY_BUILD_DATE:-}" DIRTY: "${PROXY_DIRTY:-unknown}" image: steady-proxy:latest - # Run as the host user so PROXY_REQUEST_LOG_DIR dumps are owned by you, not - # root (readable without sudo). Bash does not export UID and has no GID, so - # this falls back to 1000:1000 unless you set both in .env (or export them). + # Run as host user so ./logs archives are owned by you instead of root. + # Set UID and GID in .env if your user is not 1000:1000. user: "${UID:-1000}:${GID:-1000}" environment: PROXY_LISTEN_ADDR: "0.0.0.0:8789" PROXY_UPSTREAM_URL: "${PROXY_UPSTREAM_URL:?set PROXY_UPSTREAM_URL in .env to your real gateway}" - PROXY_SDK_RETRY_CAP: "${PROXY_SDK_RETRY_CAP:-100}" # Backstop for the Stainless retry counter; not the client retry budget. + PROXY_SDK_RETRY_CAP: "${PROXY_SDK_RETRY_CAP:-100}" PROXY_TRANSACTIONAL_LOCAL_RETRIES: "${PROXY_TRANSACTIONAL_LOCAL_RETRIES:-0}" PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS: "${PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS:-10000}" PROXY_KEEPALIVE_MS: "${PROXY_KEEPALIVE_MS:-600000}" - PROXY_WORKFLOW_KEEPALIVE_MS: "${PROXY_WORKFLOW_KEEPALIVE_MS:-10000}" # Shorter window for Workflow content progress; see docs/ARCHITECTURE.md. - PROXY_RESPONSES_KEEPALIVE_MS: "${PROXY_RESPONSES_KEEPALIVE_MS:-30000}" # Silent-start window when early commit is enabled. - PROXY_RESPONSES_BUFFER_MS: "${PROXY_RESPONSES_BUFFER_MS:-600000}" # Full-buffer window when early commit is disabled. - PROXY_GEMINI_BUFFER_MS: "${PROXY_GEMINI_BUFFER_MS:-600000}" # Native Gemini SSE buffering window; 0 disables the timed live switch. - PROXY_RESPONSES_EARLY_COMMIT: "${PROXY_RESPONSES_EARLY_COMMIT:-1}" # 0 buffers Responses; client idle timeout must exceed the buffer window. - PROXY_RESPONSES_REPLAY_DELTA_BYTES: "${PROXY_RESPONSES_REPLAY_DELTA_BYTES:-65536}" # Combine compatible text deltas during replay; 0 keeps raw replay. - PROXY_RESP_HEADER_TIMEOUT_MS: "${PROXY_RESP_HEADER_TIMEOUT_MS:-660000}" # Overrides the binary default of 60000 ms for slow gateway headers. + PROXY_WORKFLOW_KEEPALIVE_MS: "${PROXY_WORKFLOW_KEEPALIVE_MS:-10000}" + PROXY_RESPONSES_KEEPALIVE_MS: "${PROXY_RESPONSES_KEEPALIVE_MS:-30000}" + PROXY_RESPONSES_BUFFER_MS: "${PROXY_RESPONSES_BUFFER_MS:-600000}" + PROXY_GEMINI_BUFFER_MS: "${PROXY_GEMINI_BUFFER_MS:-600000}" + PROXY_RESPONSES_EARLY_COMMIT: "${PROXY_RESPONSES_EARLY_COMMIT:-1}" + PROXY_RESPONSES_REPLAY_DELTA_BYTES: "${PROXY_RESPONSES_REPLAY_DELTA_BYTES:-65536}" + PROXY_RESP_HEADER_TIMEOUT_MS: "${PROXY_RESP_HEADER_TIMEOUT_MS:-660000}" PROXY_UPSTREAM_BYTE_IDLE_MS: "${PROXY_UPSTREAM_BYTE_IDLE_MS:-600000}" PROXY_VALIDATE_JSON: "${PROXY_VALIDATE_JSON:-1}" PROXY_NORMALIZE_TOOL_JSON: "${PROXY_NORMALIZE_TOOL_JSON:-1}" diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 7fe2cec..a245989 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -1,266 +1,109 @@ # Architecture -steady-proxy handles the connection between an API client and its gateway. The gateway -still owns provider routing and availability. The proxy holds responses long -enough to catch failures, then either delivers the response or makes another -attempt possible within the configured budgets. +`steady-proxy` sits between an API client and an upstream gateway. Your gateway still handles model routing and provider selection; the proxy buffers streaming responses so transient upstream failures can be caught and retried before the client commits to a turn. ## Client compatibility -The proxy selects stream behavior from the HTTP method, API path, and request. -Messages and Responses use the body's `stream` field; Gemini uses the -`streamGenerateContent` method with `alt=sse`. These streaming `POST` requests -share one buffering and retry engine, with a separate interpreter per protocol. -No specific user agent is required. +The proxy selects a protocol interpreter based on the HTTP method, path, and request parameters: -Other agents, SDKs, and applications can use these routes when they support the -proxy's timeout, retry, and stream behavior. Integration tests exercise all three -protocols using ordinary HTTP requests. The live CLI harness uses Claude Code; -it does not establish compatibility with every client. +- **Anthropic Messages:** `POST /v1/messages` with `"stream": true` +- **OpenAI Responses:** `POST /v1/responses` with `"stream": true` +- **Gemini GenerateContent:** `POST /{version}/models/{model}:streamGenerateContent?alt=sse` under `/v1`, `/v1beta`, `/v1alpha`, or `/v1beta1` (including Vertex AI `/projects/{project}/locations/{location}/publishers/google/models/{model}` paths) -| Client behavior | Requirement or effect | +All three streaming routes share one buffering and retry engine (`wireModel` in `wire.go`) and do not require a specific `User-Agent`. Other routes (`/v1/chat/completions`, `generateContent`, `countTokens`, non-SSE `streamGenerateContent`, `/v1/models`, and `stream: false` requests) are forwarded as single-shot requests with HTTP error normalization. Synthesized HTTP errors use the route's native error envelope: Google's `error.code` / `error.status` on known Gemini model methods, OpenAI's error object on `/v1/responses`, and Anthropic's `type: "error"` envelope on `/v1/messages` and all other routes. + +| Requirement | Client expectation | | --- | --- | -| Base URL and transport | Send HTTP requests to the proxy using the expected API paths and SSE for streaming. The proxy does not translate between APIs or implement WebSocket streaming. | -| Timeouts | Allow the initial buffered wait and the chosen live keepalive interval; see [timeouts](CONFIGURATION.md#timeouts). | -| HTTP retries | Retry HTTP `503` with `Retry-After`. Honor `X-Should-Retry: false` where supported. Local proxy retries can recover some failures even if the client does not retry. | -| Stream recovery | Handle an incomplete response after headers have been sent. The proxy cannot restart a committed response for the client. | -| Responses events | Tolerate the additional `response.proxy_keepalive` event. Completion checks and content handling follow Codex's parser expectations. | -| Gemini events | Use native `GenerateContentResponse` SSE frames and tolerate SSE comment keepalives. The proxy preserves tool arguments, thought signatures, and unknown fields. | -| Retry counter | Without `X-Stainless-Retry-Count`, the proxy treats the client retry count as zero; a positive `PROXY_SDK_RETRY_CAP` cannot bound those client retries. | - -Some compatibility choices apply to every request on a protocol's routes. -Responses errors classified as non-retryable become HTTP `400`, and a -`response.failed` event with `context_length_exceeded` is forwarded for the client -to handle. Codex uses that event for compaction; other clients need their own -handling. The shorter Workflow window specifically depends on Claude Code's -system-prompt marker. - -Other routes, including `/v1/chat/completions`, are forwarded with one upstream -attempt per client call. They do not receive stream validation, transactional -buffering, or model fallback. HTTP errors can still be normalized for retry. -Known Gemini model methods use Google's error envelope; Responses uses OpenAI's, -and remaining routes use the Anthropic-style envelope. +| **Transport & paths** | Standard HTTP + SSE. The proxy does not translate between protocols or support WebSocket streaming. | +| **Timeouts** | Header, idle, and per-attempt timeouts must exceed the active buffering window (and live keepalive interval); see [Timeouts](CONFIGURATION.md#timeouts). | +| **HTTP retries** | Retry HTTP `503` honoring `Retry-After`, and stop on `X-Should-Retry: false` (or terminal `400`). In-proxy retries (`PROXY_TRANSACTIONAL_LOCAL_RETRIES`) can also absorb pre-commit failures transparently. | +| **Post-commit recovery** | Handle interrupted streams after `200 OK` headers are sent. Once committed, the proxy cannot restart the stream on the client's behalf. | +| **Responses events** | Ignore unknown SSE event types (`response.proxy_keepalive`), which the proxy emits because SSE comments do not reset Codex's idle timer. | +| **Gemini events** | Accept standard SSE comment keepalives (`: keepalive`). Tool calls, thought signatures, and unknown JSON fields are preserved unchanged. | +| **Retry counter** | `PROXY_SDK_RETRY_CAP` reads `X-Stainless-Retry-Count`. Clients that do not send that header (such as Codex and Gemini SDKs) must enforce their own retry limit. | ## Buffering and the point of no return -A supported streaming request starts in **buffered mode**. The proxy reads the upstream -server-sent events (SSE) without sending response headers or content to the client. -If the response completes, it sends `200 OK` and replays the buffered events. -If the attempt fails, it can discard the buffer and retry without exposing a -partial reply. +Supported streaming requests start in **buffered mode**: the proxy reads upstream SSE frames into memory (spilling to disk beyond 1 MiB) without writing HTTP headers to the client. -The moment it sends response headers is the **commit point**. From then on, it -cannot replace the response with an HTTP error. If a stream fails after that -point, the proxy ends it and logs `DROP`; recovery depends on the client's stream -retry support and remaining budget. +- If the stream completes within the buffering window, the proxy sends `200 OK` and replays the validated buffer. +- If the stream fails before headers are sent, the proxy discards the buffer and either retries upstream locally or returns a retryable `503` to the client. -The protocols choose that point differently: +Writing `200 OK` response headers is the **commit point**. Once headers are sent, the proxy can no longer convert a failure into an HTTP error; any subsequent upstream failure terminates the stream and logs `DROP`. -| Request | Default behavior | Successful completion | +| Request type | Default commit policy | Completion criterion | | --- | --- | --- | -| Anthropic Messages | Buffer for up to 600 seconds, then switch to live output. | A valid stream ending in `message_stop`. | -| Claude Code Workflow agent | Use a 10-second window, then switch once real content is available. | The same Messages completion. | -| OpenAI Responses | Switch to live output as soon as an output event arrives; a 30-second window bounds a silent start. | A valid `response.completed`. | -| Responses with `PROXY_RESPONSES_EARLY_COMMIT=0` | Buffer for up to 600 seconds, then switch to live output. | A valid `response.completed`. | -| Gemini `streamGenerateContent?alt=sse` | Buffer for up to 600 seconds, then switch to live output. | Clean transport EOF after every requested candidate has a `finishReason`, or a native prompt block. | - -Buffering trades immediate output for a chance to recover before the client sees -anything. Choose full buffering when that protection matters more than seeing -tokens arrive live. - -Gemini supports native model methods under `/v1`, `/v1beta`, `/v1alpha`, and -`/v1beta1`, including standard Vertex AI -`/projects/{project}/locations/{location}/publishers/google/models/{model}` paths. -Only `streamGenerateContent?alt=sse` receives transactional protection; -`generateContent`, `countTokens`, and non-SSE streaming transports remain -single-shot. Gemini has no separate terminal SSE event, so the proxy waits for -clean EOF, retaining usage-only frames after candidate completion. Missing -candidates, unfinished SSE frames, and read faults fail the attempt. SSE control -frames and empty data messages do not advance candidate completion. Native -`MAX_TOKENS`, safety stops, and prompt blocks without candidates are delivered -unchanged, not retried as transport failures. A prompt block accompanied by -candidates is contradictory and cannot satisfy completion. See Google's -[GenerateContent contract](https://ai.google.dev/api/generate-content). -Candidate counts follow [ProtoJSON](https://protobuf.dev/programming-guides/json/): -camelCase and snake_case field names, quoted integers, exponent notation, and zero -fractional parts are accepted without rewriting the request. - -After a Gemini response commits, a failure aborts the downstream HTTP transport: -a normal EOF could otherwise appear successful to an SDK. No committed attempt -is replayed. Whether the client recovers depends on its stream-retry behavior. +| **Anthropic Messages** | Buffer up to 600s (`PROXY_KEEPALIVE_MS`), then switch to live streaming. | Valid SSE stream ending in `message_stop`. | +| **Claude Code Workflow agent** | Buffer up to 10s (`PROXY_WORKFLOW_KEEPALIVE_MS`), then switch to live streaming once forwardable content arrives. | Valid SSE stream ending in `message_stop`. | +| **OpenAI Responses (default)** | Commit and stream live as soon as the first output event arrives; 30s (`PROXY_RESPONSES_KEEPALIVE_MS`) bounds a silent reasoning start. | Valid `response.completed` event. | +| **OpenAI Responses (`PROXY_RESPONSES_EARLY_COMMIT=0`)** | Buffer up to 600s (`PROXY_RESPONSES_BUFFER_MS`), then switch to live streaming. | Valid `response.completed` event. | +| **Gemini `streamGenerateContent?alt=sse`** | Buffer up to 600s (`PROXY_GEMINI_BUFFER_MS`), then switch to live streaming. | Clean transport EOF after every requested candidate has a `finishReason` (or a prompt block without candidates). | -### One window per client request +### Gemini completion details + +Because Gemini SSE has no dedicated terminal event, the proxy waits for clean EOF after all requested `candidateCount` candidates report a `finishReason`, preserving trailing usage-metadata frames (SSE control frames and empty data messages do not advance candidate completion). `generationConfig` / `generation_config` settings accept ProtoJSON camelCase or `snake_case` in JSON field order (later objects replace earlier ones), and candidate counts accept quoted integers, exponents, and `.0` fractions without mutating the request body. -The buffering window starts when the client request reaches the proxy. Waiting -for upstream headers, retry delays, extra attempts, and model fallback all use -that same window. None of them restart its clock. +Missing candidates, truncated frames, contradictory prompt blocks (a block accompanied by candidates), and read errors fail the attempt. Native `MAX_TOKENS`, safety stops, and standalone prompt blocks are valid completions and are delivered as-is. If a Gemini stream fails after committing to live output, the proxy aborts the downstream HTTP connection so the SDK does not treat a premature close as clean EOF. -When the window ends, a pending header or error-body read is cancelled and becomes -eligible for a client retry. An active SSE capture switches to live output; -Workflow requests also wait for real forwardable content. In live mode, keepalives -help keep the client connection open. Messages and Gemini clients receive SSE -comments; Responses clients receive a `response.proxy_keepalive` event. This -event is skippable in Codex, whose SSE reader discards comments without resetting -its idle timer; other Responses clients must also tolerate it. +### One window per client request -Keepalives cannot extend an absolute request deadline or make a silent upstream -produce content. The separate upstream idle timeout still applies. See -[timeouts](CONFIGURATION.md#timeouts) for the settings that must move together. +The buffering timer starts when the client request arrives. Upstream header waits, local retry backoffs, and model fallback attempts all share that single window. -## Retries +When the window expires: +- A pending header or error-body read is cancelled and returned for client retry. +- An active SSE stream flushes its buffered prefix and switches to live streaming (Workflow requests wait until forwardable content is present). +- In live mode, silent gaps emit `: keepalive` comments on Messages/Gemini and `response.proxy_keepalive` events on Responses. Keepalives prevent client idle disconnects, but `PROXY_UPSTREAM_BYTE_IDLE_MS` and the total request deadline still apply. -Before commit, a recoverable failure has two possible paths: +## Retries -1. If extra local retries are enabled and there is enough time, the proxy waits - and sends the request again. The client continues waiting on its original call. -2. Otherwise, the proxy returns HTTP `503` with `X-Should-Retry: true` and - `Retry-After`. Messages and Responses use `api_error`; Gemini uses its native - numeric `error.code: 503` and `error.status: "UNAVAILABLE"` envelope. +When an attempt fails before commit: -Using one generic error shape matters for Claude Code. Recognizable overload and -rate-limit errors can enter special client handlers with smaller retry budgets. -The proxy keeps those failures in the ordinary retry path and records the original -cause in its logs. +1. **Local retry:** If `PROXY_TRANSACTIONAL_LOCAL_RETRIES` has remaining budget and enough time remains in the buffering window and request deadline, the proxy waits and re-issues the upstream request transparently. +2. **Client retry:** Otherwise, the proxy returns HTTP `503` with `X-Should-Retry: true` and `Retry-After` (`api_error` envelope for Anthropic/Responses; `error.code: 503` / `UNAVAILABLE` for Gemini). -Connection failures, truncation, overload, rate limits, and most temporary service -errors are retryable. Authentication, billing, and unknown errors also usually -retry, since a gateway may recover from them. A truly invalid key or empty balance -therefore takes longer to surface. +Masking all retryable failures—including `429` rate limits and `529` overloads—into a generic `503` prevents Claude Code from routing them into special-case handlers that ignore `X-Should-Retry` and abort after ~3 tries. The real failure code is preserved in the proxy logs. -Recognized invalid requests are returned without a retry request. Explicit -[gateway retry headers](CONFIGURATION.md#optional-upstream-headers) can override -HTTP error classification. The Responses path also has exceptions designed for -Codex's native context compaction and terminal policy errors; see the -[error guide](TROUBLESHOOTING.md#invalid-requests). +By default, connection errors, truncated streams, 5xx errors, rate limits, overloads, auth/billing failures, and unknown 4xx responses are retried so transient gateway issues do not kill long turns. Deterministic request-shape errors (context length, schema/tool validation, unknown model) fail immediately without retry; see [Troubleshooting](TROUBLESHOOTING.md#invalid-requests). ### Retry budgets -`PROXY_TRANSACTIONAL_LOCAL_RETRIES=N` allows up to `N` extra attempts for a -recoverable streaming failure before commit. With `N > 0`, a request that keeps -using the same model can make at most: +With `PROXY_TRANSACTIONAL_LOCAL_RETRIES=N` (`N > 0`), a turn using the same model can make up to: ```text (client retries + 1) × (N + 1) upstream attempts ``` -The actual number may be lower because attempts and backoff must fit within the -buffering window, request deadline, and proxy retry cap. Local retries wait for -upstream `Retry-After` **plus** an exponential delay starting at one second. The -extra delay is capped at 10 seconds by default; that cap does not shorten an -upstream-requested wait. - -The default `N=0` disables the general local retry budget. It still permits one -short retry for selected fast connection or HTTP failures: the failure must arrive -within three seconds, and the proxy waits 250 ms before trying again. The formula -above does not describe that default path. - -`PROXY_SDK_RETRY_CAP` is a separate backstop based on the Anthropic SDK's -`X-Stainless-Retry-Count` header. It does not increase the client's own budget. -Claude Code 2.1.191 was observed to default to 10 retries and clamp -`CLAUDE_CODE_MAX_RETRIES` to 15; client versions can change these limits. Codex does -not send the Stainless counter, so any positive proxy cap leaves its retry limit -to `request_max_retries` and `stream_max_retries`. Other clients without that -header likewise need to enforce their own retry limit. - -Gemini clients likewise own their retry budgets when they do not send the -Stainless counter. If conversion is disabled or exhausted, a transient Gemini -failure becomes terminal HTTP `400`, since its SDK may ignore `X-Should-Retry`. -Explicitly non-retryable Gemini `408`, `429`, and `5xx` errors also become `400` -so a gateway's `false` retry verdict cannot trigger an SDK's status-based retry. - -The same counter determines the `Retry-After` sent to the client when the upstream -provides no delay. It grows exponentially from one second to 30 seconds, plus -0–1 seconds of jitter. Without the counter, the synthesized delay remains -1–2 seconds on each client request. Local retries use the proxy's own attempt -counter for their backoff. - -There is no circuit breaker. Retries are controlled by backoff, time limits, and -attempt budgets. +bounded by the buffering window, total request deadline, and `PROXY_SDK_RETRY_CAP`. Local retries sleep for the upstream `Retry-After` **plus** an exponential backoff (1s, 2s, 4s, …, capped by `PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS`, default 10s). -## Model fallback - -For streaming Anthropic requests, a complete response with -`stop_reason: "refusal"` can trigger a new attempt using -`PROXY_REFUSAL_FALLBACK_MODEL` instead. The default model is `claude-opus-5`. -Recognized Fable safeguards errors can trigger the same substitution before -commit, even without a complete response stream. +When `N = 0` (the default), general local retries are off, except for a single fast retry (250 ms wait) when a connection error or selected HTTP status (`408`, `409`, `502`, `503` without `Retry-After`) arrives within 3 seconds of the attempt start. -Only the request's top-level `model` value changes; the other fields retain their -values. The proxy logs `WARN` with the original and fallback model. Choose a model -your gateway supports, or set the variable to `off`, `none`, `disabled`, or an -empty value to disable fallback. +`PROXY_SDK_RETRY_CAP` (default `100`) bounds retry conversion using the `X-Stainless-Retry-Count` header (it does not raise client limits; for example, Claude Code 2.1.191 defaults to 10 retries and clamps `CLAUDE_CODE_MAX_RETRIES` to 15). When the upstream provides no `Retry-After`, the synthesized client delay scales exponentially from 1s to 30s (+ up to 1s jitter) based on that header (or stays 1–2s when the header is absent). If retry conversion is disabled (`PROXY_SDK_RETRY_CAP=0`) or exhausted, Responses and transient Gemini failures return terminal HTTP `400` because their SDKs may retry `5xx`/`429` regardless of `X-Should-Retry: false`. -The substitution happens at most once per client request. A refusal from the -fallback model is delivered normally, as is a refusal that arrives after commit. -The fallback receives a fresh local retry budget, but it shares the original -request's time limits. Setting `PROXY_SDK_RETRY_CAP=0` does not disable this -separate model substitution. +## Model fallback -Recognized safeguards errors can also trigger fallback when the upstream sends -`x-gateway-retryable: false` or `x-should-retry: false`. Those headers govern retry -classification; disable fallback separately with `PROXY_REFUSAL_FALLBACK_MODEL=off` -if the original safeguards error should reach the client. +On streaming `POST /v1/messages` requests, if a buffered response finishes with `stop_reason: "refusal"` or fails pre-commit with a recognized Fable safeguard error, the proxy rewrites the top-level `"model"` field to `PROXY_REFUSAL_FALLBACK_MODEL` (default `claude-opus-5`) and re-issues the request once. -Fallback does not apply to OpenAI Responses, Gemini, or non-streaming requests. It detects -the structured refusal and recognized safeguards messages; it does not inspect -ordinary assistant text for a refusal. +- All other JSON fields (`max_tokens`, `thinking`, `tools`, etc.) preserve their JSON values. +- The proxy logs `WARN` with the original and fallback models. +- Model substitution happens at most once per client request, gets a fresh local retry counter within the remaining request window, and remains enabled even when `PROXY_SDK_RETRY_CAP=0` or when a recognized safeguard error carries `x-gateway-retryable: false` / `x-should-retry: false`. +- Set `PROXY_REFUSAL_FALLBACK_MODEL` to `off`, `none`, `disabled`, or `""` to disable it. Refusal fallback does not apply to Responses, Gemini, or non-streaming calls. ## Workflow agents -Claude Code's Workflow `agent()` calls have a per-agent stall watchdog. In the -observed runtime, its default budget is 180 seconds without forwarded content. -A 600-second buffering window could therefore hide an otherwise healthy response -until the watchdog aborts it. - -The proxy recognizes the Workflow runtime marker, `workflow orchestration script`, -in the request's `system` field and uses `PROXY_WORKFLOW_KEEPALIVE_MS` instead. -The default is 10 seconds. Once that window has elapsed and real content is -available, it forwards the buffered prefix and streams live. Interactive sessions -and ordinary subagents retain their usual window. Workflow requests appear as -`/wf` in the log. - -Keep this window below the agent's `stallMs`. Keepalive comments and ping events -do not count as Workflow progress, so the proxy cannot fix a long gap in actual -upstream content. A slow first output or a mid-turn pause that exceeds `stallMs` -needs a larger script-side budget, such as `agent(prompt, {stallMs: 300000})`. -`CLAUDE_ASYNC_AGENT_STALL_TIMEOUT_MS` controls a different path; there is no global -override for this watchdog in the observed runtime. +Claude Code's Workflow `agent()` runtime enforces a per-agent content-stall watchdog (180s by default; `CLAUDE_ASYNC_AGENT_STALL_TIMEOUT_MS` controls a different path) that only resets when actual assistant content is delivered—SSE pings and comments do not count. -## Stream validation and replay +When the proxy detects the Workflow system-prompt marker (`workflow orchestration script` inside `system`), it labels the request `/wf` in the logs and uses `PROXY_WORKFLOW_KEEPALIVE_MS` (default 10s) instead of the 10-minute window. Once 10s have elapsed and forwardable content is buffered, the proxy commits and streams live. If the upstream model itself goes silent for longer than the script's `stallMs`, raise `stallMs` in the workflow script (e.g., `agent(prompt, {stallMs: 300000})`). -For Anthropic, the proxy checks stream order, event JSON, and accumulated tool or -server-tool input JSON before delivering a buffered success. It combines tool -`input_json_delta` fragments into one complete JSON delta so the client does not -try to parse incomplete fragments. Buffered replay can also fill initial input -and cache usage from final usage when a gateway supplied zero placeholders. +## Stream validation and replay -For Responses, validation focuses on completion and error events. Other content -frames are handled to match Codex's lenient parser. During buffered or prefix replay, -adjacent compatible text deltas can be combined to avoid flooding the client's -notification queue. Live text and non-text frames are preserved. Frames with -non-empty logprobs, ambiguous structure, or incompatible routing stay separate. -Responses tool arguments are not combined by the proxy. +- **Anthropic Messages:** Validates event sequence, event JSON, and accumulated `tool_use` / `server_tool_use` input JSON before replaying a buffered stream. Fragmented `input_json_delta` chunks are coalesced into a single valid JSON delta per tool block, and zero-placeholder `message_start` token counts are backfilled from `message_delta` usage when present. +- **OpenAI Responses:** Requires a valid `response.completed` event (with non-empty string `id` and complete integer usage fields when present). During buffered/prefix replay, adjacent compatible `response.output_text.delta` frames are merged up to `PROXY_RESPONSES_REPLAY_DELTA_BYTES` (64 KiB) to avoid flooding the client queue; frames with logprobs or mismatched metadata stay untouched. +- **Gemini SSE:** Validates JSON framing, candidate `finishReason` completion, and absence of in-band error objects, then replays original bytes unchanged. -The [stream settings](CONFIGURATION.md#stream-processing) control these behaviors. -Response buffers spill from memory to temporary files and are released when the -attempt finishes. Size limits also bound unfinished SSE events before parsing. +See [Stream processing](CONFIGURATION.md#stream-processing) to tune or disable these checks. ## Retry limits and side effects -The full buffering and local retry engine applies to streaming `POST` requests -on the Messages and Responses routes. Other calls, including `count_tokens`, model -listing, and `stream: false`, use one upstream attempt per client request. Their -HTTP and connection failures can still be converted into client retries. If copying -a successful body fails, the proxy aborts the response and logs `DROP`. - -Retrying an upstream request may repeat work already performed by server-side or -remote tools. Buffering prevents delivery of a failed attempt to the client; it -does not provide provider-level idempotency. Use idempotency protection for those -tools or keep such requests outside the proxy. To disable proxy retry conversion -and local retries, set `PROXY_SDK_RETRY_CAP=0`; disable model substitution -separately with `PROXY_REFUSAL_FALLBACK_MODEL=off`. Clients may still apply their -own recovery after a committed stream fails. - -The proxy does not resume a partially delivered response. It cannot combine full -response validation with immediate streaming for the same attempt without support -from the gateway. +Transactional buffering and local retries apply only to streaming `POST` requests on Messages, Responses, and Gemini SSE. Single-shot routes use one upstream attempt per client call; if a response body copy fails mid-transfer, the proxy aborts the connection and logs `DROP`. + +Because retries re-send the upstream request, any server-side or remote MCP tool invoked by the provider may run more than once. Client-local tools are safe because the client only executes them after receiving a complete response. For remote tools with side effects, enforce idempotency at the tool layer or set `PROXY_SDK_RETRY_CAP=0` and `PROXY_REFUSAL_FALLBACK_MODEL=off`. diff --git a/docs/CONFIGURATION.md b/docs/CONFIGURATION.md index a9e5a85..7763c5b 100644 --- a/docs/CONFIGURATION.md +++ b/docs/CONFIGURATION.md @@ -1,183 +1,99 @@ # Configuration -Start with the [quick start](../README.md#quick-start). Most installations only -need a gateway URL and the matching client timeouts. This reference covers the -remaining settings when you need to change the defaults. +Most setups only need `PROXY_UPSTREAM_URL` in `.env` and matching client timeouts (see the [README quick start](../README.md#quick-start)). This page documents all environment variables and upstream gateway headers. -## Applying settings +## How settings are loaded -Docker Compose reads `.env` and passes the variables declared in -[`docker-compose.yml.example`](../docker-compose.yml.example) to the container. After editing -`.env`, run `docker compose up -d` to apply the changes. Exported shell values take -precedence over `.env`. +- **Docker Compose** reads `.env` and passes the variables declared in `docker-compose.yml` into the container. Run `docker compose up -d` after editing `.env`. Shell environment variables take precedence over `.env`. +- **Native binary** reads its process environment directly and does not read `.env`. -The binary reads its process environment; it does not load `.env` itself. Set -variables when launching it or in your service configuration. See -[running from source](DEVELOPMENT.md#build-and-run). +### Binary vs. Compose defaults -Tables below show **binary defaults**. Compose changes three of them: +The tables below list **binary defaults**. `docker-compose.yml.example` overrides three values for container use: -| Variable | Binary | Compose | -| --- | --- | --- | -| `PROXY_LISTEN_ADDR` | `127.0.0.1:8789` | Fixed to `0.0.0.0:8789` inside the container, with the host port bound to `127.0.0.1`. | -| `PROXY_RESP_HEADER_TIMEOUT_MS` | `60000` | `660000`, to allow gateways that wait for model output before sending headers. | -| `PROXY_VERBOSE` | Off | `1`. | +| Variable | Binary default | Compose default | Reason | +| --- | --- | --- | --- | +| `PROXY_LISTEN_ADDR` | `127.0.0.1:8789` | `0.0.0.0:8789` | Listens inside the container network while publishing `127.0.0.1:8789` on the host. | +| `PROXY_RESP_HEADER_TIMEOUT_MS` | `60000` | `660000` | Accommodates gateways that withhold HTTP headers until generation starts. | +| `PROXY_VERBOSE` | `0` | `1` | Enables local-retry and archive diagnostic logs by default in Docker. | -Compose does not currently forward the storage-limit variables, -`PROXY_SPOOL_DIR`, `PROXY_MAX_REQUEST_DURATION_MS`, or `PROXY_DEADLINE_MARGIN_MS`. -To set one of those in Docker, add it to the service's `environment` mapping; -putting it in `.env` alone has no effect. Mount any custom storage directory too. +Storage limits (`PROXY_MAX_*_BYTES`), `PROXY_SPOOL_DIR`, `PROXY_MAX_REQUEST_DURATION_MS`, and `PROXY_DEADLINE_MARGIN_MS` are not listed in `docker-compose.yml.example`—add them under `environment:` in `docker-compose.yml` if you need to override them in Docker. -All `_MS` settings use integer milliseconds, and `_BYTES` settings use integer -bytes. Invalid numeric values or values below the permitted minimum stop startup -with an error naming the offending variables. Empty numeric settings use their -defaults. Use `0` and `1` for feature switches; refusal fallback has its own -disable values below. +All `_MS` variables take integer milliseconds; `_BYTES` variables take integer bytes; boolean switches use `1` and `0`. Startup fails immediately if `PROXY_UPSTREAM_URL` is missing/invalid or if any numeric variable is malformed or below its minimum. ## Connection -| Variable | Default | Purpose | +| Variable | Default | Description | | --- | --- | --- | -| `PROXY_UPSTREAM_URL` | Required | Gateway URL: `http(s)://host[:port][/prefix]`. Query strings and fragments are rejected. | -| `PROXY_LISTEN_ADDR` | `127.0.0.1:8789` | Address to listen on. Keep access restricted to loopback. | - -The proxy appends the incoming request path to the upstream URL. For example, -`https://gateway.example.com/api` plus `/v1/responses` becomes -`https://gateway.example.com/api/v1/responses`. Authentication stays in the client; -its authorization headers are forwarded to this gateway. +| `PROXY_UPSTREAM_URL` | *Required* | Upstream gateway root (`http(s)://host[:port][/prefix]`). Query strings and fragments are rejected. Incoming request paths are appended verbatim. | +| `PROXY_LISTEN_ADDR` | `127.0.0.1:8789` | Bind address. Keep restricted to loopback (`127.0.0.1`). | ## Timeouts -A **buffering window** is how long the proxy may withhold response headers and -content before switching to live output. It starts at client request arrival and -includes upstream waits, retry delays, and fallback attempts. See -[the buffering lifecycle](ARCHITECTURE.md#buffering-and-the-point-of-no-return). +A **buffering window** is how long the proxy holds a streaming request before switching to live output. It starts when the client request arrives and covers header waits, local retries, backoff, and model fallback. -| Variable | Default | Purpose | +| Variable | Default | Description | | --- | --- | --- | -| `PROXY_KEEPALIVE_MS` | `600000` (10 min) | Messages buffering window. `0` disables the timed switch to live output. | -| `PROXY_WORKFLOW_KEEPALIVE_MS` | `10000` (10 s) | Workflow agent window; live output also requires real content. `0` keeps these requests buffered without a timed switch. | -| `PROXY_RESPONSES_KEEPALIVE_MS` | `30000` (30 s) | Responses silent-start window when early commit is on. `0` disables this timer; output can still trigger early commit. | -| `PROXY_RESPONSES_BUFFER_MS` | `600000` (10 min) | Responses window when early commit is off. `0` disables the timed switch to live output. | -| `PROXY_GEMINI_BUFFER_MS` | `600000` (10 min) | Native Gemini SSE buffering window. `0` disables the timed switch to live output; clean completion, idle limits, and request deadlines still apply. | -| `PROXY_RESP_HEADER_TIMEOUT_MS` | `60000` (1 min) | Wait for upstream response headers. `0` disables this timeout, but other deadlines still apply. | -| `PROXY_UPSTREAM_BYTE_IDLE_MS` | `600000` (10 min) | Maximum gap without upstream bytes during SSE capture or error-body reads. Must be at least `1`. | -| `PROXY_MAX_REQUEST_DURATION_MS` | `1500000` (25 min) | Total request deadline, including local retries. Must be at least `1`. | -| `PROXY_DEADLINE_MARGIN_MS` | `25000` (25 s) | Time reserved before a client's `X-Stainless-Timeout` deadline. `0` removes the margin. | - -On supported streaming requests, `X-Stainless-Timeout`, when present, is read in -**seconds**. The effective duration is the smaller of the configured request -ceiling and the client timeout minus the margin. For short client timeouts, the -margin is limited to half their budget. Other routes use the configured ceiling. +| `PROXY_KEEPALIVE_MS` | `600000` (10m) | Anthropic Messages buffering window. `0` disables the timed switch to live output. | +| `PROXY_WORKFLOW_KEEPALIVE_MS` | `10000` (10s) | Claude Code Workflow agent buffering window (commits once forwardable content is present). `0` disables the timed switch. | +| `PROXY_RESPONSES_KEEPALIVE_MS` | `30000` (30s) | OpenAI Responses silent-start window when `PROXY_RESPONSES_EARLY_COMMIT=1`. `0` disables the timer. | +| `PROXY_RESPONSES_BUFFER_MS` | `600000` (10m) | OpenAI Responses full-buffering window when `PROXY_RESPONSES_EARLY_COMMIT=0`. `0` disables the timed switch. | +| `PROXY_GEMINI_BUFFER_MS` | `600000` (10m) | Gemini `streamGenerateContent?alt=sse` buffering window. `0` disables the timed switch. | +| `PROXY_RESP_HEADER_TIMEOUT_MS` | `60000` (1m) | Max wait for upstream HTTP response headers (`660000` in Compose). `0` disables this timer. | +| `PROXY_UPSTREAM_BYTE_IDLE_MS` | `600000` (10m) | Max gap without upstream bytes during SSE streaming or error reads (minimum `1`). | +| `PROXY_MAX_REQUEST_DURATION_MS` | `1500000` (25m) | Hard ceiling for an entire client request, including local retries (minimum `1`). | +| `PROXY_DEADLINE_MARGIN_MS` | `25000` (25s) | Margin subtracted from the client's `X-Stainless-Timeout` (in seconds) so the proxy can return a clean retry before the client aborts. Capped at half the client timeout for short requests; `0` disables the margin. | ### Match the client to the window -For any client, keep response-header, idle, and total request timeouts long enough -to cover the active buffering window. Also allow for keepalive intervals after -the response goes live. The setting names depend on the client or SDK. - -For Claude Code with the default 600,000 ms Messages window, set: - -| Claude Code setting | Value | Why | -| --- | --- | --- | -| `CLAUDE_CODE_CONNECT_TIMEOUT_MS` | `660000` | Allows the proxy to wait before sending response headers. | -| `API_TIMEOUT_MS` | `720000` | Keeps the per-attempt timeout above the buffering window. | -| `API_FORCE_IDLE_TIMEOUT` | `0` | Prevents the optional idle watchdog from aborting the buffered wait. | - -Both timeouts must be strictly above the window. Matching them exactly risks an -abort at the boundary. The response-header timeout was observed to default to -about 60 seconds in Claude Code; disabling its idle watchdog does not change -that limit. `CLAUDE_API_TIMEOUT` is not a recognized setting for this purpose. - -For Codex, keep `stream_idle_timeout_ms` above `PROXY_RESPONSES_KEEPALIVE_MS` in -early-commit mode, or above `PROXY_RESPONSES_BUFFER_MS` in full-buffer mode. The -README's `660000` covers either default. A zero window does not disable the -client's timeouts or the proxy's request deadline. +Client response-header, stream-idle, and per-attempt timeouts must be strictly greater than the active buffering window: -For Gemini, keep SDK response-header, idle, and total timeouts above -`PROXY_GEMINI_BUFFER_MS`. A timeout of 660,000 ms or more covers the default -buffering window. Keepalives after the live switch are standard SSE comments; -clients that ignore comments for idle accounting must allow their actual content -gaps too. Non-SSE `streamGenerateContent` transports are forwarded without buffering. - -Claude Code Workflow agents have an additional content stall timer. Keep their proxy window -small and below the script's `stallMs`; see [Workflow agents](ARCHITECTURE.md#workflow-agents). +- **Claude Code (`~/.claude/settings.json`):** Set `CLAUDE_CODE_CONNECT_TIMEOUT_MS=660000`, `API_TIMEOUT_MS=720000`, and `API_FORCE_IDLE_TIMEOUT=0` for the default 600,000 ms window (`CLAUDE_API_TIMEOUT` has no effect). +- **Codex (`~/.codex/config.toml`):** Set `stream_idle_timeout_ms = 660000` to cover both the 30s early-commit window and the 600s full-buffer window. +- **Gemini SDKs:** Set HTTP/stream timeouts to at least `660000` ms to cover `PROXY_GEMINI_BUFFER_MS`. +- **Workflow agents:** Keep `PROXY_WORKFLOW_KEEPALIVE_MS` below the workflow script's `stallMs`; see [Workflow agents](ARCHITECTURE.md#workflow-agents). ## Retries and fallback -| Variable | Default | Purpose | +| Variable | Default | Description | | --- | --- | --- | -| `PROXY_TRANSACTIONAL_LOCAL_RETRIES` | `0` | Extra attempts for recoverable streaming failures before commit. `0` retains only the limited fast retry path. | -| `PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS` | `10000` | Maximum extra delay added to upstream `Retry-After` between local retries. `0` removes the extra delay. | -| `PROXY_SDK_RETRY_CAP` | `100` | Stop retry conversion when the Stainless retry counter reaches this value. `0` disables conversion and local retries. | -| `PROXY_REFUSAL_FALLBACK_MODEL` | `claude-opus-5` | Alternate model for an Anthropic refusal before commit. `off`, `none`, `disabled`, or an explicit empty value disables substitution. | - -Local retries need enough time left in the current request and buffering window. -The SDK cap uses the Stainless counter; it does not raise client retry limits. -Clients without that header, including Codex and Gemini SDKs, must enforce their -own retry limits when the proxy cap is positive. With a zero cap, Responses and -transient Gemini failures are returned as terminal HTTP `400` because those -clients may not honor `X-Should-Retry: false` alone. - -Model fallback has a separate budget and remains enabled when the SDK cap is -zero. See [retry budgets](ARCHITECTURE.md#retry-budgets) and -[model fallback](ARCHITECTURE.md#model-fallback) for the exact scope. +| `PROXY_TRANSACTIONAL_LOCAL_RETRIES` | `0` | Extra in-proxy upstream attempts before commit on recoverable streaming failures. `0` keeps only a single fast retry (<3s failure, 250ms delay). | +| `PROXY_LOCAL_RETRY_EXTRA_BACKOFF_CAP_MS` | `10000` (10s) | Max exponential delay added to upstream `Retry-After` between local retries. `0` adds no extra delay. | +| `PROXY_SDK_RETRY_CAP` | `100` | Stops converting errors into `503` retries once `X-Stainless-Retry-Count` reaches this value. `0` disables retry conversion and local retries. | +| `PROXY_REFUSAL_FALLBACK_MODEL` | `claude-opus-5` | Fallback model for pre-commit Anthropic refusals and recognized Fable safeguard errors. Set to `off`, `none`, `disabled`, or `""` to disable. | + +See [Retries](ARCHITECTURE.md#retries) and [Model fallback](ARCHITECTURE.md#model-fallback) for behavior details. ## Stream processing -| Variable | Default | Purpose | +| Variable | Default | Description | | --- | --- | --- | -| `PROXY_RESPONSES_EARLY_COMMIT` | `1` | Stream Responses output as soon as it arrives. `0` enables full buffering under `PROXY_RESPONSES_BUFFER_MS`. | -| `PROXY_RESPONSES_REPLAY_DELTA_BYTES` | `65536` (64 KiB) | Limit for combining adjacent compatible Responses text deltas during replay. `0` preserves raw replay. | -| `PROXY_VALIDATE_JSON` | `1` | Validate Anthropic event and accumulated tool-input JSON. `0` disables these checks and tool-JSON normalization. | -| `PROXY_NORMALIZE_TOOL_JSON` | `1` | Combine Anthropic tool and server-tool input fragments into complete JSON deltas. Requires JSON validation. `0` disables this combination only. | - -The Responses replay limit counts source JSON bytes, not just visible text. It -bounds grouped payloads and reduces bursts of tiny notifications when replaying a -buffered response. Live deltas are unchanged. Incompatible, oversized, malformed, -or ambiguous frames and frames with non-empty logprobs remain raw. - -Responses completion/error detection, Gemini JSON/completion/error checks, and -Anthropic structured refusal detection remain active with `PROXY_VALIDATE_JSON=0`. +| `PROXY_RESPONSES_EARLY_COMMIT` | `1` | Commit and stream `/v1/responses` as soon as the first output event arrives. Set `0` for full buffering under `PROXY_RESPONSES_BUFFER_MS`. | +| `PROXY_RESPONSES_REPLAY_DELTA_BYTES` | `65536` (64 KiB) | Max source JSON bytes when merging adjacent compatible `response.output_text.delta` events on replay. `0` replays raw frames. | +| `PROXY_VALIDATE_JSON` | `1` | Validate Anthropic SSE event JSON and accumulated tool-input JSON. `0` disables Anthropic JSON validation and tool-JSON normalization (terminal/error detection on Responses, Gemini, and Anthropic refusals stays active). | +| `PROXY_NORMALIZE_TOOL_JSON` | `1` | Coalesce fragmented Anthropic `input_json_delta` chunks into a single valid JSON delta on buffered replay (requires `PROXY_VALIDATE_JSON=1`). | ## Storage and diagnostics -| Variable | Default | Purpose | +| Variable | Default | Description | | --- | --- | --- | -| `PROXY_MAX_BUFFER_MEM_BYTES` | `1048576` (1 MiB) | Response buffer memory threshold before spilling to a temporary file. `0` spills immediately. | -| `PROXY_MAX_RESPONSE_BYTES` | `134217728` (128 MiB) | Maximum buffered response size; also bounds each SSE event, including unfinished lines. Must be positive. | -| `PROXY_MAX_REQUEST_BYTES` | `67108864` (64 MiB) | Maximum client request body. Larger bodies are rejected with `413`, or `400` on Responses. Must be positive. | -| `PROXY_SPOOL_DIR` | OS temporary directory | Location for response spill files; typically `$TMPDIR` or `/tmp`. | -| `PROXY_REQUEST_LOG_DIR` | Off | Directory for full request/response JSON archives. Empty or unset disables archiving. | -| `PROXY_VERBOSE` | Off | `1` adds internal retry and archive diagnostic messages. Request outcome logs remain enabled regardless. | - -Archives contain unencrypted payloads and have no automatic retention. See -[saving request and response data](OPERATIONS.md#saving-request-and-response-data) -for their format and data-handling limits. +| `PROXY_MAX_BUFFER_MEM_BYTES` | `1048576` (1 MiB) | In-memory buffer limit per request before spilling to disk in `PROXY_SPOOL_DIR`. `0` spills immediately. | +| `PROXY_MAX_RESPONSE_BYTES` | `134217728` (128 MiB) | Max buffered response size and max single SSE event size (minimum `1`). | +| `PROXY_MAX_REQUEST_BYTES` | `67108864` (64 MiB) | Max client request body size (minimum `1`). Oversized bodies fail with `413` (`400` on Responses). | +| `PROXY_SPOOL_DIR` | OS temp dir | Directory for temporary response spill files (unlinked immediately on release). | +| `PROXY_REQUEST_LOG_DIR` | *Off* | Directory for full request/response JSON archives. Empty or unset disables archiving; see [Saving request and response data](OPERATIONS.md#saving-request-and-response-data). | +| `PROXY_VERBOSE` | `0` | Set `1` to log internal `[local-retry]` and `[request-log]` diagnostics (`1` in Compose). | ## Optional upstream headers -A gateway can set these headers on a non-2xx response to override error -classification: +Your gateway can attach these headers to non-2xx HTTP responses to override the proxy's error heuristics: -| Header | Effect | +| Upstream header | Effect | | --- | --- | -| `x-gateway-retryable: true` | Mark the failure retryable, subject to the remaining budget. | -| `x-gateway-retryable: false` | Mark the failure non-retryable. | -| `x-should-retry: true` or `false` | Apply the same decision if `x-gateway-retryable` did not provide one. | -| `Retry-After` | Supply the retry delay in seconds or as an HTTP date. | - -`x-gateway-retryable` takes precedence over `x-should-retry`, which takes precedence -over status and body heuristics. A terminal Responses error is rendered as HTTP -`400`. Gemini also normalizes non-retryable `408`, `429`, and `5xx` errors to `400` -because its SDKs may ignore `X-Should-Retry: false`. Other non-retryable statuses -are retained. - -These headers control retry classification. Recognized Anthropic safeguards -errors can still trigger a separate [model fallback](ARCHITECTURE.md#model-fallback), -even with a `false` verdict. Set `PROXY_REFUSAL_FALLBACK_MODEL=off` to disable that -substitution too. - -The proxy strips `x-gateway-retryable`, `x-gateway-error-stage`, and -`x-gateway-error-code` from client requests, so clients cannot supply these gateway -hints themselves. +| `x-gateway-retryable: true` / `false` | Force the error to be treated as retryable or terminal (highest precedence). | +| `x-should-retry: true` / `false` | Same effect when `x-gateway-retryable` is not set. | +| `Retry-After` | Delay in seconds or HTTP-date format for local and client retries. | + +Non-retryable Responses errors normalize to HTTP `400`. Non-retryable Gemini `408`, `429`, and `5xx` errors also normalize to `400` so Gemini SDKs do not auto-retry them by status code. + +The proxy strips `x-gateway-retryable`, `x-gateway-error-stage`, and `x-gateway-error-code` from incoming client requests so clients cannot spoof gateway hints. diff --git a/docs/DEVELOPMENT.md b/docs/DEVELOPMENT.md index 305cafe..23481b1 100644 --- a/docs/DEVELOPMENT.md +++ b/docs/DEVELOPMENT.md @@ -1,38 +1,24 @@ # Development -steady-proxy requires Go 1.23 or later and uses only the standard library. -Docker is optional for native development; the live CLI harness requires it. +`steady-proxy` requires Go 1.23+ and uses only the Go standard library (`go.mod` has no external dependencies). Docker with Compose v2 is only needed for container builds and the live CLI test harness. ## Build and run -From a checkout of the repository: - ```bash make build ./steady-proxy --version PROXY_UPSTREAM_URL=https://your-gateway.example.com ./steady-proxy ``` -The binary listens on `127.0.0.1:8789` by default. It reads environment variables -directly and does not load `.env`. The README includes setup for -[Claude Code](../README.md#claude-code), [Codex](../README.md#codex), -[Gemini](../README.md#gemini), and [other API clients](../README.md#other-api-clients). +The binary listens on `127.0.0.1:8789` by default and reads `PROXY_*` variables directly from the process environment (it does not load `.env`). See the [README](../README.md#client-setup) for client configuration. -`make build` stamps the binary with `VERSION`, the Git commit, build date, and -working-tree status. To build a versioned Docker image with explicit build -metadata, run: +`make build` embeds `VERSION`, the Git commit, build timestamp, and working-tree status into the binary (exposed via `--version` and `GET /__version`). To build a Docker image with version metadata: ```bash make docker-build ``` -This tags both `steady-proxy:` and `steady-proxy:latest`. Ordinary -`docker compose up -d --build` also records the version, commit from the minimal -Git metadata in its build context, and build timestamp. The running proxy exposes -that information at `/__version`. - -For version bumps, local release previews, and tag-triggered publishing, see -[releasing](RELEASING.md). +This tags both `steady-proxy:` and `steady-proxy:latest`. For release packaging and publishing, see [Releasing](RELEASING.md). ## Tests and checks @@ -42,36 +28,20 @@ go build ./... make test ``` -`make test` runs `go test -race ./...`, including unit and integration tests. -CI also checks `gofmt` and builds the Docker image. Use focused tests for the -behavior you change, then run the relevant broader checks before submitting it. +`make test` runs `go test -race ./...` across all unit and integration suites. CI also enforces `gofmt`, runs `make release-test`, and verifies the Docker build. ### Live client harness -[`test/live.sh`](../test/live.sh) runs a real `claude -p` client through a -dockerized proxy against a mock gateway with controlled failures. It requires -Docker with Compose v2, `curl`, `claude`, and GNU `timeout` on `PATH`. On macOS, -install GNU coreutils and add its `gnubin` directory to `PATH` so the command is -available as `timeout`. The default run takes about 10 minutes, including a -generation longer than 300 seconds: - -```bash -./test/live.sh -``` - -To skip the long-generation case: +[`test/live.sh`](../test/live.sh) exercises a real `claude -p` CLI client through a containerized proxy against a fault-injecting mock gateway (`testupstream`). It requires Docker with Compose v2, `curl`, `claude`, and GNU `timeout` (on macOS, install `coreutils` and add `gnubin` to `PATH`). ```bash -DO_LONG=0 ./test/live.sh +./test/live.sh # full suite (~10m, includes a >300s long turn) +DO_LONG=0 ./test/live.sh # skip the 300s+ long-turn case ``` -The harness uses a separate Compose project and test image, random loopback ports, -and isolated client settings. By default, it makes no real model requests. It -removes its test project on exit or interruption and retains failed-run logs in -the temporary directory it prints. +The harness runs in an isolated Compose project on random loopback ports, makes no real upstream API calls by default, and cleans up containers on exit (retaining logs in a temporary directory on failure). -To add one real, potentially billed gateway call, export `ANTHROPIC_AUTH_TOKEN` -and run: +To include one live (billed) gateway test, export `ANTHROPIC_AUTH_TOKEN` and run: ```bash DO_REAL=1 PROXY_UPSTREAM_URL=https://your-gateway.example.com ./test/live.sh @@ -79,23 +49,20 @@ DO_REAL=1 PROXY_UPSTREAM_URL=https://your-gateway.example.com ./test/live.sh ## Finding the code -Read the [architecture guide](ARCHITECTURE.md) for the request lifecycle and -[troubleshooting](TROUBLESHOOTING.md) for expected failure behavior. +All production code lives in `package main`: | File | Responsibility | | --- | --- | -| [`main.go`](../main.go) | Configuration, routing, deadlines, local retries, fallback, and access logs. | -| [`wire.go`](../wire.go) | Shared protocol interface and Anthropic stream behavior. | -| [`responses.go`](../responses.go) | OpenAI Responses completion, errors, keepalives, and replay. | -| [`gemini.go`](../gemini.go) | Native Gemini candidate completion, usage, and unchanged SSE replay. | -| [`gemini_errors.go`](../gemini_errors.go) | Google numeric/canonical error classification and rendering. | -| [`sse.go`](../sse.go) | SSE capture, buffering, validation, tool JSON, and the transition to live output. | -| [`classify.go`](../classify.go) | Failure classification and client retry responses. | -| [`transport.go`](../transport.go) | Upstream deadline and byte-idle handling. | -| [`requestlog.go`](../requestlog.go) | Payload archives, body capture, and credential redaction. | -| [`version.go`](../version.go) | CLI and HTTP version reporting. | -| [`testupstream/main.go`](../testupstream/main.go) | Mock gateway used by the live harness. | - -Protocol implementations share the buffering and retry engine through `wireModel`. -When changing one, check both buffered and live delivery: an error before commit -can become an HTTP retry, while an error after commit must terminate the stream. +| [`main.go`](../main.go) | Config parsing, HTTP routing, request deadlines, local retry loop, refusal fallback, and access logging. | +| [`wire.go`](../wire.go) | Shared `wireModel` protocol interface, route matching, and Anthropic stream interpretation. | +| [`responses.go`](../responses.go) | OpenAI Responses stream completion, error handling, `response.proxy_keepalive`, and delta coalescing. | +| [`gemini.go`](../gemini.go) | Native Gemini SSE candidate tracking, usage extraction, and replay. | +| [`gemini_errors.go`](../gemini_errors.go) | Google numeric/canonical error classification, `RetryInfo` parsing, and error envelopes. | +| [`sse.go`](../sse.go) | Shared SSE capture engine (`captureSSECore`), memory/disk spooling, JSON validation, tool-input normalization, and live cutover. | +| [`classify.go`](../classify.go) | Error classification and `503` retry response rendering. | +| [`transport.go`](../transport.go) | Upstream HTTP transport, header timeouts, and byte-idle reader. | +| [`requestlog.go`](../requestlog.go) | Opt-in JSON payload archiving and credential redaction. | +| [`version.go`](../version.go) | `--version` flag and `/__version` endpoint. | +| [`testupstream/main.go`](../testupstream/main.go) | Deterministic fault-injecting mock gateway used by `test/live.sh`. | + +When modifying protocol behavior, check both buffered (`buffered()`) and post-commit (`live()`) paths: pre-commit failures can be discarded and retried as HTTP `503`, whereas post-commit failures must terminate the stream cleanly. diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index a8c363b..a1ddb59 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -1,29 +1,19 @@ -# Running and observing steady-proxy +# Operations -The [quick start](../README.md#quick-start) covers installation and client setup. -This guide covers routine operation, request logs, and payload capture. +See the [README quick start](../README.md#quick-start) for initial setup. This guide covers day-to-day operation, log inspection, payload archiving, and running `steady-proxy` as a `systemd` service. ## Docker Compose -Run these commands from the repository directory after copying -`docker-compose.yml.example` to your local `docker-compose.yml` during setup: +After copying `.env.example` to `.env` and `docker-compose.yml.example` to `docker-compose.yml`: ```bash -docker compose up -d --build -docker compose logs -f proxy +docker compose up -d --build # build and start +docker compose logs -f proxy # tail request logs +docker compose up -d # reload .env changes +docker compose down # stop ``` -After changing `.env`, apply it with `docker compose up -d`. To stop the service, -run `docker compose down`. - -Compose runs the container as `${UID:-1000}:${GID:-1000}` so archived files can be -owned by your host user. If your IDs differ, run `id -u` and `id -g`, then add -their numeric results as `UID` and `GID` in `.env`. Bash does not export its `UID` -variable by default and has no built-in `GID` variable. Exported values take -precedence over `.env`. - -The example mounts the host's `./logs` directory at `/logs` in the container. The container -working directory is `/`, so `PROXY_REQUEST_LOG_DIR=./logs` uses that mount. +The container runs as `${UID:-1000}:${GID:-1000}` so payload archives in `./logs` are owned by your host user. If your host UID/GID is not `1000`, check `id -u` and `id -g` and set `UID` and `GID` in `.env` (Bash does not export `UID` by default and has no `GID` variable). The host `./logs` directory is mounted at `/logs` with container working directory `/`, so `PROXY_REQUEST_LOG_DIR=./logs` writes directly to `./logs`. ## Check the running version @@ -31,12 +21,7 @@ working directory is `/`, so `PROXY_REQUEST_LOG_DIR=./logs` uses that mount. curl -fsS http://127.0.0.1:8789/__version ``` -This returns build metadata without making a gateway request. It confirms that -the proxy is reachable, not that gateway authentication or model routing works. - -To check the full Messages path, send a short prompt from Claude Code, or run the -following after exporting `ANTHROPIC_AUTH_TOKEN` and replacing `` with -a model supported by your gateway. This makes a real, potentially billed request. +`/__version` returns build metadata locally without contacting your gateway. To test the end-to-end gateway path with a real (billed) request: ```bash curl -sS http://127.0.0.1:8789/v1/messages \ @@ -48,9 +33,7 @@ curl -sS http://127.0.0.1:8789/v1/messages \ ## Reading the logs -Request outcome logs are always enabled. `PROXY_VERBOSE=1` adds internal retry -decisions and archive diagnostics. Each line has a date and time prefix; these -illustrative examples omit it: +Every proxied request logs one summary line. Setting `PROXY_VERBOSE=1` adds internal `[local-retry]` and `[request-log]` lines. ```text OK claude-sonnet-4-6/main high in=1.2k out=437 tok end_turn buffered 3.41s @@ -61,110 +44,60 @@ WARN claude-fable-5/main refusal -> retry with claude-opus-5 in=258.8k out=2. FAIL claude-sonnet-4-6/main request_shape 400 0.3s ``` -| Outcome | Meaning | +### Outcomes + +| Tag | Meaning | | --- | --- | -| `OK` | The proxy delivered the response. `buffered` or `live` tells you how. | -| `RETRY` | It returned a retryable error with a delay for the client. This does not prove the client retried. | -| `WARN` | It switched to the fallback model; the later outcome reports that attempt's result. | -| `FAIL` | It returned an error without requesting another attempt, usually because of an invalid request or the retry backstop. | -| `DROP` | The response had already started when delivery failed. The client must handle the incomplete stream. | +| `OK` | Response delivered (`buffered` or `live`). | +| `RETRY` | Recoverable pre-commit failure; returned HTTP `503` with `Retry-After` so the client retries. | +| `WARN` | Pre-commit Anthropic refusal or safeguard error triggered model fallback; a subsequent line logs the fallback outcome. | +| `FAIL` | Non-retryable error (invalid request shape or exhausted retry cap) returned to the client. | +| `DROP` | Stream or body transfer failed *after* `200 OK` headers were already sent; the client must handle recovery. | -The fields after the outcome explain the request: +### Log fields | Field | Meaning | | --- | --- | -| `model/main`, `model/sub`, `model/wf` | Requested model and detected caller kind. `requested->resolved` shows a different model reported by the upstream. | -| `high` or another effort value | Requested Messages `output_config.effort`, Responses `reasoning.effort`, or Gemini `generationConfig.thinkingConfig.thinkingLevel` (lowercased). | -| `in=… out=… tok` | Input and output token counts. Input includes cache reads and creation. | -| `end_turn`, `completed`, etc. | The reported stop reason. | -| `truncated_stream`, `sse_overloaded`, etc. | The classified failure code; use it to identify the actual cause. | -| `502->503` | Original or classified failure status followed by the status sent to the client. A single value means no status change. | -| `retry-after=2s` | Delay requested from the client. | -| `attempt=N` | Client retry count from `X-Stainless-Retry-Count`, shown when positive. | -| `proxy-retries=N` | Extra retries made inside the proxy under the configured local budget. | -| `prun=N` | Longest sequence of consecutive upstream ping events; a large value can indicate a gap in useful content. | -| `id=` | Correlation ID, present when payload archiving is enabled. | - -Caller labels use Claude Code headers and the Workflow system-prompt marker. -Requests without those signals appear as `main`, including requests from other -applications; the label does not identify or restrict the client software. - -Gemini thinking-level logging also accepts ProtoJSON snake_case field names. -Only an explicit named level is reported; the proxy does not infer one from -`thinkingBudget` or a model's default. The original request is forwarded unchanged. -Gemini model labels and archived attempt models come from the model URL, not a -body-level `model` field. - -Verbose `[local-retry]` lines include the cause, wait, and retry index. -`skip=not-enough-time` explains why another attempt could not fit the time budget. -The response headers also expose `X-Steady-Proxy-Mode` on streaming delivery and -`X-Steady-Proxy-Reason` on synthesized errors. +| `model/main`, `model/sub`, `model/wf` | Requested model and caller type (`requested->resolved` if the upstream reports a different model ID). Gemini model names come from the URL path. | +| `high`, `medium`, etc. | Explicit effort level from Messages `output_config.effort`, Responses `reasoning.effort`, or Gemini `generationConfig.thinkingConfig.thinkingLevel` (`minimal`/`low`/`medium`/`high` in camelCase or `snake_case`, lowercased; never inferred from `thinkingBudget` or model defaults). | +| `in=… out=… tok` | Input (including cache read/creation) and output token counts. | +| `end_turn`, `completed`, etc. | Upstream stop/finish reason. | +| `truncated_stream`, `sse_overloaded`, etc. | Classified failure code before status normalization. | +| `502->503` | Upstream/classified HTTP status `->` status sent to the client. | +| `retry-after=2s` | Backoff sent to the client in `Retry-After`. | +| `attempt=N` | Client retry index from `X-Stainless-Retry-Count` (omitted when 0). | +| `proxy-retries=N` | Extra pre-commit attempts made inside the proxy (`PROXY_TRANSACTIONAL_LOCAL_RETRIES`). | +| `prun=N` | Longest run of consecutive upstream SSE pings (high values indicate a content-silent gap). | +| `id=` | Archive correlation ID when `PROXY_REQUEST_LOG_DIR` is enabled. | + +Caller suffixes (`/main`, `/sub`, `/wf`) are derived from Claude Code headers and the Workflow system-prompt marker; other clients appear as `/main`. + +With `PROXY_VERBOSE=1`, `[local-retry]` lines log each internal retry cause, wait duration, and skips (such as `skip=not-enough-time`). Responses also carry `X-Steady-Proxy-Mode` on streamed deliveries and `X-Steady-Proxy-Reason` on synthesized error responses. ## Saving request and response data -Enable payload capture only when you need the exact requests and upstream replies: +To capture full request and response payloads for debugging, set: ```dotenv PROXY_REQUEST_LOG_DIR=./logs ``` -For Compose, add this to `.env` and run `docker compose up -d`. For the binary, -set it in the launch environment. The proxy attempts to write one JSON archive per -proxied client request, including non-streaming routes. Hidden retries and model -fallback appear as separate entries in the same archive's `attempts` array. - -An archive filename looks like `v1-messages-20260621t143005-a1b2c3d4.json`. Its -suffix matches both the log's `id=a1b2c3d4` and the archive's top-level `id`. To -locate it, use `ls logs/*a1b2c3d4*.json` with the ID from your log. +The proxy writes one JSON archive per client request (e.g., `v1-messages-20260621t143005-a1b2c3d4.json`, matching `id=a1b2c3d4` in the access log). Local retries and model fallback attempts are recorded inside the same file's `attempts` array. -Archives use schema `steady-proxy.payload.v2`. They contain the original client -request envelope, redacted upstream URL, proxy build metadata, upstream attempts, -response or transport failures, retry waits, stream statistics, model changes, -and final outcome. Bodies use base64 in `data_base64`, with `encoding`, byte -`size`, and `sha256` metadata. Repeated request bodies may be referenced through -`request_body_ref` instead of duplicated. +Archives use schema `steady-proxy.payload.v2` and include the client request envelope, redacted upstream URL, build metadata, per-attempt headers/bodies, retry delays, stream stats, and final outcome. Payloads are stored lossless in `data_base64` with `encoding`, `size`, and `sha256` metadata (unchanged retry bodies reference `request_body_ref`). Interrupted reads record `capture_error`. -Captured bytes can be restored exactly, including binary and image payloads. -A failed or interrupted read may only capture part of a response; a -`capture_error` field records a read interruption when available. Error-body -capture obeys the upstream idle and request deadlines. - -Archiving is best effort. It does not fail the proxied request if the directory -cannot be created or an archive cannot be written. Enable `PROXY_VERBOSE=1` to -see `[request-log]` errors when an expected archive is missing. +Archiving is best-effort and never fails a proxied call; set `PROXY_VERBOSE=1` to see `[request-log]` write errors. ### Data handling -The proxy has no authentication of its own. Keep its listening port on loopback; -Compose's published port already is. Anyone with access to the port can relay -requests with credentials they supply and read `/__version`. - -Client authorization and provider-specific headers, including `x-goog-api-key`, -are forwarded to `PROXY_UPSTREAM_URL`, along with query authentication such as `key`. -The proxy does not follow redirects. Its HTTP transport honors -`HTTP_PROXY`/`HTTPS_PROXY` and `NO_PROXY` in the proxy process environment, so a -configured forward proxy is also part of the connection path. - -Request bodies are held in memory. Buffered responses spill beyond the configured -memory threshold to temporary files in `PROXY_SPOOL_DIR`; those files are unlinked -and closed when released. Use a tmpfs-backed spool directory if payloads must stay -off persistent disk. - -Payload archives are different: they are persistent and **unencrypted**. Known -credential headers and query parameters are redacted, including `Authorization`, -`x-api-key`, `x-goog-api-key`, `Proxy-Authorization`, `key`, `api_key`, `api_token`, -`access_token`, and `client_secret`. Redacted names are listed in `redactions`. Other headers and -parameters remain, and **bodies are not inspected or redacted**. Base64 is an -encoding, not encryption. - -There is no archive rotation, size cap, or retention policy. Turn capture off -after reproducing a problem, or manage retention yourself. Keep archives in a -directory you control and out of version control; `/logs/` is already ignored. +- **Network exposure:** The proxy has no authentication. Keep `PROXY_LISTEN_ADDR` bound to loopback (`127.0.0.1`). +- **Forwarded credentials & proxies:** Client auth headers (`Authorization`, `x-api-key`, `x-goog-api-key`, etc.) and query parameters (`key`) are forwarded to `PROXY_UPSTREAM_URL`. Upstream redirects are not followed. Standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables are honored. +- **Spool files:** Request bodies stay in memory; buffered responses over `PROXY_MAX_BUFFER_MEM_BYTES` spill to temporary files in `PROXY_SPOOL_DIR` and are unlinked immediately after the attempt. Point `PROXY_SPOOL_DIR` at `tmpfs` if spill files must not touch disk. +- **Archive sensitivity & retention:** Archives are **unencrypted**. Known credential headers and query params (`Authorization`, `x-api-key`, `x-goog-api-key`, `Proxy-Authorization`, `key`, `api_key`, `api_token`, `access_token`, `client_secret`) are redacted and listed under `redactions`, but **request and response bodies are stored raw** (base64-encoded). There is no automatic rotation or retention—disable `PROXY_REQUEST_LOG_DIR` once debugging is complete. ## Run as a user service -For a native installation, place a built binary at the path used below, then -create `~/.config/systemd/user/steady-proxy.service`: +For a native Linux install, copy the `steady-proxy` binary to `/opt/steady-proxy/steady-proxy` and create `~/.config/systemd/user/steady-proxy.service`: ```ini [Unit] @@ -183,8 +116,7 @@ RestartSec=2 WantedBy=default.target ``` -Replace the binary path and gateway URL for your installation, and configure the -client as shown in the README. Enable the service and read its logs with: +Enable and follow the service logs: ```bash systemctl --user daemon-reload @@ -192,6 +124,4 @@ systemctl --user enable --now steady-proxy journalctl --user -u steady-proxy -f ``` -To keep a user service running after logout, the account needs systemd lingering -enabled: `loginctl enable-linger "$USER"`. This may require administrator -authorization on your host. +To keep the service running when logged out, enable lingering: `loginctl enable-linger "$USER"`. diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 178d420..9fd602f 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -1,20 +1,14 @@ # Releasing -Choose the version and release timing manually. Pushing a matching version tag -starts the [release workflow](../.github/workflows/release.yml), which runs CI, -builds archives, generates checksums and release notes, and publishes to GitHub. +Releases are triggered manually by pushing an annotated version tag (`v`). The [release workflow](../.github/workflows/release.yml) runs CI, builds cross-platform archives, verifies checksums, publishes the GitHub Release, and triggers the GHCR container build. -The workflow supports stable versions such as `0.1.1` and `0.2.0`. Release-candidate -tags such as `v0.2.0-rc.1` are rejected. `VERSION` is the source of truth; the tag -must be exactly `v` followed by that value. +Only stable semantic versions (such as `0.1.1` or `0.2.0`) are accepted; pre-release tags like `v0.2.0-rc.1` are rejected. `VERSION` is the single source of truth, and the Git tag must match `v$(cat VERSION)`. ## Prepare a release -1. Update `VERSION`. Use a patch increment for fixes, or a minor increment for new - features while the project is at `0.x`. -2. Commit and merge the change into `main`. Wait for CI to pass, and check that - your local checkout contains the intended commit and has no uncommitted files. -3. Create an annotated tag. For example, after changing `VERSION` to `0.1.1`: +1. Update `VERSION` (patch bump for bug fixes, minor bump for features while on `0.x`). +2. Merge to `main`, wait for CI to pass, and ensure your local `main` checkout is clean and up to date. +3. Create an annotated tag matching `VERSION`: ```bash git tag -a v0.1.1 -m "Release v0.1.1" @@ -22,86 +16,48 @@ must be exactly `v` followed by that value. ## Preview locally -Local packaging needs Go, Git, Make, and Python 3.9 or later on Linux or macOS. -Run it from the clean, tagged checkout: +Requires Go, Git, Make, and Python 3.9+ on Linux or macOS: ```bash make release TAG=v0.1.1 ``` -This only creates files under `dist/v0.1.1/`; it does not push the tag, contact -GitHub, or upload anything. The directory contains four archives for Linux and -macOS on amd64 and arm64, plus `SHA256SUMS`. Each archive includes the executable, -`VERSION`, and the MIT license. The executable reports its version, commit, build -time, and clean working-tree status. +This builds `dist/v0.1.1/` locally without pushing or uploading anything: +- Four archives (`linux`/`darwin` × `amd64`/`arm64`), each containing `steady-proxy`, `VERSION`, and `LICENSE`. +- `SHA256SUMS` for all four archives. -The packager checks that the tag points to the current commit and that the -checkout is clean. It also runs `--version` for the host platform's binary to -check its metadata. It refuses to replace an existing output directory. Failed -builds clean up their temporary files without leaving a completed release folder. - -The output directory is ignored by Git. To package the same tag again, move or -remove only that tag's local output directory first. Packaging does not replace -the test suite; `make release-test` checks packaging failures and archive contents, -while normal CI checks the application and Docker build. +The packager verifies that `TAG` points to `HEAD`, the working tree is clean, and the host binary's `--version` output matches the tag metadata. It refuses to overwrite an existing `dist/` directory (remove `dist/v0.1.1` first if re-running). Run `make release-test` to execute the packaging test suite. ## Publish -When the version, code, and local preview are ready, push the tag: +Push the tag to trigger the release workflow: ```bash git push origin v0.1.1 ``` -The workflow then: - -1. Runs the existing CI workflow, including formatting, vet, build, race tests, - packaging tests, and the Docker build. -2. Packages the tagged source and retains the archives as a workflow artifact for - seven days. -3. Verifies the downloaded archives against `SHA256SUMS` in a separate job with - permission to create releases. -4. Refuses to proceed if any release already uses the tag, including a draft. -5. Creates a draft with generated GitHub release notes and all five assets, then - publishes it after uploading succeeds. -6. Calls the container publishing workflow to build the tagged source and push - `ghcr.io/xllent-ai/steady-proxy:v0.1.1` (using the actual release tag). - -The workflows use GitHub's automatic token; no personal token is needed. The -container job has `packages: write` permission. Review the published release notes -for any migration or configuration guidance that commit and pull-request summaries -do not explain. +The workflow: +1. Runs the full CI suite (`gofmt`, `go vet`, `go build`, `go test -race`, `make release-test`, and Docker build). +2. Builds the release archives and uploads them as a 7-day workflow artifact. +3. Verifies `SHA256SUMS` in a release-publishing job, refuses to overwrite any existing release or draft for that tag, creates a draft release with generated release notes, and publishes it once all assets upload. +4. Invokes the container workflow to build and push `ghcr.io/xllent-ai/steady-proxy:v0.1.1`. + +All steps use GitHub's built-in `GITHUB_TOKEN`; no external secrets are required. ## Container images -The [container workflow](../.github/workflows/docker-publish.yml) publishes Linux -amd64 images to `ghcr.io/xllent-ai/steady-proxy`: +The [container workflow](../.github/workflows/docker-publish.yml) publishes Linux `amd64` images to `ghcr.io/xllent-ai/steady-proxy`: -- Every push to `main` updates `latest` from that push's commit. -- Publishing a GitHub release builds its tag and publishes an image with that - tag, such as `v0.1.1`. Release images never move `latest`. -- Both paths also publish `sha-` and stamp the executable with - `VERSION`, the checked-out commit, build time, and clean working-tree status. +- **Push to `main`:** Publishes `latest` and `sha-`. +- **Release published:** Publishes `` (e.g., `v0.1.1`) and `sha-`. Release builds never overwrite `latest`. -The release workflow calls the container workflow directly because releases -created with `GITHUB_TOKEN` do not trigger another workflow via a `release` event. -Manually published releases use the `release: published` trigger instead. +Note: Because releases created with `GITHUB_TOKEN` do not fire `release` workflow events, `release.yml` calls `docker-publish.yml` directly via `workflow_call`. Manually published GitHub releases trigger `docker-publish.yml` via `release: published`. -After the first image is published, check the package's visibility in GitHub's -package settings; change it to public to allow unauthenticated pulls. The image's -source label links it to this repository. If a package with the same name already -exists, grant this repository Actions access to it before publishing. +After the first package push, verify in GitHub package settings that `ghcr.io/xllent-ai/steady-proxy` visibility is set to **Public** so unauthenticated pulls work. ## If a release fails -Use the failed Actions run to identify the failing step. A test or packaging -failure creates no GitHub release. An upload or publication failure can leave a -draft; inspect it and delete that draft before rerunning the failed jobs. The -workflow will not overwrite its assets automatically. - -If only the container job fails after the release is published, rerun that failed -job; do not recreate the release. It rebuilds the same release tag. - -If a published release needs a fix, use a new version and tag. Keep existing -published tags and assets unchanged so downloads continue to identify the same -code and checksums. +- **CI or packaging failed:** No GitHub release was created. Fix the issue, bump to a new version if commits changed, or re-run if transient. +- **Asset upload failed mid-publish:** Delete the leftover draft release in GitHub before re-running the failed job (the workflow refuses to overwrite existing drafts). +- **Container job failed after release published:** Re-run only the failed container job in GitHub Actions; do not delete or recreate the published release. +- **Bug found after publishing:** Never move or overwrite a published tag. Cut a new patch version in `VERSION` and tag a new release. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 8651899..9e45bf4 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -1,233 +1,126 @@ # Troubleshooting -Start with the request's outcome and failure code in the -[access log](OPERATIONS.md#reading-the-logs). The code preserves the cause even -when the client receives a generic `503`. +Start by checking the outcome tag and failure code in the [access log](OPERATIONS.md#reading-the-logs). Even when the client receives a generic `503`, the log line records the underlying cause. -Before the proxy sends response headers, a recoverable failure can be retried -locally or returned as a client retry. After headers have been sent, the proxy can -only end a failed stream and log `DROP`. The tables below assume the failure -happens **before commit** unless stated otherwise. Recovery still depends on the -remaining time and retry budgets. +Failures **before commit** (before `200 OK` headers are sent to the client) can be retried locally or returned as a `503` client retry. Failures **after commit** cannot be replaced with an HTTP error; the proxy closes the stream and logs `DROP`. ## Find the symptom -| Symptom | Start here | +| Symptom | Section | | --- | --- | -| EOF, socket closure, incomplete reply, or JSON parse error | [Broken streams](#broken-streams) | -| Repeated overload, rate-limit, authentication, or billing errors | [Service errors](#service-errors) | -| Invalid tools, context, schema, or model | [Invalid requests](#invalid-requests) | -| No visible output, timeout, or `client_gone` | [Long or silent requests](#long-or-silent-requests) | -| Workflow agent reports “stalled … no progress” | [Workflow stalls](#workflow-stalls) | -| A different model answers after a refusal | [Model fallback](#model-fallback) | -| Size-limit failure, missing archive, or disk error | [Storage and payload capture](#storage-and-payload-capture) | +| EOF, socket reset, truncated reply, or JSON parse error | [Broken streams](#broken-streams) | +| Overload (`529`), rate limit (`429`), `5xx`, auth, or billing errors | [Service errors](#service-errors) | +| Invalid tool blocks, schema error, context overflow, or unknown model | [Invalid requests](#invalid-requests) | +| Request hangs, times out, or logs `client_gone` / `deadline` | [Long or silent requests](#long-or-silent-requests) | +| Claude Code Workflow agent aborts with “stalled … no progress” | [Workflow stalls](#workflow-stalls) | +| A different model answers after a refusal (`WARN`) | [Model fallback](#model-fallback) | +| `request_too_large`, `response_too_large`, `spool_error`, or missing archive | [Storage and payload capture](#storage-and-payload-capture) | ## Broken streams -These failures often appear after an upstream has already returned HTTP `200`. -Holding the response lets the proxy detect the failure before the client accepts -that success status. +Upstreams often fail mid-stream *after* sending HTTP `200`. Buffering catches these failures before the client sees them: -| Message or condition | What the proxy detects | Result before commit | +| Upstream failure | Proxy classification | Pre-commit action | | --- | --- | --- | -| `JSON Parse error: Unexpected EOF` | Connection closes before the terminal event, or tool input ends as incomplete JSON. | Retry as `truncated_stream` or `malformed_sse`. | -| `Request was aborted.` or `The socket connection was closed unexpectedly.` | Interrupted upstream connection. | Retry a transport or stream-read failure. | -| `Connection closed mid-response` or `Connection closed while thinking` | Missing completion after partial output or reasoning. | Retry the incomplete stream. | -| HTTP/2 `INTERNAL_ERROR; received from peer` | Upstream resets a stream. | Retry as a stream-read or transport error. | -| `JSON Parse error: Unexpected identifier` | Invalid JSON in a complete Anthropic event. | Retry as `malformed_sse`. | -| Tool/server-tool fragments never form valid JSON | Invalid accumulated Anthropic tool input. | Retry as `malformed_sse`. | -| HTML or another non-SSE response to a streaming request | Unexpected content type. | Retry as `unexpected_content_type`. | - -For Anthropic, leave `PROXY_VALIDATE_JSON=1` to catch malformed event and tool -payloads. `PROXY_NORMALIZE_TOOL_JSON=1` also combines fragmented tool input before -forwarding it. Turning validation off disables that normalization too. - -For Responses streams, a valid `response.completed` is required for successful completion. -A malformed terminal, including duplicate keys or invalid Unicode, does not count -as success. Ordinary content frames are handled according to Codex's lenient -parser rather than Anthropic's validation rules. A `response.incomplete` event -also triggers a retry; it is not a successful completion. - -Gemini `streamGenerateContent?alt=sse` requires a clean transport EOF after every -requested candidate finishes, or a prompt block. A `finishReason` alone does not -end capture: a later usage frame or error still matters. Missing candidates, -missing finish reasons, partial SSE frames, malformed JSON, and read faults cause -a retry before commit. After commit, the proxy aborts HTTP instead of sending a -clean EOF that could look successful. Native `MAX_TOKENS` and safety stops are -delivered unchanged; they are not stream failures. - -If the log shows `DROP`, the client had already received headers. Default Responses -mode reaches that point as soon as output appears. Enable full buffering with -`PROXY_RESPONSES_EARLY_COMMIT=0` to protect more of the response, and set the -[client idle timeout](CONFIGURATION.md#match-the-client-to-the-window) above the -buffering window. For Messages, check whether the turn outlasted its window or -used the shorter Workflow window. +| Stream closes before terminal event (`JSON Parse error: Unexpected EOF`, `Connection closed mid-response` / `while thinking`) | `truncated_stream` | Discard buffer and retry. | +| Socket reset, `Request was aborted.`, or HTTP/2 `INTERNAL_ERROR; received from peer` | Transport / stream-read error | Discard buffer and retry. | +| Invalid JSON in an Anthropic event (`JSON Parse error: Unexpected identifier`) or incomplete/invalid accumulated `tool_use` / `server_tool_use` JSON | `malformed_sse` | Discard buffer and retry. | +| HTML error page or non-SSE `Content-Type` on a streaming call | `unexpected_content_type` | Discard buffer and retry. | + +### Protocol-specific validation + +- **Anthropic Messages:** Keep `PROXY_VALIDATE_JSON=1` and `PROXY_NORMALIZE_TOOL_JSON=1` enabled so malformed events are retried and fragmented `input_json_delta` chunks are combined before replay. +- **OpenAI Responses:** Requires a valid `response.completed` event (duplicate keys or invalid Unicode in the terminal event are rejected; `response.incomplete` is treated as a retryable failure). Ordinary content events follow Codex's lenient parser. +- **Gemini `streamGenerateContent?alt=sse`:** Requires clean transport EOF after all requested candidates report a `finishReason` (or a standalone prompt block). Trailing usage frames are preserved. Missing candidates, missing finish reasons, partial frames, or malformed JSON trigger a retry before commit, and abort the HTTP transport after commit so the client does not mistake a drop for clean EOF. Native `MAX_TOKENS` and safety stops are valid completions and pass through unchanged. + +If you see frequent `DROP` logs on `/v1/responses`, enable full buffering with `PROXY_RESPONSES_EARLY_COMMIT=0` and ensure the client's `stream_idle_timeout_ms` exceeds `PROXY_RESPONSES_BUFFER_MS`. ## Service errors -The proxy treats temporary service failures as retryable and renders them as a -generic `503` with retry headers. This keeps recognizable overload or rate-limit -shapes from triggering Claude Code's smaller special-case retry budgets. +Recoverable upstream errors are normalized into a generic HTTP `503` with `X-Should-Retry: true` and `Retry-After` so clients like Claude Code stay on their standard retry loop instead of aborting early on `529` or `429`. -| Upstream error | Expected handling | +| Upstream error | Handling | | --- | --- | -| `529`, `overloaded_error`, “API is at capacity,” or “No available providers” | Retry; the log retains the overload or HTTP failure code. | -| `500`, `502`, `503`, or another server failure | Retry, subject to gateway override headers. | -| `429`, `rate_limit_error`, or “Rate limited” | Retry with the upstream delay when available. | -| An Anthropic SSE `error` reporting overload or rate limit | Retry before commit; end the stream after commit. | -| A Responses `error` or `response.failed` reporting overload, timeout, or rate limit | Retry before commit; end the stream after commit. | -| A Gemini SSE `error` envelope reporting `UNAVAILABLE`, `RESOURCE_EXHAUSTED`, or a server failure | Retry before commit with Gemini-shaped HTTP `503`; abort HTTP after commit. | -| Invalid API key, disabled organization, billing block, or unknown `4xx` | Usually retry, since gateway access or capacity may recover. | - -A Responses error can carry `type: "invalid_request_error"` while its more -specific `code` is `request_timeout`, `rate_limit_exceeded`, or -`server_is_overloaded`. The recognized transient code takes precedence. For -example, `request_timeout` with “stream disconnected before completion” remains -retryable instead of becoming an invalid-request failure. - -For Anthropic SSE `event: error`, the proxy also recognizes one verified transport -disconnect mislabeled `invalid_request_error`. The complete message must be exactly: - -```text -stream error: stream disconnected before completion: stream closed before response.completed -``` - -This failure is classified as `sse_stream_disconnect`. Before commit, failed -output is discarded and the existing local and client retry budgets apply, within -the current deadline and buffering window. After commit, the stream ends without -transparent replay. This exception does not change retry limits or timeouts. - -If retries continue, check the gateway's credentials, account state, availability, -and rate limits. A permanently bad key or exhausted balance will not improve -without intervention, even though the proxy allows time for recovery. Generic -policy errors usually follow this retry policy; recognized Responses `bio_policy` -and `cyber_policy` codes are terminal. - -Use [retry budgets](ARCHITECTURE.md#retry-budgets) to understand the remaining -attempts. Increasing the budget cannot fix a persistent configuration error. -A gateway that knows an error is permanent can use -[`x-gateway-retryable: false`](CONFIGURATION.md#optional-upstream-headers). +| `529`, `overloaded_error`, “API is at capacity”, “No available providers” | Retry (`503`); original overload code logged. | +| `500`, `502`, `503`, `504` | Retry (`503`), unless overridden by [gateway headers](CONFIGURATION.md#optional-upstream-headers). | +| `429`, `rate_limit_error`, “Rate limited” | Retry (`503`), preserving upstream `Retry-After` or `RetryInfo`. | +| In-stream SSE `error` / `response.failed` (overload, timeout, rate limit, `UNAVAILABLE`, `RESOURCE_EXHAUSTED`) | Retry (`503`) before commit; terminate stream (`DROP`) after commit. | +| Auth failure, billing block, or unknown `4xx` | Retried by default so temporary gateway/account blips do not abort a turn. | + +### Special error-classification rules + +- **Responses transient codes:** If a `/v1/responses` error has `type: "invalid_request_error"` but carries a transient `code` (`request_timeout`, `rate_limit_exceeded`, or `server_is_overloaded`), the transient code wins and the request is retried. +- **Anthropic SSE disconnect signature:** An Anthropic in-stream `event: error` with `type: "invalid_request_error"` is normally terminal, except for the exact gateway disconnect message: + ```text + stream error: stream disconnected before completion: stream closed before response.completed + ``` + which is classified as `sse_stream_disconnect` and retried before commit. +- **Policy errors:** Generic policy errors retry by default, but Responses `bio_policy` and `cyber_policy` codes are terminal. + +Because auth and billing failures are retried by default, a permanently invalid API key or empty balance will retry until the retry budget expires. Gateways that can distinguish permanent failures should return `x-gateway-retryable: false`. ## Invalid requests -An identical retry cannot repair an invalid request. Recognized request errors -are returned with `X-Should-Retry: false`; terminal Responses errors use HTTP -`400` so Codex stops retrying them. +Deterministic request-shape errors cannot succeed on an identical retry. The proxy returns them immediately with `X-Should-Retry: false` (and normalizes terminal Responses errors and non-retryable Gemini `408`/`429`/`5xx` errors to HTTP `400`). -| Error or message | What to fix | +| Error condition | Fix | | --- | --- | -| Missing tool result, duplicate `tool_use` ID, or unexpected `tool_use_id` | Repair or rewind the conversation's tool history. | -| Tool-use concurrency error with “Run /rewind” | Restore a consistent conversation state. | -| “Extra inputs are not permitted,” `context_management`, or `input_examples` | Align the request schema with what the gateway accepts. | -| Unexpected `anthropic-beta` header value | Use a beta/header combination supported by the gateway. | -| `max_tokens must be greater than thinking.budget_tokens` | Increase the output limit or reduce the thinking budget. | -| Thinking blocks “cannot be modified” | Preserve the original thinking blocks in conversation history. | -| Image dimensions exceed the allowed size | Resize or replace the image. | -| “Prompt is too long” or context-length error | Compact or shorten the input. | -| Explicit `model_not_found` code | Choose a model the gateway supports. | -| Responses `unknown_parameter`, `invalid_type`, or another recognized validation code | Correct the named field or value. | -| Gemini `INVALID_ARGUMENT`, `NOT_FOUND`, or a bare numeric `400`/`404` | Correct the request, model, or referenced resource. The documented `API_KEY_INVALID` ErrorInfo is an authentication exception and remains retryable. | - -Anthropic HTTP errors use recognized message signatures. A bare HTTP -`invalid_request_error` without one can still retry; an Anthropic in-stream -`invalid_request_error` is terminal except for the exact transport-disconnect -signature above. Invalid schemas, malformed tool messages, and context-limit -failures remain terminal. Responses classification uses error codes, types, and -recognized messages, with known transient codes taking precedence over generic -invalid-request types. Gateway retry headers override HTTP heuristics. - -Gemini decodes numeric `error.code`, canonical `error.status`, and `error.message` -on both HTTP and SSE errors, including the array-wrapped HTTP errors returned by -non-SSE `streamGenerateContent`. `FAILED_PRECONDITION` is not automatically permanent: -it can indicate billing or regional access. The documented Google -`google.rpc.ErrorInfo` reason `API_KEY_INVALID` remains retryable even when its -status is `INVALID_ARGUMENT`, preserving the proxy's authentication policy. -Gateway retry headers still take precedence on HTTP errors. See Google's -[GenerateContent error reference](https://ai.google.dev/gemini-api/docs/generate-content/api-errors). - -Gemini's native [`google.rpc.RetryInfo`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto) -cooldown is honored for HTTP and in-band SSE errors when no valid positive HTTP -`Retry-After` is present. Fractional seconds round up. A delay that cannot fit the -remaining local retry window is returned to the client without an early retry. - -Interrupted HTTP error-body reads are transport failures, not proof of an invalid -request. Incomplete or malformed Gemini error JSON likewise cannot establish a -validation error. Explicit retry headers remain authoritative unless the request -context itself has ended. -Proxy buffering deadlines remain timeout failures during HTTP/2 body reads; they -are not treated as client cancellations. +| Missing `tool_result`, duplicate `tool_use` ID, or unexpected `tool_use_id` | Rewind or repair the conversation's tool history. | +| Tool-use concurrency error (`Run /rewind`) | Rewind the session to a consistent state. | +| `Extra inputs are not permitted`, `context_management`, or `input_examples` | Remove unsupported fields for the target gateway/model. | +| Invalid `anthropic-beta` header | Use beta flags supported by your gateway. | +| `max_tokens must be greater than thinking.budget_tokens` | Raise `max_tokens` or lower `thinking.budget_tokens`. | +| Thinking blocks `cannot be modified` | Preserve historical thinking blocks unmodified. | +| Oversized image dimensions | Resize or remove the image. | +| `Prompt is too long` / context-length exceeded | Compact or shorten the conversation. | +| `model_not_found` | Configure a model supported by your gateway. | +| Responses `unknown_parameter`, `invalid_type`, or schema validation codes | Fix the rejected request field. | +| Gemini `INVALID_ARGUMENT`, `NOT_FOUND`, or bare HTTP `400`/`404` | Fix the request or model name (`API_KEY_INVALID` remains retryable). | + +### Protocol-specific validation details + +- **Anthropic:** HTTP `invalid_request_error` responses only fail permanently when they match a known request-shape signature; unrecognized HTTP `400`s remain retryable. In-stream SSE `invalid_request_error` events are terminal (except for the exact `sse_stream_disconnect` string above). +- **Gemini:** Parses `error.code`, `error.status`, and `error.message` across HTTP and SSE errors (including JSON-array HTTP error bodies from non-SSE `streamGenerateContent`). `FAILED_PRECONDITION` (often billing or region routing) and `google.rpc.ErrorInfo` with `reason: "API_KEY_INVALID"` remain retryable. Native [`google.rpc.RetryInfo`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto) delays are honored when no positive `Retry-After` header is present (fractional seconds round up). +- **Interrupted error reads & HTTP/2 deadlines:** Truncated error bodies, malformed error JSON, and proxy buffering deadlines during HTTP/2 reads are treated as retryable transport/timeout failures—never as validation errors or `client_gone`. ### Codex context compaction -A Responses `response.failed` event whose error code is exactly -`context_length_exceeded` is forwarded to Codex so its native compaction path can -handle it. This is an intentional exception to converting errors into HTTP -responses. A top-level `error` event with the same code, or another recognized -context-error shape, is returned as terminal HTTP `400` instead. +When a `/v1/responses` stream emits `response.failed` with `error.code: "context_length_exceeded"`, the proxy forwards that SSE event to the client instead of converting it into an HTTP error, allowing Codex's native auto-compaction to trigger. A top-level `error` event with the same code (or an HTTP context-length error) is returned as terminal HTTP `400`. ## Long or silent requests -Silence while buffering is expected. The proxy has not sent response headers -because it is still waiting for a complete response. If the client cancels during -that wait, the proxy cannot keep the abandoned call running for it. +While buffering, the proxy sends no HTTP headers to the client. If the client's own timeout is shorter than the buffering window, the client will disconnect (`client_gone`). -| Log or timing | Likely cause | Next step | +| Symptom | Cause | Fix | | --- | --- | --- | -| Claude aborts around 60 seconds with `client_gone` | Its response-header timeout is shorter than the buffering window. | Apply both Claude timeout settings from the [README](../README.md#claude-code). | -| Claude aborts near 600 seconds | Its per-attempt timeout may equal the default window. | Keep `API_TIMEOUT_MS` strictly above the window. | -| Codex idles out in full-buffer mode | `stream_idle_timeout_ms` is too short. | Set it above `PROXY_RESPONSES_BUFFER_MS`. | -| `upstream_idle` | No upstream bytes arrived within the byte-idle limit. | Check the gateway for a stalled generation or connection. | -| `deadline` before streaming begins | A header timeout, buffering-window deadline, or total request deadline elapsed. | Compare the duration with the [timeout settings](CONFIGURATION.md#timeouts). | -| `DROP` after a long live stream | The response failed after the proxy had committed. | Check the cause code and the client's stream recovery. | - -The buffering window includes header waits and all hidden retries. More local -retries do not extend it. Keepalive messages help an open client connection, but -cannot override its absolute timeout or make an upstream produce useful content. +| Claude Code aborts around ~60s with `client_gone` | Default response-header timeout (~60s) fired while buffering. | Set `CLAUDE_CODE_CONNECT_TIMEOUT_MS=660000`, `API_TIMEOUT_MS=720000`, and `API_FORCE_IDLE_TIMEOUT=0` in `~/.claude/settings.json`. | +| Client aborts right at ~600s with `client_gone` | Client timeout equals `PROXY_KEEPALIVE_MS` with no margin. | Keep client timeouts strictly above the buffering window (e.g., `660000` ms+). | +| Codex disconnects in full-buffer mode | `stream_idle_timeout_ms` is shorter than `PROXY_RESPONSES_BUFFER_MS`. | Set `stream_idle_timeout_ms = 660000` in `~/.codex/config.toml`. | +| `upstream_idle` in proxy log | No bytes received from gateway within `PROXY_UPSTREAM_BYTE_IDLE_MS`. | Check gateway health or increase `PROXY_UPSTREAM_BYTE_IDLE_MS`. | +| `deadline` before streaming starts | Header timeout (`PROXY_RESP_HEADER_TIMEOUT_MS`), buffering window, or total request deadline expired. | Compare the logged duration against your [timeout settings](CONFIGURATION.md#timeouts). | +| `DROP` after a long live stream | Upstream failed after the proxy committed to live output. | Check the logged failure code; recovery depends on client stream retries. | ## Workflow stalls -Workflow `agent()` calls have a separate watchdog based on real forwarded content. -The proxy's default 10-second Workflow window reduces buffering delay before the -first useful output. A workflow should appear as `/wf` in the log. +Claude Code Workflow `agent()` requests (logged as `/wf`) use `PROXY_WORKFLOW_KEEPALIVE_MS` (default 10s) so buffered content reaches the client before the Workflow runtime's 180s content-stall watchdog fires. -If it still stalls, keep `PROXY_WORKFLOW_KEEPALIVE_MS` below its `stallMs`, then -check for a long gap in actual upstream content. Pings do not reset the Workflow -watchdog; a high `prun` value can help identify such a gap. A larger script-side -`stallMs` is needed when the upstream itself remains content-silent too long. -See [Workflow behavior](ARCHITECTURE.md#workflow-agents) for detection and limits. +If a Workflow agent still stalls: +1. Verify `PROXY_WORKFLOW_KEEPALIVE_MS` is well below the script's `stallMs`. +2. Check the log for a high `prun=N` value, which indicates the upstream sent SSE pings without real content tokens. Because SSE pings do not reset the Workflow watchdog, increase `stallMs` in your workflow script (e.g., `agent(prompt, {stallMs: 300000})`). ## Model fallback -A `WARN` line with `refusal -> retry with ` or -`safeguard -> retry with ` records an Anthropic model substitution. The -following outcome is for the fallback attempt. Ensure that the configured model -is available through your gateway. +A `WARN` log with `refusal -> retry with ` or `safeguard -> retry with ` means the proxy intercepted a pre-commit Anthropic refusal and re-issued the request with `PROXY_REFUSAL_FALLBACK_MODEL`. The next log line shows the fallback attempt's result. -The substitution only works before commit and happens at most once per client -request. It does not apply to Responses requests. To keep the original refusal response, set -`PROXY_REFUSAL_FALLBACK_MODEL=off`; see [model fallback](ARCHITECTURE.md#model-fallback). +Ensure your gateway supports the configured fallback model, or set `PROXY_REFUSAL_FALLBACK_MODEL=off` to pass refusals through unchanged. See [Model fallback](ARCHITECTURE.md#model-fallback). ## Storage and payload capture -`request_too_large` means the incoming body exceeded `PROXY_MAX_REQUEST_BYTES`. -It is rejected without truncation. `response_too_large` means a buffered response -or an individual SSE event exceeded `PROXY_MAX_RESPONSE_BYTES`; it is returned as -a non-retryable failure before commit. Reduce the payload or deliberately adjust -the relevant limit. - -`spool_error` means a storage operation failed, not that the size cap was reached. -Check permissions and free space for `PROXY_SPOOL_DIR`. Storage I/O failures remain -retryable, and temporary response buffers are released when attempts end. - -For a missing or incomplete payload archive, enable `PROXY_VERBOSE=1` and look for -`[request-log]` errors. A `capture_error` in an archive indicates a read problem. -Archives have no automatic retention; see -[payload capture and data handling](OPERATIONS.md#saving-request-and-response-data). +- **`request_too_large`:** Client request body exceeded `PROXY_MAX_REQUEST_BYTES` (64 MiB default); rejected immediately without truncation. +- **`response_too_large`:** Buffered response or single SSE event exceeded `PROXY_MAX_RESPONSE_BYTES` (128 MiB default); returned as a non-retryable error before commit. +- **`spool_error`:** Disk I/O failure in `PROXY_SPOOL_DIR` (check free space and permissions); treated as retryable. +- **Missing or partial payload archives:** Run with `PROXY_VERBOSE=1` and check for `[request-log]` errors, or inspect `capture_error` inside the JSON archive. ## Non-streaming calls -Successful non-streaming requests, token counting, and model listing are forwarded -without full response buffering. Their HTTP or connection errors can still become -client retries, but there are no hidden transactional retries. If a successful -body copy fails, the proxy logs `DROP` and aborts the HTTP response so a truncated -body does not look like clean success. +Non-streaming requests (`stream: false`), `count_tokens` / `countTokens`, and model listings bypass SSE buffering and local retries, using a single upstream attempt per call. Recoverable HTTP and connection failures still convert into `503` client retries. If a response body transfer fails mid-copy after `200 OK` headers are written, the proxy logs `DROP` and aborts the connection so the client does not accept a truncated JSON body.