Summary
OpenAICompatClient.send() in forge/clients/openai_compat.py catches httpx.ReadTimeout and converts it to a clean BackendError(408, "Read timeout"). send_stream() -- used whenever the client requests streaming, the normal case for most chat clients -- has no equivalent handling. Confirmed present in both 0.8.3 and 0.9.0 (current latest).
Observed symptom in production
Running forge-proxy in front of Ollama (external mode, --backend-url http://<host>:11434/v1) with a large (~19K token) system prompt, a request hit the backend read timeout mid-stream. Instead of a clean timeout error, the client received:
can only concatenate list (not "str") to list
journalctl on the proxy confirmed the underlying exception was httpx.ReadTimeout; the TypeError is a secondary failure in error handling downstream of the escaped exception.
Root cause (traced in source)
send() (non-streaming) wraps the POST in try/except httpx.ReadTimeout as exc: raise BackendError(408, "Read timeout") from exc.send_stream()'s async with self._http.stream(...) block and the async for line in response.aiter_lines() loop have no equivalent try/except, so a ReadTimeout during streaming propagates raw instead of becoming a BackendError.- Something downstream that expects a
BackendError receives the raw httpx.ReadTimeout instead and mishandles it, producing the TypeError: can only concatenate list (not "str") to list -- consistent with code concatenating a list-typed partial-stream accumulator with a string error message rather than appending/wrapping it.
Suggested fix
Wrap send_stream()'s stream/read loop in the same except httpx.ReadTimeout as exc: raise BackendError(408, "Read timeout") from exc used in send().
Environment
- forge-guardrails 0.8.3 (production) and 0.9.0 (freshly installed and diffed) -- same gap in both
- Backend: Ollama via its OpenAI-compatible
/v1 endpoint (external/--backend-url mode)
- Proxy invocation:
forge-proxy --backend-url http://<ollama-host>:11434/v1 --model <model> --budget-mode manual --budget-tokens 98304 --backend-timeout 600 --inject-respond-tool
Happy to share the full journalctl excerpt or a minimal repro if useful.
Summary
OpenAICompatClient.send()inforge/clients/openai_compat.pycatcheshttpx.ReadTimeoutand converts it to a cleanBackendError(408, "Read timeout").send_stream()-- used whenever the client requests streaming, the normal case for most chat clients -- has no equivalent handling. Confirmed present in both 0.8.3 and 0.9.0 (current latest).Observed symptom in production
Running forge-proxy in front of Ollama (external mode,
--backend-url http://<host>:11434/v1) with a large (~19K token) system prompt, a request hit the backend read timeout mid-stream. Instead of a clean timeout error, the client received:journalctlon the proxy confirmed the underlying exception washttpx.ReadTimeout; the TypeError is a secondary failure in error handling downstream of the escaped exception.Root cause (traced in source)
send()(non-streaming) wraps the POST intry/except httpx.ReadTimeout as exc: raise BackendError(408, "Read timeout") from exc.send_stream()'sasync with self._http.stream(...)block and theasync for line in response.aiter_lines()loop have no equivalent try/except, so aReadTimeoutduring streaming propagates raw instead of becoming aBackendError.BackendErrorreceives the rawhttpx.ReadTimeoutinstead and mishandles it, producing theTypeError: can only concatenate list (not "str") to list-- consistent with code concatenating a list-typed partial-stream accumulator with a string error message rather than appending/wrapping it.Suggested fix
Wrap
send_stream()'s stream/read loop in the sameexcept httpx.ReadTimeout as exc: raise BackendError(408, "Read timeout") from excused insend().Environment
/v1endpoint (external/--backend-urlmode)forge-proxy --backend-url http://<ollama-host>:11434/v1 --model <model> --budget-mode manual --budget-tokens 98304 --backend-timeout 600 --inject-respond-toolHappy to share the full journalctl excerpt or a minimal repro if useful.