[bug] Closed provider streams are never closed — abandoned streaming responses leak HTTP pool slots and surface as APITimeoutError/PoolTimeout

[meffmadd]

Summary

OGX never explicitly closes the OpenAI AsyncStream returned by a provider's streaming endpoint. When a downstream consumer abandons a stream early (client disconnect, proxy cancellation, one of our wrapping generators being closed), the underlying httpx.Response socket stays open. The httpcore connection pool keeps the connection marked in-use, so it cannot be reused; with enough leaks the pool is exhausted and subsequent requests raise httpcore.PoolTimeout, which the OpenAI SDK surfaces as openai.APITimeoutError.

A socket in this state is observed as CLOSE-WAIT in the process (peer sent FIN, local side never closed). This was investigated against a production incident — full notes in http-connection-pool-close-wait-investigation.md (repo root, untracked).

Root cause

The shared streaming wrapper owns the provider stream but never closes it:

• src/ogx/providers/utils/inference/openai_mixin.py — _postprocess_chunk() defines _gen() as async for chunk in resp: yield chunk with no try/finally. Closing the wrapper does not close resp (the OpenAI AsyncStream).
• Router wrappers inherit the gap:
  • src/ogx/core/routers/inference.py — _rewrite_completion_stream_model_id() (no finally), stream_tokens_and_compute_metrics_openai_chat() (finally only records metrics/storage, never closes its response).
  • src/ogx_api/inference/fastapi_routes.py, src/ogx_api/messages/fastapi_routes.py, src/ogx_api/responses/fastapi_routes.py — _sse_generator/_preserve_context_for_sse close the direct downstream generator on CancelledError/GeneratorExit but not on generic exceptions, and no layer ever reaches the OpenAI AsyncStream.close().

The OpenAI SDK does close the response — AsyncStream.close() (async) → response.aclose(), and __stream__ has its own finally: await response.aclose() — but that only runs if the AsyncStream itself is iterated to completion or explicitly close()d. Our wrapper prevents that from happening on early abandonment.

Verification

Confirmed against current source and the vendored stack (httpx 0.28.1, httpcore 1.0.9, openai 2.43.0):

1. Pool-slot lease — live probe: local server sends Content-Length then FIN without the declared body; client with max_connections=1 reads one chunk, abandons the stream, then issues a second request → second request raises httpx.PoolTimeout. httpcore only evicts connections that are is_closed() or idle-expired; an in-use peer-closed connection is neither.
2. Wrapper leak — live probe modeled on _postprocess_chunk._gen(): consuming one chunk then aclose()-ing the wrapper leaves the upstream stream open (closed=False).
3. Timeout surfacing — openai _base_client.py maps httpx.TimeoutException (incl. PoolTimeout) → APITimeoutError.
4. Per-provider pools — OpenAIMixin._cached_client is a per-provider, per-worker PrivateAttr; pools are not shared across provider instances (relevant for attribution, not for the fix).

Proposed fix (not yet implemented)

1. Shared idempotent close helper supporting both APIs: generic async iterators (.aclose()) and openai.AsyncStream async close().
2. _postprocess_chunk._gen(): wrap in try/finally → close resp. This must be chained through every wrapping layer, because abrupt cancellation can leave _gen itself unclosed:
   • stream_tokens_and_compute_metrics_openai_chat() and _rewrite_completion_stream_model_id(): close their response in finally.
   • The three _preserve_context_for_sse/_sse_generator variants: also close on the generic except Exception path, not just cancellation.
3. Regression tests: (a) one-chunk-then-close leaks no upstream stream; (b) upstream truncates body + client disconnects mid-stream → next request completes without PoolTimeout.
4. Optional hardening (deployment): per-provider network.limits (already supported by LimitsConfig) to bound blast radius; provider-ID/base-URL attribution on timeout logs to pinpoint exhausted pools.

Notes

• Recommended bug label; open to tech-debt if we prefer to scope it as hardening.
• The bug_report.yml template (torch collect_env, paste-traceback) doesn't fit this networking/resource-leak bug; filing with inline body rather than the template.
• No code changes made in this repo; the investigation doc is untracked at repo root.

[meffmadd]

Live production attribution update (2026-08-25)

The current incident provides stronger provider/worker attribution than the original report.

Confirmed live behavior

• The llm-d gateway + Qwen 3.6 InferencePool returned successful synthetic completions in roughly 45–66 ms.
• All four Qwen 3.6 vLLM engines returned successful direct completions in roughly 42–64 ms; EPP reported four ready endpoints and no queue.
• Requests through OGX intermittently returned no bytes until client timeout.
• Correlating localhost requests with /proc/<pid>/fd showed every sampled failure landing on the same Uvicorn worker. On that exact worker:
  • Qwen 3.5 returned HTTP 200 in 140 ms.
  • Qwen 3.6 timed out.
• That worker accumulated 111 CLOSE-WAIT plus 24 established sockets to the llm-d gateway. Other workers remained able to serve Qwen 3.6.
• The live OGX traceback now explicitly reaches httpcore.PoolTimeout; the pod logs contained 197 PoolTimeout occurrences and 40 surfaced openai.APITimeoutError occurrences in the inspected interval.

This establishes that the incident is a per-provider, per-worker Qwen 3.6 pool exhaustion, not an InferencePool or vLLM-engine outage. Uvicorn routing between healthy and poisoned workers explains the intermittent symptom.

Is this specifically caused by multimodality?

Current evidence says no—not at the HTTP/SSE protocol level:

1. Fully consumed Qwen 3.6 text and image streams both had valid chunked SSE framing, one [DONE] marker, and reusable keep-alive connections.
2. Repeated synthetic multimodal requests with a 1.64 MB inline PNG succeeded through the gateway, both streaming and non-streaming, while reusing one connection.
3. The same request through OGX's Responses API also completed for Qwen 3.6 text, Qwen 3.6 image, and Qwen 3.5 text.
4. A one-slot real AsyncOpenAI/httpx pool using the same wrapper shape reproduced the leak after one-chunk-then-wrapper-close for all three cases:
   • Qwen 3.6 text -> next request APITimeoutError
   • Qwen 3.6 image -> next request APITimeoutError
   • Qwen 3.5 text -> next request APITimeoutError

Multimodal requests are nevertheless a plausible triggering path: aggregate stored data shows that Qwen 3.6 handles substantially more inline-image requests, with successful stored response objects averaging about 1.9 MB and reaching about 4 MB. Those requests traverse additional Responses API generators. We have not yet established that image handling causes more early abandonment, however; normal large-image completion does not reproduce an upstream defect.

The defensible conclusion is therefore:

The cleanup defect is model-agnostic. Qwen 3.6 is the provider pool observed exhausting first. Multimodal/Responses lifecycle may determine which streams are abandoned, but there is currently no evidence that Qwen 3.6 or vLLM emits malformed/truncated responses during normal multimodal completion.

Affected-version corrections

• The live ogx:1.2.2 image contains openai 2.53.0, httpx 0.28.1, and httpcore 1.0.9.
• Although current OGX main supports network.limits, OGX 1.2.2's NetworkConfig schema does not contain limits, and build_network_client_kwargs() does not apply them. Deployment network.limits values are therefore ignored by the affected image. httpx still defaults to 100 max connections / 20 keep-alive connections / 5 s keep-alive expiry.
• The OpenAI SDK's live default pool/read timeout is 600 seconds with two retries, explaining the user-visible indefinite hangs once a provider pool is exhausted.

Remaining attribution work

To identify the Qwen 3.6-specific trigger rather than only the exhausted pool, the next run after clearing the pod should capture:

• provider ID + sanitized base URL/path on pool acquisition/timeout;
• per-provider/per-worker pool totals (active, idle, waiting, peer-closed/unreleased);
• stream API (chat.completions vs responses), multimodal flag, and close reason (normal exhaustion, downstream cancellation, parser/upstream exception);
• socket deltas during a matched text/image and complete/cancel matrix.

Because all llm-d providers currently share one gateway ClusterIP and differ only by URL path, old sockets cannot be attributed to a model from /proc/net/tcp alone. That instrumentation is needed to distinguish a multimodal-specific abandonment path from stochastic accumulation in the Qwen 3.6 pool.