System Info
- OGX version:
1.2.2
- Image:
ghcr.io/tu-wien-datalab/ogx:1.2.2
- Deployment: Kubernetes, 8 Uvicorn workers
- Inference backends: remote OpenAI-compatible/vLLM providers
- OpenAI SDK:
2.43.0
- httpx:
0.28.1
- httpcore:
1.0.9
- Python/OS details: not yet captured
Information
The issue was observed in a production deployment rather than an example script.
🐛 Describe the bug
The OGX process accumulates TCP connections in the CLOSE_WAIT state during normal request handling. In one incident, 34 connections to a remote inference endpoint remained stable in CLOSE_WAIT, including while no recent requests to that endpoint were visible in the logs. The connections remained until the OGX process was restarted; after restart, they were gone.
During the same period, requests repeatedly failed with httpcore.PoolTimeout and openai.APITimeoutError, and some chat-completion requests returned HTTP 500. It is not yet established whether every timeout came from the same provider pool as the observed CLOSE_WAIT connections.
A possible explanation is that OGX does not consistently close a provider HTTP response/connection when a streaming request is interrupted, abandoned, or closed by the upstream peer.
A minimal probe modeled after OGX's streaming wrapper also showed that consuming one chunk and then closing the wrapper did not close the wrapped upstream stream.
Error logs
The recurring exception types were:
httpcore.PoolTimeout
openai.APITimeoutError
Expected behavior
Provider responses and TCP connections should be released when a request completes, is cancelled, or is abandoned. Connections should leave CLOSE_WAIT promptly, connection counts should remain stable, and later requests should not fail because stale connections occupy pool slots.
System Info
1.2.2ghcr.io/tu-wien-datalab/ogx:1.2.22.43.00.28.11.0.9Information
The issue was observed in a production deployment rather than an example script.
🐛 Describe the bug
The OGX process accumulates TCP connections in the
CLOSE_WAITstate during normal request handling. In one incident, 34 connections to a remote inference endpoint remained stable inCLOSE_WAIT, including while no recent requests to that endpoint were visible in the logs. The connections remained until the OGX process was restarted; after restart, they were gone.During the same period, requests repeatedly failed with
httpcore.PoolTimeoutandopenai.APITimeoutError, and some chat-completion requests returned HTTP 500. It is not yet established whether every timeout came from the same provider pool as the observedCLOSE_WAITconnections.A possible explanation is that OGX does not consistently close a provider HTTP response/connection when a streaming request is interrupted, abandoned, or closed by the upstream peer.
A minimal probe modeled after OGX's streaming wrapper also showed that consuming one chunk and then closing the wrapper did not close the wrapped upstream stream.
Error logs
The recurring exception types were:
Expected behavior
Provider responses and TCP connections should be released when a request completes, is cancelled, or is abandoned. Connections should leave
CLOSE_WAITpromptly, connection counts should remain stable, and later requests should not fail because stale connections occupy pool slots.