Skip to content

fix(pixel-relay): return 502 bad gateway when upstream model router is unavailable - #5605

Open
vaibhavsrv wants to merge 1 commit into
Osmantic:public-betafrom
vaibhavsrv:fix/pixel-relay-upstream-502
Open

vaibhavsrv wants to merge 1 commit into
Osmantic:public-betafrom
vaibhavsrv:fix/pixel-relay-upstream-502

Conversation

@vaibhavsrv

Copy link
Copy Markdown
Contributor

Why this matters

When Pixel agent forwards chat or model discovery requests to the internal model router through ods/extensions/services/pixel-model-relay/relay.py, an unavailable, terminating, or restarting model-router container causes upstream_task to raise an uncaught aiohttp.ClientConnectorError (or ClientError / OSError / TimeoutError). Because the reverse proxy endpoint _inference in relay.py evaluated upstream = await upstream_task without connection error handling, transport failures bubble up to the aiohttp framework, resulting in an unhandled HTTP 500 "Server got itself in trouble" error rather than a structured HTTP 502 Bad Gateway response. Additionally, sending data=body on GET requests (such as /v1/models) passes an empty string body with a forced JSON content-type header, which can break upstream endpoints expecting clean GET requests.

This fix surgically guards await upstream_task in _inference against (ClientError, OSError, asyncio.TimeoutError) and returns a standardized JSON 502 Bad Gateway payload ({"error": {"message": "upstream model router unavailable", "type": "bad_gateway", "code": "502"}}). It also sanitizes GET requests to avoid sending empty request bodies, and cleanly catches client errors during chunk consumption. Existing routing, authentication, authorization checks, and stream piping semantics remain untouched.

Validation

  • Baseline reproduction: Pointing relay.UPSTREAM to an unreachable port raised unhandled ClientConnectorError inside _inference, failing with HTTP 500 Internal Server Error (500 != 502).
  • Post-fix behavior: Requests to /v1/chat/completions and /v1/models against an unreachable upstream cleanly return HTTP 502 Bad Gateway with standard JSON error payload.
  • Telemetry statement: "[Pixel-model-relay] suites: 2 passed. New-test Ruff, ShellCheck, and diff checks pass; new regressions wired into Linux CI."

Overlap check

Risk / AI disclosure

AI-assisted investigation, implementation and CLI regressions. This strengthens reverse proxy upstream connection error handling, not runtime admission. Independent human review and platform/runtime qualification remain gates. No running configuration, deployment or upstream merge changed.

Follow-up integration evidence

Composed with #5604 at 8e30bba without conflicts. Production and test diffs passed together; pixel model relay checks remain intact.
Backlog composition was local-only (production/test diffs, excluding workflow/Makefile wiring); it is not an upstream merge or independent human approval. Declared live-review gates remain open.

@vaibhavsrv
vaibhavsrv force-pushed the fix/pixel-relay-upstream-502 branch from 762bfe7 to 81f6b4d Compare September 17, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant