Skip to content

fix(sentry): drop ClientDisconnect, downgrade transport/malformed-JSON logs - #814

Draft
sjawhar wants to merge 6 commits into
LuthienResearch:mainfrom
trajectory-labs-pbc:fix/sentry-expected-client-and-network-noise
Draft

sjawhar wants to merge 6 commits into
LuthienResearch:mainfrom
trajectory-labs-pbc:fix/sentry-expected-client-and-network-noise

Conversation

@sjawhar

@sjawhar sjawhar commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Problem

With Sentry enabled, three more categories of expected client/network conditions still surface as error-level events, on top of the two already-open fixes (#807 OTel logger noise, #809 upstream provider status codes):

  • ClientDisconnect (starlette.requests) — raised while reading the client's request body after the client has already disconnected. Unhandled anywhere in this codebase; the client is gone, there's nothing to respond to. Three separate Sentry issues (LUTHIEN-3/4/E) for the three call sites that read the body (/v1/messages, /v1/{path}, /openai/{path}).
  • Mid-stream network errorshttpx.RemoteProtocolError, ReadTimeout, ReadError raised directly by httpx (not wrapped in AnthropicConnectionError) when the upstream connection drops while we're reading a streaming response. These fell through _build_error_event's generic else branch, which logs at error — that branch is meant for genuine proxy bugs, but it also caught this backend-network condition (LUTHIEN-A/B/G).
  • Malformed JSON in an incoming request — the client sent a body that doesn't parse as JSON. Already correctly returned to the client as an HTTP 400; the logger.error call at that site was the only reason it shows up in Sentry (LUTHIEN-7/9).

None of these are proxy defects. Combined they're several dozen events across six issues, not huge individually, but stacking with the other two fixes to help clear the org's exhausted error quota.

Fix

  • _sentry_before_send: new _DROPPED_EXCEPTION_TYPES set matched by (module, type) against the built event's exception list (not hint["exc_info"], which for a client-disconnect is the outer ExceptionGroup anyio's TaskGroup wraps it in — the SDK has already flattened the group's members into event["exception"]["values"] by the time before_send runs). Currently just ("starlette.requests", "ClientDisconnect").
  • _build_error_event: new elif isinstance(e, AnthropicUpstreamTransportError) branch, sitting next to the existing AnthropicConnectionError branch it mirrors — same error_type/message, logger.warning instead of the generic branch's logger.error. (AnthropicClient wraps the raw httpx.TransportError into AnthropicUpstreamTransportError before it reaches this function — see the client-boundary bullet below — so this branch matches the wrapper type, not the raw httpx exception.)
  • _process_request: the Malformed JSON in Anthropic request log call downgraded from logger.error to logger.warning.
  • AnthropicClient.complete()/.stream(): raw httpx.TransportError raised during the actual upstream call is now wrapped in a new AnthropicUpstreamTransportError type at the client boundary, so the pipeline can tell a genuine upstream network flap apart from a raw httpx.TransportError raised elsewhere (e.g. a policy's own outbound HTTP call), which must stay on the generic error-level path.
  • Streaming path (_handle_execution_streaming's mid-stream exception handler): AnthropicUpstreamTransportError joins the existing AnthropicConnectionError branch that sets the internal completion final_status to 503 (previously fell to the generic else: final_status = 500) — this feeds the completion webhook's http_status field and the request-log recorder's inbound/outbound status, not an HTTP status returned to the client (headers are already flushed for an in-progress stream; the client only sees the mid-stream SSE error event from _build_error_event above).
  • Non-streaming path: _handle_anthropic_error classifies AnthropicUpstreamTransportError into a 502 BackendAPIError — previously this type wasn't classified at all in the non-streaming path and propagated as an unclassified 500. This one is the literal HTTP status returned to the client, since no response has been sent yet at this point.
  • DirectApiProvider.complete(): new except AnthropicUpstreamTransportError branch translating it into InferenceProviderError, mirroring the existing anthropic.APIConnectionError branch — the new type isn't an anthropic.APIConnectionError subclass, so without this it escaped the provider's error boundary unclassified instead of following the same connection-error path every other backend failure in this provider takes.

Genuine proxy bugs are unaffected — the generic except Exception/else branches still log at error, pinned by test_builds_generic_error_event_logs_at_error and test_builds_generic_error_event_for_policy_origin_transport_error.

No behavior change to proxying itself for a successful response, and no change to what bytes reach the client mid-stream: the SSE error event's JSON shape is identical, only its error.type/message and log level change. Two internal status values used for accounting (not returned to the client) do change for this exception class: the non-streaming path's unclassified 500 becomes the correct 502, and the streaming completion webhook/request-log's unclassified 500 becomes 503. This only changes what Sentry captures plus that internal accounting.

Verification

  • uv run pytest tests/luthien_proxy/unit_tests/test_sentry_scrubbing.py tests/luthien_proxy/unit_tests/pipeline/test_anthropic_processor.py tests/luthien_proxy/unit_tests/llm/test_anthropic_client.py — 164 passed (new tests: test_drops_client_disconnect, test_drops_client_disconnect_flattened_from_exception_group, test_keeps_same_named_exception_from_different_module, test_keeps_mixed_exception_group_with_non_client_disconnect_member, test_builds_transport_error_event_and_logs_at_warning, test_builds_generic_error_event_logs_at_error, test_builds_generic_error_event_for_policy_origin_transport_error, test_upstream_transport_error_raises_backend_api_error, plus the TestAnthropicClientTransportErrorWrapping class in test_anthropic_client.py and a caplog assertion added to test_malformed_json_returns_400).
  • uv run pytest tests/luthien_proxy/unit_tests/inference/test_direct_api.py — 29 passed (new: test_upstream_transport_error_becomes_provider_error).
  • ruff format --check / ruff check clean on every touched file.
  • pyright: 0 errors, 0 warnings, 0 informations on the touched source files (pyrightconfig's include is src/saas_infra only; the touched test files are outside pyright's scope in this repo).

Issues silenced

LUTHIEN-3, LUTHIEN-4, LUTHIEN-E (ClientDisconnect), LUTHIEN-A, LUTHIEN-B, LUTHIEN-G (mid-stream transport errors), LUTHIEN-7, LUTHIEN-9 (malformed JSON).

Related

Part of the same Sentry-quota bug sweep as #807 (OTel logger noise → broadened to opentelemetry.* wildcard in a follow-up commit on that branch, also silences LUTHIEN-C/F) and #809 (upstream provider status codes → broadened to include 400/404/401 in a follow-up commit on that branch, also silences LUTHIEN-6/2/D).

@sjawhar
sjawhar marked this pull request as draft August 29, 2026 22:48
…N logs

Three symptom groups of the same shape: a client or network condition the
pipeline already handles correctly still surfaces as a Sentry error.

- ClientDisconnect (starlette.requests): raised while reading the client
  request body after the client has already left. Completely unhandled
  anywhere in this codebase — there's no one to send a response to, and no
  proxying behavior depends on it. Dropped in _sentry_before_send by
  (module, type) so an unrelated same-named exception isn't swallowed by
  accident. Silences LUTHIEN-3/4/E.

- Mid-stream httpx.TransportError (RemoteProtocolError, ReadTimeout,
  ReadError, ...): raised directly by httpx, not wrapped in
  AnthropicConnectionError, when the upstream connection drops while
  _build_error_event's caller is mid-read of a streaming response. Added as
  a new branch logging at warning, matching the existing
  AnthropicConnectionError treatment right above it. Silences LUTHIEN-A/B/G.

- Malformed JSON in an incoming request: the client's fault, already
  returned to the client as an HTTP 400 by the surrounding code. Downgraded
  logger.error to logger.warning at the one call site. Silences LUTHIEN-7/9.

Genuine proxy bugs are unaffected: the generic except-Exception branch in
_build_error_event still logs at error for anything that isn't one of the
above, and a new test pins that. No behavior change to proxying itself —
response codes, retry behavior, and client-visible errors are unchanged;
only what Sentry captures changes.
…ientDisconnect

_sentry_before_send dropped the whole event if ANY exception entry matched
ClientDisconnect, so a mixed ExceptionGroup(ClientDisconnect, RuntimeError)
would be swallowed even though the RuntimeError is a genuine proxy-side
failure riding alongside the disconnect. Require every non-group-wrapper
(leaf) exception entry to match _DROPPED_EXCEPTION_TYPES instead, treating
ExceptionGroup/BaseExceptionGroup wrapper entries as structural.

Addresses thermonuclear-deep-review Medium finding on PR LuthienResearch#814.
legion-implementer Bot pushed a commit to trajectory-labs-pbc/luthien-proxy that referenced this pull request Aug 29, 2026
The httpx.TransportError warning-downgrade lived in _build_error_event, which
also receives policy-origin exceptions via _handle_execution_streaming's
broad except around the emissions iterator, so a policy's own
httpx.TransportError got misclassified as an upstream network flap; the
non-streaming path (_handle_anthropic_error) didn't classify TransportError
at all and it surfaced as an unclassified 500.

Move the classification to the upstream client boundary:
AnthropicClient.complete/stream now wrap a genuine httpx.TransportError from
the actual Anthropic call into a new AnthropicUpstreamTransportError.
_build_error_event (streaming) and _handle_anthropic_error (non-streaming)
both downgrade that type to a warning-level 502/api_connection_error,
matching the existing AnthropicConnectionError handling. A raw
httpx.TransportError from anywhere else (e.g. a policy's own outbound call)
is not this type and stays on the error-level internal path.

Addresses thermonuclear-deep-review High-quality finding on PR LuthienResearch#814.
@legion-implementer
legion-implementer Bot force-pushed the fix/sentry-expected-client-and-network-noise branch from 3227f60 to d4ce431 Compare August 29, 2026 23:32
The httpx.TransportError warning-downgrade lived in _build_error_event, which
also receives policy-origin exceptions via _handle_execution_streaming's
broad except around the emissions iterator, so a policy's own
httpx.TransportError got misclassified as an upstream network flap; the
non-streaming path (_handle_anthropic_error) didn't classify TransportError
at all and it surfaced as an unclassified 500.

Move the classification to the upstream client boundary:
AnthropicClient.complete/stream now wrap a genuine httpx.TransportError from
the actual Anthropic call into a new AnthropicUpstreamTransportError.
_build_error_event (streaming) and _handle_anthropic_error (non-streaming)
both downgrade that type to a warning-level 502/api_connection_error,
matching the existing AnthropicConnectionError handling. A raw
httpx.TransportError from anywhere else (e.g. a policy's own outbound call)
is not this type and stays on the error-level internal path.

Addresses thermonuclear-deep-review High-quality finding on PR LuthienResearch#814.
…nal_status

The mid-stream exception handler in _handle_execution_streaming only
special-cased AnthropicStatusError and AnthropicConnectionError before
falling through to `else: final_status = 500`. AnthropicUpstreamTransportError
(AnthropicClient's own wrapper around a real upstream network failure,
added earlier in this PR) fell into that generic branch, so a mid-
stream Anthropic network outage recorded final_status=500 in the
completion webhook and request-log rows — misclassifying an upstream
outage as a proxy bug, even though the emitted SSE error event already
correctly classified it as api_connection_error.

Fold AnthropicUpstreamTransportError into the same branch as
AnthropicConnectionError (503), matching how both are already treated
identically in _build_error_event and _handle_anthropic_error.
@legion-implementer
legion-implementer Bot force-pushed the fix/sentry-expected-client-and-network-noise branch from d4ce431 to ddd3bbc Compare August 30, 2026 00:26
The 4 code comments that attributed these regression tests to an internal
review tool name are rewritten to describe the finding generically. No
behavior or test-logic change.

red: grep -rn "thermonuclear-deep-review" tests/luthien_proxy/unit_tests/llm/test_anthropic_client.py tests/luthien_proxy/unit_tests/pipeline/test_anthropic_processor.py tests/luthien_proxy/unit_tests/test_sentry_scrubbing.py
  -> 4 hits (test_anthropic_client.py:441, test_anthropic_processor.py:879+1169, test_sentry_scrubbing.py:202)
green: same grep -> 0 hits
Omp-Session: 01a081d4-2514-7000-b8e2-ff8548776146
…iProvider

DirectApiProvider.complete() calls AnthropicClient.complete(), which this PR's
earlier commit changed to raise AnthropicUpstreamTransportError (a plain,
gateway-owned Exception subclass — not an anthropic.APIConnectionError) for a
raw httpx.TransportError from the actual upstream call. DirectApiProvider's
catch chain only translated anthropic.APIConnectionError into
InferenceProviderError, so the new type escaped the documented provider error
boundary as an unclassified exception instead of InferenceProviderError, same
as every other backend-connection failure this provider handles.

Adds the missing except branch, mirroring the existing APIConnectionError one.

red: uv run pytest tests/luthien_proxy/unit_tests/inference/test_direct_api.py::TestErrorTranslation::test_upstream_transport_error_becomes_provider_error
  -> AnthropicUpstreamTransportError escapes uncaught (1 failed)
green: same command -> 1 passed; full test_direct_api.py -> 29 passed
Omp-Session: 01a081d4-2514-7000-b8e2-ff8548776146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant