Skip to content

fix(sentry): never capture opentelemetry.context detach noise - #807

Open
sjawhar wants to merge 2 commits into
LuthienResearch:mainfrom
trajectory-labs-pbc:fix/sentry-otel-context-noise
Open

fix(sentry): never capture opentelemetry.context detach noise#807
sjawhar wants to merge 2 commits into
LuthienResearch:mainfrom
trajectory-labs-pbc:fix/sentry-otel-context-noise

Conversation

@sjawhar

@sjawhar sjawhar commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator

Problem

With Sentry enabled, the gateway captures an error event on nearly every streaming request: ValueError: <Token ...> was created in a different Context, logged by opentelemetry.context. In our production deployment this single logger produced roughly 86k Sentry error events in 13 hours and exhausted the organization's error quota the day after Sentry was turned on.

Root cause

OpenTelemetry detaches context tokens in finally blocks that can run in a different asyncio context than the one that attached them, which happens routinely on streaming paths. OTel already catches the ValueError and logs it itself (logger.exception("Failed to detach context") in opentelemetry/context/__init__.py), and the request completes normally: the captured events show 200 responses in their breadcrumbs. The Sentry logging integration then promotes each of those log records into an error event.

Fix

Ignore the opentelemetry.context logger in init_sentry(), next to the existing opentelemetry.sdk.trace.export ignore that exists for the same reason. The ignore-logger test now pins both loggers. Changelog fragment included.

Verification

  • uv run pytest tests/luthien_proxy/unit_tests/test_sentry_scrubbing.py passes at this commit
  • ruff check and ruff format --check clean on the touched files
  • The dev_checks.sh pytest stage shows three failures on this machine (test_onboard.py::test_find_docker_ports_respects_env_vars, two in test_config_registry.py) that reproduce identically on a clean main checkout with pytest -n 4, so they are unrelated to this change

Update: broadened to the whole opentelemetry.* namespace

Two more Sentry issues turned up on the same shared root cause but a different sub-logger, so exact-matching two logger names wasn't enough: LUTHIEN-C (opentelemetry.exporter.otlp.proto.http.trace_exporter, 2,044 events) and LUTHIEN-F (opentelemetry.exporter.otlp.proto.http.metric_exporter, 998 events) — the OTel collector rejecting exported spans/metrics with HTTP 403.

sentry_sdk's ignore_logger matches with fnmatch, so ignore_logger("opentelemetry.*") replaces both exact-match calls and covers every current and future OTel sub-logger — exporter/instrumentation noise included — without having to chase each one individually. A new behavioral test (test_opentelemetry_wildcard_ignores_every_sub_logger) exercises the real sentry_sdk EventHandler._can_record against five logger names (including a hypothetical future one) plus a non-OTel logger, pinning the fnmatch semantics this depends on. This also matches the precedent set by the Datadog monitor for luthien-proxy errors, which already excludes @logger:opentelemetry.* for the same reason (telemetry-pipeline health is the collector's own monitors' job).

Silences: LUTHIEN-C, LUTHIEN-F (new), plus the original LUTHIEN context-detach noise this PR already covered.

OTel swallows 'Failed to detach context' (ValueError on cross-task token
resets during streaming) and the proxied request is unaffected, but the
Sentry logging integration captured it on ~every request: ~86k events in
13h on 2026-08-05, exhausting the org error quota (LUTHIEN-1).

Ignore the logger at init, same as opentelemetry.sdk.trace.export.
Broadens the ignore_logger calls from the two specific loggers
(opentelemetry.context, opentelemetry.sdk.trace.export) to the whole
opentelemetry.* namespace via sentry_sdk's fnmatch-based ignore_logger
matching.

This also silences LUTHIEN-C and LUTHIEN-F, which are the OTel collector
rejecting exported spans/metrics with HTTP 403
(opentelemetry.exporter.otlp.proto.http.trace_exporter /
.metric_exporter logging at ERROR) — a different sub-logger than either
existing ignore, so it kept burning quota. Telemetry-pipeline health is
the collector's own Datadog monitors' job (which already exclude
@logger:opentelemetry.* for the same reason); Sentry should match.

No behavior change to proxying or telemetry export itself — this only
changes what Sentry captures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant