fix(sentry): never capture opentelemetry.context detach noise - #807
Open
sjawhar wants to merge 2 commits into
Open
fix(sentry): never capture opentelemetry.context detach noise#807sjawhar wants to merge 2 commits into
sjawhar wants to merge 2 commits into
Conversation
OTel swallows 'Failed to detach context' (ValueError on cross-task token resets during streaming) and the proxied request is unaffected, but the Sentry logging integration captured it on ~every request: ~86k events in 13h on 2026-08-05, exhausting the org error quota (LUTHIEN-1). Ignore the logger at init, same as opentelemetry.sdk.trace.export.
legion-implementer
Bot
force-pushed
the
fix/sentry-otel-context-noise
branch
from
August 5, 2026 16:15
ecfa384 to
745f9e7
Compare
Broadens the ignore_logger calls from the two specific loggers (opentelemetry.context, opentelemetry.sdk.trace.export) to the whole opentelemetry.* namespace via sentry_sdk's fnmatch-based ignore_logger matching. This also silences LUTHIEN-C and LUTHIEN-F, which are the OTel collector rejecting exported spans/metrics with HTTP 403 (opentelemetry.exporter.otlp.proto.http.trace_exporter / .metric_exporter logging at ERROR) — a different sub-logger than either existing ignore, so it kept burning quota. Telemetry-pipeline health is the collector's own Datadog monitors' job (which already exclude @logger:opentelemetry.* for the same reason); Sentry should match. No behavior change to proxying or telemetry export itself — this only changes what Sentry captures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With Sentry enabled, the gateway captures an error event on nearly every streaming request:
ValueError: <Token ...> was created in a different Context, logged byopentelemetry.context. In our production deployment this single logger produced roughly 86k Sentry error events in 13 hours and exhausted the organization's error quota the day after Sentry was turned on.Root cause
OpenTelemetry detaches context tokens in
finallyblocks that can run in a different asyncio context than the one that attached them, which happens routinely on streaming paths. OTel already catches the ValueError and logs it itself (logger.exception("Failed to detach context")inopentelemetry/context/__init__.py), and the request completes normally: the captured events show 200 responses in their breadcrumbs. The Sentry logging integration then promotes each of those log records into an error event.Fix
Ignore the
opentelemetry.contextlogger ininit_sentry(), next to the existingopentelemetry.sdk.trace.exportignore that exists for the same reason. The ignore-logger test now pins both loggers. Changelog fragment included.Verification
uv run pytest tests/luthien_proxy/unit_tests/test_sentry_scrubbing.pypasses at this commitruff checkandruff format --checkclean on the touched filesdev_checks.shpytest stage shows three failures on this machine (test_onboard.py::test_find_docker_ports_respects_env_vars, two intest_config_registry.py) that reproduce identically on a cleanmaincheckout withpytest -n 4, so they are unrelated to this changeUpdate: broadened to the whole
opentelemetry.*namespaceTwo more Sentry issues turned up on the same shared root cause but a different sub-logger, so exact-matching two logger names wasn't enough: LUTHIEN-C (
opentelemetry.exporter.otlp.proto.http.trace_exporter, 2,044 events) and LUTHIEN-F (opentelemetry.exporter.otlp.proto.http.metric_exporter, 998 events) — the OTel collector rejecting exported spans/metrics with HTTP 403.sentry_sdk'signore_loggermatches withfnmatch, soignore_logger("opentelemetry.*")replaces both exact-match calls and covers every current and future OTel sub-logger — exporter/instrumentation noise included — without having to chase each one individually. A new behavioral test (test_opentelemetry_wildcard_ignores_every_sub_logger) exercises the realsentry_sdkEventHandler._can_recordagainst five logger names (including a hypothetical future one) plus a non-OTel logger, pinning thefnmatchsemantics this depends on. This also matches the precedent set by the Datadog monitor for luthien-proxy errors, which already excludes@logger:opentelemetry.*for the same reason (telemetry-pipeline health is the collector's own monitors' job).Silences: LUTHIEN-C, LUTHIEN-F (new), plus the original LUTHIEN context-detach noise this PR already covered.