Skip to content

fix(logging): redact OpenTelemetry setup exception telemetry - #1713

Draft
seonghobae wants to merge 3 commits into
fix/structured-exception-telemetryfrom
fix/otel-exception-telemetry
Draft

seonghobae wants to merge 3 commits into
fix/structured-exception-telemetryfrom
fix/otel-exception-telemetry

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Refs #1698.

Current authority

Verified finding

setup_telemetry() retained logger.exception("OpenTelemetry setup failed; continuing without tracing."). Python logging renders the current exception information on that path. OTLP/exporter/instrumentation failures may carry connection strings, provider response text, tokens, or internal paths, so the original value and traceback can enter the ordinary application log. This is a bounded CWE-532 / OWASP Logging Cheat Sheet confidentiality gap, not a telemetry architecture rewrite.

RED → minimal causal repair

Test-first commit d7289799e2661ad3ebbf5c309579ec8bdc45738b injects a secret-bearing RuntimeError at the real OpenTelemetry import boundary and formats the resulting log record. It requires the operation label, exception type, and exception_fingerprint= while rejecting a token-shaped value, PostgreSQL connection string, internal path, and traceback header. The predecessor logger.exception() behavior cannot satisfy this contract.

Production repair 96446d87fdbf59de1868a55f01be999990d31df8 consumes #1700's core.safe_logging.redacted_exception_info() helper. Log severity and the fixed operation message remain unchanged; raw exception value/traceback are no longer attached. Application behavior still fails open only with respect to tracing itself: telemetry setup failure continues without tracing exactly as before.

Doctoring 76d54e20a020fb96e31554fd43fa3de2bf5053af records scope, rejected alternatives, evidence limits, and traceability to Python 3.14 logging, CWE-532, and OWASP Logging Cheat Sheet. This PR intentionally does not alter the existing OTEL bootstrap environment-variable boundary or create a second sanitizer.

Independent review

CodeRabbit reviewed exact unchanged range 2d6800aea110ac04eeedcd5faa91479f62877408 → 76d54e20a020fb96e31554fd43fa3de2bf5053af and submitted formal APPROVED review PRR_kwDOSNjZ2s8AAAABN4zxWA at 2026-09-16T18:48:02Z. Fresh review-thread inventory is empty. This satisfies the current-head independent-review lane only while the head remains unchanged; it does not substitute for hosted required contexts.

Evidence boundary

Fresh exact-head repository workflow lookup for 76d54e20... still returns zero PR-triggered repository workflows despite the material three-file delta and exact valid stacked parent. This is another #1691 stacked-admission RED, not hosted GREEN. Parent #1700 receipts/reviews do not transfer.

A focused source-equivalent regression harness previously reproduced the predecessor disclosure and passed after the bounded-helper repair, but this connector session has not executed the full repository test suite or a hosted CI job. Keep that distinction explicit.

Keep Draft until #1700 reaches valid protected ancestry and this unchanged effective delta obtains every then-live required context terminal-success with zero valid findings/threads and qualifying post-last-push independent review. No temporary develop retarget, workflow copy, dummy/no-op evidence commit, synthetic status, self-approval, force push, destructive rebase, or gate weakening.

Summary by CodeRabbit

  • Bug Fixes

    • Improved telemetry setup error logging to prevent sensitive credentials, tokens, and internal file paths from appearing in logs.
    • Applications continue operating without tracing when telemetry setup fails.
  • Documentation

    • Added documentation describing secure telemetry failure logging and its verification coverage.

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Advanced

Run ID: 07c4e732-3956-479c-b4f8-caa04b529431

📥 Commits

Reviewing files that changed from the base of the PR and between 2d6800a and 76d54e2.

📒 Files selected for processing (3)
  • backend/core/telemetry.py
  • backend/tests/test_telemetry_exception_redaction.py
  • docs/doctoring/otel-exception-telemetry.md

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The OpenTelemetry setup failure path now logs redacted exception details instead of raw tracebacks. A regression test checks sensitive-value exclusion and unchanged application state. Documentation records the contract, verification boundary, references, and integration gate.

Changes

Telemetry exception redaction

Layer / File(s) Summary
Redacted setup logging
backend/core/telemetry.py
The setup failure handler passes the exception to redacted_exception_info() through logger.error, replacing logger.exception().
Regression coverage and documentation
backend/tests/test_telemetry_exception_redaction.py, docs/doctoring/otel-exception-telemetry.md
The test verifies exception type and fingerprint logging without secrets, connection strings, paths, or tracebacks. The application state remains unconfigured. The documentation records the logging contract and integration conditions.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 76d54

The telemetry setup failure path retains bounded diagnostic information without writing exception secrets or tracebacks to application logs. No actionable current-head risk remains.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: redacting OpenTelemetry setup exception data from logs.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/otel-exception-telemetry

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review exact base 2d6800aea110ac04eeedcd5faa91479f62877408 → exact head 76d54e20a020fb96e31554fd43fa3de2bf5053af. Focus on whether the formatter-level regression actually falsifies predecessor logger.exception() disclosure, whether redacted_exception_info() preserves bounded RCA evidence without raw exception value/traceback, and whether this 3-file slice changes any OpenTelemetry setup behavior beyond logging confidentiality.

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='2d6800aea110ac04eeedcd5faa91479f62877408'
head='76d54e20a020fb96e31554fd43fa3de2bf5053af'

echo '== Commit topology =='
git merge-base "$base" "$head"
git rev-list --left-right --count "$base...$head"
git diff --name-status "$base" "$head"

echo '== Exact diff =='
git diff --find-renames --find-copies --no-ext-diff "$base" "$head" -- \
  backend/core/telemetry.py \
  backend/tests/test_telemetry_exception_redaction.py \
  docs/doctoring/otel-exception-telemetry.md

echo '== telemetry module outline =='
ast-grep outline backend/core/telemetry.py --items all --view expanded || true

echo '== safe_logging candidates =='
fd -a -i 'safe_logging' backend || true
rg -n -C 5 '\bdef\s+redacted_exception_info\b|\bredacted_exception_info\s*\(' backend/core

echo '== Relevant source ranges =='
sed -n '1,280p' backend/core/telemetry.py
safe_file=$(fd -a -i 'safe_logging.py' backend | head -n1 || true)
if [ -n "$safe_file" ]; then
  echo "== $safe_file =="
  ast-grep outline "$safe_file" --items all --view expanded || true
  sed -n '1,320p' "$safe_file"
fi
echo '== Regression test =='
cat -n backend/tests/test_telemetry_exception_redaction.py
echo '== Doctoring =='
cat -n docs/doctoring/otel-exception-telemetry.md

Length of output: 24083


🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='2d6800aea110ac04eeedcd5faa91479f62877408'
head='76d54e20a020fb96e31554fd43fa3de2bf5053af'
repo='ContextualWisdomLab/naruon'
pr='1713'

echo '== Static patch checks =='
git diff --check "$base" "$head"
git diff --word-diff=porcelain "$base" "$head" -- backend/core/telemetry.py

echo '== PR head and merge state =='
gh pr view "$pr" --repo "$repo" \
  --json number,state,isDraft,headRefOid,baseRefOid,mergeStateStatus,reviewDecision,statusCheckRollup \
  --jq '{number,state,isDraft,headRefOid,baseRefOid,mergeStateStatus,reviewDecision,statusCheckRollup: [.statusCheckRollup[] | {name,conclusion,status}]}' || true

echo '== Reviews and review threads =='
gh api "repos/$repo/pulls/$pr/reviews" --paginate \
  --jq '.[] | {user: .user.login, state, commit_id, submitted_at}' || true
gh api graphql -F owner='ContextualWisdomLab' -F name='naruon' -F number="$pr" -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes { isResolved comments(first: 10) { nodes { author { login } body path originalLine } } }
      }
    }
  }
}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false)' || true

Length of output: 2776


@seonghobae: Static review of exact base 2d6800aea110ac04eeedcd5faa91479f62877408 to exact head 76d54e20a020fb96e31554fd43fa3de2bf5053af found no actionable issues.

  • The regression test falsifies the predecessor behavior. logger.exception() attaches the active exception information. Formatting its LogRecord would render the injected RuntimeError value and traceback. The test rejects each injected secret value and the traceback header after formatter execution. The predecessor cannot satisfy these assertions.
  • redacted_exception_info(error) preserves bounded RCA evidence. It emits the original exception class name and a 16-character SHA-256-derived failure-site fingerprint. It creates a new RuntimeError and returns None for the traceback. It does not attach error, str(error), repr(error), or the original traceback to the log record.
  • The setup_telemetry() control flow is unchanged. The edit retains the enablement checks, endpoint validation, OTLP configuration, instrumentation calls, success-state assignment, error level, fixed operation message, and fail-open behavior for tracing setup failures. The only production behavior change is exception-log confidentiality.
  • git diff --check reported no patch whitespace errors.

I did not execute repository tests. GitHub currently reports this PR as Draft. The queried status rollup has no terminal conclusions, and no submitted reviews or unresolved review threads were returned. The required exact-head hosted checks and independent review remain pending.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Copy link
Copy Markdown
Contributor Author

Focused source-equivalent RED/GREEN receipt (not hosted merge evidence): I materialized the exact #1700 helper + #1713 telemetry/test contents into an isolated local Python harness and ran only test_telemetry_exception_redaction.py with warnings-as-errors. The repaired source passed 1/1 in 0.17s. Replacing only the exception handler with the predecessor logger.exception(...) implementation reproduced 1 failure: the rendered record contained Traceback, the injected token, PostgreSQL connection string, and internal path, and lacked exception_type=RuntimeError/fingerprint evidence.

This receipt is deliberately scoped: it validates the causal formatter contract, not the full repository checkout, hosted required contexts, dependency lock, or merge authority. Exact-head workflow inventory remains zero because this is a stacked PR. CodeRabbit selected the exact 3-file range but reported its review-capacity limit, so no formal independent approval is claimed.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

Please review the unchanged exact range 2d6800aea110ac04eeedcd5faa91479f6287740876d54e20a020fb96e31554fd43fa3de2bf5053af. The earlier request was rate-limited before a formal review was produced. Focus on the three-file logging-confidentiality slice only: formatter-level falsification of predecessor logger.exception() disclosure, preservation of OpenTelemetry setup/fail-open tracing behavior, and whether redacted_exception_info() leaves enough bounded RCA evidence without raw exception values or traceback.

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor
🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base='2d6800aea110ac04eeedcd5faa91479f62877408'
head='76d54e20a020fb96e31554fd43fa3de2bf5053af'

echo '== Topology and changed paths =='
git merge-base "$base" "$head"
git rev-list --left-right --count "$base...$head"
git diff --name-status "$base" "$head"
git diff --check "$base" "$head"

echo '== Exact 3-file diff =='
git diff --find-renames --find-copies --no-ext-diff "$base" "$head" -- \
  backend/core/telemetry.py \
  backend/tests/test_telemetry_exception_redaction.py \
  docs/doctoring/otel-exception-telemetry.md

echo '== telemetry.py outline and relevant content =='
ast-grep outline backend/core/telemetry.py --items all --view expanded || true
sed -n '1,320p' backend/core/telemetry.py

echo '== safe_logging definition and all call contexts =='
safe_file=$(fd -a -i 'safe_logging.py' backend | head -n1 || true)
if [ -n "$safe_file" ]; then
  echo "== $safe_file outline =="
  ast-grep outline "$safe_file" --items all --view expanded || true
  sed -n '1,360p' "$safe_file"
fi
rg -n -C 4 '\bredacted_exception_info\s*\(' backend --glob '*.py'

echo '== Regression test =='
cat -n backend/tests/test_telemetry_exception_redaction.py

echo '== Doctoring =='
cat -n docs/doctoring/otel-exception-telemetry.md

echo '== Prior handler and current handler context =='
git show "$base:backend/core/telemetry.py" | sed -n '1,320p'
echo '--- current ---'
git show "$head:backend/core/telemetry.py" | sed -n '1,320p'

Length of output: 36684


@seonghobae: I reviewed the exact range 2d6800aea110ac04eeedcd5faa91479f6287740876d54e20a020fb96e31554fd43fa3de2bf5053af. I found no actionable issues in the three-file slice.

  • The regression test falsifies the predecessor behavior at the formatter boundary. The predecessor logger.exception() attaches the active exception information. logging.Formatter.format() then renders the original RuntimeError value and traceback. The test formats the captured records and rejects the injected token, PostgreSQL connection string, internal path, and traceback header. The predecessor cannot satisfy these assertions.

  • redacted_exception_info(error) preserves bounded RCA evidence. It creates a new RuntimeError whose message contains exception_type=RuntimeError and a 16-character SHA-256-derived exception_fingerprint. It returns None for the traceback. It does not attach the original exception object, exception value, or traceback to the log record.

  • setup_telemetry() keeps its existing behavior. The enablement checks, endpoint validation, importer and exporter setup, instrumentation calls, success-state assignment, error severity, and fixed operation message are unchanged. A setup failure still returns normally and continues without tracing. The production change only replaces raw exception logging with redacted exception telemetry.

  • The exact diff changes only the stated three files. git diff --check reported no whitespace errors.

I did not execute repository code or tests in this review environment. The focused local receipt in the discussion supports the formatter-level causal contract, but it does not replace exact-head hosted checks or an independent review.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@seonghobae seonghobae added bug Something isn't working priority: high High-priority or P1 work labels Sep 19, 2026 — with ChatGPT Codex Connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: high High-priority or P1 work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant