Skip to content

fix(telemetry): retain actionable provider failure evidence - #989

Merged
seonghobae merged 5 commits into
fix/provider-embedding-current-mainfrom
fix/capability-probe-actionable-telemetry
Sep 1, 2026
Merged

fix(telemetry): retain actionable provider failure evidence#989
seonghobae merged 5 commits into
fix/provider-embedding-current-mainfrom
fix/capability-probe-actionable-telemetry

Conversation

@seonghobae

@seonghobae seonghobae commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Outcome

Capability-probe failures now retain enough secret-free evidence to diagnose upstream 4xx responses instead of logging only a broad error type.

  • records provider status, stable error type, bounded sanitized error summary, provider-neutral model group, and fallback observation state in logs/spans
  • preserves actionable schema diagnostics such as the Azure json_object message contract
  • continues to reject prompts, message/content payload assignments, credentials, URLs, and IP addresses
  • does not infer LiteLLM internal fallback outcomes from unstructured exception text

Verification

  • 145 focused telemetry/provider error/orchestrator tests passed
  • compileall passed
  • git diff --check passed

Stacked on #970 because it uses that branch’s capability-probe path.


Devin Review

Signed-off-by: Seongho Bae <me@seonghobae.me>
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 2b3820fc-7ec3-4b17-9504-aa51a4c5b2f3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T02:47:56.320171Z bb98f0b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

devin-ai-integration[bot]

This comment was marked as resolved.

chatgpt-codex-connector[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head security follow-up ccbbaf13: telemetry now exports only a package-owned canonical JSON-object schema diagnostic; all other provider prose collapses to the stable error code. Quoted "messages": / "content": assignments, suffix/fallback prose, classified exception prose, prompts, bodies, URLs, credentials, and IPs remain non-exportable. Relevant suite: 147 passed; compileall and diff-check passed.

Signed-off-by: Seongho Bae <me@seonghobae.me>
@seonghobae

Copy link
Copy Markdown
Contributor Author

Independent exact-head security audit found and fixed two concrete gaps in 7d3601a9: (1) the real LiteLLM/Azure prefix (AzureException BadRequestError -) previously prevented the JSON-object contract from reaching the package-owned canonical diagnosis; matching now recognizes that bounded phrase anywhere but still exports only the fixed summary, never upstream prose, and (2) server.address no longer exports IPv4/IPv6 literals into OTLP failure spans. The provided failure shape now retains invalid_request_error, provider status 400, model group, and messages must mention json when response_format is json_object, while tests prove the prefix, fallback prose, caller text, URL, and IP are absent. Focused: 113 passed. Full suite: 2936 passed, 2 skipped. Python compile and git diff --check passed; Ruff is unavailable in this worktree (Failed to spawn: ruff).

@seonghobae
seonghobae merged commit d81be27 into fix/provider-embedding-current-main Sep 1, 2026
2 checks passed
@seonghobae
seonghobae deleted the fix/capability-probe-actionable-telemetry branch September 1, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant