Skip to content

Add observability tracing for callback agent via Braintrust proxy #58

Description

@fyang0507

Problem

When the callback agent runs headless via sundial, there's no visibility into what it's doing — no trace of API calls, tool usage, latency, or token consumption. The only way to inspect a session today is to manually read the Claude session JSONL file after the fact.

Proposed solution: Braintrust AI proxy

Braintrust provides an AI proxy that intercepts LLM API calls, logs full request/response traces, and forwards to the provider. Claude Code respects ANTHROPIC_BASE_URL, so routing through Braintrust requires zero code changes — just environment variables.

Verified behavior

# Claude Code reads ANTHROPIC_BASE_URL
ANTHROPIC_BASE_URL=test claude -p "hi"
# → API Error: "test/v1/messages?beta=true" cannot be parsed as a URL.

Implementation (config-only)

callback_command: "ANTHROPIC_BASE_URL=https://braintrustproxy.com/v1 claude --dangerously-skip-permissions -p"

With BRAINTRUST_API_KEY either inline or in the shell profile (sundial runs commands via /bin/zsh -l -c, so login profile env vars are inherited).

What you get in Braintrust dashboard

  • Full API request/response for every LLM call
  • Tool call traces (what the agent read, what commands it ran)
  • Token usage per turn
  • Latency breakdown
  • Session-level grouping

Alternatives considered

Approach Pros Cons
Braintrust proxy via ANTHROPIC_BASE_URL Zero code changes, config-only, full API tracing Requires Braintrust account, adds network hop
Claude Code hooks (--include-hook-events) Built-in, no external service Only lifecycle events (tool use, session start), not API-level traces
Post-hoc JSONL parsing No dependencies Manual, no dashboard, no real-time visibility
Claude Code plugin Could be rich No Braintrust plugin exists in official marketplace yet

Scope

This is a config/documentation change — document the Braintrust proxy pattern in the watch config comments and potentially in the skills docs. No code changes to outreach-cli needed.

Relationship

Complements #57 (cost analysis) — Braintrust would give real-time cost tracking per callback rather than manual JSONL parsing.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions