Repository navigation
Conversation
Observability is the missing half of an agent harness: the audit log records what happened, but nothing emits it as a trace. Each model run is now an `invoke_agent` span with `chat <model>` and `execute_tool <name>` children, and the money path gets `txn_agent.purchase` / `txn_agent.checkout_cart` spans carrying the gate decision, its reasons, the outcome and the idempotency key. Approval verdicts get their own child span. The core stays stdlib-only: OpenTelemetry is imported lazily, so an unconfigured provider means every helper is a no-op, and tool arguments and prompts are never recorded as attributes.
Reviewer's GuideAPIAdd the optional tracing module and its public configuration, span, and attribute helpers for agent runs.
Uses of the APIInstrument checkout and single-offer purchase paths with gate, approval, outcome, and idempotency details.
Wrap model runs, chat turns, and tool calls in spans while recording model usage and run metadata.
Instrument tool execution while excluding arguments and marking fail-soft errors.
ConfigurationAdd the OpenTelemetry API extra and include its SDK in the development dependencies for testing.
DocumentationDocument the optional tracing setup, emitted spans, privacy behavior, and provider configuration.
TestsCover configured and no-op tracing, checkout and approval facts, tool errors, and model usage spans.
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 9 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="txn_agent/tools.py" line_range="128" />
<code_context>
+ # Every tool execution is a span: the arguments are deliberately NOT recorded
+ # (they can carry user data), only the tool name, the outcome and an error
+ # class when the call failed.
+ with telemetry.span(telemetry.tool_span_name(name), {
+ telemetry.ATTR_OPERATION: "execute_tool",
+ telemetry.ATTR_TOOL_NAME: name,
</code_context>
<issue_to_address>
**Sensitive details reach traces**
When tool arguments, gate reasons, or payment or checkout errors contain caller- or provider-supplied text, tool names, verticals, error text, gate reasons, and payment or checkout messages are copied into span attributes or events, so trace exporters retain dynamic details excluded by the privacy guarantees.
Record only fixed classifications and bounded facts in telemetry, omitting or sanitizing caller- and provider-supplied text.
Also at `txn_agent/tools.py:130-131`, `txn_agent/tools.py:141-142`, `txn_agent/agent.py:118`, `txn_agent/agent.py:134`, `txn_agent/agent.py:159-164`, `txn_agent/agent.py:210`, `txn_agent/agent.py:233`, `txn_agent/agent.py:238`.
</issue_to_address>
### Comment 2
<location path="txn_agent/agent.py" line_range="107" />
<code_context>
+ telemetry.ATTR_CART_ID: cart.cart_id,
+ telemetry.ATTR_MERCHANT: cart.merchant,
+ telemetry.ATTR_VERTICAL: cart.vertical.value,
+ telemetry.ATTR_CART_TOTAL: cart.total,
+ telemetry.ATTR_CART_LINES: len(cart.items),
+ }) as active:
</code_context>
<issue_to_address>
**Purchase traces report the wrong total**
When revalidation changes an offer price and the purchase proceeds, the purchase spans retain the original cart or quote total while checkout and payment use the revalidated live total, so traces report a different amount from the one charged.
Update the purchase and checkout span totals to use the revalidated live total.
Also at `txn_agent/agent.py:192`.
</issue_to_address>
### Comment 3
<location path="txn_agent/telemetry.py" line_range="115" />
<code_context>
+ except ImportError:
+ _TRACER = _NullTracer()
+ else:
+ _TRACER = trace.get_tracer(INSTRUMENTATION_NAME)
+ return _TRACER
+
</code_context>
<issue_to_address>
**Configured tracer gets overwritten**
When the first `_tracer()` lookup runs concurrently with `configure()` or `reset()`, `_tracer()` can assign `trace.get_tracer(...)` after `configure()` installs its tracer, so later spans go to the wrong provider and the configured exporter misses them.
Synchronize tracer initialization and configuration so an in-flight lookup cannot overwrite a configured tracer.
Also at `txn_agent/telemetry.py:121-122`.
</issue_to_address>
### Comment 4
<location path="tests/test_telemetry.py" line_range="201-203" />
<code_context>
+ assert chat.attributes[telemetry.ATTR_OUTPUT_TOKENS] == 30
+
+
+@pytest.mark.skipif(importlib.util.find_spec("opentelemetry") is not None,
+ reason="opentelemetry-api is installed; the no-op path is not the default")
+def test_span_is_a_noop_without_opentelemetry() -> None:
+ with telemetry.span("anything", {"a": 1}) as active:
+ active.set_attribute("b", 2)
</code_context>
<issue_to_address>
**No-op path goes untested**
When the test suite runs with the project's development dependencies installed, the OpenTelemetry package is installed, so the skip condition skips the no-op test and CI does not detect regressions in that path.
Run the no-op assertions in an environment or test setup that exercises the absent-SDK path even when OpenTelemetry is installed.
</issue_to_address>
### Comment 5
<location path="README.md" line_range="177" />
<code_context>
+
+```bash
+pip install 'txn-agent[otel]' # opentelemetry-api; bring your own exporter/SDK
+```
+
+Every agent run becomes a trace — `invoke_agent txn-agent` → `chat <model>` and
</code_context>
<issue_to_address>
**README setup snippet raises NameError**
When a reader runs the tracing setup snippet as shown, `txn_agent.telemetry.reset()` references `txn_agent` without importing or binding it, so the snippet raises `NameError` instead of resetting telemetry.
Import or otherwise bind `txn_agent` in the setup snippet before calling `txn_agent.telemetry.reset()`.
Also at `README.md:176`.
</issue_to_address>
### Comment 6
<location path="txn_agent/agent.py" line_range="159" />
<code_context>
payment_ref, charged = self._pay(live, key, live.cart_id)
except PaymentError as e:
- return Outcome(Status.FAILED, live.cart_id, cart=live, reasons=(f"payment: {e}",))
+ telemetry.add_event(active, "txn_agent.payment_failed", {"error": str(e)})
+ return done(Outcome(Status.FAILED, live.cart_id, cart=live, reasons=(f"payment: {e}",)))
try:
</code_context>
<issue_to_address>
**Failed purchases look like successful spans**
When a payment authorization or merchant checkout fails, `_checkout_cart` and `_purchase` catch payment and checkout exceptions and return normally from the span context; recording an exception only adds an event, so these failed purchase spans keep an unset status and standard OpenTelemetry error queries miss them.
Set the active span status to OpenTelemetry ERROR for failed payment and checkout outcomes, including payment failures that only add an event.
Also at `txn_agent/agent.py:164`, `txn_agent/agent.py:233`, `txn_agent/agent.py:238`.
</issue_to_address>
### Comment 7
<location path="txn_agent/telemetry.py" line_range="139-140" />
<code_context>
+ """
+ try:
+ ctx = _tracer().start_as_current_span(name)
+ except Exception: # pragma: no cover - defensive: a broken tracer is not fatal
+ yield _NullSpan()
+ return
+ with _ctx(ctx) as active:
</code_context>
<issue_to_address>
**Tracing failures disappear silently**
When a configured tracer raises while creating a span or writing attributes, `telemetry.span` substitutes a null span when span creation fails, and `set_attributes` silently skips failed writes; a configured but broken tracer therefore produces no useful trace or diagnostic signal, leaving operators unaware that tracing has stopped.
Keep tracing failures isolated from purchases but emit a rate-limited diagnostic or expose a failure callback so operators can detect broken tracing.
Also at `txn_agent/telemetry.py:179`.
</issue_to_address>
### Comment 8
<location path="txn_agent/llm.py" line_range="78" />
<code_context>
+ resp = client.messages.create(model=model, max_tokens=1024, system=SYSTEM_PROMPT,
+ messages=messages, tools=TOOL_SCHEMAS)
+ usage = getattr(resp, "usage", None)
+ if usage is not None:
+ telemetry.set_attributes(chat_span, {
+ telemetry.ATTR_INPUT_TOKENS: getattr(usage, "input_tokens", None),
</code_context>
<issue_to_address>
**OpenAI chat spans omit token usage**
When the agent uses the OpenAI-compatible adapter and the endpoint returns token usage, when the OpenAI-compatible endpoint returns usage, `LLMResponse` does not expose it and the adapter discards it, so this check sees `None` and the chat span has no input or output token attributes.
Add usage to `LLMResponse` and copy the endpoint's usage fields into it so the shared tracing path can record them.
</issue_to_address>
### Comment 9
<location path="tests/test_telemetry.py" line_range="102" />
<code_context>
+ assert span.attributes[telemetry.ATTR_OUTCOME] == "purchased"
+ assert span.attributes[telemetry.ATTR_MERCHANT] == "AppliancePartsCo"
+ # The key is a hash, not a cart dump: safe to put on a span.
+ assert len(str(span.attributes[telemetry.ATTR_IDEMPOTENCY_KEY])) == 32
+ # Decimals are stringified so the attribute type stays valid.
+ assert isinstance(span.attributes[telemetry.ATTR_CART_TOTAL], str)
</code_context>
<issue_to_address>
**Wrong idempotency key goes unnoticed**
When the span records any incorrect key that is still 32 characters long, `test_checkout_records_gate_decision_outcome_and_key` only checks the key's length on line 102, so a constant or otherwise incorrect 32-character value passes and the test misses incorrect telemetry.
Assert that the recorded attribute equals the idempotency key computed for the checkout cart.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 8 findings to address first, and with a configured exporter, traces include cart IDs, the idempotency key, and payment or checkout exception details, which may expose transaction information outside the service. Reverting stops future traces but cannot retract data already exported.
Blocking findings: txn_agent/tools.py:128, txn_agent/agent.py:107, txn_agent/telemetry.py:115, tests/test_telemetry.py:203, txn_agent/agent.py:159, and 3 more
| # Every tool execution is a span: the arguments are deliberately NOT recorded | ||
| # (they can carry user data), only the tool name, the outcome and an error | ||
| # class when the call failed. | ||
| with telemetry.span(telemetry.tool_span_name(name), { |
There was a problem hiding this comment.
🟡 Medium · Sensitive details reach traces
When tool arguments, gate reasons, or payment or checkout errors contain caller- or provider-supplied text, tool names, verticals, error text, gate reasons, and payment or checkout messages are copied into span attributes or events, so trace exporters retain dynamic details excluded by the privacy guarantees.
Record only fixed classifications and bounded facts in telemetry, omitting or sanitizing caller- and provider-supplied text.
Also at txn_agent/tools.py:130-131, txn_agent/tools.py:141-142, txn_agent/agent.py:118, txn_agent/agent.py:134, txn_agent/agent.py:159-164, txn_agent/agent.py:210, txn_agent/agent.py:233, txn_agent/agent.py:238.
Prompt for AI agents
In `txn_agent/tools.py` at line 128:
**Sensitive details reach traces**
When tool arguments, gate reasons, or payment or checkout errors contain caller- or provider-supplied text, tool names, verticals, error text, gate reasons, and payment or checkout messages are copied into span attributes or events, so trace exporters retain dynamic details excluded by the privacy guarantees.
Record only fixed classifications and bounded facts in telemetry, omitting or sanitizing caller- and provider-supplied text.
Also at `txn_agent/tools.py:130-131`, `txn_agent/tools.py:141-142`, `txn_agent/agent.py:118`, `txn_agent/agent.py:134`, `txn_agent/agent.py:159-164`, `txn_agent/agent.py:210`, `txn_agent/agent.py:233`, `txn_agent/agent.py:238`.| telemetry.ATTR_CART_ID: cart.cart_id, | ||
| telemetry.ATTR_MERCHANT: cart.merchant, | ||
| telemetry.ATTR_VERTICAL: cart.vertical.value, | ||
| telemetry.ATTR_CART_TOTAL: cart.total, |
There was a problem hiding this comment.
🟡 Medium · Purchase traces report the wrong total
When revalidation changes an offer price and the purchase proceeds, the purchase spans retain the original cart or quote total while checkout and payment use the revalidated live total, so traces report a different amount from the one charged.
Update the purchase and checkout span totals to use the revalidated live total.
Also at txn_agent/agent.py:192.
Prompt for AI agents
In `txn_agent/agent.py` at line 107:
**Purchase traces report the wrong total**
When revalidation changes an offer price and the purchase proceeds, the purchase spans retain the original cart or quote total while checkout and payment use the revalidated live total, so traces report a different amount from the one charged.
Update the purchase and checkout span totals to use the revalidated live total.
Also at `txn_agent/agent.py:192`.| except ImportError: | ||
| _TRACER = _NullTracer() | ||
| else: | ||
| _TRACER = trace.get_tracer(INSTRUMENTATION_NAME) |
There was a problem hiding this comment.
🟡 Medium · Configured tracer gets overwritten
When the first _tracer() lookup runs concurrently with configure() or reset(), _tracer() can assign trace.get_tracer(...) after configure() installs its tracer, so later spans go to the wrong provider and the configured exporter misses them.
Synchronize tracer initialization and configuration so an in-flight lookup cannot overwrite a configured tracer.
Also at txn_agent/telemetry.py:121-122.
Prompt for AI agents
In `txn_agent/telemetry.py` at line 115:
**Configured tracer gets overwritten**
When the first `_tracer()` lookup runs concurrently with `configure()` or `reset()`, `_tracer()` can assign `trace.get_tracer(...)` after `configure()` installs its tracer, so later spans go to the wrong provider and the configured exporter misses them.
Synchronize tracer initialization and configuration so an in-flight lookup cannot overwrite a configured tracer.
Also at `txn_agent/telemetry.py:121-122`.| @pytest.mark.skipif(importlib.util.find_spec("opentelemetry") is not None, | ||
| reason="opentelemetry-api is installed; the no-op path is not the default") | ||
| def test_span_is_a_noop_without_opentelemetry() -> None: |
There was a problem hiding this comment.
🟡 Medium · No-op path goes untested
When the test suite runs with the project's development dependencies installed, the OpenTelemetry package is installed, so the skip condition skips the no-op test and CI does not detect regressions in that path.
Run the no-op assertions in an environment or test setup that exercises the absent-SDK path even when OpenTelemetry is installed.
Prompt for AI agents
In `tests/test_telemetry.py` at lines 201-203:
**No-op path goes untested**
When the test suite runs with the project's development dependencies installed, the OpenTelemetry package is installed, so the skip condition skips the no-op test and CI does not detect regressions in that path.
Run the no-op assertions in an environment or test setup that exercises the absent-SDK path even when OpenTelemetry is installed.|
|
||
| trace.set_tracer_provider(TracerProvider()) # your exporter of choice | ||
| txn_agent.telemetry.reset() # re-detect the configured provider | ||
| ``` |
There was a problem hiding this comment.
⚪ Low · README setup snippet raises NameError
When a reader runs the tracing setup snippet as shown, txn_agent.telemetry.reset() references txn_agent without importing or binding it, so the snippet raises NameError instead of resetting telemetry.
Import or otherwise bind txn_agent in the setup snippet before calling txn_agent.telemetry.reset().
Also at README.md:176.
Prompt for AI agents
In `README.md` at line 177:
**README setup snippet raises NameError**
When a reader runs the tracing setup snippet as shown, `txn_agent.telemetry.reset()` references `txn_agent` without importing or binding it, so the snippet raises `NameError` instead of resetting telemetry.
Import or otherwise bind `txn_agent` in the setup snippet before calling `txn_agent.telemetry.reset()`.
Also at `README.md:176`.| payment_ref, charged = self._pay(live, key, live.cart_id) | ||
| except PaymentError as e: | ||
| return Outcome(Status.FAILED, live.cart_id, cart=live, reasons=(f"payment: {e}",)) | ||
| telemetry.add_event(active, "txn_agent.payment_failed", {"error": str(e)}) |
There was a problem hiding this comment.
🟡 Medium · Failed purchases look like successful spans
When a payment authorization or merchant checkout fails, _checkout_cart and _purchase catch payment and checkout exceptions and return normally from the span context; recording an exception only adds an event, so these failed purchase spans keep an unset status and standard OpenTelemetry error queries miss them.
Set the active span status to OpenTelemetry ERROR for failed payment and checkout outcomes, including payment failures that only add an event.
Also at txn_agent/agent.py:164, txn_agent/agent.py:233, txn_agent/agent.py:238.
Prompt for AI agents
In `txn_agent/agent.py` at line 159:
**Failed purchases look like successful spans**
When a payment authorization or merchant checkout fails, `_checkout_cart` and `_purchase` catch payment and checkout exceptions and return normally from the span context; recording an exception only adds an event, so these failed purchase spans keep an unset status and standard OpenTelemetry error queries miss them.
Set the active span status to OpenTelemetry ERROR for failed payment and checkout outcomes, including payment failures that only add an event.
Also at `txn_agent/agent.py:164`, `txn_agent/agent.py:233`, `txn_agent/agent.py:238`.| except Exception: # pragma: no cover - defensive: a broken tracer is not fatal | ||
| yield _NullSpan() |
There was a problem hiding this comment.
🟡 Medium · Tracing failures disappear silently
When a configured tracer raises while creating a span or writing attributes, telemetry.span substitutes a null span when span creation fails, and set_attributes silently skips failed writes; a configured but broken tracer therefore produces no useful trace or diagnostic signal, leaving operators unaware that tracing has stopped.
Keep tracing failures isolated from purchases but emit a rate-limited diagnostic or expose a failure callback so operators can detect broken tracing.
Also at txn_agent/telemetry.py:179.
Prompt for AI agents
In `txn_agent/telemetry.py` at lines 139-140:
**Tracing failures disappear silently**
When a configured tracer raises while creating a span or writing attributes, `telemetry.span` substitutes a null span when span creation fails, and `set_attributes` silently skips failed writes; a configured but broken tracer therefore produces no useful trace or diagnostic signal, leaving operators unaware that tracing has stopped.
Keep tracing failures isolated from purchases but emit a rate-limited diagnostic or expose a failure callback so operators can detect broken tracing.
Also at `txn_agent/telemetry.py:179`.| resp = client.messages.create(model=model, max_tokens=1024, system=SYSTEM_PROMPT, | ||
| messages=messages, tools=TOOL_SCHEMAS) | ||
| usage = getattr(resp, "usage", None) | ||
| if usage is not None: |
There was a problem hiding this comment.
🟡 Medium · OpenAI chat spans omit token usage
When the agent uses the OpenAI-compatible adapter and the endpoint returns token usage, when the OpenAI-compatible endpoint returns usage, LLMResponse does not expose it and the adapter discards it, so this check sees None and the chat span has no input or output token attributes.
Add usage to LLMResponse and copy the endpoint's usage fields into it so the shared tracing path can record them.
Prompt for AI agents
In `txn_agent/llm.py` at line 78:
**OpenAI chat spans omit token usage**
When the agent uses the OpenAI-compatible adapter and the endpoint returns token usage, when the OpenAI-compatible endpoint returns usage, `LLMResponse` does not expose it and the adapter discards it, so this check sees `None` and the chat span has no input or output token attributes.
Add usage to `LLMResponse` and copy the endpoint's usage fields into it so the shared tracing path can record them.| assert span.attributes[telemetry.ATTR_OUTCOME] == "purchased" | ||
| assert span.attributes[telemetry.ATTR_MERCHANT] == "AppliancePartsCo" | ||
| # The key is a hash, not a cart dump: safe to put on a span. | ||
| assert len(str(span.attributes[telemetry.ATTR_IDEMPOTENCY_KEY])) == 32 |
There was a problem hiding this comment.
🟡 Medium · Wrong idempotency key goes unnoticed
When the span records any incorrect key that is still 32 characters long, test_checkout_records_gate_decision_outcome_and_key only checks the key's length on line 102, so a constant or otherwise incorrect 32-character value passes and the test misses incorrect telemetry.
Assert that the recorded attribute equals the idempotency key computed for the checkout cart.
Prompt for AI agents
In `tests/test_telemetry.py` at line 102:
**Wrong idempotency key goes unnoticed**
When the span records any incorrect key that is still 32 characters long, `test_checkout_records_gate_decision_outcome_and_key` only checks the key's length on line 102, so a constant or otherwise incorrect 32-character value passes and the test misses incorrect telemetry.
Assert that the recorded attribute equals the idempotency key computed for the checkout cart.
What this adds
Observability is the missing half of an agent harness: the audit log records what happened, but nothing emits it as a trace, so an agent run is invisible in Grafana/Jaeger/Tempo.
invoke_agent txn-agent, withgen_ai.operation.name,gen_ai.agent.nameandgen_ai.request.model.chat <model>spans carrygen_ai.usage.input_tokens/output_tokens;execute_tool <name>spans carrygen_ai.tool.nameand an error attribute when the router fails soft.txn_agent.purchaseandtxn_agent.checkout_cartcarry the gate decision, its reasons, cart total, outcome and the idempotency key;txn_agent.approvalrecords the verdict.txn-agent[otel](opentelemetry-api). Nothing is imported at module load, so the stdlib-only core invariant holds and an unconfigured provider means every helper is a no-op.Why the attribute names live in one file
txn_agent/telemetry.pycentralises every span and attribute name. The GenAI semantic conventions are still moving, so when they rename something the fix is one file, not a search across the package.Verification
ruff check .cleanmypy txn_agent— strict, no issues (20 files)pytest— 74 passed, 1 skipped, coverage 87.36% against the 85% floorInMemorySpanExporter:chatandexecute_toolspans reportinvoke_agent txn-agentas their parent, and a purchase recordstxn_agent.gate.decision=auto_buy,txn_agent.outcome=purchasedand a 32-char idempotency key.🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com
Summary by Sourcery
Add optional, privacy-conscious OpenTelemetry tracing across agent execution and transaction workflows.
New Features:
Enhancements:
Build:
txn-agent[otel]optional dependency and include the OpenTelemetry SDK in development dependencies.Documentation:
Tests:
Chores: