Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions docs/en/docs/how-to/trace-with-phoenix.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,12 +91,6 @@ PowerContext configures inference instrumentation to exclude content. Spans carr
durations, and error categories. Prompts, model responses, Memory content, and vectors are excluded, and message
attributes record only the shape of each message rather than its text.

One exception applies to generation. When a model returns output that does not satisfy the requested schema,
Pydantic AI retries with feedback that quotes the model's own invalid output, and it records that feedback in the
`gen_ai.input.messages` and `pydantic_ai.all_messages` attributes regardless of the content setting. For Memory
extraction, that quoted output is the proposed Memory content. Treat the tracing backend as a system that may receive
model output on this retry path, and restrict access to it accordingly.

## Stop Phoenix

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/en/docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ that do not use the `powercontext` command may omit the `cli` extra.

Enabling tracing also produces spans for the generation and embedding calls that PowerContext constructs, without
recording prompts, model responses, Memory content, or vectors. See
[Trace with Phoenix](../how-to/trace-with-phoenix.md) for a working configuration and the one documented exception.
[Trace with Phoenix](../how-to/trace-with-phoenix.md) for a working configuration.

To use OceanBase, provide its URL through your environment or secret manager:

Expand Down
5 changes: 0 additions & 5 deletions docs/zh/docs/how-to/trace-with-phoenix.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,11 +87,6 @@ span 是批量导出的,刷新前请稍等几秒。MCP 请求会用 `MCP mcp.t
PowerContext 在配置推理 instrumentation 时关闭了内容记录。span 只携带模型标识、token 用量、耗时和错误类别;
prompt、模型响应、Memory 内容和向量都不会被导出,消息类属性只记录每条消息的结构,不记录正文。

generation 有一个例外。当模型返回的输出不满足所要求的 schema 时,Pydantic AI 会带着反馈重试,而这段反馈里
引用了模型自己的非法输出;无论内容开关如何设置,它都会被写入 `gen_ai.input.messages` 和
`pydantic_ai.all_messages` 属性。对 Memory extraction 来说,被引用的那段输出就是候选的 Memory 内容。
因此应把 tracing 后端视为在这条重试路径上可能收到模型输出的系统,并相应限制其访问权限。

## 停止 Phoenix

```bash
Expand Down
3 changes: 1 addition & 2 deletions docs/zh/docs/reference/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,7 @@ OpenTelemetry 环境变量进行配置。不使用 `powercontext` command 的 pr
`cli` extra。

启用 tracing 后,PowerContext 自己构造的 generation 与 embedding 调用也会产生 span,且不记录 prompt、模型响应、
Memory 内容或向量。可运行的配置和唯一一处已记录的例外见
[用 Phoenix 查看 trace](../how-to/trace-with-phoenix.md)。
Memory 内容或向量。可运行的配置见 [用 Phoenix 查看 trace](../how-to/trace-with-phoenix.md)。

使用 OceanBase 时,通过环境或 secret manager 提供 URL:

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ classifiers = [
builtin = [
"aiosqlite>=0.22,<1",
"apscheduler>=3.11,<4",
"pydantic-ai-slim[anthropic,openai]>=2.14.1,<3",
"pydantic-ai-slim[anthropic,openai]>=2.27.1,<3",
"pydantic-settings>=2.7,<3",
"pyobvector>=0.2.28,<0.3",
"sqlalchemy[asyncio]>=2,<3",
Expand Down
57 changes: 56 additions & 1 deletion tests/builtin/inference/test_pydantic_ai.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from pydantic_ai.exceptions import ModelHTTPError
from pydantic_ai.messages import ModelMessage, ModelResponse, TextPart
from pydantic_ai.models.function import AgentInfo, FunctionModel
from pydantic_ai.models.instrumented import InstrumentationSettings
from pydantic_ai.models.instrumented import InstrumentationSettings, InstrumentedModel
from pydantic_ai.models.test import TestModel
from pydantic_ai.usage import RequestUsage

Expand Down Expand Up @@ -52,6 +52,19 @@ class Answer:
value: str


# Nested on purpose: retry feedback only carries the raw model output when the validation
# error location is longer than one element, which a flat output type cannot produce.
@dataclass(frozen=True, slots=True)
class Candidate:
text: str
intent: str


@dataclass(frozen=True, slots=True)
class Proposal:
candidates: tuple[Candidate, ...]


TEST_PROFILE = EmbeddingProfile(
profile_id="test-v1",
model="test:test",
Expand Down Expand Up @@ -271,6 +284,48 @@ async def scenario() -> None:
asyncio.run(scenario())


def test_instrumented_generation_spans_exclude_schema_retry_content() -> None:
exporter = InMemorySpanExporter()
provider = TracerProvider(shutdown_on_exit=False)
provider.add_span_processor(SimpleSpanProcessor(exporter))
instrumentation = InstrumentationSettings(
tracer_provider=provider,
include_content=False,
include_binary_content=False,
include_model_request_parameters=False,
)
# The first response drops the required `intent`, so the retry feedback quotes it back.
pending = [
'{"candidates":[{"text":"traveler prefers aisle seats"}]}',
'{"candidates":[{"text":"redacted","intent":"add"}]}',
]

async def reply(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
del messages, info
return ModelResponse(parts=[TextPart(pending.pop(0))])

async def scenario() -> None:
generator = PydanticAIStructuredGenerator(
model=InstrumentedModel(FunctionModel(reply), instrumentation),
instructions="Propose candidates.",
input_type=Question,
output_type=Proposal,
)

result = await generator.generate(Question("bounded evidence"))

assert result.output.candidates[0].intent == "add"

asyncio.run(scenario())

spans = exporter.get_finished_spans()
# Both responses consumed and two chat spans recorded prove the retry path ran.
assert not pending
assert len([span for span in spans if span.name.startswith("chat ")]) == 2
for span in spans:
assert "traveler prefers aisle seats" not in str(span.attributes)


def test_embedding_adapter_returns_validated_vectors_and_usage() -> None:
async def scenario() -> None:
model = PydanticAIEmbeddingModel(
Expand Down
22 changes: 12 additions & 10 deletions uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.