Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 64 additions & 0 deletions docs/mkdocs/en/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Models in tRPC-Agent have the following core features:
- **Multimodal capabilities**: Supports multimodal content processing including text, images, etc. (e.g., Hunyuan multimodal models)
- **Prompt Cache support**: Provides unified prompt cache configuration across OpenAI, Anthropic, and LiteLLM routes to reduce repeated input cost for long prompts and multi-turn conversations
- **Model retry support**: Supports configuring retry at the model layer. The SDK automatically retries when exceptions such as rate limits occur, and backs off using an exponential backoff strategy
- **Provider metadata extraction**: Allows OpenAI-compatible provider-specific response fields to be allowlisted and attached to model responses and tracing spans
- **Extensible configuration**: Supports custom configuration options such as GenerateContentConfig, HttpOptions, client_args to meet various scenario requirements

## Quick Start
Expand Down Expand Up @@ -167,6 +168,69 @@ see the OpenAI reasoning guide for model-specific support. The SDK deliberately
`thinking_budget` to `effort`; it only requests `reasoning.summary` when thinking is enabled so the
reasoning output stays readable.

#### Extracting Provider-Specific Response Fields

Some OpenAI-compatible providers add proprietary fields to the response, such as a request or tracing
identifier. Configure `response_metadata_extractor` to explicitly allowlist and normalize the fields
that should be exposed to application code and tracing:

```python
from typing import Any

from trpc_agent_sdk.models import OpenAIModel


def extract_provider_metadata(
response_data: dict[str, Any],
) -> dict[str, Any] | None:
some_marker = response_data.get("some_marker")
if not isinstance(some_marker, dict):
return None
some_field = some_marker.get("some_field")
if not isinstance(some_field, str) or not some_field:
return None
return {
"some_marker": {
"some_field": some_field,
},
}


model = OpenAIModel(
model_name="your-model",
api_key="your-api-key",
base_url="https://your-openai-compatible-endpoint/v1",
response_metadata_extractor=extract_provider_metadata,
)
```

The callback receives the complete `model_dump()` dictionary for the provider response. It may read
top-level or nested fields, but should return only the small, JSON-serializable values required by the
application. Returning `None` means that no metadata was extracted. Invalid return values and callback
exceptions are logged and ignored without failing the model call.

For streaming requests, the extractor is invoked exactly once with the first response event to avoid
per-token overhead. The compatible provider must therefore include its metadata in the first event.

The extracted value is stored under the stable `provider_response_metadata` namespace:

```python
from trpc_agent_sdk.models import PROVIDER_RESPONSE_METADATA

async for event in runner.run_async(...):
provider_metadata = (event.custom_metadata or {}).get(
PROVIDER_RESPONSE_METADATA
)
if provider_metadata:
print(provider_metadata)
```

The same metadata is also attached to the model invocation tracing span. Do not return the full raw
provider response from the extractor, because it may contain sensitive or unnecessarily large data.

For a complete runnable example, see
[examples/llmagent_with_model_extra_fields](../../../examples/llmagent_with_model_extra_fields/README.md).

#### Advanced Usage

Since version `1.1.10`, `OpenAIModel` supports passing a shared HTTP client provider to enable connection reuse. By default, `OpenAIModel` creates a temporary HTTP client for each model-service request. If you want to reuse connections, use the following configuration:
Expand Down
62 changes: 62 additions & 0 deletions docs/mkdocs/zh/model.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ tRPC-Agent 内的模型具有以下核心特性:
- **多模态能力**:支持文本、图像等多模态内容处理(如 hunyuan 多模态模型)
- **Prompt Cache 支持**:支持跨 OpenAI、Anthropic 与 LiteLLM 路由的统一 prompt cache 配置,降低长提示词和多轮会话的重复输入成本
- **模型重试支持**:支持在模型层配置重试,SDK 将在限流等异常发生时自动重试,并按指数退避策略进行退避
- **Provider 元数据提取**:支持将 OpenAI 兼容服务返回的厂商扩展字段加入白名单,并附加到模型响应和 tracing span
- **可扩展配置**:支持 GenerateContentConfig、HttpOptions、client_args 等自定义配置项,满足不同场景需求

## 快速上手
Expand Down Expand Up @@ -164,6 +165,67 @@ model = OpenAIModel(
具体以 OpenAI reasoning 指南及模型文档为准。SDK 刻意不做 `thinking_budget` → `effort` 的映射;
仅在开启 thinking 时请求 `reasoning.summary`,保证推理输出可读。

#### 提取厂商响应扩展字段

部分 OpenAI 兼容服务会在响应中增加请求标识、链路标识等厂商扩展字段。可以配置
`response_metadata_extractor`,明确选择并规范化允许暴露给业务代码和 tracing 的字段:

```python
from typing import Any

from trpc_agent_sdk.models import OpenAIModel


def extract_provider_metadata(
response_data: dict[str, Any],
) -> dict[str, Any] | None:
some_marker = response_data.get("some_marker")
if not isinstance(some_marker, dict):
return None
some_field = some_marker.get("some_field")
if not isinstance(some_field, str) or not some_field:
return None
return {
"some_marker": {
"some_field": some_field,
},
}


model = OpenAIModel(
model_name="your-model",
api_key="your-api-key",
base_url="https://your-openai-compatible-endpoint/v1",
response_metadata_extractor=extract_provider_metadata,
)
```

回调接收厂商响应完整的 `model_dump()` 字典,可以读取顶层或嵌套字段,但应只返回业务所需的、
体积较小且可 JSON 序列化的数据。返回 `None` 表示没有提取到元数据。返回值非法或回调抛出异常时,
SDK 会记录日志并忽略该元数据,不会导致模型调用失败。

对于流式请求,为避免产生逐 token 开销,提取器只会使用第一个响应事件调用一次。因此兼容服务
必须在首个事件中携带需要提取的厂商字段。

提取结果存放在稳定的 `provider_response_metadata` 命名空间中:

```python
from trpc_agent_sdk.models import PROVIDER_RESPONSE_METADATA

async for event in runner.run_async(...):
provider_metadata = (event.custom_metadata or {}).get(
PROVIDER_RESPONSE_METADATA
)
if provider_metadata:
print(provider_metadata)
```

同一份元数据也会附加到模型调用的 tracing span。不要从提取器返回完整原始响应,以免泄漏敏感信息
或引入不必要的大体积数据。

完整可运行示例参见
[examples/llmagent_with_model_extra_fields](../../../examples/llmagent_with_model_extra_fields/README.md)。

#### 高级用法

从版本 `1.1.10`之后 OpenAIModel 支持传入共享的 http client 来解决连接复用的场景,当前的 OpenAIModel 默认每次都会创建临时的 http client 去访问模型服务;如果期望连接复用可以使用如下的方式
Expand Down
2 changes: 1 addition & 1 deletion examples/fastapi_server/_runner_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ def new_session_id() -> str:

async def close(self) -> None:
"""Gracefully close the runner and release resources."""
self._runner.close()
await self._runner.close()
logger.info("RunnerManager closed: app=%s", self.app_name)

# ------------------------------------------------------------------
Expand Down
8 changes: 8 additions & 0 deletions examples/llmagent_with_model_extra_fields/.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Copy this file or edit it in place before running the example.
# The example uses an OpenAI-compatible endpoint.

TRPC_AGENT_API_KEY=
TRPC_AGENT_BASE_URL=
TRPC_AGENT_MODEL_NAME=


145 changes: 145 additions & 0 deletions examples/llmagent_with_model_extra_fields/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# LLM Agent 模型额外字段上报示例

本示例演示如何通过 `OpenAIModel.response_metadata_extractor`,从 OpenAI
兼容服务的响应中提取厂商扩展字段,并通过 `LlmResponse.custom_metadata`
传递给业务代码和 tracing 链路。

## 关键特性

- **显式白名单提取**:业务通过回调决定允许上报哪些厂商字段,避免透传完整原始响应。
- **兼容流式与非流式响应**:回调接收响应的完整 `model_dump()` 字典;流式模式只读取第一个事件。
- **稳定的元数据命名空间**:结果保存在
`custom_metadata["provider_response_metadata"]` 中。
- **失败不影响模型调用**:回调返回 `None`、非字典、不可 JSON 序列化数据或抛出异常时,SDK 会忽略该元数据。
- **自动进入 tracing**:提取结果会作为 provider response metadata 上报到模型调用 span。

## Agent 层级结构说明

本例是单 Agent 示例,额外字段提取器绑定在模型上:

```text
weather_agent (LlmAgent)
├── model: OpenAIModel(..., response_metadata_extractor=_extract_some_field)
├── tool: get_weather_report(city)
└── runner: 从 Event.custom_metadata 读取并打印厂商元数据
```

关键文件:

- [agent/agent.py](./agent/agent.py):定义白名单提取器并注入 `OpenAIModel`
- [agent/config.py](./agent/config.py):读取模型连接环境变量
- [agent/tools.py](./agent/tools.py):天气工具实现
- [agent/prompts.py](./agent/prompts.py):Agent 提示词
- [run_agent.py](./run_agent.py):运行入口,读取并打印 provider metadata

## 关键代码解释

这里的 `some_marker` `some_field` 表示厂商返回的额外字段,具体视模型本身返回的真实数据为例,这里只是一个描述

### 1) 定义厂商字段提取器

```python
def _extract_some_field(
response_data: dict[str, Any],
) -> dict[str, Any] | None:
some_marker = response_data.get("some_marker")
if not isinstance(some_marker, dict):
return None
some_field = some_marker.get("some_field")
if not isinstance(some_field, str) or not some_field:
return None
return {
"some_marker": {
"some_field": some_field,
},
}
```

`response_data` 是 OpenAI SDK 响应对象的完整 `model_dump()` 结果。回调只返回
允许进入业务事件和 tracing 的字段,并可在这里统一转换命名格式。

> 流式模式只会使用第一个响应事件执行一次提取,因此兼容服务需要在首个事件中携带厂商字段。

### 2) 将提取器注入模型

`agent/agent.py` 将回调传给 `OpenAIModel`:

```python
OpenAIModel(
model_name=model_name,
api_key=api_key,
base_url=base_url,
response_metadata_extractor=_extract_some_field,
)
```

### 3) 从事件中读取元数据

```python
from trpc_agent_sdk.models import PROVIDER_RESPONSE_METADATA

provider_metadata = (event.custom_metadata or {}).get(
PROVIDER_RESPONSE_METADATA
)
```

本例的一次天气问答包含两次模型调用:第一次模型选择天气工具,第二次模型根据
工具结果生成最终回答。因此运行输出中会看到两个不同的 `some_field`。

## 环境要求

- Python3.10+,推荐 Python3.12

## 构建步骤

```bash
git clone https://github.com/trpc-group/trpc-agent-python.git
cd trpc-agent-python
./build.sh
source .venv/bin/activate
```

## 运行步骤

### 配置环境变量

在 [`.env`](./.env) 中配置(或通过 `export` 设置):

```bash
TRPC_AGENT_API_KEY=<your-api-key>
TRPC_AGENT_BASE_URL=<openai-compatible-base-url>
TRPC_AGENT_MODEL_NAME=<model-name>
```

模型服务需要返回提取器识别的扩展字段。本例期望响应中包含:

```json
{
"some_marker": {
"some_field": "<provider-span-id>"
}
}
```

### 运行命令

```bash
cd examples/llmagent_with_model_extra_fields
python3 run_agent.py
```

## 运行结果示例

```text
Session ID: 926d3fa3...
User: What's the current weather in Beijing?
Assistant:
Provider metadata: {"some_marker": {"some_field": "d3f87b419c8e871c"}}

Invoke Tool: get_weather_report({'city': 'Beijing'})
Tool Result: {'temperature': '25°C', 'condition': 'Sunny', 'humidity': '60%'}
Assistant: It's currently **25°C and sunny** in Beijing.
Provider metadata: {"some_marker": {"some_field": "5ca6786f4a74f5b1"}}
```

`some_field` 由模型服务生成,每次运行都会不同。
Empty file.
55 changes: 55 additions & 0 deletions examples/llmagent_with_model_extra_fields/agent/agent.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Agent module for the model extra fields example."""

from typing import Any

from trpc_agent_sdk.agents import LlmAgent
from trpc_agent_sdk.models import LLMModel
from trpc_agent_sdk.models import OpenAIModel
from trpc_agent_sdk.tools import FunctionTool

from .config import get_model_config
from .prompts import INSTRUCTION
from .tools import get_weather_report

def _extract_some_field(response_data: dict[str, Any]) -> dict[str, Any] | None:
"""Allowlist Venus tracing metadata from an OpenAI-compatible response."""
marker = response_data.get("venusMarker")
if not isinstance(marker, dict):
return None
span_id = marker.get("spanId")
if not isinstance(span_id, str) or not span_id:
return None
return {
"venusMarker": {
"span_id": span_id,
},
}

def _create_model() -> LLMModel:
"""Create an OpenAI-compatible model with SDK-managed extra fields enabled."""
api_key, base_url, model_name = get_model_config()
return OpenAIModel(
model_name=model_name,
api_key=api_key,
base_url=base_url,
response_metadata_extractor=_extract_some_field,
)


def create_agent() -> LlmAgent:
"""Create a weather agent that uses model-level extra fields."""
return LlmAgent(
name="weather_agent",
description="A weather assistant with SDK-managed extra fields enabled.",
model=_create_model(),
instruction=INSTRUCTION,
tools=[FunctionTool(get_weather_report)],
)


root_agent = create_agent()
21 changes: 21 additions & 0 deletions examples/llmagent_with_model_extra_fields/agent/config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Tencent is pleased to support the open source community by making tRPC-Agent-Python available.
#
# Copyright (C) 2026 Tencent. All rights reserved.
#
# tRPC-Agent-Python is licensed under Apache-2.0.
"""Configuration helpers for the model extra fields example."""

import os

def get_model_config() -> tuple[str, str, str]:
"""Get model connection settings from environment variables."""
api_key = os.getenv("TRPC_AGENT_API_KEY", "")
base_url = os.getenv("TRPC_AGENT_BASE_URL", "")
model_name = os.getenv("TRPC_AGENT_MODEL_NAME", "")
if not api_key or not base_url or not model_name:
raise ValueError("TRPC_AGENT_API_KEY, TRPC_AGENT_BASE_URL, and "
"TRPC_AGENT_MODEL_NAME must be set in environment variables")
return api_key, base_url, model_name



Loading
Loading