Skip to content

test: port the upstream Open Responses tests - #347

Merged
eric8810 merged 41 commits into
masterfrom
test/port-the-upstream-open-responses-tests
Oct 27, 2026
Merged

eric8810 merged 41 commits into
masterfrom
test/port-the-upstream-open-responses-tests

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

按 RFC-0045(#281)把 @ai-sdk/open-responses@2.0.58 的上游测试迁完,按厂商分批的第九个(Mistral #295、Cohere #296、DeepSeek #299、Groq #301、Hugging Face #303、xAI #312、Anthropic AWS #319、Azure #324)。

上游 5 个测试文件共 124 个用例:

  • 93 个有对应测试并通过:
    • 新写 57 个:provider 1 个、输入转换 20 个、语言模型 36 个。
    • 原有的 tests/open_responses_test.rs 里已经完整覆盖的 36 个(finish reason 8 个、输入转换 17 个、语言模型 11 个)直接加上 /// upstream: … 标记。另有 31 处只覆盖了一部分,加 /// partial: … 注释指向完整用例。这两类改动单独一个提交,函数体不变。
  • 19 个写进 layout.toml 的手写排除:
    • 18 个 sub-feature(Provider 构造与配置:迁移后遗留缺口 #275):
      • 扩展注册表相关的 10 个:历史回放 5 个、provider tool 编码 3 个、流事件解码 2 个。
      • customToolId 自定义工具 3 个。
      • strictResponseInput 2 个。
      • getResponseErrorMetadata 2 个。
      • 带扩展时拒绝 workflow 序列化 1 个。
    • 1 个 registered(SSE 预读):流的第一个事件就是 error 时,aimux 的 do_stream 直接返回 Err,不进流。
  • 12 个未迁:open-responses-extension.test.ts 整个文件。它测的 createOpenResponsesExtensionRegistry 在 aimux 里没有,layout.toml 里这个文件和 open-responses-tools.ts 都标成 missing,挂 Provider 构造与配置:迁移后遗留缺口 #275。

迁移中发现 16 处行为差异,都修在 #346 里,一处一个提交(每个提交带一个在 master 上失败的测试),#346 已合入 master:

提交 差异
996eaea1 回放 assistant 文本时丢了 item id 和 url_citation 标注
4cfcd02d 用户图片不带 detail(上游默认 auto,可由 providerOptions 指定)
cfac970a 用户内联文件没有按完整 media type 解析
53ea2cf6 用户文件是 provider 引用或文本文件时,上游报 UnsupportedFunctionality,aimux 只给警告
9e54ca4a 回放 tool call 时丢了 item id
787f45ef 输出的 reasoning、文本、tool call 不带 provider metadata(itemId 等)
3eb0e5f0 只有 summary 的 reasoning item 不产出 reasoning
506be6b7 流里的 response.reasoning_summary_text.delta 被忽略
278fb9ec 流结束时没收尾的 reasoning,结束事件不带 item id
2aab71e8 流里 tool call 的参数取累积的 delta,上游取完成事件里的完整参数
0b1433db 流里的 response.failed 和 error 事件没有变成 error part
d3865fa7 无 schema 的 JSON 输出没有发 { "type": "json_object" }
61e808d9 provider-defined tool 没有警告,tool choice 指向它时没有清空
38892609 不读 providerOptions.<name>.reasoningEffort
c8afdf3f 成功响应里带 error 时,status 和 data 跟上游不一致
211a1385 响应里缺 output 时,错误类型跟上游不一致

位置按新定的布局:新测试全放 src/open_responses/tests/,一个上游测试文件对应一个 Rust 文件。

一、体系概览:Open Responses 的测试现在怎么跑(master b91829af)

cargo test -p aimux-providers
  ├─ unittests src/lib.rs                                       open_responses 一个都没有
  └─ tests/*.rs
       ├─ open_responses_test.rs                70 个   11 个出处写成 /// TS: `…test.ts:行号`,不带 describe 链
       └─ package_user_agent_test.rs             1 处   ai-sdk-open-responses/2.0.58
            create_open_responses(OpenResponsesProviderSettings { .. })        open_responses.rs:181
            └ OpenResponsesProvider::responses(model_id)                       :230
              └ OpenResponsesModel
                ├ do_generate                                                  :294
                │ ├ build_request_body(options)                                :852
                │ │ └ convert_to_open_responses_input(prompt)                  :1039
                │ │   └ _with_namespace(prompt, provider_options_name)        :1045
                │ └ 解析 output → content / usage / finish reason
                │     └ map_open_responses_finish_reason                       :823
                └ do_stream                                                    :434
                  └ OpenResponsesStream(SSE 事件 → StreamPart)               :518

上游 packages/open-responses/src/ 对应:

open-responses-extension.test.ts                      12   createOpenResponsesExtensionRegistry
open-responses-provider.test.ts                        3   createOpenResponses(...),getResponseErrorMetadata
responses/convert-to-open-responses-input.test.ts     41   convertToOpenResponsesInput
responses/map-open-responses-finish-reason.test.ts     8   mapOpenResponsesFinishReason
responses/open-responses-language-model.test.ts       60   OpenResponsesLanguageModel.doGenerate / doStream,createTestServer + __fixtures__

二、本 PR 的改动(细节)

1. 新测试目录(open_responses.rs 末尾 #[cfg(test)] mod tests;,:1788)

cargo test -p aimux-providers --lib open_responses::                    57 个
  src/open_responses/tests/mod.rs        mod provider; mod responses;
  provider.rs                    1 个   mod endpoint_specific_response_error_metadata
  responses/mod.rs                      mod convert_to_input; mod language_model; mod map_finish_reason;
  responses/convert_to_input.rs 20 个   mod convert_to_open_responses_input › system_messages / user_messages /
                                         assistant_messages / tool_messages / provider_reference /
                                         top_level_only_media_type_resolution
    convert(prompt) / convert_with(name, prompt) → convert_to_open_responses_input_with_namespace   :30 / :39
  responses/language_model.rs   36 个   mod open_responses_language_model
                                         › do_generate › basic_generation / manual_history_replay / tools /
                                           provider_options_reasoning / tool_call_parsing / tool_choice /
                                           system_messages / pdf_input_file
                                         › do_stream › basic_generation / reasoning_with_tool_call / pdf_input_file
    create_model(model_id, responses) → provider.responses(..) + MockFetch                    :46
    json_fixture / chunks_fixture:fixtures/*.json、*.chunks.txt 回放                        :55 / :75
    snapshot!(name):与 snapshots/*.json 比(上游 __snapshots__ 转成 JSON)                    :37
  responses/map_finish_reason.rs         只有文档注释:8 个用例在 tests/open_responses_test.rs 里已经覆盖并标了出处
  fixtures/(7 个)、snapshots/(13 个)  上游 __fixtures__ 原样拷贝;快照每个 toMatchSnapshot 一个文件
  src/test_support/mod.rs                #[path = "../../tests/common/mock_fetch.rs"] mod mock_fetch;   lib.rs 末尾 #[cfg(test)] mod test_support;
scripts/check_provider_boundaries.sh     规则 7c、8b 的 find 跳过 */tests/*(同 #[cfg(test)] 块)

2. aimux-providers/layout.toml(只有 [open_responses])

src/index.ts                                         mapped → open_responses.rs
src/open-responses-extension.ts                      missing,issue = 275
src/open-responses-provider.ts                       mapped,sub-feature 2:getResponseErrorMetadata
src/open-responses-tools.ts                          missing,issue = 275
src/responses/convert-to-open-responses-input.ts     mapped,sub-feature 4:strictResponseInput ×2、自定义工具结果 ×2
src/responses/map-open-responses-finish-reason.ts    mapped
src/responses/open-responses-{api,config,language-model-options}.ts   mapped
src/responses/open-responses-language-model.ts       mapped
  registered 1,ref = "SSE 预读":preserves $expectedType stream errors
  sub-feature 12:workflow 序列化、扩展回放 ×5、扩展 tool ×3、扩展流事件 ×2、自定义工具输入
src/version.ts                                       not-applicable

3. tests/open_responses_test.rs:36 个测试加 /// upstream: @ai-sdk/open-responses src/<file> › <describe 链>,原来的 11 行 /// TS: …:行号 换成这个;31 处加 /// partial: …; full port in src/open_responses/tests/<file>.rs。

harness 上的差别(期望值都没改):

  • 上游的 createModel 直接 new OpenResponsesLanguageModel,这里用 create_open_responses(..).responses(..),差别只在这些用例不比的 URL 和 headers。
  • 错误比 AiMuxError 的变体和字段,不比同一个错误对象;UnsupportedFunctionalityError 对应 UnsupportedFunctionality(functionality)。
  • 快照里值为 undefined 的键在 JSON 里省略。

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change
  • New provider
  • New / updated binding
  • Docs / RFC
  • Tests(上游测试迁移)

Checklist

  • cargo +1.99 fmt -p aimux-providers -- --check passes
  • cargo +1.99 clippy -p aimux-providers --lib --tests -- -D warnings passes
  • cargo +1.99 test -p aimux-providers:lib 192(其中 open_responses 57),129 个测试二进制全过
  • scripts/check_provider_boundaries.sh passes
  • Commit messages follow Conventional Commits

Notes for reviewers


Generated by Claude Code

cunninghamcard-bit and others added 18 commits October 20, 2026 10:11
@ai-sdk/open-responses 2.0.58 reads `providerOptions.<name>.itemId` and
`annotations` off an assistant text part (convert-to-open-responses-input.ts
:279-305): the item id becomes the message `id`, `url_citation` annotations
go back on the `output_text`, and a change of item id starts a new message.
aimux sent the text without either.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 sets `detail` on every user `input_image`
from `providerOptions.<name>.imageDetail` (`low`, `high` or `auto`, else
`auto`; convert-to-open-responses-input.ts:88-98). aimux only did this for
tool-result images; the two now share `image_detail`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 builds the data URL of an inline user file
from `resolveFullMediaType` (convert-to-open-responses-input.ts:91-108), which
detects a partial type such as `image` from the bytes and throws when it
cannot. aimux used the media type as given. The pre-send check that tool
results already had now covers user files too.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 throws UnsupportedFunctionalityError for a
user file given as a provider reference ("file parts with provider
references") or as text ("text file parts"),
convert-to-open-responses-input.ts:75-84. aimux dropped the part with a
warning and sent the rest. The pre-send check now fails the call.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 puts an assistant tool call's
`providerOptions.<name>.itemId` on the replayed `function_call` as `id`
(convert-to-open-responses-input.ts:306-336). aimux sent the call without it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 puts `{ <name>: { itemId } }` on generated and
streamed tool calls, adds `url_citation` annotations on text
(getOutputTextAnnotations), and gives reasoning
createReasoningProviderMetadata: item id, summary, content, encrypted
content (open-responses-language-model.ts:446-505, :757-779, :876-917).
aimux returned every part without provider metadata, so a replayed
response lost its item ids.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 turns a reasoning item without content into
one reasoning part whose text is the joined summary
(open-responses-language-model.ts:448-473), so summary and encrypted-only
items survive a replay. aimux only read `content` and dropped such items.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 turns both `response.reasoning_summary_text.delta`
and LM Studio's `response.reasoning_text.delta` into reasoning deltas
(open-responses-language-model.ts:878-892). aimux ignored the summary
deltas.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 remembers the active reasoning item id and
closes it with that id in flush (open-responses-language-model.ts:997-1003).
aimux kept only a flag and emitted `reasoning-end` with id `reasoning-0`,
which matched no `reasoning-start`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 uses the `response.output_item.done` item's
`arguments` unless they are empty, and only then the streamed ones
(open-responses-language-model.ts:758-781). aimux preferred the
accumulated arguments, which hold the empty string from
`response.output_item.added` when no argument events arrive.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 enqueues createOpenResponsesStreamError for a
`response.failed` event with `response.error` and for every `error` event
(message, code, the event as data), and an `error` event sets the finish
reason to error with the code (open-responses-language-model.ts:957-990).
aimux only set the finish reason for `response.failed` and ignored `error`
events after the first one. The first-event case stays the registered SSE
pre-read: do_stream returns it as Err.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 sends `{ type: 'json_object' }` when the JSON
response format has no schema (open-responses-language-model.ts:304-315).
aimux sent `{ type: 'json_schema' }` without a schema, which servers reject.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 warns `provider-defined tool <id>` for every
provider tool no extension encodes, and leaves `tool_choice` out when it
names such a tool (open-responses-language-model.ts:166-300). aimux has no
extensions; it dropped the tools without a warning and sent the choice as
a function choice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 resolves the effort as
`openResponsesOptions.reasoningEffort ?? <mapped top-level reasoning>`
(open-responses-language-model.ts:330-347), so a provider-native value such
as `max` goes out as is. aimux only read `reasoningSummary` from the
provider options.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 throws an APICallError with
`statusCode: errorMetadata?.statusCode ?? 400` and `data: response.error`
when a 200 response carries `error` (open-responses-language-model.ts
:411-426). aimux used the HTTP status, 200, and left `data` empty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/open-responses 2.0.58 throws an APICallError with status 500,
`isRetryable: false`, the request and the raw response when the response
has no `output` (open-responses-language-model.ts:428-441). aimux returned
InvalidResponseData with the same message.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Port @ai-sdk/open-responses 2.0.58 per RFC-0045 into
src/open_responses/tests/: provider.rs and responses/{language_model,
convert_to_input, map_finish_reason}.rs, one file per upstream test file,
with the upstream fixtures and the snapshots as JSON. layout.toml gets the
[open_responses] table: the extension registry and custom tools are
missing (#275), plus 18 sub-feature exclusions and the registered SSE
pre-read.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Old tests in tests/open_responses_test.rs that assert every expectation of
an upstream case move, unchanged, into mod trees named after its describe
chain and carry its `/// upstream:` marker (36). Tests that cover a case
only in part get a `/// partial:` note pointing at the full port (31).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
cunninghamcard-bit and others added 2 commits October 24, 2026 09:54
Keeps this PR's API call error for a response without output (master's #290 had turned the old message into InvalidResponseData), and keeps both this PR's tests and master's raw-chunk tests (#285) at the end of open_responses_test.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…t/port-the-upstream-open-responses-tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
cunninghamcard-bit and others added 10 commits October 25, 2026 08:49
…t-is-an-api-call-error' into test/port-the-upstream-open-responses-tests
…d request

The missing output ApiCallError was built in do_generate from the raw
request URL and body, so an inline image's base64 went into
request_body_values and from there into FFI errors and recordings. The
check now runs in open_responses_successful_response_handler, next to the
response body error check, and uses the URL and request values the HTTP
layer has already sanitized, as the other API call errors do.

Test: missing_output_error_does_not_carry_inline_image_data
(tests/open_responses_test.rs) fails before the change with the image data
in request_body_values.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…missing-output-is-an-api-call-error

# Conflicts:
#	CHANGELOG.md
…t-is-an-api-call-error' into test/port-the-upstream-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml
…missing-output-is-an-api-call-error

# Conflicts:
#	CHANGELOG.md
… into test/port-the-upstream-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml
…missing-output-is-an-api-call-error

# Conflicts:
#	CHANGELOG.md
… into test/port-the-upstream-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml
…missing-output-is-an-api-call-error

# Conflicts:
#	CHANGELOG.md
… into test/port-the-upstream-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml
@cunninghamcard-bit
cunninghamcard-bit changed the base branch from fix/open-responses-missing-output-is-an-api-call-error to master October 25, 2026 23:33

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the post-approval commits on this head: only additive tests/docs or tightened boundary checks; no weakening or behavior regression. Re-approving the current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase is content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; tests and CI unchanged. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master merges and layout.toml conflict resolution; CI 16/16. Re-approving current head.

…eam-open-responses-tests

# Conflicts:
#	aimux-providers/layout.toml

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.

@eric8810
eric8810 merged commit 25e83ea into master Oct 27, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants