Skip to content

test: port the upstream OpenAI-compatible tests - #355

Merged
eric8810 merged 8 commits into
masterfrom
test/port-the-upstream-openai-compatible-tests
Oct 25, 2026
Merged

eric8810 merged 8 commits into
masterfrom
test/port-the-upstream-openai-compatible-tests

Conversation

@cunninghamcard-bit

Copy link
Copy Markdown
Contributor

Summary

按 RFC-0045(#281)把 @ai-sdk/openai-compatible@3.0.62 的上游测试迁完,按厂商分批的第十一个(Mistral #295、Cohere #296、DeepSeek #299、Groq #301、Hugging Face #303、xAI #312、Anthropic AWS #319、Azure #324、Open Responses #347、Google Vertex #352)。

上游 10 个测试文件共 268 个用例:

  • 266 个新写了测试并通过,放在 src/openai_compatible/tests/,一个上游测试文件对应一个 Rust 文件(chat 语言模型 107 个、消息转换 50 个、工具 12 个、usage 3 个、content array 2 个、驼峰键 17 个、completion 29 个、嵌入 9 个、图像 26 个、provider 11 个)。
  • 2 个写进 layout.toml 的 ts-only:supportsStructuredOutputs 和 supportsMultiPartToolContent 两个 provider 设置“只传给 chat 模型”。上游查的是 mock 掉的模型构造函数收到的参数;aimux 的 completion、嵌入、图像模型共用 provider 的 chat 设置,但从不读这两个字段,行为上没有可观察的差别。
  • 原有的旧测试没有一个完整覆盖某个上游用例,所以都没加 /// upstream: 标记;其中 20 处只覆盖了一部分,加了 /// partial: … 指向完整用例,这部分单独一个提交,函数体不变。

迁移中发现 1 处行为差异,修在 #354 里(带一个在 master 上失败的测试)。本 PR 叠在 #354 上:

提交 差异
cb0b764d 流里某个事件不是 JSON 时,aimux 的 JsonParse 只带 serde 的报错;上游是 JSON parsing failed: Text: {unparsable}.\nError message: …。改在 provider-utils 共用的 SSE 解析里,所有走 create_event_source_response_handler 的厂商一起对齐

一、体系概览:OpenAI-compatible 的测试现在怎么跑(master 6c5ff9f6)

cargo test -p aimux-providers
  ├─ unittests src/lib.rs                       openai_compatible 6 个(config.rs、convert.rs 里的 #[cfg(test)])
  └─ tests/*.rs
       ├─ openai_compatible_upstream_e2e_test.rs    11 个
       ├─ openai_compatible_factory_test.rs         11 个
       └─ openai_compatible_test.rs                 17 个(Groq / DeepSeek 预设)
            create_openai_compatible(OpenAICompatibleProviderSettings { .. })   openai_compatible/mod.rs:142
            └ OpenAICompatibleProvider
              ├ chat → OpenAICompatibleChatModel                               :283   chat.rs
              │   └ build_request_body / convert_messages / prepare_function_tools   convert.rs
              ├ completion → OpenAICompatibleCompletionModel                   :291   completion.rs
              ├ embedding → OpenAICompatibleEmbeddingModel                     :300   embedding.rs
              └ image → OpenAICompatibleImageModel                             :309   image.rs
            共用配置 CompatModelConfig { chat: ChatSettings, error_structure, .. }   config.rs
            流:post_json_to_api + create_event_source_response_handler          aimux-provider-utils/src/response_handler.rs

上游 packages/openai-compatible/src/ 的 10 个测试文件(用例数):

chat/openai-compatible-chat-language-model.test.ts          107
chat/convert-to-openai-compatible-chat-messages.test.ts      50
chat/openai-compatible-prepare-tools.test.ts                 12
chat/convert-openai-compatible-chat-usage.test.ts             3
chat/openai-compatible-chat-content-array.test.ts             2
utils/to-camel-case.test.ts                                  17
completion/openai-compatible-completion-language-model.test.ts  29
embedding/openai-compatible-embedding-model.test.ts           9
image/openai-compatible-image-model.test.ts                  26
openai-compatible-provider.test.ts                           13

二、本 PR 的改动(细节)

1. 新测试目录(openai_compatible/mod.rs 末尾 #[cfg(test)] mod tests;,:368)

cargo test -p aimux-providers --lib openai_compatible::tests::          266 个
  src/openai_compatible/tests/mod.rs       mod chat; mod completion; mod embedding; mod image; mod provider; mod utils;
  chat/chat_language_model.rs      107   mod config / do_generate / do_stream / metadata_extraction / raw_chunks / transform_request_body
  chat/convert_to_chat_messages.rs  50   convert::convert_messages(&prompt, &MessageSpec { .. })
  chat/prepare_tools.rs             12   convert::prepare_function_tools(&CallOptions)
  chat/convert_chat_usage.rs         3   chat::usage_from_raw
  chat/chat_content_array.rs         2   provider.chat(..).do_stream 加 MockFetch
  utils/to_camel_case.rs            17   convert::to_camel_case;resolveProviderOptionsKey / warnIfDeprecated… 经 build_request_body 测
  completion/completion_language_model.rs  29
  embedding/embedding_model.rs       9
  image/image_model.rs              26
  provider.rs                       11
  chat/fixtures/(6 个)            上游 __fixtures__ 原样拷贝
  chat/snapshots/(4 个)           上游 toMatchSnapshot 转成 JSON
  src/test_support/mod.rs           #[path = "../../tests/common/mock_fetch.rs"] mod mock_fetch;   lib.rs 末尾 #[cfg(test)] mod test_support;
可见性                              chat.rs:35、embedding.rs:23、image.rs:27 的 config 字段,completion.rs:28 的 CompletionConfig 和 :37 的 config 字段改成 pub(super)
scripts/check_provider_boundaries.sh  规则 7c、8b 的 find 跳过 */tests/*;规则 4c 允许 crate::test_support(openai_compatible 只许引用共用层,测试要引用 mock)

2. 这些测试动到的进程级状态

新测试编进 aimux-providers 的 lib 测试二进制,和其他模块的单元测试同进程并行跑。它们不读也不改环境变量,没有 #[serial],没有全局计数器,不写临时文件;每个测试用自己的 MockFetch。

3. aimux-providers/layout.toml(只加 [openai_compatible],26 个上游源文件)

25 个 mapped 到 openai_compatible/*.rs,src/version.ts not-applicable
src/openai-compatible-provider.ts   ts-only 2:supportsStructuredOutputs / supportsMultiPartToolContent 只传给 chat 模型

chat/openai-compatible-chat-content-array.test.ts 没有同名源文件,用例全迁了,不需要写排除。

4. 旧测试的标记(单独一个提交,函数体不变)

tests/openai_compatible_upstream_e2e_test.rs   13 处 /// partial:(换掉原来的 /// TS: 行)
tests/openai_compatible_factory_test.rs         7 处 /// partial:
tests/openai_compatible_test.rs                 不动:测的是 Groq / DeepSeek 预设,不对应上游用例

harness 上的差别(期望值都没改):

  • 上游直接 new OpenAICompatibleChatLanguageModel(...) 等;这里用 create_openai_compatible 建,config › 用例用 CompatModelConfig 加 from_config 建,读 model.config。
  • provider(id) 和 provider.languageModel(id) 返回的模型读不到 config,includeUsage 等从它发出的请求体里看。includeUsage: undefined 在 Rust 里是 bool,按 false 断言。
  • 上游 doGenerate 的 request.body 是 JSON 字符串,这里是 JSON 值,比之前先解析快照里的字符串。
  • mock 响应带上测试服务器的默认响应头(content-type、content-length 或 cache-control、connection)。
  • 流里的错误 part 比 ApiCallError.data(上游的原始 error 对象);抛出的错误比 {name, message}。坏 JSON 的报错只比 \nError message: 之前的部分,后面是 JS 运行时 JSON.parse 自己的文字。
  • 图像的两个 response metadata 用例:aimux 没有 currentDate 钩子,只断言时间戳落在调用期间。
  • toThrow("'X' functionality not supported") 比 UnsupportedFunctionality(X) 的 functionality 字符串。
  • user agent 钉住的是 ai-sdk-openai-compatible/3.0.62 ai-sdk-provider-utils/5.0.53(上游 mock 成 0.0.0-test),单独比。快照里值为 undefined 的键在 JSON 里省略。

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change
  • New provider
  • New / updated binding
  • Docs / RFC
  • Tests(上游测试迁移)

Checklist

  • cargo +1.99 fmt --all -- --check passes
  • cargo +1.99 clippy -p aimux-providers -p aimux-provider-utils -p aimux-ops --lib --tests --all-features -- -D warnings passes
  • cargo +1.99 test -p aimux-providers -p aimux-provider-utils -p aimux-ops --all-features:177 个测试二进制,3091 个全过(aimux-providers lib 406,其中 openai_compatible 272:新 266、原有 6)
  • scripts/check_provider_boundaries.sh passes
  • Commit messages follow Conventional Commits

Notes for reviewers


Generated by Claude Code

cunninghamcard-bit and others added 3 commits October 20, 2026 14:36
… error

A streamed SSE event whose data is not JSON yielded `JsonParse` with only
the parser's message. The AI SDK's `parseJsonEventStream` reads each event
with `safeParseJSON`, whose `JSONParseError` reads
`JSON parsing failed: Text: <text>.\nError message: <message>`
(`@ai-sdk/provider` src/errors/json-parse-error.ts:16-19). Found while
porting the `@ai-sdk/openai-compatible` 3.0.62 tests (`doStream › should
handle unparsable stream parts`, chat and completion); every stream read
through `create_event_source_response_handler` now reports it the same way.
Well-formed JSON of the wrong shape stays `InvalidResponseData`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Ports the tests of @ai-sdk/openai-compatible 3.0.62 (RFC-0045) into
aimux-providers/src/openai_compatible/tests/, one Rust file per upstream
test file, one mod per describe level and an `/// upstream:` marker on
every test: 266 tests for 266 of the 268 upstream cases. The other 2 are
ts-only in aimux-providers/layout.toml (`[openai_compatible]`): they read
the settings handed to mocked model constructors, which aimux's
completion, embedding and image models never read.

Upstream fixtures are copied unchanged into chat/fixtures/, the four
`toMatchSnapshot` values become chat/snapshots/*.json. The model `config`
fields (and `CompletionConfig`) are widened to `pub(super)` so the tests
can read the configuration a model was built with. Boundary rule 4c lets
the in-crate tests name `crate::test_support`.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Old tests in openai_compatible_upstream_e2e_test.rs and
openai_compatible_factory_test.rs that cover part of an upstream case get a
`/// partial:` line naming the case and the file with its full port (20
lines). None of them covers a whole case, so none gets an `/// upstream:`
marker. Test bodies are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
cunninghamcard-bit and others added 2 commits October 24, 2026 10:05
…ai-sdk-parse-errors

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…-errors into test/port-the-upstream-openai-compatible-tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@cunninghamcard-bit
cunninghamcard-bit marked this pull request as ready for review October 24, 2026 11:08
@cunninghamcard-bit
cunninghamcard-bit changed the base branch from fix/openai-compatible-unparsable-stream-chunks-are-ai-sdk-parse-errors to master October 25, 2026 08:10

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenAI-compatible test port:266 tests/266 unique markers、fixtures/snapshots 精确,无生产行为变化;rebase 可去掉已合入 #354 的噪声。Approve。

layout.toml: master's [mistral] table (#295) and this PR's
[openai_compatible] table are both kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase 后 clean,provider-utils 无 #354 残留;266 tests/266 markers,断言完整。Approve。

…eam-openai-compatible-tests

# Conflicts:
#	aimux-providers/layout.toml
@eric8810
eric8810 merged commit 291cb8c into master Oct 25, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants