Skip to content

test: port the upstream Cohere tests - #296

Merged
eric8810 merged 13 commits into
masterfrom
test/port-the-upstream-cohere-tests
Oct 25, 2026
Merged

eric8810 merged 13 commits into
masterfrom
test/port-the-upstream-cohere-tests

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

按 RFC-0045(#281)把 @ai-sdk/cohere@4.0.52 的上游测试迁完,是按厂商分批的第二个(第一个是 Mistral,#295)。

上游 5 个测试文件共 80 个用例,全部有完整对应的测试并通过:新增 72 个,原有 12 个只改出处标记,没有排除项。原有测试里有 3 个只断言了上游用例的一部分(text、tool call 的生成结果和 embedding 的原始响应),它们改标 partial:,完整断言在 src/cohere/tests/ 的新测试里。不改任何非测试代码。

位置按新定的布局:Cohere 的新测试全放 src/cohere/tests/,一个上游测试文件对应一个 Rust 文件。原有的 2 个独立测试文件留在原地,由包拆分线程的搬移 PR 合进来。

一、体系概览:Cohere 的测试现在怎么跑(master 5743a40a)

cargo test -p aimux-providers
  ├─ unittests src/lib.rs                                    Cohere 一个都没有(src/cohere/*.rs 没有 #[cfg(test)])
  └─ tests/*.rs(126 个独立测试程序,各自 #[path] 引 tests/common/mock_fetch.rs)
       ├─ cohere_embedding_test.rs        5 个    /// TS: "<it 文本>"
       └─ cohere_upstream_e2e_test.rs     8 个    /// TS: "<it 文本>" (文件名)
            create_cohere(CohereProviderSettings { fetch: Some(fetch.transport()), .. })
                                                     cohere/mod.rs:107
            ├ CohereModel::do_generate / do_stream    cohere/model.rs:106 / :242
            │ └ build_request_body                    cohere/convert.rs:343
            │   ├ convert_prompt_to_cohere            cohere/convert.rs:129
            │   └ prepare_tools                       cohere/convert.rs:39
            ├ CohereEmbeddingModel::do_embed          cohere/embedding.rs:50
            └ CohereRerankingModel::do_rerank         cohere/reranking.rs:82
            MockFetch::seen()[0].json_body()          tests/common/mock_fetch.rs:67   与上游请求体比

上游 packages/cohere/src/ 对应:

cohere-chat-language-model.test.ts          41   createTestServer → provider(...).doGenerate / doStream
cohere-embedding-model.test.ts               7   doEmbed
reranking/cohere-reranking-model.test.ts    12   doRerank
convert-to-cohere-chat-prompt.test.ts       13   直接调 convertToCohereChatPrompt
cohere-prepare-tools.test.ts                 7   直接调 prepareTools

改动前 80 个用例里只有 12 个有对应测试("should extract embedding" 还被两个文件各标了一次),出处只写 it 文本;两个内部函数文件的 20 个用例一个都没迁。

二、本 PR 的改动(细节)

1. 新测试(src/cohere/mod.rs:273 加 #[cfg(test)] mod tests;)

cargo test -p aimux-providers --lib cohere::                       72 个
  src/cohere/tests/mod.rs    mod chat_language_model; mod convert_to_chat_prompt; mod embedding_model; mod prepare_tools; mod reranking;
  src/cohere/tests/convert_to_chat_prompt.rs                        13 个
    mod convert_to_cohere_chat_prompt › mod file_processing
      should_extract_documents_from_file_parts()                    :40
        /// upstream: @ai-sdk/cohere src/convert-to-cohere-chat-prompt.test.ts › convert to cohere chat prompt › file processing › should extract documents from file parts
        convert_prompt_to_cohere(&prompt)                           cohere/convert.rs:129
  src/cohere/tests/prepare_tools.rs                                  7 个
    mod tool_choice_handling
      should_handle_tool_type_tool_choice_by_filtering_tools()      :141
        prepare_tools(..)                                           cohere/convert.rs:39
  src/cohere/tests/chat_language_model.rs                           39 个
    mod do_generate › mod text / mod tool_call                      整个结果与 .snap 快照比(含 response 的 body、headers)
    mod do_generate › mod request › should_pass_tools()             :608
      /// upstream: @ai-sdk/cohere src/cohere-chat-language-model.test.ts › doGenerate › request › should pass tools
      CohereModel::do_generate                                      cohere/model.rs:106
      fetch.seen()[0].json_body()["tools"] 与上游比
    mod do_stream › text / reasoning / tool_call / empty_tool_call / request   :887 起
  src/cohere/tests/embedding_model.rs                                3 个   do_embed › input type、output dimension、raw response(body、headers 整份比)
  src/cohere/tests/reranking/reranking_model.rs                     10 个   路径对应上游 reranking/ 子目录
    mod do_rerank › mod json_documents › should_send_request_with_the_correct_headers()   :137
      CohereRerankingModel::do_rerank                               cohere/reranking.rs:82
  fixtures/、reranking/fixtures/   上游 __fixtures__ 原样拷贝
  snapshots/                       上游 .snap 里的 10 份快照
src/test_support/mod.rs             #[path = "../../tests/common/mock_fetch.rs"] mod mock_fetch;   lib.rs 末尾 #[cfg(test)] mod test_support;
scripts/check_provider_boundaries.sh   规则 7c、8b 的 find 跳过 */tests/*(同 #[cfg(test)] 块)

2. 原有 12 个测试只改标记

/// TS: "should extract tool calls" (cohere/src/cohere-chat-language-model.test.ts)
  → mod do_generate { mod tool_call { /// upstream: @ai-sdk/cohere src/cohere-chat-language-model.test.ts › doGenerate › tool call › should extract tool calls } }

函数名、函数体不变,每层 describe 包一个 mod。其中 3 个只断言了部分 response/header(e2e 的 generate_text、generate_tool_calls,embedding 的 should_expose_raw_response),标记改为 /// partial: `<用例>`; full port in src/cohere/tests/<文件>.rs.;它们用 wiremock,wiremock 会多加响应头,所以整份比较放在用 MockFetch 的新测试里。e2e 文件里的 embed_values 同时查请求体、向量、用量和原始响应,比上游那个用例多,标记让给 cohere_embedding_test.rs 的同名测试,自己改成普通注释;chat_supports_http_image_urls 测的是 supportedUrls,不是上游用例,注释写明是 aimux 自己的测试。

3. aimux-providers/layout.toml(RFC-0046 §4.4 格式,只有 [cohere])

16 个上游源文件:15 个 mapped(rust 是现在的源文件名,搬移 PR 改名时一起改),version.ts 为 not-applicable。所有 tests 表为空,没有手写排除。

harness 上的差别(期望值都没改):

  • TS 测试服务器的 requestHeaders 不含 user-agent,比较请求头时去掉。
  • 快照里值为 undefined 的键在 JSON 里省略。
  • convertToCohereChatPrompt 返回的 warnings: []:Rust 的 convert_prompt_to_cohere 不返回告警,改为断言 build_request_body 的告警为空。
  • PreparedTools 没有 PartialEq,逐字段比较。
  • 内部测试里 providerOptions 的键用 options::NAMESPACE(边界检查不许在 src/cohere/ 里写字面量)。

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change
  • New provider
  • New / updated binding
  • Docs / RFC
  • Tests(上游测试迁移)

Checklist

  • cargo +1.99 fmt --all -- --check passes
  • cargo +1.99 clippy -p aimux-providers --lib --tests -- -D warnings passes
  • cargo +1.99 test -p aimux-providers --lib --test cohere_embedding_test --test cohere_upstream_e2e_test(合入带 test: port the upstream Mistral tests #295 的 master 后、head 2cdd716d):3 个测试程序共 304 个,全过(cohere 新测试 72、embedding 5、e2e 8);layout.toml 与 master 的 [mistral] 表合并
  • scripts/check_provider_boundaries.sh passes
  • Commit messages follow Conventional Commits

Notes for reviewers


Generated by Claude Code

cunninghamcard-bit and others added 4 commits October 20, 2026 04:27
The 12 existing ports in the embedding and e2e test files now carry an
RFC-0045 marker and sit in one mod per upstream describe. Bodies and fn names
are unchanged. The e2e embed test checks more than the upstream case that
cohere_embedding_test.rs ports, so it no longer claims it; the supportedUrls
test is aimux's own and says so.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
All 20 cases of convert-to-cohere-chat-prompt and cohere-prepare-tools
(@ai-sdk/cohere@4.0.52) in src/cohere/convert/tests.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
49 cases (chat 37, embedding 2, reranking 10) of @ai-sdk/cohere@4.0.52 under
tests/it/cohere/, built as the single tests/it/main.rs program; the other 11
are already in cohere_embedding_test.rs and cohere_upstream_e2e_test.rs.
Upstream fixtures and snapshots sit beside the tests.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Maps each upstream source file of @ai-sdk/cohere@4.0.52 to its Rust file
(RFC-0046 §4.4) with the RFC-0045 tests table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
cunninghamcard-bit and others added 2 commits October 20, 2026 06:36
All Cohere tests of this port now live in one directory, one file per
upstream test file: chat_language_model, convert_to_chat_prompt,
embedding_model, prepare_tools and reranking/reranking_model, with their
fixtures/ and snapshots/. src/cohere/convert/tests.rs and the tests/it/
binary are gone; the shared mock fetch is reached through
src/test_support/. The boundary script skips files under tests/ dirs as
it skips #[cfg(test)] blocks. Test bodies and markers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
convert_to_chat_prompt.rs and prepare_tools.rs still wrapped their tests
in the module of the old src/cohere/convert/tests.rs. The file is that
module now, so only the upstream describe levels stay; the module_inception
allow goes with the wrapper. Test bodies and markers are unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@cunninghamcard-bit
cunninghamcard-bit marked this pull request as ready for review October 24, 2026 11:22

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cohere test port:generate_text / generate_tool_calls 和 embedding 的 upstream snapshot 只断言了部分 response/header map;请补全 response body/headers 的完整断言,或明确说明缩减。其余 69 个新测试与 fixtures 一致。

…h the whole snapshot

eric's review of #296: `should extract text response`, `should extract tool
calls` and the embedding `should expose the raw response` asserted only part
of upstream's snapshot (selected fields, a trimmed embedding body, one
header). They are now ported in src/cohere/tests/ against the .snap entries
(snapshots/do_generate_text.json, do_generate_tool_call.json,
do_embed_raw_response.json): the whole result, response body and headers
included. The older cases in tests/cohere_upstream_e2e_test.rs and
tests/cohere_embedding_test.rs stay and carry `partial:` notes pointing at
the full ports.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5

cunninghamcard-bit commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor Author
eric 的意见(原话) 处理
generate_text / generate_tool_calls 和 embedding 的 upstream snapshot 只断言了部分 response/header map;请补全 response body/headers 的完整断言,或明确说明缩减。 已补完整断言(0852cc0b)。src/cohere/tests/ 新增 3 个测试,把整个结果与上游 .snap 快照比较:do_generate › text › should_extract_text_response、do_generate › tool_call › should_extract_tool_calls、do_embed › should_expose_the_raw_response。比较范围包括 response 的 body 和 headers(含 content-length,长度与上游一致)。原来这 3 个测试用 wiremock,wiremock 会多加响应头,没法逐字比 headers,所以标记改为 partial:,并指向完整测试所在文件。
其余 69 个新测试与 fixtures 一致。 不变。新测试现在共 72 个,80 个上游用例各有一个 upstream: 标记,另有 3 个 partial:。之后 master 合入了 #295,layout.toml 冲突,已合并 master 解决(2cdd716d,保留 [mistral] 表,再加 [cohere] 表)。合并后 lib + cohere* 共 304 个测试全过,fmt、clippy 和边界检查也都通过。PR 描述里的数字已同步。

Generated by Claude Code

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

layout.toml 的 merge 解决错了:mistral 下出现空的 [[mistral.files]],原来的 version.ts not-applicable 行被移到 cohere。请恢复 mistral 的 version.ts 行并删除空 entry。其余测试全量断言和 fixtures 没问题。

cunninghamcard-bit and others added 2 commits October 25, 2026 11:11
The master merge took the shared trailing version.ts lines once, which left
an empty [[mistral.files]] entry and moved the Mistral entry under Cohere.
layout.toml is now master's file with the Cohere table appended after it,
so the diff against master only adds the Cohere table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5

Copy link
Copy Markdown
Contributor Author
eric 的意见(原话) 处理
layout.toml 的 merge 解决错了:mistral 下出现空的 [[mistral.files]],原来的 version.ts not-applicable 行被移到 cohere。请恢复 mistral 的 version.ts 行并删除空 entry。 已修,5c9d9e0f。layout.toml 重建为 master 的原文(mistral 18 项、vertex 44 项,version.ts 的 not-applicable 行回到 mistral 下)后面追加 cohere 表(16 项),不再有空 entry;用 tomllib 逐表核对过条目数
其余测试全量断言和 fixtures 没问题。 没有改动。分支已合入最新 master(fa9ce2df,master 209be26),fmt、clippy、边界脚本和 cohere 测试都通过

Generated by Claude Code


Generated by Claude Code

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

layout.toml 已恢复 mistral version.ts、删除空 entry;Cohere 表与上游 fixtures/snapshots 精确,80 markers、无 ignore。Approve。

…eam-cohere-tests

# Conflicts:
#	aimux-providers/layout.toml

Copy link
Copy Markdown
Contributor Author

Flutter example (android) failed on c92e311 before any build step ran: sdkmanager got HTTP 502 from dl.google.com while downloading NDK 28.2 (android-ndk-r28c-linux.zip), so CI result went red with it. This PR changes only Cohere tests and layout.toml, nothing the Flutter example builds. I re-ran the failed jobs once.


Generated by Claude Code


Generated by Claude Code

…eam-cohere-tests

# Conflicts:
#	aimux-providers/layout.toml
@eric8810
eric8810 merged commit 04f4255 into master Oct 25, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants