Skip to content

test: port the upstream Google Vertex tests - #352

Merged
eric8810 merged 9 commits into
masterfrom
test/port-the-upstream-google-vertex-tests
Oct 25, 2026
Merged

eric8810 merged 9 commits into
masterfrom
test/port-the-upstream-google-vertex-tests

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

按 RFC-0045(#281)把 @ai-sdk/google-vertex@5.0.100 的上游测试迁完,按厂商分批的第十个(Mistral #295、Cohere #296、DeepSeek #299、Groq #301、Hugging Face #303、xAI #312、Anthropic AWS #319、Azure #324、Open Responses #347)。

上游 22 个测试文件共 251 个用例:

  • 204 个有对应测试并通过:
    • 新写 199 个,放在 src/vertex/tests/,一个上游测试文件对应一个 Rust 文件(provider 与鉴权 48 个、Anthropic 30 个、MaaS 24 个、xAI 18 个、嵌入 16 个、图像 10 个、视频 23 个、转写 27 个、location 校验 2 个、语言模型 1 个)。
    • 原有的 tests/vertex_location_validation_test.rs 里已经完整覆盖的 5 个直接加上 /// upstream: … 标记。另有 37 处旧测试只覆盖了一部分,加 /// partial: … 注释指向完整用例(其中 4 处是合并 master 后评审要求补的)。这两类改动在单独的提交里,函数体不变。
  • 31 个写进 layout.toml 的手写排除:
    • 14 个 sub-feature(Provider 构造与配置:迁移后遗留缺口 #275):
      • 边缘鉴权从 GOOGLE_CLIENT_EMAIL / GOOGLE_PRIVATE_KEY / GOOGLE_PRIVATE_KEY_ID 读凭据 5 个。
      • interactions 模型 2 个,Cloud TTS 的 chirp 路由 2 个。
      • tools(Vertex 和 Vertex Anthropic)2 个,工具结果下载上限 1 个。
      • 图像结果的 is_retryable 1 个,视频操作的 error 状态 1 个。
    • 17 个 ts-only:
      • JS 的 new 3 个。
      • 数 mock 的 createOpenAICompatible 调用次数或参数 5 个,查 mock 的 base provider 收到的参数 6 个。
      • 边缘鉴权模块自己的错误文本和 user agent 3 个:aimux 只有一条鉴权路径,跟 google-auth-library(Node)走。
  • 16 个未迁:google-vertex-cloud-tts-speech-model.test.ts 整个文件。aimux 没有 Vertex 的 Cloud Text-to-Speech 模型,layout.toml 里这个文件和 google-vertex-tools.ts 都标成 missing,挂 Provider 构造与配置:迁移后遗留缺口 #275。

迁移中发现 2 处行为差异,都修在 #351 里,一处一个提交(每个提交带一个在 master 上失败的测试)。#351 已合入 master,本 PR 合并了 master(b7ce78a3,之后又合了一次带 #295 的 master,7f719f0b),不再重复 #351 的提交和测试,那两个测试按 master 的版本保留:

提交 差异
4ddae9c2 Gemini Live 转写在音频发完前被服务端关闭时,aimux 透传 socket 的 websocket closed by peer (…),上游报 … closed unexpectedly before finishing (code 1011, reason: internal error).(Google 和 Vertex 都是)
a995f016 三个 Gemini 图像模型上游声明支持文件输入、不支持 mask,aimux 两项都是未知

一、体系概览:Google Vertex 的测试现在怎么跑(master 6c5ff9f6)

cargo test -p aimux-providers
  ├─ unittests src/lib.rs                                       vertex 一个都没有
  └─ tests/*.rs
       ├─ vertex_upstream_e2e_test.rs           19 个
       ├─ google_vertex_image_test.rs            8 个
       ├─ vertex_location_validation_test.rs     8 个
       ├─ vertex_transcription_test.rs           5 个
       └─ vertex_video_test.rs                   3 个
            create_google_vertex(VertexProviderSettings { .. })               vertex/mod.rs:368
            └ VertexProvider
              ├ chat / embedding / image / video                               :477 / :492 / :502 / :512
              │   chat → Google 语言模型读 Vertex 命名空间(#291)
              ├ speech / transcription                                         :527 / :544
              │   └ gemini 转写 → google::transcription::live_stream          vertex/gemini_transcription.rs:259
              └ 鉴权 GoogleAuth(服务账号 JWT / ADC / 外部凭据)               vertex/auth.rs、auth_certificate.rs、auth_external.rs
            create_google_vertex_anthropic                                     vertex/anthropic_provider.rs:45
            create_google_vertex_maas / create_google_vertex_xai               vertex/partner_provider.rs:213 / :296

上游 packages/google-vertex/src/ 的 22 个测试文件(用例数):

google-vertex-provider-base.test.ts          35   google-vertex-provider.test.ts                 7
edge/google-vertex-provider-edge.test.ts      5   edge/google-vertex-auth-edge.test.ts          11
google-vertex-auth-google-auth-library.test.ts 5  google-vertex-location-validation.test.ts      7
anthropic/google-vertex-anthropic-provider{,-node}.test.ts   13 / 10   anthropic/edge/…-edge.test.ts   9
maas/google-vertex-maas-provider{,-node}.test.ts             12 / 10   maas/edge/…-edge.test.ts        9
xai/google-vertex-xai-provider{,-node}.test.ts               11 /  6   xai/edge/…-edge.test.ts         6
google-vertex-embedding-model.test.ts        16   google-vertex-image-model.test.ts             11
google-vertex-video-model.test.ts            24   google-vertex-transcription-model.test.ts     18
gemini-transcription/google-vertex-gemini-transcription-model.test.ts   9
google-vertex-language-model.test.ts          1   google-vertex-cloud-tts-speech-model.test.ts  16(未迁)

二、本 PR 的改动(细节)

1. 新测试目录(vertex/mod.rs 末尾 #[cfg(test)] mod tests;,:614)

cargo test -p aimux-providers --lib vertex::                            199 个
  src/vertex/tests/mod.rs                  每个上游测试文件一个 mod,上游子目录保留成子 mod
  provider_base.rs 28 · provider.rs 7 · edge/provider_edge.rs 5
  auth_google_auth_library.rs 5 · edge/auth_edge.rs 3 · location_validation.rs 2
  anthropic/anthropic_provider.rs 11 · anthropic_provider_node.rs 10 · anthropic/edge/anthropic_provider_edge.rs 9
  maas/maas_provider.rs 9 · maas_provider_node.rs 8 · maas/edge/maas_provider_edge.rs 7
  xai/xai_provider.rs 8 · xai_provider_node.rs 5 · xai/edge/xai_provider_edge.rs 5
  embedding_model.rs 16 · image_model.rs 10 · video_model.rs 23
  transcription_model.rs 18 · gemini_transcription/gemini_transcription_model.rs 9 · language_model.rs 1
  cloud_tts_speech_model.rs                只有文档注释:整个上游文件挂 #275
  fixtures/google-vertex-embedding.json    上游 __fixtures__ 原样拷贝
  fixtures/aimux-test-rsa-key.pem          测试用 RSA 私钥:上游 mock 掉 Web Crypto,aimux 自己签服务账号 JWT
  snapshots/embedding_{full_result,raw_response}.json   上游 toMatchSnapshot 转成 JSON
  src/test_support/mod.rs                  #[path = "../../tests/common/mock_fetch.rs"] mod mock_fetch;   lib.rs 末尾 #[cfg(test)] mod test_support;
vertex/auth.rs:101                         GoogleAuth.options 改成 pub(super),测试读鉴权器拿到的选项
scripts/check_provider_boundaries.sh       规则 7c、8b 的 find 跳过 */tests/*(同 #[cfg(test)] 块)

2. 这些测试动到的进程级状态

新测试编进 aimux-providers 的 lib 测试二进制,跟其他模块的单元测试同一个进程、并行跑,所以单列出来:

环境变量(都经 test_support::mock_fetch::EnvVar 守卫,离开作用域还原)
  GOOGLE_VERTEX_API_KEY            provider_base.rs 全部 28 个(上游 beforeEach 删它)、provider.rs、edge/provider_edge.rs
  GOOGLE_VERTEX_ACCESS_TOKEN       provider.rs、edge/provider_edge.rs
  GOOGLE_VERTEX_LOCATION           location_validation.rs、maas/maas_provider.rs
  GOOGLE_APPLICATION_CREDENTIALS   maas/maas_provider_node.rs、xai/xai_provider_node.rs、auth_google_auth_library.rs、anthropic/edge/anthropic_provider_edge.rs
  CLOUDSDK_CONFIG、GCE_METADATA_HOST、GCE_METADATA_IP    同上几个 node 测试,清掉 ADC 的回退来源
#[serial]                          改环境变量的 46 个测试全部 #[serial](serial_test 已是 dev-dependency)
计数器                             provider.rs、edge/provider_edge.rs 里的 AtomicUsize 都是每个测试自己的 Arc,没有全局 static
临时文件                           没有;私钥用 include_str! 读 fixtures/

3. aimux-providers/layout.toml(只加 [vertex],44 个上游源文件)

大部分 mapped 到 vertex/*.rs;edge 和 node 入口与普通入口映射到同一个 Rust 文件(aimux 只有一个入口)
src/google-vertex-cloud-tts-speech-model.ts      missing,issue = 275
src/google-vertex-tools.ts                       missing,issue = 275
src/version.ts                                   not-applicable
sub-feature 14、ts-only 17                       见 Summary

language model 和 location validation 的测试没有同名源文件,不需要写排除。

4. 旧测试的标记(单独一个提交,函数体不变)

tests/vertex_location_validation_test.rs   5 个按 describe 链放进 mod 并加 /// upstream:;2 处 /// partial:
tests/vertex_upstream_e2e_test.rs          18 处 /// partial:
tests/google_vertex_image_test.rs          9 处 /// partial:(含 master 上 #351 加的能力测试)
tests/vertex_transcription_test.rs         5 处 /// partial:
tests/vertex_video_test.rs                 3 处 /// partial:

合并 master 后补的 4 处:vertex_video_test.rs 的 should_generate_video(对应 should return completed with GCS URI video),e2e 里的 maas_sub_provider_builds_on_openai_compatible、xai_sub_provider_builds_on_openai_compatible、partner_headers_win_over_per_call_headers(各自对应 MaaS / xAI provider 测试里的几个用例,完整版在 src/vertex/tests/maas/、src/vertex/tests/xai/)。e2e 里其余没有标记的测试(transcription 的 region 校验、executable 凭据、partner project、user agent 规则、4 个 gemini_*)不对应 @ai-sdk/google-vertex 的用例,是 aimux 自己的测试(gemini_* 的行为在 @ai-sdk/google 里测)。

harness 上的差别(期望值都没改):

  • 上游把模型类换成 vi.fn() 查构造参数(provider、baseURL、headers、generateId)。aimux 的模型配置是私有的,这里从模型和它发出的请求上看同样的值:provider()、请求 URL 的前缀、注入的 access token 出现在 Authorization 头里、自定义 generateId 出现在 tool call id 里。
  • Express Mode(API key)下不支持的模型,上游在创建时就报错,aimux 在第一次请求时报,错误文本相同。
  • 鉴权:generateAuthToken 返回 null 时上游的头是 Bearer null,这里让生成器返回字符串 "null",得到同样的头。
  • 嵌入向量用 f32 比;TooManyEmbeddingValuesForCall 对应 InvalidArgument。
  • Gemini 转写用脚本化的 WsConnector 代替上游的 mock WebSocket;TranscriptionResult 没有 usage 字段,上游对 usage 的断言没有对应。
  • user agent 钉住的是 ai-sdk-google-vertex/5.0.100 ai-sdk-provider-utils/5.0.53(上游 mock 成 0.0.0-test),单独比。
  • 快照里值为 undefined 的键在 JSON 里省略;header 名比小写。

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change
  • New provider
  • New / updated binding
  • Docs / RFC
  • Tests(上游测试迁移)

Checklist

  • cargo +1.99 fmt --all -- --check passes
  • cargo +1.99 clippy -p aimux-providers -p aimux-provider-utils -p aimux-ops --lib --tests --all-features -- -D warnings passes
  • cargo +1.99 test -p aimux-providers -p aimux-provider-utils -p aimux-ops --all-features:177 个测试二进制,3024 个全过(aimux-providers lib 339,其中 vertex 199)
  • 合并 master(含 test: port the upstream Mistral tests #295)后(head 7f719f0b):fmt、clippy、边界检查通过,cargo test -p aimux-providers --lib 加 *vertex* 测试程序共 6 个、462 个测试全过;layout.toml 与 master 的 [mistral] 表合并
  • scripts/check_provider_boundaries.sh passes
  • Commit messages follow Conventional Commits

Notes for reviewers


Generated by Claude Code

cunninghamcard-bit and others added 4 commits October 20, 2026 13:44
…d like the AI SDK

A server close before the input audio ended now fails the Gemini Live
transcription stream (Google and Vertex) with `InvalidResponseData`:
"<label> Live transcription WebSocket closed unexpectedly before finishing
(code <code>, reason: <reason>).", the text of the `onClose` handler in
@ai-sdk/google 4.0.87 (google-transcription-model.ts:577) and
@ai-sdk/google-vertex 5.0.100 (google-vertex-gemini-transcription-model.ts:600).
It used to pass on the socket's "websocket closed by peer (code ...)" error.

`aimux_provider_utils::ws::peer_close` reads the close frame back from that
error, so the format stays in ws.rs.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
`VertexImageModel` reports `supports_file_inputs: Some(true)` and
`supports_mask_inputs: Some(false)` for the ids in
`googleVertexImageModelsWithFileInputSupport` (@ai-sdk/google-vertex 5.0.100,
google-vertex-image-model.ts:29-62) and `None` for the rest. It used the
trait default, `None`, for every id.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Ports the tests of @ai-sdk/google-vertex 5.0.100 (RFC-0045) into
aimux-providers/src/vertex/tests/, one Rust file per upstream test file,
one mod per describe level and an `/// upstream:` marker on every test:
199 tests for 199 of the 251 upstream cases. 5 more are covered by old
tests, marked in the next commit.

The other 47 are in aimux-providers/layout.toml (`[vertex]`): 14
sub-feature (#275) and 17 ts-only exclusions, and the 16 cases of the Cloud
Text-to-Speech model, whose source file is `missing` (#275).

Upstream fixtures are copied unchanged into fixtures/, `toMatchSnapshot`
values become snapshots/*.json. fixtures/aimux-test-rsa-key.pem is a test
key: aimux signs the service-account JWT itself where upstream mocks Web
Crypto. `GoogleAuth.options` is widened to `pub(super)` so the provider
tests can read the options the credential resolver was built with.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Old tests in tests/vertex_location_validation_test.rs that assert every
expectation of an upstream case move, unchanged, into mod trees named after
its describe chain and carry its `/// upstream:` marker (5). Tests that
cover a case only in part get a `/// partial:` note pointing at the full
port (the `/// TS:` lines they had are replaced).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
cunninghamcard-bit and others added 2 commits October 24, 2026 09:52
…i-sdk

ws.rs and google/transcription.rs resolve to the same tree as the Google fix PR #362 after its master merge: master's ws_error takes the error source, and closed_before_finishing builds InvalidResponseData with AiMuxError::invalid_response_data.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…est/port-the-upstream-google-vertex-tests

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@cunninghamcard-bit
cunninghamcard-bit marked this pull request as ready for review October 24, 2026 11:23
@cunninghamcard-bit
cunninghamcard-bit changed the base branch from fix/vertex-image-and-live-transcription-match-the-ai-sdk to master October 25, 2026 08:10
…eam-google-vertex-tests

# Conflicts:
#	aimux-providers/tests/google_vertex_image_test.rs

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Vertex test port:199 个测试/199 个 marker、fixtures/snapshots 精确;但 PR 当前 DIRTY,且重复了已合入 #351 的测试与 fix commit。请 rebase 到 master,保留 master 的测试并补 partial marker。

cunninghamcard-bit and others added 2 commits October 25, 2026 08:40
eric's review of #352: four old tests check part of an upstream case
without saying so. They now carry `partial:` notes naming the cases and
their full ports: should_generate_video (vertex_video_test.rs), and in
vertex_upstream_e2e_test.rs the MaaS and xAI sub-provider tests and
partner_headers_win_over_per_call_headers.

The other unmarked old tests have no upstream test case: the region option
check, executable credentials, the partner project and user-agent rules, and
the Gemini behaviours that @ai-sdk/google tests rather than
@ai-sdk/google-vertex. They stay aimux-own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5

Copy link
Copy Markdown
Contributor Author
eric 的意见(原话) 处理
但 PR 当前 DIRTY,且重复了已合入 #351 的测试与 fix commit。 已合并 master(b7ce78a3)。之后 master 合入了 #295,又产生 layout.toml 冲突,所以再合并一次(7f719f0b),冲突的解决方式是保留 master 的 [mistral] 表,再加上本 PR 的 [vertex] 表。现在相对 master 的 diff 不再带 #351 的 fix 提交和 CHANGELOG,只有测试、fixtures、layout.toml、src/test_support/ 和边界脚本这几处。
请 rebase 到 master,保留 master 的测试并补 partial marker。 用的是合并而不是 rebase:分支已经推送过,合并不改写历史。master 上 #351 的两个测试按 master 的版本保留:google_vertex_image_test.rs 的能力测试已标 partial:,google_transcription_test.rs 属于 Google,不在 Vertex 的用例范围内。另外补了 4 处 partial:(3095391e):vertex_video_test.rs 的 should_generate_video,以及 e2e 里 MaaS、xAI 子 provider 和 partner headers 的 3 个测试,它们各自对应 src/vertex/tests/ 里几个完整用例的一部分。e2e 里其余没有标记的测试(region 校验、executable 凭据、partner project、user agent、4 个 gemini_*)不对应 @ai-sdk/google-vertex 的用例,是 aimux 自己的测试。现在共有 37 处 partial:。
Vertex test port:199 个测试/199 个 marker、fixtures/snapshots 精确 新测试没有改。合并后 lib + *vertex* 共 462 个测试全过,fmt、clippy 和边界检查也都通过。PR 描述已同步。

Generated by Claude Code


Generated by Claude Code

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase 后 clean,重复的 #351 测试已移除,diff 为 test-only;199 tests/200 markers,fixtures 精确。Approve。

@eric8810
eric8810 merged commit 209be26 into master Oct 25, 2026
16 checks passed
cunninghamcard-bit pushed a commit that referenced this pull request Oct 25, 2026
Master now carries the Google Vertex test port (#352, squashed). The merge
uses the test branch head 7f719f0 as its base, so only master's own
changes since then come in; nothing conflicted.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants