Skip to content

refactor: move the Google files to the upstream layout - #367

Merged
eric8810 merged 10 commits into
masterfrom
refactor/move-the-google-files-to-the-upstream-layout
Oct 27, 2026
Merged

eric8810 merged 10 commits into
masterfrom
refactor/move-the-google-files-to-the-upstream-layout

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

把 src/google/ 按 @ai-sdk/google 一个文件对一个文件重排(RFC-0046 §4.2),照上游建 transcription/ 子目录。九个老测试程序和 utils.rs 里的内联测试并进厂商测试目录 src/google/tests/(RFC-0046 §5)。函数体只改了路径和可见性。crate 根和 google 模块根的再导出不变;google 下有公开模块路径变了,列在“公开路径”一节。测试数量和函数名前后一致;tests/files.rs 的 mock 服务器改成不走 wiremock 的池(见“共享进程状态”)。叠在测试 PR #363 上。实施进度在 #328 跟踪。

已经测试分支 b49a3ba3 合上 master b69137eb,冲突按 master 的改动搬到新位置解决(convert.rs 的改动落到 language_model.rs、language_model_options.rs,speech、transcription 的改动落到各自的模型和 options 文件以及 get_realtime_base_url.rs,老测试程序里的错误模式改成结构体变体);测试数前后仍相等(584 个:lib 569 个加留在 tests/ 的 15 个,名字多重集相同)。

一、体系现状(概览,测试 PR #363 头 54e6fd31)

aimux_providers::google::create_google              src/google/mod.rs:239
 └ GoogleProvider::model_config                     mod.rs:296
   ├ chat → GoogleModel                             mod.rs:313 → model.rs:43
   │  ├ do_generate / do_stream                     model.rs:167、:245
   │  ├ build_request_body_with_warnings_for_namespace(getArgs)
   │  │                                             convert.rs:1028
   │  │  ├ validate_call_options_for_namespace      convert.rs:1353(providerOptions 校验从 :1430 起)
   │  │  ├ get_google_model_capabilities            convert.rs:637
   │  │  ├ convert_to_google_messages_for_namespace convert.rs:86
   │  │  ├ prepare_all_tools                        convert.rs:730
   │  │  └ sanitize_response_json_schema            convert.rs:954
   │  ├ GoogleLanguageModelStream(Transformer)     model.rs:763
   │  │  └ GoogleJsonAccumulator                    utils.rs:121
   │  ├ extract_content_from_candidate              model.rs:905
   │  ├ parse_finish_reason                         convert.rs:1704
   │  ├ convert_usage                               convert.rs:1736
   │  ├ extract_sources                             convert.rs:1776
   │  ├ google_stream_error                         mod.rs:67
   │  └ google_failed_response_handler             mod.rs:95
   ├ embedding → GoogleEmbeddingModel               mod.rs:331 → embedding.rs:25
   ├ image → GoogleImageModel                       mod.rs:340 → image.rs:32(GoogleImageSettings :28)
   ├ video → GoogleVideoModel                       mod.rs:361 → video.rs:26
   ├ speech → GoogleSpeechModel                     mod.rs:368 → speech.rs:230
   │  ├ parse_options                               speech.rs:115
   │  └ google_speech_input                         speech.rs:152
   ├ transcription → GoogleTranscriptionModel       mod.rs:380 → transcription.rs:173
   │  ├ TranscriptionOptions                        transcription.rs:64
   │  ├ live_web_socket_url                         transcription.rs:399
   │  └ live_stream                                 transcription.rs:451
   ├ files → GoogleFiles                            mod.rs:391 → files.rs:137
   └ decision_model                                 mod.rs:440
ProviderDiscovery::list_models → list_models_once   mod.rs:458 → :474
supported_urls                                      mod.rs:155

上游同一条路:createGoogle(google-provider.ts)建 GoogleLanguageModel(google-language-model.ts,getArgs、doGenerate、doStream 和流的解析都在这个文件)、GoogleEmbeddingModel、GoogleImageModel、GoogleVideoModel、GoogleSpeechModel、transcription/ 下的 GoogleTranscriptionModel、GoogleFiles;提示词转换、工具准备、能力表、schema 清理、finish reason、usage、错误、API 类型、选项 schema、JSON 累加器各在自己的文件。aimux 把这些放在 11 个文件里:mod.rs(provider、设置、错误处理、模型列表)、model.rs、convert.rs(提示词转换、工具、能力表、schema 清理、getArgs、选项校验、finish reason、usage、sources)、types.rs(API 类型和错误信封)、utils.rs(get_model_path、is_supported_file_url、GoogleJsonAccumulator)、embedding.rs、image.rs、video.rs、speech.rs、transcription.rs、files.rs,加上 aimux 自己的 options.rs。

测试:#363 的新用例在 src/google/tests/(google:: 下 439 个,含 utils.rs 的 1 个内联测试);老用例在九个独立测试程序里:tests/google_embedding_test.rs(7)、tests/google_evaluation_test.rs(7)、tests/google_factory_test.rs(6)、tests/google_files_test.rs(17)、tests/google_image_test.rs(20)、tests/google_speech_test.rs(40)、tests/google_transcription_test.rs(16)、tests/google_upstream_e2e_test.rs(13)、tests/google_video_test.rs(4)。

二、本 PR 改动(细节)

aimux_providers::google::create_google              src/google/provider.rs:161               ← google-provider.ts
 └ GoogleProvider::model_config                     provider.rs:218
   ├ chat → GoogleModel                             provider.rs:235 → language_model.rs:48   ← google-language-model.ts
   │  ├ do_generate / do_stream                     language_model.rs:172、:250
   │  ├ build_request_body_with_warnings_for_namespace(getArgs)
   │  │                                             language_model.rs:1209
   │  │  ├ validate_call_options_for_namespace      language_model.rs:1534
   │  │  │  └ validate_provider_options             language_model_options.rs:9             ← google-language-model-options.ts
   │  │  ├ get_google_model_capabilities            model_capabilities.rs:24                 ← google-model-capabilities.ts
   │  │  ├ convert_to_google_messages_for_namespace convert_to_messages.rs:44                ← convert-to-google-generative-ai-messages.ts
   │  │  ├ prepare_all_tools                        prepare_tools.rs:97                      ← google-prepare-tools.ts
   │  │  └ sanitize_response_json_schema            sanitize_response_json_schema.rs:7       ← sanitize-response-json-schema.ts
   │  ├ GoogleLanguageModelStream(Transformer)     language_model.rs:768
   │  │  └ GoogleJsonAccumulator                    json_accumulator.rs:82                   ← google-json-accumulator.ts
   │  ├ extract_content_from_candidate              language_model.rs:910
   │  ├ parse_finish_reason                         map_finish_reason.rs:10                  ← map-google-generative-ai-finish-reason.ts
   │  ├ convert_usage                               convert_usage.rs:11                      ← convert-google-generative-ai-usage.ts
   │  ├ extract_sources                             language_model.rs:1733
   │  ├ google_stream_error                         error.rs:25                              ← google-error.ts
   │  └ google_failed_response_handler             error.rs:53
   ├ embedding → GoogleEmbeddingModel               provider.rs:253 → embedding_model.rs:25  ← google-embedding-model.ts
   ├ image → GoogleImageModel                       provider.rs:262 → image_model.rs:29      ← google-image-model.ts
   │                                                (GoogleImageSettings image_settings.rs:4 ← google-image-settings.ts)
   ├ video → GoogleVideoModel                       provider.rs:283 → video_model.rs:26      ← google-video-model.ts
   ├ speech → GoogleSpeechModel                     provider.rs:290 → speech_model.rs:33     ← google-speech-model.ts
   │  ├ parse_options                               speech_model_options.rs:93               ← google-speech-model-options.ts
   │  ├ google_speech_input                         speech_input.rs:19                       ← google-speech-input.ts
   │  └ SpeechResponseBody 等                        speech_api.rs:32                         ← google-speech-api.ts
   ├ transcription → GoogleTranscriptionModel       provider.rs:302 → transcription/transcription_model.rs:69
   │                                                ← transcription/google-transcription-model.ts
   │  ├ TranscriptionOptions                        transcription/transcription_model_options.rs:43
   │  │                                             ← transcription/google-transcription-model-options.ts
   │  ├ live_web_socket_url                         get_realtime_base_url.rs:8               ← get-realtime-base-url.ts
   │  └ live_stream                                 transcription/transcription_model.rs:329
   ├ files → GoogleFiles                            provider.rs:313 → files.rs:137           ← google-files.ts
   └ decision_model                                 provider.rs:362
ProviderDiscovery::list_models → list_models_once   provider.rs:380 → :396
supported_urls                                      provider.rs:77
现在 搬到 上游
mod.rs 的模块文档、mod、pub use mod.rs(只有模块声明和导出) index.ts、internal/index.ts
mod.rs 的常量、EXTERNAL_URL_MEDIA_TYPES、supported_urls、GoogleProviderSettings、create_google、google、GoogleProvider 和 Provider / ProviderDiscovery 实现、list_models_once provider.rs google-provider.ts(list_models_once 是 aimux 的模型发现,跟 provider 放一起)
mod.rs 的 google_stream_error、google_failed_response_handler;types.rs 的 GoogleErrorEnvelope、GoogleError error.rs google-error.ts
types.rs 其余 api_types.rs google-api-types.ts(RFC-0046 §4.2:zod 响应 schema 放在 api_types)
model.rs;convert.rs 的 build_vertex_request_body、build_request_body_with_warnings*、validate_call_options*、validate_file_part、resolve_thinking、code_execution_tool_name、extract_sources language_model.rs google-language-model.ts(getArgs、extractSources 都在这个文件)
convert.rs 里 validate_call_options_for_namespace 的 providerOptions 校验(原 :1430 起) language_model_options.rs 的 validate_provider_options google-language-model-options.ts(上游 zod schema 的手写校验)
convert.rs 的提示词转换、contains_schema_reference convert_to_messages.rs convert-to-google-generative-ai-messages.ts
convert.rs 的 GooglePrompt prompt.rs google-generative-ai-prompt.ts
convert.rs 的 prepare_tools、prepare_all_tools、build_function_declaration prepare_tools.rs google-prepare-tools.ts
convert.rs 的 GoogleModelCapabilities、get_google_model_capabilities model_capabilities.rs google-model-capabilities.ts
convert.rs 的 sanitize_response_json_schema sanitize_response_json_schema.rs sanitize-response-json-schema.ts
convert.rs 的 parse_finish_reason map_finish_reason.rs map-google-generative-ai-finish-reason.ts
convert.rs 的 convert_usage convert_usage.rs convert-google-generative-ai-usage.ts
utils.rs 的 get_model_path / is_supported_file_url / GoogleJsonAccumulator get_model_path.rs / supported_file_url.rs / json_accumulator.rs get-model-path.ts / google-supported-file-url.ts / google-json-accumulator.ts
embedding.rs、video.rs embedding_model.rs、video_model.rs(原样) google-embedding-model.ts、google-video-model.ts(它们的 *-options.ts / *-settings.ts 在 aimux 里没有单独的内容,留在模型文件里)
image.rs 的 GoogleImageSettings / 其余 image_settings.rs / image_model.rs google-image-settings.ts / google-image-model.ts
speech.rs 的选项结构和 parse_options / GoogleSpeechInput、google_speech_input / 响应结构 / 其余 speech_model_options.rs / speech_input.rs / speech_api.rs / speech_model.rs google-speech-model-options.ts / google-speech-input.ts / google-speech-api.ts / google-speech-model.ts
transcription.rs 的 live_web_socket_url get_realtime_base_url.rs(cfg(feature = "realtime")) get-realtime-base-url.ts
transcription.rs 的 TranscriptionOptions、parse_offset_seconds / 其余 transcription/transcription_model_options.rs / transcription/transcription_model.rs transcription/google-transcription-model-options.ts / transcription/google-transcription-model.ts
files.rs 不动 google-files.ts
options.rs 不动 aimux 独有:Gemini 和 Vertex 共用的 providerOptions / providerMetadata 命名空间(layout.toml 不列)
  • 公开路径(RFC-0046 §4.5,按 refactor: move the Mistral files to the upstream layout #297 / refactor: move the Cohere files to the upstream layout #298 / refactor: move the xAI files to the upstream layout #313 / refactor: move the Amazon Bedrock files to the upstream layout #359 / refactor: move the Anthropic files to the upstream layout #364 的做法只列出来):
    • google::embedding、google::image、google::speech、google::video 不再公开;GoogleEmbeddingModel、GoogleImageModel、GoogleImageSettings、GoogleSpeechModel、GoogleVideoModel 仍可从 google:: 和 crate 根拿到。
    • google::convert、google::types、google::utils 不再公开:convert_to_google_messages、build_request_body、build_request_body_with_warnings、prepare_tools、prepare_all_tools、get_google_model_capabilities、sanitize_response_json_schema、parse_finish_reason、convert_usage、extract_sources、响应类型、get_model_path、is_supported_file_url、GoogleJsonAccumulator 变成 crate 内部,没有公开的替代。speech::google_speech_input 同样变成 crate 内部。
    • 没有任何调用方的两个包装函数 convert::convert_to_google_messages、convert::build_request_body 删掉了(模块不公开后它们是死代码;文档合到 convert_to_google_messages_for_namespace、build_request_body_with_warnings 上)。只有测试用的 get_model_path、is_supported_file_url 现在是 #[cfg(test)] pub(super)(aimux 的请求路径不调用上游的 getModelPath);ProcessResult.current_json 只有测试读,按 xai 的先例加了 cfg_attr(not(test), allow(dead_code, reason = …))。
    • google::files、google::transcription 照旧公开,google::transcription::GoogleTranscriptionModel 照旧。
  • crate 内的调用方不用改:Vertex 用的 google::convert::build_vertex_request_body 由 mod.rs 的 pub(crate) use language_model as convert; 接住,google_failed_response_handler 由 pub(crate) use error::google_failed_response_handler; 导出,default_providers.rs 的 API_KEY_ENV_VAR / DEFAULT_BASE_URL 和 Vertex 的 list_models_once 由 pub(crate) use provider::{…} 导出,google::options::* 不动,Vertex 转写用的 TranscriptionOptions、parse_offset_seconds、live_stream、validate_live_input_audio_format 由 transcription/mod.rs 的 pub(crate) use 导出。src/vertex/ 一个文件都没改,Vertex 自己的搬移 PR 因此不会和这里在 import 上冲突。
  • 可见性:拆到兄弟文件后要互相调用的项改为 pub(super):matches_prefix_boundary、speech_api.rs 和 speech_model_options.rs 的结构体及字段、parse_options、live_web_socket_url、interactions_config、validate_provider_options;language_model 模块改为 pub(crate)(给 convert 别名用),其余新模块私有。transcription_model.rs 里的 super:: 改成 crate::google::(多了一层目录)。
  • 测试:
    • 每个老程序并进对应上游文件的测试,放在末尾一个 older_* 模块里:tests/google_embedding_test.rs → tests/embedding_model.rs 的 older_embedding;google_evaluation_test.rs → tests/evaluation.rs 的 older_evaluation;google_factory_test.rs → tests/provider.rs 的 older_factory;google_files_test.rs → tests/files.rs 的 older_files;google_image_test.rs → tests/image_model.rs 的 older_image;google_transcription_test.rs → tests/transcription/transcription_model.rs 的 older_transcription;google_upstream_e2e_test.rs → tests/language_model.rs 的 older_upstream_e2e;google_video_test.rs → tests/video_model.rs 的 older_video。
    • google_speech_test.rs 按它的上游来源拆:getGoogleSpeechInput 那节 7 个进 tests/speech_input.rs 的 older_speech_input,其余 33 个进 tests/speech_model.rs 的 older_speech。
    • utils.rs 的内联测试进 tests/json_accumulator.rs 的 older_utils。
    • 每节前一行注释写明原来的位置,老文件头的说明改成普通注释。mock_fetch 改用 crate::test_support::mock_fetch,aimux_providers:: 改成 crate::。test: port the upstream Google tests #363 测试里的 convert::、types::、utils::、speech:: import 改到新路径。断言都没改。
  • 留在 tests/ 的:tests/google_pse_test.rs(另一个厂商)、tests/google_vertex_image_test.rs(Vertex 的)和其余跨厂商的测试程序。
  • layout.toml 的 [google] 29 个 rust 字段改到新路径,两条 reason 里的 google/convert.rs、google/transcription.rs 改成 google/prepare_tools.rs、google/transcription/transcription_model.rs。scripts/check_provider_boundaries.sh 不用改,照样通过。

共享进程状态:老测试原来各是单独的测试程序,现在和 #363 的用例同在 lib 的一个测试程序里。

  • 32 线程下连跑 40 次,有 4 次 older_files / older_image 的用例在 received_requests 里多看到一个 1 字节的上传请求。原因是 test: port the upstream Google tests #363 的 tests/files.rs 的 should_reject_promptly_when_aborted_during_the_polling_delay 在请求途中丢掉客户端,create_mock_files 用的是 wiremock 池里的服务器(MockServer::start()),晚到的请求落在已经还回池、又被下一个用例拿走的服务器上。修法是改那个泄漏的 setup:create_mock_files(tests/files.rs:142)改用 MockServer::builder().start(),自己一个不进池的服务器,加了注释说明。改后 32 线程连跑 100 次(60 + 40)没有再出现。用例本身没改,没加 #[serial]。
  • 环境变量:lib 测试里没有用例设 GOOGLE_GENERATIVE_AI_API_KEY,Google 也没有 base URL 的环境变量,没有相互影响。没有全局计数器,没有换进程默认 transport 的用例;fixture 路径按 CARGO_MANIFEST_DIR 算,位置不影响。

CHANGELOG

CHANGELOG Unreleased 的 Breaking 末尾加了一条,列出本家变了的公开子模块路径和替代入口(RFC-0046 §4.5 第 2 条):

- Google's sub-modules follow the `@ai-sdk/google` files (RFC-0046 §4.2):
  `google::embedding`, `google::image`, `google::speech`, `google::video`,
  `google::convert`, `google::types` and `google::utils` are no longer public.
  The models and settings stay at the `google` root and the crate root:
  `google::{GoogleEmbeddingModel, GoogleImageModel, GoogleImageSettings,
  GoogleSpeechModel, GoogleVideoModel}`. The request and response helpers of
  `convert`, `types` and `utils` (`convert_to_google_messages`,
  `build_request_body`, `build_request_body_with_warnings`, `prepare_tools`,
  `prepare_all_tools`, `get_google_model_capabilities`,
  `sanitize_response_json_schema`, `parse_finish_reason`, `convert_usage`,
  `extract_sources`, the response types, `get_model_path`,
  `is_supported_file_url`, `GoogleJsonAccumulator`) and
  `speech::google_speech_input` become crate-internal, with no public
  replacement. `google::files` and `google::transcription` keep their paths.

验证

  • cargo test -p aimux-providers --lib 加原来九个 --test:前后都是 710 个测试(前 lib 580 + 130,后 lib 710),函数名多重集相同;google:: 下 439 → 569。改了 create_mock_files 之后 lib 测试程序默认线程连跑 20 次、32 线程连跑 100 次,710 个全过。
  • cargo +1.99 clippy -p aimux-providers --lib --tests -- -D warnings、cargo +1.99 check -p aimux-providers --lib --tests --all-features、RUSTDOCFLAGS="-D warnings" cargo +1.99 doc -p aimux-providers --no-deps、cargo fmt --all -- --check、scripts/check_provider_boundaries.sh、cargo +1.99 check --workspace、cargo test -p aimux-providers(2817 过,5 ignored)通过。
  • --no-default-features 下有一条 LIVE_WEB_SOCKET_PATH is never used 的警告,test: port the upstream Google tests #363 上就有,没动。

Type of change

  • Refactor (no behavior change)

Checklist

  • Tests pass locally
  • Commit messages follow Conventional Commits

https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW


Generated by Claude Code

Split src/google/ into one file per @ai-sdk/google 4.0.87 source file
(RFC-0046 §4.2), with transcription/ as upstream. Function bodies only
change paths and visibility; the crate-root and google-root exports are
unchanged, and the sub-modules that were public are listed in the
CHANGELOG. The old tests/google_*_test.rs programs and the inline
utils test move into src/google/tests/ with equal test names and counts.

The files tests use a wiremock server of their own: the abort case drops
the client mid-request, and a request landing after the test ended reached
the next test that took the pooled server once all tests share one binary.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
@cunninghamcard-bit
cunninghamcard-bit marked this pull request as ready for review October 24, 2026 11:35

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Google layout refactor:584→584 tests,move-only,公开 re-export 保留;仅需修正 audit 文档里指向已删除 google_files_test.rs 的链接。Approve。

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW

Copy link
Copy Markdown
Contributor Author
eric 的意见(原话) 处理
「仅需修正 audit 文档里指向已删除 google_files_test.rs 的链接」 已改,c2b292a4:docs/internal/audit/rust-architecture-audit-2026-07-31.md 第 245 行的链接改指 aimux-providers/src/google/tests/files.rs#L1061-L1149,也就是原来那三个上传错误用例(初始化失败、没有上传地址、上传失败)现在所在的 error_handling 模块。第 235 行的 google_files_test:2 是当时测试程序名的失败计数,不是链接,没动。

Generated by Claude Code

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW

@eric8810 eric8810 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the post-approval commits on this head: only additive tests/docs or tightened boundary checks; no weakening or behavior regression. Re-approving the current head.

…sts' into refactor/move-the-google-files-to-the-upstream-layout

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0116HPMoqaLfFr1TyqDq1hrW
@eric8810
eric8810 changed the base branch from test/port-the-upstream-google-tests to master October 27, 2026 04:37
@eric8810
eric8810 merged commit a23a636 into master Oct 27, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants