Repository navigation
test: port the upstream Amazon Bedrock tests - #358
Conversation
@ai-sdk/amazon-bedrock 5.0.100 decodes each frame with @smithy/eventstream-codec (amazon-bedrock-event-stream-decoder.ts:42), whose splitMessage throws `The prelude checksum specified in the message (…) does not match the calculated CRC32 checksum (…)` and `The message checksum (…) did not match the expected value of …`. decode_stream reported both as `Invalid Amazon Bedrock event-stream frame`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/amazon-bedrock 5.0.100 rejects a provider-reference file part as `file parts with provider references`, checks a URL file for `s3:` before its media type and rejects a non-S3 or non-image/video URL as `File URL data` (convert-to-amazon-bedrock-chat-messages.ts:238-290), and rejects a tool-result file that is not data or an S3 image/video URL as `tool result file data of type "<type>"` (:612-650). aimux said `File reference data`, checked the media type first and said `file media type or data: …`, and said `tool result file data`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/amazon-bedrock 5.0.100 returns the usage object itself as `raw` (convert-amazon-bedrock-usage.ts:42). convert_usage re-serialised the typed BedrockUsage, so explicit `null` keys were lost; doGenerate and doStream patched `raw` afterwards from the response JSON. BedrockUsage now keeps the object it was parsed from, and its field errors name the field. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/amazon-bedrock 5.0.100 takes `part.toolUse?.toolUseId || this.config.generateId()` (amazon-bedrock-chat-language-model.ts:796-797), so an empty id is replaced. aimux's doGenerate kept the empty id; its stream already generated one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
… topP @ai-sdk/amazon-bedrock 5.0.100 drops temperature, topK and topP for a model that rejects sampling parameters, warning in that order, before clamping and the thinking checks (amazon-bedrock-chat-language-model.ts:204-230). aimux handled the three in one pass with the other checks and warned temperature, topP, topK. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/amazon-bedrock 5.0.100 warns `maxOutputTokens is not supported by
${this.modelId} when high reasoning is enabled and will be ignored`
(amazon-bedrock-chat-language-model.ts:511). aimux's warning left out the
model id.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
With no fetch setting, @ai-sdk/amazon-bedrock 5.0.100 calls `globalThis.fetch` as it is when each request is made (amazon-bedrock-sigv4-fetch.ts:38, :129), and the settings' own doc says the process default is resolved per request. create_amazon_bedrock instead captured default_fetch() once at creation, so a later set_default_transport never reached the provider's chat, embedding, image or reranking models. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Ports the tests of @ai-sdk/amazon-bedrock 5.0.100 (RFC-0045) into aimux-providers/src/bedrock/tests/, one Rust file per upstream test file, one mod per describe level and an `/// upstream:` marker on every test: 408 new tests. With the 28 old tests marked in the next commit, 436 of the 457 upstream cases outside the Bedrock-Anthropic and Mantle sub-providers have a test. aimux-providers/layout.toml (`[bedrock]`) lists the other 21: 3 registered (SigV4 also signs GET and bodyless requests), 4 sub-feature (#275), 10 ts-only, and the 4 cases of the inject-fetch-headers test helper, which is not-applicable; the anthropic/ files are not-applicable (alignment doc §5) and the mantle/ files missing (#275). Upstream fixtures are copied unchanged into fixtures/ and reranking/fixtures/, the `toMatchSnapshot` values become snapshots/*.json. `convert_prompt` and the chat and embedding models' `model_family` are widened to `pub(super)` for the tests. The provider tests that emulate upstream's settings mocks set AWS_* variables, and two SigV4 tests swap the process default transport; all of them are `#[serial]`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Old tests that cover a whole upstream case move into mods mirroring its describe chain and get its `/// upstream:` marker (28: 16 in bedrock_convert_test.rs, 9 in bedrock_remaining_test.rs, 2 in bedrock_embedding_test.rs, 1 in amazon_bedrock_image_test.rs). Old tests that cover part of a case get a `/// partial:` line naming the case and the file with its full port (87 lines, one of them in event_stream.rs). Test names and bodies are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…ort-match-the-ai-sdk Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…e-ai-sdk into test/port-the-upstream-amazon-bedrock-tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…ort-match-the-ai-sdk Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
eric8810
left a comment
There was a problem hiding this comment.
Bedrock test port:408 新测试 + 28 旧测试 + 21 exclusions = 457 覆盖,fixtures 逐字节一致,无 runtime 变化;layout 中 error.ts 的映射应改到 bedrock/mod.rs 或后续 error.rs。Approve。
e4ecf49 reports both @smithy/eventstream-codec checksum errors, but only the message checksum had a test. stream_prelude_checksum_mismatch_error flips a frame's prelude CRC and expects the codec's text ("The prelude checksum specified in the message (…) does not match the calculated CRC32 checksum (…)"); with master's decoder it fails with InvalidResponseData("Invalid Amazon Bedrock event-stream frame"). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…equest 9abb06e's test covers the chat model only. The embedding, image and reranking models share the same fetch (amazon-bedrock-provider.ts:207-215 hands one fetchFunction to every model, :334, :359, :367, :380). non_chat_models_look_up_the_default_transport_per_request creates the provider while the default transport points at a closed port, then sends an embedding, an image and a reranking request through a proxy set afterwards; without 9abb06e the embedding request fails. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…t.toml bedrock_failed_response_handler, the port of amazonBedrockFailedResponseHandler, lives in bedrock/mod.rs, not bedrock/model.rs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Generated by Claude Code Generated by Claude Code |
…gs-usage-and-transport-match-the-ai-sdk' into test/port-the-upstream-amazon-bedrock-tests
…errors-warnings-usage-and-transport-match-the-ai-sdk # Conflicts: # CHANGELOG.md
…match-the-ai-sdk' into test/port-the-upstream-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
…eam-amazon-bedrock-tests
eric8810
left a comment
There was a problem hiding this comment.
Verified the post-approval commits on this head: only additive tests/docs or tightened boundary checks; no weakening or behavior regression. Re-approving the current head.
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase is content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; tests and CI unchanged. Re-approving current head.
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; CI 16/16. Re-approving current head.
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-amazon-bedrock-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.
Summary
按 RFC-0045(#281)把
@ai-sdk/amazon-bedrock@5.0.100的上游测试迁完,按厂商分批的第十二个(Mistral #295、Cohere #296、DeepSeek #299、Groq #301、Hugging Face #303、xAI #312、Anthropic AWS #319、Azure #324、Open Responses #347、Google Vertex #352、OpenAI-compatible #355)。上游 21 个测试文件共 518 个用例。
anthropic/和mantle/两个子 provider 的 61 个不在范围里:Bedrock-Anthropic(InvokeModel 走 Anthropic 消息体)aimux 没做,是对齐文档 §5 登记的产品选择;Mantle 在 aimux 里只是 registry 预设,做成完整 provider 挂在 #275。其余 457 个:src/bedrock/tests/,一个上游测试文件对应一个 Rust 文件。/// upstream:标记,函数名和函数体不变;另有 87 处旧测试只覆盖了一部分,加了/// partial: …指向完整用例。这部分单独一个提交。layout.toml:3 个registered(SigV4 也给 GET 和无 body 的请求签名,§9.1 已知差距),4 个sub-feature(Provider 构造与配置:迁移后遗留缺口 #275:没有createAmazonBedrockCachePoint、provider 上没有anthropicTools),10 个ts-only,另外inject-fetch-headers.ts是上游包内的测试 helper(不导出),条目是not-applicable,它的 4 个用例不迁,helper 本身在tests/support.rs里留了一份 Rust 版给嵌入、图像、重排序的测试用。迁移中发现 7 处行为差异,修在 #357 里,一处一个提交,各带一个在 master 上失败的测试。#357 已合入 master:
e4ecf494@smithy/eventstream-codec的原文,不再是Invalid Amazon Bedrock event-stream frame469c5403file parts with provider references、File URL data(先查s3:再查媒体类型)、tool result file data of type "…"38bc4a3cconvert_usage的raw保留 Bedrock 发来的原对象,显式的null键不再丢cb6aab13toolUseId为空时生成一个 id(流式本来就这样)4a3354c3866ce154maxOutputTokens的警告带上模型 id9abb06ebfetch时,每次请求再取进程默认 transport(同 #318 对 Anthropic AWS 的改法)一、体系概览:Amazon Bedrock 的测试现在怎么跑(master
e115e723)上游
packages/amazon-bedrock/src/的 21 个测试文件(用例数):二、本 PR 的改动(细节)
1. 新测试目录(
bedrock/mod.rs末尾#[cfg(test)] mod tests;,:723)2. 这些测试动到的进程级状态
新测试编进
aimux-providers的 lib 测试二进制,和其他模块的单元测试同进程并行跑。provider.rs里模拟上游 settings mock 的用例会设置或清掉AWS_REGION、AWS_ACCESS_KEY_ID、AWS_SECRET_ACCESS_KEY、AWS_SESSION_TOKEN、AWS_BEARER_TOKEN_BEDROCK、AWS_ENDPOINT_URL、AWS_ENDPOINT_URL_BEDROCK_RUNTIME、AWS_ENDPOINT_URL_BEDROCK_AGENT_RUNTIME,用完恢复原值。sigv4_fetch.rs的 2 个用例用set_default_transport换成本地假代理,测完换回默认。#[serial](共 26 个),和库里其他#[serial]用例互斥;其余新测试不读这些变量,也不走默认 transport(都传自己的MockFetch)。没有全局计数器,不写临时文件。3.
aimux-providers/layout.toml(只加[bedrock],35 个上游源文件)amazon-bedrock-chat-language-model-event-stream.test.ts和amazon-bedrock-usage-raw.test.ts没有同名源文件,用例全迁了,不需要写排除。4. 旧测试的标记(单独一个提交,函数名和函数体不变)
harness 上的差别(期望值都没改,除了下面写明的错误文字比法):
new BedrockChatLanguageModel(...);这里用create_amazon_bedrock加 API key、region、base_url、fetch建。上游 mock 掉模型类、fetch 工厂和 settings loader 的用例,改成发一次请求,从线上看到的 URL、头和签名断言 mock 本该收到的参数。…Exception用:message-type exception)。message、type、statusCode、isRetryable、data)。should handle failed chunk parsing上游经 mock 注入错误值,Rust 注入不了,改用一个 payload 不是 JSON 的帧,两边都不比错误值本身。UnsupportedFunctionality只带一个字符串,即上游的functionality,没有单独的message:上游默认文案'X' functionality not supported.和三个 mime type 错误的自定义 message,都比抛出时的 functionality。inputTokens: invalid type: …),不是 zod 的 issue 列表,所以只查 cause 里有字段名,不要求带引号。generateId、currentDate在 Rust 里注入不了:空toolUseId用例先断言生成的 id 非空再比其余字段;图像时间戳断言落在调用期间。ai-sdk-amazon-bedrock/5.0.100(上游 mock 成0.0.0-test),单独比。快照里值为undefined的键在 JSON 里省略。Type of change
Checklist
cargo +1.99 fmt --all -- --checkpassescargo +1.99 clippy -p aimux-providers -p aimux-provider-utils -p aimux-ops --lib --tests --all-features -- -D warningspassescargo +1.99 test -p aimux-providers -p aimux-provider-utils -p aimux-ops --all-features:177 个测试二进制,3240 个全过(aimux-providers lib 550,其中 bedrock 414:新 408、原有 6)scripts/check_provider_boundaries.shpassesNotes for reviewers
src/test_support/mod.rs、aimux-providers/layout.toml,并改了边界脚本规则 7c、8b 的同两行;这几处在各 PR 里内容相同,后合入的只需合并layout.toml的厂商表。layout.toml里。5a4bbf7e,master 5157fdc);layout.toml是 master 的原文后接 bedrock 表。Generated by Claude Code