Repository navigation
test: port the upstream Anthropic tests - #361
Conversation
@ai-sdk/anthropic 4.0.71 drops the caller of assistant blocks before the last user message whose source code execution server_tool_use is not among the emitted blocks, warning once per tool call (convert-to-anthropic-prompt.ts:1463-1510). aimux sent the caller as is. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 warns `provider executed tool result output type <type> for tool <name> is not supported` for the MCP, code execution, web fetch/search, tool search and advisor results it cannot send back (convert-to-anthropic-prompt.ts:1029-1032, 1380-1383), and `provider executed tool result for tool <name> is not supported` for any other tool (:1435-1438). aimux left out the type and used the former for unknown tools. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 pushes the mcp_tool_result block without breaking out of the tool-result case, so it falls through to the `provider executed tool result for tool <name> is not supported` warning (convert-to-anthropic-prompt.ts:1025-1046, 1435-1438). aimux sent the block without the warning. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 turns a response `fallback` block into a custom `anthropic.fallback` part in doGenerate and doStream (anthropic-language-model.ts:1724-1737, 1976-1989) and converts that part back into a `fallback` block, a thinking-order boundary, warning on invalid metadata (convert-to-anthropic-prompt.ts:829-853, 1600-1606). aimux sent the `fallbacks` option but dropped the block on both sides. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 copies `content.citations` through as is (anthropic-language-model.ts:1508, :2289; schema anthropic-api.ts:866 makes it optional), so a document without citations has no key, and the prompt converter (convert-to-anthropic-prompt.ts:1266) drops it via JSON.stringify. aimux wrote `"citations": null` in both places. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 maps `providerOptions.anthropic.safeguards` to the body's `safeguards` (`classifierContext` -> `classifier_context`, anthropic-language-model.ts:812-820) and adds `dangerous-tool-use-2026-09-03` (:970-975). aimux validated the option but left it out of the request. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 matches `/claude-sonnet-4(?:-|@)/` and `/claude-opus-4(?:-|@)/` (anthropic-language-model.ts:3311, :3324), so the Vertex id `claude-sonnet-4@20250514` is known with 64000 max output tokens. aimux matched only `claude-sonnet-4-` / `claude-opus-4-` and treated the `@` ids as unknown (128000 max tokens plus a warning). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 keeps the betas in a Set and joins it as built (anthropic-language-model.ts:1082-1087, :1118): prompt betas, the option betas in the order of :963-1037, tool betas, header betas (config, then request, :1122-1140), then `anthropicBeta`. aimux kept a BTreeSet, so the header came out sorted. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 emits a streamed provider tool call's input as
the accumulated delta text, only re-stringifying a code_execution
`{ code }` input that lacks a `type` (anthropic-language-model.ts:2638-2657).
aimux parsed every code_execution input and re-serialized it compactly.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 has no flush: every `message_stop` enqueues its finish and the stream keeps reading, so a later `message_start` opens the next message (anthropic-language-model.ts:2849-2877, :3031-3087), and the usage accumulator only takes the input side from `message_start`. aimux terminated the stream at the first `message_stop`. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 reads `message.content` of `message_start` and emits each `tool_use` block there (deferred programmatic tool calls) as tool-input-start/delta/end plus a tool-call with its caller metadata (anthropic-language-model.ts:2905-2955). aimux ignored that content. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 throws a first-chunk error as an APICallError
with `responseBody: JSON.stringify(error.data)` and `statusCode:
error.statusCode ?? 500` (anthropic-language-model.ts:3150-3160). aimux
used the whole `{"type":"error","error":…}` frame as the body and left an
unmapped type without a status; the upstream e2e test now expects the 500.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
… error A streamed SSE event whose data is not JSON yielded `JsonParse` with only the parser's message. The AI SDK's `parseJsonEventStream` reads each event with `safeParseJSON`, whose `JSONParseError` reads `JSON parsing failed: Text: <text>.\nError message: <message>` (`@ai-sdk/provider` src/errors/json-parse-error.ts:16-19). Found while porting the `@ai-sdk/openai-compatible` 3.0.62 tests (`doStream › should handle unparsable stream parts`, chat and completion); every stream read through `create_event_source_response_handler` now reports it the same way. Well-formed JSON of the wrong shape stays `InvalidResponseData`. Same change as #354 (the OpenAI-compatible fix PR): the Anthropic stream port hits it too (`doStream › code execution 20250825 tool › should include file id list in code execution tool call result`). It is a no-op once #354 is merged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
Ports the tests of @ai-sdk/anthropic 4.0.71 (RFC-0045) into aimux-providers/src/anthropic/tests/, one Rust file per upstream test file, one mod per describe level and an `/// upstream:` marker on every test: 533 new tests. With the 29 old tests marked in the next commit, 562 of the 614 upstream cases have a test. aimux-providers/layout.toml (`[anthropic]`) lists the other 52: anthropic-batch.ts and the tool/*.ts factories are missing (#275), so their 40 cases are not ported; 10 cases are sub-feature (#275: no web_fetch / web_search output schemas, no per-call file upload headers, no NoSuchProviderReference error kind) and 2 ts-only. Upstream fixtures are copied unchanged into fixtures/ and skills/fixtures/, the `toMatchSnapshot` values become snapshots/*.json. `get_model_capabilities` and `ModelCapabilities` are widened to `pub(super)` for the tests. Boundary rule 6b now skips comment lines, so a marker may quote an upstream test name that contains "anthropic". The new tests touch no environment variable and no process-wide state. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
29 old tests that assert everything an upstream @ai-sdk/anthropic 4.0.71 case asserts get its `/// upstream:` marker and move into mods named after its describe chain; their names and bodies are unchanged. 22 `/// partial:` lines point the old tests that cover a case only in part at the full port in src/anthropic/tests/. Old `/// TS:` lines that the markers replace are removed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
11 ported tests built the provider without `base_url`, so it read `ANTHROPIC_BASE_URL` when created. The lib test binary also runs `default_providers::an_invalid_env_base_url_fails_each_call_not_creation`, which sets that variable to "not a url" for a moment, so under parallel threads those tests could pick it up. They now pass `https://api.anthropic.com/v1`, the default they already expect. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…match-the-ai-sdk Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…-sdk into test/port-the-upstream-anthropic-tests Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…match-the-ai-sdk Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
@ai-sdk/anthropic 4.0.71 parses an error chunk with `code`, `statusCode` and `isRetryable` as `nullish()` (anthropic-api.ts:1450-1459), so a field sent as `null` stays on the parsed error, and createAnthropicStreamError uses that parsed error as the data when the frame has none (`data: 'data' in error ? error.data : error`, anthropic-language-model.ts:91); a first-chunk error is thrown with `responseBody: JSON.stringify(error.data)` (:3135-3139). aimux read the three fields as plain options, so a `null` was dropped from that data and from the response body. They are now presence-aware; a `null` status or retry flag still falls back to the type's (:89-90). StreamErrorData::is_retryable becomes Option<Option<bool>> (CHANGELOG Breaking). Test: first_chunk_stream_error_keeps_explicit_null_fields (tests/anthropic_remaining_test.rs) fails without the change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…guards-and-streams-match-the-ai-sdk' into test/port-the-upstream-anthropic-tests
…through warning Upstream's MCP branch in convertToAnthropicPrompt has no break, so an MCP tool named web_search falls into `if (providerToolName === 'web_search')` (convert-to-anthropic-prompt.ts:1284-1336), which breaks before the unsupported-result warning at :1435-1438. The test pins that: the mcp_tool_result block is sent and no warning is raised. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…guards-and-streams-match-the-ai-sdk' into test/port-the-upstream-anthropic-tests # Conflicts: # aimux-providers/layout.toml
Upstream's MCP branch in the assistant tool-result conversion has no `break`, so after the `mcp_tool_result` the result goes through the per-tool checks: a provider tool name adds its typed result block, tool search warns about an `error-json` result, and any other name ends at the unsupported-result warning. The guard skipped the whole per-tool step for provider tool names; this follows upstream by output type instead. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RoxcKU4zBD64Lri3e4Lrq5
…ts-betas-safeguards-and-streams-match-the-ai-sdk
…h-the-ai-sdk' into test/port-the-upstream-anthropic-tests # Conflicts: # aimux-providers/layout.toml
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml # aimux-providers/tests/anthropic_remaining_test.rs
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
…eam-anthropic-tests
eric8810
left a comment
There was a problem hiding this comment.
Verified the post-approval commits on this head: only additive tests/docs or tightened boundary checks; no weakening or behavior regression. Re-approving the current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase is content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; tests and CI unchanged. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and CHANGELOG/layout conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
|
Generated by Claude Code Generated by Claude Code |
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master/parent merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master merges and layout.toml conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master merges and conflict resolution; CI 16/16. Re-approving current head.
…eam-anthropic-tests # Conflicts: # aimux-providers/layout.toml
eric8810
left a comment
There was a problem hiding this comment.
Rebase verified content-preserving: only master merges and layout.toml union; CI 16/16. Re-approving current head.
Summary
按 RFC-0045(#281)把
@ai-sdk/anthropic@4.0.71的上游测试迁完,按厂商分批的第十三个(Mistral #295、Cohere #296、DeepSeek #299、Groq #301、Hugging Face #303、xAI #312、Anthropic AWS #319、Azure #324、Open Responses #347、Google Vertex #352、OpenAI-compatible #355、Amazon Bedrock #358)。上游 15 个测试文件共 614 个用例:
src/anthropic/tests/,一个上游测试文件对应一个 Rust 文件。/// upstream:标记,函数名和函数体不变;另有 22 处旧测试只覆盖了一部分,加了/// partial: …指向完整用例。这部分单独一个提交。layout.toml:anthropic-batch.ts(Message Batches API,38 个)和tool/*.ts的工具工厂(bash_20241022、bash_20250124各 1 个)aimux 没有,文件条目是missing,挂 Provider 构造与配置:迁移后遗留缺口 #275,用例不迁。sub-feature(Provider 构造与配置:迁移后遗留缺口 #275):web_fetch / web_search 工具没有输出 schema(8 个),文件上传没有逐次调用的 headers(1 个),aimux-core 没有NoSuchProviderReference错误类型(1 个)。ts-only:anthropicErrorDataSchema是导出的 zod schema;往uploadFile传 ReadableStream 是 JS 运行时检查,Rust 类型表达不了。迁移中发现 12 处行为差异,修在 #360 里,一处一个提交,各带一个在 master 上失败的测试;#360 还带上了 #354 的共用 SSE 解析修复(Anthropic 的流式用例也用到)。#360 已合入 master:
8a66e7f4caller去掉并警告45440e7bd231d61c1d933c42fallback块保留成anthropic.fallbackcustom part,并能发回去13da917a"citations": nulla152cdd1safeguards选项连同 beta 一起发出去(之前校验后丢掉)95d14ebbclaude-sonnet-4@…、claude-opus-4@…模型 idc5943d44anthropic-beta按上游的插入顺序拼,不再排序7c45448a1751a69fmessage_stop之后继续读后面的 message7f4a253dmessage_start里自带的tool_use块也流出来df932bbaresponseBody是错误数据本身e94beeaeJSON parsing failed: …(与 #354 同一改动)一、体系概览:Anthropic 的测试现在怎么跑(master
24977c43)上游
packages/anthropic/src/的 15 个测试文件(用例数):二、本 PR 的改动(细节)
1. 新测试目录(
anthropic/mod.rs末尾#[cfg(test)] mod tests;,:413)2. 这些测试动到的进程级状态
新测试编进
aimux-providers的 lib 测试二进制,和其他模块的单元测试同进程并行跑。它们不读也不改环境变量,没有#[serial],不换进程默认 transport,没有全局计数器,不写临时文件;每个测试用自己的MockFetch,需要 URL 的都显式传base_url。挪进 mod 的旧 baseURL 用例仍在anthropic_remaining_test.rs自己的二进制里,照旧#[serial]改ANTHROPIC_BASE_URL并恢复。3.
aimux-providers/layout.toml(只加[anthropic],44 个上游源文件)anthropic-message-lifecycle.test.ts和anthropic-unknown-model-max-output-tokens.test.ts没有同名源文件,用例全迁了,不需要写排除。4. 旧测试的标记(单独一个提交,函数名和函数体不变)
harness 上的差别(期望值都没改,除了下面写明的比法):
new AnthropicMessagesLanguageModel(...);这里用create_anthropic加api_key、base_url、fetch建,generateId能注入的地方用 provider 设置注入。10.0等于10);mock 响应带上测试服务器的content-length。type、message、statusCode、isRetryable、data)。ApiCallError::is_retryable是 bool:上游对推不出元数据的错误类型statusCode、isRetryable都是 undefined,aimux 是没有状态码和false,两者按相等处理。\nError message:之前的部分,后面是 JS 运行时JSON.parse自己的文字。convert-to-anthropic-prompt用toEqual(new Set(...))比 betas,与顺序无关,两边都排序再比;anthropic-prepare-tools的 inline snapshot 按插入顺序打印,按顺序比。toolsetNames: { X: 'computer' }在 aimux 里由调用的工具推出来,改成传一个叫 X 的computer_toolset_20260801工具。UnsupportedFunctionality、InvalidArgument只带一个字符串,上游的argument: 'providerOptions'等字段查不了,只比错误类型和消息。mockId计数器,web search fixture 快照里的 id 从id-14开始,这个用例的生成器也从 14 起。ai-sdk-anthropic/4.0.71 ai-sdk-provider-utils/5.0.53(上游 mock 成0.0.0-test),单独比。快照里值为undefined的键在 JSON 里省略。crate::anthropic::options::CANONICAL(边界规则 6b 不许源码里出现"anthropic"字面量)。Type of change
Checklist
cargo +1.99 fmt --all -- --checkpassescargo +1.99 clippy -p aimux-providers -p aimux-provider-utils -p aimux-ops --lib --tests --all-features -- -D warningspassescargo +1.99 test -p aimux-providers -p aimux-provider-utils -p aimux-ops --all-features:167 个测试二进制,3373 个测试通过(比 fix: Anthropic prompts, betas, safeguards and streams match the AI SDK #360 多 533 个)scripts/check_provider_boundaries.shpassesNotes for reviewers
src/test_support/mod.rs、aimux-providers/layout.toml,并改了边界脚本规则 7c、8b 的同两行;这几处在各 PR 里内容相同,后合入的只需合并layout.toml的厂商表。规则 6b 的改动只在本 PR。layout.toml的 tests 表里,40 个属于missing的源文件。5e4ae0d2,master 25b9038);layout.toml是 master 的原文后接 anthropic 表,tests/anthropic_remaining_test.rs的冲突取本 PR 的版本(master 那边的omits_orphaned_historical_programmatic_caller_with_warning本 PR 已经有)。Generated by Claude Code