Skip to content

[PR] test(wire): pin all four protocols against their official SDK consumers - #7466

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-11-21_wire-sdk-consumers
Open

ranxianglei wants to merge 1 commit into
masterfrom
2026-11-21_wire-sdk-consumers

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Wires the official client SDKs as hermetic CI consumers for the three protocols that lacked the guarantee #7402 established for Responses: openai chat (openai 6.26.0), anthropic (@anthropic-ai/sdk 0.91.1), and google (@google/genai 1.52.0). Each suite feeds the proxy's synthesized byte stream (adapter emit* + stream-error legacy completion + a proxy-mode re-emitted upstream) to the real SDK and asserts the stream is accepted and the content / usage / finish fields survive — the same "official client must eat our stream" contract as tests/responses-content-index.test.ts.

This is direction 2 of the investigation issue #7461. Directions 1 and 3 are answered in the issue thread.

Bug found and fixed while wiring

The official OpenAI SDK rejects a stream whose choice 0 never carries a role (missing role for choice 0) — the same failure class as #7401 (Responses missing content_index). When the upstream dies before sending any bytes, the proxy's synthesized error completion is the client's first frame, and both emit sites (relay-level emitStreamError and the loop-level adapter.emitError) omitted the role, so the SDK rejected the whole response and the client saw a connection error instead of the failure text. Both now open with the assistant role; a duplicate role (upstream already sent one) is harmless — the SDK just re-sets the already-"assistant" role. Pinned by the two error-completion tests.

How the consumers are wired

  • SDK resolution — through the in-tree @earendil-works/pi-coding-agent (the [PR] fix(responses): preserve content indices in synthesized streams #7402 precedent): createRequire(import.meta.resolve("@earendil-works/pi-coding-agent")) → await import(pathToFileURL(req.resolve(pkg)).href). No package.json / lockfile changes. If pi-ai ever drops one of these transitive deps the suite fails loudly (a visible, deliberate diff), not silently.
  • openai / anthropic — new OpenAI({ fetch }) / new Anthropic({ fetch }) with a fetch that returns the synthesized bytes. Fully hermetic, no network.
  • google — @google/genai's models.generateContentStream path uses the global fetch (no client-level fetch option — verified in the SDK source), so the consumer runs against a local HTTP server on listen(0) that serves the synthesized bytes. Still hermetic (loopback only, no external network).
  • CI — the new tests/*.test.ts files are auto-globbed by scripts/run-ci-tests.mjs (no registration). None of the suites spawn a process, so no LIVE_FILES tripwire entry is needed.

Files

  • tests/openai-chat-sdk-consumer.test.ts (new, 7 tests)
  • tests/anthropic-sdk-consumer.test.ts (new, 7 tests)
  • tests/google-sdk-consumer.test.ts (new, 5 tests)
  • src/stream-error.ts (openai legacy completion opens with the assistant role)
  • src/loop/adapter-openai.ts (new buildRole(); emitError completion shape opens with it)

Closes #7461

Wires the official client SDKs as hermetic CI consumers for the three
protocols that lacked the guarantee #7402 established for Responses:
openai chat (openai 6.26.0), anthropic (@anthropic-ai/sdk 0.91.1), and
google (@google/genai 1.52.0). Each suite feeds the proxy's synthesized
byte stream (adapter emit* + stream-error legacy completion + a
proxy-mode re-emitted upstream) to the real SDK and asserts the stream
is accepted and the content/usage/finish fields survive — the same
"official client must eat our stream" contract as
tests/responses-content-index.test.ts.

SDKs are resolved through the in-tree pi-coding-agent (the #7402
precedent) so no package.json/lockfile changes; the google consumer runs
against a local HTTP server because @google/genai's streaming path uses
the global fetch (no client-level fetch option).

The wiring surfaced a real bug: the official OpenAI SDK rejects a stream
whose choice 0 never carries a role ("missing role for choice 0"), the
same class as #7401. When the upstream dies before sending any bytes the
synthesized error completion is the client's first frame, and both emit
sites (relay-level emitStreamError and the loop-level adapter emitError)
omitted the role. Both now open with the assistant role; a duplicate role
is harmless. Pinned by the two error-completion tests.

Closes #7461
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-21_wire-sdk-consumers (350e963)

Latest build — install this exact version

npm install -g billion-context@0.0.11219-pr.7466.36474ea

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
350e963 billion-context@0.0.11219-pr.7466.36474ea build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7466.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title test(wire): pin all four protocols against their official SDK consumers [PR] test(wire): pin all four protocols against their official SDK consumers Nov 21, 2026
@ranxianglei ranxianglei added bug Something isn't working enhancement New feature or request p3 scope/ci labels Nov 21, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Dedupe: not a duplicate. #7466 is the only open PR wiring official SDK consumers for the wire protocols, and the only open PR citing #7461. The other open PRs matching "consumer" in the title (#5944, #4844) are unrelated domains (window sizing, dsh WS-lane gate).

Live status (verified just now):

Labels: the mirror shim does not implement the label endpoints in this environment (endpoint not implemented in ework shim), so I could not apply the pickup triage labels here. My triage judgment for this PR, for when labeling is possible: bug (it fixes the missing-role error-completion defect), scope/ci (primary purpose is CI test wiring), p2 (the role bug only bites when the upstream dies before the first byte — the official OpenAI SDK then rejects the whole stream; workaround exists, not a total outage).

I am now reviewing the branch (code + regression surface) and will follow up with the structured review.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Independent review of PR #7466 — done. I re-read the diff from scratch (not from the issue thread), re-ran the suites locally, and probed the one claim the tests do not cover. Verdict: review clean, mergeable once CI goes green (it is still running: 29 success / 23 skipped / 8 in progress / no failures; the 40 "cancelled" rows are the superseded earlier run).

What I verified beyond the PR's own claims:

  1. The "duplicate role is harmless" claim is NOT pinned by any test — both error-completion tests only cover the first-frame case. The real duplicate-role scenario is: upstream sends role+content, then dies (truncation at src/loop/core.ts:1719, re-request HTTP error at src/loop/core.ts:1925, empty body at src/loop/core.ts:1931) and the legacy completion is appended after the upstream's role. I probed this directly against the official openai 6.26.0 SDK: both the relay-level (emitStreamError) and adapter-level (emitError) duplicate-role streams are accepted, content concatenates correctly, finish=stop. Claim holds.
  2. Blast radius confirmed narrow: the role chunk only affects openai-protocol clients that explicitly opt in to compat.streamErrorShape="completion" (src/config.ts:1817, default "protocol"). The native error shape (default) is byte-identical to before — pinned by the new "still rejects the default native error" negative test. No other protocol branch in legacyCompletionChunk changed.
  3. Regression suites on the touched paths: 63/63 pass locally (proxy-stream-error, google-stream-errors, upstream-truncation, loop-sse-framing-scan, verif-adapter-wire-stability, terminal-sse-framing, upstream-error-log). New suites: 19/19 pass locally. npm run typecheck clean on the PR head.
  4. Wiring claims verified: all three SDKs resolve through the createRequire(import.meta.resolve("@earendil-works/pi-coding-agent")) path exactly as the tests use it; the new files contain no SPAWN_RE pattern (no LIVE_FILES tripwire needed) and are auto-globbed by scripts/run-ci-tests.mjs.
  5. Master drift: 6 commits ahead (WorkBuddy lane WorkBuddy Desktop: route its model traffic through bili (settings-file lane) so the wb app gets in-app compression like dsh #7456/[PR] feat: WorkBuddy Desktop lane — settings-file routing + auto-compact gate (#7456) #7457), zero overlap with the PR's files — no semantic conflict, rebase would be clean.
  6. Wire-contract ledger ([CI/testing] wire-contract anti-regression system: validation-parity fake upstreams + golden schema snapshots + constraint ledger + staged 4-protocol x 3-mode client smoke matrix #1304): the role constraint is a whole-stream property ("choice 0 carries a role at least once"), not a per-frame one, so the WIRE_RULES per-frame validator shape doesn't fit (unlike WC-026, which is per-frame). The SDK consumer tests are the right permanent pin and they live in CI. Noted, not a defect.
  7. Test style: the new files follow the tests/responses-content-index.test.ts precedent exactly (collector/consume pattern, SDK resolution). The triplicated collector() is the established repo convention for test isolation, not a new duplication defect.

Residual risk (low, for the record): the completion shape exists for hosts whose SDK cannot surface in-band errors — those hosts may not be the official OpenAI SDK. A non-standard client that rejected a second role delta would be the only failure mode; role-in-delta is legal on any chunk per the OpenAI spec, and such hosts already accepted the synthesized completion before, so I rate it low.

Labels: the PR already carries bug / enhancement / p3 / scope/ci on GitHub — correct set. Correcting my earlier in-thread p2 guess: p3 is the right call, the bug's trigger is a double edge case (explicit compat opt-in + upstream dies before the first byte).

Auto-merge gate: not eligible — it changes a wire shape (error completion) and is a test+fix combo, so this is a human-merge PR. CI green + your merge is all that's left.


八问

  1. 是什么问题:四条出站 wire 协议此前没有"官方客户端必须能吃我们合成的流"的持续自动化校验([Bug] Synthesized Responses text events lose content_index and crash SDK consumers #7401/[Bug] Fake-completion buffering silently truncates successful responses #7405/[PR] fix(openai): keep provider usage fields in the synthesized completion #7136 都是事后才知道的);接线时还发现 openai legacy error completion 缺 assistant role,官方 SDK 直接拒收整个流。
  2. 什么原因导致的:adapter 手搓发射帧,缺官方 SDK 消费端回归;两处 error completion 发射点(relay 级 emitStreamError、loop 级 adapter.emitError)合成流时漏了 role 开场帧。
  3. 影响范围与严重程度:质量缺口覆盖四条协议(测试面);missing-role bug 仅 compat.streamErrorShape="completion" 显式 opt-in 的 openai 客户端 + 上游首字节前死亡,双边缘条件,p3。
  4. 怎么修复的:三条协议各接一个 hermetic 官方 SDK 消费端(openai/anthropic 注入 fetch,google 走 listen(0) 本地回环 server);两处发射点补 role 开场帧。
  5. 是否有回归问题:无。我独立复查了触碰路径的既有行为清单:native 错误形状逐字节未变(有负向测试钉住)、其余协议分支未动、重复 role 场景实测官方 SDK 接受、触碰路径的 63 个既有测试全绿、typecheck 干净。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(adapter 仍手搓,方向 1 结论不变;本 PR 只加校验 + 修一个发射缺陷)。
  8. 是否引入了新的配置:不涉及(无新 env/config/flag;SDK 经 in-tree pi-coding-agent 解析,package.json/lockfile 未动)。

收工判定:审核通过,无拦截项;等 CI 全绿 + owner 合并(人工合并,wire shape 变更不走 auto-merge)。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working enhancement New feature or request p3 scope/ci

Projects

None yet

1 participant