Skip to content

fix: pause the Responses WS lane under backpressure instead of failing the exchange - #2093

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-04_responses-ws-backpressure-pause
Oct 4, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-04_responses-ws-backpressure-pause

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

ResponsesWsUpstream.fetch (src/responses-ws.ts:273) hard-failed the whole exchange whenever the consumer's stream backlog exceeded ~1x MAX_REQUEST_BYTES (~100 MiB): if ((controller?.desiredSize ?? 0) < 0) throw new Error("Responses WebSocket upstream buffer limit exceeded"). A slow client (or the pipeline legitimately stalling between chunks while doing async kernel work — src/loop/adapter-responses.ts reads via getReader()/for await) plus a large/large-terminal-frame Responses stream aborted mid-generation even though both ends were healthy.

Root cause & fix approach

The undici WebSocket socket cannot be paused, so "pause the source" has to be a software gate. The fix makes the ReadableStream's own native queue the backpressure buffer: frames arriving while the consumer lags simply wait in the queue — delivery is consumer-paced by construction, which IS the pause. No pull-promise gating (a pending pull that only resolves on the next WS message deadlocks in the tail state: all messages received, backlog parked, consumer stalled, no further message to wake the engine — reproduced before shipping).

A consumer that NEVER catches up must still be bounded, so a retained estimate tracks the backlog: exact while the queue is under the highWaterMark (requestBytes - desiredSize, Node's byte-sized formula), conservative accumulation beyond. The guard now trips at retained > 2 * requestBytes with the SAME error message and the SAME shared-catch path (reason=invalid-event, checkpoint reset, socket teardown) as before.

requestBytes is an internal constructor parameter (default MAX_REQUEST_BYTES, identical to the previous hard-coded value) so tests can exercise the backpressure window with small budgets; it is NOT user-facing configuration.

Behavior change (old -> new)

  • Backpressure tolerance: hard-fail when stream backlog exceeded ~1x requestBytes (~100 MiB production) -> pause/buffer in the stream queue; fail only if retained backlog exceeds 2x requestBytes (~200 MiB). Worst-case peak retention during a pathological permanent stall rises from ~100 MiB+frame to ~200 MiB+frame; steady-state behavior is unchanged (HWM stays 100 MiB, healthy consumers keep the queue far below it).
  • Everything else (error message text, reason=invalid-event log path, cancel/close semantics, non-streaming branch) is unchanged.

Config surface

N/A — no user-facing config, env var, or CLI flag added or changed. The new constructor parameter is internal test injection with a default equal to the previous hard-coded constant.

Tests

  • New regression tests in tests/responses-websocket.test.ts: (1) a stalled consumer pauses the lane instead of failing — crosses the old hard-fail point, drains later, receives the full ordered event sequence, and reuses the socket for the next turn; (2) a permanently stalled consumer still trips the buffer guard with the original error and teardown. Verified the first test FAILS against the old logic (teeth check) and passes with this fix.
  • Targeted suites green locally: responses-websocket.test.ts 38/38, codex-responses-ws.test.ts 5/5, typecheck clean. Full regression left to CI.

Closes #2076

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-04_responses-ws-backpressure-pause (b166a29)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2132-pr.2093.b9c4d7a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2093.tgz
npm install -g package

Install instructions are refreshed on each push.

Review of #2093: simulating the exact retained formula against the fixture frame sizes shows the 2x guard trips during the 17th delta (index 16), not around delta 23. Comment-only fix.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewed #2093 against #2076. Verdict: mergeable — root-cause fix, teeth-verified regression tests, clean diff, all 14 CI checks green, base is current master.

Verification performed (not just read-the-diff):

  • Ran targeted suites on the branch head: responses-websocket.test.ts 38/38, codex-responses-ws.test.ts 5/5, typecheck clean.
  • Teeth check reproduced independently: temporarily reverted the guard to the old desiredSize < 0 logic, new test 1 failed with exactly Responses WebSocket upstream buffer limit exceeded, restored. The new test genuinely pins the bug.
  • Trip-point measured, not estimated: drove the real ResponsesWsUpstream (not a model) through the exact fixture frames with a stalled consumer — the 2× guard trips at delta index 16, not the ~23 the test comment claimed. See minor-fix note below.
  • Full CI matrix already green on head (ubuntu/windows × node 22/24, real codex/dsh/pi/opencode-native lanes, OpenCode V2 Responses-WS ACP, build-artifact, version-guard). mergeable: True, base a311a23c.

Structured seven questions:

  1. 是什么问题: the Responses WS transport hard-failed an in-flight exchange whenever the consumer's stream backlog crossed ~1× requestBytes (~100 MiB prod), aborting a healthy mid-generation stream just because the ACP loop legitimately stalled between chunks doing async kernel work.
  2. 什么原因导致的: the undici WebSocket socket cannot be paused, so "pause the source" must be a software gate; the old code treated "queue above HWM" as fatal instead of buffering. Root layer = missing consumer-paced delivery bound, not the throw site.
  3. 怎么修复的: make the ReadableStream's native byte queue the backpressure buffer (frames arriving while the consumer lags wait in-queue — delivery is consumer-paced by construction, which is the pause). A retained estimate bounds a never-catches-up consumer: exact under HWM via Node's linear byte-sized desiredSize, conservative accumulation beyond. Guard now trips at retained > 2×requestBytes with the same error string and same reason=invalid-event catch → checkpoint-reset → socket-teardown path. Pull-promise gating deliberately avoided (deadlocks in the tail state — reproduced pre-ship).
  4. 是否有回归问题: one intentional behavior change, disclosed and bounded — backpressure tolerance 1×→2× requestBytes; worst-case peak retention on a permanent stall rises ~100 MiB+frame → ~200 MiB+frame; steady-state unchanged (HWM stays 100 MiB, healthy consumers keep queue far below it). Everything else audited unchanged: non-streaming branch, cancel/close semantics, terminal-frame history accounting, error text, reason=invalid-event log path, wire bytes (SSE block construction hoisted into a var, byte-identical), FIFO ordering, single-active-exchange invariant (:189). Constructor param defaults to the old constant → production HWM identical. No unintended regressions found.
  5. 是否破坏缓存: not applicable / no impact. Session/prefix-cache and compression untouched. The ResponsesWsHistory continuation cursor (previous_response_id) commit/reset semantics are unchanged — on failure it still resets via the same resetHistory("invalid-event"); on success-after-stall it commits normally. No dedup-key or retry-strategy drift.
  6. 是否改了设计方向: no. Stays inside the existing transport; buffers in the stream's own queue rather than adding a new data flow or boundary.
  7. 是否引入了新的配置: none user-facing. requestBytes is an internal constructor param (default = previous hard-coded MAX_REQUEST_BYTES) used only for small-budget test injection. Config surface N/A, matching the PR's claim.

Minor fix applied directly (labeled follow-up commit b166a29d, pushed to the branch): test 2's comment claimed the held backlog crosses the 128 KiB ceiling "around delta 23"; driving the real class shows it trips at delta index 16. Comment-only, zero behavior change — CI will re-confirm on the new head.

Sequencing note (not a blocker): open PR #2004 also touches src/responses-ws.ts (+16 lines, purely additive observability methods in a different region) plus the same two test files. No semantic conflict with this PR, but whichever of #2093/#2004 merges second should rebase onto the other first.

Closes #2076. Ready to merge.

@ranxianglei
ranxianglei merged commit 93d3677 into master Oct 4, 2026
14 checks passed
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.13-master.93d3677 (rolling master tag).

npm install -g billion-context@0.0.13-master.93d3677  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei ranxianglei mentioned this pull request Oct 5, 2026
@ranxianglei
ranxianglei deleted the 2026-10-04_responses-ws-backpressure-pause branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ws] Responses WS-lane backpressure hard-fails the exchange instead of pausing (src/responses-ws.ts:273)

1 participant