Skip to content

[PR] fix(responses-ws): bound the prewarm probe and make busy-lane rejections reconstructible (#6172) - #6292

Open
ranxianglei wants to merge 2 commits into
masterfrom
2026-11-12_ws-busy-observability-probe-watchdog
Open

ranxianglei wants to merge 2 commits into
masterfrom
2026-11-12_ws-busy-observability-probe-watchdog

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #6172

What this fixes

The prewarm probe (input: [] + generate: false, sent by Codex on every fresh connection) was the one unbounded upstream call in a Responses WS session: no abort signal, no watchdog. A wedged handshake or a silent first event pinned busy=true forever while the connection stayed open, so every later turn got 409 response-in-progress until the client disconnected. That is the residual-state path the issue's 29 rejections cannot rule out — and the old one-line client rejected reason=response-in-progress log made it impossible to tell that case apart from legitimate concurrency protection.

The main turn path was already bounded (client-disconnect abort + two-phase idle watchdog); this PR holds the probe to the same discipline.

Changes (src/responses-ws.ts only)

  1. Bounded probe. The probe fetch is wrapped in withFetchTransport(fetchWithTimeout(...)) exactly like the turn path. The first-byte budget covers handshake + first event; the idle budget covers mid-stream silence. An abort surfaces as the existing explicit Probe relay failed error frame, and the lane releases either way. No new knobs — it reuses network.upstreamFirstByteMs / network.upstreamTimeoutMs.
  2. Centralized release. Every admission releases exactly its own state via settle(kind, outcome): a per-session generation counter, activeMeta (kind / prev_id / items / bytes / age) and an 8-entry transition ring replace the two duplicated finally blocks. The stale-callback fear from the issue is structurally impossible here (one session per connection; each closure's settle belongs to its own admission), but it is now auditable in one place.
  3. Reconstructible rejection line. The warn keeps the exact prefix client rejected reason=response-in-progress and appends: req_bytes / req_prev_id / req_items (best-effort identifiers of the REJECTED frame — parse failure just omits them) plus active_gen / active_kind / active_age_ms / active_prev_id / active_items / active_terminal (turns only) / admitted / transitions=[…]. Identifiers only — message bodies never reach the log.
  4. Debug settle lines per exchange (exchange settled kind=… gen=… outcome=… duration_ms=…) and enriched client closed phase line when a close lands mid-exchange.

Tests (tests/responses-websocket.test.ts):

  • overlap-rejection test now pins the enriched fields (and asserts no body text reached the log);
  • new black-hole-upstream test: a wedged probe fails within the first-byte budget with the explicit error frame, the lane releases, and a second probe on the same connection is admitted instead of 409'd.

Behavior changes (old → new)

# Old New Why
1 Hung prewarm probe: silent forever-hang, lane pinned busy until client disconnect Fails after the first-byte budget (default 12 min) with the existing Probe relay failed frame; lane released Bounds the only unbounded upstream call in the session
2 client rejected reason=response-in-progress (bare) Same prefix + diagnostic fields (ids/counts/bytes/age/transitions) The issue's core ask: distinguish legitimate protection from stuck state without leaking bodies
3 No per-exchange settle record Debug-level exchange settled … + transition ring Reconstructable state machine

Wire protocol otherwise unchanged. No config schema change.

Config surface

None added. The probe reuses the existing network.upstreamFirstByteMs / network.upstreamTimeoutMs budgets (BILI_UPSTREAM_FIRST_BYTE_MS / BILI_UPSTREAM_TIMEOUT_MS) — no new env vars, keys, or defaults.

…ons reconstructible (#6172)

The prewarm probe was the one unbounded upstream call in a Responses WS
session: no signal, no watchdog. A wedged handshake or a silent first event
pinned busy=true forever and 409'd every later turn until the client
disconnected — the residual-state path behind the 29 response-in-progress
rejections in #6172. The turn path was already bounded (client-disconnect
abort + two-phase idle watchdog); the probe is now held to the same
discipline with the existing budgets (first-byte covers handshake + first
event, idle covers mid-stream silence). An abort lands as the explicit
'Probe relay failed' frame and the lane releases either way.

- Centralize admission release in settle(kind, outcome): per-admission
  generation counter, activeMeta (kind/prev_id/items/bytes/age), and an
  8-entry transition ring so every release is exactly its own admission's.
- Enriched rejection line keeps the exact prefix
  'client rejected reason=response-in-progress' and adds req_bytes /
  req_prev_id / req_items plus active_gen / active_kind / active_age_ms /
  active_prev_id / active_items / active_terminal / admitted / transitions.
  Identifiers only; message bodies never reach the log.
- Tests: the overlap-rejection line pins the enriched fields; a new
  black-hole-upstream test proves a wedged probe fails within budget and
  releases the lane (a second probe on the same connection is admitted).
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-12_ws-busy-observability-probe-watchdog (9199534)

Latest build — install this exact version

npm install -g billion-context@0.0.9186-pr.6292.90e6364

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
9199534 billion-context@0.0.9186-pr.6292.90e6364 build
becc7ac billion-context@0.0.9179-pr.6292.4a518a4 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6292.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(responses-ws): bound the prewarm probe and make busy-lane rejections reconstructible (#6172) [PR] fix(responses-ws): bound the prewarm probe and make busy-lane rejections reconstructible (#6172) Nov 11, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Picked up this [PR] thread to verify the existing agent PR (body carries the ework-agent-pr marker → not re-doing the work; verifying branch + CI + targeted tests instead).

查重: 不重复。Only one open PR targets #6172 — this one (#6292, branch 2026-11-12_ws-busy-observability-probe-watchdog, head becc7ac). The other open prewarm-related PR (#4595, routing context-only prewarms through the upstream proxy) addresses a different mechanism (proxy routing, not bounding/observability), so no parallel-fix collision on this thread.

标签: intended set = bug · codex · scope/wiring · p2 (bug class; Codex Responses-WS lane = client protocol adapter; conditional failure with a reconnect workaround → p2, not data-loss/unrecoverable). The mirror label API is not implemented in this shim (endpoint not implemented in ework shim), so I could not apply them here; the GitHub PR already carries bug/p2/ws/codex/scope/wiring.

In progress now: full diff review of the single commit, live CI status read, and a targeted local run of tests/responses-websocket.test.ts. I will follow up with the structured verdict (八问) once those are done.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verification complete. I reviewed the full single-commit diff (src/responses-ws.ts + tests/responses-websocket.test.ts only), traced the bounding mechanism end-to-end, ran the targeted suite locally, and read live CI state.

What I verified

  • Mechanism is sound. The probe now runs inside withFetchTransport(this.upstreamFetch, () => fetchWithTimeout(...)). fetchWithTimeout resolves the ALS transport and routes through ResponsesWsUpstream.fetch as before — so the probe's network path (hop/tunnel stamping, connection key, HTTP fallback on refused upgrade) is unchanged; the wrapper adds the two-phase watchdog (first-byte budget covers handshake+first event, idle budget covers mid-stream silence). Abort propagates via options.signal into the transport's handshake/stream handlers → AbortError → the existing Probe relay failed error frame → settle() releases the lane. The turn path keeps using the same shared marked transport instance (egress-shed WeakSet tag semantics identical to the old per-call wrapping).
  • Tests have teeth. The overlap-rejection test pins every enriched field AND asserts the rejected frame's body never reached the log (negative assertion). The new black-hole test reproduces the exact wedge (upgrade accepted, then silence): probe fails bounded with the explicit frame, exchange settled kind=probe … outcome=aborted, and a second probe on the same connection is admitted instead of 409'd — i.e. it fails pre-fix and proves the fix.
  • Local targeted run: tests/responses-websocket.test.ts → 42/42 pass on head becc7ac.
  • Live CI (mirror cache): all checks finished — 18 success / 12 skipped / 0 failed. The skipped lanes are the real-client e2e suites, skipped by design for plain agent PRs under the three-layer CI gate (feat!: remove image pre-compression and the image_full tool (#3266) #3295). mergeable=true; base is 5 commits behind current master but conflict-free (BEHIND is not a blocker).

Regression inventory (existing behaviors on the touched paths): rejection wire frame byte-identical (response_in_progress/409/same message, exact log prefix kept); probe success-path relay unchanged (transport gains signal/dispatcher/redirect-manual/accept-encoding restriction — standard bili egress discipline, matching every other fetchWithTimeout call site); release semantics equivalent (old duplicated finally bodies centralized into settle()); onClose actions unchanged (log enrichment only); debug-only additions (settle lines, 8-entry transition ring, bounded memory); plugin-host WS lane only — proxy mode untouched; kernel/, update.ts, identity logic untouched. No regressions found.

Two non-blocking observations

  1. Default bound is the first-byte knob's default (= full idle budget, 12 min): a wedged probe now pins the lane for at most ~12 min instead of forever. Operators who want faster release tighten BILI_UPSTREAM_FIRST_BYTE_MS. No new knob was added (config-surface discipline), which is correct.
  2. Pre-existing asymmetry, NOT introduced by this PR: when an upstream proxy is configured, turns' WS handshakes ride the proxy dispatcher (src/loop/core.ts:660) while the probe connects direct (WsBridgeContext carries no proxy info — wiring it needs plumbing across ws-bridge.ts). Pre-fix that asymmetry meant the probe could hang forever; post-fix it fails fast/bounded (TCP-level refusals fail in seconds; black-hole case is budget-bounded). Strictly better than before; whether to route probes through the proxy too is your call — noted here per the no-new-issue rule rather than filed.

判定: 可合并 — CI 全绿、定向测试全过、无回归、行为变更已在 PR 内披露。等 owner 合并即可(合并权在 human)。


结构化八问

  1. 是什么问题:Codex 每次新连接发的 prewarm probe(input:[] + generate:false)是 Responses WS 会话里唯一无上界的上游调用——握手或首个事件卡住时 busy 永久置顶,之后每一轮都吃 409 直到客户端断开;且旧日志只有一行裸 reason=response-in-progress,无法区分合法并发保护与卡死状态(issue 里那 29 条拒绝无法归因)。
  2. 什么原因导致的:probe 路径当初没接入 turn 已有的有界纪律(client-disconnect abort + 两阶段 idle watchdog);准入状态(谁占着车道、占多久、怎么来的)完全没有记录,拒绝发生时不可重建。
  3. 影响范围与严重程度:仅走 Responses WS 车道的 Codex/opencode 客户端,且上游在 probe 阶段黑洞(握手成功但不发事件,或握手挂起)。触发时该连接不可用直到重连(有绕行手段),无数据丢失 → p2。
  4. 怎么修复的:probe 的 fetch 包进与 turn 完全相同的 withFetchTransport + fetchWithTimeout(first-byte 预算覆盖握手+首事件,idle 预算覆盖流中静默),abort 落到既有的显式错误帧,车道无论如何都经统一的 settle(kind, outcome) 释放(gen 计数 + activeMeta + 8 条转移环,替代两处重复 finally);拒绝日志保留原前缀并追加被拒帧与活跃准入的标识字段(ids/counts/bytes/age/transitions,正文永不入日志)。
  5. 是否有回归问题:逐项清点过——拒绝线帧字节不变;probe 成功路径帧中继不变(仅按 bili 出站惯例补 signal/dispatcher/redirect-manual/accept-encoding 收窄);egress-shed 标签语义等价(每调用包装 → 每会话共享一个标记实例,WeakSet 成员判定相同);释放语义等价;onClose 行为不变;proxy mode 未触碰;kernel/update/身份逻辑未触碰。未发现回归。
  6. 是否破坏缓存:不涉及。不改消息重建/fold 锚点/前缀字节,prefix cache 不受影响;重试、增量游标、去重键均不受影响。
  7. 是否改了设计方向:没有。是把 probe 拉齐到 turn 已有的有界纪律 + 可观测性(debug 级 settle 行、有界转移环),架构/边界/数据流未转向。
  8. 是否引入了新的配置:没有。复用既有 network.upstreamFirstByteMs / network.upstreamTimeoutMs(BILI_UPSTREAM_FIRST_BYTE_MS / BILI_UPSTREAM_TIMEOUT_MS),无新 env/key/默认值。

收尾判定:可合并,等 owner 点合(PR #6292,head becc7ac)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

The full verdict is already in this thread (my comment above, comment id 73214) — this nudge appears to have raced the post. Restating the bottom line only:

Outcome: PR #6292 verified, no code changes made (agent-PR marker → verify-only mode).

  • Diff reviewed end-to-end (single commit, src/responses-ws.ts + tests/responses-websocket.test.ts); bounding mechanism traced through fetchWithTransport → ALS transport → ResponsesWsUpstream.fetch — network path unchanged, watchdog added.
  • Tests have teeth (enriched-field pins + body-leak negative assertion; new black-hole test reproduces the wedge and proves lane release).
  • Local targeted suite: 42/42 pass on head becc7ac; live CI: 18 success / 12 skipped (by design) / 0 failed; mergeable=true, base conflict-free.
  • Regression inventory all clear; behavior changes disclosed in the PR body; no config surface added.

判定:可合并,等 owner 点合 PR #6292。Nothing remains open from this session.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant