Skip to content

[PR] fix: Codex/opencode WebSocket delta turns no longer rebuild the full history before compression (#4804) - #4817

Merged
ranxianglei merged 13 commits into
masterfrom
2026-10-26_issue4804-ws-delta-continuation
Oct 27, 2026
Merged

ranxianglei merged 13 commits into
masterfrom
2026-10-26_issue4804-ws-delta-continuation

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Part 2 of 2 for #4804 (root fix). Supersedes PR #4808: this branch already contains its stopgap commits, so merging this PR makes #4808 redundant — please close #4808 when merging this one.

What was broken

A previous_response_id continuation expanded into the complete reconstructed history before admission and processing: every turn re-materialized all retained originals (O(N) JSON rebuild plus repeated serialization), and once that reconstruction crossed the byte budget the turn died with 413 request_too_large before ACP compression ever ran — stranding sessions whose model view had already been compressed to <1 MiB. The client's actual wire frame was tens of KiB (real capture: 37,739 B); the cost was entirely internal expansion.

How it works

The delta rides the unified transcript:

  1. Session (src/responses-ws.ts): on an eligible continuation (matching checkpoint id, unified storage readable, all prior outputs echoable), dispatches the small body [prior outputs, new input] instead of the O(N) reconstruction, and stamps the envelope with the commit-time identity signature of the dispatched body's last core message (non-enumerable symbol property, zero wire pollution).
  2. Prepare (src/server/prepare-responses.ts + buildAnchoredView in src/server/snapshot-render.ts): renders the outbound prefix from the server's own transcript rows — same loader/anchor machinery as snapshot-render, no chain walk (the client vouches for the prefix by protocol) — and appends the delta tail verbatim.
  3. Honest seams: the anchor is verified against the stored row's identity signature; any mismatch (missing store / non-echoable output / rollout off / seam miss) falls back to the legacy full-reconstruction path, and an engine-side seam miss answers 400 previous_response_not_found — the client's standard recovery (full replay) re-anchors losslessly. Nothing is dropped: the store keeps every original.
  4. Chained checkpoints (base/tail/tailSig) make steady-state turns grow memory O(1); expand() walks the chain only on fallback. Echoing prior outputs into the delta body also gets them ingested into the unified transcript (rows are request-side only), keeping the server-rendered prefix complete across restarts.

Behavior changes (disclosed)

Surface Old New Why
WS wire frame cap (ws-bridge.ts maxPayload) hardcoded 100 MiB limitsReceiveBytes() (default 5 GiB, network.receiveBodyBytes / BILI_RECEIVE_BODY_BYTES) post-boundary reconnect full-replay must not die at the wire layer (acceptance 2+3); bounded by per-connection single-flight
WS admission budget (responses-ws.ts) hardcoded 100 MiB on reconstructed body same knob one budget surface, zero new config fields
WS checkpoint retention cleared at 100 MiB combined same knob near-cap checkpoints survive instead of stranding the next continuation
413 refusal message fixed "100 MiB" names the live budget honest output
Delta continuation path always O(N) full reconstruction server-rendered prefix + echoed outputs + tail, gated by new rollout feature wsDeltaContinuation, DEFAULT OFF (0%) root fix; ramp is the owner's lever (BILI_ROLLOUT_WS_DELTA_CONTINUATION=1 or rollout.features.wsDeltaContinuation)
Delta-turn error surface — new 400 previous_response_not_found when the transcript seam can't be verified standard client recovery = full replay, zero loss

With the rollout OFF (the merged default) the only effective changes are the first four rows — identical to PR #4808. Eligibility additionally requires storage.unified: "full" (shadow mode reads return null; no 400-chatter).

Config surface

No new config fields. Reuses network.receiveBodyBytes / BILI_RECEIVE_BODY_BYTES (Part 1) and the existing owner-gated rollout registry (new entry wsDeltaContinuation, percent 0). Env vars: BILI_ROLLOUT_WS_DELTA_CONTINUATION follows the established BILI_ROLLOUT_* convention.

Item-ID evidence (thread floor 8)

v1 is deliberately client-ID-independent: the anchor is a server-side identity signature computed at commit time on the exact dispatched body (same converter both sides ⇒ byte-exact parity). The reporter's stability data (10/10 items across 17 replay comparisons; but 8/128 assistant messages omitting id in the natural thread) confirms ID-keying cannot carry alone — recorded here as future hardening, not implemented.

Tests

kernel-change: none — this branch makes zero changes under kernel/ (an earlier lockfile conflict from a swept ci-fixes sync was resolved by taking master's kernel 0.0.166 regeneration verbatim).

Review notes (§7.4)

Client↔bili wire shape unchanged; bili→upstream content is what compression already produced. No config schema, persistence format, or kernel changes. Touched the session/compression-judgment path with a new rollout-gated behavior, so this stays a human merge decision.

ci-check added 7 commits October 26, 2026 13:45
- export ResponsesWsSession so the onMessage admission seam is testable
  directly (same seam as the issue's offline repro harness)
- tests/issue4804-ws-admission-boundary.test.ts expresses the acceptance
  behavior: full replay above the old 100 MiB cap reaches dispatch; small
  previous_response_id continuation over the reconstructed cap reaches
  dispatch; checkpoint accounting retains a completed turn whose combined
  size crosses the old cap. Verified to fail on unfixed code with the exact
  production signatures (request_too_large 413, previous_response_not_found).
  Controls (small turns pass, foreign id refused 400) stay green.
…4804)

The Responses WebSocket lanes (codex-responses-ws, responses-ws) measured
the RECONSTRUCTED full history against a hardcoded 100 MiB cap before
dispatch ran, so a long session whose processed outbound was far below the
cap died with 413 request_too_large on the next turn even when the client
sent only a few MiB of delta; checkpoint accounting shared the same cap and
cleared checkpoints near it, stranding continuations with
previous_response_not_found. #3202 made the HTTP budgets configurable but
explicitly left the WS surface fixed at 100 MiB.

- ws maxPayload, reconstructed-body admission, upstream frame/backpressure/
  output guards and checkpoint accounting all resolve limitsReceiveBytes()
  (BILI_RECEIVE_BODY_BYTES > network.receiveBodyBytes > 5 GiB) - zero new
  config fields
- one full serialization per turn: admission measures the same wire string
  the pipeline envelope pushes (was two)

Behavior changes (old -> new, defaults):
- WS wire frame cap 100 MiB -> 5 GiB
- WS reconstructed-body admission cap 100 MiB -> 5 GiB
- checkpoint retention cap 100 MiB -> 5 GiB (near-cap checkpoints survive)
- 413 message names the live budget instead of a constant "100 MiB"
Restore the old value with BILI_RECEIVE_BODY_BYTES=104857600 (this also sets
the HTTP receive budget, as it always did).
…4804)

The codec comment claimed codex "never sends previous_response_id", but the
#4804 real-session capture shows another codex-family build sending
previous_response_id with a tiny input tail as its normal operating mode.
Scope the claim to the 0.147.0 capture it came from; both shapes are live
production traffic.
master window commits 293ae65/de5e03659 bumped kernel/package.json to
0.0.165 without regenerating the lockfiles; lock-sync re-resolution then
rewrites the stale recorded versions (root 0.0.163, kernel 0.0.164) and
fails. Pure recorded-version sync via npm install --package-lock-only;
no dependency or semantic change.
…ry reconstruction (#4804)

A previous_response_id continuation used to expand into the complete
reconstructed history before admission/processing: every turn re-materialized
all retained originals (O(N) JSON rebuild plus repeated serialization), and
once the reconstruction crossed the byte budget the turn was refused with 413
before ACP ever ran — stranding sessions whose model view had already been
compressed to <1 MiB. The client's actual wire frame was tens of KiB; the
cost was entirely bili's internal expansion.

The delta now rides the unified transcript:

- On an eligible continuation (matching checkpoint id, unified storage
  readable, all prior outputs echoable), the session dispatches the small
  body [prior outputs, new input] instead of the O(N) reconstruction and
  stamps the envelope with the commit-time identity signature of the
  dispatched body's last core message (non-enumerable symbol property).
- prepare-responses renders the outbound prefix from the server's own
  transcript rows (buildAnchoredView: same loader/anchor machinery as
  snapshot-render, no chain walk — the client vouches for the prefix by
  protocol) and appends the delta tail verbatim.
- The seam is verified against the stored row's identity signature; any
  mismatch (or missing store / non-echoable output / rollout off) falls back
  to the legacy full-reconstruction path, and an engine-side seam miss
  answers 400 previous_response_not_found — the client's standard recovery
  (full replay) re-anchors losslessly. Nothing is dropped: the store keeps
  every original.
- Checkpoints chain (base/tail/tailSig) so steady-state turns grow memory
  O(1); the legacy expand() walks the chain only on fallback.
- Echoing the prior outputs into the delta body also gets them ingested into
  the unified transcript (rows are request-side only), keeping the
  server-rendered prefix complete across restarts.

Behavior changes (disclosed):
- New rollout feature wsDeltaContinuation, DEFAULT OFF (0%): zero behavior
  change until ramped; force on via BILI_ROLLOUT_WS_DELTA_CONTINUATION=1 or
  rollout.features.wsDeltaContinuation.
- When ON and eligible: the upstream request for a delta turn carries the
  server-rendered folded prefix plus echoed outputs plus the tail instead of
  the client-resent full history (removes the internal O(N) materialization
  and its 413 window).
- A new 400 previous_response_not_found can be returned on a delta turn when
  the transcript seam cannot be verified; the client recovers with a full
  replay (no data loss).

Tests: tests/issue4804-ws-delta-anchor.test.ts (engine/history/session units,
both gate directions) and tests/issue4804-ws-delta-e2e.test.ts (real proxy +
fake WS upstream: delta keeps the conversation without resending it; unknown
checkpoint -> 400 -> lossless replay).
@ranxianglei ranxianglei changed the title fix: Codex/opencode WebSocket delta turns no longer rebuild the full history before compression (#4804) [PR] fix: Codex/opencode WebSocket delta turns no longer rebuild the full history before compression (#4804) Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_issue4804-ws-delta-continuation (4baf5b3)

Latest build — install this exact version

npm install -g billion-context@0.0.6271-pr.4817.cde034a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
4baf5b3 billion-context@0.0.6271-pr.4817.cde034a build
bd32f90 billion-context@0.0.6240-pr.4817.d8e4d61 build
f72dbfe billion-context@0.0.6215-pr.4817.aed7d2f build
72e8d44 billion-context@0.0.6213-pr.4817.5541b59 build
5404d9c billion-context@0.0.6211-pr.4817.facacb7 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4817.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 是 #4804(WS 车道长会话在压缩前被 100 MiB 拦死)的根治 Part 2,与 #4808(同 issue 的止血 PR,仍 open)是「根治 supersede 止血」关系,非重复立项;按 PR 描述合并本 PR 时应关闭 #4808。另有 #4788/#3643 追踪 RSS/耗时基线,属后续验证跟踪,不重复。

标签: bug / codex / opencode / scope/wiring / p1(缺陷修复;涉及 Codex 与 OpenCode V2 两条 WS 车道;改动面在请求管道/WS 桥接层;底层缺陷会让长会话续轮直接 413 死掉、会话搁浅,属"不能用"级)。

状态: 已拉取 head 5404d9c2(5 commits, +1070/-87, base = master cdac1c09),CI 23 个 check 全部 queued。开始独立审阅(diff 逐段 + 定向测试),结论按八问体例随后给出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 同机制已登记,非新问题——系统报的 kernel-guard 红落在旧 head 72e8d44450,与 #4599 floor 265 登记的签名同一机制(ci-fixes sweep 带入 kernel/package-lock.json delta → Kernel Boundary Rule 1 要求正文声明 kernel-change:),且前一轮会话已按既有路径处置完(补声明 + 重触发)。本轮拉了现 head f72dbfedbc1 的全部 52 个 check 与 10 个红分片的完整日志逐项核对,结论如下。

① 是什么问题:CI 门禁 kernel-guard 红——PR diff 因 sweep 带入的 lockfile 版本同步提交(dfadc21b3)触碰了 kernel/**,而旧 head 跑 CI 时 PR 正文还没有 kernel-change: 声明。

② 什么原因导致的:Kernel Boundary 规则「任何触碰 kernel/ 的 PR 必须在正文声明」;ci-fixes 的 lockfile 同步使三点 diff 非空。不是内核代码变更——本 PR 零 kernel 源码改动、零版本 bump,被触碰的只有 lockfile 记录行(0.0.164→0.0.165)。

③ 影响范围与严重程度:仅 CI 门禁级,无用户面影响;只影响本 PR 的可合并判定,不影响任何已发布行为。

④ 怎么修复的:零代码改动——PR 正文补 kernel-change: 声明(写明是 sweep 带入的 lockfile 记录同步)+ 空 commit f72dbfedbc1 重触发门禁。实测新 head 上 kernel-guard ✅ 绿,lock-sync ✅、unused-exports ✅、static-gates ✅×2、全部 guard 与 13 条 e2e workflow 全绿,新增 #4804 套件 30/30 通过(含 e2e 真代理+假 WS 上游两条)。

⑤ 是否有回归问题:无。剩余 10 个 test 分片红经逐日志核对全部为继承性 born-red(master/ci-fixes 同款,哨兵正在 ci-fixes 上修,#4599 floors 270/271):#1492 B preflight ×3、#847 exhaustion ×3、#4125 give-up reason ×3、storage-engine-proofs full-mode restart(google)×3、storage-render golden(google)×3;另 1 条 Windows EBUSY teardown flake(issue4732 测试自己的 rmSync 撞 SQLite 锁,与本 PR diff 零交集)已登记 #4599 待哨兵裁定。按 CI 协调协议不在本分支修 master 级失败;ci-fixes 收敛后 re-merge 即绿(或等哨兵 help-merge sweep)。

⑥ 是否破坏缓存:不涉及(本轮零代码改动)。

⑦ 是否改了设计方向:不涉及。

⑧ 是否引入了新的配置:不涉及(rollout 开关 wsDeltaContinuation 默认 0% 属 PR 原有披露面,本轮未动)。

判定:本 lane 收工——唯一可操作项(kernel-guard)已修复并实测绿,无需再推新 commit;剩余红全部继承/环境因素,等哨兵在 ci-fixes 收敛后本分支 re-merge 即转全绿,届时可请 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 与 attempt 1 同一继承集群,非新问题——本轮是同一批 10 个 test 分片的重跑结果(新 job,09:09Z 起跑),失败签名与首轮日志审计逐条一致,确认确定性继承而非 flake 风暴;ci-fixes 仍为 1eb592a1e(已全部合入本分支),哨兵修复未落笔。

① 是什么问题:10 个 test 分片红 = master/ci-fixes 继承的 born-red 集群 + 1 条 Windows teardown 缺陷。

② 什么原因导致的:全部失败源自 master 上既有的代码/测试(preflight 措辞漂移钉死 #1492/#847/#4125、storage golden 漂移家族 ×2),与本 PR diff 零交集(本 PR 只动 responses-ws / snapshot-render / prepare-responses / ws-bridge / knobs / rollout / persist 纯 getter)。其中 tests/issue4732-plugin-render-tail-anchor.test.ts 的 EBUSY 重跑复现第二次(同形:该测试自己的 test.after(rmSync) 撞未释放的 SQLite handle)——由「疑似 flake」上修为「倾向确定性 Windows teardown race」,已登记 #4599 待哨兵收敛进 ci-fixes。

③ 影响范围与严重程度:仅 CI 门禁级,无用户面影响;只影响本 PR 的可合并判定。

④ 怎么修复的:本轮零代码改动——按 CI 协调协议(2026-10-24),master 级失败不在此分支修,统一收敛 ci-fixes(哨兵已接管,#4599 floors 270/271);本分支已含当前 ci-fixes tip,修复落笔后 re-merge(或等哨兵 help-merge sweep)即全绿。不开 decision-request issue:这些失败无一需要设计裁决,且 owner 政策明令禁止为 CI 问题开新 issue,状态一律在 #4599 汇报(本 lane 两条评论均已留痕,非静默等待)。

⑤ 是否有回归问题:无。本 PR 自身车道全绿(kernel-guard ✅、lock-sync ✅、unused-exports ✅、static-gates ✅×2、全部 guard ✅、e2e workflow ×13 ✅、新增 #4804 套件 30/30 ok);重跑未引入任何新失败签名。

⑥ 是否破坏缓存:不涉及。

⑦ 是否改了设计方向:不涉及。

⑧ 是否引入了新的配置:本轮零改动。一项依赖披露:kernel-guard 目前靠 PR 正文的 kernel-change: 声明(sweep 带入的 lockfile 同步)保持绿;收敛载体 #4814 合并后本分支 kernel/** delta 消失、声明即失去必要;若按授权卡点新政(2026-10-26)该声明被否导致门禁再红,将按政策开 kernel authorization request,不自行补标。

判定:auto-fix lane(attempt 2/2)收工——本分支无可操作项(全部红为继承/环境因素,修复所有权在 ci-fixes 哨兵);#4814 + 哨兵修复落 master/ci-fixes 后 re-merge 转全绿,届时请 owner 合并。

ci-check and others added 3 commits October 26, 2026 18:36
…4-ws-delta-continuation

# Conflicts:
#	kernel/package-lock.json
#	package-lock.json
…k so the 400 full-replay recovery is reachable (#4817)

The #4804 recovery branch in responses-ws.ts answers a seam-miss delta with
the client's standard 400 previous_response_not_found frame and clears the
checkpoint chain, but it only fires when dispatch lets the
SnapshotAnchorFallbackError through. server.ts's dispatch catch used to
swallow every error into a generic 503 — the 400 branch was dead code in
production (the unit test stubs dispatch to rethrow directly), the
checkpoint stayed live, and the client retried the same deterministic miss
forever. Rethrow for exactly the request the WS session stamped
(WS_ANCHORED_REQ_PROP); plain-HTTP lanes keep the 503 shape.

Adds the end-to-end happy-path pin: a delta continuation anchors through the
REAL dispatch and completes without any error frame. The miss path itself
stays unit-pinned; driving it through the real pipeline needs a same-session
side-lane primitive the harness lacks (a second WS peer's transcript rows are
invisible to the main peer's snapshot view) — recorded in the PR thread.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

跟进:审查发现的「400 恢复分支是死代码」已修复并推到本 PR 分支(4baf5b3)。

修了什么:server.ts 的 dispatch catch 把所有错误吞成通用 503,导致 #4804 里精心写的「锚点失配 → 400 previous_response_not_found → 客户端全量重放」恢复分支在生产永远走不到(单测是靠 stub dispatch 直接 rethrow 才测到的)。现在对 WS 会话打过标记的请求(WS_ANCHORED_REQ_PROP)原样 rethrow,普通 HTTP 车道保持 503 形状。

测试:新增端到端 happy-path 钉(增量续写经真实 dispatch 锚定成功、无任何 error 帧);失配路径维持单测钉。想把失配路径也做成真管线 e2e 时发现测试基建缺口:同一 session id 的第二个 WS peer 写的 transcript 行,主 peer 的快照视图看不到(每连接独立会话上下文),且轮间折叠发生在下一次 dispatch 内部(在 seam 检查之后)——两种确定性失配手段都构造不出来。这个缺口先记录在案,不阻塞本 PR。

WS 家族回归 30/30 绿,typecheck 干净。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI lane 确认(head 4baf5b335,含死代码修复):全部 55 个 check 绿(39 pass / 16 skip / 0 fail)——12 个 test 分片全过,kernel-guard / lock-sync / unused-exports / static-gates / 各 guard / e2e workflow ×13 全绿。此前登记的继承红集群已消解:master 经 #4807/#4815 合入吸收了哨兵修复——#847/#1492B preflight 契约(938407427)、#4125 give-up reason(247b92f49+f180e5cc7)、以及我在 #4599 报的 #4732 EBUSY 也落了双层修复(31b292dda resilient rmrf + b258b6e7d teardown 先关 SessionStore——提交说明印证了「held index.db handle」的诊断方向)。分支已含 master@341e76c28,无需再动。

八问(本轮为 CI 状态确认,零代码改动):
① 是什么问题:无新问题——确认死代码修复推送后的 CI 终态。
② 什么原因导致的:不涉及(本轮未发现缺陷)。
③ 影响范围与严重程度:不涉及。
④ 怎么修复的:本 lane 无需动作——继承红的收敛载体(#4807/#4815 进 master)已被分支包含,CI 重跑即全绿。
⑤ 是否有回归问题:无——本 PR 自身车道全绿(新增 e2e happy-path 钉在内);失配路径的测试基建缺口按你的记录接受为非阻塞项。
⑥ 是否破坏缓存:不涉及。
⑦ 是否改了设计方向:不涉及。
⑧ 是否引入了新的配置:不涉及。

**判定:分支全绿,可合并。**请 owner 合并;按 PR 正文约定,#4808 在本 PR 合并后冗余,可一并关闭。

@ranxianglei
ranxianglei merged commit e91e113 into master Oct 27, 2026
55 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.871-master.e91e113 (rolling master tag).

npm install -g billion-context@0.0.871-master.e91e113  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant