Skip to content

feat: termination diagnostics on the plugin-lane truncation signal (#2328 Q2/Q4) - #2367

Merged
ranxianglei merged 4 commits into
masterfrom
2026-10-08_terminal-diag
Oct 8, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-10-08_terminal-diag

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Implements the remaining quarters of #2328 (Q2 + Q4). Companion to #2351 (Q1 raw-SSE dump) — both are independent now; whichever merges second gets a small src/server.ts call-site conflict to resolve (same lines: the tee vs the upstreamMeta argument).

What

The #721 in-band upstream_stream_truncated error frame said a stream was cut, but never why it could not be completed. The pipes already know: parsed-event count, last event types, the dangling EOF buffer, retry-budget state, upstream HTTP identity — none of it reached the client or the log (#2347 needed a raw-SSE dump to tell the two failure classes apart).

Q4 — classification, computed from factual bytes at the cut (tailLooksTerminal in src/plugin.ts):

  • no-terminal-seen — clean EOF / read error, nothing terminal in flight.
  • terminal-bytes-unrecognized — the dangling partial event carries the opening bytes of the wire's own terminal: data: [DO…, event: message_stop, a quoted (non-null) finish_reason/finishReason chunk head, event: response.completed|failed|incomplete / "type":"response.comp…" head.

Q2 — metadata (TruncationDiag in src/stream-error.ts), attached to the error frame as meta and to the log line as one grep-able JSON blob:

field source
cause eof | read-error (which pipe path fired)
classification Q4 above
events, unparseableForwarded, bytes per-stream counters (unparseable = #2190 raw-forward path)
lastEventTypes last ≤5 parsed event labels (wire-native type; chunk/chunk:finish=<r> for type-less wires)
eofBufferBytes, eofBufferHead dangling partial SSE event at the cut
finishReason (chat) / responseStatus + heldEvents (responses) wire-specific terminal context
visibleChars, sawStrippedEcho what the client actually assembled
retryZeroByteCutSpent, retryDegenerateSpent #2171/#732 one-shot budget state
upstreamStatus, upstreamContentType threaded from server.ts via new optional UpstreamMeta param (5 call sites); cites the first upstream response (a resolveFakeCompletion re-send is not reflected)

Wire-shape change ⚠️ (human-merge item)

meta is a new field on error-only frames (openai/anthropic/google: nested as error.meta; responses: top-level meta; google keeps its numeric-code {code:503,status:"UNAVAILABLE"} object shape). No non-error frame changes. Frames emitted without diag (loop adapters, other emitUpstreamTruncation callers) are byte-identical to before. Per house rule this PR is left for human merge even if CI is green.

Tests

Rebased on current master, so it sits on top of #2329's stateful SSE normalizer: byte counts see the raw upstream bytes pre-normalize; a lone-CR tail is resolved by finalizeSseLineEndings before the classification runs, so the classifier only ever sees a genuinely dangling event.

Refs #2328. Human merge required (wire shape).

@ranxianglei
ranxianglei force-pushed the 2026-10-08_terminal-diag branch from 3b3d083 to 603f418 Compare October 7, 2026 16:52
awork and others added 3 commits October 8, 2026 00:53
…iter throw paths)

The non-rewriter branch awaited dumpRaw inline, so a rejecting pipe would skip it and leave the raw-SSE dump half-written — while the plugin lane and the rewriter lane both await in their own finallys. Move the non-rewriter await into the shared safety-net finally (idempotent for the rewriter lane) so all three lanes honor the 'full upstream tail before turn close' contract on every exit path. Success-path behavior unchanged.
…2328 Q2/Q4)

The #721 in-band upstream_stream_truncated error frame said a stream was
cut, but never WHY it could not be completed: the pipes know the event
count, the last event types, the dangling EOF buffer, the retry budget
state and the upstream HTTP identity — none of it reached the client or
the log. #2347 forensics had to fall back to raw-SSE dumps to tell
"upstream never sent a terminal" from "the terminal's bytes never
completed".

- stream-error.ts: emitUpstreamTruncation takes an optional TruncationDiag;
  it rides the error frame as `meta` (openai/anthropic/google nested in
  error.meta, responses top-level meta — google keeps its numeric-code
  object shape) and the log line as classification + cause + full JSON.
- plugin.ts: both strip pipes (chat + responses) track events/bytes/
  unparseable-forwards/last-5-event-types; at EOF and read-error they
  build the diag and classify the cut (#2328 Q4):
  - no-terminal-seen — EOF/read-error with no terminal bytes in flight;
  - terminal-bytes-unrecognized — the dangling partial event carries the
    opening bytes of the wire's own terminal ([DO(ne), event:
    message_stop, a quoted finish_reason/finishReason chunk, response.
    completed-family head). Factual byte evidence only.
  Responses adds responseStatus + held done-family count; both add the
  #2171/#732 retry-budget state and the upstream status/content-type
  (threaded as optional UpstreamMeta from server.ts, 5 call sites).
- Wire-shape note: `meta` is a NEW field on error-only frames — flagged
  for human review per house rule; no non-error frame changes.

Tests: tests/issue2328-terminal-diag.test.ts — 8 cases across openai/
anthropic/google/responses: both classifications, cause eof/read-error,
counters, upstreamMeta threading, log-line shape; existing truncation
suites (#2171/#1452/#732/#1061/#1039) unchanged and green.
@ranxianglei
ranxianglei force-pushed the 2026-10-08_terminal-diag branch from 603f418 to 4f71fef Compare October 7, 2026 17:01
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

⚠️ merge_conflict 唤醒已处理:分支已 rebase 到当前 master 并 push,冲突已解除(mergeable: true,CI 运行中)。

发生了什么

冲突解法(3 处,均为 union)

  1. imports:保留 incoming 的 TruncationDiag import + UpstreamMeta + tailLooksTerminal,degenerate-turn 导入取两侧并集(degenerateTurnWarning, endsWithDraftClose)。
    2./3. 两个 pipe 读循环:保留 master 的 pendingFinal 结构,[Bug] Responses SSE 的 CRLF 跨 chunk 归一化会拆坏多行 data,产生 upstream_stream_truncated(v0.1.187) #2323 的 EOF-resolve-redrive 路径原样不动;diagBytes += value.byteLength 放进收 chunk 的 else 分支——计数语义与 rebase 前逐字节一致(每个非空收到 chunk 计一次原始字节;EOF redrive 路径不重复计数)。

结构化七问

  1. 是什么问题:PR 分支与 master 的机械性合并冲突,不是代码缺陷。
  2. 什么原因导致的:[Bug] Responses SSE 的 CRLF 跨 chunk 归一化会拆坏多行 data,产生 upstream_stream_truncated(v0.1.187) #2323 把两个 strip-pipe 循环重构为 pendingFinal 形式、fix: retry terminal turns ending in a compression-draft closing tag (#2303) #2304 改了同一 import 行,与本 PR 新增的字节计数器/导入文本重叠。
  3. 怎么修复的:整栈 cherry-pick 到当前 master,三处冲突按"两侧意图并集"手解(见上),未丢弃任何一方语义。
  4. 是否有回归问题:未发现。定向验证全绿:tsc --noEmit clean;50/50 通过(issue2328-terminal-diag 8 例、issue2347-dump-sse-plugin-lane、issue2171-zero-byte-truncation-retry、issue2323-sse-crlf-cross-chunk、loop-degenerate-retry、issue1452-socket-hardening、upstream-truncation)。既有行为逐项过:[Bug] Responses SSE 的 CRLF 跨 chunk 归一化会拆坏多行 data,产生 upstream_stream_truncated(v0.1.187) #2323 CRLF holdback/redrive 保持、plugin lanes: auto-retry zero-client-visible upstream truncations (frequent since 2026-10-05, cf. #2154) #2171 zero-byte retry 仍先于截断判定、计数器仍按设计统计 normalize 前的原始字节。全局回归交给 CI(本地测试纪律)。注:作者已声明 tests/plugin-agent.test.ts 有 2 个环境相关失败在 master 上同样复现,若 CI 出现属已知项。
  5. 是否破坏缓存:不涉及——未触碰缓存/增量游标/去重键;一次性重试预算(plugin lanes: auto-retry zero-client-visible upstream truncations (frequent since 2026-10-05, cf. #2154) #2171/[BUG] In some cases, with Codex's auto model + DeepSeek, it will suddenly terminate automatically. #732)状态机未变。
  6. 是否改了设计方向:不涉及——rebase 保持原架构:仅 error-only frame 带 meta,非 error frame 逐字节不变。
  7. 是否引入了新的配置:不涉及(UpstreamMeta 是内部参数透传,非配置面)。

判定:冲突已解除,等 CI 转绿即可;wire-shape 变更(error frame 新增 meta)按 house rule 仍需人工合并。

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-08_terminal-diag (26c4e9b)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2547-pr.2367.bef7b67

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2367.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

CI 全绿(4f71fef):主 CI(typecheck + 全量测试 + build)✅,全部 13 条 e2e/registry/image lane ✅。其中 e2e-native-dsh 首跑被基础设施在 27s 处 cancel(runner 侧,非测试失败),已通过 API re-run 转绿。

无阻塞项,等人工合并(wire-shape:error frame 新增 meta)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

⚠️ merge_conflict 唤醒核查:本次为陈旧通知,无需 rebase——分支当前与 base 零冲突,未做任何 push(避免无谓的 SHA 重写和 CI 重跑)。

核查证据(2026-10-07 17:1xZ):

  • GitHub API:mergeable: true、mergeable_state: clean;head 4f71fef86 @ 2026-10-08_terminal-diag,base master @ 2495ad25d(与我上次 rebase 所用 base 完全一致,master 此后未前进)。
  • 本地证明:git merge-base(分支, master) = 2495ad25d = master 本身,即分支恰好领先 master 3 个提交(353ba2586、286a85713、4f71fef86,可快进);git merge-tree --write-tree 产出干净合并树、无冲突文件。
  • 推测成因:平台镜像在我 force-push 后的 GitHub 状态重算窗口内采样到了瞬时 dirty(push 后我观测到 mergeable_state: unstable,随后转 clean)。按 house rule「冲突解完状态会自动转干净,不重复 rebase」处置。

七问(本轮为核查型唤醒):

  1. 是什么问题:系统转发的 PR 冲突告警;实际是通知时序竞态,非真实冲突。
  2. 什么原因导致的:force-push 后 GitHub 短暂处于状态重算期,镜像在该窗口采样到 dirty。
  3. 怎么修复的:无需修复——API 状态 + 本地 merge-base/merge-tree 双重验证分支干净,不做 no-op rebase。
  4. 是否有回归问题:不涉及(未改任何代码/引用)。
  5. 是否破坏缓存:不涉及。
  6. 是否改了设计方向:不涉及。
  7. 是否引入了新的配置:不涉及。

判定:PR 保持 4f71fef86 不动,CI 已全绿(见上一条评论);若后续 master 真的前进并再次产生冲突,届时再走 rebase 流程。等人工合并(wire-shape)。

@ranxianglei
ranxianglei merged commit 196e3b6 into master Oct 8, 2026
27 checks passed
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.125-master.196e3b6 (rolling master tag).

npm install -g billion-context@0.0.125-master.196e3b6  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei
ranxianglei deleted the 2026-10-08_terminal-diag branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant