Skip to content

[PR] fix(#1706): recognize unterminated final [DONE] at EOF; add truncation forensics logs - #1709

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-09-29_issue1706-truncation-fix
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-09-29_issue1706-truncation-fix

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Ref: #1706 (no auto-close — the reported incident is likely a genuine mid-thought disconnect; this PR fixes one verified bili-side false-positive window and makes every future occurrence self-diagnosable. Closing once the cutting hop is confirmed from the owner's log line.)

What changed

Two changes, both in src/plugin.ts, both on the #721 truncation path:

1. Recognize an unterminated final data: [DONE] at EOF (false-positive fix). The SSE parser only recognizes [DONE] inside complete blank-line-delimited events. A gateway that closes right after the final data: [DONE] line — no terminating blank line — left the sentinel dangling in the buffer at EOF; if the stream had also omitted finish_reason, bili reported a semantically completed turn as truncated. At EOF the single-data-line [DONE] payload is now recognized with the same normalization as the in-stream form (data:[DONE], extra spaces), the sentinel is forwarded with its terminator restored, and no error is emitted. Recognition is deliberately strict: exactly one data line whose normalized payload is [DONE]. Multi-line dangling blocks, other protocols, and the read-failure (catch) path are untouched and stay conservative.

2. Truncation forensics logging. Both emission sites in both lanes (chat + responses) now include <bytes>B in <elapsed>ms, last byte <silence>ms ago. The silence duration names the cutting hop's idle budget: ~300s/600s ⇒ a middlebox/proxy, ~720s ⇒ bili's own 12-minute upstream watchdog (BILI_UPSTREAM_TIMEOUT_MS), 0B ⇒ the connection produced nothing. Log-only; wire output unchanged by this part.

Behavior changes (old → new)

  1. Wire behavior, chat lane: OpenAI chat stream ending with an unterminated final data: [DONE] + no finish_reason anywhere — old: in-band truncation error frame ("this turn may be incomplete"); new: treated as complete, sentinel forwarded terminated, no error. Rationale: [DONE] is an unambiguous completion marker; a missing terminator is framing detail, not semantics. Everything else (genuine cuts, multi-line dangling blocks, catch path) behaves exactly as before — pinned by control tests.
  2. Log format (log-only): new warn line on EOF truncation; the read-failure warn line gains the forensics suffix. No client-visible impact beyond change 1.

Why (issue context)

#1706 reports the error during opencode's thinking phase, frequently but not deterministically. Thinking = long wire silence ⇒ classic idle-timeout disconnect signature; the intermittent pattern matches "only when silence exceeds some hop's idle budget". While investigating, two bili-side gaps were verified locally:

  • the false-positive window above (reproduced pre-fix, fixed here);
  • undiagnosable genuine cuts: prior log lines said nothing about how long the last byte was silent — exactly the data needed to attribute the cut to bili's watchdog vs a middlebox vs the provider. Owner's specific incident is most likely a real disconnect; the next occurrence's log line will say which hop.

Evidence

Probe matrix against pipePluginChatWithStrip (openai lane), before → after:

Scenario Before After
finish_reason + proper [DONE] clean clean
finish_reason + dangling [DONE] silent synthesis silent synthesis
no finish_reason + dangling data: [DONE]\n error frame (false positive) clean, sentinel forwarded terminated
no finish_reason + bare data: [DONE] error frame (false positive) clean, sentinel forwarded terminated
no finish_reason + data:[DONE] (no space) error frame (false positive) clean
genuine cut, no terminal error frame error frame + forensics log
dangling block with 2 data lines error frame error frame (conservative, unchanged)

New regression suite tests/issue1706-dangling-done.test.ts: 8 tests pinning the whole matrix plus both forensics log shapes (EOF path and read-failure path).

Preflight

  • npm run typecheck — clean
  • npm test — 3240 pass / 0 fail / 12 skip (full suite, rebased head = current master 8d3d1bf)
  • npm run build — success

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-29_issue1706-truncation-fix (de91d23)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1665-pr.1709.67e2678

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1709.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR review — verified against branch 2026-09-29_issue1706-truncation-fix @ 922abc33 (branch was renamed from pr-1709 mid-review; same commit).

① Issue direction — correct. Two separate problems, both handled at their decision point:

② Regression inventory (existing behaviors on touched paths):

  • Complete [DONE] mid-stream handling — untouched ✔
  • Silent synthesis (finish_reason delivered, [DONE] absent entirely) — untouched; the finish_reason+dangling-[DONE] combo now routes through forwarding instead, wire output byte-equivalent (data: [DONE]\n\n final frame either way) ✔
  • Genuine truncation error frame (Plugin mode: upstream stream truncated before terminal event → client gets bare cut-off SSE (no completion/error event) #721) — identical emission; log format extended (disclosed) ✔
  • Read-failure path — identical emission; log suffix added (disclosed) ✔
  • Non-openai protocols — guarded by protocol === "openai", unchanged ✔
  • Client-abort cases (res.destroyed/writableEnded) — unchanged ✔
  • Success-path residual-buf write — byte-identical when danglingDone=false ✔
  • settleUsage() / warn ordering — unchanged (all run before the truncated branch in old and new) ✔
  • Responses lane — log-only additions; no dangling-[DONE] recognition there, defensible: on that wire [DONE] follows response.completed, which already sets sawTerminal, so the false-positive shape is unreachable without a gateway stripping the completed event too ✔
    → No unintended regressions. Both disclosed behavior changes are intentional and justified.

③ Code — clean. Normalization (slice(5).replace(/^ /,"").trim()) matches the in-stream form exactly; tab/space variants covered. Conservative boundary: multi-data-line dangling blocks stay on the error path (a partial JSON chunk + [DONE] is genuinely ambiguous). Forensics counters minimal and correct; loggerLog used throughout (AGENTS-compliant).

④ Tests — real, with teeth. 8 behavioral tests drive the actual pipePluginChatWithStrip; assertions check exact tail bytes (out.endsWith("data: [DONE]\n\n")), error-frame absence/presence, and forensics regex via setLogCapture. Controls pin every adjacent behavior (silent synthesis, genuine cut, read failure, conservative multi-line case, clean termination). A regression of either the fix or the conservative boundary fails a specific test. Minor gaps (non-blocking): responses-lane forensics log shape unpinned; partial-sentinel edge (data: [DO) unpinned.

⑤ Diff — clean. 2 files, every line on-purpose.

Behavior changes (explicit, per policy):

  1. Chat lane, openai protocol, clean EOF with exactly one unterminated final data: [DONE] data line and no finish_reason: old = in-band truncation error frame ("upstream stream ended before a completion event…") → new = stream completes normally, sentinel forwarded with restored terminator. Reason: the sentinel was delivered; framing-only loss is not truncation, and the new output equals proper-framing output byte-for-byte.
  2. Log format: two warn lines gain a <bytes>B in <ms>, last byte <ms> ago forensics suffix (one previously-silent path now logs). Log-only.

Local verification at 922abc33: typecheck clean · npm test 3240 pass / 0 fail / 12 skip (matches your numbers) · build OK.

CI red explained — not caused by this PR: all four core test-matrix jobs fail on #1101 T9 in tests/issue1086-chain-self-exemption.test.ts with ReferenceError: rmSync is not defined — a file this diff never touches. Decisively, those job logs contain an older version of that test (title "default OFF, 1 enables") than the one present in commit 922abc33 ("default ON, 0 disables", import intact) → the red jobs ran against a stale tip from before the rebase onto 8d3d1bfc. All e2e lanes on the current head are green. Corroboration: PR #1710 (merged minutes ago, dcbc9449) fixes exactly this master-side bug class — "two bare rmSync calls left undefined (breaks #1101 T9 on CI)".
→ One action needed: re-run the stale test-matrix jobs (or rebase onto latest master ca981936) to get a fresh green on the current head. Given the local full-suite pass at exactly this commit, expect green.

Verdict: code is merge-ready (wire-shape change ⇒ human merge per §7.4); only a fresh green CI run on the current head is outstanding. No changes made to the branch — review only.

中文摘要:审核了 PR #1709(识别 EOF 处未终止的最后一个 data: [DONE],消除 #1706 误报,并为真截断增加字节/时序取证日志)——方向正确、无回归、测试有牙、diff 干净,本地在提交 922abc3 上 typecheck/3240 项测试/build 全绿;CI 四个红 job 是 rebase 前的旧 tip 上已存在的 rmSync is not defined 测试 bug(与本项目无关,#1710 已在 master 修复),只需对当前 head 重跑 CI 即可合并。

@ranxianglei
ranxianglei force-pushed the 2026-09-29_issue1706-truncation-fix branch from 922abc3 to 0fc0aa8 Compare September 29, 2026 16:05
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Follow-up (continuing the review above):

While verifying, the branch was rebased onto latest master (ca981936, which includes the #1710 rmsync-test fix) — new tip 0fc0aa8a. Good: that rebase alone puts the PR on a healthy base and triggers a fresh CI run, which is exactly what was needed to clear the stale red matrix jobs (those ran against the pre-rebase tip and failed on the now-fixed rmSync is not defined bug in tests/issue1086-chain-self-exemption.test.ts).

I also picked up one of the two minor gaps flagged in my review and fixed it directly on the branch (small-problem rule):

  • New commit de91d23e — test(#1709): pin partial-sentinel EOF edge: a stream whose final block is cut mid-token (data: [DO\n) must stay on the truncation path, i.e. NOT be recognized as [DONE]. 8 lines added to tests/issue1706-dangling-done.test.ts using the existing harness; guards against a future refactor loosening the exact-match check to prefix/regex matching.
  • The other gap (responses-lane forensics log shape unpinned) needs a new harness for pipePluginResponsesWithStrip — beyond a small passing fix, left noted rather than bundled here.

Verified locally at de91d23e (rebased head): typecheck clean · targeted file 9/9 · full suite 3290 pass / 0 fail / 17 skip · build OK.

Status: waiting on the fresh CI run triggered by this push; once green, the PR is ready for human merge (wire-shape change ⇒ §7.4 human-only). No other changes made.

中文摘要:PR 已被 rebase 到最新 master(含 #1710 的 rmsync 测试修复),我在分支上补了一个小测试提交 de91d23(钉住「哨兵被截断在 token 中间」仍走截断路径的边界),rebase 后本地全量验证通过(typecheck/3290 项测试/build 全绿);等这次推送触发的新 CI 变绿即可由人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

孤儿票清理:标题指向的 issue 已关闭,修复已由其他路径落地。关单;误判请 reopen。

@ranxianglei ranxianglei changed the title fix(#1706): recognize unterminated final [DONE] at EOF; add truncation forensics logs [PR] fix(#1706): recognize unterminated final [DONE] at EOF; add truncation forensics logs Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant