Skip to content

[PR] perf: frame large loop SSE events without repeated prefix scans - #6853

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_stream-frame-scan
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_stream-frame-scan

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Problem

Fixes #6849.

The OpenAI, Anthropic, Responses, and Google core-loop SSE readers normalize and search the entire unfinished event after every network chunk. A single large event, such as Gemini inline image output, repeatedly scans the same prefix and blocks the event loop for progressively longer periods.

Changes

  • Add a shared incremental framer in the existing sse-util.ts module and use it in the four loop SSE readers.
  • Normalize only the newly decoded chunk plus a possible split CR; collect line fragments until the line completes.
  • Preserve the previous LF-normalized frame boundaries, leading/empty lines, comments, data whitespace, multiline fields, Unicode decoding, and EOF residual behavior.
  • Keep protocol payload interpretation, signature handling, tool arguments, and Google's JSON-array mode unchanged.

There are no new configuration fields, dependencies, source modules, or kernel changes.

Reproduction and measured result

The parent is master 2e3cfc973b8620b1f7bc8a415ddcbd31e7f4aaee (0.1.215).

The deterministic regression passes a 1 MiB image-shaped Google event through the real adapter in 4 KiB chunks and counts the characters examined by newline searches. Unchanged master examines 271,581,497 characters for 1,048,717 input bytes and fails the linear-work assertion; the candidate examines 2,097,465 characters and forwards the exact same frame. The three one-byte Unicode/LF/CRLF/CR controls also pass on the baseline.

The included benchmark measures the Google adapter with one warmup and the median of three timed passes, receiving 16 KiB chunks:

Inline base64 payload Wire bytes Master median Candidate median
1 MiB 1,048,765 43.53 ms 7.44 ms
4 MiB 4,194,493 487.13 ms 16.70 ms
8 MiB 8,388,797 2,796.53 ms 46.09 ms
node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-stream-frame-scan.mjs 1 4 8

These are local adapter/framing measurements using synthetic image-shaped payloads, with no image codec, network, or model involved. CI uses the deterministic scan bound rather than a timing threshold.

Validation

137 focused tests passed:

node --import ./tests/preload-test-env.mjs --import tsx --test --test-concurrency=1 tests/loop-sse-framing-scan.test.ts tests/issue2323-sse-crlf-cross-chunk.test.ts tests/loop-multiline-sse.test.ts tests/loop-multiline-sse-http.test.ts tests/google-complete-frame.test.ts tests/loop-adapters.test.ts tests/adapter-google.test.ts tests/google-stream-errors.test.ts

node --import ./tests/preload-test-env.mjs --import tsx --test --test-concurrency=1 tests/anthropic-thinking-replay.test.ts tests/anthropic-round2-thinking-hold.test.ts tests/responses-round2-lifecycle.test.ts tests/e2e-google-compress.test.ts

The new framer is also compared against the previous whole-buffer framing for all short combinations of ordinary text, CR, and LF at every two-chunk split, plus one-character chunking, multiline/metadata fields, trailing CR, and EOF residuals. Existing controls cover real local HTTP responses, signed thinking replay, tool continuations, all four adapters, and hermetic Google compression.

Also passed:

  • tsc --noEmit --project tsconfig.json
  • npm run build
  • node scripts/check-test-import-graph.mjs
  • git diff --check

The live Codex four-phase E2E has not been run. E2E_CHECK=1 confirms the built distribution and installed CLI, but the configured http://127.0.0.1:8199/v1/models upstream is unreachable. The HTTP tests above are hermetic.

Related work checked: #6579 optimizes the separate native-compaction terminal observer, not these loop readers. #2329 and #6619/#6654 establish the CRLF and data-field behavior retained here.

Preserve existing newline and EOF semantics while scanning only newly
received text in the four SSE loop adapters.

Fixes ranxianglei#6849
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: not a duplicate. No other open issue or PR targets the growing-prefix scans in the core-loop SSE readers. Related work checked and confirmed distinct: #6579 (closed) made the native-compaction terminal observer linear — different parser, different path; #2329 / #6619 / #6654 established the split-CRLF and data-field contracts this PR preserves (their tests are in the validation set and pass). The author's other open PRs (#6821/#6843/#6845/#6848/#6854) cover different mechanisms.

Labels: bug, scope/wiring, p2, pr — performance defect in the wire-protocol adaptation layer; usable but degrades under large single SSE events.

Review

Direction (issue #6849). The issue identifies the right root cause: each loop reader appends the chunk to buf, re-normalizes the WHOLE buffer, and re-runs indexOf("\n\n") from the start, so every chunk revisits the entire already-received event. The PR fixes the producer of the quadratic work (repeated prefix scans) rather than masking the symptom — this is the 疏, not 堵, path. The issue's "Expected behavior" section describes exactly what landed.

Independent verification (I re-ran everything, not just trusted the numbers):

  • Baseline reproduced on master code: 271,581,497 newline-search chars examined for 1,048,717 wire bytes (259× input) — matches the issue/PR claim exactly.
  • PR branch: 2,097,465 chars (2.0× input), frame forwarded byte-exact — matches the claim exactly.
  • Benchmark on this machine: 1 MiB image-shaped Google event, 16 KiB chunks → 4.93 ms median (PR reports 7.44 ms; master 43.53 ms).
  • All 137/137 focused tests pass locally; tsc --noEmit clean; check-test-import-graph.mjs clean.
  • CI on head 7a1f401b: fully green (27 success / 1 skipped / 0 failed, verified live a moment ago).

Correctness analysis. I traced the framer's state machine against the old semantics by hand for the tricky cases — split CRLF across chunks, lone-CR terminators, CR-terminated streams, blank-line-only input, whitespace-only EOF residuals, empty chunks — and it matches. The key invariant it builds on is the #2323 hold-back in normalizeSseLineEndings (trailing lone \r not normalized until its partner byte can no longer arrive), which the framer's pendingCr reproduces per-chunk instead of per-whole-buffer. The differential test is the real anchor: all 1,093 strings over {a, \r, \n} up to length 6, at every two-chunk split plus one-character chunking, compared against finalizeSseLineEndings(text).split("\n\n") — which is exactly the old code's complete-stream output. One semantic nuance, verified harmless: the framer yields whitespace-only frames (the old inline code filtered them at that spot); all four call sites apply the same trim/dataLine filter, so adapter behavior is unchanged.

Regression review (existing behaviors on touched paths, item by item):

  1. Frame sequences of the four loop readers — preserved (byte-exact 1 MiB frame + exhaustive differential test).
  2. Frame yield timing — unchanged: frames yield at the same chunk boundaries as before; EOF residual still after the read loop; the split-CR hold means a frame ending \r + next chunk \n\n still yields on the next chunk, as before.
  3. Google SSE-vs-JSON-array mode detection — untouched; the framer is only used in SSE mode, and the pre-mode whitespace accumulation path (continue without resetting buf) is intact, so no bytes are lost.
  4. Other sse-util consumers (v1internal, responses-ws, reasoning-guard, plugin, external-summary-http) — normalizeSseLineEndings/finalizeSseLineEndings are unchanged and still used there.
  5. Config surface, persistence, wire message shape, kernel, version — none touched. No new dependencies.
  6. High-risk patterns: no semantic drift, no fail-fast→swallow, no threshold/timing/output-format drift.
    → No regressions found.

Test validity. The scan-bound test has teeth: it fails on master (271M chars ≫ 8× bound) and passes on the PR — a deterministic operation count, not a timing threshold, so it's CI-stable. The Unicode one-byte-split tests cover multi-byte characters across all three terminator styles. The hermetic HTTP suites cover real local responses; the PR is transparent that the live Codex four-phase E2E was not run (upstream unreachable) — acceptable given the hermetic coverage and green CI full suite.

Non-blocking notes:

  • The branch is 4 commits behind master (README-only changes, no file overlap, mergeable: true) — a rebase before merge is the clean path.
  • [PR] fix: preserve signed Google text on replies and tool replay #6845 (same author) also touches src/loop/adapter-google.ts and tests/google-complete-frame.test.ts — whichever of the two merges first, the other will need a rebase. Merge-ordering consideration only.
  • The benchmark script is dev-only: scripts/ is not in the npm files list, so nothing ships.

Auto-merge gate: not eligible — perf: commit (not pure fix:), spans the four hot loop-pipeline files, and it's an external contribution. Human merge required.

结构化八问

  1. 是什么问题:四个核心循环 SSE 读取器(OpenAI/Anthropic/Responses/Google)每收到一个网络块,就把整个未完结事件重新归一化、从头重新扫描,直到事件的结尾空行到达为止;单个大事件(如 Gemini 行内图片,数 MiB)导致同一前缀被反复扫描,事件循环阻塞时间随数据量平方增长。
  2. 什么原因导致的:旧实现把每个块追加进 buf 后对整个 buf 执行 normalizeSseLineEndings + indexOf("\n\n") 从头搜——每个块都重做已收字节的全部工作。根因是「重扫累积前缀」的缓冲设计本身,不是事件大小。
  3. 影响范围与严重程度:所有走这四个循环读取器的流式响应;严重程度随单个 SSE 事件大小增长。普通文本流(小事件)无感;8 MiB 单事件在 master 上阻塞事件循环约 2.8 秒(实测)。p2:可用,特定条件(大单事件,主要是 Gemini 行内图片)下性能明显下降。
  4. 怎么修复的:在既有 sse-util.ts 中新增共享增量分帧器 createSseFramer——只归一化新解码块(加一个按 [Bug] Responses SSE 的 CRLF 跨 chunk 归一化会拆坏多行 data,产生 upstream_stream_truncated(v0.1.187) #2323 不变量保留的跨块 split-CR 候选),行片段累积到行完结才拼行,空行到达才出帧;四个读取器改为 push/finish,不再重扫前缀。
  5. 是否有回归问题:未发现回归。帧序列逐字节一致(1 MiB 图片帧 byte-exact 验证 + 1093 个短 CR/LF 串全切分差分测试);出帧时机不变;Google 数组模式未动;sse-util 其他消费方(v1internal/responses-ws/reasoning-guard/plugin/external-summary-http)未受影响;配置/持久化/线格式/kernel 均未触碰。高危模式逐项过:无语义漂移、无 fail-fast 变吞错、无阈值/时序/输出格式漂移。
  6. 是否破坏缓存:不涉及——缓存、重试、增量游标、去重键均未改动。
  7. 是否改了设计方向:不涉及——同一管道、同一模块,仅分帧机制从「整缓冲重扫」变为「增量分帧」,无架构转向。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值,无新依赖。

判定:可合(建议先 rebase 到当前 master——落后 4 个 README-only 提交,无冲突;与 #6845 的合入顺序注意一下即可)。

@ranxianglei

Copy link
Copy Markdown
Owner

This fix is included verbatim (original commits and authorship preserved) in the merged batch PR #7074 — the changes are already on master. Closing to avoid a duplicate merge. 该修复已随批量 PR #7074 原样合入 master,为避免重复合并,关闭此 PR。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Large loop SSE frames repeatedly rescan their growing prefix

2 participants