Skip to content

[PR] perf: scan native compaction SSE frames incrementally - #6665

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-terminal-sse
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-terminal-sse

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6664

Problem

Native compaction can include a large encrypted-content field in a single SSE event. observeSseTerminalState currently normalizes and searches the full unfinished event after every chunk, so a fragmented event is repeatedly flattened and scanned before its terminal decision can finish.

Changes

  • Normalize only the new decoded chunk and a pending CR. Carry a final LF separately so a blank-line boundary can still span chunks.
  • Keep unfinished frame fragments and join them once when a complete event arrives. Preserve the existing JSON classification, first-terminal rule, failure precedence and EOF behavior.
  • Cover mixed line endings, byte-fragmented Unicode, malformed events followed by valid events, large compaction frames and missing final blank lines.
  • Extend the real HTTP proxy test with a 2 MiB compaction frame: the full payload reaches the client, and the native-compaction boundary is scheduled only after completion.
  • Add scripts/bench-terminal-sse.mjs to measure the exported observer with configurable frame size, chunk size and line endings.

Validation

  • 64 focused cases verified across stream-terminal, terminal-sse-framing, terminal-sse-fragments, native-compact-terminal, and codex-compact-e2e (the native integration case was rerun after the final payload assertion adjustment).
  • A deterministic 1,000-case local differential oracle produced identical terminal decisions against the unchanged base, including empty chunks, mixed line endings, fragmented UTF-8, malformed JSON and truncated input.
  • Focused strict TypeScript check, benchmark syntax check, npm run build, and git diff --check: pass.
  • Full repository typecheck/test matrix and real-provider Codex E2E remain for CI/maintainer validation.

Local benchmark

Node v24.22.1, Linux, 16 MiB encrypted-content field, 64 KiB chunks, three iterations per case. Each iteration validates completed; frame construction/import are outside the interval and decoding/framing/JSON classification are included.

Endings Baseline median This change median
LF 1,097.1 ms 117.7 ms
CRLF 1,251.4 ms 38.0 ms

Reproduce with node --import tsx scripts/bench-terminal-sse.mjs 16 64 lf and the same command ending in crlf. These measurements concern this observer and vary with runtime/load; they are not end-to-end proxy throughput claims. The observer still retains one unfinished event and validates its complete JSON; this change introduces no new frame-size limit.

Compatibility

The SSE protocol decisions from #6138 and JSON status handling from #6136 are unchanged. No configuration, wire shape, kernel logic or dependency changes.

@ranxianglei ranxianglei changed the title perf: scan native compaction SSE frames incrementally [PR] perf: scan native compaction SSE frames incrementally Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Duplicate screening: this PR duplicates an earlier open PR by the same author — 查重: 重复 #6579.

Both are by @Afloat16, both modify the same file src/stream-terminal.ts, and both fix the same root cause: observeSseTerminalState re-normalizes and re-scans an ever-growing buffer on every network chunk, so a large response.completed compaction frame does quadratic copy/scan work before the terminal decision can finish. Same symptom (slow terminal observation on multi-MiB frames), same mechanism, same root cause → 100% duplicate. The author's own note on #6578 ("Implemented in #6579") confirms #6579 is the first implementation.

Disposition (先来后到 / first-come-first-served): the earlier #6579 is the canonical one and should be kept; the later #6665 should be closed in its favor (likewise #6664 is a duplicate of #6578). Per the no-source-mutation rule I am not relabeling or closing #6665 myself — that call is yours. Say the word and I'll apply the duplicate label to #6665/#6664.

The two fixes differ in approach, so there is a real choice to make:

I'm running a full correctness comparison of both (mixed line endings, chunk-boundary CRLF, fragmented UTF-8, blank-line spanning chunks) and will post the detailed review + 8-question next so you have the evidence to pick which one to keep.

@Afloat16

Copy link
Copy Markdown
Contributor Author

Thanks for the duplicate screening. I am keeping #6579 as the canonical PR and #6578 as the tracking issue, and closing #6665 and #6664 in their favor.

The commits, benchmark, and regression cases on this branch remain available for the correctness comparison you described. Any remaining findings can be tracked on #6579 so the two approaches do not create parallel landing paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native compaction SSE terminal observation rescans growing frames

2 participants