Repository navigation
[PR] perf: meter streamed handoff replies incrementally - #6748
Merged
ranxianglei merged 1 commit intoNov 14, 2026
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6746
Problem
Handoff capture meters both the new cumulative message and its previous value on every text/reasoning delta. The saved response is capped at 1.5 MB, but the work before reaching that cap grows quadratically with the number of deltas. A 57,600-character reply split into 2,400 fragments makes
Buffer.byteLengthscan 138,240,000 characters. The same mechanism occurs through the production SSE parser and a real local HTTP client.Change
Keep each captured entry's text byte count, total accounted size and final UTF-16 code unit. Meter only the new text fragment on append, subtracting two bytes when an old lone high surrogate and a new leading low surrogate join into one four-byte UTF-8 code point. The cached final code unit avoids inspecting or flattening the accumulated prefix.
Tool and whole-response replacements still measure their complete new values and release the previous entry's budget. The shared cap, per-entry overhead, first over-budget discard, entry ordering, stale-request guards and saved
CoreMessageshape stay intact. Saves expose message objects, and later appends replace those objects so earlier saved snapshots stay unchanged. No outgoing wire transformation or runtime dependency is introduced.Verification
pipePluginChatWithStrip→ saved capture. The HTTP test also compares every forwarded byte with capture disabled.npm run typecheck,npm run build, strict TypeScript checking of the benchmark script, andgit diff --checkpass.Measured effect
The committed benchmark measures a real local HTTP client reading the complete response through the production SSE parser and handoff capture. Fixtures are built before timing; complete wire bytes plus saved reply metadata are hashed after timing. Three alternating old/new pairs, with one warmup per scenario in each process, produce identical complete hashes in all 18 measured outputs.
The short case shows no improvement. These measurements cover the local HTTP/parser/capture/read path; they do not measure real model generation or remote network latency. Node 24.22.1, TZ=UTC, otherwise idle host.
Remaining merge gate
The repository requires real Codex four-phase coverage before merging request-pipeline changes. The official zero-token
E2E_CHECK=1preflight foundcodex-cli 0.167.0-alpha.4and the built proxy, but the configured default upstream athttp://127.0.0.1:8199/v1was unreachable. The real four-phase test remained skipped behind its existing gate. Real-provider Codex E2E has not passed and remains required before merge. No gate was weakened.History check
Checked Afloat16's indexed issue history and actual related PR file patches; none previously changes this capture module. The complete #4248 implementation introduces the repeated cumulative byte metering and is unchanged in this part of master. #4860 concerns capture acceptance after the client closes, not byte accounting. Focused all-state searches found no equivalent incremental-metering change.
Base: master
a8bbb8d23108ba9148d09512927041364625d910(0.1.214). Full-suite status remains hosted CI on the exact head.