Skip to content

[PR] perf: meter streamed handoff replies incrementally - #6748

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-handoff-capture
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-handoff-capture

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Closes #6746

Problem

Handoff capture meters both the new cumulative message and its previous value on every text/reasoning delta. The saved response is capped at 1.5 MB, but the work before reaching that cap grows quadratically with the number of deltas. A 57,600-character reply split into 2,400 fragments makes Buffer.byteLength scan 138,240,000 characters. The same mechanism occurs through the production SSE parser and a real local HTTP client.

Change

Keep each captured entry's text byte count, total accounted size and final UTF-16 code unit. Meter only the new text fragment on append, subtracting two bytes when an old lone high surrogate and a new leading low surrogate join into one four-byte UTF-8 code point. The cached final code unit avoids inspecting or flattening the accumulated prefix.

Tool and whole-response replacements still measure their complete new values and release the previous entry's budget. The shared cap, per-entry overhead, first over-budget discard, entry ordering, stale-request guards and saved CoreMessage shape stay intact. Saves expose message objects, and later appends replace those objects so earlier saved snapshots stay unchanged. No outgoing wire transformation or runtime dependency is introduced.

Verification

  • 13 new tests: 3 work-bound failures / 10 passing behavior controls on unchanged master.
  • Work assertions cover both text and reasoning, plus real HTTP → production pipePluginChatWithStrip → saved capture. The HTTP test also compares every forwarded byte with capture disabled.
  • Exact-cap tests cover split surrogate pairs, lone surrogates, complete emoji, BMP multibyte text, multiple entries, tool replacement, tool-to-text replacement, whole-response replacement and immutable saved snapshots.
  • 64/64 related tests pass, including existing four-protocol stream/JSON capture, incomplete/error/oversize cases, loop-core handling and CLI export.
  • Full npm run typecheck, npm run build, strict TypeScript checking of the benchmark script, and git diff --check pass.
  • Independent read-only review checked production logic, callers and all 13 tests; no blocker. The reviewer did not rerun tests.
node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx --test --test-concurrency=1 tests/handoff-capture-work.test.ts tests/handoff-response.test.ts tests/cli-export.test.ts tests/loop-core.test.ts

Measured effect

The committed benchmark measures a real local HTTP client reading the complete response through the production SSE parser and handoff capture. Fixtures are built before timing; complete wire bytes plus saved reply metadata are hashed after timing. Three alternating old/new pairs, with one warmup per scenario in each process, produce identical complete hashes in all 18 measured outputs.

Scenario Old median (range), ms New median (range), ms
3,000 Unicode deltas, 336,000 text bytes 622.664 (620.083–749.757) 28.687 (28.426–29.382)
3,000 ASCII deltas, 288,000 text bytes 519.929 (463.008–571.017) 19.642 (17.787–20.905)
8 Unicode deltas, 896 text bytes 1.469 (1.093–1.471) 1.496 (1.193–1.703)

The short case shows no improvement. These measurements cover the local HTTP/parser/capture/read path; they do not measure real model generation or remote network latency. Node 24.22.1, TZ=UTC, otherwise idle host.

# The optional module selects the historical capture implementation;
# the production HTTP parser and benchmark stay identical on both sides.
node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx scripts/bench-handoff-capture.ts [historical-capture-module.ts]

Remaining merge gate

The repository requires real Codex four-phase coverage before merging request-pipeline changes. The official zero-token E2E_CHECK=1 preflight found codex-cli 0.167.0-alpha.4 and the built proxy, but the configured default upstream at http://127.0.0.1:8199/v1 was unreachable. The real four-phase test remained skipped behind its existing gate. Real-provider Codex E2E has not passed and remains required before merge. No gate was weakened.

History check

Checked Afloat16's indexed issue history and actual related PR file patches; none previously changes this capture module. The complete #4248 implementation introduces the repeated cumulative byte metering and is unchanged in this part of master. #4860 concerns capture acceptance after the client closes, not byte accounting. Focused all-state searches found no equivalent incremental-metering change.

Base: master a8bbb8d23108ba9148d09512927041364625d910 (0.1.214). Full-suite status remains hosted CI on the exact head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Handoff reply capture repeatedly scans the full text after every stream delta

2 participants