Skip to content

[PR] perf(stream): serialize request evidence only for filtered JSON replies - #6597

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_lazy-json-echo-history
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-13_lazy-json-echo-history

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #6596

Why

Each non-streaming JSON reply currently serializes ctx.messages in full, even for ordinary short prose and when absorb is disabled. This is synchronous history-sized work on the shared proxy process. The string is only needed by absorb's quote-preservation check when a reply actually enters the prose filter.

Change

Delay request-history serialization behind the existing filter gate and absorb-enabled check across Anthropic, OpenAI Chat, Responses and Gemini. Reuse evidence across Anthropic blocks. Gemini prechecks its response parts and captures any needed request evidence before yielding proxy tools, preserving the existing snapshot timing. Filtering rules, user quotations, signed thoughts and real tool arguments retain their behavior.

The patch includes a reproducible benchmark of the real async rewriters and compatibility coverage. No new configuration or protocol behavior.

Measurements

Base e1f31ad9d5b47500362699a9867819b861c7de96 (0.1.213); Node 24.22.1/Linux. For each history size, three fresh before/after process pairs alternate ordering. Each process runs two warmups and five measured calls per protocol/mode. The 8 MiB input contains 1,024 valid CoreMessage entries and the response is a fixed short answer.

Median of per-process medians:

Protocol 8 MiB, absorb off: base → change (ms) 8 MiB, absorb on: base → change (ms)
Anthropic 10.216 → 0.009 9.515 → 0.002
OpenAI Chat 9.881 → 0.003 10.388 → 0.002
Responses 9.902 → 0.003 10.560 → 0.003
Gemini 10.633 → 0.027 10.287 → 0.019

Median whole-process peak RSS for the combined eight-case run: 183,504 KiB → 128,184 KiB. Every per-protocol full-response SHA-256 digest matched across revisions and samples.

The 1 KiB control is mixed, with all medians below 0.05 ms: Gemini absorb-off 0.031 → 0.044 ms and absorb-on 0.019 → 0.027 ms. These results establish removal of the large-history serialization cost in this synthetic rewriter workload; they are not production HTTP latency measurements. Output probing/filtering still runs, and armed replies requiring quote evidence still serialize history.

Reproduce one fresh process with:

node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx scripts/bench-json-rewrite-history.ts 8388608

Verification

  • 26 compatibility controls pass on unchanged base and on the change: four-protocol Unicode prose; absorb on/off; preserved literal user quote; distinct-ref negative control; render tags; multiple prose fields; untouched signed Gemini thoughts and real tool arguments. These are behavior controls, while the benchmark demonstrates the performance defect.
  • Final changed tree: 165 tests pass, 0 fail, across json-rewrite-history, tag-echo, absorb-emission and external-summary-runtime suites.
  • Typecheck, complete build, unused-export gate, test-import-graph gate and git diff --check pass.
  • Remote tree matches the tested local tree exactly. Full regression remains hosted CI.

The linked issue records all-state duplicate screening and triage. Topic N (server response processing), cross-client; suggested p2, scope/wiring.

@ranxianglei ranxianglei changed the title perf(stream): serialize request evidence only for filtered JSON replies [PR] perf(stream): serialize request evidence only for filtered JSON replies Nov 13, 2026
@ranxianglei
ranxianglei merged commit 22d4767 into ranxianglei:master Nov 14, 2026
45 of 50 checks passed
@ranxianglei

Copy link
Copy Markdown
Owner

Superseded by #6677 (merged as 3858b6956, 2026-11-14).

Landed into master via the real merge commit 22d47678eba (Merge Afloat16/2026-11-13_lazy-json-echo-history (PR #6597)), so your original commits are preserved verbatim in master history — only hashes of the batch wrapper changed, your authorship and commit messages are intact.

The fix for #6596 is now in master. Closing this PR as superseded — thanks for the careful, well-tested fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Non-streaming response rewriters serialize full request histories for ordinary prose

2 participants