Stream message_update as deltas: cumulative partial broadcasts are O(L^2) (208 MiB wire for one 10k-token response)

[ashwin-pc]

Problem

pi emits the full cumulative partial assistant message on every message_update, and pi-web broadcasts that envelope verbatim to every connected WebSocket client (server/session/hostEvents.ts forwards the pi event; server/realtime.ts:39-43 does one JSON.stringify + send per client). Over a response of length L this serializes and transmits every prefix of the message — O(L²) total bytes per response per client.

The RealtimeHub replay log also retains the last 1,000 envelopes for reconnect catch-up, which under streaming means holding many large cumulative partials in memory even though a reconnecting client re-syncs state anyway.

Measured evidence (isolated benchmark, origin/main @ 32f7afb)

Fixed-rate local provider (100 tok/s), one WS client, isolated instance:

Response length	WebSocket wire bytes
1,000 tokens	~3.68 MiB
10,000 tokens	208.44 MiB (~56.7× for 10× the tokens)

Related pressure at fan-out (same benchmark, 3×1,000-token turns per session):

Concurrent sessions	Per-session tok/s	Bystander HTTP p99	Server CPU mean
1	100.10	0.76 ms	6.7%
8	99.85	64.2 ms	24.1%
16	99.86	200.4 ms	43.6%

Throughput is not yet limited (per-session rate stays flat through N=16, CPU has headroom, and delivery lag stayed ~1 ms p99 even at token 10,000 of a 10k response). But the superlinear wire volume is the first known scaling cliff: faster models, longer responses, or more connected clients multiply it directly, and it is the most plausible driver of the p99 growth above.

Proposed fix

1. Delta-only streaming transport: for message_update, broadcast only the text/thinking delta plus session id, message index, and a sequence number. Clients accumulate locally. The complete message still arrives on message_end (already broadcast), so any drift self-heals at message boundaries; the existing seq/sync_required machinery covers reconnects.
2. Exclude message_update envelopes from the reconnect replay log — a reconnecting client refreshes state/messages anyway; replaying stale partials is pure cost.
3. Optional hardening: coalesce/throttle message_update broadcasts (e.g. ≥30–50 ms per session) so worst-case fan-out work is bounded independent of provider chunk rate.

Wire-compat note: gate on a client capability flag or bump the envelope version so older clients keep receiving cumulative partials until updated.

Acceptance criteria

• WebSocket bytes for one streamed response scale ~linearly with token count (10k-token response within ~2× of 10× a 1k-token response, not ~57×).
• Client transcript rendering during streaming is unchanged (deltas accumulate correctly across tool calls, thinking blocks, and multi-message turns), and message_end reconciles the final message.
• Reconnect during an active stream converges to the correct transcript (replay log without message_update + state refresh).
• Bystander p99 at N=16 concurrent streaming sessions improves materially from the ~200 ms baseline.
• Regression check: the parallel-session benchmark driver (fake 100 tok/s provider + WS byte accounting) is added under tests/ or scripts/ and asserts the linear-bytes budget.

Relation to prior issues

• pi-web server becomes unresponsive over time: event-loop saturation from per-token broadcast fan-out + session-file fd leak #24 — earlier event-loop saturation incident; fixed lifecycle/fd leaks and identified per-event broadcast fan-out, but the cumulative-partial transport remained.
• Session performance contract: constant-time opens, shallow reconciliation, and isolated history work #74 — session performance contract; this was documented there as the known remaining streaming-path flaw (finding F5) and explicitly left out of scope.