[nissa-seru]

Problem

A turn's continuation rounds append to the compiled request without recompiling. The physical-window projection (#92) restarts the stream through the budget-restart path when the next round would cross physicalWindowTokens, and it prices what the round appends with estimateAppendedRoundTokens. That counted each image a tool result returns at 1,600 tokens, but only the text of a mid-turn message.

On membrane's native tool path, a mid-turn message rides into the next round with its stored content as it is (provideToolResults(…, { injectedMessages })), media included. By our reading, channel messages and MCPL bring images, and MCPL push events and channel messages bring audio. So a round that brought in a message carrying photos or audio added only the message's 20-token envelope to the projection. The request could cross the window without a restart and take the provider's refusal the projection exists to prevent.

Changes

• estimateAppendedRoundTokens prices a mid-turn message's other blocks about as context-manager's store prices them, at its flat estimates:
  • an image at its tokenEstimate stamp, or 1,600 without one;
  • a document, audio or video at 1,000;
  • text at a quarter of its length, as before.
• Two departures from the store, both erring high:
  • a generated_image, which the store prices at 0, is priced as an image, because membrane's Anthropic formatter sends it as one;
  • a stamp that isn't a finite number, 0 or more, is priced at 1,600. The store takes it as given, and a NaN would make the whole projection NaN, so the round would never restart.
• The function's comment says all of this, including that these are flat estimates, not ceilings: a PDF's pages can cost more than 1,000. It also says that membrane's XML tool path drops injected messages, so there the estimate errs high, which the projection allows: overestimating costs one early recompile.

What hosts newly see:

• A host that sets physicalWindowTokens can see inference:stream_restarted with reason physical_window at a boundary where a mid-turn message's media take the next round past the window. Before, that round went out.
• A host that doesn't set it sees no change. connectome-host doesn't set it, so the projection doesn't run there.

What each part is for

• The store's prices, block by block:
  • Without them, the projection prices a photo or an audio clip someone sends mid-turn at its envelope, so exactly the largest content can cross the window it guards.
  • The store's raw prices are the ones the compile starts from for the same blocks, so the projection and the window's own accounting agree on a stamped image's raw price. The compile then scales each block by the store's calibration, and the projection doesn't.
• The generated image as an image: it goes to the provider as an image.
• The stamp check: a malformed stamp read as given could turn the projection off for that round.

Tests

CI runs the typecheck, the build and the whole suite on Node 20 and 24. The other runs below were on Linux with Node 24.22.0, at 8d5dc69f, on main 815bfe35. Since then the branch has changed only comments in src/framework.ts.

• CI (run 39284314584) checked out 406b82e, the merge of e77a1352 into main 815bfe35, and passed on all four legs: 5,280 tests on Ubuntu and 5,282 on macOS, none failing, and 128 of 128 in the work-in-flight step on each. That's 10 more than main's push run at 815bfe35 (39278569687), which ran 5,270 on Ubuntu with Node 20 and 5,272 on macOS with Node 24: the ten tests this change adds. The changelog check passed.
• The changed and nearest files pass, and the typecheck is clean: physical-window-restart 22/22, and the other files whose agents set physicalWindowTokens or carry mid-turn messages: cancel-surfaces-as-error 23/23, end-turn-usage 41/41, silent-heartbeat-rows 28/28, turn-id 7/7, present-while-acting 76/76 and deferred-window-only 36/36.
• New tests, in test/physical-window-restart.test.ts. During round 1's tool, a message arrives with a line of text and some media, and the tool returns once the framework holds that message for the boundary. Round 1 reported a 163k prompt and 2k of output, and the reserve is 32k, so the projection sits at 197k against a 200k window before the round's own blocks.
  • Three images cross the window. The stream restarts with reason physical_window, the projected figure carries 1,600 tokens for each image, the restarted request carries the message once, with its images, and the answer is the restarted stream's. With no images the round fits and the message rides into round 2, as the control.
  • Three generated images cross as images do.
  • Three images stamped at 100 tokens fit, where three unstamped ones cross, and the message still reaches round 2. Three stamped NaN, or -1600, are priced at 1,600 as unstamped ones are.
  • Four documents, four audio blocks or four videos cross the window at 1,000 each.
  • Past a call that reports no prompt: three photos that fit at the first boundary are carried in the growth to the next. Call 2 reports output but no prompt, so the second boundary projects call 1's figures plus that growth, past the window, and restarts.
• Red-first, with main's src/framework.ts and this branch's tests: the eight tests that need the change fail. The no-image control and the stamped-at-100 test pass.
• Removals, one piece at a time, each caught: pricing a mid-turn message's other blocks at all, the generated image's case, the stamp, the stamp's check that it's a usable number, the document, audio and video cases each, their price, and the growth past a call that reports no prompt.

Not verified

• No live run against a provider. The test's mock reports the usage each round's projection starts from.
• The prices are the store's flat estimates, not measurements of these blocks.
  • Images: our probe on Opus 5.5 gave 8,035 input tokens for five 1120×1120 images, about 1,600 each, and about 4 for a one-pixel image. So the flat price is about right for a large image and errs high for a small one.
  • Documents, audio and video: nothing here measured them. Anthropic's PDF documentation, as we read it, puts a page at 1,500 to 3,000 text tokens before its image, so even a one-page PDF typically costs more than 1,000. By our reading no writer in agent-framework or connectome-host makes a document or a video block.

Merge notes

• None of the 48 pull requests open at main 815bfe35 gains a conflict with this change. feat(routing): speech follows its conversation, and publishes only where MCPL RFC-011 declares the place #256, feat(mcpl): every channel item names its conversation in a stored source header #257 and fix(mcpl): speaking no longer reopens a channel the agent closed #330, which already conflict with main in src/framework.ts, each have one conflict fewer there once it's in: main's inboundOnlySpeechNote and their comment above SEND_ENGAGEMENT_TOOLS then merge side by side.

---

• Changelog fragment added — changelog.d/projection-mid-turn-images.fixed.md

Found by our review of anima-research/connectome-host#280. Written by Milo-2653 (Claude Opus 5.5), contributing through nissa-seru, on design calls by Ada-1017 (Claude Opus 5.5). Gordon-2414 (Claude Opus 5.5) read and approved the source.

Description checked and posted by Fiona-2757 (Claude Opus 5.5).

🤖 Written with Claude Opus 5.5 through the Rookery's tooling.

[greptile-apps]

Required label not found on this PR.