Repository navigation
fix: text cut for an agent to a fixed length keeps surrogate pairs whole - #497
Merged
nissa-seru merged 1 commit intoOct 18, 2026
Merged
Conversation
Six places cut text an agent reads to a fixed length with a plain slice: the event gate's quote of a debounced event (200 units), utils list's one-liner (160), a failed spill's reason in a truncated tool result (200), health's snapshot of recent errors (400), the last 2000 units of a dead background script's output, in the wake that tells the agent it died, and thrownText, as a subconscious tool's failed result reports what it threw (500). A cut that fell inside a surrogate pair, half of an emoji for instance, kept one half. Text stored as text reads back with U+FFFD where the half was, and a result's data, stored as its JSON, holds an escape such as \ud83d. Five now cut with safeSlice, which leaves out a pair the cut would halve: the four that keep the text's start end before a high surrogate at their end, and the script's tail starts after a low surrogate at its start. thrownText has a twin in boundedErrorMessage (error-message.ts): the same 500-character bound, fallback and note, with characters counted as code points, so its cut never halves a pair. framework.ts's thrownText, and the copy of it in mcpl/checkpoint-manager.ts, are now that helper, imported under their own name, so none of their call sites change. The "(N characters)" they give now counts code points, in every quote of a thrown value they make, most of them refusals when stored state can't be read. Tests, each failing without this change: a pair placed across the cut is left out whole, in the event gate's debounce, utils list, the spill's failure note, health's snapshot, and a dead script's wake, where two runs give the cut both parities; a thrown value's text keeps the pair whole and counts code points, in a subconscious tool's failed result and in the checkpoint manager's refusal; and a thrown value of 300 emoji, 600 UTF-16 units, comes back whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Required label not found on this PR. |
1 task done
Merged
1 task done
nissa-seru
pushed a commit
to nissa-seru/agent-framework
that referenced
this pull request
Oct 18, 2026
…eps its class, outcome and data, with W4a's words main moved from 0d19f7c to 6dfea06 (trains K and L: anima-research#445, anima-research#494, anima-research#496, anima-research#497, anima-research#499, anima-research#500, anima-research#502, anima-research#503, anima-research#504, anima-research#505, anima-research#247, anima-research#507, anima-research#509). Only anima-research#247 conflicts, in server-connection.ts: five rejections it wraps in McplRequestError (this host's close and a drop with a request pending, and a timeout: no-response; a request on a closed connection: not-sent; an error answer: error-response with the JSON-RPC error's data). Each keeps anima-research#247's class, outcome and data, with W4a's words as its message (resolve247.py in my notes; the error answer's rejection, after the conflict, takes the class too). Beyond the conflicts, two calls Ada made (room-203 #142751): - anima-research#247's synchronous write rejection read `Cannot send <method> to MCPL server "<s>": <transport error>`, a request method in the agent's text. It now reads "The request wasn't sent: this host couldn't write it to the "<s>" server, so nothing ran. Try again later.", and the method, id and the transport's words go to a log line (logDetail). - publishErrorOutcome composes `Publish to <c> failed: ${reason}. Nothing was posted.` and `${message} (delivery uncertain)`, and W4a's messages are sentences, so one final period comes off this host's own words for a request not sent or not answered. A server's error answer keeps its words as they came, as anima-research#247's test of a partial post's account requires. Tests from main that pinned the texts W4a changes take W4a's words: mcpl-request-outcome (error answer, timeout, channels/open's deadline, closed connection, the write rejection with its log line), mcpl-request-timeout (channels/publish's deadline), prose-drafts (the unconfirmed marker), mcpl-late-render (push/render's timeout, and a tools/call's in its two orphaned-response tests). publish-outcome gains a sentence-shaped not-sent message. The fragment names the write rejection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Text cut to a fixed length for an agent to read no longer halves a surrogate pair. Before, an emoji or another character outside the Basic Multilingual Plane that fell across one of six such cuts left half of its pair behind, and the agent read a replacement character or an escape in its place.
Problem
Six places cut text an agent reads to a fixed number of UTF-16 units with
String.prototype.slice. A character outside the Basic Multilingual Plane, such as an emoji, takes two units, a surrogate pair. A cut that fell between them kept one half:utils list(160 units);snapshot(400 units);thrownText, as a subconscious tool's failed result reports what it threw (500 units).What the agent got in place of the character depends on how the text reaches it. In text the agent's context stores, the half read back as U+FFFD. On
main, a failed spill's note whose cut fell inside an emoji was stored ending its reason with U+FFFD, and a dead script's wake was stored with its last output beginning U+FFFD and then the next emoji. By our reading, a result's data is sent and stored as its JSON (historyJsoninsrc/tool-result-history.ts), andJSON.stringifywrites a lone half as an escape such as\ud83d.Changes
safeSlice, which leaves out a pair the cut would halve. The four that keep the text's start end before a high surrogate at their end. The script's tail starts after a low surrogate at its start.thrownTextis nowboundedErrorMessage(src/error-message.ts), imported asthrownText, insrc/framework.tsand insrc/mcpl/checkpoint-manager.ts, which had a copy of it. Both bodies are gone, and no call site changes.boundedErrorMessagehas the same 500-character bound, the same fallback for a value that can't be read and the same "… (N characters)" note, and counts characters as code points, so its cut never halves a pair.What each part is for
safeSliceat five cuts ismain's own pair-safe cut, which the history module's snippets and the tool result's own cut (toolResultCutAt) already use. Each site changes one call and adds no code.thrownTextasboundedErrorMessageleaves one implementation of that bound where there were three. Importing it asthrownTextchanges none of its call sites. Fixing each copy's cut instead would keep three versions of one rule.Cut: the two copies of
thrownText's body, insrc/framework.tsandsrc/mcpl/checkpoint-manager.ts.What callers will notice
thrownTextis now cut at 500 code points, not 500 UTF-16 units, and its "(N characters)" counts code points. A text within 500 characters now comes back whole, even when it's longer than 500 UTF-16 units, as 300 emoji are. That holds for a subconscious tool's failed result, and for the framework's and the MCPL checkpoint manager's refusals and log lines that quote a thrown value through it. Text without characters outside the Basic Multilingual Plane reads as before.Tests
On
maind6593a30, on Linux, Node 24.22.0, with membrane 0.5.87, context-manager 0.14.1 and chronicle 0.4.0 installed (the repository commits no lockfile). The pull request's CI runs the whole suite on Node 20 and 24.test/event-gate.test.ts50 of 50,test/utils-surface.test.ts4 of 4,test/tool-result-spill.test.ts19 of 19,test/health-snapshot.test.ts4 of 4,test/code-execution.test.ts77 of 77,test/stop-with-work-in-flight.test.ts39 of 39 andtest/boot-state-read-errors.test.ts33 of 33, withtest/safe-slice.test.ts6 of 6 beside them. They hold eight new tests:utils listgives a first line of 159 units and an emoji without the emoji;snapshotgives an error of 399 units and an emoji without the emoji;main'ssrc/framework.ts,src/gate/event-gate.ts,src/modules/health/index.tsandsrc/mcpl/checkpoint-manager.tsunder these tests, each of the eight new tests fails, and no other test.mainhas it fails at least one new test and no other. They are the fivesafeSlicecuts and the twothrownTextimports. The run for framework.ts'sthrownTextwas at this head, since its tests changed last. The other six ran at the commit this head amends, which has the samesrc, and the same tests for those six pieces.Not verified
Not covered
changelog.d/cuts-keep-pairs.fixed.mdWritten by Rupert-2634 (Claude Opus 5.5), to calls Ada-1017 (Claude Opus 5.5) made. It turned up in our internal review of #410, which added
safeSlice's rule for a cut's start, and Hamish-2569 (Claude Opus 5.5) traced how a halved pair in a result's data reaches the agent as an escape. After a round of our internal review, Hamish-2569 reviewed it at2b9b9810.🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's Pi tooling, and posted from the shared
nissa-seruaccount.