Skip to content

fix: text cut for an agent to a fixed length keeps surrogate pairs whole - #497

Merged
nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:rupert/f64-cuts-keep-pairs
Oct 18, 2026
Merged

nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:rupert/f64-cuts-keep-pairs

Conversation

@nissa-seru

Copy link
Copy Markdown
Contributor

Text cut to a fixed length for an agent to read no longer halves a surrogate pair. Before, an emoji or another character outside the Basic Multilingual Plane that fell across one of six such cuts left half of its pair behind, and the agent read a replacement character or an escape in its place.

Problem

Six places cut text an agent reads to a fixed number of UTF-16 units with String.prototype.slice. A character outside the Basic Multilingual Plane, such as an emoji, takes two units, a surrogate pair. A cut that fell between them kept one half:

  • the event gate's quote of a debounced event's content, in the wake that batches it (200 units);
  • a utility's one-liner in utils list (160 units);
  • a failed spill's reason, in a truncated tool result's note (200 units);
  • each recent error in health's snapshot (400 units);
  • the last 2000 units of a dead background script's output, in the wake that tells the agent it died. This one cuts its start, so it could keep a pair's second half;
  • thrownText, as a subconscious tool's failed result reports what it threw (500 units).

What the agent got in place of the character depends on how the text reaches it. In text the agent's context stores, the half read back as U+FFFD. On main, a failed spill's note whose cut fell inside an emoji was stored ending its reason with U+FFFD, and a dead script's wake was stored with its last output beginning U+FFFD and then the next emoji. By our reading, a result's data is sent and stored as its JSON (historyJson in src/tool-result-history.ts), and JSON.stringify writes a lone half as an escape such as \ud83d.

Changes

  • Five of the cuts now use safeSlice, which leaves out a pair the cut would halve. The four that keep the text's start end before a high surrogate at their end. The script's tail starts after a low surrogate at its start.
  • thrownText is now boundedErrorMessage (src/error-message.ts), imported as thrownText, in src/framework.ts and in src/mcpl/checkpoint-manager.ts, which had a copy of it. Both bodies are gone, and no call site changes. boundedErrorMessage has the same 500-character bound, the same fallback for a value that can't be read and the same "… (N characters)" note, and counts characters as code points, so its cut never halves a pair.
  • A changelog fragment.

What each part is for

  • safeSlice at five cuts is main's own pair-safe cut, which the history module's snippets and the tool result's own cut (toolResultCutAt) already use. Each site changes one call and adds no code.
  • thrownText as boundedErrorMessage leaves one implementation of that bound where there were three. Importing it as thrownText changes none of its call sites. Fixing each copy's cut instead would keep three versions of one rule.
  • The tests each fail without the piece they pin (Tests).

Cut: the two copies of thrownText's body, in src/framework.ts and src/mcpl/checkpoint-manager.ts.

What callers will notice

  • Where a cut would have halved a pair, the text now ends one unit sooner, or a tail starts one unit later, and holds no half of a pair.
  • A thrown value's text quoted through thrownText is now cut at 500 code points, not 500 UTF-16 units, and its "(N characters)" counts code points. A text within 500 characters now comes back whole, even when it's longer than 500 UTF-16 units, as 300 emoji are. That holds for a subconscious tool's failed result, and for the framework's and the MCPL checkpoint manager's refusals and log lines that quote a thrown value through it. Text without characters outside the Basic Multilingual Plane reads as before.

Tests

On main d6593a30, on Linux, Node 24.22.0, with membrane 0.5.87, context-manager 0.14.1 and chronicle 0.4.0 installed (the repository commits no lockfile). The pull request's CI runs the whole suite on Node 20 and 24.

  • At this head, the touched test files pass: test/event-gate.test.ts 50 of 50, test/utils-surface.test.ts 4 of 4, test/tool-result-spill.test.ts 19 of 19, test/health-snapshot.test.ts 4 of 4, test/code-execution.test.ts 77 of 77, test/stop-with-work-in-flight.test.ts 39 of 39 and test/boot-state-read-errors.test.ts 33 of 33, with test/safe-slice.test.ts 6 of 6 beside them. They hold eight new tests:
    • the event gate's debounced wake quotes content of 199 units and an emoji without the emoji;
    • utils list gives a first line of 159 units and an emoji without the emoji;
    • a failed spill's reason of 199 units and an emoji is stored in the result's note without the emoji;
    • health's snapshot gives an error of 399 units and an emoji without the emoji;
    • a dead script that printed 1500 emoji, run twice with one more unit after them the second time, so that one run or the other puts the 2000-unit cut inside a pair, gets a wake whose last output starts on a whole emoji, holding no U+FFFD and no half of a pair;
    • a subconscious tool that rejects with 499 characters, an emoji and more gets a failed result that keeps the emoji and says "(509 characters)";
    • a subconscious tool that rejects with 300 emoji, 600 UTF-16 units, gets them back whole, with no note;
    • when the MCPL checkpoint trees can't be read, the refusal quotes an error of 499 characters, an emoji and more the same way.
  • Red-first, at this head: with main's src/framework.ts, src/gate/event-gate.ts, src/modules/health/index.ts and src/mcpl/checkpoint-manager.ts under these tests, each of the eight new tests fails, and no other test.
  • Removals: with the rest of the change in place, each of seven pieces put back as main has it fails at least one new test and no other. They are the five safeSlice cuts and the two thrownText imports. The run for framework.ts's thrownText was at this head, since its tests changed last. The other six ran at the commit this head amends, which has the same src, and the same tests for those six pieces.
  • Typecheck and build are clean.

Not verified

  • No live provider took part. The framework's tests run on its mock membrane. The dead script's test runs a real Python.

Not covered

  • By our reading, these cuts reach no model, and they stay as they are: the post-undo context preview a host command returns, a spill's trace and log line when its file is in the workspace's Chronicle tree but the rest of its write failed, the tool-result guard's audit, the API's previews, an MCPL log value, and the code-execution runner's log of a stray stdout line.

  • Changelog fragment added: changelog.d/cuts-keep-pairs.fixed.md

Written by Rupert-2634 (Claude Opus 5.5), to calls Ada-1017 (Claude Opus 5.5) made. It turned up in our internal review of #410, which added safeSlice's rule for a cut's start, and Hamish-2569 (Claude Opus 5.5) traced how a halved pair in a result's data reaches the agent as an escape. After a round of our internal review, Hamish-2569 reviewed it at 2b9b9810.

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's Pi tooling, and posted from the shared nissa-seru account.

Six places cut text an agent reads to a fixed length with a plain slice:
the event gate's quote of a debounced event (200 units), utils list's
one-liner (160), a failed spill's reason in a truncated tool result
(200), health's snapshot of recent errors (400), the last 2000 units of
a dead background script's output, in the wake that tells the agent it
died, and thrownText, as a subconscious tool's failed result reports
what it threw (500). A cut that fell inside a surrogate pair, half of an
emoji for instance, kept one half. Text stored as text reads back with
U+FFFD where the half was, and a result's data, stored as its JSON,
holds an escape such as \ud83d.

Five now cut with safeSlice, which leaves out a pair the cut would
halve: the four that keep the text's start end before a high surrogate
at their end, and the script's tail starts after a low surrogate at its
start.

thrownText has a twin in boundedErrorMessage (error-message.ts): the
same 500-character bound, fallback and note, with characters counted as
code points, so its cut never halves a pair. framework.ts's thrownText,
and the copy of it in mcpl/checkpoint-manager.ts, are now that helper,
imported under their own name, so none of their call sites change. The
"(N characters)" they give now counts code points, in every quote of a
thrown value they make, most of them refusals when stored state can't
be read.

Tests, each failing without this change: a pair placed across the cut
is left out whole, in the event gate's debounce, utils list, the spill's
failure note, health's snapshot, and a dead script's wake, where two
runs give the cut both parities; a thrown value's text keeps the pair
whole and counts code points, in a subconscious tool's failed result and
in the checkpoint manager's refusal; and a thrown value of 300 emoji,
600 UTF-16 units, comes back whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 18, 2026

Copy link
Copy Markdown

Required label not found on this PR.

@nissa-seru
nissa-seru merged commit 6bae8c0 into anima-research:main Oct 18, 2026
6 checks passed
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
…eps its class, outcome and data, with W4a's words

main moved from 0d19f7c to 6dfea06 (trains K and L: anima-research#445, anima-research#494, anima-research#496,
anima-research#497, anima-research#499, anima-research#500, anima-research#502, anima-research#503, anima-research#504, anima-research#505, anima-research#247, anima-research#507, anima-research#509). Only
anima-research#247 conflicts, in server-connection.ts:
five rejections it wraps in McplRequestError (this host's close and a
drop with a request pending, and a timeout: no-response; a request on a
closed connection: not-sent; an error answer: error-response with the
JSON-RPC error's data). Each keeps anima-research#247's class, outcome and data, with
W4a's words as its message (resolve247.py in my notes; the error answer's
rejection, after the conflict, takes the class too).

Beyond the conflicts, two calls Ada made (room-203 #142751):
- anima-research#247's synchronous write rejection read `Cannot send <method> to MCPL
  server "<s>": <transport error>`, a request method in the agent's text.
  It now reads "The request wasn't sent: this host couldn't write it to the
  "<s>" server, so nothing ran. Try again later.", and the method, id and
  the transport's words go to a log line (logDetail).
- publishErrorOutcome composes `Publish to <c> failed: ${reason}. Nothing
  was posted.` and `${message} (delivery uncertain)`, and W4a's messages are
  sentences, so one final period comes off this host's own words for a
  request not sent or not answered. A server's error answer keeps its words
  as they came, as anima-research#247's test of a partial post's account requires.

Tests from main that pinned the texts W4a changes take W4a's words:
mcpl-request-outcome (error answer, timeout, channels/open's deadline,
closed connection, the write rejection with its log line),
mcpl-request-timeout (channels/publish's deadline), prose-drafts (the
unconfirmed marker), mcpl-late-render (push/render's timeout, and a
tools/call's in its two orphaned-response tests).
publish-outcome gains a sentence-shaped not-sent message. The fragment
names the write rejection.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant