feat(workspace)!: read, grep, edit and edit_many answer as the text the model reads

changelog.d/cedric-f55-workspace-text.breaking.md

@@ -0,0 +1,49 @@
+- **Scripts, hosts and modules that read the workspace's `read`, `grep`,
+  `edit` or `edit_many` results:** each now answers as the text the model
+  reads, with its fields beside it in `structured`. The model read each as
+  one line of JSON, a page's lines escaped inside it. It now reads a header
+  of the result's fields, one `key: value` to a line (`path: …`,
+  `totalLines: …`, `nextOffsetChars: null`), then labelled sections: a
+  page's `--- content ---`, grep's matches by file and version (`N:` for a
+  match, `N-` for a line of its context, as grep writes them), what grep
+  skipped, and a receipt's `--- files ---` and `--- changed ---`. A value
+  outside a page's content and a receipt's changed lines that holds a line
+  break (LF, CR, VT, FF, NEL, U+2028 or U+2029) or starts with `"`, such as
+  a path or a line grep matched, is written as a JSON string, so a value
+  shown as it is never starts with `"`. A control character in a file
+  reaches the model as itself, where JSON wrote an escape such as `\u001b`.
+
+  **Who needs to act:**
+  - a `code_execution` script that parses one of these results' text as
+    JSON (`JSON.parse(result.content[0].text)`): read
+    `result.structuredContent`, which holds the fields the JSON held;
+  - a host or module that reads these results' fields from `result.data`,
+    through `executeToolCall` or `ctx.callTool`: read `result.structured`.
+    connectome-host reads either since 0.11.0
+    (anima-research/connectome-host#324), and needs that version or later:
+    before it, its file viewers show such a file as empty.
+
+  **What a page holds:** a page is held to the call's result limit as its
+  text measures it, where it was measured as JSON, and so is a receipt.
+  - A model's page holds more of a file whose lines JSON escapes: at the
+    default 24,000 characters, a page of this repository's
+    `src/modules/workspace/index.ts` now ends at line 475, where it ended at
+    line 454. A spilled result's notice sizes the read-back page it suggests
+    the same way, so below a cap of about 12,300 characters it suggests a
+    longer one than before.
+  - A script's page is never shorter than before, since a page's text is
+    shorter than its JSON, unless the text quotes its path or a note. A script's answer carries the result's fields, a
+    page's content among them, beside the text, and the limit counts the
+    text alone. A script's limit counts each character at the six bytes it
+    takes at most in the answer, and only a control character takes more
+    than three, so the two together can pass what the script takes only on
+    a page of more than half that limit (about 3.7 million characters at
+    the default 128 MiB of script memory) that holds control characters.
+    Such a page answers `ResultTooLarge`, which cuts nothing, and the script
+    reads a smaller page.
+
+  **Unchanged:** the tools' descriptions and inputs; the words of their
+  notes; their errors, which were text already; the results of `write`,
+  `delete`, `ls`, `glob`, `status`, `materialize`, `sync` and the other
+  workspace tools, whose fields stay in `data`; and a page for a call made
+  in process without a result limit, which holds to its line limit alone.

changelog.d/feat-workspace-read.changed.md

@@ -1,6 +1,6 @@
 - The workspace's `read` shows a file's text as it is, and says so. A line page shows a CRLF file's lines without the CR before each line feed, and leaves out a BOM. A line whose own text ends in a CR shows its line end's CR too, since `edit` reads a CR before a line feed as part of the line end, so text copied from the line still matches it. Its note says when the file has CRLF ends or a BOM, counting a mix of CRLF and LF ends and the lines that show both CRs. It also says when the file isn't valid UTF-8: each invalid byte sequence shows as U+FFFD, `edit` refuses the file, and `write` replaces it whole. Before, each line of a CRLF file ended in a CR, a BOM came first in the first line as U+FEFF, and invalid bytes showed as U+FFFD with no note. A character page still reads the stored text as it is, BOM and CRs included.
 - On a page for a call that carries a `resultMaxChars`, such as a model's or a running script's, a line over 2,000 characters is cut, with a marker that gives the `offsetChars` and `limitChars` that read its rest. Before, a line page showed every line whole, however long.
-- A `read` page stops before its result would pass `resultMaxChars`, the most of it that can reach its caller whole. For a model's call that's its agent's inline cap, 24,000 characters by default, within which a result is stored whole unless its round's results together pass the readers' bound; for a running script's call it's what the script's memory takes, 7,456,527 characters at the default 128 MiB. Its note says where to continue: the `offset` after its last line, or for a character page the `offsetChars` it also gives as `nextOffsetChars`, with the `limitChars` left of what it was asked for, so a reader that follows the notes ends where it asked to. Where even a page's first line would pass the limit, that line is cut shorter, at the most the page has room for. Where a page's last line is cut, its note names the `offsetChars` where that line's rest starts and the `limitChars` that reads it to the line's end, before the next line's `offset` if the file goes on. A page always moves on, so where its own fields (the path and its notes) and the least it can show pass a small limit, it passes the limit. At 1,000 characters, a page with no notes does that only for a path of about 680 characters or more. Its notes take room from that: with all three notes on a file's text and a conflict's, a path of 93 to 129 characters does it, and a note that names a symlink's target, as one saying disk couldn't be checked can, takes as much room as the target is long. Before, a page of up to 2,000 lines, or of `limitChars` characters, went out whole, and for a result that passed the limit the framework kept only its start, cut mid-JSON, writing the whole of it to a workspace file where the agent has one. A caller that pages 2,000 lines at a time and starts the next page at line 2,001 now starts where the note says. A running `code_execution` script's call has that larger limit, so its line pages still stop at the line limit, 2,000 by default, and its character pages at `limitChars`, unless a page would pass that. Its line pages cut a line over 2,000 characters and leave out CRs and a BOM, as a model's do, and a character page gives it the stored text exactly. A call made in process, through `handleToolCall`, `executeToolCall` or `ModuleContext.callTool`, carries no `resultMaxChars` unless its caller sets one. Its line pages stop at the line limit alone and show every line whole, as they did before, though now without the CRs and the BOM.
+- A `read` page stops before its result would pass `resultMaxChars`, the most of it that can reach its caller whole. For a model's call that's its agent's inline cap, 24,000 characters by default, within which a result is stored whole unless its round's results together pass the readers' bound; for a running script's call it's what the script's memory takes, 7,456,527 characters at the default 128 MiB. Its note says where to continue: the `offset` after its last line, or for a character page the `offsetChars` it also gives as `nextOffsetChars`, with the `limitChars` left of what it was asked for, so a reader that follows the notes ends where it asked to. Where even a page's first line would pass the limit, that line is cut shorter, at the most the page has room for. Where a page's last line is cut, its note names the `offsetChars` where that line's rest starts and the `limitChars` that reads it to the line's end, before the next line's `offset` if the file goes on. A page always moves on, so where its own fields (the path and its notes) and the least it can show pass a small limit, it passes the limit. At 1,000 characters, a page with no notes does that only for a path of about 690 characters or more. Its notes take room from that: with all three notes on a file's text and a conflict's, a path of 101 to 137 characters does it, and a note that names a symlink's target, as one saying disk couldn't be checked can, takes as much room as the target is long. Before, a page of up to 2,000 lines, or of `limitChars` characters, went out whole, and for a result that passed the limit the framework kept only its start, cut mid-JSON, writing the whole of it to a workspace file where the agent has one. A caller that pages 2,000 lines at a time and starts the next page at line 2,001 now starts where the note says. A running `code_execution` script's call has that larger limit, so its line pages still stop at the line limit, 2,000 by default, and its character pages at `limitChars`, unless a page would pass that. Its line pages cut a line over 2,000 characters and leave out CRs and a BOM, as a model's do, and a character page gives it the stored text exactly. A call made in process, through `handleToolCall`, `executeToolCall` or `ModuleContext.callTool`, carries no `resultMaxChars` unless its caller sets one. Its line pages stop at the line limit alone and show every line whole, as they did before, though now without the CRs and the BOM.
 - The workspace's `grep` leaves out a CRLF end's CR, for matching as well as display, as `read` does, so `foo$` finds a CRLF file's line too, as it does under ripgrep's `--crlf`. A match's text and context are its lines as `read` shows them, but each line whole, where a `read` page for a call with a result limit cuts a line over 2,000 characters. A BOM isn't part of the first line, and a file with CRLF ends says so in its result's `lineEndings` (`CRLF`, or `mixed`). Before, a CRLF file's lines ended in a CR, so `foo$` missed them and a match's text copied into an edit carried the CR. A pattern written to find the CR of a CRLF line end no longer matches, except on a line whose own text ends in a CR, which keeps its line end's CR as `read` shows it. When a pattern names a CR, as a literal one or as `\r`, `\x0d`, `\u000d`, `\cM` or `\015`, and a file it searched has CRLF ends, the result's `note` says how many files searched have them and names them, each as `CRLF` or `mixed`, up to 20 and then how many more. It also says that a character `read` shows every CR in the workspace's copy, or in disk's file on a direct mount where the host has one. A pattern that names U+FEFF, over a file that starts with a BOM, gets a note that names the files that start with one and says a BOM isn't part of the first line. So a `grep` for `\r` or `^\uFEFF` still finds which of the files it searches have CRLF ends or a BOM. A CR that doesn't come before a line feed is still part of its line.
 - The workspace's `read` first takes in a change on disk to a file the workspace holds, as a listing and an `edit` do, and shows that text. Its note says so when disk's copy had changed since the workspace last agreed with it. It also says when disk's copy couldn't be checked, or disk has none and nothing records whether it was deleted there, so it shows the workspace's copy; where the check gave up because the selected branch kept changing, it says to read again. A file disk deleted answers that disk deleted it and the deletion was taken in, and a file disk made binary or too large answers that it's on disk but not stored. Where the path's stored copy is a draft and disk changed or deleted it too, the read shows the draft, and its note names the conflict and the ways to settle it, a sync, or a forced materialize on a mount that isn't read-only: for a deletion, a sync of the path takes it and removes the workspace's copy. Before, a read of a path the workspace held didn't look at disk, so it could show a copy another writer had since changed; a path it didn't hold was observed, as now.
 - A direct mount's `read` and `grep` show a file's text as a store-backed mount's do, with the same pages, cut lines, notes and `lineEndings`. Only the disk check is a store-backed mount's alone, since a direct mount reads disk as it is and keeps no copy of its own.

src/framework.ts

@@ -21876,11 +21876,12 @@ export class AgentFramework {
    *  the inline cap would show more: where a cap at the readers' bound would
    *  keep a longer head, or the whole result. */
   private spillPageAdvice(path: string, cap: number, canRaise: boolean): string {
-    // JSON serialization can expand each read-back code unit to six
-    // chars (\\u0000). Leave room for the page's metadata and a possible
-    // extra code unit at a surrogate-pair boundary.
-    const pageOverhead = JSON.stringify(path).length + 256;
-    const readLimit = Math.max(1, Math.min(2000, Math.floor((cap - pageOverhead) / 6) - 1));
+    // A page is held to the cap as its text measures it, each code unit one
+    // character (the workspace's read-page.ts). Leave room for the page's
+    // fields and notes, and a possible extra code unit at a surrogate-pair
+    // boundary.
+    const pageOverhead = path.length + 256;
+    const readLimit = Math.max(1, Math.min(2000, cap - pageOverhead - 1));
     return `Page with workspace--read ${JSON.stringify({ path, offsetChars: 0, limitChars: readLimit })}; `
       + 'use nextOffsetChars as offsetChars until null, keeping limitChars.'
       + (canRaise ? ' Or raise the inline cap via agent_settings update tool_result_inline_max_chars.' : '');