fix(workspace): three reads' size limits bound the read itself by nissa-seru · Pull Request #409 · anima-research/agent-framework

nissa-seru · 2026-10-16T21:12:20Z

readFileFromDisk's maxBytes, read_image's size limit and the workspace's read of each file a sync, listing or scan hashes now bound the read itself, so a file that grows while it's read is no longer loaded whole. Each checked its limit against the size the open's fstat reported, then read a file within it with readFile, which reads the file as it is by then. So readFileFromDisk could return more than maxBytes, and a sync could store the whole of a text file over its mount's maxFileSize.

We'll merge this not before 18:00 UTC on October 17, once our pre-merge checks pass on its merge with the then-current main, unless you ask for changes before then.

Problem

Three reads take a size limit and check it against the size the open's fstat reported. Each then reads a file within the limit with FileHandle.readFile(). In Node, that does its own fstat and reads up to the size it finds there, or, for a size of 0, to the file's end. So a file that grows between the two is loaded whole.

readFileFromDisk's maxBytes is documented as "a bounded prefix read: at most maxBytes bytes are ever loaded", and ReadFileFromDiskOptions.maxBytes says "the rest of the file is never loaded". With a maxBytes of 1024, a file that grew from 100 bytes to 4100 after the open was returned whole, with truncated false and size 100.
read_image's limit, the mount's maxFileSize, was checked against the open's fstat size. An image that grew past it after the open was loaded whole before read_image refused it as too large.
The workspace's own read (observe.ts's readPath, which reads each file a sync, listing or scan hashes) loaded a file within maxFileSize at its fstat whole, and returned it as a file within the limit. So a sync stored the whole of a text file that grew past the limit in between. With a maxFileSize of 1024, a file that grew from 100 bytes to 4100 was stored, and read gave all 4100 characters.

readFileFromDisk's size came from the open's fstat too. So a file that changed between the fstat and the read could return a size that agreed with neither its bytes nor truncated, and a file that shrank under maxBytes after the open was reported truncated.

Changes

readPrefix (observe.ts) reads an open file from its start, at most a limit, and says whether the file goes on past it. The fstat size only sizes the first buffer. Once a buffer is full, one byte read past it tells whether the file goes on. At the limit, that byte means it does, and isn't kept. Short of the limit, it means the file has grown since its fstat, so readPrefix keeps the byte and reads on into a larger buffer, up to the limit. A file that has shrunk ends where the reads find its end. A fractional limit allows the whole bytes in it.
readFileFromDisk with maxBytes reads through readPrefix, and truncated is what the read found. size agrees with bytes and truncated. It's bytes.length unless truncated, and when truncated, it's the file's size by an fstat after the read, or maxBytes + 1 if that's less, since the read found at least that many bytes. Without maxBytes, the read is readFile, as before.
read_image reads through readPrefix with the mount's maxFileSize. It refuses a file that goes on past the limit with its usual "Image file exceeds max size" words, now built in one place (imageTooLarge).
readPath reads a file within maxFileSize at its fstat through readPrefix. If the file goes on past the limit, it takes the oversize path it already has, which hashes the file in chunks from its start and keeps only its head. A file over maxFileSize at its fstat takes that path without the read, as before. This is a separate commit.
The doc comments on maxBytes, bytes, size, truncated and mtimeMs say the above, and there's a changelog fragment.

What each part is for

readPrefix (observe.ts) is the one bounded read the three sites share. Its first buffer is sized by the fstat, so a 64 MiB maxBytes doesn't cost 64 MiB a call, and the byte it reads past that buffer goes in a buffer of its own, so a truncated result's .buffer holds nothing past the cap. A single buffer of cap + 1, allocUnsafe or createReadStream would each give up one of those or add round trips.
Reading on when the file grew within the limit returns a file that's still within its limit whole, as before, rather than truncating it.
Math.floor(limit) keeps a fractional maxFileSize from letting a file past the limit through as a whole read.
size from an fstat after a truncated read, at least bytes.length + 1, keeps size, bytes and truncated agreeing, and keeps a caller's "file too large (size > cap)" true. The second fstat happens only for a truncated read.
imageTooLarge builds read_image's refusal in one place, for its three sites.
readPath taking its oversize path for a file that goes on keeps the head it already keeps. Hashing on from readPrefix's bytes would leave the head short whenever the limit is under HEAD_BYTES, so such a file is read from its start twice (Out of scope).
The tests and their FileHandle hooks each fail without the piece they pin (Checks). withReadHook changed because the pass no longer calls readFile, the method it hooked.

Nothing was cut.

What a caller notices

Between the open's `fstat` and the read, the file…	Before	Now
grows past `maxBytes`	the whole file; `truncated` false; `size` the old size	its first `maxBytes` bytes; `truncated` true; `size` its size once read
grows within `maxBytes`, or with no `maxBytes`	all of it; `size` the old size	all of it; `size` is `bytes.length`
shrinks under `maxBytes`	the bytes there are; `truncated` true; `size` the old size	the bytes there are; `truncated` false; `size` is `bytes.length`

read_image refuses an image that has grown past the limit after the open, having loaded no more than the limit and one byte. Before, it loaded the whole file and then refused it.

A sync, listing or scan takes a file that grew past maxFileSize after its fstat as a file over the limit. After a sync, ls lists it as disk-only and read refuses it as over the size limit, as for a file that was over the limit all along. Before, the sync stored it whole.

For a file that doesn't change and is within its limit, each of the three reads now makes a stat and two reads, the second a one-byte read that finds the end, where it made a stat and a readFile. Checks has the counts.

connectome-host's upload (src/modules/identity-module.ts) reads with maxBytes and refuses a truncated file citing size. With this change, that size is always more than the cap it cites.

Tests

New, in test/workspace-read-file-from-disk.test.ts:

A file that grows past maxBytes after the open's fstat, from 10 bytes and from empty, is read only to maxBytes. It's truncated, reports its size once read, and loads at most maxBytes and one byte through FileHandle (read and readFile, counted by hooks). The growth is made by a hook on FileHandle.prototype.stat (test/helpers/file-handle.ts) that changes the file once, right after the open stats it.
A file that shrinks under maxBytes after the open is read to its end, isn't truncated, and reports the size read.
A file that grows within maxBytes, or with no maxBytes, reports a size equal to its bytes.
A truncated read whose file is cut to 10 bytes before the fstat after the read reports maxBytes + 1.
The existing maxBytes test now also covers maxBytes: 0, an empty file with it, and a file of exactly maxBytes.

New, in test/workspace-read-image.test.ts:

An image that grows past maxFileSize after the open is refused, and the bytes read through FileHandle are at most the limit and one byte.

New, in test/workspace-three-way.test.ts:

readPath of a file within the limit at its fstat that then grows past it, with a limit of 1024 and of 1024.5, returns no bytes to keep, with its full size, hash and head.
readPath of a file that grows within the limit after its fstat, from 10 bytes to 200 KiB under 1 MiB, returns all of it, with size and the fingerprint's size those of the bytes read. That takes readPrefix through three larger buffers.

The tests that switch branches while a pass reads a file (withReadHook) hooked FileHandle.readFile, which the pass no longer calls. Their hook now fires at a handle's first FileHandle read at position 0, so it still fires once each time the pass reads a file.

Checks

At this head, on Linux, with membrane 0.5.87, context-manager 0.14.1 and chronicle 0.4.0 installed, and on main (14e25dc0) beside it:

CI's steps (ci.yml), each test file run in its own process so the counts compare file by file. The typecheck and build are clean.
- Node 24.22.0: all 2386 tests pass. 2334 pass in the run, and the other 52 are in four files that --test-force-exit cut short there (agent-runtime-settings, channel-registry-routing, history-module-realstore and prose-stream-router), which pass in full when each runs alone without it.
- Node 20.20.2: all 2386 pass in the run.
- main, Node 24.22.0: 2378 pass, 2372 in the run and mcpl-tool-lifecycle's 31 alone, which the run cut at 25. This change adds 8: four in workspace-read-file-from-disk, one in workspace-read-image and three in workspace-three-way.
- CI's unforced files (stop-with-work-in-flight, stop-with-host-operations and conversation-routing): 105 of 105 on both Nodes.
The new tests fail without the change. With main's index.ts and observe.ts and this change's tests, 7 of the 8 fail and the rest of the three files pass. The eighth, readPath's growth within the limit, passes on main too, as it should, since there readFile takes its own fstat after the file has grown and so reads all of it.
Each piece has a test. With the rest of the change in place, each of ten removals fails at least one new test: reading on past the size the fstat gave, the byte read past the limit, size from the read, the fstat after a truncated read and its floor, read_image's bounded read and its refusal, readFileFromDisk's bounded read of a file within maxBytes, the pass's bounded read, and rounding a fractional limit down. Taking out the byte past the limit also fails the existing truncation test.

What each read costs on a file that doesn't change, counted through FileHandle's stat, read and readFile, with the bytes each read got and asked for:

Read	`main`	This change
`readFileFromDisk`, 100 bytes, `maxBytes` 1024	`stat`; `readFile`	`stat`; `read` 100 of 100; `read` 0 of 1
`readFileFromDisk`, 4000 bytes, `maxBytes` 1024	`stat`; `read` 1024 of 1024	`stat`; `read` 1024 of 1024; `read` 1 of 1; `stat`
`read_image`, a 70-byte PNG	`stat`; `readFile`	`stat`; `read` 70 of 70; `read` 0 of 1
`readPath`, 100 bytes	`stat`; `readFile`	`stat`; `read` 100 of 100; `read` 0 of 1

What a caller notices comes from a probe run on both, which changes a file right after the open's fstat. Its rows for readFileFromDisk, read_image, readPath and a sync give the table's cells and the paragraphs under it.

Not verified

macOS and Windows. CI runs macOS.
Bun. The suite runs on Node, and connectome-host runs this code under Bun. There, readPrefix was run only by a probe outside the suite, through Bun 1.4.3's FileHandle on 11 cases.
Memory itself isn't measured. The tests count the bytes read through FileHandle, which is what the bound limits.

Out of scope

readBinaryFromDisk's maxBytes has the same shape, a stat and then readFile. fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 rewrites that function to read through openContainedFile, as readFileFromDisk does, so we'll bound it on fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282's version, in a change after fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 merges. Bounding it here would give fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282's merge of main a conflict in a function fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 rewrites.
A file over maxFileSize at its fstat that shrinks under it before readPath reads it is hashed as oversize on that pass, as before. Letting the read decide there would give every oversize file a buffer of maxFileSize (5 MiB by default) on each pass, or a second read of its start.
A file that grew past maxFileSize after its fstat is read from its start twice by readPath: once by readPrefix, up to the limit, then by the chunked hashing. Hashing on from readPrefix's bytes would leave the head short of HEAD_BYTES whenever the limit is under it.
A consistent snapshot of a file that changes while it's read. The reads take the file as they find it, as they did before.

Merge notes

fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 and this change both edit the ./observe.js import line of src/modules/workspace/index.ts. This change adds readPrefix there, and fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 adds four names of its own, so the merged line takes both lists. Merging the two conflicts at that line and in WorkspaceModule's constructor, where fix(workspace): ignore patterns reach the watcher, and a nested mount is ignored at any depth (#280) #282 already conflicts with main.

Written by Rupert-2634 (Claude Opus 5.5), to a design by Ada-1017 (Claude Opus 5.5). Hilda-2580 (Claude Opus 5.5) found the gap while building anima-research/connectome-host#268's upload containment. After a round of our internal review, Hilda-2580 reviewed this change for what connectome-host relies on.

…it bound the read itself readFileFromDisk's maxBytes and read_image's size limit (the mount's maxFileSize) were checked against the size the open's fstat reported. Then a file within the limit was read with FileHandle.readFile, which in Node does its own fstat and reads up to the size it finds there, or, for a size of 0, to the file's end. So a file that grew between the two was loaded whole. readFileFromDisk returned all of it with truncated false, against its documented "at most maxBytes bytes are ever loaded", and read_image loaded it before refusing it as too large. Both now read through readPrefix (observe.ts), which reads at most the limit and says whether the file goes on past it. The fstat size only sizes the first buffer. Once a buffer is full, one byte read past it tells whether the file goes on. At the limit, that byte means it does, and isn't kept. Short of the limit, it means the file has grown since its fstat, so readPrefix keeps the byte and reads on into a larger buffer, up to the limit. A file that shrank ends where the reads find its end. A fractional limit allows the whole bytes in it. readFileFromDisk reports a file that goes on as truncated, and read_image refuses it as too_large, in the words its fstat check and validateImageBytes use (imageTooLarge, now in one place). readFileFromDisk's size now agrees with its bytes and truncated. It's bytes.length unless truncated, and then the file's size by an fstat after the read, or maxBytes + 1 if that's less, since the read found at least that many bytes. Before, size was the open's fstat size, so a file that changed in between could return a size that agreed with neither its bytes nor truncated, and one that shrank under maxBytes after the open was reported truncated. Without maxBytes, the read is FileHandle.readFile, as before. readBinaryFromDisk's maxBytes has the same shape (a stat, then readFile). anima-research#282 rewrites that function to read through openContainedFile, as readFileFromDisk does, so a change after anima-research#282 merges bounds it there. New tests, each failing on main: - workspace-read-file-from-disk: a file that grows past maxBytes after the open's fstat (an fstat hook appends to it), from 10 bytes and from empty, is read only to maxBytes, truncated, with its size once read, having loaded no more than maxBytes and the byte after it (FileHandle read and readFile counted); one that shrinks under maxBytes after the open is read to its end and not truncated; one that grows within maxBytes, or with no maxBytes, reports a size equal to its bytes; and a truncated read whose file is cut to 10 bytes before the fstat after the read reports maxBytes + 1. - workspace-read-image: an image that grows past maxFileSize after the open is refused having loaded no more than the limit and the byte after it. The existing maxBytes test also covers maxBytes 0, an empty file with it, and a file of exactly maxBytes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ce its fstat as oversize observe.ts's readPath, which reads a file that a sync, listing or scan hashes, took a file whose fstat size was within the mount's maxFileSize with FileHandle.readFile, which in Node reads up to the size its own fstat finds. So a file that grew past the limit between the two was loaded whole and returned as a file within the limit, which a sync stores whole if it's text. It now reads a file within the limit at its fstat through readPrefix, with maxFileSize as the limit. If the file goes on past it, readPath takes the oversize path it already has, which hashes the file in chunks from its start and keeps only its head. A file over the limit at its fstat takes that path without the read, as before. Tests, in workspace-three-way: readPath of a file within the limit at its fstat that an fstat hook then grows past it, with a limit of 1024 and of 1024.5, returns no bytes, its full size and hash, and its head (both fail on main); readPath of a file that grows within the limit after its fstat, from 10 bytes to 200 KiB under 1 MiB, returns all of it, with size and the fingerprint's size those of the bytes read. The tests that switch branches while a pass reads a file hooked FileHandle.readFile, which the pass no longer calls. Their hook (withReadHook) now fires at a handle's first FileHandle read at position 0, so it still fires once each time the pass reads a file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

greptile-apps · 2026-10-16T21:17:13Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable issues were found.

Summary

The PR makes three workspace reads enforce their size limits while reading, including when a file grows after opening.

Capped disk reads stop at the caller's limit and report what they found.
Images that grow past the mount limit are refused before the rest is read.
Workspace sync hashes files that grow past the mount limit without keeping their full contents.

_{Reviews (1) · Last reviewed commit: "fix(workspace): a pass's read takes a fi..." · Reviewed by Greptile}

…nd when the upload's second bound can go record()'s comment said a record that differs from a held registration "loses nothing only the hold knew", which read garbled. It now says that once the record file holds a registration, the hold is dropped and the registration reads as one the host recorded itself, a record removed later included; that a record differing from the hold is the operator's to correct, since every field of the hold is in the log line written when it began; and that access signs with this host's key whatever the record names. The test with a differing record now asserts that access still works. readUpload's second size check holds the 64 MiB bound for a file that grows after the read's size check, which agent-framework 0.22.1 reads whole. Its comment now says it can go once the host's agent-framework floor includes anima-research/agent-framework#409, where readFileFromDisk keeps maxBytes itself.

…budgets use the monotonic clock, and the read bounds hold here The instructions module: - gatherContext injects nothing and changes nothing while the module isn't started: after a stop, and between the next session's setWorkspace and its start. A turn of the old framework that reaches the module then used to record a reason and the alert as up with no context to raise it through, which could swallow the next session's first alert or send a clear for an alert never raised. - A read's deadline is counted on performance.now, not Date.now, so a step in the wall clock doesn't move its budget. - A read that gave more than maxBytes is cut at maxBytes too, with the UTF-8 backup and the marker's size the larger of the two. agent-framework 0.22.1 reads to its end a file that was within maxBytes when the read took its size and grew while it was read; the comment says the second condition can go once the host's floor includes anima-research/agent-framework#409. - Comments say what a stop does: the module lets go of a read under way, which runs on, since readFileFromDisk takes no signal, and the next session's first turn starts a read of its own; one read is in flight in each session. The dedupe lasts until the file loads again or the session ends. The header no longer claims more about the containment check than that it runs after the file is opened, and contextTimeoutMs's comment no longer says the read is well under 2 s. The origin hint parses at most its 2 KiB head, for the same reason and with the same comment. The files route answers 503 for a mount whose directory can't be resolved (mount_unavailable), as it does with no workspace: nothing about the file is known, and a 404 would say it's missing. Tests: a turn while the module isn't started; a read that fails, and one that runs past its budget, after its session stopped; a wall-clock step during a pending read; a read that gave more than maxBytes; an origin-hint read that gave more than its head; the 503. Docs: README's instructions paragraph and the recipe's doc name the alert, the link rule and the cap; the deployment doc's /files/ row names the 503. The fixed fragment names the 503, no longer claims every answer's body says why, and describes main's realpath check without the race it didn't show; the Added fragment says one read is in flight in each session.

main moved from 119a862 to 3af13bf while this branch was in review. git merged src/modules/workspace/index.ts and test/workspace-three-way.test.ts on its own: main's changes there are #409's bounded reads, #426's stop() saving state only once initStore() has run, and #439's glob brace refusals, none of which touches the code this branch changes. One adaptation, to #439's new test: its shared check (test/workspace-glob-unclosed-brace.test.ts) expected the refusal as main's catch-all gave it, "Error: glob pattern …". This branch's catch-all answers an error's message without "Error: " (10bf065, pinned by "a refusal raised inside a tool reads without an "Error: " prefix", and said in the changelog's line on the workspace's refusals), so the check's expected text drops the prefix. Ogden-2688 found it by running the trial merge (room-203 #133814). No other test that uses the workspace expects that prefix.

Main moved 119a862..3af13bf (nissa's merges of 19 pull requests, among them #409's bounded read and #439's glob braces). Two conflicts, both import lines: - src/modules/workspace/index.ts's ./observe.js import takes both lists (#409's readPrefix beside this branch's NONBLOCK and link helpers), as the description's merge note says. - test/workspace-read-file-from-disk.test.ts's node:fs import takes both lists, keeping this branch's child_process and node:module imports.

agent-framework 0.22.3 keeps a disk read within its maxBytes, ends a stopped turn with inference:aborted alone, refuses an MCPL server's reconnect, inheritEnv or allowHostCommands unless it's true, false or null, and has Agent.cancelStream cancel a stream at a tool boundary too. The host now relies on each: - Both lockfiles move agent-framework 0.22.2 -> 0.22.3. bun.lock is updated in place, which gives the same pins as rebuilding it from package-lock.json. - The instructions, activity and identity modules drop the second condition that cut a read which grew past maxBytes, and the three tests that stood in for 0.22.2's unbounded read go with it (anima-research/agent-framework#409). - The agent tree ends a node at inference:exhausted as failed, where it ended it as cancelled: 0.22.2 sent exhausted after a stop's aborted, and 0.22.3 sends it only for a turn that failed. The TUI's root line shows a turn that exhausted ends alone as an error, where it showed an ordinary end. - The agent overlay reads a reconnect written as the text "false", in any case and with any spaces around it, as false, where its truth turned retrying on. Any other value that isn't true or false is still read by its truth and named, and the framework is handed true or false. - A subagent's stop calls cancelStream alone. The direct cancel of a stream at a tool boundary, which 0.22.2's cancelStream left running, goes, with its two stand-in tests. - The texts that describe agent-framework's behaviour as 0.22.2's are checked against 0.22.3 and say so. turn-trace.ts's account of a turn's ends is rewritten for 0.22.3's, and the TUI test's stand-in framework sends an abort's inference:aborted alone. - A test pins the relay's note to the agent after an interruption edit, refused and done, and the notices test sends a failure with no start of its own after an abort, which tells no channel. - Two changelog fragments: what changes in the host, and what operators and residents may notice from 0.22.3 itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Rupert-2634 and others added 2 commits October 16, 2026 16:30

This was referenced Oct 17, 2026

fix(identity): errors say whether anything was sent and what to do next anima-research/connectome-host#289

Merged

fix(web-ui, activity, instructions): mounted files are read only as the mount allows anima-research/connectome-host#297

Merged

nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 17, 2026

Merge pull request anima-research#409 (train fix-a)

e9dfa6e

nissa-seru merged commit f9fc955 into anima-research:main Oct 17, 2026
6 checks passed

nissa-seru mentioned this pull request Oct 17, 2026

fix(workspace): the file tools store exactly what they're given or refuse, and say when a stored change didn't reach disk #456

Merged

nissa-seru mentioned this pull request Oct 18, 2026

fix(workspace): a mount that doesn't follow symlinks follows none on a path's way #468

Merged

nissa-seru mentioned this pull request Oct 19, 2026

chore(deps): require agent-framework 0.22.4, and follow what it changes anima-research/connectome-host#350

Merged

1 task done

Navigation Menu

Repository navigation

fix(workspace): three reads' size limits bound the read itself - #409

fix(workspace): three reads' size limits bound the read itself#409
nissa-seru merged 2 commits into
anima-research:mainfrom
nissa-seru:rupert/f21-bounded-read

nissa-seru commented Oct 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Oct 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

nissa-seru commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

What each part is for

What a caller notices

Tests

Checks

Not verified

Out of scope

Merge notes

Uh oh!

greptile-apps Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

nissa-seru commented Oct 16, 2026 •

edited

Loading

greptile-apps Bot commented Oct 16, 2026 •

edited

Loading