Repository navigation
fix(deferral): a deferred write whose landing can't be read back isn't stored twice - #607
Conversation
… stored twice When a drained deferred write threw, writeDrainedInOrder read the agent's store back from where it stood before the write, for copies carrying the entry's deferredWriteId, so a write that landed before the throw counted as written. But storedCopiesOf returned [] when that read threw, and a count that couldn't be read before the write left nothing to read from, so the drain took either as "nothing landed": it cut the target and requeued the entry, and a later flush stored it a second time. An unknown landing is now kept apart from an empty one. storedCopiesOf returns undefined when the store can't be read back, and so does the catch when the count before the write couldn't be read. Such an entry is marked (DeferredWrite.landing = 'unknown') before it's requeued, so the queue persists the mark. Before a marked entry is written again, the store is searched for its id with queryMessages (landedCopiesById), which needs no position: - found, it counts as written, as when the readback works: its copies joined (asCompiled), logged, and passed to onLanded; - not found, the mark goes and it's written; - when the search throws, the target is cut and the entry requeued, still marked, holding what's behind it, as a write that keeps failing does. handOffDeferredWrites sets its agent's boot scan position to 0 for a marked entry, since an earlier copy could sit anywhere before the store's present count; boot's check scans from there. A write found later is carried at the flush that finds it, after what the turn stored since, while the store holds it earlier. The success path, where the write returned and its sharded body can't be read back, still carries the body as given. logDeferredWriteError names the three new cases. The drain's doc now says what an unknown landing does. test/deferred-window-only.test.ts: at a tool boundary, a write that throws after storing, whose readback then throws, is stored once and found by its id; and four tests of the drain itself: a count that couldn't be read before a throwing write, an unknown entry the store doesn't hold, an unknown entry whose search throws, and a hand-off of an unknown entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, read or changed From our internal review of 1ff5b35, and a colleague's runs: - releaseWindow stored a released agent's queue with cm.addMessage, outside the drain, so a marked entry was stored twice there. It's now looked up by its id first. Found, it's neither stored again nor passed to the primary, since release's first pass has passed its stored copy if the agent hadn't read it. Where the store can't be searched, it's stored anyway, with a line saying it may now be stored twice, since no later flush will try again. - A marked entry whose agent isn't registered was held at every flush, since the lookup took a missing agent as a failed search. The lookup now runs only with an agent, so addMessage drops the entry as it does an unmarked one. - A module's handle to a marked held write reached only the queued entry: an edit was lost when the flush found the stored copy, a removal left the copy, and a query showed the write twice. The handle now reaches the stored copy too. An edit and a removal apply to both, looked up first, so a lookup that throws throws with nothing changed; a query shows the write once, as its held row. The coalescer's read and shown checks read the copy's own row, a failed search counting as read and as shown, and its removal takes the copy only where that row may never have been shown. - The hand-off no longer records 0 as boot's scan position for a marked entry: the persisted mark already has the first flush after a restart look it up, so scanning the whole store at boot bought nothing. - deferredMessages is typed DeferredWrite[], which took the inline type's docs, so the mark is in the queue's one shape. - The line for an unknown landing says the store couldn't be read, which covers a count that failed before the write. The boundary's comment and the test file's header name the found-later exception to the window matching what the stream carried. Tests: the end-to-end test checks the window's order, and the count test checks the id the write counts under and its onLanded. New tests cover a found write carried as stored, another target's entry behind a held one, an unregistered agent, a mark kept across a restart, release (a found entry, a found failed reply, a blind search), a module's edit, removal and query, both under a blind lookup, and the coalescer's removal by the copy's shown check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… changed Our internal review of 1741955 found the two copies that round kept side by side, a marked held entry and its stored copy, disagreeing wherever a path reached only one: an edit or removal by the copy's own id never reached the queued entry, so a removed write was written again; the wake check read any queued write as unread; a sharded copy was seen by its first shard only; an edit or removal that committed and then threw left the two apart; the tune-out backlog showed the write twice; and the handle and the copy's id read as different rows. So a marked entry is now settled wherever a path that reads or changes held writes meets it (settleUnknownLandings): looked up by its id in its target's store. Found, the write landed: it leaves the queue, logged by its id, and is a stored write from then on, so the existing paths for a landed write apply to it, and it isn't carried into a turn. Not found, the mark goes, and it's an ordinary held write. Where the store can't be searched, it stays marked, and each path reads that doubt by its own rule: an edit or removal throws with nothing changed (refuseUnsettled), the coalescer reads it as read and as shown and doesn't remove it, and the wake check reads it as unread. A change the settling makes to the queue is persisted, as an edit's is. The paths that settle: heldWritesFor, so a module's reads, edits and removals, by a handle or a stored id; the coalescer's isUnreadPlacement and removePlacement; deferredWriteReadOrGone; and the tune-out backlog's held writes. Release settles in its own loop, in the released agent's store, and now logs a store made without its lookup only once the store succeeded. The copy-side code of the round before goes: the edit, removal and query of both copies, and the coalescer's checks on the copy. The docs that said a queued write is unstored, or unshown, name the doubt: the module API's held rows, editMessage and removeMessage, the coalescer host's isUnshown, removeOccupant's walk and isUnreadPlacement. Tests: the handle tests now check a write that landed is met as its stored row, by its handle and by its stored id, for an edit, a removal, a query and getMessage; a held write the store doesn't hold is edited as an ordinary one; settling inside a read persists the queue; the wake check reads a copy the agent read as read; the coalescer reads and removes a settled write as its stored placement, and a doubt as read and shown without removing it; the tune-out backlog leaves a write that landed to the stored rows; the restart test runs with and without a landed copy; and release leaves a failed reply the agent had read unpassed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… by a copy's row Our internal review of bb1f0e3, and a colleague's read: - Settling dropped a found write from the queue and persisted that, so a crash before the store synced could lose a write whose copy wasn't durable yet. A found write now moves to the un-acked set, as a drained write does (settleFound), so the durable queue keeps it until the next ack has synced the store. - A found write now takes the notes a store takes (noteStored), which its first write missed when it threw, as a flush that finds such a write takes them. So an addressed one is weighed for an open turn's re-pin and for the other residents, wherever it's found. - An edit, removal or getMessage by a stored id now settles the held write that row is a copy of (settleByCopy), read from the row's deferredWriteId, so it needs no search: a removal by the copy's id no longer leaves the queued entry to be written again, even where the store can't be searched. - The turn's draft notice settles the agent's held writes before it reads what they name. - Release's catch says, for an entry whose store couldn't be searched, that it may be stored once, twice or not at all; its comment no longer claims settleUnknownLandings runs there. - refuseUnsettled names the held write by its handle, and says the edit or removal is refused the same way while the store can't be searched and that the host log names the store's error. - The coalescer host's isUnshown contract reads "in the deferred queue and not already stored". Tests: settling a found write keeps it un-acked and durable, takes its notes and logs it by its id; one not found is persisted unmarked; an edit and a removal by a stored id settle by the row while the store can't be searched; getMessage settles by a handle and by a stored id, answering the stored row; a targeted settle leaves another marked write alone; the draft notice settles; the coalescer's removal settles on its own; release settles in an ephemeral run's own store, and says what it can't know when it can neither search nor store; and the tune-out backlog's test checks the stored rows too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n; pin two more cases From our internal review of 5216b34, text and two tests: - The coalescer host's isUnshown contract says a queued placement is unshown only when it's known not to be stored already, which a doubt isn't. - releaseWindow's doc names the line for a store that fails as well as the search: it may be stored once, twice or not at all. - The settle test checks the found write keeps its mark in the un-acked set, so a restart before the ack looks it up again. - A new test has a stored row settle only the held write whose id it carries, beside another marked write that stays marked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…meet A module context's editMessage and removeMessage can now throw for a held write whose landing is unknown, while its store can't be searched. The fixed fragment names that with the error's words, so a module author whose code edits or removes held writes knows it can throw there. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Medium impact] The PR appears safe to merge. SummaryThe PR keeps a deferred write marked when its write throws and the store cannot confirm whether it landed.
Acknowledgments from
Reviews (1) · Last reviewed commit: "docs(changelog): name the refusal a modu..." · Reviewed by Greptile |
Anarchid
left a comment
There was a problem hiding this comment.
Nightly dual review · round R1 at f80c0e1 · reviewers: Codex gpt-5.6-sol (xhigh) and Claude claude-opus-5-5, merged and every claim re-verified by the Claude leg · posted by the maintainer's campaign runner. Verdict: COMMENT (the formal review event is left to the maintainer). Finding ids R1-n are stable; cite them in fix commits.
Reviewed at f80c0e1 against main f8c23e1 (merge base d8cc690)
Verdict: COMMENT. The fix is sound and closes the double write it names. One major follow-up, three minors and a nit, none of them a regression against d8cc6909.
The core change holds up under tracing. An unreadable landing is now kept apart from an empty one, the mark persists with the queue and survives a restart, and every reader and mutator of held writes we could find settles the mark first. What it leaves is about where a write found later sits. That write stays where it first landed, ahead of replies whose requests never held it. That is the state this codebase's own boundary comment says a signed-thinking provider refuses (context-manager #155). The description prices it as one prompt-cache miss. In most cases a one-line lookup in the catch would avoid it. The other findings are a read asymmetry in releaseWindow, the cost of the new by-id lookup, and how the new refusal travels through a process response.
This PR was merged at ea4297ab before this review, so everything below is for a follow-up. We had no earlier review on this PR. The premise we were given holds: the positional readback this PR fixes came in with #286 (eadf9a90), and #461 (4ccaf3a3) documented this exact gap ("if it had landed, a later flush stores it again").
Major
R1-1. A write found later stays stored ahead of replies minted without it. Per context-manager #155 that is a refusal, not a cache miss, and it is avoidable in the common case.
- Where it lands. The write that throws has already stored its message right after the round's tool_result (
src/framework.ts:21300). The catch can't read it back (21313), so it marks it and requeues it. Onlywrittenentries are injected (11476-11492), so this boundary carries nothing, and the stream's next round answers without it. Later, the turn-end flush, a later boundary, or a reader's settle finds it by id (21290-21294,21843-21844), and it counts as written where it already sits. - The PR's own e2e test pins this.
test/deferred-window-only.test.ts:599-623asserts that the window endsuser: <tool_result>,Antra: look left,assistant: Done.,Antra: look right.Done.was minted by a request that held neither interjection. - The rule it breaks is stated in the same files. The test header (
test/deferred-window-only.test.ts:16-18) and the boundary comment (src/framework.ts:11440-11445) both say: "A provider that binds signed thinking to its prefix (context-manager #155) refuses a reply whose stored prefix holds a message its request didn't." context-manager #155 (open) reports that refusal as a 400 on every later turn, on accounts where the binding is enforced. The new exception text sits next to that rule (11423-11427, test header line 18). The description says "that costs one prompt-cache miss", and the changelog says only "later requests place it earlier". Neither names the refusal. - Not a regression. On
d8cc6909the same stray copy sits in the same place, with a duplicate after it. The PR removes the duplicate but not the misplacement, and it documents the misplacement as cheap. - Fix. The landing can usually be settled at the same boundary. When
storedCopiesOfreturnsundefined, orbeforeisundefined, search by id right there before marking, and mark only if that search also throws. We tried that one line locally,if (landed === undefined && agent) landed = this.landedCopiesById(agent, msg.id);after21313. The e2e test's window then becomesuser: <tool_result>,Antra: look left,Antra: look right,assistant: Done.. Both interjections are stored at that boundary and returned as written, so the boundary injects them, and the reply follows them as the stream carried it. The test's expectation would change to match. That leaves the case where both reads fail. There, either remove the stray copy where no reader can have been shown it (isUnreadStoredMessage(…, 'shown')) and store the entry at the current position, or at least name the #155 consequence in the boundary comment, the test header and the changelog. The first-tier fix also mostly retires the ordering exception the description lists under "What callers notice".
Minor
R1-2. releaseWindow's two reads can disagree. A found failed reply can then reach the primary zero times or twice.
releaseWindowfirst passes on unread failed replies by reading the window (src/framework.ts:21518).unreadFailedRepliesturns a read failure into[](21580-21597). Later, a marked queued entry is looked up by id (21533).- (a) The first read throws and the lookup finds the copy.
continueat21537skips the fallback at21561-21563, the queue is acked at21566, and the primary is never told. The description acknowledges this leg. - (b) The first read succeeds and passes the landed copy on.
passFailedReplyToPrimarymakes a fresh write that doesn't carry the entry's id (21675-21683). Then the lookup throws, so the entry is stored again andqueuedFailedReplyForPrimarypasses it a second time (21562-21565). The primary gets two notices. This leg isn't acknowledged.
- (a) The first read throws and the lookup finds the copy.
- Both legs need the two reads to disagree inside one synchronous release. In context-manager 0.14.1 both reads are
MessageStore.getAll(), so this takes a transient failure, and it's rare. - Fix. Have the first pass report whether it read the window, and which
deferredWriteIds it passed (the stored copy carries the id in its metadata). In the found branch, pass the copy on when the first pass failed and the copy is above the floor. In the unsearchable branch, skip the fallback for an id the first pass already passed. Add tests for both asymmetric cases. The existing release tests cover only two successful reads, or a lookup failure on an ordinary write.
R1-3. The by-id lookup reads the whole branch with blobs resolved, once per marked entry per call, and logs every time it fails.
landedCopiesById(src/framework.ts:21363-21368) callsqueryMessages. In context-manager 0.14.1, that isgetAll()and then a filter (src/message-store.ts:1111-1133).getAll()maps every row throughinternalToStoredwith blobs resolved (770-810,1815-1837), and that throwsBlob not foundfor any missing blob on the branch (src/blob-manager.ts:270-276). context-manager 0.17.1 has the same shape (message-store.ts:1180).- It runs from hot paths: each coalescer
isUnread/isUnshown, each module read, edit, removal and query, and each flush. Each failure logs a line (21367), and a flush adds another (21287). - Boot's own check for landed writes already scans blob-free windows (
landedDeferredWriteIds,6593-6626). - Fix. Use one blob-free windowed scan per store for all marked ids, resolve content only for the rows that match, and throttle the repeated line.
landedRow(21821) predates this PR and has the same shape.
R1-4. The new refusal is documented on ModuleContext, but it also reaches applyProcessResponse, where nothing catches it.
refuseUnsettled(src/framework.ts:21892-21899) is thrown fromeditMessageandremoveMessage(21950,21972). It is documented onModuleContext(src/types/module.ts:314-328).EventResponse.editMessagesandremoveMessages(src/types/module.ts:523-533) reach the same functions fromapplyProcessResponse(src/framework.ts:11849-11859). That function has no per-edit guard, anddispatchProcessEventToModuleshas no per-response guard (11805-11811).- Scenario: a module's
onProcessreturns an edit of a held handle it got fromaddMessage, while that write is a doubt. The refusal then aborts, for that event, the remaining modules' responses, this module's state update and inference request, the tool-call dispatch when the event is atool-call(11810-11811), theprocess:completedtrace and the process log. - Fix. Catch each refused edit or removal in
applyProcessResponseand log it. The module can't catch it on that path. Failing that, say onEventResponsethat an edit can be refused.
Nit
R1-5. settleByCopy logs every copy as one unsharded write ({ count: 1, sharded: false }, src/framework.ts:21885), even when the row a module named is one shard of a body. The "found by its id" line then says "it was stored". The row's bodyGroupId is right there.
Verified fine
- The defect is real on
d8cc6909. Withd8cc6909'ssrc/framework.tsunder this PR's tests, 29 of them fail. At head, both touched test files pass 72/72 (Node 22.21, context-manager 0.14.1, membrane 0.5.87, chronicle 0.4.0). - The drain's ordering is right. The mark is set before
requeueDrainedpersists the entry (21323-21326,21182-21190), and a marked entry is looked up before any rewrite (21282-21297). The success path is unchanged (21305-21308). - The mark survives a restart. Restore spreads each persisted entry (
7072,7112). The durable record is pending plus un-acked (6826-6828), so moving a settled-found entry into the un-acked set needs no re-persist, and the entry stays there until an ack has synced (21864-21868,6536-6555). - Every
drainDeferredForcaller goes throughwriteDrainedInOrder/readdDrained, exceptreleaseWindow, which has its own policy. - Every module-context entry point settles first (
addMessage,editMessage,removeMessage,getMessage,queryMessages,2852-2856), viaheldWritesFor/heldRowsorsettleByCopy. So do the coalescer's checks and removal, the wake check, the draft notice and the tune-out backlog. - Settling is synchronous. There are no awaits between a lookup and its decision, and loops that remove settled entries iterate a filtered snapshot.
- context-manager 0.14.1
queryMessagesdoes what the lookup needs. It searches the current branch, compares metadata by strict equality, returns every match in store order, and shards carry the body's metadata. - One copy is carried exactly as stored.
asCompiledreturns a single copy unchanged (788-793). - Nothing merged alongside touches deferral. Neither #564 nor the membrane-version comment updates touch the deferral code. Main CI at
ea4297abis green, and the PR's CI at head is green on ubuntu/macos × Node 20/24. - The description's notes on #582 and #598 hold. #582 adds a malformed-write drop to the same catch.
Method
Two reviewers, each finding re-verified before adoption.
- Codex
gpt-5.6-solatxhighreviewed the diff in a read-only sandbox: about 10 minutes, 50 commands, 6.63M input tokens (6.41M cached) and 28.2k output (17.9k reasoning). It could not install dependencies, run tests or reach the network (no CI, no issue lookups). - Claude fetched current main, the PR head and context-manager (its
mainand thev0.14.1tag the^0.14.1range resolves to). It installed dependencies in a scratch worktree, ran the two touched test files at head and againstd8cc6909'sframework.ts, and ran the one-line probe for R1-1. It also read context-manager's query,getAlland blob paths, and checked #155, #286, #461, #582 and CI withgh. - Who found what:
- R1-1, R1-4 and R1-5: Claude only.
- R1-2: Sol only. Adopted, but downgraded from major, because both legs need two
getAll()-backed reads to disagree inside one synchronous release, and one leg is acknowledged in the description. - R1-3: found independently by both.
- Not run: the full suite (5,641 tests per the description). We relied on CI for it.
…ove) main moved by 32 pull requests since this branch's last merge of it (8225e4b, in 4a0cd8b). Only src/framework.ts is changed on both sides, and git merges it without conflicts. Main's hunks come within 40 lines of this branch's in two places, each read in the merged file: - removePlacement: anima-research#607 settles a write whose landing is unknown before the placement resolves, and removes a deferred write from the queue only when its landing is known. This branch's hold check comes after both, before a stored message is touched, so under a hold a write still queued is withdrawn from the queue as before, and a stored one, including one the settle found landed, is declined. - readdDrained: anima-research#607 reworks writeDrainedInOrder beneath it. This branch's traceAs option traces what that returns as stored, after it. main's test/mcpl-coalescing.test.ts:1571 (3253dab, after 8225e4b) writes right after undoLastTurn(). On this branch that write is deferred while the store's reservation holds for the managers' re-initialization, so the test fails at this merge. R3-1 of the maintainer's round 3 adapts it, in the commit after this one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cked notices From an internal review of the merge with main. - stop()'s pass for a release the quiesce postponed: where an agent's pass throws, the drain now sends that agent's un-acked notices to readdDrained too, beside its pending ones, so they go on as boot would send them, read or not. Before, the pass alone passed un-acked entries, and the drain step's ack then dropped them, though the comment said a throwing pass's queue went on as boot would. They're taken, the agent's own alone, before the drain hands its pending entries to the un-acked set. - Comments: stop()'s doc, the run's end, drainDeferredFor, and the pass's docs now say a queued notice whose copy the agent read is kept back, a throwing pass's notices go on as boot would, and why the un-acked entries the pass reads have all landed. anima-research#616's and anima-research#607's pending changelog fragments say so too, the latter that the release's once holds as long as its sync went through, and that a crash or a failed sync leaves the entry to boot; anima-research#616's that a channel notice the agent could act on goes on with failed replies. Tests: - route-failure-targeting, for a failed reply and for a channel notice: - a throwing pass's pending and un-acked notices both reach the queue for the primary, and the primary's and another run's un-acked notices don't go on as the run's; - a release passes on only its own agent's un-acked entries, beside the primary's and another run's; - the stop() cases count what stop() left queued for the primary before the restart; - the un-acked case checks the release acks it, and is named for where its crash falls: between the release's sync and its queue's rewrite; - its two waits for the run's registration are bounded in time. - deferred-window-only's test that a released window doesn't pass on a failed reply the agent had read now releases a run, and checks the release took the entry from the queue. Released as the primary itself, the copied landing mark found the agent's own copy under the entry's id and kept the notice back without the read check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts own id A fork's or run's failed reply can still be in the deferred-write queue when the agent is released and also have a copy in the agent's window: its earlier write threw after storing and the store couldn't be read back to tell (`landing: 'unknown'`, #607), or a lookup has since found it and it waits for its ack. The release passed that copy to the primary as a fresh write under a new id, while it skipped or kept the entry, so the notice had two identities, and each path judged on its own whether the other would pass it: - a crash between the release's sync and its rewrite of the queue, or a crash before the ack of an entry a lookup had found, left the entry to boot, which passed it again: two notices; - stop() after a quiesce postponed the release passed the copy and left the entry queued as a gone agent's, which boot passed again: two; - a release whose window couldn't be read, while the lookup worked, passed neither: none; - a release whose store couldn't be searched, while the window could be read, stored the entry a second time and passed both: two, and one where the agent had read the copy. Now the entry is the notice's one identity while it's in the durable queue, pending or un-acked. The window's read passes no copy of such an entry, and notes whether the agent read it; the release passes the entry itself, under its own id and seq, unless the agent read its copy, and passes it where the window can't be read. Boot's check for landed writes knows that id, so a crash or a restart replays it once. Where the store can't be searched, a copy the window's read saw counts as found, so the entry isn't stored again. stop()'s pass for a postponed release shares this: an un-acked entry goes on under its own id there, since stop's ack would otherwise drop it, and pending ones stay queued for boot, which passes each once. Unchanged: boot can't read a gone agent's window, so where a stop() or a crash leaves such an entry to it, it passes the entry even where the agent had read the copy. flushDeferredWrites' comment now says so. Tests (route-failure-targeting): each path above reaches the primary once, or not at all where the run read the copy, beside not-landed controls, and stop()'s pass carries an un-acked entry. The window's read is now releasedWindow; the tests of its read-or-unread rule and stop-step-throws' row for a postponed release's pass call it by that name. #607's changelog fragment says what the release now does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers the review of #607 (R1-1, R1-3, R1-4, R1-5). - A drained write that throws after it's stored, whose readback can't say whether it landed, is looked up by its id at the boundary that wrote it. Found, it counts as written there, and a tool boundary injects it with what it stores. Before, it was marked and counted at a later flush where it first landed, ahead of replies made without it. - One that can be neither read back nor looked up there still counts where it sits when a later lookup finds it, and a tool boundary doesn't inject it: the agent reads it at its next compile. The comments, the test header, the changelog and the log line name that as a known limit: on an account where the provider enforces its binding of signed thinking to its prefix, a later request replaying a reply stored after it, whose request didn't carry it, is refused with a 400 (anima-research/context-manager#155). - The lookup by id keeps queryMessages and, where it throws, as one missing blob anywhere on the branch makes it do, scans the branch in blob-free windows (landedPositions), reading only the matching rows with their blobs. A window that comes back short answers that the store can't be searched, never that the write is absent. Lines saying a write can't be looked up are said once per write until a lookup of it succeeds. - A module's onProcess response that throws as it's applied stops there, as before, and no longer ends the event: a line names the module and where it threw, process:completed and the process log mark it with where it threw (ModuleProcessResponse.threw), and the other responses, a tool call's dispatch and the trace go on. EventResponse's docs, the README and two comments say a response applies in order up to a throw, not atomically. - A held write settled through one shard a module names says a shard was stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@Anarchid The findings of your review of this PR are answered in #726, merged as
The description of #726 gives the tests, the measurements and the limits that remain. Written by Ansel-2428 (Claude Opus 5.5). — Fiona-2757 |
A deferred context write whose write threw, and whose store then couldn't be read to tell whether it had landed, is no longer requeued as unstored and written a second time. It's marked as unknown, and it's looked up by its id in its store before it's written again, and before anything reads or changes it as a held write.
Problem
When a drained deferred write throws,
writeDrainedInOrderreads the agent's store back from where it stood before the write, for copies carrying the entry'sdeferredWriteId. A write that landed before the throw then counts as written and isn't stored again. ButstoredCopiesOfreturned[]when that read threw. A store that couldn't say how many messages it held before the write left no position to read from, and also gave[]. The drain took either as "nothing landed": it cut the target and requeued the entry, and a later flush stored it a second time.Reproduced on
maind8cc6909(see Tests). At a tool boundary, the strategy throws oncelook leftis stored, and the store's next read throws. The window then endsuser: <tool_result>,Antra: look left,assistant: Done.,Antra: look left,Antra: look right. Solook leftis stored twice, and the boundary carries neither message.Changes
storedCopiesOfreturnsundefinedwhen the store can't be read back, and the catch treats a count that couldn't be read before the write the same way. Such an entry is marked (DeferredWrite.landing = 'unknown') before it's requeued, so the persisted queue carries the mark, across a restart too.queryMessages(landedCopiesById), which needs no position:asCompiled), and it's logged and passed toonLanded;addMessagedrops it where its agent is gone.settleUnknownLandings), by the same lookup:settleFound). It moves from the pending queue to the un-acked set, as a drained write does, so the durable queue keeps it until the next ack has synced the store that holds it. It takes the notes a store takes, which its first write missed when it threw, as a flush that finds such a write takes them, and it's logged by its id. From then on it's a stored write, so the existing paths for a landed write apply to it;refuseUnsettled). The coalescer reads it as read and as shown, and doesn't remove it. The wake check reads it as unread.heldWritesFor, which serves a module context's reads, edits and removals; the coalescer'sisUnreadPlacementandremovePlacement; the wake checkdeferredWriteReadOrGone; the tune-out backlog's held writes; and the turn's draft notice. A mark that goes is persisted, as an edit is.getMessage,editMessageandremoveMessagealso settle the held write that row is a copy of, read from the row'sdeferredWriteId(settleByCopy). That needs no search, so a removal by the copy's id doesn't leave the queued entry to be written again, even where the store can't be searched.releaseWindow), stored outside the drain, settles a marked entry in the released agent's own store:DeferredWrite, which took the inline type's docs, so the mark is in the queue's one shape.logDeferredWriteErrornames the new cases: a write whose landing is unknown, one found later by its id, and one that still can't be looked up.editMessageandremoveMessage, the coalescer host'sisUnshown,removeOccupant's walk andisUnreadPlacement.What callers notice besides the fix:
What each part is for, and what was left out
undefinedfromstoredCopiesOf: the one place that knew the read failed now says so, rather than looking like an empty answer.queryMessagesfinds the entry's copies wherever they are on the current branch, aslandedRowalready finds a held write by its id.StoredCopyshape andstoredCopy: the readback and the search by id return the same rows, a sharded body's carrying itsbodyGroupId, so one projection serves both.readdDrained's rule), so an addressed one settled by a reader is weighed for an open turn's re-pin and for the other residents, as one a flush finds is. Release takes none, since the agent it releases has no turn left to weigh it.settleByCopy: a module that names a stored row by its id has the copy in hand, so the write it belongs to can be settled without a search, which is what keeps a removal by that id from being undone.refuseUnsettled: an edit or removal of a write that may already be stored would otherwise change only the queued entry and leave the stored copy as it was.DeferredWrite.landing, the module API's held rows,editMessageandremoveMessage, the coalescer host'sisUnshown,removeOccupant's walk,isUnreadPlacement): each said or implied that a queued write is unstored or unshown. A module author or a host implementer reading them would otherwise not know a held write can be a doubt, or that an edit of one can be refused.drainInternals,entry,heldAndStored,copiesOf), since the unknown states need a store that throws at a chosen moment:noteStoredstub that records them) and its line; one not found persisted unmarked; a stored row settling only the write whose id it carries; a module's edit and removal by handle and by stored id, also by stored id while the store can't be searched; a query, andgetMessageby each; a held write the store doesn't hold; an edit and removal refused while the store can't be searched; the wake check, leaving another marked write alone; the draft notice; the coalescer's checks and its removal on its own, with the store searchable and not; and the tune-out backlog on a tuned-out rig, its stored rows included.main. Telling that apart from a write that never landed would need a record of removals bydeferredWriteId.mainwrites any un-acked write whose row went before its ack; for a released agent's failed reply, the primary is then told a second time. Also by our reading, settling can run inside a flush, from a store listener: an ack there would drop that flush's unwritten entries from the record, and a sync would run while the store is in the middle of an append.Tests
maind8cc6909(Node 24.22.0): the end-to-end test againstmain's source fails withlook leftstored twice, the window as quoted above.413b429a, whose code and tests this head shares: the one commit after it changes only the changelog fragment. All ran on Node 24.22.0.test/deferred-window-only.test.tsand twelve other test files that write, hold, release or coalesce deferred entries pass: 680 tests.main'ssrc/framework.ts(d8cc6909) under these tests, 29 of the 30 new tests fail. The 30th, a marked write for an agent that isn't registered, passes there too, sincemaindrops such a write as it drops any.undefined, the lookup before writing, holding on a failed search, the found write's content andonLanded, and the missing-agent guard: the drain's own cases, and for the mark, the readback and the lookup, the end-to-end test too;settleByCopyat an edit, a removal andgetMessage, and settling only the write whose id the row carries: the stored-id tests;f80c0e15: Build & Test passes on ubuntu-latest and macos-latest, each with Node 20 and 24, beside the packed-release install check (https://github.com/anima-research/agent-framework/actions/runs/39351051363) and the changelog check (https://github.com/anima-research/agent-framework/actions/runs/39351051361).Not verified
Out of scope
src/framework.ts.releaseWindow's doc comment, as this change does, and the two conflict there.changelog.d/deferred-write-landing-unknown.fixed.md.Written by Nigel-2654 (Claude Opus 5.5). Milo-2653 (Claude Opus 5.5) found the defect in an internal review of another change, Ada-1017 (Claude Opus 5.5) made the design calls, and Gordon-2414 (Claude Opus 5.5) reviewed each round, through the code and tests at this head. Four rounds of our internal review read it, the last before the two commits that changed only comments, tests and the fragment.
Description checked and posted by Fiona-2757 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.