diff --git a/changelog.d/mid-turn-window-only-held.fixed.md b/changelog.d/mid-turn-window-only-held.fixed.md new file mode 100644 index 000000000..6f897a684 --- /dev/null +++ b/changelog.d/mid-turn-window-only-held.fixed.md @@ -0,0 +1,127 @@ +- A message deferred during a turn that can't join the live stream at a tool + boundary — one carrying tool blocks, or named as the agent itself — now + waits for the turn's end, after its last round, instead of being stored at + the boundary between rounds the stream never showed it. Messages queued + behind it wait with it, so the queue keeps its order; messages queued + before it are still heard at the boundary. The cost: a message addressed + to the agent but queued behind a held one isn't heard mid-turn, and waits + for the turn's end, and unless the queue is persisted (below), a held + message and what is queued behind it live only in memory until then, so a + crash during the turn loses them. Stored between rounds, a held message + sat before the turn's later replies in every later compile although their + requests never held it: a prompt-cache divergence, and on a provider that + binds signed thinking to the prefix it was minted under + (anima-research/context-manager#155, whose binding + anima-research/context-manager#160 walks), a reply whose thinking is + refused. +- A message injected into the live stream at a tool boundary now counts as + read for RFC-006 coalescing when the stream says it carries injected + messages (membrane's `ToolContext.supportsInjectedMessages`, which + membrane's main sets true on its native path): the consumed watermark moves + when messages are injected, not only at a compile. Before, an injected + message sat above the watermark of the turn's compile, so a coalesced + replacement for its subject could remove a message the agent had heard and + answered. Within a branch the watermark only advances: a compile that + started before such a mark doesn't roll it back when it finishes. A stream + that doesn't say, as on membrane 0.5.87, still gets injected messages, and + they stay unread, as before: 0.5.87's XML tool path doesn't carry them, and + warns that they'll come next turn instead. +- A stream that says it can't carry injected messages (as membrane's main says + of its XML prefill path) gets none: everything deferred for the agent during + the turn waits for the turn's end, after its last round, as a held message + does, instead of being stored between rounds the stream never showed it. +- A deferred write that a flush defers again does so whatever its content, + under its id and in its place in the order writes arrived. That happens + while its target's own turn, stream or tool cycle is alive, or the host is + quiesced. (A write whose store throws before storing it goes back into the + queue too; see below.) A `tool_result` used to land regardless, apart from + the `tool_use` it answers, which went back into the queue. +- A write that waits now waits on its own target's state: the target's + turn, stream and tool cycle, or a quiesced host. (A fresh `tool_result` + still doesn't wait.) Before, a write for the primary agent waited out every + agent's tool cycle, while one addressed to an agent waited only on that + agent's. So a turn's end could store a later addressed message ahead of an + earlier one for the primary, and a fork's, subagent's or the + subconscious's tools held the primary's writes, which protected nothing in + the primary's window. Each flush, at a turn's end, after surgery, after a + withdrawn turn, after a start that ends before its stream and after a + puppet call, now delivers the agent's deferred messages while another + agent's tool cycle is pending; only the agent's own state holds them. The + flush after surgery stores several agents' entries in the order they + arrived, where it went agent by agent: residents share one message slot, + so that reordered their shared transcript. +- `stop()` now stores what is still deferred for each registered agent + before the store closes, in the order it arrived; before, a memory-only + queue lost it. Once `stop()` begins, a new ephemeral run is refused, and + an ephemeral run whose turn is refused because `stop()` has closed + admission, or whose retry `stop()` cancels, ends at once rather than + waiting for a watchdog. Since no turn starts or registers a stream once + `stop()` begins (unless `start()` is called before it finishes), no reply + can then follow a stored entry in a stream, so `stop()` stores each one + even while a frame it doesn't wait for still holds its target's turn, such + as a turn started outside the event loop, or a puppet call still running + after `stop()`'s wait for operator operations timed out; that call's pair + then lands after them. A quiesced host keeps what is deferred in its + persisted queue for boot recovery. An entry for a target no longer + registered (a disposed fork) is dropped from a memory-only queue at exit, + as before. +- What is deferred for a conversation fork whose closure turn ends, or for + an ephemeral run that ends, is now stored in that agent's store before the + fork is disposed or the run deregistered. Before, the turn-end flush ran + after, found no agent to take it, and dropped it. A closing fork is now + disposed by the frame that ends its closure turn: one that hands the turn + to a successor (a guard or budget restart) leaves it. Before, that frame + disposed it, so the successor found no agent and the turn never finished. + A quiesced host defers what is queued for an ending run or a closing fork + again instead, and with the agent gone, the flush that meets it later + drops it. +- A deferred write whose store throws without storing it goes back into the + queue under its id, and so does every entry queued behind it for that + agent, so none of them lands ahead of an earlier one, and a queued + `tool_use` that can't be stored keeps the `tool_result` queued behind it + from being stored alone. (A stored `tool_use` whose `tool_result` then + can't be stored stays alone until the result lands at a later flush.) + Other agents' entries go on, in the order they arrived. The log line names + the entry at each flush, and one that keeps throwing holds what is + deferred behind it for that agent until it can be stored. The hold is the + queue's: a write that isn't deferred is stored at once and can land ahead + of a held entry. That's a fresh `tool_result`, a message assembled into + the agent's turn before its compile (RFC-006 coalescing), and any write + while the agent's turn, stream and tool cycle are idle and the host isn't + quiesced. So an entry that keeps throwing holds only what is deferred + while the agent's turn, stream or tool cycle is alive, or the host is + quiesced. Before, it stayed in the un-acked set and the next + acknowledgement dropped it from the persisted queue (at a turn start or a + flush it was lost even from memory). At a tool boundary, nothing from it + on is injected. While the framework runs it is kept; a memory-only queue + still loses it when the process exits. A persisted queue (one a quiesce + persisted, or boot recovery restored) keeps it across restarts: each boot + restores it and tries it again with the rest of the queue, at once or at + the resume after a quiesced boot, and logs it again, unless it's one boot + drops (below). While it's held the queue stays persisted, so every later + deferral is written to the recovery file, up to the file's size cap; past + it the file isn't rewritten, as the host log warns, and what's deferred + after that is kept in memory only. Only the host log names the held write. +- A deferred write that throws after its message was stored, such as one + whose store listener throws, counts as written: it isn't put back into the + queue and stored a second time, and at a tool boundary what was stored is + injected with the rest. For a body the strategy shards, that can be its + first shards alone, since the throw stops the store's append loop: the + rest of the body isn't stored, and the request carries what landed, joined + as the strategy joins a body's shards when it compiles. The log line says + it was stored, or how many shards were. +- Boot recovery drops a recovered deferred write whose participant isn't + text, whose content isn't an array, or that holds a block that isn't an + object, and logs it by id and participant, rather than holding it in the + queue. The store can't take a null block. It keeps a string or number + block as given, but the context manager's token estimate throws on such a + block, so once one is stored, every compile for that agent fails, and a + restart doesn't clear it. Before, an entry whose participant isn't text or + whose content isn't an array was dropped without a line, and one with a + block that isn't an object was kept: a null block could never be stored, + and a string or number block, once stored, made every compile for its + agent fail. Boot recovery also drops an entry whose `forAgent` is present + but empty or not text, keeps the first well-formed entry with an id and + drops and names a later one, names an entry with no id as it gives it one, + and numbers an entry whose `seq` isn't a positive whole number up to + `Number.MAX_SAFE_INTEGER` after the others, naming it unless it has none. diff --git a/src/agent.ts b/src/agent.ts index e6e0129c5..d6728715b 100644 --- a/src/agent.ts +++ b/src/agent.ts @@ -817,20 +817,42 @@ export class Agent { /** * RFC-006 consumed watermark: the newest stored message at the last compile - * of a model request, per branch. Everything above it has never been in a + * of a model request, or at the last tool boundary that injected messages + * into a live one, per branch. Everything above it has never been in a * request and may still be replaced or withdrawn in place by its sender. */ private consumedWatermark: { branch: string; sequence: number } | null = null; + /** Counts marks: a compile writes its watermark back only if no mark was + * made while it ran (a tool boundary's, or another compile's). */ + private watermarkMarks = 0; getConsumedWatermark(): { branch: string; sequence: number } | null { return this.consumedWatermark; } - /** Mark everything currently stored as consumed (boot, branch switch). */ + /** Mark everything currently stored as consumed (the agent's creation, at + * boot or later, and a tool boundary that injected messages into the live + * stream). */ markContextConsumed(): void { // Tolerates partial context-manager doubles (tests build Agents over // stubs): without a watermark nothing is ever treated as unread. this.consumedWatermark = this.pendingWatermark(); + this.watermarkMarks++; + } + + /** + * A finished compile's watermark, taken before it awaited. Within a branch + * the watermark only advances: if a mark was made while the compile ran, + * the compile's (older) one is applied only where it is later on that + * mark's branch, so a boundary's mark is never rolled back by a compile + * that started before it (a preview during a live turn). + */ + private applyCompileWatermark(watermark: { branch: string; sequence: number }, marksBefore: number): void { + const current = this.consumedWatermark; + if (this.watermarkMarks !== marksBefore && current + && (current.branch !== watermark.branch || current.sequence >= watermark.sequence)) return; + this.consumedWatermark = watermark; + this.watermarkMarks++; } /** The watermark a successful compile will establish (taken before, applied after). */ @@ -862,9 +884,10 @@ export class Agent { // RFC-006 §3.3: consumption is determined at assembly — a compile that // fails assembled nothing, so the watermark moves only on success. const watermark = this.pendingWatermark(); + const marksBefore = this.watermarkMarks; const compile = () => this.contextManager.compile(this.compileBudget(budget)); const result = budget ? await compile() : await this.compileAtLiveBudget(compile); - if (watermark) this.consumedWatermark = watermark; + if (watermark) this.applyCompileWatermark(watermark, marksBefore); return result; } @@ -903,9 +926,10 @@ export class Agent { // compression hold whose deadline has passed, which `compile` does for // any compile before it selects. const watermark = opts?.dryRun ? null : this.pendingWatermark(); + const marksBefore = this.watermarkMarks; const compile = () => this.contextManager.compile(this.compileBudget(budget), injections, opts as never); const result = budget || opts?.dryRun ? await compile() : await this.compileAtLiveBudget(compile); - if (watermark) this.consumedWatermark = watermark; + if (watermark) this.applyCompileWatermark(watermark, marksBefore); return result; } @@ -1069,7 +1093,12 @@ export class Agent { // If all tools done, transition to ready if (this._state.pending.size === 0) { - this._state = { status: 'ready', toolResults: this._state.completed, stream: this._state.stream }; + this._state = { + status: 'ready', + toolResults: this._state.completed, + stream: this._state.stream, + supportsInjectedMessages: this._state.supportsInjectedMessages, + }; } } @@ -1363,9 +1392,10 @@ export class Agent { /** * Transition to waiting_for_tools when stream yields tool calls. - * Called by framework's driveStream. + * Called by framework's driveStream, with what the stream declared about + * carrying injected messages with this round's results (see AgentState). */ - enterWaitingForTools(calls: ToolCall[], stream: YieldingStream): void { + enterWaitingForTools(calls: ToolCall[], stream: YieldingStream, supportsInjectedMessages?: boolean): void { const pending = new Map(); for (const call of calls) { pending.set(call.id, { @@ -1375,7 +1405,7 @@ export class Agent { startedAt: Date.now(), }); } - this._state = { status: 'waiting_for_tools', pending, completed: [], stream }; + this._state = { status: 'waiting_for_tools', pending, completed: [], stream, supportsInjectedMessages }; } /** diff --git a/src/framework.ts b/src/framework.ts index 94f1035ff..30420ed6a 100644 --- a/src/framework.ts +++ b/src/framework.ts @@ -489,6 +489,31 @@ interface DeferredWrite { } const bySeq = (a: { seq: number }, b: { seq: number }): number => a.seq - b.seq; +/** A drained entry as writeDrainedInOrder wrote it: the id its store gave it + * (empty when addMessage deferred it again; a sharded body's first shard) + * and what the store holds of it. For an entry whose write threw after + * storing it, found in the store and counted as written, the content is + * what landed, as later compiles render it, which for a sharded body can be + * its first shards alone. */ +interface DrainedWrite { + id: MessageId; + content: ContentBlock[]; +} + +/** What later compiles render from the store for the copies of one write + * that landed: one copy as it is, and two or more, the shards of a sharded + * body, as context-manager's autobiographical strategy (the one that + * shards on ingest) joins a body's consecutive shards: their non-text + * blocks, then one text block of their joined text. The join is that + * strategy's `mergeAdjacentBodyGroupRaw`, which context-manager doesn't + * export, so the rule is copied here, and a test compares the two. */ +function asCompiled(landed: ReadonlyArray<{ content: ContentBlock[] }>): ContentBlock[] { + if (landed.length === 1) return landed[0]!.content; + const blocks = landed.flatMap((m) => m.content); + const text = blocks.filter((b) => b.type === 'text').map((b) => (b as { text: string }).text).join(''); + return [...blocks.filter((b) => b.type !== 'text'), { type: 'text', text } as ContentBlock]; +} + /** Stamp a deferred write's durable id into the metadata it is stored with * (boot recovery dedups replays by it). Idempotent for an already-stamped * message. */ @@ -496,6 +521,30 @@ function withDeferredWriteId(metadata: MessageMetadata | undefined, id: string): return { ...(metadata ?? {}), deferredWriteId: id } as MessageMetadata; } +/** A deferred entry as a log line names it: its id, participant and block + * types. A block that isn't an object is named by its JavaScript type, and + * an object whose type isn't a string as untyped, so the line names whatever + * a queued entry holds. */ +function describeDeferredWrite(msg: { id: string; participant: string; content: ContentBlock[] }): string { + const kind = (b: unknown): string => { + if (b === null) return 'null'; + if (typeof b !== 'object') return typeof b; + const type = (b as { type?: unknown }).type; + return typeof type === 'string' ? type : 'untyped'; + }; + return `${msg.id} (${msg.participant}: ${[...new Set(msg.content.map(kind))].join('+') || 'empty'})`; +} + +/** Whether a deferred message can join a live stream at a tool boundary: + * no tool blocks (the membrane's tool cycle) and not named as the agent + * itself (it would render as an assistant turn, a prefill). */ +function isInjectableMidTurn(msg: { participant: string; content: ContentBlock[] }, agentName: string): boolean { + // Exact, as membrane's role test is: only a participant equal to the + // request's assistantParticipant (Agent's name) renders as the assistant. + return msg.participant !== agentName + && !msg.content.some((b) => b.type === 'tool_use' || b.type === 'tool_result'); +} + /** * A prose block's envelopes, as explicit mode routes them: every line whose * text, after its leading whitespace, is `>>` and then a non-space opens a @@ -2824,6 +2873,42 @@ export class AgentFramework { } } + // No stream is left: from the start of stop() none registers and no + // agent turn starts (providerAdmissionClosed, checked for every agent), + // and every stream registered before was cancelled and awaited above. + // (A start() made before this stop finishes reopens admission, so a turn + // it begins can be alive here and take one of these entries mid-stream; + // #315 asks how start() after stop() should behave.) + // What is still deferred (a message held for a turn's end whose flush + // found the target busy again, or one deferred behind its target's own + // state) would wait for a flush that may never come. Land each at its + // registered target now, while the store is open; a memory-only queue + // would otherwise lose it at exit. Every registered agent's entries go in + // queue order (residents share one message slot), each target cut at its + // own first entry that can't be stored (writeDrainedInOrder). They go + // past the deferral: a frame stop() doesn't wait for (a turn started + // outside the event loop) can still hold its target's turn token, but no + // reply can follow an entry in any stream. + // + // A puppet call in flight is one of the operator operations stop() waited + // for above, so its pair and the flush after it have landed by now. If + // that wait timed out, the call may still be running and holding its + // target's turn: its pair lands after these entries, on the store a + // timed-out wait leaves open. + // + // A quiesced host defers each entry again instead, into the persisted + // queue, for boot recovery. Past the deferral, a write that throws stays + // queued with what follows it for its target, and a memory-only queue + // still loses it at exit. An entry for a target no longer registered (a + // disposed fork) is left: dropped with a memory-only queue at exit, and + // logged and dropped from a persisted one by the boot or resume flush + // that meets it. + if (this.deferredMessages.length > 0) { + const entries = [...this.agents.keys()].flatMap((name) => this.drainDeferredFor(name)); + this.readdDrained(entries, 'stop', { bypassDeferral: !this.quiesced }); + this.ackDeferredWrites(); + } + // Final sync before closing try { this.store.sync(); @@ -4859,12 +4944,18 @@ export class AgentFramework { } /** - * Land every deferred write whose target has no turn alive, one at a time, - * acknowledging each durably as it lands (the persisted queue is rewritten - * after every message). A crash mid-flush therefore replays exactly the - * un-acked remainder at the next boot — no loss, no duplicates. A write - * the store REJECTS is logged and acked too: replaying a poison message - * forever would wedge every later boot on it. + * Land every deferred write whose target has no turn alive, as one batch + * acknowledged once: the chronicle is synced, then the persisted queue is + * rewritten without the batch (ackDeferredWrites). Until then the batch + * stays in the durable queue, so a crash mid-flush replays it at the next + * boot, which skips each entry that already landed, by its deferredWriteId + * — no loss, no duplicates. The batch is stored in queue order, whatever + * its targets (writeDrainedInOrder). A write the store rejects stays + * queued, logged, with what is queued behind it for its target, so the + * flush never stores that target's later entries ahead of it, and other + * targets' entries go on. (A write that isn't deferred can still land + * ahead of it, such as one for an idle agent once the host has resumed.) + * A boot that meets it again logs it and goes on. * * Per-target: an agent whose turn is still alive keeps ITS messages * deferred for its own turn boundary; everyone else's flush now. @@ -4887,22 +4978,7 @@ export class AgentFramework { flush.sort(bySeq); this.deferredMessages = keep; this.handOffDeferredWrites(flush); - let stored = 0; - for (const msg of flush) { - try { - this.addMessage(msg.participant, msg.content, msg.metadata, { - deferredWriteId: msg.id, - ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), - }); - stored++; - } catch (err) { - console.error( - `[host-mode] ${label}: failed to store a deferred context write ` + - `(participant=${msg.participant}, forAgent=${msg.forAgent ?? 'primary'}):`, - err, - ); - } - } + const stored = this.readdDrained(flush, `host-mode ${label}`).length; // Ack = sync the chronicle FIRST, then rewrite the durable queue without // the batch. Never the other way round. this.ackDeferredWrites(); @@ -5042,7 +5118,12 @@ export class AgentFramework { * window is operator-length, not turn-length. Lives in the * branch-independent recovery file so a rollback cannot orphan it. * Deferrals outside a quiesce window (turn-alive, mid-tool-cycle) are - * still memory-only: they flush within the turn, as before. + * mirrored only while such a queue is still persisted; otherwise they are + * memory-only: they land within the turn, at a tool boundary or at its + * end, and a crash before then loses them. That window runs to the turn's + * end for a message held there and what is queued behind it, and, on a + * stream that can't carry injected messages, for every message deferred + * during the turn. */ private persistDeferredWrites(): void { try { @@ -5105,18 +5186,93 @@ export class AgentFramework { if (typeof v === 'number' && Number.isFinite(v)) scanFrom.set(k, Math.max(0, Math.floor(v))); } } - const candidates = data - .filter((m): m is Omit & { id?: string; seq?: number } => - !!m && typeof m === 'object' && typeof (m as { participant?: unknown }).participant === 'string' - && Array.isArray((m as { content?: unknown }).content)) - .map((m, i) => ({ - ...m, - id: typeof m.id === 'string' ? m.id : randomUUID(), - // Files that predate `seq` are in write order already. - seq: typeof m.seq === 'number' ? m.seq : i + 1, - })) + // An entry the queue's readers can't handle has no place in it: one whose + // participant isn't text, whose content isn't an array, that holds a + // block that isn't an object, or whose `forAgent` is present but not a + // name, is dropped here with a line naming it, rather than held. Reading + // a null block's type throws, at the tool boundary and in the store. The + // store keeps a string or number block as given, but the context + // manager's token estimate keys a WeakMap on each block, which such a + // value can't be: once one is stored, every compile for that agent + // throws, and a restart doesn't clear it. What a block holds, such as a + // tool result's content, isn't checked here, as it isn't for a fresh + // write. The framework writes `forAgent` only as a non-empty name, so + // another value comes from damage or a hand edit, and the agent it + // meant can't be known: sent to the primary, as a falsy one would be, it + // could put another agent's message in the primary's window. + const wellFormed = (m: unknown): m is Omit & { id?: unknown; seq?: unknown } => { + if (!m || typeof m !== 'object') return false; + const { participant, content, forAgent } = m as { participant?: unknown; content?: unknown; forAgent?: unknown }; + return typeof participant === 'string' + && Array.isArray(content) && content.every((b) => !!b && typeof b === 'object') + && (forAgent === undefined || (typeof forAgent === 'string' && forAgent !== '')); + }; + const malformed = data.filter((m) => !wellFormed(m)); + for (const m of malformed) { + // What the entry holds there: a value that isn't text by its type. + const field = (key: string): string => { + const value = m && typeof m === 'object' ? (m as Record)[key] : undefined; + if (typeof value === 'string') return value; + if (value === undefined) return `no ${key}`; + return `${key} ${value === null ? 'null' : `of type ${typeof value}`}`; + }; + console.error( + `[host-mode] a recovered deferred context write is malformed and was dropped: ` + + `${field('id')} (${field('participant')})`, + ); + } + // One entry per id: the boot dedup and the ack each take an id as one + // write, so an id that comes again is dropped with a line, and the first + // entry with it is kept. An entry with no id is given a fresh one, with + // a line: a copy of it that already landed carries the id it lost, so + // the boot dedup can't find it, and it may be stored again. + const seen = new Set(); + const kept: Array & { id: string; seq?: unknown }> = []; + let duplicates = 0; + for (const m of data.filter(wellFormed)) { + const hasId = typeof m.id === 'string' && m.id !== ''; + const id = hasId ? (m.id as string) : randomUUID(); + if (!hasId) { + console.error(`[host-mode] a recovered deferred context write (${m.participant}) has no id; it is given ${id}`); + } + if (seen.has(id)) { + duplicates++; + console.error( + `[host-mode] a recovered deferred context write repeats id ${id} (${m.participant}) and was dropped; ` + + `the first entry with that id is kept`, + ); + continue; + } + seen.add(id); + kept.push({ ...m, id }); + } + // A seq is a positive whole number no larger than + // Number.MAX_SAFE_INTEGER, the entry's place in arrival order. + // An entry without one (a file that predates seq, which is in write + // order already) or with any other value (damage or a hand edit: + // JSON's 1e400 reads as Infinity) is numbered after every valid seq, in + // file order, so the counter stays finite and the rest keep their order. + const validSeq = (seq: unknown): seq is number => Number.isSafeInteger(seq) && (seq as number) > 0; + let lastSeq = 0; + for (const m of kept) if (validSeq(m.seq)) lastSeq = Math.max(lastSeq, m.seq); + let renumbered = 0; + const candidates: DeferredWrite[] = kept + .map((m) => { + if (validSeq(m.seq)) return { ...m, seq: m.seq }; + renumbered++; + if (m.seq !== undefined) { + // A value that isn't a number is named by its type: it could be + // any size. + const seq = typeof m.seq === 'number' ? String(m.seq) : m.seq === null ? 'null' : `of type ${typeof m.seq}`; + console.error( + `[host-mode] a recovered deferred context write (${m.id}) has seq ${seq}, not a positive ` + + `whole number up to Number.MAX_SAFE_INTEGER; it is queued after the others`, + ); + } + return { ...m, seq: ++lastSeq }; + }) .sort(bySeq); - this.deferredSeq = Math.max(this.deferredSeq, ...candidates.map((m) => m.seq)); + this.deferredSeq = Math.max(this.deferredSeq, lastSeq); // Exactly-once: a message that landed before its ack was written is // already in the store under its deferredWriteId — do not replay it. // The scan covers every slot the interrupted batch could occupy (from @@ -5133,8 +5289,9 @@ export class AgentFramework { this.deferredMessages.push(...restored); this.deferredWritesPersisted = true; // Rewrite the durable queue without the already-landed entries so a - // second interrupted boot does not re-scan them. - if (landed.size > 0) this.persistDeferredWrites(); + // second interrupted boot does not re-scan them, without the malformed + // and repeated ones, and with the seqs given here. + if (landed.size > 0 || malformed.length > 0 || duplicates > 0 || renumbered > 0) this.persistDeferredWrites(); return restored.length; } catch (err) { console.error('[host-mode] failed to restore persisted deferred context writes:', err); @@ -5909,6 +6066,15 @@ export class AgentFramework { } } + /** A turn refused because stop() has begun: an ephemeral run waiting on it + * ends now, rather than when its startup watchdog fires. */ + private endEphemeralRunRefused(agent: Agent): void { + const run = this.ephemeralRuns.get(agent.name); + if (run && this.agents.get(agent.name) === agent) { + run.settle.reject(new Error(`Ephemeral agent "${agent.name}" refused: the framework is stopping`)); + } + } + private createDeferred(): Deferred { let resolve!: (value: T) => void; let reject!: (reason?: unknown) => void; @@ -6032,6 +6198,11 @@ export class AgentFramework { `refusing new ephemeral run for ${agent.name} — resume() first`, ); } + // Once stop() has begun no turn starts (startAgentStream), so a run + // admitted now could only wait out its startup watchdog. + if (this.providerAdmissionClosed) { + throw new Error(`framework is stopping: refusing new ephemeral run for ${agent.name}`); + } // Only a fresh object returned by createEphemeralAgent may enter this path. // Never overwrite a resident/conversation owner or a concurrent run: their // cleanup is name-keyed and could cancel/deregister the legitimate owner. @@ -6127,6 +6298,18 @@ export class AgentFramework { agent.cancelStream(); if (this.ephemeralRuns.get(agent.name) === run) this.ephemeralRuns.delete(agent.name); if (this.agents.get(agent.name) === agent) { + // What is still queued for the run, such as a message held for its + // turn's end, goes to its store now, past its deferral: once the run + // is deregistered, a flush finds no agent to take it and drops it. + // The run's namespace stays in the store for investigation, so the + // messages stay with it. A quiesced host defers them again instead, + // as stop() does, and with the run gone, the flush that meets them + // later drops them. + const queued = this.drainDeferredFor(agent.name); + if (queued.length > 0) { + this.readdDrained(queued, `end of ephemeral run ${agent.name}`, { bypassDeferral: !this.quiesced }); + this.ackDeferredWrites(); + } // Exact-generation disposal owns these name-keyed entries. Remove them // before deregistration; the late driveStream finally will see the // generation mismatch and leave any future owner untouched. @@ -7874,19 +8057,13 @@ export class AgentFramework { } // Writers that deferred behind the reservation land now, on the branch // the operator chose (or the restored source) — the same end-of-turn - // flush driveStream/puppet perform, under the same guard. - if (this.deferredMessages.length > 0 && this.pendingAssistantBlocks.size === 0) { - for (const name of reserved.keys()) { - if (this.activeTurnTokens.has(name)) continue; - for (const msg of this.drainDeferredFor(name)) { - try { - this.addMessage(msg.participant, msg.content, msg.metadata, - msg.forAgent ? { forAgent: msg.forAgent } : undefined); - } catch (error) { - console.error(`[operator] post-surgery deferred flush failed for ${name}: ${error instanceof Error ? error.message : String(error)}`); - } - } - } + // flush driveStream/puppet perform, under the same guard — for every + // agent free to take them, in queue order (writeDrainedInOrder). + if (this.deferredMessages.length > 0) { + const free = [...reserved.keys()] + .filter((name) => !this.activeTurnTokens.has(name) && !this.pendingAssistantBlocks.has(name)); + this.readdDrained(free.flatMap((name) => this.drainDeferredFor(name)), 'post-surgery flush'); + this.ackDeferredWrites(); } }; } @@ -8869,36 +9046,51 @@ export class AgentFramework { // target's live stream (deaf agent + window/live divergence). Left // queued, they flush at the target's own next boundary — with // injection — or in driveStream's finally when its turn ends. + // + // Only what can be injected is drained here. Tool blocks would + // corrupt the tool-cycle structure the membrane enforces, and a + // message named as the agent itself would render as an ASSISTANT + // turn on the wire (an unintended prefill the model would + // continue). Stored here without being injected, such a message + // would sit in the window before the turn's later replies although + // their requests never held it: the divergence above, and a reply + // whose signed thinking the provider then refuses, since it binds + // thinking to the prefix it was minted under (context-manager + // #155). So the drain stops at the first such message, which waits + // with everything queued behind it — the queue keeps its order — + // for the turn-end flush, where no reply follows it in this stream. + // + // A stream that declares it can't carry injected messages + // (supportsInjectedMessages false: membrane's XML prefill path) + // gets none, so nothing is drained here. Everything queued for + // the agent waits for the turn-end flush, as a held message does, + // rather than sitting in the window between rounds the stream + // never showed it. const midTurnInjections: Array<{ participant: string; content: ContentBlock[]; metadata?: MessageMetadata }> = []; // The injected messages' ids, for a boundary that leaves them to the // turn's next start as arrivals. const injectedIds = new Map(); - { - const deferred = this.drainDeferredFor(agent.name); - if (deferred.length > 0) { - for (const msg of deferred) { - const id = agent.getContextManager().addMessage(msg.participant, msg.content, withDeferredWriteId(msg.metadata, msg.id)); - // Every other resident reads what this boundary stores; for its - // own agent, the boundary weighs what it injects below. - this.noteForOtherResidents(agent, msg.metadata, id); - // Injection guards: tool blocks would corrupt the tool-cycle - // structure the membrane enforces, and a message named as the - // agent itself would render as an ASSISTANT turn on the wire - // (an unintended prefill the model would continue). Such - // messages stay window-only — visible next turn. - const hasToolBlocks = msg.content.some( - (b) => b.type === 'tool_use' || b.type === 'tool_result' - ); - if (!hasToolBlocks && msg.participant !== agent.name) { - const injection = { - participant: msg.participant, - content: msg.content, - ...(msg.metadata ? { metadata: msg.metadata } : {}), - }; - midTurnInjections.push(injection); - injectedIds.set(injection, id); - } - } + if (currentState.supportsInjectedMessages !== false) { + const deferred = this.drainDeferredFor(agent.name, (msg) => !isInjectableMidTurn(msg, agent.name)); + // Stored in queue order, cut at the first that can't be stored: it + // and what follows wait for the turn's end, and only what was + // stored is injected (the round's results go on either way). A + // write that threw after storing counts as stored, and what landed + // of it is what's injected, so the request carries what the store + // holds (writeDrainedInOrder). + const stored = this.writeDrainedInOrder(deferred, `tool boundary for ${agent.name}`, (msg) => + agent.getContextManager().addMessage(msg.participant, msg.content, withDeferredWriteId(msg.metadata, msg.id))); + for (const [msg, { id, content }] of stored) { + // Every other resident reads what this boundary stores; for its + // own agent, the boundary weighs what it injects below. + this.noteForOtherResidents(agent, msg.metadata, id); + const injection = { + participant: msg.participant, + content, + ...(msg.metadata ? { metadata: msg.metadata } : {}), + }; + midTurnInjections.push(injection); + injectedIds.set(injection, id); } } @@ -9022,7 +9214,7 @@ export class AgentFramework { } if (shouldEndTurn) { - // endTurn: messages already stored above, cancel stream, reset to idle. + // endTurn: what this boundary stored stays; cancel stream, reset to idle. // The batch is never submitted, so nothing can refuse it: admit // it now instead of leaving it pending across the idle gap. agent.toolResultGuard.settleTurnEnded(); @@ -9094,6 +9286,29 @@ export class AgentFramework { agent.toolResultGuard.submissionResults(membraneResults), midTurnInjections.length > 0 ? { injectedMessages: midTurnInjections } : undefined, ); + // RFC-006 §3.3: consumption is determined at assembly, and this + // is assembly too when the stream carries the injected messages + // into its next request. Without the mark they would sit above + // the watermark of the turn's compile and read as unread, so a + // coalesced replacement could remove a message the agent heard + // and answered (and that its later replies were minted after). + // Only a stream that declares it carries them is marked + // (supportsInjectedMessages true: membrane's native path). One + // that doesn't say may be membrane 0.5.87's XML path, which + // drops them with a warning, and a message marked read there + // could no longer be replaced though the agent never saw it. So + // without the declaration they stay unread, as before this mark. + // The mark is the store's head, a scalar, as a compile's is: it + // also covers what was stored since the compile without being + // carried, such as another slot's messages, and a write that + // bypassed deferral into this slot (a fresh tool_result, a + // routed channel message). That over-cover can't be left out. + // Under context-manager#160 such a write already gives the + // replies stored after it stamps their requests didn't carry, + // whatever the mark says; deferring those writers is the fix. + if (midTurnInjections.length > 0 && currentState.supportsInjectedMessages === true) { + agent.markContextConsumed(); + } agent.setStreaming(currentState.stream); this.emitTrace({ type: 'inference:stream_resumed', @@ -11454,9 +11669,12 @@ export class AgentFramework { const ownsProviderGate = !this.ephemeralRuns.has(agent.name) && !this.conversationAgentHomes.has(agent.name); // A stopping framework starts no turn, whoever owns provider admission - // (conversation agents and ephemeral runs do not take the gate). + // (conversation agents and ephemeral runs do not take the gate). An + // ephemeral run waiting on the turn ends now, as at every refusal of + // stop()'s. if (this.providerAdmissionClosed) { if (ownsProviderGate && providerGateAlreadyHeld) this.releasePrimaryProviderGate(agent.name); + this.endEphemeralRunRefused(agent); return; } if (ownsProviderGate && !providerGateAlreadyHeld) { @@ -11589,24 +11807,13 @@ export class AgentFramework { // A start that ends before its stream exists lands what was deferred // during it, as a stream's teardown does. Left queued, those messages // would land only at the agent's next start, after any stored meanwhile. - // Per message, as the turn-start flush does: one store failure must - // neither drop the messages behind it nor replace the start's own error. + // In queue order and cut at the first that can't be stored, as every + // flush is (readdDrained), so a store failure neither lands the + // messages behind it out of order nor replaces the start's own error. + // Gated on this agent's own tool cycle, as the turn-end flush is. if (!tokenHandedOff && !this.activeTurnTokens.has(agent.name) - && this.deferredMessages.length > 0 && this.pendingAssistantBlocks.size === 0) { - for (const msg of this.drainDeferredFor(agent.name)) { - try { - this.addMessage(msg.participant, msg.content, msg.metadata, { - deferredWriteId: msg.id, - ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), - }); - } catch (err) { - console.error( - `[deferred-flush] failed-start flush failed to store a message for ${agent.name} ` + - `(participant=${msg.participant}, id=${msg.id}):`, - err, - ); - } - } + && this.deferredMessages.length > 0 && !this.pendingAssistantBlocks.has(agent.name)) { + this.readdDrained(this.drainDeferredFor(agent.name), 'failed-start flush'); this.ackDeferredWrites(); } } @@ -11669,25 +11876,17 @@ export class AgentFramework { this.deferredMessages.length > 0 ) { { - const deferred = this.drainDeferredFor(agent.name); - for (const msg of deferred) { - // Per-message try/catch (mirrors announceLocusIfChanged): one - // poison message — or a transient store-write failure — must not - // abort the turn or drop the messages behind it in the queue. - try { - const id = agent.getContextManager().addMessage( - msg.participant, msg.content, withDeferredWriteId(msg.metadata, msg.id), - ); - this.noteArrival(agent.name, msg.metadata, id); - this.noteForOtherResidents(agent, msg.metadata, id); - this.emitTrace({ type: 'message:added', messageId: id, source: 'deferred-flush:turn-start' }); - } catch (err) { - console.error( - `[deferred-flush] turn-start flush failed to store a message for ${agent.name} ` + - `(participant=${msg.participant}):`, - err, - ); - } + // One poison message, or a transient store-write failure, must not + // abort the turn or drop the messages behind it in the queue: the + // drain cuts there, and it and what follows stay queued, in order + // (writeDrainedInOrder). What was stored is weighed for the turn now + // starting and for every other resident's open turn. + const stored = this.writeDrainedInOrder(this.drainDeferredFor(agent.name), `turn-start flush for ${agent.name}`, (msg) => + agent.getContextManager().addMessage(msg.participant, msg.content, withDeferredWriteId(msg.metadata, msg.id))); + for (const [msg, { id }] of stored) { + this.emitTrace({ type: 'message:added', messageId: id, source: 'deferred-flush:turn-start' }); + this.noteArrival(agent.name, msg.metadata, id); + this.noteForOtherResidents(agent, msg.metadata, id); } this.ackDeferredWrites(); } @@ -11727,13 +11926,8 @@ export class AgentFramework { this.activeTurnTriggers.delete(agent.name); } if (!this.activeTurnTokens.has(agent.name) - && this.deferredMessages.length > 0 && this.pendingAssistantBlocks.size === 0) { - for (const msg of this.drainDeferredFor(agent.name)) { - this.addMessage(msg.participant, msg.content, msg.metadata, { - deferredWriteId: msg.id, - ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), - }); - } + && this.deferredMessages.length > 0 && !this.pendingAssistantBlocks.has(agent.name)) { + this.readdDrained(this.drainDeferredFor(agent.name), 'withdrawn-turn flush'); this.ackDeferredWrites(); } return false; @@ -11943,6 +12137,7 @@ export class AgentFramework { this.channelRegistry?.stopTyping(); this.eventGate?.onInferenceEnded(agent.name); this.abortDroppedTurn(agent.name, turnToken, reason, turnStartedAt); + if (this.providerAdmissionClosed) this.endEphemeralRunRefused(agent); return false; } @@ -11970,9 +12165,13 @@ export class AgentFramework { // A stop followed by start must not revive this earlier turn. // Membrane's yielding stream starts its request when it is first // iterated, so cancelling it here, before driveStream, sends nothing. - if (this.agents.get(agent.name) !== agent || this.providerAdmissionClosed || lifecycle !== this.lifecycleGeneration) { + // A turn that was mid-compile when stop() cancelled every stream would + // otherwise register after that pass and outlive stop(). The caller's + // finally releases the turn token. + const stopping = this.providerAdmissionClosed; + if (this.agents.get(agent.name) !== agent || stopping || lifecycle !== this.lifecycleGeneration) { const why = this.agents.get(agent.name) !== agent ? 'the agent was replaced or removed' - : this.providerAdmissionClosed ? 'the framework was stopping' + : stopping ? 'the framework was stopping' : 'the framework was stopped and started again'; const reason = `a turn compiled while ${why} is dropped before it sends anything`; console.error(`[turn] ${agent.name}: ${reason}`); @@ -11989,6 +12188,7 @@ export class AgentFramework { this.channelRegistry?.stopTyping(); this.eventGate?.onInferenceEnded(agent.name); this.abortDroppedTurn(agent.name, turnToken, reason, turnStartedAt); + if (stopping) this.endEphemeralRunRefused(agent); return false; } // A conversation agent takes no provider gate, so its own compression @@ -12065,11 +12265,13 @@ export class AgentFramework { if (this.providerAdmissionClosed) { // stop() ended the wait: that cancels the retry, it does not hasten // it. Nothing is written (the store is closing); this frame still - // owns its token, since no retry frame replaced it. + // owns its token, since no retry frame replaced it. An ephemeral + // run waiting on the turn ends now, as at every refusal of stop()'s. if (this.activeTurnTokens.get(agent.name) === turnToken) { this.activeTurnTokens.delete(agent.name); this.activeTurnTriggers.delete(agent.name); } + this.endEphemeralRunRefused(agent); return false; } // The retry re-enters startAgentStream, which replaces this frame's @@ -12507,7 +12709,15 @@ export class AgentFramework { // Truncated responses yield 'complete' with stop_reason 'max_tokens' directly. // So all calls here have valid input — {} is legitimate for parameterless tools. - agent.enterWaitingForTools(event.calls, stream); + // Whether this stream can carry messages injected with the round's + // results: true on membrane's native path, false on its XML + // prefill path, absent before membrane declared it. The tool + // boundary reads it from the agent's state. + agent.enterWaitingForTools( + event.calls, + stream, + (event.context as { supportsInjectedMessages?: boolean }).supportsInjectedMessages, + ); for (const call of event.calls) { // RFC-007: register at the one dispatch point for model-issued @@ -13293,7 +13503,9 @@ export class AgentFramework { await this.waitForRetry(action.delayMs); // stop() ending the wait cancels the retry (this frame's finally // clears the token); it is never permission to retry at once. + // An ephemeral run waiting on the turn ends now. if (!this.providerAdmissionClosed) await this.startAgentStream(agent, trigger, attempt + 1); + else this.endEphemeralRunRefused(agent); } else { this.settleAgent(agent.name, { stopReason: 'exhausted', @@ -13700,12 +13912,6 @@ export class AgentFramework { this.pendingAssistantBlocks.delete(agent.name); } - // A conversation fork whose TTL closure turn just finished is done for - // good — dispose it so the agent map doesn't grow monotonically. - if (ownsPhysicalStream && this.closingConversationAgents.has(agent.name)) { - this.disposeConversationAgent(agent.name); - } - // The turn is torn down — release the turn-alive marker BEFORE the // deferred flush below, so the flush appends at the settled tail. // Token-matched: if a successor turn already owns the agent (endTurn @@ -13721,17 +13927,26 @@ export class AgentFramework { // Flush any deferred messages (e.g. if stream failed while tools were // pending). Only THIS agent's messages: other targets' entries wait // for their own boundaries (re-adding via addMessage re-defers if the - // target has meanwhile started a turn). - if (frameReachedTerminal && this.deferredMessages.length > 0 && this.pendingAssistantBlocks.size === 0) { - const deferred = this.drainDeferredFor(agent.name); - for (const msg of deferred) { - this.addMessage(msg.participant, msg.content, msg.metadata, { - deferredWriteId: msg.id, - ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), - }); - } + // target has meanwhile started a turn). Gated on this agent's own tool + // cycle, not every agent's: messages held for this turn's end depend on + // this flush, and another agent's pending tools say nothing about this + // window. addMessage judges each entry by the same rule, so the drained + // prefix is stored in order, or goes back into the queue whole under + // its ids when a successor turn has already taken the agent. + if (frameReachedTerminal && this.deferredMessages.length > 0 && !this.pendingAssistantBlocks.has(agent.name)) { + this.readdDrained(this.drainDeferredFor(agent.name), 'turn-end flush'); this.ackDeferredWrites(); } + + // A conversation fork whose TTL closure turn just finished is done for + // good — dispose it so the agent map doesn't grow monotonically. The + // frame that ends the turn disposes it, after the flush above, so what + // was held for the turn's end lands in the fork's store instead of + // being dropped as addressed to no agent. A frame that hands the turn + // to a successor (a guard or budget restart) leaves the fork to it. + if (frameReachedTerminal && ownsPhysicalStream && this.closingConversationAgents.has(agent.name)) { + this.disposeConversationAgent(agent.name); + } } } @@ -13896,6 +14111,9 @@ export class AgentFramework { { id: randomUUID(), seq: ++this.deferredSeq, participant: agentName, content: toolUse, forAgent: agentName }, { id: randomUUID(), seq: ++this.deferredSeq, participant: 'user', content: blocks, forAgent: agentName }, ); + // As addMessage's own deferral does: a crash before the next persist + // must not lose the pair from a persisted queue. + if (this.quiesced || this.deferredWritesPersisted) this.persistDeferredWrites(); } this.emitTrace({ @@ -13916,19 +14134,14 @@ export class AgentFramework { if (this.activeTurnTokens.get(agentName) === turnToken) this.activeTurnTokens.delete(agentName); // Messages deferred while we held the token land now, after the pair // — the same end-of-turn flush driveStream performs, under the same - // tool-cycle guard. (No wake is requested: the agent sees the pair, - // and anything that arrived meanwhile, on its next turn.) Keyed on - // "no turn alive", not on token ownership: if a turn did replace us - // and has already ended, its flush ran before our pair was queued — - // nobody else will flush it, so we do. + // guard: this agent's own tool cycle. (No wake is requested: the agent + // sees the pair, and anything that arrived meanwhile, on its next + // turn.) Keyed on "no turn alive", not on token ownership: if a turn + // did replace us and has already ended, its flush ran before our pair + // was queued — nobody else will flush it, so we do. if (!this.activeTurnTokens.has(agentName) - && this.deferredMessages.length > 0 && this.pendingAssistantBlocks.size === 0) { - for (const msg of this.drainDeferredFor(agentName)) { - this.addMessage(msg.participant, msg.content, msg.metadata, { - deferredWriteId: msg.id, - ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), - }); - } + && this.deferredMessages.length > 0 && !this.pendingAssistantBlocks.has(agentName)) { + this.readdDrained(this.drainDeferredFor(agentName), 'puppet flush'); this.ackDeferredWrites(); } } @@ -15414,13 +15627,14 @@ export class AgentFramework { */ forAgent?: string; /** - * Set when the message is being flushed FROM the durable deferred - * queue: it is the entry's one durable identity. A write stamps it - * into `metadata.deferredWriteId` (boot dedup); a re-deferral moves - * the same logical entry back to pending under the same id instead of - * minting a second, independently replayable one. + * Set when the message comes back from the deferred queue: the entry's + * one durable identity and its place in arrival order. A write stamps + * the id into `metadata.deferredWriteId` (boot dedup). A re-deferral + * moves the same logical entry back to pending under the same id and + * seq, instead of minting a second, independently replayable one, or + * moving it behind entries that arrived after it. */ - deferredWriteId?: string; + fromQueue?: { id: string; seq: number }; /** Out-param: where the message landed (RFC-006 needs to find it again). */ placement?: { agent?: string; messageId?: MessageId; deferredId?: string }; /** @@ -15431,12 +15645,7 @@ export class AgentFramework { bypassDeferralFor?: string; } ): MessageId { - // Route to the named agent, else the primary (not ephemeral subagents). - const agent = opts?.forAgent - ? this.agents.get(opts.forAgent) - : this.primaryAgentName - ? this.agents.get(this.primaryAgentName) - : this.agents.values().next().value; + const agent = this.deliveryAgent(opts?.forAgent); if (!agent) { if (opts?.forAgent) { console.error( @@ -15468,32 +15677,40 @@ export class AgentFramework { // against a FROZEN context — module events and api message.send must not // append mid-surgery. Deferred here, flushed by resume(). tool_result // still lands (a draining turn's continuation depends on it). - const hasToolResult = content.some(b => b.type === 'tool_result'); - // Explicitly-targeted deliveries scope the tool-cycle check to the - // target (pendingAssistantBlocks is keyed by agent); the default path - // keeps the historical GLOBAL check byte-for-byte (conservative: any - // agent's pending cycle defers primary-bound messages). - const midToolCycle = opts?.forAgent - ? this.pendingAssistantBlocks.has(agent.name) - : this.pendingAssistantBlocks.size > 0; + // + // A write coming back from the deferred queue (fromQueue) is + // re-deferred whatever its content: its tool_use went back into the + // queue just before it, in the same drain, and a tool_result stored + // without it would split the pair. (A turn-end flush can run after a + // successor turn has taken the agent: the 2026-07-31 interleave.) + const freshToolResult = opts?.fromQueue === undefined && content.some(b => b.type === 'tool_result'); + // One rule for every write, primary-bound or targeted: it waits while ITS + // TARGET's own tool cycle is pending (pendingAssistantBlocks is keyed by + // agent). Another agent's cycle says nothing about this window. The + // default path used to check every agent's cycle (efc7e7af, from when + // all agents shared one message store), which held the primary's writes + // behind a fork's, subagent's or the subconscious's tools, and judged a + // drained queue by two rules, so a later targeted entry could land ahead + // of an earlier primary-bound one. Residents that share the un-namespaced + // slot with overlapping turns put each other's writes inside the other's + // live turn whatever this check does: the limit #286 states. + const midToolCycle = this.pendingAssistantBlocks.has(agent.name); if ( - !hasToolResult && + !freshToolResult && opts?.bypassDeferralFor !== agent.name && (this.quiesced || midToolCycle || this.activeTurnTokens.has(agent.name) || this.activeStreams.has(agent.name)) ) { - // A re-deferral of a drained entry keeps its id and leaves the un-acked - // set: one logical message, one durable identity — never two entries. - const id = opts?.deferredWriteId ?? randomUUID(); - let seq: number | undefined; - if (opts?.deferredWriteId) { - const prior = this.unackedDeferredWrites.find((m) => m.id === id); - seq = prior?.seq; + // A re-deferral of a drained entry keeps its id and seq, and leaves the + // un-acked set: one logical message, one durable identity, one place + // in arrival order, whether or not the queue is persisted. + const id = opts?.fromQueue?.id ?? randomUUID(); + if (opts?.fromQueue) { this.unackedDeferredWrites = this.unackedDeferredWrites.filter((m) => m.id !== id); } - this.deferredMessages.push({ id, seq: seq ?? ++this.deferredSeq, participant, content, metadata, forAgent: opts?.forAgent }); + this.deferredMessages.push({ id, seq: opts?.fromQueue?.seq ?? ++this.deferredSeq, participant, content, metadata, forAgent: opts?.forAgent }); if (this.quiesced || this.deferredWritesPersisted) this.persistDeferredWrites(); if (opts?.placement) { opts.placement.agent = agent.name; opts.placement.deferredId = id; } return '' as MessageId; // Deferred — flushed at the target's next boundary @@ -15502,32 +15719,43 @@ export class AgentFramework { const stored = agent.getContextManager().addMessage( participant, content, - opts?.deferredWriteId ? withDeferredWriteId(metadata, opts.deferredWriteId) : metadata, + opts?.fromQueue ? withDeferredWriteId(metadata, opts.fromQueue.id) : metadata, ); if (opts?.placement) { opts.placement.agent = agent.name; opts.placement.messageId = stored; } - // Stored while none of the agent's turns is alive, or assembled for the - // start now beginning: no tool boundary will weigh it. - if (!this.activeTurnTokens.has(agent.name) || opts?.bypassDeferralFor === agent.name) { + this.noteStored(agent, metadata, stored, opts?.bypassDeferralFor); + return stored; + } + + /** Where addMessage delivers: the named agent, else the primary, else the + * first registered agent. Undefined when there is no such agent. */ + private deliveryAgent(forAgent?: string): Agent | undefined { + return forAgent + ? this.agents.get(forAgent) + : this.primaryAgentName + ? this.agents.get(this.primaryAgentName) + : this.agents.values().next().value; + } + + /** The notes addMessage takes for a message it stored: an arrival for the + * agent when none of its turns is alive or the message was assembled for + * the start now beginning (no tool boundary will weigh it), and one for + * every other resident's open turn. */ + private noteStored(agent: Agent, metadata: MessageMetadata | undefined, stored: MessageId, bypassDeferralFor?: string): void { + if (!this.activeTurnTokens.has(agent.name) || bypassDeferralFor === agent.name) { this.noteArrival(agent.name, metadata, stored); } this.noteForOtherResidents(agent, metadata, stored); - return stored; } /** * Splice out the deferred messages addressed to `agentName` (explicitly * via forAgent, or implicitly when it is the primary), leaving other - * targets' messages queued for their own boundaries. + * targets' messages queued for their own boundaries. With `stopAt`, the + * drain ends at the first of them (in queue order) that it matches: that + * message and every later one stay queued, so what is drained is always a + * prefix of the agent's queue. */ - private drainDeferredFor(agentName: string): Array<{ - /** Durable identity — every write of this entry must carry it - * (`withDeferredWriteId` / `opts.deferredWriteId`). */ - id: string; - participant: string; - content: ContentBlock[]; - metadata?: MessageMetadata; - forAgent?: string; - }> { + private drainDeferredFor(agentName: string, stopAt?: (msg: DeferredWrite) => boolean): DeferredWrite[] { if (this.deferredMessages.length === 0) return []; const mine: typeof this.deferredMessages = []; const rest: typeof this.deferredMessages = []; @@ -15535,14 +15763,187 @@ export class AgentFramework { const target = msg.forAgent ?? this.primaryAgentName; (target === agentName ? mine : rest).push(msg); } - this.deferredMessages = rest; mine.sort(bySeq); + const stop = stopAt ? mine.findIndex(stopAt) : -1; + const drained = stop === -1 ? mine : mine.slice(0, stop); + this.deferredMessages = stop === -1 ? rest : [...rest, ...mine.slice(stop)]; // The drained messages are about to be written by the caller; they stay // in the DURABLE queue (as un-acked) until the caller's // ackDeferredWrites() has synced the chronicle. Removing them here would // acknowledge before the write is durable. - this.handOffDeferredWrites(mine); - return mine; + this.handOffDeferredWrites(drained); + return drained; + } + + /** + * Put drained entries back into the queue under their ids and seq: out of + * the un-acked set, and persisted when the queue is. A memory-only queue + * keeps them while the framework runs. + */ + private requeueDrained(entries: DeferredWrite[]): void { + if (entries.length === 0) return; + const ids = new Set(entries.map((m) => m.id)); + this.unackedDeferredWrites = this.unackedDeferredWrites.filter((m) => !ids.has(m.id)); + for (const msg of entries) { + if (!this.deferredMessages.some((m) => m.id === msg.id)) this.deferredMessages.push(msg); + } + if (this.quiesced || this.deferredWritesPersisted) this.persistDeferredWrites(); + } + + /** + * Write drained entries in queue (seq) order, whatever their targets: + * residents share one message slot, so the order entries arrived in is the + * order they're stored. Each target is cut at its own first entry that + * can't be stored: that entry and every later one for the same target go + * back into the queue under their ids and seq (requeueDrained) before + * anything is logged, and other targets' entries go on. Left in the + * un-acked set, the next ack would drop them from the durable queue. + * + * So none of a target's entries lands ahead of an earlier one, and a queued + * tool_use that can't be stored keeps the tool_result queued behind it from + * being stored alone. (A tool_use that is stored, followed by a tool_result + * that can't be, stays stored without it until the tool_result lands at a + * later flush. The only queued pair is puppetToolCall's defensive one, + * which no path reaches today.) An entry that keeps failing is logged by + * id at each flush, and holds its target's later queued writes rather than + * letting them land out of order. + * + * The hold is the queue's alone. A write that isn't deferred is stored at + * once and can land ahead of a held entry: a fresh tool_result, whatever + * its agent's state, and any write while its agent's turn, stream and tool + * cycle are all idle and the host isn't quiesced. So an entry that keeps + * failing holds only what is deferred behind it. + * + * A write can throw after its message is stored: a store listener, the + * strategy's handling of the add, or a later shard of a sharded body can + * throw once a record is written. Before an entry that threw is put back, + * its target's store is read from where it stood before the write, for + * messages carrying the entry's `deferredWriteId`. What it finds counts as + * written, at most once: the entry isn't requeued and stored a second + * time, and what landed, as later compiles render it (asCompiled), is what + * it returns as the entry's content. For a sharded body, that can be its + * first shards alone, since the append loop stops at the throw; the rest + * of the body isn't stored. (Requeued, the + * landed shards would be stored twice, and only the rest could not be + * appended later without splitting the body.) Boot's scan for landed + * writes counts a landed shard the same way. + * + * Returns what was written (stored, or deferred again by addMessage, whose + * id is then empty), in queue order: for one target, a prefix of its + * entries. `onLanded` hears an entry whose write threw after storing it, + * with the id it landed under, as soon as it's found: before the writes + * after it, so whatever it records keeps their order. + */ + private writeDrainedInOrder( + entries: DeferredWrite[], + where: string, + write: (msg: DeferredWrite) => MessageId, + onLanded?: (msg: DeferredWrite, id: MessageId) => void, + ): Map { + const ordered = [...entries].sort(bySeq); + // An entry's target as addMessage resolves it, keyed by name for the cut. + const targetOf = (msg: DeferredWrite): string | null => msg.forAgent ?? this.deliveryAgent()?.name ?? null; + const cut = new Set(); + const written = new Map(); + for (let i = 0; i < ordered.length; i++) { + const msg = ordered[i]!; + const target = targetOf(msg); + if (cut.has(target)) continue; + const agent = this.deliveryAgent(msg.forAgent); + const before = agent ? this.storedCountOf(agent) : undefined; + try { + written.set(msg, { id: write(msg), content: msg.content }); + } catch (error) { + const landed = agent && before !== undefined ? this.storedCopiesOf(agent, before, msg.id) : []; + if (landed.length > 0) { + written.set(msg, { id: landed[0]!.id, content: asCompiled(landed) }); + this.logDeferredWriteError(msg, where, error, { count: landed.length, sharded: landed[0]!.bodyGroupId !== undefined }); + onLanded?.(msg, landed[0]!.id); + continue; + } + cut.add(target); + this.requeueDrained(ordered.slice(i).filter((m) => targetOf(m) === target)); + this.logDeferredWriteError(msg, where, error); + } + } + return written; + } + + /** How many messages the agent's store holds: undefined when its store + * can't say. */ + private storedCountOf(agent: Agent): number | undefined { + try { + return agent.getContextManager().getMessageCount(); + } catch { + return undefined; + } + } + + /** The messages carrying `deferredWriteId` among those the agent's store + * gained from position `from` on, in store order, with their blobs + * resolved, as a request carries them. A sharded body's carry its + * `bodyGroupId`. */ + private storedCopiesOf(agent: Agent, from: number, deferredWriteId: string): Array<{ id: MessageId; content: ContentBlock[]; bodyGroupId?: string }> { + try { + const cm = agent.getContextManager(); + return cm.getMessageWindow(from, Math.max(0, cm.getMessageCount() - from)).messages + .filter((m) => m.metadata?.deferredWriteId === deferredWriteId) + .map((m) => ({ id: m.id, content: m.content, ...(m.bodyGroupId !== undefined ? { bodyGroupId: m.bodyGroupId } : {}) })); + } catch (error) { + console.error(`[deferral] could not read ${agent.name}'s store for a deferred write that threw:`, error); + return []; + } + } + + /** The line naming a drained write that threw: one that couldn't be stored, + * which is already back in the queue, or, with `landed`, one stored before + * the throw (for a sharded body, how many of its shards). Logging mustn't + * fail the flush that met it, so a thrown value that has no text is named + * as such. */ + private logDeferredWriteError( + msg: DeferredWrite, + where: string, + error: unknown, + landed?: { count: number; sharded: boolean }, + ): void { + let reason: string; + try { + reason = error instanceof Error ? error.message : String(error); + } catch { + reason = 'a thrown value that has no text'; + } + console.error( + `[deferral] ${where}: deferred write ${describeDeferredWrite(msg)} for ` + + `${msg.forAgent ?? this.primaryAgentName ?? 'the primary'} ` + + (landed + ? `threw after ${landed.sharded + ? `${landed.count} shard${landed.count === 1 ? '' : 's'} of its body ${landed.count === 1 ? 'was' : 'were'} stored` + : 'it was stored'}; what was stored counts as written and isn't stored again: ` + : `could not be stored; it stays queued, holding what is queued behind it for that agent: `) + + reason, + ); + } + + /** Re-add drained entries through addMessage, which stores each or defers + * it again, in queue order and cut per target (writeDrainedInOrder). With + * `bypassDeferral`, each is stored past its target's deferral (stop()). A + * write that threw after storing missed the notes addMessage takes after a + * store, so they're taken for what landed as soon as it's found, before + * the writes after it: an open turn's arrivals keep the order the writes + * landed in, and its newest stays the newest (remainingArrival). */ + private readdDrained(entries: DeferredWrite[], where: string, opts?: { bypassDeferral?: boolean }): DeferredWrite[] { + const bypassFor = (msg: DeferredWrite): string | undefined => + opts?.bypassDeferral ? this.deliveryAgent(msg.forAgent)?.name : undefined; + const written = this.writeDrainedInOrder(entries, where, (msg) => + this.addMessage(msg.participant, msg.content, msg.metadata, { + fromQueue: { id: msg.id, seq: msg.seq }, + ...(msg.forAgent ? { forAgent: msg.forAgent } : {}), + ...(bypassFor(msg) ? { bypassDeferralFor: bypassFor(msg) } : {}), + }), (msg, id) => { + const agent = this.deliveryAgent(msg.forAgent); + if (agent) this.noteStored(agent, msg.metadata, id, bypassFor(msg)); + }); + return [...written.keys()]; } private editMessage(id: MessageId, content: ContentBlock[]): void { diff --git a/src/types/agent.ts b/src/types/agent.ts index 708c7002f..3e71079a2 100644 --- a/src/types/agent.ts +++ b/src/types/agent.ts @@ -307,13 +307,18 @@ export interface InferenceOptions { /** * Internal state of an agent. + * + * `supportsInjectedMessages` is what the stream declared with the round's + * tool calls (membrane's `ToolContext.supportsInjectedMessages`): whether it + * can carry messages injected with that round's results. It's absent when + * the stream doesn't say, as on membrane 0.5.87 and earlier. */ export type AgentState = | { status: 'idle' } | { status: 'inferring'; promise: Promise; abortController: AbortController } | { status: 'streaming'; stream: YieldingStream } - | { status: 'waiting_for_tools'; pending: Map; completed: CompletedToolCall[]; stream?: YieldingStream } - | { status: 'ready'; toolResults: CompletedToolCall[]; stream?: YieldingStream }; + | { status: 'waiting_for_tools'; pending: Map; completed: CompletedToolCall[]; stream?: YieldingStream; supportsInjectedMessages?: boolean } + | { status: 'ready'; toolResults: CompletedToolCall[]; stream?: YieldingStream; supportsInjectedMessages?: boolean }; /** * A tool call that's in progress. diff --git a/test/conversation-routing.test.ts b/test/conversation-routing.test.ts index c6c6c6b0a..50db20df3 100644 --- a/test/conversation-routing.test.ts +++ b/test/conversation-routing.test.ts @@ -10,7 +10,7 @@ import { tmpdir } from 'node:os'; import { mkdtempSync, rmSync } from 'node:fs'; import { join } from 'node:path'; import { AgentFramework } from '../src/index.js'; -import type { ProcessEvent, ConversationRouterConfig, TraceEvent } from '../src/index.js'; +import type { ProcessEvent, ConversationRouterConfig, TraceEvent, Module, ModuleContext, ToolCall, ToolResult } from '../src/index.js'; import { Membrane, OpenAIResponsesAPIAdapter, OpenAIResponsesFormatter } from '@animalabs/membrane'; import type { ContentBlock } from '@animalabs/membrane'; import { MockMembrane, createMockResponse } from './helpers/mock-membrane.js'; @@ -629,6 +629,108 @@ describe('Conversation routing', () => { await framework.stop(); }); + it('a write deferred for a fork during its closure turn is stored in the fork before it is disposed', async () => { + const IDLE_TTL_MS = 60_000; + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'hello!' }])); + const framework = await makeFramework({ idleTtlMs: IDLE_TTL_MS }); + framework.pushEvent(incomingEvent({ channelId: 'slack:D4', text: 'hi', channelType: 'im' })); + await framework.runUntilIdle(); + const g1 = framework.getAgent('conversation-slack-D4-g1')!; + assert.ok(g1, 'g1 fork should exist after spawn'); + + forceExpiry(framework, 'slack:D4', IDLE_TTL_MS); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Final report posted.' }])); + // While the closure turn is alive, a write addressed to the fork is + // deferred to that turn's end. + const internals = framework as unknown as { + addMessage(participant: string, content: ContentBlock[], metadata?: undefined, opts?: { forAgent?: string }): string; + }; + const streamYielding = membrane.streamYielding.bind(membrane); + let deferred: string | undefined; + membrane.streamYielding = (request, options) => { + deferred = internals.addMessage('alice', [{ type: 'text', text: 'one more thing' }], undefined, + { forAgent: 'conversation-slack-D4-g1' }); + return streamYielding(request, options); + }; + framework.pushEvent({ type: 'external-message', source: 'test', content: 'tick', metadata: {} } as unknown as ProcessEvent); + await framework.runUntilIdle(); + + assert.equal(deferred, '', 'deferred: the closure turn was alive'); + assert.equal(framework.getAgent('conversation-slack-D4-g1'), null, 'disposed once its closure turn finished'); + const stored = g1.getContextManager().getAllMessages().map((m) => JSON.stringify(m.content)); + assert.ok(stored[stored.length - 2]!.includes('Final report posted.'), stored.join('\n')); + assert.ok(stored[stored.length - 1]!.includes('one more thing'), 'stored after the closure reply, before the fork was disposed'); + + await framework.stop(); + }); + + it('a closure turn that restarts is disposed by the frame that ends it, after storing what was held for it', async () => { + const IDLE_TTL_MS = 60_000; + // One tool, for the closure turn's tool round. + const tools: Module = { + name: 'test', + async start(_ctx: ModuleContext) {}, + async stop() {}, + getTools: () => [{ name: 'echo', description: 'Echo input', inputSchema: { type: 'object', properties: {} } }], + async handleToolCall(_call: ToolCall): Promise { return { success: true, data: 'echoed' }; }, + async onProcess() { return {}; }, + } as unknown as Module; + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'test.chronicle'), + membrane: membrane.asMembrane(), + // The mock's tool round reports 10 input tokens, over this budget, + // which the fork inherits from its template: the closure turn's tool + // round restarts its stream. + agents: [{ name: 'trunk', model: 'test-model', systemPrompt: 'You are the trunk.', maxStreamTokens: 5 }], + modules: [tools], + conversations: { templateAgent: 'trunk', idleTtlMs: IDLE_TTL_MS }, + }); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'hello!' }])); + framework.pushEvent(incomingEvent({ channelId: 'slack:D6', text: 'hi', channelType: 'im' })); + await framework.runUntilIdle(); + const g1 = framework.getAgent('conversation-slack-D6-g1')!; + assert.ok(g1, 'g1 fork should exist after spawn'); + + forceExpiry(framework, 'slack:D6', IDLE_TTL_MS); + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Checking.' }, + { type: 'tool_use', id: 'e1', name: 'test--echo', input: {} }, + ] as ContentBlock[], 'tool_use')); + const internals = framework as unknown as { + addMessage(participant: string, content: ContentBlock[], metadata?: undefined, opts?: { forAgent?: string }): string; + }; + const streamYielding = membrane.streamYielding.bind(membrane); + let streams = 0; + let deferred: string | undefined; + membrane.streamYielding = (request, options) => { + const first = streams++ === 0; + if (first) { + // While the closure turn is alive, a write addressed to the fork is + // deferred to that turn's end. + deferred = internals.addMessage('alice', [{ type: 'text', text: 'one more thing' }], undefined, + { forAgent: 'conversation-slack-D6-g1' }); + } + const stream = streamYielding(request, options); + // A mock stream takes every response queued when it starts, so the + // restart's stream gets its reply only once the first has started. + if (first) membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Final report posted.' }])); + return stream; + }; + framework.pushEvent({ type: 'external-message', source: 'test', content: 'tick', metadata: {} } as unknown as ProcessEvent); + await framework.runUntilIdle(); + + assert.equal(deferred, '', 'deferred: the closure turn was alive'); + assert.equal(streams, 2, 'the closure turn restarted once'); + assert.equal(framework.getAgent('conversation-slack-D6-g1'), null, 'disposed once its closure turn finished'); + const stored = g1.getContextManager().getAllMessages().map((m) => JSON.stringify(m.content)); + // The restart's turn start stores what was held for the turn, before its + // compile, so the restarted stream carries it and its reply follows. + assert.ok(stored[stored.length - 2]!.includes('one more thing'), stored.join('\n')); + assert.ok(stored[stored.length - 1]!.includes('Final report posted.'), 'the restarted closure turn finished in the fork'); + + await framework.stop(); + }); + it('restart does not reuse generation names or re-seed an existing namespace', async () => { membrane.pushResponse(createMockResponse([{ type: 'text', text: 'noted' }])); const fw1 = await makeFramework({}, 'restart.chronicle'); diff --git a/test/deferred-window-only.test.ts b/test/deferred-window-only.test.ts new file mode 100644 index 000000000..44145ca24 --- /dev/null +++ b/test/deferred-window-only.test.ts @@ -0,0 +1,1017 @@ +/** + * What the window stores mid-turn is what the live stream carried. + * + * A message deferred during a turn is stored at the agent's next tool + * boundary and injected into the resumed stream (hear-while-acting), so the + * next compile's prefix is the request the turn's later replies were minted + * under. Two kinds of message can't be injected: one carrying tool blocks + * (it would break the tool cycle membrane enforces) and one named as the + * agent itself (a prefill). Those wait for the turn's end, and so does + * everything queued behind them, so the queue's order holds. A provider that + * binds signed thinking to its prefix (context-manager #155) refuses a reply + * whose stored prefix holds a message its request didn't. + * + * A message the stream carried has been read: RFC-006's consumed watermark + * covers it, so a coalesced replacement can no longer remove it. Only a + * stream that declares it carries injected messages (membrane's native + * path) is known to have carried them. One that declares it can't (membrane's + * XML prefill path) gets none, and what is queued waits for the turn's end. + * One that doesn't say (membrane 0.5.87) gets them, and they stay unread. + */ + +import { describe, it, beforeEach, afterEach } from 'node:test'; +import assert from 'node:assert/strict'; +import { tmpdir } from 'node:os'; +import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import type { + Module, + ModuleContext, + ProcessState, + ProcessEvent, + EventResponse, + ToolDefinition, + ToolCall, + ToolResult, +} from '../src/index.js'; +import { AgentFramework } from '../src/index.js'; +import { AutobiographicalStrategy } from '@animalabs/context-manager'; +import { MockMembrane, createMockResponse } from './helpers/mock-membrane.js'; +import type { ContentBlock } from '@animalabs/membrane'; + +interface Interjection { + participant: string; + content: ContentBlock[]; +} + +/** The installed context-manager's version. Its package's exports don't + * include package.json, so walk up from its entry point to it. */ +function contextManagerVersion(): string { + let dir = dirname(fileURLToPath(import.meta.resolve('@animalabs/context-manager'))); + for (;;) { + const file = join(dir, 'package.json'); + if (existsSync(file)) { + const pkg = JSON.parse(readFileSync(file, 'utf8')) as { name?: string; version?: string }; + if (pkg.name === '@animalabs/context-manager' && pkg.version) return pkg.version; + } + const up = dirname(dir); + if (up === dir) throw new Error("can't find @animalabs/context-manager's package.json"); + dir = up; + } +} + +/** Set, with the reason, when shards can't land before a throw: from + * context-manager 0.15, a sharded write that fails partway removes what it + * wrote. The tests of carrying landed shards pin 0.14.x. */ +const NO_SHARDS_LAND_BEFORE_A_THROW = (() => { + const version = contextManagerVersion(); + const [major, minor] = version.split('.').map(Number); + return major! > 0 || minor! >= 15 + ? `context-manager ${version} removes what a sharded write that fails partway wrote, so no shards land before a throw` + : false; +})(); + +/** Tools that deliver messages while they run, the way a module or a channel + * would while the agent is busy: `move` continues the turn, `halt` ends it. */ +class InterjectingModule implements Module { + readonly name = 'robot'; + framework: AgentFramework | null = null; + /** Delivered, in order, by the next tool call before it returns. */ + interjections: Interjection[] = []; + /** Run inside the next tool call, after the interjections are delivered. */ + duringTool: (() => void) | null = null; + + async start(_ctx: ModuleContext): Promise {} + async stop(): Promise {} + + getTools(): ToolDefinition[] { + return [ + { name: 'move', description: 'Move', inputSchema: { type: 'object', properties: {} } }, + { name: 'halt', description: 'End the turn', inputSchema: { type: 'object', properties: {} } }, + ]; + } + + async handleToolCall(call: ToolCall): Promise { + const pending = this.interjections; + this.interjections = []; + for (const interjection of pending) { + this.framework!.pushEvent({ type: 'interjection', source: 'test', interjection } as unknown as ProcessEvent); + } + // Let the run loop handle them while this round's tools are pending, and + // return only once each is in the deferred queue, so the tool-result + // boundary meets them there (bounded: a run that never queues them fails). + if (pending.length > 0) { + const queue = () => (this.framework as unknown as { deferredMessages: unknown[] }).deferredMessages.length; + const end = Date.now() + 5000; + while (queue() < pending.length) { + if (Date.now() > end) throw new Error(`only ${queue()} of ${pending.length} interjection(s) were queued`); + await new Promise((r) => setTimeout(r, 5)); + } + } + const during = this.duringTool; + this.duringTool = null; + during?.(); + return call.name.endsWith('halt') + ? { success: true, data: { halted: true }, endTurn: true } + : { success: true, data: { ok: true } }; + } + + async onProcess(event: ProcessEvent, _state: ProcessState): Promise { + if (event.type === 'external-message') { + return { + addMessages: [{ participant: 'Antra', content: [{ type: 'text', text: 'go' }] }], + requestInference: true, + }; + } + if ((event as { type: string }).type === 'interjection') { + const { participant, content } = (event as unknown as { interjection: Interjection }).interjection; + // No wake: this suite is about where the message lands in this turn. + return { addMessages: [{ participant, content }] }; + } + return {}; + } +} + +const text = (t: string): ContentBlock[] => [{ type: 'text', text: t } as ContentBlock]; + +/** One line per stored message: participant, then its text or block types. */ +function windowOf(framework: AgentFramework): string[] { + return framework.getAgent('assistant')!.getContextManager().getAllMessages().map((m) => { + const parts = m.content.map((b) => (b.type === 'text' ? (b as { text: string }).text : `<${b.type}>`)); + return `${m.participant}: ${parts.join(' ')}`; + }); +} + +describe('mid-turn deliveries keep the window what the stream carried', () => { + let tempDir: string; + let membrane: MockMembrane; + let module: InterjectingModule; + + beforeEach(() => { + tempDir = mkdtempSync(join(tmpdir(), 'dwo-test-')); + membrane = new MockMembrane(); + module = new InterjectingModule(); + }); + + afterEach(() => { + rmSync(tempDir, { recursive: true, force: true }); + }); + + async function createFramework(extra: Partial[0]> = {}): Promise { + const framework = await AgentFramework.create({ + storePath: join(tempDir, 'test.chronicle'), + membrane: membrane.asMembrane(), + agents: [{ name: 'assistant', model: 'test-model', systemPrompt: 'You are a robot pilot.' }], + modules: [module], + ...extra, + }); + module.framework = framework; + return framework; + } + + /** A turn of two rounds: `tool` with the interjections, then a reply. */ + let drains: Array<{ atBoundary: boolean; queued: string[] }> = []; + async function runTurn(tool: 'move' | 'halt', interjections: Interjection[]): Promise { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: `robot--${tool}`, input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + drains = recordDrains(framework); + module.interjections = interjections; + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + return framework; + } + + /** What the tool boundary's drain found queued. */ + const atBoundary = () => drains.find((d) => d.atBoundary)?.queued; + + /** What each drain for the agent found queued, in order, and whether it + * stopped at a held message (the tool boundary's drain does). */ + function recordDrains(framework: AgentFramework): Array<{ atBoundary: boolean; queued: string[] }> { + const drains: Array<{ atBoundary: boolean; queued: string[] }> = []; + const internals = framework as unknown as { + deferredMessages: Array<{ participant: string; content: ContentBlock[] }>; + drainDeferredFor: (name: string, stopAt?: unknown) => unknown[]; + }; + const drain = internals.drainDeferredFor.bind(framework); + internals.drainDeferredFor = (name: string, stopAt?: unknown) => { + if (name === 'assistant' && internals.deferredMessages.length > 0) { + drains.push({ + atBoundary: stopAt !== undefined, + queued: internals.deferredMessages.map((m) => `${m.participant}: ${m.content.map((b) => (b.type === 'text' ? (b as { text: string }).text : `<${b.type}>`)).join(' ')}`), + }); + } + return drain(name, stopAt); + }; + return drains; + } + + function injected(): string[] { + const options = membrane.lastStream!.receivedToolResultOptions[0]; + return (options?.injectedMessages ?? []).map((m) => + `${m.participant}: ${(m.content as Array<{ text?: string }>).map((b) => b.text).join(' ')}`); + } + + it('a message named as the agent waits for the turn to end, after its last round', async () => { + const framework = await runTurn('move', [{ participant: 'assistant', content: text('my own words') }]); + assert.deepEqual(atBoundary(), ['assistant: my own words'], 'it was queued when the boundary drained'); + assert.deepEqual(injected(), []); + assert.deepEqual(windowOf(framework), [ + 'Antra: go', + 'assistant: Moving. ', + 'user: ', + 'assistant: Done.', + 'assistant: my own words', + ]); + await framework.stop(); + }); + + it('a message carrying tool blocks waits too', async () => { + const framework = await runTurn('move', [{ + participant: 'Antra', + content: [{ type: 'tool_use', id: 'x1', name: 'robot--move', input: {} } as ContentBlock], + }]); + assert.deepEqual(injected(), []); + assert.deepEqual(windowOf(framework).slice(2), ['user: ', 'assistant: Done.', 'Antra: ']); + await framework.stop(); + }); + + it('what is queued behind a held message waits with it, in order', async () => { + const framework = await runTurn('move', [ + { participant: 'assistant', content: text('my own words') }, + { participant: 'Antra', content: text('look left') }, + ]); + assert.deepEqual(atBoundary(), ['assistant: my own words', 'Antra: look left'], 'both were queued when the boundary drained'); + assert.deepEqual(injected(), []); + assert.deepEqual(windowOf(framework).slice(2), [ + 'user: ', + 'assistant: Done.', + 'assistant: my own words', + 'Antra: look left', + ]); + await framework.stop(); + }); + + it('what is queued before a held message is still heard at the boundary', async () => { + const framework = await runTurn('move', [ + { participant: 'Antra', content: text('look left') }, + { participant: 'assistant', content: text('my own words') }, + ]); + assert.deepEqual(atBoundary(), ['Antra: look left', 'assistant: my own words'], 'both were queued when the boundary drained'); + assert.deepEqual(injected(), ['Antra: look left']); + assert.deepEqual(windowOf(framework).slice(2), [ + 'user: ', + 'Antra: look left', + 'assistant: Done.', + 'assistant: my own words', + ]); + await framework.stop(); + }); + + /** Whether the stored message with this text reads as unread (RFC-006). */ + function unread(framework: AgentFramework, textOf: string): boolean { + const agent = framework.getAgent('assistant')!; + const stored = agent.getContextManager().getAllMessages() + .find((m) => m.content.some((b) => (b as { text?: string }).text === textOf))!; + assert.ok(stored, `'${textOf}' is stored`); + return (framework as unknown as { + isUnreadStoredMessage: (a: typeof agent, id: string) => boolean; + }).isUnreadStoredMessage.call(framework, agent, stored.id); + } + + it('a message injected into a stream that declares it carries injections is read: the consumed watermark covers it', async () => { + membrane.supportsInjectedMessages = true; // membrane's native path declares it + const framework = await runTurn('move', [{ participant: 'Antra', content: text('look left') }]); + assert.deepEqual(injected(), ['Antra: look left']); + assert.equal(unread(framework, 'look left'), false, 'a coalesced replacement must not remove what the agent heard'); + await framework.stop(); + }); + + it('a stream that doesn\'t say whether it carries injections gets them, and they stay unread', async () => { + // membrane 0.5.87 doesn't declare it, on either path, and its XML path + // drops what's injected: marked read there, a message the agent never + // saw could no longer be replaced. + const framework = await runTurn('move', [{ participant: 'Antra', content: text('look left') }]); + assert.deepEqual(injected(), ['Antra: look left']); + assert.deepEqual(windowOf(framework).slice(2), ['user: ', 'Antra: look left', 'assistant: Done.']); + assert.equal(unread(framework, 'look left'), true); + await framework.stop(); + }); + + it('a stream that can\'t carry injections gets none: what is queued waits for the turn to end', async () => { + membrane.supportsInjectedMessages = false; // membrane's XML prefill path declares it + const framework = await runTurn('move', [{ participant: 'Antra', content: text('look left') }]); + assert.deepEqual(atBoundary(), undefined, 'the boundary drained nothing'); + assert.equal(membrane.lastStream!.receivedToolResultOptions[0], undefined, 'nothing was injected'); + assert.deepEqual(windowOf(framework).slice(2), ['user: ', 'assistant: Done.', 'Antra: look left']); + assert.equal(unread(framework, 'look left'), true); + await framework.stop(); + }); + + it('a message stored at a boundary that ends the turn was never carried, and stays unread', async () => { + const framework = await runTurn('halt', [{ participant: 'Antra', content: text('too late') }]); + const agent = framework.getAgent('assistant')!; + const late = agent.getContextManager().getAllMessages() + .find((m) => m.content.some((b) => (b as { text?: string }).text === 'too late'))!; + const isUnread = (framework as unknown as { + isUnreadStoredMessage: (a: typeof agent, id: string) => boolean; + }).isUnreadStoredMessage.bind(framework); + assert.equal(isUnread(agent, late.id), true); + await framework.stop(); + }); + + /** The framework's own queue and its record of turns and tool cycles. */ + function internalsOf(framework: AgentFramework) { + return framework as unknown as { + pendingAssistantBlocks: Map; + activeTurnTokens: Map; + activeStreams: Map; + deferredMessages: Array<{ id: string; seq: number; participant: string }>; + deferredWritesPersisted: boolean; + drainDeferredFor: (name: string, stopAt?: unknown) => unknown[]; + readdDrained: (entries: unknown[], where: string) => unknown[]; + addMessage(participant: string, content: ContentBlock[], metadata?: undefined, opts?: { forAgent?: string; fromQueue?: { id: string; seq: number } }): string; + }; + } + const otherCycle = (): ContentBlock[] => [{ type: 'tool_use', id: 'b1', name: 'robot--move', input: {} } as ContentBlock]; + + it('a message held for the turn\'s end lands then, while another agent\'s tool cycle is pending', async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + const internals = internalsOf(framework); + // Whether the other cycle was pending at the turn-end flush itself. + const pendingAtFlush: boolean[] = []; + const drain = internals.drainDeferredFor.bind(framework); + internals.drainDeferredFor = (name: string, stopAt?: unknown) => { + if (name === 'assistant' && stopAt === undefined && internals.deferredMessages.length > 0) { + pendingAtFlush.push(internals.pendingAssistantBlocks.has('builder')); + } + return drain(name, stopAt); + }; + module.duringTool = () => { + // Another agent is mid tool cycle when this turn ends, and a write + // named as this agent (held: it can't join the live stream) waits. + internals.pendingAssistantBlocks.set('builder', otherCycle()); + assert.equal(internals.addMessage('assistant', text('my own words'), undefined, { forAgent: 'assistant' }), '', 'deferred'); + }; + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + assert.deepEqual(pendingAtFlush, [true], 'the other cycle was pending at the turn-end flush'); + assert.deepEqual(windowOf(framework).slice(-2), ['assistant: Done.', 'assistant: my own words']); + internals.pendingAssistantBlocks.delete('builder'); + await framework.stop(); + }); + + it("a write waits on its target's own turn and tool cycle, not another agent's, primary-bound or targeted", async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + internals.pendingAssistantBlocks.set('builder', otherCycle()); + try { + assert.notEqual(internals.addMessage('Antra', text('primary-bound')), '', 'stored: the primary is idle'); + assert.notEqual(internals.addMessage('Antra', text('targeted'), undefined, { forAgent: 'assistant' }), '', 'stored too'); + internals.pendingAssistantBlocks.set('assistant', otherCycle()); + assert.equal(internals.addMessage('Antra', text('in its own cycle')), '', "deferred: the primary's own cycle is pending"); + } finally { + internals.pendingAssistantBlocks.delete('builder'); + internals.pendingAssistantBlocks.delete('assistant'); + } + await framework.stop(); + }); + + it("the puppet's flush lands what was deferred behind it while another agent's tool cycle is pending", async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + module.duringTool = () => { + internals.pendingAssistantBlocks.set('builder', otherCycle()); + assert.equal(internals.addMessage('Antra', text('while puppeted')), '', 'deferred behind the puppet'); + }; + try { + await framework.puppetToolCall('assistant', 'robot--move', {}); + assert.equal(internals.deferredMessages.length, 0); + assert.deepEqual(windowOf(framework).slice(-3), ['assistant: ', 'user: ', 'Antra: while puppeted']); + } finally { + internals.pendingAssistantBlocks.delete('builder'); + } + await framework.stop(); + }); + + it('a turn mid-compile when the framework stops never starts a stream, and what was deferred for it is stored', async () => { + membrane.pushResponse(createMockResponse(text('too late'))); + const framework = await createFramework(); + const internals = internalsOf(framework); + const agent = framework.getAgent('assistant')!; + let compiling!: () => void; + const inCompile = new Promise((resolve) => { compiling = resolve; }); + let release!: () => void; + const released = new Promise((resolve) => { release = resolve; }); + const start = agent.startStreamWithInjections.bind(agent); + (agent as unknown as { startStreamWithInjections: typeof start }).startStreamWithInjections = async (...args) => { + compiling(); + await released; + return start(...args); + }; + // The event loop starts the turn, as in a host (stop() awaits the loop). + framework.start(); + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await inCompile; + assert.equal(internals.addMessage('Antra', text('late word')), '', 'deferred: the turn is alive'); + const stopped = framework.stop(); + release(); + await stopped; + assert.equal(internals.activeStreams.size, 0, 'no stream outlived stop()'); + + const reopened = await createFramework(); + assert.ok(windowOf(reopened).includes('Antra: late word'), 'stored before the store closed'); + assert.ok(!windowOf(reopened).includes('assistant: too late'), 'the refused turn produced nothing'); + await reopened.stop(); + }); + + it('a drained write whose store throws stays in the persisted queue, and is stored on the next boot', async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + internals.deferredWritesPersisted = true; + internals.activeTurnTokens.set('assistant', 1); + assert.equal(internals.addMessage('Antra', text('kept')), '', 'deferred'); + internals.activeTurnTokens.delete('assistant'); + const cm = framework.getAgent('assistant')!.getContextManager(); + const add = cm.addMessage.bind(cm); + (cm as unknown as { addMessage: typeof add }).addMessage = (...args) => { + if ((args[1] as Array<{ text?: string }>).some((b) => b.text === 'kept')) throw new Error('disk full'); + return add(...args); + }; + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { await framework.stop(); } finally { console.error = error; } + assert.ok(errors.some((e) => e.includes('could not be stored; it stays queued') && e.includes('disk full')), errors.join('\n')); + assert.equal(internals.deferredMessages.length, 1, 'still queued'); + + const reopened = await createFramework(); + assert.ok(windowOf(reopened).includes('Antra: kept'), 'boot recovery stored it'); + await reopened.stop(); + }); + + it('at a tool boundary, a write whose store throws cuts the drain: it and what follows wait for the turn end, in order', async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + const cm = framework.getAgent('assistant')!.getContextManager(); + const add = cm.addMessage.bind(cm); + let failOnce = true; + (cm as unknown as { addMessage: typeof add }).addMessage = (...args) => { + if (failOnce && (args[1] as Array<{ text?: string }>).some((b) => b.text === 'look left')) { + failOnce = false; + throw new Error('transient'); + } + return add(...args); + }; + module.interjections = [ + { participant: 'Antra', content: text('look left') }, + { participant: 'Antra', content: text('look right') }, + ]; + const error = console.error; + console.error = () => {}; + try { + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + } finally { console.error = error; } + assert.deepEqual(injected(), [], 'nothing from the failed write on is injected'); + assert.deepEqual(windowOf(framework).slice(2), [ + 'user: ', + 'assistant: Done.', + 'Antra: look left', + 'Antra: look right', + ], 'both were kept, and stored at the turn end in queue order'); + await framework.stop(); + }); + + /** The strategy's onNewMessage throws once, synchronously, for the record + * with this text. The manager's own store listener calls it as each record + * is written, so the write throws though its message is stored. A host's + * onMessage listener can't do this on context-manager 0.15, which logs its + * throw and lets the write stand. */ + function throwOnceAfterStoring(framework: AgentFramework, textOf: string): void { + const strategy = (framework.getAgent('assistant')!.getContextManager() as unknown as { + strategy: { onNewMessage?: (message: { content: ContentBlock[] }, ctx: unknown) => Promise }; + }).strategy; + const onNewMessage = strategy.onNewMessage; + let thrown = false; + strategy.onNewMessage = function (this: unknown, message, ctx) { + if (!thrown && message.content.some((b) => (b as { text?: string }).text === textOf)) { + thrown = true; + throw new Error('strategy failed'); + } + return onNewMessage ? onNewMessage.call(this, message, ctx) : Promise.resolve(); + }; + } + + it('at a tool boundary, a write that throws after its message is stored counts as stored: injected, and not stored again', async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + throwOnceAfterStoring(framework, 'look left'); + module.interjections = [ + { participant: 'Antra', content: text('look left') }, + { participant: 'Antra', content: text('look right') }, + ]; + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + } finally { console.error = error; } + assert.deepEqual(injected(), ['Antra: look left', 'Antra: look right'], 'the stored copy is carried too'); + assert.deepEqual(windowOf(framework).slice(2), [ + 'user: ', + 'Antra: look left', + 'Antra: look right', + 'assistant: Done.', + ], 'stored once, where the stream carried it'); + assert.ok(errors.some((e) => e.includes("threw after it was stored; what was stored counts as written and isn't stored again") + && e.includes('strategy failed')), errors.join('\n')); + await framework.stop(); + }); + + /** Shard a message whose first block is `marker` into one shard per block, + * as a strategy's ingress chunking does. */ + function shardOn(framework: AgentFramework, marker: string): void { + const cm = framework.getAgent('assistant')!.getContextManager() as unknown as { strategy: Record }; + cm.strategy.chunkIngressMessage = (_participant: string, content: ContentBlock[]) => + (content[0] as { text?: string } | undefined)?.text === marker + ? { bodyGroupId: `body-${marker}`, shards: content.map((block, shardIndex) => ({ content: [block], shardIndex })) } + : null; + } + + it("at a tool boundary, a sharded body that throws after its first shard: what landed is carried and counts as written, and the rest isn't stored", { skip: NO_SHARDS_LAND_BEFORE_A_THROW }, async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + shardOn(framework, 'part one'); + // The strategy throws on the first shard's add, which stops the append + // loop before the second shard. + throwOnceAfterStoring(framework, 'part one'); + module.interjections = [ + { participant: 'Antra', content: [...text('part one'), ...text('part two')] }, + { participant: 'Antra', content: text('look right') }, + ]; + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + } finally { console.error = error; } + assert.deepEqual(injected(), ['Antra: part one', 'Antra: look right'], 'the request carries what the store holds'); + assert.deepEqual(windowOf(framework).slice(2), [ + 'user: ', + 'Antra: part one', + 'Antra: look right', + 'assistant: Done.', + ], 'the first shard alone, stored once'); + assert.equal(internalsOf(framework).deferredMessages.length, 0, 'nothing went back into the queue'); + assert.ok(errors.some((e) => e.includes("threw after 1 shard of its body was stored; what was stored counts as written and isn't stored again") + && e.includes('strategy failed')), errors.join('\n')); + await framework.stop(); + }); + + it('at a tool boundary, the shards that landed before a throw are carried as the next compile renders them', { skip: NO_SHARDS_LAND_BEFORE_A_THROW }, async () => { + // Probe by Ansel-2428: the autobiographical strategy shards a long + // message on ingest and joins a body's consecutive shards when it + // compiles, so the boundary must carry what landed in that shape. The + // body with an image between two texts fails here if the join changes + // where it puts a non-text block, so the copy of the join can't drift. + const long = Array.from({ length: 60 }, (_, i) => `Sentence ${i} of a long message from Antra.`).join(' '); + const ofLong = (messages: Array<{ content: unknown }>) => + messages.filter((m) => JSON.stringify(m.content).includes('Sentence 0 ')); + const PNG = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg=='; + const image = { type: 'image', source: { type: 'base64', data: PNG, mediaType: 'image/png' } } as ContentBlock; + const cases: Array<{ name: string; content: ContentBlock[]; throwAt: number }> = [ + { name: 'text, throw at shard 2', content: text(long), throwAt: 2 }, + { name: 'text, throw at shard 3', content: text(long), throwAt: 3 }, + { name: 'a short text, an image, then the long text, throw at shard 2', content: [...text('Intro. '), image, ...text(long)], throwAt: 2 }, + ]; + for (const [i, { name, content, throwAt }] of cases.entries()) { + membrane = new MockMembrane(); + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework({ + storePath: join(tempDir, `shards-${i}.chronicle`), + membrane: membrane.asMembrane(), + agents: [{ + name: 'assistant', model: 'test-model', systemPrompt: 'You are a robot pilot.', + strategy: new AutobiographicalStrategy({ + adaptiveResolution: true, foldingStrategy: 'kv-stable', recentWindowTokens: 30_000, kvStableReachTokens: 8_000, + targetChunkTokens: 50, + }), + } as never], + }); + const cm = framework.getAgent('assistant')!.getContextManager(); + let shards = 0; + cm.onMessage((event) => { + if (event.type !== 'add' || !(event.message as { bodyGroupId?: string }).bodyGroupId) return; + if (++shards === throwAt) throw new Error('listener failed'); + }); + module.interjections = [{ participant: 'Antra', content }]; + const error = console.error; + console.error = () => {}; + try { + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + const landed = cm.getAllMessages().filter((m) => (m as { bodyGroupId?: string }).bodyGroupId); + assert.equal(landed.length, throwAt, `${name}: ${throwAt} shards landed before the throw stopped the body`); + const injected = ofLong(membrane.lastStream!.receivedToolResultOptions[0]?.injectedMessages ?? []); + // The next turn's request is compiled from the store. + membrane.pushResponse(createMockResponse(text('Okay.'))); + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + const next = ofLong(membrane.calls.at(-1)!.messages); + assert.equal(injected.length, 1, `${name}: one message injected`); + assert.equal(next.length, 1, `${name}: one message in the next request`); + assert.deepEqual(injected[0]!.content, next[0]!.content, `${name}: the boundary carried what the next compile renders`); + } finally { + console.error = error; + await framework.stop(); + } + } + }); + + it("a write that throws after it's stored, re-added from the queue, is weighed for its agent's open turn as a stored write is", async () => { + const framework = await createFramework(); + throwOnceAfterStoring(framework, 'from beta'); + const internals = framework as unknown as { + openLogical: Map; + readdDrained(entries: Array<{ id: string; seq: number; participant: string; content: ContentBlock[]; metadata?: Record }>, where: string): unknown[]; + }; + // A logical turn still open between its streams, engaged with chan-beta: + // no turn is alive, so no tool boundary weighs what is stored now, and + // addMessage notes it as one of the turn's arrivals. + const open = { engaged: new Set(['chan-beta']), arrivals: [] as Array<{ messageId: string; channelId: string }> }; + internals.openLogical.set('assistant', open); + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + let written: unknown[]; + try { + written = internals.readdDrained( + [{ id: 'w1', seq: 1, participant: 'Antra', content: text('from beta'), metadata: { channelId: 'chan-beta' } }], + 'test flush', + ); + } finally { console.error = error; } + assert.ok(errors.some((e) => e.includes('test flush: deferred write') + && e.includes("threw after it was stored; what was stored counts as written and isn't stored again") + && e.includes('strategy failed')), errors.join('\n')); + assert.equal(written.length, 1, 'it counts as written'); + const copies = framework.getAgent('assistant')!.getContextManager().getAllMessages() + .filter((m) => (m.metadata as { deferredWriteId?: string } | undefined)?.deferredWriteId === 'w1'); + assert.equal(copies.length, 1, 'stored once'); + assert.deepEqual(open.arrivals, [{ messageId: copies[0]!.id, channelId: 'chan-beta' }], 'and noted as one of the open turn\'s arrivals'); + await framework.stop(); + }); + + it("writes re-added from the queue are noted for an open turn in the order they landed, one that throws after it's stored among them", async () => { + const framework = await createFramework(); + throwOnceAfterStoring(framework, 'from alpha'); + const agent = framework.getAgent('assistant')!; + const internals = framework as unknown as { + openLogical: Map; + readdDrained(entries: Array<{ id: string; seq: number; participant: string; content: ContentBlock[]; metadata?: Record }>, where: string): unknown[]; + remainingArrival(agent: unknown, routing: unknown): string | undefined; + }; + // An open turn engaged with both channels, between its streams: the + // earlier write throws after it's stored, and the later one lands. + const open = { engaged: new Set(['chan-alpha', 'chan-beta']), arrivals: [] as Array<{ messageId: string; channelId: string }> }; + internals.openLogical.set('assistant', open); + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { + internals.readdDrained([ + { id: 'w1', seq: 1, participant: 'Antra', content: text('from alpha'), metadata: { channelId: 'chan-alpha' } }, + { id: 'w2', seq: 2, participant: 'Antra', content: text('from beta'), metadata: { channelId: 'chan-beta' } }, + ], 'test flush'); + } finally { console.error = error; } + assert.equal(errors.filter((e) => e.includes('threw after it was stored')).length, 1, errors.join('\n')); + const idOf = (writeId: string) => agent.getContextManager().getAllMessages() + .find((m) => (m.metadata as { deferredWriteId?: string } | undefined)?.deferredWriteId === writeId)!.id; + assert.deepEqual(open.arrivals, [ + { messageId: idOf('w1'), channelId: 'chan-alpha' }, + { messageId: idOf('w2'), channelId: 'chan-beta' }, + ]); + assert.equal(internals.remainingArrival(agent, open), 'chan-beta', "the turn's newest arrival is the later write's"); + await framework.stop(); + }); + + it('at the turn end, a write that throws after its message is stored is not queued again', async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + throwOnceAfterStoring(framework, 'look left'); + // Held behind a message named as the agent, both wait for the turn end. + module.interjections = [ + { participant: 'assistant', content: text('my own words') }, + { participant: 'Antra', content: text('look left') }, + ]; + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + } finally { console.error = error; } + assert.equal(internalsOf(framework).deferredMessages.length, 0, 'nothing went back into the queue'); + assert.ok(errors.some((e) => e.includes("turn-end flush: deferred write") + && e.includes("threw after it was stored; what was stored counts as written and isn't stored again") + && e.includes('strategy failed')), errors.join('\n')); + await framework.stop(); // stop() lands whatever is still queued + + const reopened = await createFramework(); + assert.deepEqual(windowOf(reopened).slice(2), [ + 'user: ', + 'assistant: Done.', + 'assistant: my own words', + 'Antra: look left', + ], 'stored once, at the turn end'); + await reopened.stop(); + }); + + it("a queued tool_use whose store throws keeps its tool_result from being stored alone", async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + internals.deferredWritesPersisted = true; + // A pair queued for the agent, as puppetToolCall queues one behind a live turn. + const queue = framework as unknown as { + deferredSeq: number; + deferredMessages: Array<{ id: string; seq: number; participant: string; content: ContentBlock[]; forAgent?: string }>; + }; + queue.deferredMessages.push( + { id: 'pair-use', seq: ++queue.deferredSeq, participant: 'assistant', content: [{ type: 'tool_use', id: 'p1', name: 'robot--move', input: {} } as ContentBlock], forAgent: 'assistant' }, + { id: 'pair-result', seq: ++queue.deferredSeq, participant: 'user', content: [{ type: 'tool_result', toolUseId: 'p1', content: 'moved' } as ContentBlock], forAgent: 'assistant' }, + ); + const cm = framework.getAgent('assistant')!.getContextManager(); + const add = cm.addMessage.bind(cm); + (cm as unknown as { addMessage: typeof add }).addMessage = (...args) => { + if ((args[1] as ContentBlock[]).some((b) => b.type === 'tool_use')) throw new Error('disk full'); + return add(...args); + }; + const errors: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { errors.push(args.map(String).join(' ')); }; + try { await framework.stop(); } finally { console.error = error; } + assert.ok(errors.some((e) => e.includes('pair-use (assistant: tool_use)') && e.includes('disk full')), errors.join('\n')); + assert.deepEqual(queue.deferredMessages.map((m) => m.id), ['pair-use', 'pair-result'], 'both stay queued, in order'); + + const reopened = await createFramework(); + const window = windowOf(reopened); + assert.deepEqual(window.slice(-2), ['assistant: ', 'user: '], 'boot recovery stored the pair, in order'); + assert.equal(window.filter((line) => line === 'user: ').length, 1, 'no tool_result was stored alone'); + await reopened.stop(); + }); + + it("stop() stores what is deferred for an agent whose turn frame it doesn't wait for", async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + // A frame stop() doesn't await, such as a turn started outside the + // event loop, holds the agent's turn token. + internals.activeTurnTokens.set('assistant', 1); + assert.equal(internals.addMessage('Antra', text('still to land')), '', 'deferred: the turn is alive'); + await framework.stop(); + + const reopened = await createFramework(); + assert.ok(windowOf(reopened).includes('Antra: still to land'), 'stored before the store closed'); + await reopened.stop(); + }); + + for (const persisted of [false, true]) { + it(`stop() stores several agents' deferred writes in the order they arrived (${persisted ? 'persisted' : 'memory-only'} queue)`, async () => { + // Residents share one message slot, so the order writes arrived in is + // the order every resident reads. + const agents = [ + { name: 'assistant', model: 'test-model', systemPrompt: 'You are a robot pilot.' }, + { name: 'scout', model: 'test-model', systemPrompt: 'You scout.' }, + ]; + const framework = await createFramework({ agents }); + const internals = internalsOf(framework); + // A queue boot recovery restored, or one a quiesce persisted, stays + // persisted until it empties. + internals.deferredWritesPersisted = persisted; + // Frames stop() doesn't wait for hold both agents' turns. + internals.activeTurnTokens.set('assistant', 1); + internals.activeTurnTokens.set('scout', 2); + assert.equal(internals.addMessage('Antra', text('one')), '', 'deferred'); + assert.equal(internals.addMessage('Antra', text('two'), undefined, { forAgent: 'scout' }), '', 'deferred'); + assert.equal(internals.addMessage('Antra', text('three')), '', 'deferred'); + // A turn-end flush of the assistant's entries while a successor holds + // its turn defers them again (the 2026-07-31 interleave). They keep + // their places in arrival order. + internals.readdDrained(internals.drainDeferredFor('assistant'), 'turn-end flush'); + assert.equal(internals.deferredMessages.length, 3, 'deferred again'); + await framework.stop(); + + const reopened = await createFramework({ agents }); + assert.deepEqual( + windowOf(reopened).filter((line) => /^Antra: (one|two|three)$/.test(line)), + ['Antra: one', 'Antra: two', 'Antra: three'], + ); + await reopened.stop(); + }); + } + + it('a turn waiting to retry when the framework stops gives up rather than compile again', { timeout: 60_000 }, async () => { + let waiting!: () => void; + const inDelay = new Promise((resolve) => { waiting = resolve; }); + const framework = await createFramework({ + errorPolicy: { + maxRetries: 2, + onInferenceError: (_error: Error, _agent: string, attempt: number) => { + waiting(); + // Long enough that only stop() ends the wait. + return attempt < 2 ? { retry: true as const, delayMs: 60_000 } : { retry: false as const }; + }, + }, + }); + const internals = internalsOf(framework); + const agent = framework.getAgent('assistant')!; + let compiles = 0; + (agent as unknown as { startStreamWithInjections: () => Promise }).startStreamWithInjections = async () => { + compiles++; + throw new Error('boom'); + }; + // A retry re-enters startAgentStream after its wait. stop() ends the + // wait, and that cancels the retry before it re-enters: count re-entries, + // so a retry that went ahead can't pass for one that was cancelled. + const retries: Array<{ attempt: number; stopping: boolean }> = []; + const frames = framework as unknown as { + providerAdmissionClosed: boolean; + startAgentStream(agent: unknown, trigger?: unknown, attempt?: number, held?: boolean): Promise; + }; + const startAgentStream = frames.startAgentStream.bind(framework); + frames.startAgentStream = (a, trigger, attempt = 0, held = false) => { + if (attempt > 0) retries.push({ attempt, stopping: frames.providerAdmissionClosed }); + return startAgentStream(a, trigger, attempt, held); + }; + framework.start(); + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await inDelay; + assert.equal(internals.addMessage('Antra', text('while it waits')), '', 'deferred: the turn is alive'); + const error = console.error; + console.error = () => {}; + try { await framework.stop(); } finally { console.error = error; } + assert.deepEqual(retries, [], 'stop() ended the wait, and the retry was cancelled before it re-entered'); + assert.equal(compiles, 1, 'the retry was not compiled'); + + const reopened = await createFramework(); + assert.ok(windowOf(reopened).includes('Antra: while it waits'), 'what was deferred for it was stored'); + await reopened.stop(); + }); + + it('a message stored straight into the context during a round, with nothing injected at the boundary, stays unread', async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + const agent = framework.getAgent('assistant')!; + let bypassed = ''; + module.duringTool = () => { + // A writer that bypasses deferral (an older route): the stream never carried it. + bypassed = agent.getContextManager().addMessage('Antra', text('bypassed')); + }; + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await framework.runUntilIdle(); + assert.deepEqual(injected(), []); + const isUnread = (framework as unknown as { + isUnreadStoredMessage: (a: typeof agent, id: string) => boolean; + }).isUnreadStoredMessage.bind(framework); + assert.equal(isUnread(agent, bypassed), true, 'a boundary that injected nothing marks nothing read'); + await framework.stop(); + }); + + it("a compile's watermark never rolls back a mark made while it ran", async () => { + const framework = await createFramework(); + const agent = framework.getAgent('assistant')!; + const cm = agent.getContextManager(); + cm.addMessage('Antra', text('before')); + const compile = cm.compile.bind(cm); + let release!: () => void; + const released = new Promise((resolve) => { release = resolve; }); + (cm as unknown as { compile: typeof compile }).compile = async (...args) => { + await released; + return compile(...args); + }; + const compiling = agent.compileContext(); + cm.addMessage('Antra', text('heard at a boundary')); + agent.markContextConsumed(); + const marked = agent.getConsumedWatermark(); + release(); + await compiling; + assert.deepEqual(agent.getConsumedWatermark(), marked, "the boundary's mark stands"); + (cm as unknown as { compile: typeof compile }).compile = compile; + await agent.compileContext(); + assert.ok(agent.getConsumedWatermark()!.sequence >= marked!.sequence, 'a later compile still advances it'); + await framework.stop(); + }); + + it("a held message waits out a successor turn's tool cycle, and lands once it is over", async () => { + membrane.pushResponse(createMockResponse([ + { type: 'text', text: 'Moving.' }, + { type: 'tool_use', id: 'c1', name: 'robot--move', input: {} }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse(text('Done.'))); + const framework = await createFramework(); + const internals = internalsOf(framework) as ReturnType & { releasePrimaryProviderGate: (name: string) => void }; + const agent = framework.getAgent('assistant')!; + module.duringTool = () => { + assert.equal(internals.addMessage('assistant', text('my own words'), undefined, { forAgent: 'assistant' }), '', 'held'); + }; + // As the turn's stream completes, a successor turn takes the agent and is + // mid tool cycle (the 2026-07-31 interleave): the stream generation, the + // turn token and the cycle are the successor's when the first turn's + // teardown flushes. + let completed = false; + const off = framework.onTrace((event) => { + if ((event as { type: string }).type !== 'inference:completed') return; + off(); + completed = true; + (agent as unknown as { _streamId: number })._streamId++; + internals.activeTurnTokens.set('assistant', 1_000_000); + internals.pendingAssistantBlocks.set('assistant', otherCycle()); + }); + // The first turn's teardown releases its provider gate, then flushes, + // synchronously: once the release is seen, the flush has run. + let tornDown!: () => void; + const teardown = new Promise((resolve) => { tornDown = resolve; }); + const release = internals.releasePrimaryProviderGate.bind(framework); + internals.releasePrimaryProviderGate = (name: string) => { + release(name); + if (completed && name === 'assistant') tornDown(); + }; + framework.start(); + framework.pushEvent({ type: 'external-message', source: 'test', content: 'go' } as unknown as ProcessEvent); + await teardown; + await new Promise((resolve) => setImmediate(resolve)); + assert.ok(!windowOf(framework).includes('assistant: my own words'), "not stored inside the successor's cycle"); + assert.ok(internals.deferredMessages.some((m) => m.participant === 'assistant'), 'still queued'); + // The successor's cycle and turn end; what is still queued lands (here at stop's drain). + internals.pendingAssistantBlocks.delete('assistant'); + internals.activeTurnTokens.delete('assistant'); + internals.activeStreams.delete('assistant'); + await framework.stop(); + const reopened = await createFramework(); + assert.equal(windowOf(reopened).at(-1), 'assistant: my own words'); + await reopened.stop(); + }); + + it('a write coming back from the deferred queue re-defers while a turn is alive, tool_result or not', async () => { + const framework = await createFramework(); + const internals = internalsOf(framework); + internals.activeTurnTokens.set('assistant', 1); + try { + // A queued pair's tool_result must follow its tool_use back into the + // queue, or the pair splits (the turn-end flush can run after a + // successor turn has started). + const requeued = internals.addMessage('user', [ + { type: 'tool_result', toolUseId: 'p1', content: 'ok' } as ContentBlock, + ], undefined, { fromQueue: { id: 'w1', seq: 41 } }); + assert.equal(requeued, ''); + // It goes back under its id and its place in arrival order, though this + // queue is held in memory only. + assert.ok(internals.deferredMessages.some((m) => m.id === 'w1' && m.seq === 41)); + // So does plain text. + assert.equal(internals.addMessage('Antra', text('plain'), undefined, { fromQueue: { id: 'w2', seq: 42 } }), ''); + assert.ok(internals.deferredMessages.some((m) => m.id === 'w2' && m.seq === 42)); + // A fresh tool_result write keeps landing as before. + const fresh = internals.addMessage('user', [ + { type: 'tool_result', toolUseId: 'p2', content: 'ok' } as ContentBlock, + ]); + assert.notEqual(fresh, ''); + } finally { + internals.activeTurnTokens.delete('assistant'); + } + await framework.stop(); + }); +}); diff --git a/test/ephemeral-settle.test.ts b/test/ephemeral-settle.test.ts index ce029a56c..58bb2ddcc 100644 --- a/test/ephemeral-settle.test.ts +++ b/test/ephemeral-settle.test.ts @@ -17,6 +17,7 @@ import type { ToolResult, } from '../src/index.js'; import { AgentFramework } from '../src/index.js'; +import { JsStore } from '@animalabs/chronicle'; import { createMockResponse, MockMembrane, MockYieldingStream } from './helpers/mock-membrane.js'; class EphemeralToolModule implements Module { @@ -62,6 +63,30 @@ class EphemeralToolModule implements Module { } } +/** A tool that, while its call runs, has `duringCall` run: there, a test + * addresses the run as the event gate does, through the framework's + * addMessage with `forAgent`. */ +class DuringCallToolModule implements Module { + readonly name = 'test'; + duringCall: (() => void) | null = null; + + async start(_ctx: ModuleContext): Promise {} + async stop(): Promise {} + + getTools(): ToolDefinition[] { + return [{ name: 'echo', description: 'Echo input', inputSchema: { type: 'object', properties: {} } }]; + } + + async handleToolCall(_call: ToolCall): Promise { + this.duringCall?.(); + return { success: true, data: { ok: true } }; + } + + async onProcess(_event: ProcessEvent, _state: ProcessState): Promise { + return {}; + } +} + class ThrowingMembrane extends MockMembrane { readonly errors: Error[]; @@ -110,18 +135,229 @@ async function runStartedEphemeral( framework: AgentFramework, agent: Awaited>['agent'], contextManager: Awaited>['contextManager'], + watchdogs?: Parameters[2], ) { - const promise = framework.runEphemeralToCompletion(agent, contextManager); + const promise = framework.runEphemeralToCompletion(agent, contextManager, watchdogs); framework.start(); return promise; } +// For a run that stop() is to end: its turn has started inference, so if the +// refusal didn't end it, only the idle watchdog would, after fifteen minutes. +// A short one turns that into a "stalled" rejection, which the refusal's +// pattern doesn't match. +const STOPPED_RUN_WATCHDOGS = { idleTimeoutMs: 20_000, idlePollMs: 50 }; + async function stopAndRemove(framework: AgentFramework, tempDir: string): Promise { await framework.stop(); rmSync(tempDir, { recursive: true, force: true }); } describe('runEphemeralToCompletion settle signal', () => { + it('a run whose turn stop() refuses ends at once, and a run asked for once stop() has begun is refused', { timeout: 60_000 }, async () => { + const { tempDir, storePath } = tempStorePath('ephemeral-stopping-'); + const membrane = new MockMembrane(); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'too late' }])); + const framework = await AgentFramework.create({ + storePath, + membrane: membrane.asMembrane(), + agents: [], + modules: [new EphemeralToolModule()], + syncIntervalMs: 0, + }); + try { + const { agent, contextManager } = await createEphemeral(framework); + const later = await createEphemeral(framework, 'later'); + // Hold the run's compile until stop() has begun. + let compiling!: () => void; + const inCompile = new Promise((resolve) => { compiling = resolve; }); + let release!: () => void; + const released = new Promise((resolve) => { release = resolve; }); + const start = agent.startStreamWithInjections.bind(agent); + (agent as unknown as { startStreamWithInjections: typeof start }).startStreamWithInjections = async (...args) => { + compiling(); + await released; + return start(...args); + }; + const run = runStartedEphemeral(framework, agent, contextManager, STOPPED_RUN_WATCHDOGS); + await inCompile; + const stopped = framework.stop(); + release(); + await assert.rejects(run, /refused: the framework is stopping/); + await stopped; + await assert.rejects( + framework.runEphemeralToCompletion(later.agent, later.contextManager), + /framework is stopping: refusing new ephemeral run/, + ); + later.cleanup(); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('a run whose turn stop() refuses before its compile ends at once', { timeout: 60_000 }, async () => { + // The turn waits on its modules' context when stop() begins, so it is + // refused before it compiles, not after. + const { tempDir, storePath } = tempStorePath('ephemeral-stopping-precompile-'); + let gathering!: () => void; + const inGather = new Promise((resolve) => { gathering = resolve; }); + let release!: () => void; + const released = new Promise((resolve) => { release = resolve; }); + const contextModule: Module = new EphemeralToolModule(); + contextModule.gatherContext = async () => { + gathering(); + await released; + return []; + }; + const framework = await AgentFramework.create({ + storePath, + membrane: new MockMembrane().asMembrane(), + agents: [], + modules: [contextModule], + syncIntervalMs: 0, + }); + try { + const { agent, contextManager } = await createEphemeral(framework); + let compiles = 0; + const start = agent.startStreamWithInjections.bind(agent); + (agent as unknown as { startStreamWithInjections: typeof start }).startStreamWithInjections = async (...args) => { + compiles++; + return start(...args); + }; + const run = runStartedEphemeral(framework, agent, contextManager, STOPPED_RUN_WATCHDOGS); + await inGather; + const stopped = framework.stop(); + release(); + await assert.rejects(run, /refused: the framework is stopping/); + await stopped; + assert.equal(compiles, 0, 'the turn was refused before its compile'); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('a run waiting to retry when stop() begins gives up before its turn starts again, and ends at once', { timeout: 60_000 }, async () => { + // stop() ends the retry's wait, which cancels the retry before its turn + // starts again; the run ends then, not when its idle watchdog fires. + const { tempDir, storePath } = tempStorePath('ephemeral-retry-stopping-'); + let waiting!: () => void; + const inDelay = new Promise((resolve) => { waiting = resolve; }); + let gathered = 0; + const contextModule: Module = new EphemeralToolModule(); + contextModule.gatherContext = async () => { + gathered++; + return []; + }; + const framework = await AgentFramework.create({ + storePath, + membrane: new MockMembrane().asMembrane(), + agents: [], + modules: [contextModule], + syncIntervalMs: 0, + errorPolicy: { + maxRetries: 1, + onInferenceError: (_error: Error, _agent: string, attempt: number): ErrorAction => { + waiting(); + // Long enough that only stop() ends the wait. + return attempt < 1 ? { retry: true, delayMs: 60_000 } : { retry: false }; + }, + }, + }); + try { + const { agent, contextManager } = await createEphemeral(framework); + let compiles = 0; + (agent as unknown as { startStreamWithInjections: () => Promise }).startStreamWithInjections = async () => { + compiles++; + throw new Error('boom'); + }; + const run = runStartedEphemeral(framework, agent, contextManager, STOPPED_RUN_WATCHDOGS); + await inDelay; + const stopped = framework.stop(); + await assert.rejects(run, /refused: the framework is stopping/); + await stopped; + assert.equal(compiles, 1, 'the retry was refused, not compiled'); + assert.equal(gathered, 1, 'the retry was refused before its turn started'); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('a run whose stream failed and waits to retry ends at once when stop() begins', { timeout: 60_000 }, async () => { + // The stream's own retry: stop() ends its wait and cancels it there. + const { tempDir, storePath } = tempStorePath('ephemeral-stream-retry-stopping-'); + let waiting!: () => void; + const inDelay = new Promise((resolve) => { waiting = resolve; }); + const membrane = new MockMembrane(); + membrane.streamYielding = (request) => { + membrane.calls.push(request); + // A stream that fails once it has started. + const stream = new MockYieldingStream([]); + (stream as unknown as { events: unknown[] }).events.push({ type: 'error', error: new Error('overloaded') }); + return stream; + }; + const framework = await AgentFramework.create({ + storePath, + membrane: membrane.asMembrane(), + agents: [], + modules: [new EphemeralToolModule()], + syncIntervalMs: 0, + errorPolicy: { + maxRetries: 1, + onInferenceError: (_error: Error, _agent: string, attempt: number): ErrorAction => { + waiting(); + // Long enough that only stop() ends the wait. + return attempt < 1 ? { retry: true, delayMs: 60_000 } : { retry: false }; + }, + }, + }); + try { + const { agent, contextManager } = await createEphemeral(framework); + const run = runStartedEphemeral(framework, agent, contextManager, STOPPED_RUN_WATCHDOGS); + await inDelay; + const stopped = framework.stop(); + await assert.rejects(run, /refused: the framework is stopping/); + await stopped; + assert.equal(membrane.calls.length, 1, 'the retry was not compiled'); + } finally { + rmSync(tempDir, { recursive: true, force: true }); + } + }); + + it('a turn asked for once stop() has begun does not start, and its run ends at once', { timeout: 60_000 }, async () => { + // As a wake the loop dequeued before stop() ended it would be: refused + // before the turn starts. The run, whose request never reached the loop, + // ends then, not when its startup watchdog fires. + const tempDir = mkdtempSync(join(tmpdir(), 'ephemeral-late-turn-')); + // A store the framework doesn't own stays open after stop(), as the run's + // own cleanup needs it to. + const store = JsStore.openOrCreate({ path: join(tempDir, 'store') }); + let gathered = 0; + const contextModule: Module = new EphemeralToolModule(); + contextModule.gatherContext = async () => { + gathered++; + return []; + }; + const framework = await AgentFramework.create({ + store, + membrane: new MockMembrane().asMembrane(), + agents: [], + modules: [contextModule], + syncIntervalMs: 0, + }); + try { + const { agent, contextManager } = await createEphemeral(framework); + // Asked for, but the event loop never started, so its turn waits. + const run = framework.runEphemeralToCompletion(agent, contextManager, STOPPED_RUN_WATCHDOGS); + await framework.stop(); + await (framework as unknown as { startAgentStream(a: typeof agent): Promise }).startAgentStream(agent); + await assert.rejects(run, /refused: the framework is stopping/); + assert.equal(gathered, 0, 'the turn did not start'); + } finally { + store.close(); + rmSync(tempDir, { recursive: true, force: true }); + } + }); + it('resolves plain text completion and deregisters the agent', async () => { const { tempDir, storePath } = tempStorePath('ephemeral-plain-'); const membrane = new MockMembrane(); @@ -194,6 +430,49 @@ describe('runEphemeralToCompletion settle signal', () => { } }); + it('what is queued for a run when it ends is stored in its namespace, not dropped once it is deregistered', async () => { + const { tempDir, storePath } = tempStorePath('ephemeral-queued-'); + const membrane = new MockMembrane(); + membrane.pushResponse(createMockResponse([ + { type: 'tool_use', id: 'call_1', name: 'test--echo', input: {} }, + ], 'tool_use')); + membrane.pushResponse(createMockResponse([{ type: 'text', text: 'Done.' }])); + + const tools = new DuringCallToolModule(); + const framework = await AgentFramework.create({ + storePath, + membrane: membrane.asMembrane(), + agents: [], + modules: [tools], + syncIntervalMs: 0, + }); + // While the run's tool call is pending, two messages for it are deferred: + // one named as the run itself, which waits for its turn's end, and one + // queued behind it. + const internals = framework as unknown as { + addMessage(participant: string, content: Array<{ type: 'text'; text: string }>, metadata?: undefined, opts?: { forAgent?: string }): string; + }; + const returned: string[] = []; + tools.duringCall = () => { + returned.push(internals.addMessage('worker', [{ type: 'text', text: 'my own words' }], undefined, { forAgent: 'worker' })); + returned.push(internals.addMessage('Antra', [{ type: 'text', text: 'look left' }], undefined, { forAgent: 'worker' })); + }; + + try { + const { agent, contextManager } = await createEphemeral(framework, 'worker'); + const result = await runStartedEphemeral(framework, agent, contextManager); + assert.strictEqual(result.speech, 'Done.'); + assert.deepStrictEqual(returned, ['', ''], 'both were deferred: the run was waiting on its tool'); + assert.strictEqual(framework.getAgent(agent.name), null, 'deregistered'); + const stored = contextManager.getAllMessages().map((m) => + `${m.participant}: ${m.content.map((b) => (b.type === 'text' ? (b as { text: string }).text : `<${b.type}>`)).join(' ')}`); + assert.deepStrictEqual(stored.slice(-3), ['worker: Done.', 'worker: my own words', 'Antra: look left'], + 'held to the turn end, and stored after the run\'s last reply'); + } finally { + await stopAndRemove(framework, tempDir); + } + }); + it('resolves through the endTurn tool-result path', async () => { const { tempDir, storePath } = tempStorePath('ephemeral-endturn-'); const membrane = new MockMembrane(); diff --git a/test/helpers/mock-membrane.ts b/test/helpers/mock-membrane.ts index 21774f3d8..96cbc92b5 100644 --- a/test/helpers/mock-membrane.ts +++ b/test/helpers/mock-membrane.ts @@ -56,7 +56,12 @@ export class MockYieldingStream implements YieldingStream { { injectedMessages?: Array<{ participant?: string; content: unknown[] }> } | undefined > = []; - constructor(private responses: NormalizedResponse[]) { + /** + * @param supportsInjectedMessages what each tool-calls event declares, as + * membrane does after 0.5.87 (true on its native path, false on its XML + * prefill path); left out, the event doesn't say, as on 0.5.87. + */ + constructor(private responses: NormalizedResponse[], private supportsInjectedMessages?: boolean) { this.processResponse(0); } @@ -108,6 +113,9 @@ export class MockYieldingStream implements YieldingStream { // provider order — lets the framework store the assistant turn // verbatim and (≥0.5.72) route the round's prose live. roundContent: response.content, + ...(this.supportsInjectedMessages !== undefined + ? { supportsInjectedMessages: this.supportsInjectedMessages } + : {}), }, } as StreamEvent); } else { @@ -167,6 +175,8 @@ export class MockMembrane { responses: NormalizedResponse[] = []; calls: NormalizedRequest[] = []; lastStream: MockYieldingStream | null = null; + /** What the streams' tool-calls events declare (see MockYieldingStream). */ + supportsInjectedMessages: boolean | undefined = undefined; private responseIndex = 0; pushResponse(response: NormalizedResponse): void { @@ -185,7 +195,7 @@ export class MockMembrane { this.calls.push(request); const remaining = this.responses.slice(this.responseIndex); this.responseIndex = this.responses.length; - const stream = new MockYieldingStream(remaining); + const stream = new MockYieldingStream(remaining, this.supportsInjectedMessages); this.lastStream = stream; return stream; } diff --git a/test/host-quiesce.test.ts b/test/host-quiesce.test.ts index 55213e976..74951cd3e 100644 --- a/test/host-quiesce.test.ts +++ b/test/host-quiesce.test.ts @@ -758,7 +758,7 @@ test('a batch carrying a budget restart re-parks its sibling wakes instead of co }); }); -test('resume flushes deferred writes per message: one poison write neither drops the rest nor skips the reopen', async () => { +test('resume flushes deferred writes in order: a poison write stays queued with what follows it, and the reopen still completes', async () => { const membrane = new MockMembrane(); await withFramework(membrane, async (framework) => { const traces: TraceEvent[] = []; @@ -784,16 +784,317 @@ test('resume flushes deferred writes per message: one poison write neither drops const status = await framework.resume(); assert.equal(status.quiesced, false); - assert.equal(status.deferredWrites, 0, 'the queue is drained even though one write failed'); + // A drained write whose store throws is never dropped, and nothing lands + // ahead of it: it and the write queued behind it go back into the queue + // under their ids, for the next flush. + assert.equal(status.deferredWrites, 2, 'the write that failed and the one behind it are still queued'); assert.deepEqual( cm.getAllMessages().slice(before).map((m) => (m.content[0] as { text: string }).text), - ['first', 'third'], - 'the writes behind the poison one still landed', + ['first'], + 'the write queued before the poison one landed', ); assert.ok(traces.some((t) => t.type === 'host:resume'), 'resume completed its follow-through'); }); }); +test('a write that fails at resume goes back into the queue once: the recovery file holds each entry once while the batch waits for its ack', async () => { + const membrane = new MockMembrane(); + await withFramework(membrane, async (framework, dir) => { + await framework.quiesce({ reason: 'refold' }); + const cm = framework.getAgent('agent')!.getContextManager(); + const add = (framework as unknown as { + addMessage(participant: string, content: unknown[]): unknown; + }).addMessage.bind(framework); + add('user', [{ type: 'text', text: 'first' }]); + add('user', [{ type: 'text', text: 'poison' }]); + add('user', [{ type: 'text', text: 'third' }]); + + const realAdd = cm.addMessage.bind(cm); + (cm as unknown as { addMessage: typeof cm.addMessage }).addMessage = ((participant, content, metadata, causedBy) => { + if ((content[0] as { text?: string }).text === 'poison') throw new Error('store write failed'); + return realAdd(participant, content, metadata, causedBy); + }) as typeof cm.addMessage; + // The batch's ack syncs the store before it rewrites the recovery file + // without what landed. While that sync fails, the batch stays in the file + // as it was handed off, beside what went back into the queue. + const store = (framework as unknown as { store: { sync(): void } }).store; + const realSync = store.sync.bind(store); + let syncFails = true; + store.sync = () => { + if (syncFails) throw new Error('sync failed'); + realSync(); + }; + try { + await framework.resume(); + } finally { + syncFails = false; + } + + const file = join(dir, 'store', 'recovery', 'deferred-writes.json'); + const pending = (JSON.parse(readFileSync(file, 'utf8')) as { + pending: Array<{ id: string; content: Array<{ text: string }> }>; + }).pending; + assert.deepEqual( + pending.map((entry) => entry.content[0]!.text), + ['first', 'poison', 'third'], + 'the entry that landed waits for its ack, and the two that went back are in the queue, each once', + ); + assert.equal(new Set(pending.map((entry) => entry.id)).size, 3, 'no id is written twice'); + }); +}); + +test("resume stores several agents' writes in the order they arrived, and cuts each agent at its own stuck write, named at every flush", { timeout: 60_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'host-quiesce-targets-')); + const framework = await AgentFramework.create({ + storePath: join(dir, 'store'), + membrane: new MockMembrane().asMembrane(), + agents: [ + { name: 'agent', model: 'test-model', systemPrompt: 'test' }, + { name: 'scout', model: 'test-model', systemPrompt: 'test' }, + ], + modules: [], + }); + const lines: string[] = []; + const error = console.error; + console.error = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + try { + await framework.quiesce({ reason: 'refold' }); + const internals = framework as unknown as { + deferredMessages: Array<{ id: string; content: Array<{ text?: string }> }>; + addMessage(participant: string, content: unknown[], metadata?: undefined, opts?: { forAgent?: string }): unknown; + }; + const add = (text: string, forAgent: string) => + internals.addMessage('user', [{ type: 'text', text }], undefined, { forAgent }); + // Residents share one message slot, so each one's context manager reads + // what either stored; each stores through its own. + const agentCm = framework.getAgent('agent')!.getContextManager(); + const before = agentCm.getAllMessages().length; + const stored = () => agentCm.getAllMessages().slice(before).map((m) => (m.content[0] as { text: string }).text); + add('one', 'agent'); + add('two', 'scout'); + add('three', 'agent'); + // Nothing checks a fresh write's blocks beyond reading each one's type, + // so a block that isn't an object can be queued; the log line names it + // by its JavaScript type. + internals.addMessage('user', [{ type: 'text', text: 'stuck' }, 'a bare string'], undefined, { forAgent: 'agent' }); + add('for scout', 'scout'); + add('behind the stuck one', 'agent'); + add('also for scout', 'scout'); + assert.equal(framework.getHostModeStatus().deferredWrites, 7); + const stuckId = internals.deferredMessages.find((m) => m.content[0]?.text === 'stuck')!.id; + + // The store refuses agent's 'stuck' write with a thrown value that has no + // text, so the log line can't print it. + const realAdd = agentCm.addMessage.bind(agentCm); + (agentCm as unknown as { addMessage: typeof agentCm.addMessage }).addMessage = ((participant, content, metadata, causedBy) => { + if ((content[0] as { text?: string }).text === 'stuck') throw Object.create(null); + return realAdd(participant, content, metadata, causedBy); + }) as typeof agentCm.addMessage; + const named = () => lines.filter((line) => + line.includes(`deferred write ${stuckId} (user: text+string) for agent could not be stored`) + && line.includes('a thrown value that has no text')).length; + + let status = await framework.resume(); + // In the order they arrived, across both agents; from agent's stuck + // write on, agent's are held, and scout's go on. + assert.deepEqual(stored(), ['one', 'two', 'three', 'for scout', 'also for scout']); + assert.equal(status.deferredWrites, 2, "agent's stuck write and the one behind it are still queued"); + assert.equal(named(), 1, 'the flush names the stuck write: its id and block types'); + + // A second flush with the store still refusing it: held again, named again. + await framework.quiesce({ reason: 'refold' }); + status = await framework.resume(); + assert.deepEqual(stored(), ['one', 'two', 'three', 'for scout', 'also for scout'], 'still nothing lands ahead of it'); + assert.equal(status.deferredWrites, 2); + assert.equal(named(), 2, 'every flush names it'); + } finally { + try { + await framework.stop(); + } finally { + console.error = error; + rmSync(dir, { recursive: true, force: true }); + } + } +}); + +test('boot recovery drops a malformed recovered deferred write, naming it, and restores the rest, after which the agent compiles', { timeout: 60_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'host-quiesce-malformed-')); + const storePath = join(dir, 'store'); + const config = () => ({ + storePath, + membrane: new MockMembrane().asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test' }], + modules: [], + }); + let framework = await AgentFramework.create(config()); + const lines: string[] = []; + const error = console.error; + try { + await framework.quiesce({ reason: 'surgery' }); + const before = framework.getAgent('agent')!.getContextManager().getAllMessages().length; + const add = (framework as unknown as { addMessage(p: string, c: unknown[]): unknown }).addMessage.bind(framework); + add('user', [{ type: 'text', text: 'first' }]); + add('user', [{ type: 'text', text: 'second' }]); + await framework.stop(); + + // Entries no write of the framework's makes, as other code could leave + // them in the recovery file: a null block, a string block and a number + // block beside a text block, content that isn't blocks, a participant + // that isn't text, which the line names by its type, and a target that + // is empty or isn't text. Once stored, a string or number block would + // make every compile for the agent throw, since the context manager's + // token estimate keys a WeakMap on each block. + const file = join(storePath, 'recovery', 'deferred-writes.json'); + const doc = JSON.parse(readFileSync(file, 'utf8')) as { pending: Array> }; + assert.equal(doc.pending.length, 2); + const dropped = ['null-block', 'string-block', 'number-block', 'not-blocks', 'number-participant', 'empty-target', 'number-target']; + doc.pending.splice(1, 0, + { id: 'null-block', seq: 100, participant: 'user', content: [{ type: 'text', text: 'x' }, null] }, + { id: 'string-block', seq: 101, participant: 'user', content: [{ type: 'text', text: 'x' }, 'a bare string'] }, + { id: 'number-block', seq: 102, participant: 'user', content: [{ type: 'text', text: 'x' }, 42] }, + { id: 'not-blocks', seq: 103, participant: 'user', content: 'x' }, + { id: 'number-participant', seq: 104, participant: 42, content: [{ type: 'text', text: 'x' }] }, + { id: 'empty-target', seq: 105, participant: 'user', content: [{ type: 'text', text: 'x' }], forAgent: '' }, + { id: 'number-target', seq: 106, participant: 'user', content: [{ type: 'text', text: 'x' }], forAgent: 7 }, + ); + writeFileSync(file, JSON.stringify(doc)); + + console.error = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + framework = await AgentFramework.create(config()); + console.error = error; + assert.equal(framework.getHostModeStatus().deferredWrites, 2, 'the well-formed entries are restored'); + for (const entry of ['null-block (user)', 'string-block (user)', 'number-block (user)', 'not-blocks (user)', 'number-participant (participant of type number)', 'empty-target (user)', 'number-target (user)']) { + assert.ok(lines.some((line) => line.includes(`malformed and was dropped: ${entry}`)), `${entry} is named:\n${lines.join('\n')}`); + } + const rewritten = readFileSync(file, 'utf8'); + assert.ok(!dropped.some((id) => rewritten.includes(id)), 'the recovery file is rewritten without them'); + await framework.resume(); + const texts = framework.getAgent('agent')!.getContextManager().getAllMessages().slice(before) + .map((m) => (m.content[0] as { text: string }).text); + assert.deepEqual(texts, ['first', 'second']); + await assert.doesNotReject(framework.getAgent('agent')!.compileContext(), 'and the agent compiles'); + } finally { + console.error = error; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } +}); + +test('boot recovery keeps one entry per id, and numbers a seq that isn\'t a positive whole number after the others', { timeout: 60_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'host-quiesce-identity-')); + const storePath = join(dir, 'store'); + const config = () => ({ + storePath, + membrane: new MockMembrane().asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test' }], + modules: [], + }); + let framework = await AgentFramework.create(config()); + const lines: string[] = []; + const error = console.error; + try { + await framework.quiesce({ reason: 'surgery' }); + const before = framework.getAgent('agent')!.getContextManager().getAllMessages().length; + const add = (framework as unknown as { addMessage(p: string, c: unknown[]): unknown }).addMessage.bind(framework); + add('user', [{ type: 'text', text: 'first' }]); + add('user', [{ type: 'text', text: 'second' }]); + await framework.stop(); + + // What the framework never writes, as damage or a hand edit could leave + // it: an id that comes twice, an entry with no id, a seq of 1e400 (JSON + // reads it as Infinity), a negative one, zero and a string. The entries + // with a bad seq come before 'second' in the file. Nothing here is + // malformed, so only these make boot rewrite it. + const file = join(storePath, 'recovery', 'deferred-writes.json'); + const doc = JSON.parse(readFileSync(file, 'utf8')) as { pending: Array> }; + assert.equal(doc.pending.length, 2); + const entry = (id: string | undefined, seq: unknown, text: string) => + ({ ...(id === undefined ? {} : { id }), seq, participant: 'user', content: [{ type: 'text', text }] }); + doc.pending.splice(1, 0, + entry('infinite-seq', '__INFINITY__', 'infinite seq'), + entry('negative-seq', -3, 'negative seq'), + entry('twice', 50, 'first with its id'), + entry('twice', 51, 'second with its id'), + entry('zero-seq', 0, 'zero seq'), + entry('string-seq', 'seven', 'string seq'), + entry(undefined, 52, 'no id'), + ); + writeFileSync(file, JSON.stringify(doc).replace('"__INFINITY__"', '1e400')); + + console.error = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + framework = await AgentFramework.create(config()); + console.error = error; + const named = (text: string) => assert.ok(lines.some((line) => line.includes(text)), `${text}:\n${lines.join('\n')}`); + const late = 'not a positive whole number up to Number.MAX_SAFE_INTEGER; it is queued after the others'; + named('repeats id twice (user) and was dropped; the first entry with that id is kept'); + named(`(infinite-seq) has seq Infinity, ${late}`); + named(`(negative-seq) has seq -3, ${late}`); + named(`(zero-seq) has seq 0, ${late}`); + named(`(string-seq) has seq of type string, ${late}`); + named('a recovered deferred context write (user) has no id; it is given '); + assert.equal(framework.getHostModeStatus().deferredWrites, 8); + + const rewritten = JSON.parse(readFileSync(file, 'utf8')) as { pending: Array<{ id: unknown; seq: unknown }> }; + assert.equal(rewritten.pending.length, 8, 'the recovery file is rewritten without the repeated id'); + assert.ok(rewritten.pending.every((m) => Number.isSafeInteger(m.seq) && (m.seq as number) > 0), JSON.stringify(rewritten.pending)); + + await framework.resume(); + const texts = framework.getAgent('agent')!.getContextManager().getAllMessages().slice(before) + .map((m) => (m.content[0] as { text: string }).text); + assert.deepEqual(texts, ['first', 'second', 'first with its id', 'no id', 'infinite seq', 'negative seq', 'zero seq', 'string seq']); + } finally { + console.error = error; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } +}); + +test('boot recovery restores a recovery file that predates seq in file order, and rewrites it with seqs', { timeout: 60_000 }, async () => { + const dir = mkdtempSync(join(tmpdir(), 'host-quiesce-noseq-')); + const storePath = join(dir, 'store'); + const config = () => ({ + storePath, + membrane: new MockMembrane().asMembrane(), + agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test' }], + modules: [], + }); + let framework = await AgentFramework.create(config()); + const lines: string[] = []; + const error = console.error; + try { + await framework.quiesce({ reason: 'surgery' }); + const before = framework.getAgent('agent')!.getContextManager().getAllMessages().length; + const add = (framework as unknown as { addMessage(p: string, c: unknown[]): unknown }).addMessage.bind(framework); + for (const text of ['first', 'second', 'third']) add('user', [{ type: 'text', text }]); + await framework.stop(); + + // A file written before entries carried a seq is in write order. Here + // that order is third, first, second, so only the file can give it. + const file = join(storePath, 'recovery', 'deferred-writes.json'); + const doc = JSON.parse(readFileSync(file, 'utf8')) as { pending: Array> }; + assert.equal(doc.pending.length, 3); + const [first, second, third] = doc.pending.map(({ seq: _seq, ...rest }) => rest); + writeFileSync(file, JSON.stringify({ version: 1, pending: [third, first, second] })); + + console.error = (...args: unknown[]) => { lines.push(args.map(String).join(' ')); }; + framework = await AgentFramework.create(config()); + console.error = error; + assert.ok(!lines.some((line) => line.includes('has seq')), `an entry with no seq isn't named:\n${lines.join('\n')}`); + const rewritten = JSON.parse(readFileSync(file, 'utf8')) as { pending: Array<{ seq: unknown; content: Array<{ text: string }> }> }; + assert.deepEqual(rewritten.pending.map((m) => [m.content[0]!.text, m.seq]), [['third', 1], ['first', 2], ['second', 3]], + 'rewritten with seqs, in file order'); + + await framework.resume(); + const texts = framework.getAgent('agent')!.getContextManager().getAllMessages().slice(before) + .map((m) => (m.content[0] as { text: string }).text); + assert.deepEqual(texts, ['third', 'first', 'second']); + } finally { + console.error = error; + await framework.stop(); + rmSync(dir, { recursive: true, force: true }); + } +}); + test('context writes deferred while quiesced survive a restart and land at resume', async () => { const dir = mkdtempSync(join(tmpdir(), 'host-quiesce-deferred-')); const storePath = join(dir, 'store'); diff --git a/test/operator-surgery.test.ts b/test/operator-surgery.test.ts index 46bacc245..59516e60d 100644 --- a/test/operator-surgery.test.ts +++ b/test/operator-surgery.test.ts @@ -228,6 +228,105 @@ describe('live operator surgery', () => { } }); + it('a write deferred behind the surgery lands with its durable id, as every flush from the queue does', async () => { + const c = cm() as unknown as { switchBranch: (name: string) => Promise }; + const originalSwitch = c.switchBranch; + let releaseSwitch!: () => void; + const gate = new Promise((resolve) => { releaseSwitch = resolve; }); + c.switchBranch = async (name: string) => { await gate; return originalSwitch.call(c, name); }; + const internals = framework as unknown as { + surgeryHold: unknown; + deferredMessages: Array<{ id: string }>; + addMessage(participant: string, content: unknown[], metadata?: Record): string; + }; + try { + const rollback = framework.rollbackToMessage('scout', { messageId: ids[2] }); + await new Promise((resolve) => setImmediate(resolve)); + assert.ok(internals.surgeryHold, 'store hold is up while the switch is awaited'); + assert.equal(internals.addMessage('user', [{ type: 'text', text: 'arrived mid-surgery' }]), '', 'deferred'); + const queuedId = internals.deferredMessages.at(-1)!.id; + releaseSwitch(); + await rollback; + const landed = cm().getAllMessages().find((m) => (m.content[0] as { text?: string }).text === 'arrived mid-surgery'); + assert.ok(landed, 'flushed when the hold released'); + // The id makes a queued pair's tool_result re-defer with its tool_use + // rather than land alone, and lets boot recovery skip a replay. + assert.equal((landed!.metadata as { deferredWriteId?: string } | undefined)?.deferredWriteId, queuedId); + } finally { + c.switchBranch = originalSwitch; + } + }); + + it("the post-surgery flush lands what waited behind the hold while another agent's tool cycle is pending, and acknowledges it", { timeout: 60_000 }, async () => { + const c = cm() as unknown as { switchBranch: (name: string) => Promise }; + const originalSwitch = c.switchBranch; + let releaseSwitch!: () => void; + const gate = new Promise((resolve) => { releaseSwitch = resolve; }); + let entered!: () => void; + const inSwitch = new Promise((resolve) => { entered = resolve; }); + c.switchBranch = async (name: string) => { entered(); await gate; return originalSwitch.call(c, name); }; + const internals = framework as unknown as { + surgeryHold: unknown; + deferredWritesPersisted: boolean; + unackedDeferredWrites: unknown[]; + pendingAssistantBlocks: Map; + addMessage(participant: string, content: unknown[], metadata?: Record): string; + }; + internals.deferredWritesPersisted = true; + try { + const rollback = framework.rollbackToMessage('scout', { messageId: ids[2] }); + await inSwitch; + assert.ok(internals.surgeryHold, 'store hold is up while the switch is awaited'); + assert.equal(internals.addMessage('user', [{ type: 'text', text: 'waited for the hold' }]), '', 'deferred'); + // Another agent's tool cycle says nothing about scout's window. + internals.pendingAssistantBlocks.set('builder', [{ type: 'tool_use', id: 'b1', name: 'x', input: {} }]); + releaseSwitch(); + await rollback; + assert.ok(remainingTexts().includes('waited for the hold'), 'flushed when the hold released'); + assert.equal(internals.unackedDeferredWrites.length, 0, 'and acknowledged, so the recovery queue no longer holds it'); + } finally { + internals.pendingAssistantBlocks.delete('builder'); + c.switchBranch = originalSwitch; + } + }); + + it("the post-surgery flush stores several agents' writes in the order they arrived", { timeout: 60_000 }, async () => { + // Residents share one message slot, so the order writes arrived in is the + // order every resident reads. + await framework.stop(); + framework = await AgentFramework.create({ + storePath, + membrane: membrane.asMembrane(), + agents: [ + { name: 'scout', model: 'test-model', systemPrompt: 'You are scout.' }, + { name: 'builder', model: 'test-model', systemPrompt: 'You are builder.' }, + ], + modules: [], + }); + const c = cm() as unknown as { switchBranch: (name: string) => Promise }; + const originalSwitch = c.switchBranch; + let releaseSwitch!: () => void; + const gate = new Promise((resolve) => { releaseSwitch = resolve; }); + let entered!: () => void; + const inSwitch = new Promise((resolve) => { entered = resolve; }); + c.switchBranch = async (name: string) => { entered(); await gate; return originalSwitch.call(c, name); }; + const add = (framework as unknown as { + addMessage(participant: string, content: unknown[], metadata?: undefined, opts?: { forAgent?: string }): string; + }).addMessage.bind(framework); + try { + const rollback = framework.rollbackToMessage('scout', { messageId: ids[2] }); + await inSwitch; + assert.equal(add('user', [{ type: 'text', text: 'one' }]), '', 'deferred'); + assert.equal(add('user', [{ type: 'text', text: 'two' }], undefined, { forAgent: 'builder' }), '', 'deferred'); + assert.equal(add('user', [{ type: 'text', text: 'three' }]), '', 'deferred'); + releaseSwitch(); + await rollback; + assert.deepEqual(remainingTexts().filter((t) => ['one', 'two', 'three'].includes(t)), ['one', 'two', 'three']); + } finally { + c.switchBranch = originalSwitch; + } + }); + // --- strategy-initialization failure ----------------------------------------- // switchBranch()/fork() move the chronicle branch BEFORE awaiting strategy // initialization, so a rejection there leaves the store on the new branch. diff --git a/test/present-while-acting.test.ts b/test/present-while-acting.test.ts index 0d965d9e6..df37b7484 100644 --- a/test/present-while-acting.test.ts +++ b/test/present-while-acting.test.ts @@ -931,13 +931,19 @@ describe('present while acting', () => { deferredMessages: unknown[]; }; let deferredInStart = -1; + // Another agent's tool cycle is pending when the start fails: the flush + // at its end waits only on this agent's own cycle. + const cycles = (framework as unknown as { pendingAssistantBlocks: Map }).pendingAssistantBlocks; module.startHook = async () => { store.addMessage('Antra', [{ type: 'text', text: 'quick question for you' }] as ContentBlock[], addressed('discord:dm:antra')); deferredInStart = store.deferredMessages.length; + cycles.set('builder', [{ type: 'tool_use', id: 'b1', name: 'x', input: {} }] as ContentBlock[]); }; await release(framework); + cycles.delete('builder'); assert.equal(deferredInStart, 1, 'the DM was deferred while the start held the turn'); assert.equal(calls, 2, 'the second start failed before its stream'); + assert.equal(store.deferredMessages.length, 0, "the failed start's flush stored the DM while another agent's tools were pending"); // During the second hold, someone else addresses the agent. await say(framework, 'over here', addressed('discord:dm:q')); await release(framework); @@ -950,6 +956,80 @@ describe('present while acting', () => { await framework.stop(); }); + it("a failed start's flush stops at a message it can't store: it and what follows stay queued, in order", async () => { + membrane.pushResponse(createMockResponse([ + { type: 'tool_use', id: 'c1', name: 'robot--move', input: { dir: 'north' } }, + ] as ContentBlock[], 'tool_use')); + membrane.pushResponse(createMockResponse([] as ContentBlock[])); + + const framework = await createFramework(); + stubChannelRegistry(framework); + module.toolDelayMs = 25; + let hostHolds = 0; + (framework as unknown as { providerHoldHook: (error: Error) => { holdMs: number; reason: string } | undefined }) + .providerHoldHook = (error) => error.message === 'stream setup failed' && hostHolds++ === 0 + ? { holdMs: 60_000, reason: 'host hold' } : undefined; + // As above: stream one is held at its end, the start after the first + // release fails before its stream exists and is held, the third answers. + const makeStream = membrane.streamYielding.bind(membrane); + let calls = 0; + membrane.streamYielding = (request, options) => { + const n = ++calls; + if (n === 2) throw new Error('stream setup failed'); + const stream = makeStream(request, options); + const iterate = stream[Symbol.asyncIterator].bind(stream); + stream[Symbol.asyncIterator] = async function* () { + for await (const event of { [Symbol.asyncIterator]: iterate }) { + if (event.type === 'complete' && n === 1) { + yield { type: 'error' as const, error: accelerationLimit(request) }; + return; + } + yield event; + } + }; + return stream; + }; + feedStart(framework, 2, [{ type: 'text', text: 'Back now.' }] as ContentBlock[]); + + trigger(framework); + await framework.runUntilIdle(); + // During the second start, before its stream exists, two messages arrive + // and are deferred, and the store refuses the first of them. + const store = framework as unknown as { + addMessage(participant: string, content: ContentBlock[], metadata: Record): string; + deferredMessages: Array<{ content: ContentBlock[] }>; + }; + const cm = framework.getAgent('assistant')!.getContextManager() as unknown as { + addMessage(participant: string, content: ContentBlock[], metadata?: unknown): string; + }; + const add = cm.addMessage.bind(cm); + let refuse = true; + cm.addMessage = (participant, content, metadata) => { + if (refuse && (content[0] as { text?: string })?.text === 'first') throw new Error('store refused'); + return add(participant, content, metadata); + }; + module.startHook = async () => { + store.addMessage('Antra', [{ type: 'text', text: 'first' }] as ContentBlock[], ambient('discord:dm:antra')); + store.addMessage('Antra', [{ type: 'text', text: 'second' }] as ContentBlock[], ambient('discord:dm:antra')); + }; + const texts = () => framework.getAgent('assistant')!.getContextManager().getAllMessages() + .flatMap((m) => m.content.filter((b) => b.type === 'text').map((b) => (b as { text: string }).text)) + .filter((t) => t === 'first' || t === 'second'); + await release(framework); + assert.equal(calls, 2, 'the second start failed before its stream'); + assert.deepEqual(store.deferredMessages.map((m) => (m.content[0] as { text: string }).text), ['first', 'second'], + 'the message the flush could not store, and the one behind it, stay queued in order'); + assert.deepEqual(texts(), [], 'nothing behind the refused message was stored'); + + // Once the store takes it, the next start stores both, in order. + refuse = false; + await release(framework); + assert.equal(calls, 3, 'the third start compiled'); + assert.deepEqual(texts(), ['first', 'second']); + + await framework.stop(); + }); + for (const [label, withdrawn, expected] of [ ['re-pins to an addressed message its over-budget boundary left unweighed', false, 'discord:dm:antra'], ['moves nothing for such a message withdrawn before it starts', true, 'chan-live-1'], @@ -1285,63 +1365,6 @@ describe('present while acting', () => { await framework.stop(); }); - it("a failed start's flush stores each deferred message on its own, so one failure loses no other", async () => { - membrane.pushResponse(createMockResponse([ - { type: 'tool_use', id: 'c1', name: 'robot--move', input: { dir: 'north' } }, - ] as ContentBlock[], 'tool_use')); - membrane.pushResponse(createMockResponse([] as ContentBlock[])); - const framework = await createFramework(); - stubChannelRegistry(framework); - module.toolDelayMs = 25; - let hostHolds = 0; - (framework as unknown as { providerHoldHook: (error: Error) => { holdMs: number; reason: string } | undefined }) - .providerHoldHook = (error) => error.message === 'stream setup failed' && hostHolds++ === 0 - ? { holdMs: 60_000, reason: 'host hold' } : undefined; - const makeStream = membrane.streamYielding.bind(membrane); - let calls = 0; - membrane.streamYielding = (request, options) => { - const n = ++calls; - if (n === 2) throw new Error('stream setup failed'); - const stream = makeStream(request, options); - const iterate = stream[Symbol.asyncIterator].bind(stream); - stream[Symbol.asyncIterator] = async function* () { - for await (const event of { [Symbol.asyncIterator]: iterate }) { - if (event.type === 'complete' && n === 1) { - yield { type: 'error' as const, error: accelerationLimit(request) }; - return; - } - yield event; - } - }; - return stream; - }; - const store = framework as unknown as { - addMessage(participant: string, content: ContentBlock[], metadata: Record, opts?: { deferredWriteId?: string }): string; - }; - const add = store.addMessage.bind(framework); - // The store refuses the first message when the failed start's flush lands it. - store.addMessage = (participant, content, metadata, opts) => { - if (opts?.deferredWriteId && (content[0] as { text?: string }).text === 'poison') throw new Error('store refused'); - return add(participant, content, metadata, opts); - }; - - trigger(framework); - await framework.runUntilIdle(); - module.startHook = async () => { - store.addMessage('Antra', [{ type: 'text', text: 'poison' }] as ContentBlock[], {}); - store.addMessage('Antra', [{ type: 'text', text: 'second' }] as ContentBlock[], {}); - }; - await release(framework); - - assert.equal(calls, 2, 'the second start failed before its stream'); - const texts = framework.getAgent('assistant')!.getContextManager().getAllMessages() - .flatMap((m) => m.content.filter((b) => b.type === 'text').map((b) => (b as { text: string }).text)); - assert.ok(texts.includes('second'), 'the message behind the refused one was stored'); - assert.ok(!texts.includes('poison')); - - await framework.stop(); - }); - for (const [label, notice] of [ ['a re-pin', 'The conversation moved to'], ["a turn's announcement", 'Your plain speech now lands in'],