Repository navigation
fix(trace): a dropped turn ends with inference:aborted, and Discord typing stops at a normal end - #338
Conversation
beginAgentTurn emits inference:started before the compile. When the compile returns to an agent that was replaced or removed, or a framework that is stopping or was stopped and started again, the turn is dropped before it sends anything, and returned with no terminal event, against onTrace's contract that the terminal events fire once per turn. A module bracketing a turn between inference:started and a terminal event (the Discord module's typing loop, connectome-host's activity module) kept the dropped turn open. The drop now emits inference:aborted, with the reason the console line gives and the turn's duration from its start. The test of a compile from before a stop and restart records the agent's inference:* events and checks the dropped turn gets exactly one terminal event.
…rted too The check ahead of the compile (an agent replaced or removed while the modules' gatherContext and the MCPL beforeInference hooks awaited, or a framework that began stopping) also returned after inference:started with no terminal event, and without a console line. It now logs and emits inference:aborted as the drop after the compile does, with its reason. Two tests hold a turn in gatherContext, then stop the framework or remove the agent, and check its one terminal event and its reason. The fragment now covers both drops.
…s name alone The review of the drop-terminal change found that a turn dropped after a stop and start, or after its agent object was replaced, can reach its drop while a newer turn of the same name is running: the new lifecycle's, or the replacement's. inference:aborted and lastInference carry only the agent's name, so the dropped turn's event would end that newer turn's bracket: the Discord module's typing loop, keyed by name, would stop while it runs. Both drops now end their turn through abortDroppedTurn, which does nothing when another turn holds the name's turn token, and otherwise emits inference:aborted and records lastInference.endedAt, as driveStream records it, so the health snapshot no longer shows a dropped turn as running. The reason is the logged line's text without its prefix, built once. - inference:aborted's reason and durationMs get doc comments: reason now has three sources (an operator's abort, 'quiesce_abandoned', a dropped turn's sentence), and durationMs runs from this start of the turn. - The web UI in ui/ returns an agent to idle on the four terminal events onTrace's contract names, not only inference:completed and failed. - The drop tests cover both checks (before the compile and after it) and four ways each: the framework stopping, the agent removed, the agent object replaced, and a newer turn holding the name's token. The first three check the exact reason, durationMs against the time held, and lastInference.endedAt; the last checks that nothing is emitted or recorded. - The changelog fragment says what is recorded, the newer-turn rule, and the UI.
A turn's normal end is inference:completed: driveStream emits it once, when the turn's stream completes, after agent.reset(). Tool-call cycles stay inside the yielding stream (provideToolResults, inference:stream_resumed), so it does not fire between tool rounds. inference:turn_ended fires only when a tool result ends the turn (shouldEndTurn). onTrace's documentation named four terminal events (turn_ended, aborted, failed, exhausted) and left the normal end out, and the Discord module's TURN_TERMINAL_TRACES followed it. With autoTyping on, its default, the keepalive therefore went on refreshing every TYPING_REFRESH_MS after an ordinary reply, until the agent's next inference:started or TYPING_MAX_MS. - TURN_TERMINAL_TRACES includes inference:completed. - onTrace's doc names inference:completed as the normal end, and says what a bracket meets besides: a context-budget restart emits inference:started again with no terminal event for the stream it replaced, and a failing turn can emit inference:failed and then inference:exhausted. - The web UI's comment in ui/src/App.vue groups inference:completed with the other terminal events (no behaviour change: it already idled on it). - test/discord-typing-keepalive.test.ts: for each of the five terminal events, a turn's keepalive shows typing at inference:started, refreshes once while the turn runs (mocked setInterval), and refreshes no more after the event. With inference:completed removed from the set, exactly the inference:completed case fails. - Fragments: a new fixed fragment for the Discord module and the doc; the drop-terminal fragment's UI sentence now says which events the UI added.
|
@antra-tess @Anarchid Could you review this? It closes two gaps in how a turn's bracket ends. A turn the framework drops before it sends anything now ends with We'll merge it not before 2026-10-15 12:00 UTC unless you'd like changes. #333 also changes the two drop sites, and is due to merge first, so this branch will take — Ada-1017 |
|
| const owner = this.activeTurnTokens.get(agentName); | ||
| if (owner !== undefined && owner !== turnToken) return; | ||
| const now = Date.now(); | ||
| this.lastInferenceAt.set(agentName, { ...this.lastInferenceAt.get(agentName), endedAt: now }); |
There was a problem hiding this comment.
Disposed agents regain records
When an ephemeral agent's idle watchdog expires during gatherContext or compilation, runEphemeralToCompletion removes its lastInferenceAt entry and turn token. If the old wait finishes later, this write restores an orphan { endedAt } entry because the missing token passes the guard. No later cleanup removes it, so long-running hosts retain records for those timed-out subagents. Keep the terminal event, but only update a record that still belongs to this turn.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/framework.ts
Line: 11149
Comment:
**Disposed agents regain records**
When an ephemeral agent's idle watchdog expires during `gatherContext` or compilation, `runEphemeralToCompletion` removes its `lastInferenceAt` entry and turn token. If the old wait finishes later, this write restores an orphan `{ endedAt }` entry because the missing token passes the guard. No later cleanup removes it, so long-running hosts retain records for those timed-out subagents. Keep the terminal event, but only update a record that still belongs to this turn.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Addressed in 9d81e5d. The drop now ends the lastInference record only while it's still this turn's, the one beginAgentTurn stamped with this turn's startedAt, so a name whose ephemeral run was disposed meanwhile gets no orphan { endedAt } back. The terminal event still goes out. The dropped-turn tests gain a disposed-run case at both checks, and with the guard taken out, those two fail and the others pass.
— Ada-1017
Greptile's P2 on the PR: abortDroppedTurn wrote {...existing, endedAt}
whenever no other turn held the name's token. When an ephemeral run is
disposed while its turn waits (runEphemeralToCompletion deletes the name's
record and token), the drop then gave the disposed name an orphan
{ endedAt } record that nothing removes.
The drop now ends the record only when it is still this turn's, the one
beginAgentTurn stamped with this turn's startedAt; the terminal event still
goes out, so a listener's bracket closes. The dropped-turn tests gain a
'disposed' case at both checks (agent, record and token gone, as the
disposal does): inference:aborted, and no record for the name. With the
guard taken out, both disposed cases fail and the other eight pass.
main's drop before the compile now ends the dropped turn's routing (endDroppedTurn) where this branch names why the turn was dropped and logs it. Resolved by keeping both, in that order: the reason and its log line, then endDroppedTurn, then this branch's stopTyping, onInferenceEnded and abortDroppedTurn with the reason. The drop after the compile merged on its own with the same order. Checked: tsc and the build are clean; tool-result-guard 124/124, discord-typing-keepalive 5/5, trunk-channel-routing 20/20, explicit-prose-routing 64/64, present-while-acting 57/57 and physical-window-restart 12/12.
|
Pushed — Ada-1017 |
…pawn wait and transport listeners, and TypeScript 7 main since cd036e0 brings anima-research#329, anima-research#333, anima-research#273, anima-research#368, anima-research#343, anima-research#340, anima-research#365, anima-research#341, anima-research#352, anima-research#338, anima-research#274 (TypeScript 7), anima-research#347, anima-research#346, anima-research#348 and anima-research#344 (no hang after a signal kill; no uncaught transport or spawn errors). Merged, not rebased, so the commits slimepriestess and the round's readers read stay as they were (Ada-1017, room-203 #105340). `git show --remerge-diff` shows the resolution alone. Conflicts, three files: - src/mcpl/transport.ts: - anima-research#344's listeners for a read error on the child's stdout or stderr go in beside the stdin listener this branch already had. - close() stays this branch's reap (SIGTERM, then SIGKILL, then an explicit failure; one shared attempt). anima-research#344's version waited for an exit that kill() made likely, and its change there, counting a signalCode as an exit (anima-research#331), the reap already does. - src/mcpl/server-connection.ts: - handshake keeps this branch's signature (the abort signal and the negotiated protocolVersion) and its abortable open, and takes anima-research#344's logAfterSettled. That listener takes over 'error' once the initialize race settles, comes off just before a successful return, and stays on a failed handshake's transport. - The race's cleanup keeps both lines: this branch's removal of the abort listener and anima-research#344's logAfterSettled. - The reconnect attempt keeps this branch's reconnectOnce, which reaps the previous launch before another starts and closes a handshaken transport itself when close() has ended the attempt. anima-research#344's version kept that transport for close() with a logging listener. That intent carries over as a listener put on before the attempt's own close, since the transport isn't wired and the handshake's listeners are off. - package.json: devDependencies take both @babel/parser (anima-research#274) and @modelcontextprotocol/server (this branch). Beyond the conflicts, for main's async StdioTransport.spawn (anima-research#344): - SpawnerStdioTransport.start() awaits it. So a modern stdio command that can't be spawned rejects start() with its spawn error, and the connect fails with that error instead of waiting out its 30 s discover probe. This is R1-4's fix (Anarchid's review, room-364 #105270), and its test follows this merge. - The tests that wrap spawn to make a child unreapable await it, in mcp-modern-outcomes, mcp-server-ownership and mcpl-legacy-lifetime. So does the unreapable-close test's own spawn. tsc can't see the old synchronous wrappers, since they reach the child through a cast, and they hung those suites. - anima-research#344's test of a transport kept for close() during a reconnect now follows this branch's reconnectOnce. The attempt closes that transport itself and close() waits for it, so the test resolves the held close before it awaits the attempt. It's retitled for what it checks: the transport logs the errors it reports while it closes, and after. tsc (TypeScript 7) is clean. A trial of this resolution on main bdc5dc4 with anima-research#344's head 8387393, the head that merged, passed every test file on Node 24.21.0 and Node 20.20.2 (J10968). The only failures were 69 in a stale dist/test/mcp-modern-http-2025.test.js that the VPS copy kept from anima-research#337's runs. The full check of this commit follows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…research#286's branch anima-research#338's dropped-turn ending conflicted with this branch at both of startAgentStream's dropped-turn exits, in three hunks. Each exit keeps both sides: anima-research#338's abortDroppedTurn with its reason, then this branch's end of a waiting ephemeral run when stop() refused the turn. At the post-compile exit, the reason uses this branch's `stopping`, read once before the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ada-1017 asked for main to come in now rather than after anima-research#282, which lands behind the 0.22.2 release hold, so the PR's next external review reads a current base (room-203 #112135). When anima-research#282 lands, anima-research#360 takes main again. 87 commits since 92125b0, among them anima-research#338, anima-research#274 (TypeScript 7), anima-research#347, anima-research#352 and anima-research#344. The merge is clean: git merge-tree reports no conflict. The merged tree has no reader of ProseStreamRouter, roundDestination or TurnRouting.destination, and main adds no prose delivery site: framework.ts calls routeSpeech eight times on both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main took anima-research#338 (a dropped turn ends with inference:aborted), anima-research#274 (TypeScript 7), anima-research#347, anima-research#346, anima-research#348 and anima-research#344 since 22bb6ca. The merge has no conflicts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#286's round merged main at 91e70ed, which brought #338's abortDroppedTurn to beginAgentTurn's two dropped-turn exits, beside #286's per-site endEphemeralRunRefused calls. This branch removes those per-site ends, since stop()'s sweep ends every ephemeral run still registered once the event loop has ended. At both exits, the resolution keeps abortDroppedTurn, with its reason and the turn's start time, and drops the endEphemeralRunRefused call. In stop-with-work-in-flight's imports, #274 replaced typescript's parser with @babel/parser, and this branch added chronicle's JsStore. Both stay. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… it waited stop() ends a retry's wait, which cancels the retry. The turn's attempt had emitted inference:failed, which a retry usually follows, and then nothing more came: no event said the turn had ended, and when beginAgentTurn was the one waiting (a stream that couldn't start, #334's exit), lastInference kept the turn's start with no end. #338's two drops in the same function end their turns with abortDroppedTurn, and this exit now does too: inference:aborted with the reason "a retry waiting while the framework was stopping is dropped before it sends anything", and lastInference's end, unless another turn of the name holds the token. driveStream waits for its own retry when a stream fails once started. Its finally already ended the turn's record and gate entry, so only the event was missing, and it now emits the same one with the same reason (RETRY_CANCELLED_BY_STOP), under the same token check: to a listener, a retry stop() cancelled is one case whichever frame was waiting. The onTrace contract already allows a second end after inference:failed, since a failing turn can emit inference:exhausted after it. Tests (stop-with-work-in-flight): - The #334 test now also checks the turn's events, inference:started, inference:failed, then inference:aborted with that reason, and that healthSnapshot()'s lastInference has its end. - New beside it: the same for a stream that fails once started, whose retry driveStream waits for. Its lastInference end was already there. - New: when another turn holds the agent's turn token, that stream's cancelled retry emits nothing after inference:failed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… run at once main moved to agent-framework 0.22.2 with #271. This brings the host's code, comments and docs to that release; what the host installs doesn't change. - The subagent module reads a provider's stated wait the framework won't sleep only from 0.22.2's ProviderWaitNotSleptError, by its name, which a second copy of the package gives its own class too. The fallback that matched 0.22.1's two message endings goes, since 0.22.2 rejects every such run with that error. - A spawn or fork whose run ends over such a wait fails at once. It used to take the rate-limit branch first when the provider's words named a rate limit: it halved the subagents' shared concurrency, throttled new runs and waited out the 30 s cooldown, all for a run that is never tried again. - Comments that named 0.22.1's behaviour name 0.22.2's, each checked at v0.22.2, where the code they describe is unchanged: readFileFromDisk, which still reads to its end a file that grows during the read, and opens and waits on a FIFO; writeBinary and its path parser; the workspace's running state, still private; Agent.cancelStream's states; the workspace read's lstat before it resolves the root; the startup watchdog's words; driveStream's check that the agent still owns its name; and that it sends no hard-down-clear. The overBudget branch's cancel, which stopRun follows, is now cited at its 0.22.2 lines. - The agent tree's reducer names the turns 0.22.2 now ends with inference:aborted: one the framework drops after its inference:started because its agent was replaced or removed, or the framework was stopping or had been stopped and started again, unless another turn of the agent's name has begun (anima-research/agent-framework#338). - docs/LIBRARY-PIPELINE.md: a watched mount's catch-up scan at startup catches files written while an agent was offline only within agent-framework's walk cap, 5000 files, links and other entries that aren't directories. On a larger mount, such a change waits until the syncs' rotation reaches it. The paragraph no longer names a syncFromFs, which neither package has. - A changelog fragment adds what operators may notice from 0.22.2 beside #271's: a mount over 5000 files catching up after the upgrade, coalesced MCPL messages with inline images admitted, and the way back to an earlier host release on 0.22.1. Tests: test/subagent-provider-wait.test.ts runs subagents whose runs reject with agent-framework's own ProviderWaitNotSleptError, its words built by the installed framework for a long wait, one too long to represent and one that isn't a number, and checks one run, the framework's reason and no throttle in each; a plain rate-limit error is the control that throttles and is tried again. Checked on Linux with Node 24.22.0, npm 11.19.1 and Bun 1.4.3 at this commit's tree, with Bun's transpiler cache off: npm ci installs agent-framework 0.22.2, context-manager 0.14.1 and membrane 0.5.87; root tsc is clean, and the Web UI's tsc reports only the error main has at App.tsx:948; the strict Web UI build passes; the five test files whose text or tests change pass 131 of 131 in three runs; bun test passes all 1928 tests across 157 files. Reading the wait other than by the class's name fails eight tests, and leaving out the early end, in spawn and fork, fails five.
Four conflicts, one of them a fragment main has released. - agent.ts, the stream's request build: anima-research#281 wrapped it in toolResultGuard.afterAnnotations, and this branch renamed it assembleActivationRequest, which returns the request with its continuePrompt. The wrap now takes the renamed assembler and reads both. anima-research#281's refusal retry reassembles through runInference's own compile (compileWithInjections), so it calls neither assembler and needs nothing. - framework.ts, before the compile: anima-research#338 added the admission check and the dropped turn's terminal event to the ownership check, and this branch removed the ephemeralSystemPrompt injection just above it. Both stand. - framework.ts, after the compile: anima-research#251 added a provider-wait check for conversation agents, and this branch records the stream's [Continue] prompt there. The wait check comes first, so a prompt is recorded only for a stream that will be driven. - framework.ts, the non-streaming fallback: anima-research#312's note that it continues the stream's logical turn and this branch's note that it carries the turn's properties are one comment now. - changelog.d/silent-heartbeat-turn-rows.fixed.md went out in 0.22.0, after this branch had edited it. What the edit said beyond this branch's own fragments, that a tick is told apart by its server and its event id together, is in request-only-rows-stay.fixed.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'s own assembler anima-research#281's "a framework compile from before stop cannot drive after restart" and anima-research#338's "a turn dropped after its compile, with ..." cases hold a stream's compile by replacing agent.buildActivationRequest. This branch renamed the stream path's assembler to assembleActivationRequest, which returns the request with its continuePrompt, and kept buildActivationRequest as its preview. So after the merge of main the holds never fired, and each of those tests waited until its timeout. A helper, holdStreamCompile, now holds the assembler the stream path calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and the abort docs A stop still writes its inference-log row with `success: false` (`Stream cancelled (<reason>)`, or `Stream aborted: shutdown` or `: quiesce_abandoned`), so `errorsOnly`, documented as "Only show failed inferences" / "Only show failures", lists stops as well. Its docs in the MCP tool, the API params and InferenceLogQuery now say it shows inferences that didn't succeed, failed or stopped, and the log entry's `error` doc says it holds the failure's message or what stopped it. inference:aborted's `reason` doc (from #338) named only abortInference's reason, quiesce's and a dropped turn's sentence. It now also names Agent.cancelStream's reason, `'user'` when the canceller gave none, and `'shutdown'`. README's abort section says what a cancelled turn ends as: a stop, whether the stream reports the cancel as `aborted`, as an `error` or by just closing, with one inference:aborted, the [turn-interrupted] note and no inference:exhausted; a stream membrane aborts on its own (`timeout`, `error`) stays a failed inference; stop() and a quiesce abandon stop with their own reasons and no note. Documentation only; no behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main comes in now rather than after anima-research#282, which lands behind the 0.22.2 release hold, so the PR's next external review reads a current base. When anima-research#282 lands, anima-research#360 takes main again. 87 commits since 92125b0, among them anima-research#338, anima-research#274 (TypeScript 7), anima-research#347, anima-research#352 and anima-research#344. The merge is clean: git merge-tree reports no conflict. The merged tree has no reader of ProseStreamRouter, roundDestination or TurnRouting.destination, and main adds no prose delivery site: framework.ts calls routeSpeech eight times on both sides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A turn that the framework drops before it sends anything now ends with
inference:aborted. Before, it emittedinference:startedand then no terminal event, so a module that brackets a turn between the two, such as a typing indicator, kept the dropped turn open. The Discord module's typing keepalive now also stops at a turn's normal end,inference:completed, whichonTrace's documentation had left out.The gap
beginAgentTurnemitsinference:started, then runs the modules'gatherContextand the MCPLbeforeInferencehooks, then compiles. Both waits can be long, so the framework checks the turn twice before it sends anything:Either check drops the turn, and both returned with no terminal event. The check before the compile also logged nothing.
onTrace's documentation (src/types/module.ts) names the events a module ends a turn's bracket on:inference:turn_ended,inference:aborted,inference:failedandinference:exhausted. It left out the normal end,inference:completed, whichdriveStreamemits once, when the turn's stream completes. Tool-call cycles stay inside that stream, so it doesn't fire between them. The Discord module in this repository follows the documentation. Its typing keepalive stops at any of the four, and otherwise goes on refreshing until the agent's nextinference:startedorTYPING_MAX_MS, five minutes. So withautoTypingon, its default, typing went on after every ordinary reply, and after a dropped turn too.The change
inference:aborted. Itsreasonis the logged line's text without the[turn] <agent>:prefix, for example "a turn compiled while the framework was stopping is dropped before it sends anything". ItsdurationMsruns from this start of the turn, as the otherinference:abortedevents count from their own start. Nothing failed and nothing was sent, so the turn ends as aborted rather than failed, as a turn abandoned at an operator quiesce already does (reason: 'quiesce_abandoned').lastInference.endedAtis recorded too, asdriveStreamrecords it, so the health snapshot no longer shows a dropped turn as still running. It is recorded only while the record is still this turn's: an ephemeral run disposed while its turn waited has had its record deleted, and its name gets none back.lastInferencecarry only the agent's name, so the dropped turn's would end that newer turn: its typing loop would stop while it runs. The dropped turn then emits and records nothing, as before.inference:completedtoo, andonTrace's documentation names it as the normal end. The documentation also says what else a bracket meets: a context-budget restart emitsinference:startedagain with no terminal event for the stream it replaced, and a failing turn can emitinference:failedand theninference:exhausted.ui/now also returns an agent to idle oninference:turn_ended,inference:abortedandinference:exhausted, where it had looked only atinference:completedandinference:failed.inference:aborted'sreasonanddurationMsget doc comments, sincereasonnow has three sources: an operator's abort, a quiesce, and a dropped turn.The provider-wait hold after the compile already ends its turn with
inference:failed, as the failure sites' holds do, and this change leaves it alone. Ending it as aborted would leave connectome-host's activity module reading every held turn as running (below).Listeners elsewhere that handle only some of the terminal events still read a dropped turn as running. connectome-host's activity module ends its indicator on
inference:completed,inference:failedandinference:exhausted, and ignoresinference:aborted, which this change now emits for a dropped turn. The drop'sstopTyping()clears its indicator, but its own state still reads the turn as running until the agent's next end event, so a channel added to it withshow_inmeanwhile starts typing. Its owners are building that change.Tests
In
test/tool-result-guard.test.ts, each test records the agent'sinference:*events as a module's listener sees them:inference:aborted, with its reason.gatherContext) and after it (it waits in its compile). Each loses its place in one of five ways: the framework stops, the agent is removed, its run is disposed with its record and token (asrunEphemeralToCompletion's disposal does), the agent object is replaced under its name, or the replacement's own turn already holds the name's turn token.inference:aborted, with the exact reason, and itsdurationMscovers the time it was held.lastInference.endedAtis recorded, except for the disposed run, whose name gets no record back.inference:started, and records no end.In
test/discord-typing-keepalive.test.ts, for each of the five terminal events, a turn's keepalive shows typing atinference:started, refreshes once while the turn runs, and refreshes no more after the event. The module's interval runs on mocked timers.Evidence
At
9d81e5d9, onmaincd036e0d, with context-manager 0.14.0 and membrane 0.5.87 installed:tscand build. With--test-force-exit, as CI runs it, Node 20.20.2 passes 2,154/2,154 across 144 files. On Node 24.21.0 the forced runner counted 2,146, all passing: it cut two files short (history-module-realstore22/28,history-module-ux24/26), and without the force every file passes in full, 2,154/2,154.3f4cdb64, before the last commit):lastInferenceend, anddurationMsset to 0: six each;inference:completedtaken out ofTURN_TERMINAL_TRACES, exactly the test'sinference:completedcase fails.At this head,
2d090e3f, which mergesmain92125b07: the merge's one conflict was the drop before the compile, wheremainnow ends the dropped turn's routing (endDroppedTurn). The merge keeps both, in the order the drop after the compile already had: the reason and its log line,endDroppedTurn, then the typing stop, the event gate's end and the terminal event. The change againstmainis the same 9 files, +257 −12, as before the merge. Each file run on its own, the suite passes 2,244/2,244 across 145 files on Node 24.21.0, forced and unforced, and on Node 20.20.2 forced, with none cut short. Typecheck and build are clean.Not covered
Ruth-2424 reviewed it, and the merge of
main. Agnes-1009's probe found thatonTrace's documentation left out the normal end.— Ada-1017 (Claude Opus 5.5)