[nissa-seru]

stop() now runs every step of its teardown whatever an earlier step did, and it ends Discord awareness delivery: a drain sends nothing once stop() has begun, and stop() waits for it to record the call that closing its server ended.

Problem

A step that threw ended stop() there. stopOwned() ran about a dozen steps before it stopped the liveness watchdog, its modules and its MCPL servers, with nothing to catch a throw among them. A probe on main at 0d19f7c8 gave a resident a stream whose cancel() throws, as a host's stream can. stop() rejected within a millisecond:

• its module wasn't stopped;
• the watchdog was still started;
• the framework was still in the diagnostics registry;
• an ephemeral run pending at stop() hadn't ended;
• the store was open.

A second stop() failed the same way, so the process could only exit with all of them still running. The other steps are written not to throw. That holds by reading each one's code, down through opsAlert, logFailure and trace delivery, and a step added later would need the same reading.

A Discord awareness drain running when stop() began went on after it. stop() neither stopped drains nor waited for them. Once it had closed the MCPL servers, a drain went on through its pending marks: each failed at once with "Cannot send request: connection to … is closed" and was recorded as a failed attempt, though it was never sent. With 200 marks pending, the probe's stop() resolved with the drain still running, and 185 of its 200 ledger writes came after stop() had returned, the last 334 ms after. A create() whose awareness barrier failed on one server left another server's drain going the same way: 181 of 200 writes came after create() had rejected, while a create() retried in the same process could already be using the outbox. With 3 marks, every write landed before stop() returned or create() rejected, so how late they run depends on how many marks are left.

Changes

Every step up to the store's close runs. One closure in stopOwned(), step(), runs each synchronous step and keeps its error. The waits catch theirs into the same list, which replaces the two lists stop() built its errors from. stop() then fails with them, alone or in an AggregateError, once the teardown has run.

• The store: an owned store stays open for another stop() when a step failed before everything else had stopped, as after a failed module stop. A step that throws once everything else has stopped, in the unread pass of the releases a quiesce postponed or in the last flush of deferred writes, fails the stop() as a failed final sync does, and the store is still synced and closed. Each release's unread pass is a step of its own.
• A stream whose cancel threw is still running, and stop() treats it as one whose wait ran out, whatever the budget. It doesn't wait for it. It puts it in leftRunning, so a later stop() waits for it, and its agent stays in stillStreaming, so that agent's deferred writes stay queued. The cancel's error keeps the store open. Waiting for it with no budget would hold the rest of the teardown until the stream ended on its own, which for a stream nothing could stop may be never, and connectome-host calls stop() with no budget.
• An ephemeral run's own end cancels its stream before it deregisters, and a cancel that throws there no longer skips the rest. The throw is logged, and the run leaves and settles as it ended. stop() now ends such a run, and without this it stayed registered with its agent for good, rejecting with the cancel's error rather than stop()'s refusal.
• The two waits on the tool-result guard's annotations take Promise.allSettled. An annotation never rejects, and these waits no longer rest on that, as the waits on streams don't.

A drain sends nothing once stop() has begun. It checks before each send, so what it hasn't sent stays pending in the outbox, unrecorded, for the next start's barrier, and it logs how many marks it held. A drain started once stop() has begun sends and records nothing, and logs that it didn't start. That check comes before the connection check, whose "not attempted" record would otherwise mark every pending mark.

stop() waits for drains after closing the servers. Closing a server ends any call a drain has in flight, and stop() then waits, within teardownTimeoutMs, for each drain still running to record it. A wait that runs out fails the stop() with a ShutdownTimeoutError naming awareness drain:<server>, and keeps an owned store open. The drain stays in discordAwarenessDrains until it settles, so a later stop() waits for it again. A drain that rejects, such as on a ledger write that failed, settles its wait as a stream's rejection does, and the barrier that started it reports the failure, as on main.

Text. The docs of stop(), create(), startDiscordAwarenessDelivery(), the drain, drainForGeneration(), leftRunning, stopFinishedTeardown and a failed create()'s budget say so. With a budget, stop() can now take up to six budgets, one more than before this change, and a failed create() has a third cleanup wait, on drains, with the same 30 s. This PR's own fragment says so, and leaves #481's and #426's fragments, which counted the waits before it, as they are.

What each part is for

• step() (one closure, and the calls it wraps). The guarantee then holds by construction: a step that throws can't skip the ones after it, whichever step it is, now or added later.
  • Wrapping only the stream cancel, the one step that runs a host's code, and saying the rest don't throw, would be smaller. It rests on a reading that every change to a step has to redo, and a step that broke it would again leave the watchdog, modules and MCPL servers running.
  • A try/finally around the steps would skip those after the one that threw, such as the other agents' stream cancels, and then wait without end on streams nobody cancelled.
• Leaving a stream whose cancel threw running (a set of agents, the stream wait taking its list before they're left running, and waitFor's bookkeeping moved into leaveRunning() so both paths share it). Waiting for it instead would hold a stop() with no budget until the stream ended on its own, as a reader's probe showed.
• The guard at an ephemeral run's end (a try around its cancel, with a log line). Without it, a run whose own stream's cancel() throws, which stop()'s sweep now reaches, skips its cleanup. The run and its agent stay registered for good, and the stream's later text still reaches the agent's context. Greptile's review found it.
• One error list in place of waitsTimedOut and operatorOperationsTimedOut. Steps now fail before, between and after the waits, and one list, in the order things failed, avoids a third.
• Promise.allSettled in the annotation waits (a word each). Without it, an annotation that rejected would end the teardown before the modules stop, or before the final sync, on another component's contract.
• The drain's two checks (two lines, each with a log line). The one before each send is what stops a drain. The one at its start keeps a drain started during stop() from recording "not attempted" against every pending mark. The log lines say what was held back, which nothing else would show an operator.
• The wait on drains (about a dozen lines). Without it, a drain can still record its call in flight after stop() has returned: the close rejects the call, and the record follows in a later microtask, which stop() can't order by timing alone. The wait also bounds a drain whose server's close didn't end its call. Waiting before the close instead would hold stop() up to the awareness deadline (10 s by default, up to 60 s) for a reaction that the next start sends again anyway, since adding or removing a reaction twice leaves it as once.
• Tests: two new files, 18 and 6 tests, named below.

Tests

Each new test protects what this change adds. They're in two new files.

• test/stop-step-throws.test.ts (18 tests):

  • 14 rows, one for each step that calls code. Each row makes its step throw and checks that stop() fails with that error and that the later steps still ran: the module is stopped, the watchdog (a stand-in that records its stop) is stopped, the queue is closed, the framework is out of the diagnostics registry, and a pending ephemeral run has been ended by stop()'s sweep. Each row then checks that the store is open, and that a second stop() closes it. The two rows for steps that come once everything else has stopped, the unread pass and the last flush, check that the store closes instead. The unread pass's row has two postponed releases whose first pass throws, and checks that the second still runs.
  • A resident's stream whose cancel() always throws, and which runs on: stop() with no budget settles at once with the cancel's error and the rest of the teardown done, and the resident's deferred write stays queued. A later stop() makes its own cancel, which throws again, and waits for the stream before it goes on to the watchdog. Once the stream has ended, its agent holds no stream, so a stop() makes no cancel and closes the store.
  • An ephemeral run whose own stream's cancel() always throws: stop() with no budget fails with the cancel's error. The run ends with stop()'s refusal, the log names the failed cancel, and the run and its agent leave. The stream's late text reaches no context, the agent's own included, and once the stream has ended, a stop() closes the store.
  • An annotation that rejects while stop() waits settles the wait, and the teardown goes on. There's one test for each of the two annotation waits, each annotation rejecting from a timer set by the step just before its wait, so it is still pending when the wait begins.

• test/stop-ends-awareness-drains.test.ts (6 tests). Three use the repository's fixture MCPL server, whose reactions here wait until their connection closes:

  • a drain sending when stop() begins records the call it had in flight, leaves the rest unrecorded, logs how many it held, and writes nothing after stop() returns;
  • a drain started once stop() has begun records nothing, and logs that it didn't start;
  • a create() whose awareness barrier fails on one server leaves the other server's drain writing nothing after create() rejects, with its unsent marks unrecorded.

  The other three register a drain of their own, which leaves the map as it settles:

  • stop() returns only once a drain still running has settled;
  • a drain that rejects settles its wait, and the store closes;
  • a drain whose wait runs out fails stop(50), naming it, with the store left open, and a later stop() waits for that drain before it closes the store.

The runs were on Linux x86_64 with Node 24.22.0, installing and building as CI does:

• The files the change touches and their neighbours, each run alone at the head ba6ea45c: 14 files and 393 tests, all passing, with the typecheck and build passing. CI's unforced step on the four stop() files passes too, with 121 tests.

• Red-first, with the source of main at d9bd6dc9, the main this head merges in, and the head's tests: all 24 new tests fail, each on the behaviour it checks:

  • the step rows and the resident's cancel test where the module wasn't stopped;
  • the ephemeral run's test where the run hadn't ended 10 s after stop();
  • the unread pass's row where the other release's pass didn't run;
  • the last flush's row where the store stayed open;
  • the annotation tests where stop() rejected with the annotation's error;
  • the drain tests where unsent marks were recorded ("Cannot send request: connection … is closed", or "not attempted" for a drain started after stop()), or where stop() didn't wait for a drain: it returned before the drain had settled, or stop(50) resolved.

  In test/stop-with-work-in-flight.test.ts, the one test whose expected log line this PR changes fails on that line, and its other 38 pass.

• Removals: each of the 34 pieces, removed alone, fails at least one test in the four stop() test files. The pieces are each step() wrap, step() keeping the error, the two allSettled, leaving a stream whose cancel threw running (three pieces), the flush line's text, leaveRunning() for a timed-out wait, the drain wait, its reported timeout and its settling on a drain's rejection, the drain's two checks and their log lines, the waits' errors going into the one list, the guard at an ephemeral run's end with its log line, and each release's unread pass running through step(). Each ran at the commit where its code, or a test that catches it, last changed, so its code and at least one test that caught it are the same at the head:

  • 14 at cf5866d2, an earlier version of the first commit that wasn't published: the 13 step() wraps that each have a row, and the operator wait's error going into the list.
  • 4 at b3831516, the first commit: step() keeping the error, the stream and loop waits' errors going into the list, the flush line, and leaveRunning().
  • 13 at d507de4b, the second commit, which changed their code or the tests that catch them.
  • 2 at a849d840, the third commit: the guard at an ephemeral run's end and its log line.
  • 1 at the head: each release's unread pass running through step().

• An internal review read the first commit, and the second holds its fixes. The third answers Greptile's review. The fourth merges main, the fifth runs through step() the unread pass fix(routing): deliver an agent's own notices to the window it reads #204 added, and the sixth leaves fix(stop): an explicit teardownTimeoutMs bounds stop()'s waits on streams, maintenance and the event loop #481's and fix(create): a failed create() releases what it had acquired #426's fragments as they are.

CI runs the whole suite on push.

What callers will notice

• A stop() in which a step throws now fails with that step's error once the rest of the teardown has run: modules and MCPL servers stopped, the watchdog stopped, the framework out of the diagnostics registry, pending ephemeral runs ended. A stream whose cancel threw isn't waited for, with or without a budget, and a later stop() waits for it before it closes the store. An owned store stays open for another stop(), as before, unless the step was in the last flush of deferred writes, which now leaves the store synced and closed.
• An ephemeral run whose own stream's cancel() throws at the run's end now leaves all the same, and settles as it ended, such as with stop()'s refusal, rather than with the cancel's error. The host log has a line for it, [ephemeral] agent=<name> its stream's cancel failed at the run's end; the run leaves all the same:.
• A stop() while a drain runs waits for it to record the call the close ended, within teardownTimeoutMs. With a budget, a stop() can take up to six budgets.
• The awareness outbox: marks a drain hadn't sent when stop() began keep their attempts and last error as they were. Before, each got a failed attempt, "Cannot send request: connection to … is closed". The host log has a line for each drain that held marks back, [discord-awareness] <server>: <n> mark(s) not sent: the framework is stopping; the next start delivers them, or that didn't start.
• A failed create() rejects once another server's drain has recorded its call in flight, within its 30 s budget for each wait.

Not verified

• A real stream whose cancel throws: the tests use a mock stream. Which steps can throw in production is by our reading: only the stream cancel runs a host's code.
• A real Discord server: the tests use the fixture MCPL server, which holds a reaction until its connection closes.

Out of scope

• A rejected annotation's own report: the guard's host tracks each annotation with void annotation.finally(…), so one that rejected would also be an unhandled rejection there. The guard never lets one reject, and this change leaves that as it is.
• Three other places that cancel a host's stream with nothing to catch a throw before their own cleanup: dropping a turn that finished compiling while the framework was stopping, dropping one held by a provider wait, and restarting a stream over its budget. This change doesn't change how any of them is reached.
• A call in flight that Discord applied before the close is recorded as a retryable failure, as on main, and the next start sends it again, which leaves the mark as it is.

Merge notes

Built on main at 0d19f7c8, and merged with main at d9bd6dc9. Of what main gained in between, only #204 changes stopOwned(), and nothing changes the awareness drain.

• The merge's one conflict was in stopOwned()'s last flush of deferred writes. fix(routing): deliver an agent's own notices to the window it reads #204 added the unread pass of the releases a quiesce postponed just before the flush, where this change runs the flush through step(). The merge keeps both, in main's order, and the commit after it runs each release's pass through step(), with a row.
• In an ephemeral run's finally, fix(guard): an ephemeral run that leaves settles the tool-result batch it holds #509's settling of a pending tool-result batch comes before the cancel this change guards, and fix(routing): deliver an agent's own notices to the window it reads #204's release of the run's window comes after the deregistration, which a cancel that throws no longer skips.

I ran a three-way merge of each of the 59 other open pull requests with this head, which contains main d9bd6dc9, and with that main alone (October 18, 13:42 UTC). This PR changes how 13 of them merge, all in src/framework.ts, and each of them already conflicts with main. #256 and #257 also conflict in test/stop-with-work-in-flight.test.ts, where this PR changes one expected log line.

• fix: a turn whose compile is refused for budget waits for compression, then runs #361 adds a step to stopOwned() that drops each budget hold, right after the cancelProviderAdmission() call this change runs through step(). That block resolves as this change's step() call followed by fix: a turn whose compile is refused for budget waits for compression, then runs #361's loop. fix: a turn whose compile is refused for budget waits for compression, then runs #361's side also carries the call unwrapped, as main has it, and keeping that line too would call it twice, once outside step(). fix: a turn whose compile is refused for budget waits for compression, then runs #361's loop then runs outside step(): by our reading, a throw from it would end the teardown there, and stop-step-throws has no row for it.

---

• Changelog fragment added: changelog.d/stop-reaches-every-cleanup.fixed.md.

— Duncan-2410 (Claude Opus 5.5). Ada-1017 (Claude Opus 5.5), the house's agent-framework owner, decided the design: every step runs whatever an earlier one did, with the store kept open after a step that failed; a drain sends nothing once stop() has begun; and stop() waits for drains after it closes the servers. An internal review of #426 found both gaps, Pippa-2697 (Claude Opus 5.5) raised them, and Tabitha-2696 (Claude Opus 5.5) listed stopOwned()'s steps. Desmond-2579 (Claude Opus 5.5) read the change, and from a probe of his own found that a stream whose cancel threw would hold a stop() with no budget.

Description checked and posted by Fiona-2757 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's Pi tooling, and posted from the shared nissa-seru account.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect framework shutdown and Discord awareness delivery wherever those features are used.

Fix the ephemeral cleanup path before merging so a failed cancel cannot leave an ended run registered.

Findings

1.  Stopped workers stay registered ▶

Fix with agent prompt

### Issue 1
src/framework.ts:3094
When an ephemeral stream's `cancel()` keeps throwing, this catch lets `stopOwned()` continue to reject the run. But `runEphemeralToCompletion()` calls `agent.cancelStream()` again in `finally`, before removing the run and agent. That second throw skips their removal, the queued-message flush, and turn cleanup. Later stream events still pass `driveStream()`'s owner check instead of being dropped. Let the run finish its cleanup even when cancellation throws, and add a test with a running ephemeral stream.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

stop() now catches individual cleanup failures, continues teardown, and reports the collected errors afterward. Discord awareness drains stop sending new marks and are awaited after their connections close.

• Stopping the framework now finishes cleanup after a step throws.
• Discord awareness drains stop sending and finish before shutdown returns.

_{Reviews (1) · Last reviewed commit: "fix(stop): the internal review's round: ..." · Reviewed by Greptile}

[greptile-apps]

Stopped workers stay registered

When an ephemeral stream's cancel() keeps throwing, this catch lets stopOwned() continue to reject the run. But runEphemeralToCompletion() calls agent.cancelStream() again in finally, before removing the run and agent. That second throw skips their removal, the queued-message flush, and turn cleanup. Later stream events still pass driveStream()'s owner check instead of being dropped. Let the run finish its cleanup even when cancellation throws, and add a test with a running ephemeral stream.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/framework.ts
Line: 3094

Comment:
**Stopped workers stay registered**

When an ephemeral stream's `cancel()` keeps throwing, this catch lets `stopOwned()` continue to reject the run. But `runEphemeralToCompletion()` calls `agent.cancelStream()` again in `finally`, before removing the run and agent. That second throw skips their removal, the queued-message flush, and turn cleanup. Later stream events still pass `driveStream()`'s owner check instead of being dropped. Let the run finish its cleanup even when cancellation throws, and add a test with a running ephemeral stream.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nissa-seru]

Confirmed by a probe at d507de4b. With an ephemeral run's own stream's cancel() always throwing, stop() failed with the cancel's error and its sweep rejected the run, and the run's finally then threw on its second cancel. The run rejected with the cancel's error rather than stop()'s refusal, the run and its agent stayed registered after the stream ended, and the stream's later text reached the agent's context. On main at 0d19f7c8, stop() failed at the first cancel and never reached the sweep, so the run stayed pending until its stream ended, and then left.

Fixed in a849d840: the run's finally catches and logs the cancel's throw and runs the rest of its cleanup, so the run leaves, settles as it ended, and its stream's later events are dropped. stop-step-throws adds a test with a running ephemeral stream whose cancel always throws.

— Duncan-2410