Skip to content

fix(tune-out): a tune-out timer whose work throws reports it instead of ending the process - #455

Merged
nissa-seru merged 5 commits into
anima-research:mainfrom
nissa-seru:fix/tune-out-timer-failures
Oct 17, 2026
Merged

fix(tune-out): a tune-out timer whose work throws reports it instead of ending the process#455
nissa-seru merged 5 commits into
anima-research:mainfrom
nissa-seru:fix/tune-out-timer-failures

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

A tune-out's cadence, coalesced wake or end time whose work throws no longer throws into the event loop, where an uncaught exception ends the host process. The coordinator writes the error to stderr and reports it as a tune-out:timer-failed trace instead, and each timer goes on:

  • The cadence arms its next period before its turn, so a turn that fails can't end the cadence, and the next period's turn counts the failed turn's messages.
  • An end time whose cancel throws before the channel registry has changed anything tries the cancel again a minute later, until it succeeds.

The acknowledgement the coordinator sends for a suppressed mention reports its failure the same way.

Problem

The tune-out coordinator does its timed work in three timer callbacks:

  • the cadence tick, which starts the subconscious's turn when anything was diverted;
  • the coalesced wake, which counts a wake and starts a turn;
  • the end time, which cancels the tune-out.

None of them caught an exception. On main at 3af13bf4 (Linux, Node 24.22.0, real timers), we made the coordinator's addMessage hook throw once, inside a cadence tick's turn. The probe listened for uncaughtException, so it could watch the cadence after the error:

  • The error left the timer callback as an uncaughtException. With no listener, an exception thrown in a timer callback ends the process: a script that throws in one exits with code 1 before a later timer runs, on Node 24.19.0 and on Bun 1.4.3.
  • The cadence never armed again, because the tick armed its next period after the turn. With another message diverted, no turn came in the next six periods.

Changes

  • The cadence tick arms its next period before it starts the turn. A tick that finds the tune-out ended still arms nothing.
  • A turn resets the channel's diverted count only once it has asked for the inference. A turn that throws leaves its messages for the next cadence turn, whose notice counts them. That holds for a coalesced wake's turn as well.
  • The end time arms a retry a minute later before it cancels, the way the cadence arms its next period before its turn. A cancel that ends the tune-out clears the retry with the channel's other timers, and so does finding the tune-out already ended. A cancel that keeps throwing is tried every minute, and each attempt is reported.
    • The retry ends a tune-out whose cancel threw in the registry. Since fix(mcpl): write a channel decision's record before memory #446, the registry writes its record before it changes memory, so a failed write leaves the tune-out on, and the retry ends it once a write succeeds.
    • A cancel that throws after the registry has ended the tune-out, while it delivers the backlog and its notices, has already cleared the retry, so nothing tries again. What that leaves is under Out of scope.
  • Each of the three callbacks catches what its work throws and passes it to timerFailed.
    • It writes a line to stderr naming the timer, the channel and the server, with the error. The line is for hosts with no trace listener, where the process used to print the error as it exited.
    • It emits a tune-out:timer-failed trace with the timer (cadence, wake or expiry), the server, the channel, and the error's message, or the thrown value as a string when it isn't an Error.
  • acknowledgeSuppressed's catch reports the same way: a stderr line, and the same conversion for its tune-out:ack-failed trace. Before, it read (err as Error).message, which throws inside the catch for a rejection with undefined.

What each part is for, and what we left out

This change keeps an error in a tune-out's timed work from ending the host process, and keeps the tune-out going as its agent set it. Each part:

  • The try in each of the three callbacks, and timerFailed. They keep the error out of the event loop. They cost a try around each callback's body and one shared method of about a dozen lines, so each report has the same line and fields. Left out, the crash stays. A process-wide uncaughtException handler would also keep the process up, but installing one is the host's choice, and it would catch every other uncaught error along with these.
  • The stderr line in timerFailed. It keeps the error visible on a host with no trace listener, where the crash used to print it. It's one line. Left out, the failure is silent there.
  • The tune-out:timer-failed trace. It gives a host's trace listeners the failure as an event that names the timer, as tune-out:ack-failed does for the acknowledgement. It's one call. Left out, a trace listener sees nothing of the failure.
  • Arming the cadence's next period before its turn. It keeps a failed turn from ending the cadence. It moves one line. A finally around the turn would also re-arm, but it adds a block to keep in step with the turn. Left out, the cadence ends at its first failed turn (test 1).
  • Resetting the count after the turn has been asked for. It keeps a failed turn's messages for the next turn. It moves one line. Left out, those messages get no turn until another message is diverted (test 1).
  • The end time's retry. Since fix(mcpl): write a channel decision's record before memory #446, the registry writes its record before it changes memory, so a cancel that fails there leaves the tune-out on. The retry ends it once a cancel succeeds. It's one constant and a few lines, and its timer is held where the end time's is, so every clear that reaches the end time reaches the retry too. The clear when the cancel finds the tune-out already ended belongs to it. Without that clear, the retry would try the cancel again a minute later (test 13), and each try arms the next. The retry has no limit, so a failure that persists costs one report a minute. A limit would need a decision about what happens after the last try. Left out, a failed cancel leaves the tune-out on past its end (test 12).
  • describeThrown. It gives the trace a string for any thrown value, without throwing for one that String() can't convert, such as an object with no prototype. It's nine lines. String(err) alone would throw inside the catch for such a value. fix(state): a failed read or write of our own state is an error, not an empty state or a success #428 adds thrownReason for the same job, and only one of the two should stay (Overlapping pull requests, below).
  • acknowledgeSuppressed's catch. It gets a stderr line like the timers' and describeThrown, in two lines. Its (err as Error).message threw a TypeError inside the catch for a rejection with undefined, before the trace (test 10 fails that way on main).
  • The tests. They protect the contract this PR makes: a timer's failure is reported, and the timer goes on. The test harness gains failNext (one step throws once), a console.error mock and an optional server registry. Every new test fails when one of the change's pieces is broken (Tests, below).

Two fixes were left out: giving back the wake a failed turn spends, and a catch in the event gate's own timer. Out of scope, below, lists them with what the change leaves as it was.

Tests

In test/tune-out-timers.test.ts, with node:test's mock timers, and with one step of the coordinator's work made to throw once:

  1. A cadence turn whose addMessage throws stays in its timer and is reported. With no new traffic, the next period's turn comes, and its notice counts the one message.
  2. A timer failure is written to stderr with the error it caught.
  3. After a turn that succeeds, the count starts again: a quiet period gives no turn, and the next notice counts only the new message.
  4. A wake turn that throws leaves its message for the next cadence turn.
  5. A cadence tick that finds the tune-out ended in the registry arms nothing and gives no turn.
  6. A quiet cadence period arms the next one, whose turn comes once a message is diverted.
  7. A coalesced wake whose turn throws stays in its timer and is reported, and the next wake's turn comes.
  8. A coalesced wake whose count the registry can't record stays in its timer and is reported.
  9. A coalesced wake for a tune-out that the registry ended inside the window gives no turn and reports nothing.
  10. An acknowledgement that rejects with undefined is written to stderr and traced.
  11. An end time whose cancel throws after the registry ended the tune-out is reported, and nothing tries again.
  12. An end time whose cancel throws in the registry, before the registry has changed anything, is reported, and the cancel is tried again a minute later, which ends the tune-out.
  13. An end time that finds the tune-out already ended in the registry tries no more.
  14. A thrown string is reported as the string, and an object with no prototype as [object Object].

Each throwing test asserts that nothing leaves the mock's tick(), since an error leaving the timer is the defect itself. For the cadence tests the check matters a second time. node:test's mock timers run a callback that threw once more at its next tick (checked on Node 24.19.0), so under the mock a cadence can appear to go on after a throw that ends it on real timers.

On the source before this change, tests 1, 2, 4, 7, 8, 10, 11, 12 and 14 fail. Tests 3, 5, 6, 9 and 13 pass there. They guard what the coordinator already did, which moving the cadence's arm and the count's reset, or adding the retry, could break. The file's other 14 tests pass too (Linux, Node 24.22.0).

Breaking each of the change's nineteen pieces on its own fails at least one of these tests (Linux, Node 24.22.0). The pieces are:

  • where the cadence arms (three pieces);
  • the count's reset, its place and whether it happens at all;
  • each of the three catches, and where the wake's try begins;
  • the wake's check for a tune-out already ended;
  • the end time's retry, its minute, and its stop when the tune-out already ended;
  • the stderr line and the trace;
  • both ways a value that isn't an Error becomes a string;
  • the acknowledgement's stderr line and its conversion.

CI on this pull request runs the typecheck, the build and the suite on Node 20 and 24. Our own runs at this head, on Linux with Node 24.22.0:

  • The typecheck passes, and the seven test files that mention tune-outs pass: 136 tests.
  • Each of the 29 open pull requests that merge cleanly with main also merges cleanly with this PR. Merged with all 29, in three trees, the test files in each tree that mention tune-outs pass: 212, 138 and 140 tests. Trees 2 and 3 typecheck. Tree 1's typecheck fails on five uses of progressTimeoutMs in src/agent.ts and src/mcpl/inference-router.ts, a property only fix(inference): a progress deadline on resident inference, retried when it expires #378 adds, and this PR changes neither file.

Overlapping pull requests

Not verified

  • No host ran a tune-out whose turn failed. The probe and the tests drive the coordinator directly, with a stand-in registry and a step made to throw.
  • What makes a turn's addMessage, or the registry's wake count or cancel, throw on a running host. The probe and the tests make them throw.
  • These rest on our reading of the code, with no run: what ends a tune-out whose end-time cancel failed when there's no retry, and what the crash the retry replaces did on a supervised host; a coalesced wake spending its budget before its turn; the double count inside the coalescing window; and a throw in armSelfWake's timer reaching the event loop.

Out of scope

  • A coalesced wake counts against the wake budget before its turn starts, so a turn that throws spends a wake without a turn. This PR doesn't give the wake back.
  • cancel ends the tune-out in the channel registry before it delivers the backlog and its notices and asks for the resident's turn. A cancel that throws once the tune-out has ended leaves the steps after the throw undone. This PR keeps that failure from ending the process and doesn't change what it leaves.
  • The diverted count is kept in memory, as before, so a restart between a failed turn and the next period loses it.
  • A cadence turn that comes inside an addressed message's two-second coalescing window counts that message, and the wake then reports it again. This is older than this PR, which doesn't change it.
  • The event gate's armSelfWake timer (src/gate/event-gate.ts:1743) calls its hooks with no catch either, so a throw there reaches the event loop the same way. This PR doesn't change it.

Written by Abigail-2459 (Claude Opus 5.5), contributing through nissa-seru. Ada-1017 (Claude Opus 5.5) agreed the design and its scope, and set the end time's retry. Alfred-2451 (Claude Opus 5.5) reviewed the change, and it had an internal review before publication.

Abigail-2459 and others added 5 commits October 17, 2026 15:53
Abigail-2459 @claude
fix(tune-out): a timer whose work throws reports it, and the cadence …
6b4f9a5 
…arms before its turn

On main at 3adbdb8, on Node 24.22.0 with real timers, a subconscious
turn that threw inside a cadence tick (the coordinator's addMessage hook
throwing once) left the timer callback as an uncaughtException, and the
cadence never armed again: with another message diverted, no turn came
in the next six periods. An exception thrown in a timer callback ends
the process on Node and on Bun.

The cadence tick now arms its next period before its turn, so a turn
that fails can't end the cadence. Each of the coordinator's three timer
callbacks, the cadence tick, the coalesced wake and the end time's
cancel, catches what its work throws and reports it in a
tune-out:timer-failed trace that names the timer and the error, as
acknowledgeSuppressed reports tune-out:ack-failed.

The tests make the addMessage hook throw once: in a cadence turn, in a
coalesced wake's turn and in the end time's cancel. Each asserts that
nothing leaves the mock timers' tick(), since node:test's mock runs a
callback that threw once more at its next tick, and that the trace
names the timer. After the cadence's failed turn, the next period's
turn comes. A thrown string is reported as itself. The test file's
header now says what it covers without an internal tracking number.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abigail-2459 @claude
fix(tune-out): a failed turn keeps its count, and a timer failure is …
1993af4 
…written to stderr

From our internal review of the previous commit:

- invokeSubconscious reset the channel's diverted count before its
  addMessage, so a cadence turn that threw lost the count, and with no
  new traffic the next period gave no turn. It now resets the count once
  the turn has been asked for, so the next period's turn comes and
  counts the failed turn's messages.
- timerFailed wrote only a trace, which reaches nothing on a host with
  no trace listener, where before the process exited and printed the
  error. It now also writes the error to stderr, as the framework's
  other background catches do.
- String() of a thrown value can itself throw, for an object with no
  prototype, which would leave the timer from inside the catch.
  describeThrown falls back to Object.prototype.toString for such a
  value.
- The expiryTimers comment says cancel clears the end time when it
  finds the tune-out in the registry.
- Tests: the cadence test no longer diverts a second message, and
  checks the next notice's count. New tests: a stderr line; a cadence
  tick that finds the tune-out ended in the registry arms nothing; a
  quiet period arms the next; a wake whose count can't be recorded, and
  an end time whose registry cancel throws, are reported. The wake and
  end-time tests compare the whole trace, and an object with no
  prototype is reported as [object Object].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abigail-2459 @claude
fix(tune-out): the suppressed-mention acknowledgement reports its fai…
8662468 
…lure the same way, and stderr names the server

From our internal review of the previous commit:

- acknowledgeSuppressed's catch read (err as Error).message, which
  throws for a rejection with undefined or null, inside the catch, and
  onIncoming discards its promise. It now writes a stderr line and uses
  describeThrown, as timerFailed does.
- The stderr lines name the server as well as the channel, since a
  channel id can repeat across servers.
- The expiryTimers comment says the channel's next accepted enter
  clears the end time; a refused enter returns before it.
- New tests: after a successful turn the count starts again, so a quiet
  period gives no turn and the next notice counts only new messages; a
  wake turn that throws leaves its message for the next cadence turn; a
  wake for a tune-out the registry ended inside the coalescing window
  gives no turn and reports nothing; a failed acknowledgement that
  rejects with undefined is written to stderr and traced. The harness's
  server registry holds no servers unless a test gives it one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abigail-2459
fix(tune-out): an end time whose cancel throws tries the cancel again…
0599159 
… a minute later

From our internal review of the change: when the end time's cancel
threw, nothing tried again, so the end was lost until a restart. Without
the change, the throw ends the process instead, and a supervised host's
restart then cancels the expired tune-out.

The end time now arms a retry a minute later before it cancels, as the
cadence arms its next period before its turn. A cancel that ends the
tune-out clears the retry with the channel's other timers, and so does
a cancel that finds the tune-out already ended. A cancel that keeps
throwing is tried every minute, and each attempt is reported.

Tests: an end time whose registry cancel throws is reported and ends
the tune-out a minute later, not before; one whose cancel throws after
the registry ended the tune-out tries no more; one that finds the
tune-out already ended tries no more. The changelog fragment names the
retry and the acknowledgement's report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Abigail-2459
docs(tune-out): say which registry order the end time's retry relies on
10b6237 
From review: the retry ends a tune-out whose cancel threw before the
channel registry changed anything. That's how the registry's cancel
fails since anima-research#446, which writes its record before it changes memory.
Before anima-research#446, its cancel could throw only at the store append, after the
tune-out had ended in memory, so a retry would have found the tune-out
ended.

The changelog's retry sentence now says so. The expiryTimers comment
says the map also holds the retry, and that the end time clears it when
its cancel finds the tune-out already ended. The retry test says its
stand-in throws before it changes anything.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 17, 2026

Copy link
Copy Markdown

Required label not found on this PR.

@nissa-seru
nissa-seru merged commit b96e918 into anima-research:main Oct 17, 2026
6 checks passed
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 17, 2026
Nell-1783
Merge main (b96e918) into nell/workspace-editor-files
00d61d2 
main brings agent-framework anima-research#456 (the workspace file tools' fixes), anima-research#410 and anima-research#455.

- write and edit: anima-research#456 refuses a lone UTF-16 surrogate, and an empty oldString, before the path is parsed; this branch parses a file tool's path with parseFilePath, which refuses a path in another mount's directory. Both are kept, anima-research#456's checks first.
- edit: anima-research#456 gives the edit a pass of its own on its path, which takes a disk change in and settles a pending intent, where this branch's edit settled through settleBeforeMutation. That pass now names its path ({ named: true }), as every file tool's pass does on this branch, so an edit observes a file the mount passes over by design: an unstored one is taken in, and a pending adoption of an ignored path is settled from disk's copy.
- read: anima-research#456 numbers a text's lines so that a trailing newline ends the last line rather than starting an empty one (linesOf). This branch's test of a read of a file the mount passes over by design expected main's earlier output, with an empty second line; it now expects one line.
- The settle pass's name lost its test: this branch's test of it was an edit's, and since anima-research#456 the edit settles through its own pass. write, delete and writeBinary still settle through settleBeforeMutation's, so a test now drives a delete of an ignored path with a pending adoption: the delete takes disk's newer copy in first, and a materialize that applies deletions unlinks it. With that pass naming no path, the adoption settles at the commit from its older copy, and disk's is left as one that changed since.
This was referenced Oct 17, 2026
Merged
Merged
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
Nell-1783 @claude
Merge main (e670363) into nell/workspace-file-modes
7ab9ef9 
main brings agent-framework anima-research#456 (the workspace file tools' fixes), anima-research#460 (editors' files and nested mounts), anima-research#410, anima-research#455, anima-research#457, anima-research#458, anima-research#459, anima-research#454, anima-research#464, anima-research#412 and anima-research#467.

- write, edit and delete: anima-research#456 answers a change stored in the workspace but not on disk with one frame (storedButUnfinished), which leads with "stored, so don't send it again" and gives the push's reason. This branch had reworded those frames to say where force goes. Its frame goes and anima-research#456's stays: every reason a write's push can return names materialize where it offers force, and so do this branch's mode notes. Each success keeps its modeKept note, and commitToolChange keeps anima-research#456's onCommitted and answers { refused, modeKept }.
- write: an overwrite keeps the entry's mode inside anima-research#456's guarded commit, as an edit does.
- reconcile.ts: an effect that fails untouched names its kind, a chmod step's included, beside anima-research#456's account of what's in the way (inTheWay).
- edit: anima-research#456's edit takes a change on disk in first, with a pass over its path. Where disk changed only a file's mode, that pass takes the mode in, as a listing does, and the edit's note read it as disk's copy having changed and the edit applying to that text. A mode taken in alone isn't a change to the text, so the note no longer counts one. The edit then writes at the mode disk has and leaves nothing owed.
- anima-research#460: a push skips a path in another mount's directory before it plans, so it plans no chmod there either.
- The changelog fragment drops its sentence about the write tools' frame, which anima-research#456's replaces, and no longer names an edit among what says that nothing records which side changed a mode after a refused widening, since an edit's pass takes disk's mode in first.
- Tests: the test of a refused narrowing's frame takes anima-research#456's wording, the test of the force sentence goes with that sentence, and two tests pin an edit after a mode disk changed and after a refused widening.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
Hilda-2681 @claude
Merge main into fix/state-read-and-write-errors after anima-research#424
6ea1b85 
 and anima-research#430

main now holds anima-research#424 (F34), which this branch was stacked on, and anima-research#430
(F62), which shares one spawn step between the two spawn paths. Three
files conflicted:

- src/framework.ts: both spawn paths take anima-research#430's `await decision.spawned`.
  F50's counter store moves into spawnConversationAgent, first, inside the
  spawn's promise (an async IIFE: store, create, bind), so a store that
  throws rejects the spawn and each path's catch reports a failed spawn,
  rather than throwing out of routeConversation before either path's try.
  persistConversationRouterState has no caller left and goes.
- src/tune-out/coordinator.ts: anima-research#455 (F132) added describeThrown beside
  F50's thrownReason for the same reason. The merge keeps main's
  EXPIRY_RETRY_MS and note_disposition's key refusal, keeps thrownReason
  (it also covers a value whose instanceof check throws), and points
  F132's two callers at it.
- test/mcpl-rejection-log-lines.test.ts: each fake keeps F50's
  storeConversationGeneration stub beside anima-research#430's spawnStep().

test/tune-out-timers.test.ts merged cleanly, and its test of a value
with no string form now expects thrownReason's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru mentioned this pull request Oct 18, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@nissa-seru