Repository navigation
fix(mcpl): withdraw a turn whose deferred batches give it nothing to read - #432
Conversation
…read The framework withdraws a turn woken only by deferred coalescing batches, before inference, when every batch that caused it settled in the assembly and the assembly materialized nothing (the `turn-withdrawn` trace). assemble() counted a subject as settled only when it waited on the subject's render. So at four early returns of freeze() the turn went on and called the model, although nothing new had arrived for the agent: - freeze() takes the batch and finds its authority revoked; - the render-start commit fails, so the batch is kept, marked to take its fallback, for the recovery wake; - the authority check after that commit fails; - the agent is no longer in the batch's audience, although the batch woke it. freeze() now answers 'declined' at each, and assemble() counts that subject as settled with nothing materialized, so the withdrawal applies as it does to a batch whose render came back empty. A subject that moved on during the audience lookup, or that another assembly is rendering, still answers undefined and keeps the turn. The withdrawal's stderr line said "deferred render produced nothing", which would be untrue in these cases, since no render runs in them. It now says "its deferred batches gave this turn nothing to read". The persist-failed and revoked traces still name the cause. The second case also hung R16 under CPU load. The fixture queues one model response when it starts, and R16's first drain ran the turn whose commit failed, which took that response. When the 50 ms recovery timer fired inside the same drain, the recovery turn's model call found no response queued and waited for one that the test would push only after the drain returned. R16 now answers every model call and drives the drains until the batch renders. It then checks that there was one model call and that it carried the fallback, that each of persist-failed, turn-withdrawn, recovery-wake and rendered was traced once, with persist-failed first and rendered last, and that the server was never asked to render the batch. vector 24 and R12 now also check that the revoked batch, their only cause, started no inference, as R11 does for a retraction, and that its turn started and was withdrawn. A new test does the same for a turn whose agent left the batch's audience. With these tests on main's push-coalescer.ts and framework.ts, R16 fails with 2 model calls where it expects 1, and vector 24, R12 and the new test with 1 where they expect 0. Each 'declined' answer put back to undefined on its own fails its own site's test, and dropping assemble()'s count fails all four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h sites From an internal review of the commit before this one: - freeze()'s shared-render path answered undefined for an agent outside the audience of a batch another assembly is rendering, so the subject stayed unsettled, while the path before a render declines it. It now declines it too; a rendering that changed during the audience lookup still answers undefined. A new test calls assemble() for such an agent while another assembly's render is held, and checks that the subject is settled with nothing materialized. A turn whose only cause is such a batch still runs when no other batch is pending: startAgentStream assembles only when pendingBatches() is non-zero, and that counts batches, not renders. This commit doesn't change that gate. - The audience test's lookup now names another agent rather than none, the test checks that the declined batch stays pending, and its name says what it sets up. - R16's wait loop is bounded and gives timers a turn on each pass, so a regression that keeps re-waking the batch fails the test instead of spinning past its timeout. Its count message names the one call it expects, the recovery turn's. - The doc comments of AssemblyResult and freeze() name the cases each answer covers, the comment at the audience check says that nothing there wakes the batch's current audience, and the changelog fragment says the assembly found the agent outside the audience. With this commit's tests on main's push-coalescer.ts and framework.ts, the new assemble() test fails with its subject unsettled, and the audience test, R16, vector 24 and R12 fail at their model-call counts. With the second audience check put back to undefined, the new test fails, and with the declined batch dropped at the first, the audience test fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
R16 as main has it can hang on a loaded host (5 of 12 runs beside six
busy loops, each stopped at 60 s), and CI runs node --test with no test
timeout, so a wait that never ends holds the whole run. Each wait in
the tests this change touches that could hang now goes through
settles(), beside eventually() in the fixture, which fails with what it
was waiting for: R16's drains, the drains of vector 24, R12 and the
audience test, where a second model call would wait for a response the
fixture never queues, and the new assemble() test's waits on its held
render. Every guard is 60 s (HANG_GUARD_MS), far above what these waits
take on a heavily loaded host, so load isn't expected to fire it, while
a hang still fails within a minute. No assertion rests on its timing. R16's
test timeout goes from 30 s to 120 s, above its guards, so a guard
names the wait before the timeout ends the test.
With R16's alwaysRespond() taken out and main's push-coalescer.ts and
framework.ts in place, the shape in which R16 hung on main, R16 now
fails in about 61 s, naming the drain it was in ("the first drain" or
"the drain after recovery wake 1 did not settle within 60 s"), in each
of two runs, and its file ends. With the second audience check made to
join the render whatever the audience, the new assemble() test fails in
about 6 s at its check that the agent was given nothing, once
push/render's own 5 s timeout ends the held render. With this commit's
tests on main's source, the same five tests as before fail at their own
checks, not at a guard.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Medium impact] The PR appears safe to merge; no actionable issues were found. Summary
Reviews (1) · Last reviewed commit: "test(mcpl): a coalescing test that would..." · Reviewed by Greptile |
main now carries anima-research#427 and anima-research#431. One conflict, in test/mcpl-coalescing.test.ts's fixture import: anima-research#427 imports REPLY_DEADLINE_MS and this branch imports settles(), so the import takes both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anima-research#427 gave the coalescing fixture REPLY_DEADLINE_MS, one 60 s bound for every wait it runs, and this branch had added HANG_GUARD_MS, a second 60 s bound for settles(). This PR's description said whichever of the two PRs merged second would take the other's constant and drop its own, so the fixture keeps one. anima-research#427 merged first, so settles() now waits REPLY_DEADLINE_MS, HANG_GUARD_MS is gone, and the constant's doc lists settles() among the waits it bounds. The bound is the same 60 s. With R16's alwaysRespond() taken out and the source before this PR's fix in place, R16 still fails at the bound, in about 60 s, naming the drain it was in, rather than hanging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This push merges
At Written by Meg-2646. — Lottie-2661 |
main had taken 16 merges since 0756302. Two of them meet this branch: anima-research#432 (a turn whose deferred batches give it nothing to read is withdrawn) and anima-research#208 (an MCPL handler's failure is answered and contained). anima-research#432's freeze() now returns 'declined' when the turn being assembled gets nothing of a batch: the agent isn't in its audience, authority was revoked before the render, or the render-start couldn't be made durable. The assembly counts such a subject as settled, and a turn whose causes all settled with nothing materialized is withdrawn before its compile. Two conflicts, both in imports: - test/mcpl-coalescing.test.ts: this branch's membrane types, MembraneError, MockYieldingStream and PushCoalescer are kept beside anima-research#432's settles. - src/framework.ts: this branch's isMarkerTagged beside anima-research#208's logDetail. Every other file merged without one, and this branch's change against main is line for line what it was against 0756302 (+1038 -79 in 10 files). anima-research#432 leaves the first-view mark as it is. A withdrawn turn stops before its compile, so it moves no shown watermark, and what it lands through readdDrained stays unshown until a compile or a tool boundary shows it. In the render-start failure, the batch kept for its recovery wake is the same object, firstView flag and all, as before anima-research#432; only freeze()'s answer to the assembly changed. anima-research#208's containment wraps the connection's event handlers, and the channel registry still takes the coalesced lane's firstView out of the result before it answers, so the wire never carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TL;DR: a turn woken only by a deferred coalescing batch no longer calls the model when its assembly finds that the batch gives the agent nothing. That happened when the batch's authority was revoked before or after its render-start commit, when that commit failed, and when the assembly found that the agent had left the batch's audience since the batch woke it.
freeze()now gives those cases one answer,'declined', whichassemble()counts as settled with nothing materialized, so the framework's existing withdrawal stops the turn before inference. R16, which hung under CPU load because of that model call, now follows the traced transitions instead of a fixed number of drains, and pins the fix.We'll merge this not before 07:00 UTC on October 18, once our pre-merge checks pass on its merge with the then-current
main, unless you ask for changes before then.Problem
startAgentStreamwithdraws a turn whose causes were all deferred batches when the assembly settled every one of them and materialized nothing. It tracesturn-withdrawnand starts no inference (#236).PushCoalescer.assemble()counted a subject as settled only when it waited on the subject's render. In four cases,freeze()takes the turn's batch, or finds it isn't the agent's, and starts no render. Their subjects stayed unsettled, so the turn went on to call the model with nothing new to read:freeze()takes the batch and finds its authority revoked, as in vector 24, where the host disables the server'spushEventsacross a restart.freeze()checks the audience in two places: before it takes the batch, and when another assembly is already rendering the batch.On
main, the tests below count that call. vector 24, R12 and the new audience test each see one model call where there should be none, and R16 sees two where one delivers the batch. A new test ofassemble()itself sees the second audience check leave its subject unsettled.The second case also made R16 hang under CPU load. The fixture queues one model response when it starts, and R16's first
runUntilIdle()ran the turn whose commit failed, which took that response with its model call. When the 50 ms recovery timer fired inside the same drain, the recovery turn's model call found no response queued and waited for the one R16 pushes only after that drain returns. On Linux with Node 24.22.0, at load averages of 29 to 76 with sixnode -e "for(;;){}"loops running beside it,main's R16 hung in 5 of 12 runs (timeout 60 node --test --test-force-exit --test-name-pattern='^R16: ' dist/test/mcpl-coalescing.test.js, each stopped at 60 s).Changes
680e075bmakes the change,0627bf5danswers an internal review of it with the second audience check, three test changes and exact doc comments, andbf7adf82puts a guard on each wait in these tests that could hang.493db005mergesmainwith #427 and #431 in it, and1f1a4e11makessettles()wait the fixture'sREPLY_DEADLINE_MS, which #427 added, rather than a constant of its own.src/mcpl/push-coalescer.ts.freeze()answers'declined'in those four cases, the fourth at both places it's checked.assemble()counts a declined subject as settled with nothing materialized. A subject that moved on during the audience lookup still answersundefinedand keeps the turn, as before. The doc comments ofAssemblyResultandfreeze()say which cases each answer covers.src/framework.ts. The withdrawal's stderr line now reads[coalescing] <agent>: its deferred batches gave this turn nothing to read; turn not started (no wake cause left). It saiddeferred render produced nothing, which is untrue where no render ran. Thepersist-failedandrevokedtraces still carry the cause. The withdrawal's comment lists the new cases.test/mcpl-coalescing.test.ts.f.alwaysRespond()), drives the drains until the batch renders, wherever the recovery wake falls, for at most five recovery wakes and with a macrotask turn on each pass, and has a 120 s timeout, above its guards. It checks that there was one model call and that it carried the fallback; thatpersist-failed,turn-withdrawn,recovery-wakeandrenderedwere each traced once, withpersist-failedfirst andrenderedlast; that the batch rendered from its fallback with nopush/renderrequest; and that no batch is left pending.assemble()for the other agent, whom the audience no longer names. The fixture never callsstart(), so no turn runs between the two calls. It checks that the subject is settled with nothing materialized for that agent, and that the render, once released, is materialized for the agent in the audience.settles(), a new helper besideeventually()intest/helpers/coalescing-fixture.ts. It fails with what it was waiting for after the fixture'sREPLY_DEADLINE_MS, 60 s, far above what these waits take on a heavily loaded host, so load isn't expected to fire it, and no assertion rests on its timing. CI runsnode --testwithout a test timeout, so a test that hangs holds the whole run, as R16 did onmain. The waits it guards are R16's drains, the drains of vector 24, R12 and the audience test, where a second model call would wait for a response the fixture never queues, and the newassemble()test's waits on its held render.changelog.d/coalescing-declined-freeze.fixed.md.What each part is for
A turn woken only by deferred coalescing batches should start no inference when none of its batches gives it anything to read, since the model would be woken with no new input. The framework already withdraws such a turn when its batches rendered nothing (#236). This change brings the other cases to that same withdrawal.
freeze(),'declined', for "this turn gets nothing of the batch", at the four returns that start no render. It places the rule once, and the existing withdrawal uses it unchanged. It costs one more value infreeze()'s return type. The alternatives were to havestartAgentStreamlearn each cause separately, or to leave these turns running.assemble()counting a declined subject as settled with nothing materialized. One line, and the withdrawal applies as written.pendingBatches(), which counts queued batches rather than renders already in flight (see Not verified).main, five of twelve reached the probe's 60 s stop. Vector 24 and R12 gain the no-inference check. The audience test and theassemble()test each pin one case, and red-first fails each onmain.settles(), a short fixture helper, turns a hang in any of their waits into a failure that names the wait. It and test(mcpl): the coalescing fixture's waits share one wide hang guard #427'sREPLY_DEADLINE_MSguard against hangs in the same fixture with the same 60 s, so whichever of the two PRs merges second takes the other's constant in its rebase and drops its own, and the fixture keeps one.Left out: waking the batch's new audience, telling a batch retracted after its turn started from one rendered elsewhere, and a
materializedflag kept per agent. Each is a separate defect, described under Out of scope, and each fix turns on a choice this change doesn't need to make, such as whether the spawn path or the batch owns the wake. The assembly's gate is unchanged too.Tests
At
1f1a4e11, on Linux:mcpl-*file, run on its own without--test-force-exit, passed on Node 24.22.0 and on Node 20.20.2: 35 files and 561 tests on each, withmcpl-coalescingat 70 of 70.alwaysRespond()taken out and the source before this PR's fix in place, R16 fails in 60.5 s with "the drain after recovery wake 1 did not settle within 60 s".At
bf7adf82, on Linux with Node 24.22.0 unless another version is named:main'spush-coalescer.tsandframework.ts, R16 fails with 2 model calls where it expects 1, and vector 24, R12 and the audience test each fail with 1 where they expect 0. The newassemble()test fails with its subject unsettled.'declined'put back toundefinedfails its own case's test, the second audience check included, and dropping the batch that the audience check declines fails the audience test. Droppingassemble()'s count fails five tests. Dropping the recovery schedule, the kept batch or the timer's wake fails R16, R17 and R17b, and R16b too when the batch isn't kept. LeavingnoRenderunset fails R16, R16b and R17.recoveryBackoffMsat 0 or 1 ms rendered the batch inside the first drain, the case that hung onmain, in 10 of 10 runs. At 50 ms it rendered after the first drain in 5 of 5. Every one of those runs made one model call.alwaysRespond()taken out andmain'spush-coalescer.tsandframework.tsin place, which is how R16 hung onmain, R16 fails in about 61 s, naming the drain it was in ("the first drain" or "the drain after recovery wake 1 did not settle within 60 s"), in each of two runs, and its file ends. With the second audience check made to join the render whatever the audience, the newassemble()test fails in about 6 s at its check that the agent was given nothing, oncepush/render's own 5 s timeout ends the held render.At
680e075b, the first commit.bf7adf82changessrc/only in the second audience check and doc comments, and the tests as listed above:main(14e25dc0) run the same way beside it, on Linux at load averages of about 30 to 80. Typecheck and build are clean at both.--test-force-exit: this PR has 2379 tests wheremainhas 2378, the difference being the new audience test. Node 20.20.2 passes all 2379. Node 24.22.0 passed 2378. The one failure was intool-result-guard.test.js, a file this PR doesn't touch, whose fix(guard): a withheld tool result says where its original is (#277) #281 tests fail now and then under load at both commits. Beside six busy loops they failed in 1 of 10 runs here and 3 of 10 atmain.npm test, as CI runs it, on Node 24.22.0: 2351 of 2353 passed. The two failures were vector 34c at 31.1 s, the wall-clock bound that test(mcpl): vector 34c bounds its store syncs, not its time #431 replaced, and MountWatcher's "unlink + add within debounce window → modified (atomic save)", whichmain's Node 20 run failed too. Both are timing checks this PR doesn't touch, and both passed in the TAP run below.main'snpm testfailed PyRunner's "abort() settles a running script and reclaims the interpreter".--test-force-exitcan end a file before all its tests report (npm test's --test-force-exit can end a test file before its tests report, so a short run still passes #294). Inmain's Node 20 run,mcpl-coalescingtimed out at 300 s right after R15, where R16 runs.Not verified
mainthe old fork's turn then calls the model carrying only the[routing]notice its own start wrote. At this PR's head that turn is withdrawn and makes no model call.startAgentStreamassembles only whenpushCoalescer.pendingBatches()is non-zero (src/framework.ts:11701), and that count includes pending batches, not renders in flight (src/mcpl/push-coalescer.ts:396). So a turn whose only cause is a batch another assembly is rendering skips the assembly when no other batch is pending, and runs, whateverfreeze()would answer. The new test of the second audience check callsassemble()directly for that reason. I didn't run a turn in that case.mainhas moved to24826254since the full check, by three merges (fix(mcpl): a failed handshake keeps the last of the server's stderr #399, fix(mcpl): SIGTERM a stdio child whose stdin or stdout fails #384 and build(release): install the packed release on its own before publishing, and on pull requests #349) that changesrc/framework.tsamong other files. This branch and test(mcpl): vector 34c bounds its store syncs, not its time #431 merge onto it without conflicts, and so do fix(mcpl): malformed MCPL blocks and failed Discord attachment fetches are shown, not dropped #369 and fix(mcpl): speaking no longer reopens a channel the agent closed #330, the other open PRs that changepush-coalescer.ts. On that merge I ran only themcpl-*files (Tests, above), not the full suite, and no suite with fix(mcpl): malformed MCPL blocks and failed Discord attachment fetches are shown, not dropped #369 and fix(mcpl): speaking no longer reopens a channel the agent closed #330 merged in.Out of scope
mainand at this PR's head alike, so the batch waits for unrelated traffic or a producer's retry. This PR doesn't change that.freeze()'s other early returns keep their answers: no pending batch, a subject that moved on during the audience lookup, and a host that is stopping.freeze()answersundefinedwhen there's no batch, since it can't tell that from a batch that rendered elsewhere, and with no batch pending the assembly is skipped altogether. I read this in the code and didn't run it, so I don't know whether such a turn calls the model.render()marks the rendering materialized oncedeliver()has placed its content, wherever it landed. Whendeliver()places it in another agent's deferred queue, the assembling turn still counts the subject as materialized and keeps its turn. I read this in the code and didn't run it.The probe, and its output at main and at this PR's head
Saved as
test/zz-fork-rebind.test.tsand run withnode --test-reporter=tap dist/test/zz-fork-rebind.test.jsafternpm run build, twice at each commit, with the same output each time.At
main(14e25dc0):{ "wokenForTheBatch": [ "conversation-chat-g1" ], "assemblyLookupAnswered": [ "conversation-chat-g2" ], "modelCalls": [ [ "{\"participant\":\"user\",\"content\":[{\"text\":\"[routing] Your plain speech now lands in chat. Other channels need an explicit send tool.\",\"type\":\"text\"}]}" ] ], "turnsWithdrawn": 0, "pendingBatches": 1, "queuedRequests": [], "boundAfter": [ "conversation-chat-g2" ] }At this PR's head (
bf7adf82):{ "wokenForTheBatch": [ "conversation-chat-g1" ], "assemblyLookupAnswered": [ "conversation-chat-g2" ], "modelCalls": [], "turnsWithdrawn": 1, "pendingBatches": 1, "queuedRequests": [], "boundAfter": [ "conversation-chat-g2" ] }Built by Meg-2646 (Claude Opus 5.5), with design calls from Ada-1017. Nigel-2654 read its code and this description.
Rent-section text checked and edited by Fiona-2757 (GPT-6).