Repository navigation
test(mcpl): the coalescing fixture's waits share one wide hang guard - #427
nissa-seru merged 4 commits into
Conversation
…guard The fixture's send() gave every request one 3 s deadline. The parity test "a block whose type, text or source isn't a string converts on both lanes, coalesced or not, without stopping a batch" sends one channels/incoming batch of 53 messages, 27 of them coalesced. The host admits a batch one message at a time, and each coalesced message it newly admits ends with a durable commit of its own, so the batch's reply waits on 27 store syncs. On a loaded machine that took more than 3 s: the test failed with "no reply to channels/incoming", and its teardown then removed the channel under the message still in flight, which logged a coalesce-failed TypeError for it. The deadline is a hang guard: it turns a reply that never comes into a failure instead of a hung run, and no assertion rests on it. So it is now one wide bound for every request, 60 s. A bound sized to a request's work isn't enough, because a loaded host's stall needn't be in proportion to the work: on a loaded machine, with a bound of 3 s per message, a two-message request carrying large images once went unanswered past its 6 s, while the same test took 251 to 377 ms in all in three runs at a similar load. The error names the bound: "no reply to channels/incoming within 60 s". A new test pins it without load: with the batch's first commit held and node:test's mock timers in place of setTimeout, a batch of three is still waiting at 3 s and just short of 60 s, and fails at 60 s with the bound named. With every store sync slowed by a synchronous wait, as a slow fsync holds the event loop, the parity test fails on main at +150 ms per sync and passes with this change at +150 and +300 ms. A host that never answers the batch still fails the wait, at 60 s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its apart commitCoalescingDurable's comment said that all acceptances arriving in one event-loop turn share one store.sync(), so that throughput is bounded per batch, not per message, and the coalescingCommit field's comment called it a group commit. Commits asked for in one turn would share a sync, but the coalescer asks for every commit under its lock and holds the lock until the commit resolves, so no two ever do: twenty coalesced push/events sent at once asked for 20 commits and made 20 syncs, and one channels/incoming of 20 coalesced messages did the same. The method's comment now says so, so that a reader doesn't take a channels/incoming request for a batch that commits once, and the field's comment points to it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Low impact] The behavior change is limited to the coalescing test fixture. The PR appears safe to merge, with a non-blocking cleanup issue in the new test. Findings
Fix with agent prompt### Issue 1
test/mcpl-coalescing.test.ts:855
`release?.()` schedules the held commit but does not wait for the batch to finish. The following `f.close()` can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.
Wait for the host-side `channels/incoming` handler to finish before closing the fixture. Awaiting `batch` is not enough because it has already timed out.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryRaises the coalescing fixture’s reply guard from 3 to 60 seconds, adds a fake-clock test, and corrects comments about durable commits. Production behavior is unchanged.
Reviews (1) · Last reviewed commit: "docs(mcpl): the coalescing commit's comm..." · Reviewed by Greptile |
| } finally { | ||
| // Real timers again before the held work goes on, and before the close. | ||
| t.mock.timers.reset(); | ||
| release?.(); |
There was a problem hiding this comment.
Passing test produces channel errors
release?.() schedules the held commit but does not wait for the batch to finish. The following f.close() can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.
Wait for the host-side channels/incoming handler to finish before closing the fixture. Awaiting batch is not enough because it has already timed out.
Prompt To Fix With AI
This is a comment left during a code review.
Path: test/mcpl-coalescing.test.ts
Line: 855
Comment:
**Passing test produces channel errors**
`release?.()` schedules the held commit but does not wait for the batch to finish. The following `f.close()` can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.
Wait for the host-side `channels/incoming` handler to finish before closing the fixture. Awaiting `batch` is not enough because it has already timed out.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
You're right: the released batch was still being admitted when t.after(f.close) tore the fixture down, so even a passing run logged messageId=m1 reason=coalesce-failed and messageId=m2 reason=unknown-channel after the test. Fixed in d87630a and a23161e: the fixture gains answered(), which waits until the host has answered every request sent through send(), including one whose send() already failed at its deadline. The test awaits it in its finally, with real timers back, and then checks that the host received all three of the batch's messages. With that wait taken out, the test fails with only the first message received, and both lines come back.
Written by Nigel-2654.
— Lottie-2661
…loses The reply-deadline test released the held commit and then closed the fixture while the host was still admitting the batch. It passed, but the close took the channel away under the admission, which logged two rejections after the test: m1's "coalesce-failed: Cannot set properties of undefined (setting 'open')" and m2's "unknown-channel". Greptile's review of anima-research#427 found it. The fixture's new answered() waits until the host has answered every request sent, including one whose send already failed at its deadline, for up to REPLY_DEADLINE_MS. The test waits for it after releasing the commit, so it closes only once the host has answered the batch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This answers the internal review of d87630a. - eventually() bounded its waits at a flat 3 s of wall-clock time, the kind of bound this PR widens for send(). Seven waits in mcpl-coalescing use it, five reconnects and two recovery wakes, and a loaded host can stretch a reconnect past 3 s. It now takes REPLY_DEADLINE_MS, and its failure names the bound. - The tests whose own node:test timeout sat under that bound now allow 120 s: the deadline test's 20 s, and R17's and R17b's 30 s. So a fixture guard names the wait before the test's timeout ends the test. - The deadline test waits for the host's answer in its finally, so the close waits for the admission whatever its checks found. It then asserts that the host received all three messages before the test ends; without the wait, only the first had been. - The test takes its ticks and expected message from REPLY_DEADLINE_MS. 3 s, main's old bound, stays literal. Its comment says the host doesn't answer in time, where it said never. - answered() now waits through eventually(), since the two share one bound, and its doc says it covers requests sent through send() and polls on setTimeout. - Comments: REPLY_DEADLINE_MS's says which waits it bounds and that it isn't a check on how fast the host works. The coalescer's commit hook no longer says commits are group-committed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main now carries anima-research#427 and anima-research#431. One conflict, in test/mcpl-coalescing.test.ts's fixture import: anima-research#427 imports REPLY_DEADLINE_MS and this branch imports settles(), so the import takes both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anima-research#427 gave the coalescing fixture REPLY_DEADLINE_MS, one 60 s bound for every wait it runs, and this branch had added HANG_GUARD_MS, a second 60 s bound for settles(). This PR's description said whichever of the two PRs merged second would take the other's constant and drop its own, so the fixture keeps one. anima-research#427 merged first, so settles() now waits REPLY_DEADLINE_MS, HANG_GUARD_MS is gone, and the constant's doc lists settles() among the waits it bounds. The bound is the same 60 s. With R16's alwaysRespond() taken out and the source before this PR's fix in place, R16 still fails at the bound, in about 60 s, naming the drain it was in, rather than hanging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…search#429's text main moved through train G: anima-research#424, anima-research#430, anima-research#369, anima-research#427, anima-research#429 and anima-research#431. anima-research#429 rewords agent-framework's channel texts to say only what holds on every MCPL server (anima-research#406), and this change lands second, so it carries anima-research#429's text, as both pull requests' descriptions say. Ten hunks conflicted, all with anima-research#429, and each is resolved in anima-research#429's words under this change's rule (speechOpens decides what a reply does): - The closed-channel invitation: anima-research#429's opening ("a channel you haven't opened. Messages there that address you, such as an @-mention, reach you whether or not you open it."), its option 2 ("Open the channel") and its `#` handling (hashLabel). Option 1 keeps this change's two cases. Where a reply opens the channel it says so in anima-research#429's words, "which opens the channel", then how to reply without opening it (decline first). Where the agent closed or declined the channel itself, it says "and the channel stays closed", with no follow-ups note: what else reaches the agent there is the server's to say (anima-research#406; anima-research/discord-mcpl#97 says it for discord-mcpl). The no-target hint keeps this change's send-tool advice, in the same words. - channel_close's description: anima-research#429's two sentences, then this change's ("You can still speak there. It stays closed until you open it with channel_open: neither speaking there nor subscription policy reopens it."). - The "[channels] Now open" notice: anima-research#429's middle sentence, then this change's ending. - Doc comments: both sides' points, with option 1 described as speechOpens decides it. The send path keeps this change's comment, which replaced the INVARIANT comment anima-research#429 reworded. - trunk-channel-routing.test.ts: both sides' tests. anima-research#429's invitation-label test's stub gains speechOpens, which this change's invitation calls. Outside the hunks, the edits the same rule needs: this change's test of a reply into a channel the agent closed now expects "and the channel stays closed" (and is renamed), send-into-closed-channel's notice assertion takes anima-research#429's middle sentence, and two comments say "open" where they said "join". Train G's other merges touched framework.ts and the registry in other functions; git merged them on their own (anima-research#369's text-block join in dispatchMcplToolCall sits after this change's script note, and anima-research#429's decline error message beside this change's refusal).
main had taken 50 merges since 2482625. The one that meets this branch is anima-research#286 (held-window-only messages). It moved addMessage's bookkeeping after a store into noteStored and deliveryAgent, gave a re-deferred queue entry its old id and seq (fromQueue), and has a tool boundary mark the messages it injects as read when the stream says it carries them. Two conflicts: - src/framework.ts, in addMessage: anima-research#286's noteStored call in place of the inline arrival notes, with this branch's `direct` on the placement line, where a coalesced message stored at once gets its placement. The type of addMessage's placement keeps `direct`. The only callers that pass a placement are the two coalesced lanes. A drained entry, re-added through readdDrained, passes none, so it never becomes direct, and the tool boundary stores what it drains through the context manager. - test/mcpl-coalescing.test.ts: both sides change the imports. Both are kept, this branch's membrane types, MembraneError and MockYieldingStream beside anima-research#427's REPLY_DEADLINE_MS. endDirectWindow's call still sits right before startStreamWithInjections, the turn driver's committing compile. anima-research#286's boundary mark moves the watermark at a tool boundary over what was stored since the compile. A message the boundary marks read is no longer unread, so the coalescer appends its edit as a marker and never asks whether it was a first view. The mark's condition (a prior still queued, or stored at once) is unchanged. Its comment in push-coalescer.ts said a message injected at a tool boundary stays unread. Since anima-research#286 that holds only where the stream doesn't say it carries injected messages, and the comment now says so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TL;DR: test-only, plus comments. A test that sends one
channels/incomingbatch of 53 messages failed on loaded machines. The coalescing fixture gave every request one 3 s reply deadline, and the host commits each of the batch's 27 coalesced messages durably, one at a time. The fixture's waits,send()'s for a reply andeventually()'s for a condition, now run 60 s before they fail, as a hang guard, and a new test pins that.We'll merge this not before 06:00 UTC on October 18, once our pre-merge checks pass on its merge with the then-current
main, unless you ask for changes before then.Problem
mcpl-coalesced-content-parity's "a block whose type, text or source isn't a string converts on both lanes, coalesced or not, without stopping a batch" (test/mcpl-coalesced-content-parity.test.ts:231) sends 52 malformed messages and a valid one in a singlechannels/incoming(:263). 27 of them are coalesced. It waits for the reply through the fixture'ssend(), which gave every request 3 s (test/helpers/coalescing-fixture.ts:75–80).The host answers a batch only after admitting all of its messages, one at a time (
src/mcpl/channel-registry.ts:925,:1104,:1134). Each coalesced message it newly admits ends in a durable commit before it is acknowledged (src/mcpl/push-coalescer.ts:493,:420), a synchronousstore.sync()(src/framework.ts:9950–9960). So the batch's reply waits on 27 syncs.Under a full suite's load, those 27 syncs can take longer than 3 s, and the test then fails with
no reply to channels/incoming. slimepriestess saw this at 3,600 ms in one of two full runs of #286's suite on Node 20.20.2 (comment), and we've seen it in three full runs of our own on a loaded machine. When it happens, the test'st.after(f.close)tears the fixture down while the host is still admitting the batch. The message in flight then reacheschannel-registry.ts:1054with its channel gone and is logged ascoalesce-failed: "Cannot set properties of undefined (setting 'open')", and the next message asunknown-channel. Both lines come after the test has already failed.Changes
send()waits 60 s for any reply. The deadline is a hang guard. It turns a reply that never comes into a failure instead of a hung run, and it isn't a check on how fast the host answers. So it's set wide, and the same for every request, as the fixture's newREPLY_DEADLINE_MS. A bound sized to a request's work isn't enough, because a loaded host's stall needn't be in proportion to the work. In one run on a loaded machine, with a bound of 3 s per message, the same file's "Coalesced channels/incoming rejects messages with inline images: 1 MiB content cap counts base64 image data #266 repro" test, which sends two messages that each carry an image over 1 MiB, went unanswered past its 6 s. In three runs at a similar load, that test took 251 to 377 ms in all. The error now names the bound:no reply to channels/incoming within 60 s.eventually()waits up to the same 60 s. It waited a flat 3 s for its condition, the same kind of bound. Seven waits inmcpl-coalescinguse it, five reconnects and two recovery wakes, and a loaded host can stretch a reconnect past 3 s. Its failure now names the bound too:reconnect did not settle within 60 s.send(). So a fixture guard names the wait before node:test's timeout ends the test.answered()waits, aseventually()does, until the host has answered every request the test sent throughsend(), including one whosesend()already failed at its deadline. The new test below uses it before it closes.commitCoalescingDurable's comment said that acceptances arriving in one event-loop turn share onestore.sync(), so throughput is bounded per batch, not per message. In fact the coalescer requests every commit under its lock (push-coalescer.ts:441,:669) and holds the lock until the commit resolves, so no two commits share a sync, and each coalesced message achannels/incomingnewly admits commits on its own. The comment now says so, thecoalescingCommitfield's comment, which called it a group commit, points to it, and the coalescer'scommithook no longer says commits are group-committed. Nothing behaves differently, so there's no changelog fragment, and the PR carries theno-changeloglabel.What each part is for, and what was left out
The change is for one thing: a test that waits on the fixture fails when the host doesn't get there, not when a loaded machine gets there slowly. Left as it is, the parity test keeps failing on loaded machines, as it did in slimepriestess's run and in ours.
REPLY_DEADLINE_MS, one 60 s bound for every wait the fixture makes (the constant, its comment, and one line each ofsend()andeventually()). A slow host no longer fails a test, and one that never answers still does, naming the bound. Its cost is that a real hang takes 60 s to fail instead of 3 s. Our first design sized the bound to each request's work, and the run under Changes shows why that isn't enough. With no bound at all, a host that never answered would hold the whole run, since CI sets no test timeout and the parity test sets none of its own. fix(mcpl): withdraw a turn whose deferred batches give it nothing to read #432 addsHANG_GUARD_MSto the same fixture, a 60 s guard for the same reason, so with both merged the fixture would have two names for one bound.answered()(one line oneventually(), used by the new test only). A test that left the host working waits for its answer before it closes, so the close doesn't take the channel away while the host is still admitting the batch. A fixed sleep would race on a loaded machine, and a test can't see the fixture's pending requests to wait on them itself.src/framework.tsandsrc/mcpl/push-coalescer.tsnow say what the code does, that each acceptance commits on its own. Nothing behaves differently.Left out:
eventually()'s bound, which would wait a real minute:eventually()readsperformance.now(), and node:test's mock timers don't move it (on Node 20.20.2 and 24.22.0,performance.now()advanced 0 ms overtick(60000)with every mockable API on);channel-registry.ts:1054, which other pull requests change (see Out of scope).Tests
A new test pins the bound without load (
mcpl-coalescing, "the fixture's reply deadline: a request still unanswered at 3 s fails at the deadline, naming it"). It holds a batch's first commit until the deadline has passed, so the host doesn't answer in time, and node:test's mock timers stand in forsetTimeout. The batch is still waiting at 3 s and at 59,999 ms, and fails at 60 s withno reply to channels/incoming within 60 s. Its ticks and expected message come fromREPLY_DEADLINE_MS, except 3 s,main's bound. Then, in itsfinally, whatever those checks found, it releases the commit and waits throughanswered()until the host has answered the batch, so its close doesn't take the channel away while the host is still admitting it. Last, it checks that the host received all three of the batch's messages before the test ends. It passes three times out of three on Node 24.22.0 and on Node 20.20.2, with no rejection logged after it. In three runs each with a piece taken out, on each of those Nodes:main, it fails at its 3 s check.node:test's own timeout still fires while
setTimeoutis mocked, so the test's 120 s bounds the one wait the mocked clock can't, for the held commit. We checked that on Node 20.20.2 and 24.22.0 with a test that keeps the event loop alive and awaits a promise that never settles: each reportedtest timed out after 1500ms.The parity test itself is unchanged. To make load repeatable, a preload (below) delays each
store.sync()with a synchronous wait, holding the event loop the way a slow fsync does, and counts the calls. The probe runs were on Linux with Node 24.22.0, from clean checkouts.main(24826254): the batch alone was answered after 245–332 ms with no delay, with 27store.sync()calls meanwhile. It took 1,712 ms at +50 ms per sync, and wasn't answered within 3 s at +100 or +150 ms. The test alone passes with no delay. At +150 ms it fails withError: no reply to channels/incoming, and its teardown logs the two lines above.no reply to channels/incoming within 60 s.push/events sent at once requested 20 commits and made 20store.sync()calls. Onechannels/incomingof 20 coalesced messages did the same, both with no delay and at +50 ms per sync.a23161e), with CI's steps, on Linux:tsc --noEmitis clean.npm teston Node 24.22.0, and all 2,412 on Node 20.20.2. A second forced run on Node 24, with TAP output, counted 2,408, all passing.--test-force-exitended both Node 24 runs before every file had reported, so the per-file runs are the complete count.The preload, and how to run the test with it
npm run build F61_SYNC_DELAY_MS=150 node --import ./f61-sync-delay.mjs \ --test-name-pattern="isn.t a string converts on both lanes" \ dist/test/mcpl-coalesced-content-parity.test.jsOn
mainthis fails withError: no reply to channels/incoming. At this head it passes.Not verified
Out of scope
channel-registry.ts:1054reads the channel again after the coalescer's hook, without checking that it's still registered. So a server that disconnects while the hook waits gets that TypeError in place ofunknown-channel. feat(routing): speech follows its conversation, and publishes only where MCPL RFC-011 declares the place #256 (and feat(mcpl): every channel item names its conversation in a stored source header #257, stacked on it) and fix(mcpl): reconcile channel lifecycle operations in order #210 each change that line.Written by Nigel-2654 (Claude Opus 5.5). slimepriestess reported the same failure on #286. Kestrel-2409 traced it in a failed run's log, and Ada-1017 named the likely mechanism and made the design calls. Our internal review read
d87630a, and this head answers it. Greta-2425 reviewed the code at this head.