Skip to content

test(mcpl): the coalescing fixture's waits share one wide hang guard - #427

Merged
nissa-seru merged 4 commits into
anima-research:mainfrom
nissa-seru:test/coalescing-batch-reply-deadline
Oct 18, 2026
Merged

nissa-seru merged 4 commits into
anima-research:mainfrom
nissa-seru:test/coalescing-batch-reply-deadline

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: test-only, plus comments. A test that sends one channels/incoming batch of 53 messages failed on loaded machines. The coalescing fixture gave every request one 3 s reply deadline, and the host commits each of the batch's 27 coalesced messages durably, one at a time. The fixture's waits, send()'s for a reply and eventually()'s for a condition, now run 60 s before they fail, as a hang guard, and a new test pins that.

We'll merge this not before 06:00 UTC on October 18, once our pre-merge checks pass on its merge with the then-current main, unless you ask for changes before then.

Problem

mcpl-coalesced-content-parity's "a block whose type, text or source isn't a string converts on both lanes, coalesced or not, without stopping a batch" (test/mcpl-coalesced-content-parity.test.ts:231) sends 52 malformed messages and a valid one in a single channels/incoming (:263). 27 of them are coalesced. It waits for the reply through the fixture's send(), which gave every request 3 s (test/helpers/coalescing-fixture.ts:75–80).

The host answers a batch only after admitting all of its messages, one at a time (src/mcpl/channel-registry.ts:925, :1104, :1134). Each coalesced message it newly admits ends in a durable commit before it is acknowledged (src/mcpl/push-coalescer.ts:493, :420), a synchronous store.sync() (src/framework.ts:9950–9960). So the batch's reply waits on 27 syncs.

Under a full suite's load, those 27 syncs can take longer than 3 s, and the test then fails with no reply to channels/incoming. slimepriestess saw this at 3,600 ms in one of two full runs of #286's suite on Node 20.20.2 (comment), and we've seen it in three full runs of our own on a loaded machine. When it happens, the test's t.after(f.close) tears the fixture down while the host is still admitting the batch. The message in flight then reaches channel-registry.ts:1054 with its channel gone and is logged as coalesce-failed: "Cannot set properties of undefined (setting 'open')", and the next message as unknown-channel. Both lines come after the test has already failed.

Changes

  • send() waits 60 s for any reply. The deadline is a hang guard. It turns a reply that never comes into a failure instead of a hung run, and it isn't a check on how fast the host answers. So it's set wide, and the same for every request, as the fixture's new REPLY_DEADLINE_MS. A bound sized to a request's work isn't enough, because a loaded host's stall needn't be in proportion to the work. In one run on a loaded machine, with a bound of 3 s per message, the same file's "Coalesced channels/incoming rejects messages with inline images: 1 MiB content cap counts base64 image data #266 repro" test, which sends two messages that each carry an image over 1 MiB, went unanswered past its 6 s. In three runs at a similar load, that test took 251 to 377 ms in all. The error now names the bound: no reply to channels/incoming within 60 s.
  • eventually() waits up to the same 60 s. It waited a flat 3 s for its condition, the same kind of bound. Seven waits in mcpl-coalescing use it, five reconnects and two recovery wakes, and a loaded host can stretch a reconnect past 3 s. Its failure now names the bound too: reconnect did not settle within 60 s.
  • Three tests' own timeouts go from under 60 s to 120 s: the new test's, and R17's and R17b's, which wait through send(). So a fixture guard names the wait before node:test's timeout ends the test.
  • The fixture's new answered() waits, as eventually() does, until the host has answered every request the test sent through send(), including one whose send() already failed at its deadline. The new test below uses it before it closes.
  • commitCoalescingDurable's comment said that acceptances arriving in one event-loop turn share one store.sync(), so throughput is bounded per batch, not per message. In fact the coalescer requests every commit under its lock (push-coalescer.ts:441, :669) and holds the lock until the commit resolves, so no two commits share a sync, and each coalesced message a channels/incoming newly admits commits on its own. The comment now says so, the coalescingCommit field's comment, which called it a group commit, points to it, and the coalescer's commit hook no longer says commits are group-committed. Nothing behaves differently, so there's no changelog fragment, and the PR carries the no-changelog label.

What each part is for, and what was left out

The change is for one thing: a test that waits on the fixture fails when the host doesn't get there, not when a loaded machine gets there slowly. Left as it is, the parity test keeps failing on loaded machines, as it did in slimepriestess's run and in ours.

  1. REPLY_DEADLINE_MS, one 60 s bound for every wait the fixture makes (the constant, its comment, and one line each of send() and eventually()). A slow host no longer fails a test, and one that never answers still does, naming the bound. Its cost is that a real hang takes 60 s to fail instead of 3 s. Our first design sized the bound to each request's work, and the run under Changes shows why that isn't enough. With no bound at all, a host that never answered would hold the whole run, since CI sets no test timeout and the parity test sets none of its own. fix(mcpl): withdraw a turn whose deferred batches give it nothing to read #432 adds HANG_GUARD_MS to the same fixture, a 60 s guard for the same reason, so with both merged the fixture would have two names for one bound.
  2. Three test timeouts at 120 s. A test's own timeout under 60 s would end the test before the fixture's guard could name what it waited on.
  3. answered() (one line on eventually(), used by the new test only). A test that left the host working waits for its answer before it closes, so the close doesn't take the channel away while the host is still admitting the batch. A fixed sleep would race on a loaded machine, and a test can't see the fixture's pending requests to wait on them itself.
  4. The comments in src/framework.ts and src/mcpl/push-coalescer.ts now say what the code does, that each acceptance commits on its own. Nothing behaves differently.
  5. The new test pins the bound and its message without needing load, through node:test's mock timers, and pins the wait for the host's answer.

Left out:

  • a test that reproduces the load, which CI can't repeat reliably; the preload below does it by hand;
  • a bound per message, the first design;
  • a test of eventually()'s bound, which would wait a real minute: eventually() reads performance.now(), and node:test's mock timers don't move it (on Node 20.20.2 and 24.22.0, performance.now() advanced 0 ms over tick(60000) with every mockable API on);
  • channel-registry.ts:1054, which other pull requests change (see Out of scope).

Tests

A new test pins the bound without load (mcpl-coalescing, "the fixture's reply deadline: a request still unanswered at 3 s fails at the deadline, naming it"). It holds a batch's first commit until the deadline has passed, so the host doesn't answer in time, and node:test's mock timers stand in for setTimeout. The batch is still waiting at 3 s and at 59,999 ms, and fails at 60 s with no reply to channels/incoming within 60 s. Its ticks and expected message come from REPLY_DEADLINE_MS, except 3 s, main's bound. Then, in its finally, whatever those checks found, it releases the commit and waits through answered() until the host has answered the batch, so its close doesn't take the channel away while the host is still admitting it. Last, it checks that the host received all three of the batch's messages before the test ends. It passes three times out of three on Node 24.22.0 and on Node 20.20.2, with no rejection logged after it. In three runs each with a piece taken out, on each of those Nodes:

  • With the bound at 3 s, as on main, it fails at its 3 s check.
  • With the bound left out of the error, it fails at its last check.
  • Without the wait for the host's answer, it fails at its last assertion, with only the first message received, and its close leaves the two lines described under Problem.

node:test's own timeout still fires while setTimeout is mocked, so the test's 120 s bounds the one wait the mocked clock can't, for the held commit. We checked that on Node 20.20.2 and 24.22.0 with a test that keeps the event loop alive and awaits a promise that never settles: each reported test timed out after 1500ms.

The parity test itself is unchanged. To make load repeatable, a preload (below) delays each store.sync() with a synchronous wait, holding the event loop the way a slow fsync does, and counts the calls. The probe runs were on Linux with Node 24.22.0, from clean checkouts.

  • On main (24826254): the batch alone was answered after 245–332 ms with no delay, with 27 store.sync() calls meanwhile. It took 1,712 ms at +50 ms per sync, and wasn't answered within 3 s at +100 or +150 ms. The test alone passes with no delay. At +150 ms it fails with Error: no reply to channels/incoming, and its teardown logs the two lines above.
  • At this head: the test alone passes at +0, +150 and +300 ms per sync, and the whole file passes at +150 ms. The batch is answered after 4.3 s at +150 ms and after 8.3 s at +300 ms. When the coalescer's commits never resolve, so the host never answers, the wait fails after 60 s with no reply to channels/incoming within 60 s.
  • The comment's claim: 20 coalesced push/events sent at once requested 20 commits and made 20 store.sync() calls. One channels/incoming of 20 coalesced messages did the same, both with no delay and at +50 ms per sync.
  • The full suite at this head (a23161e), with CI's steps, on Linux:
    • tsc --noEmit is clean.
    • Each test file run on its own passes, 2,412 of 2,412 in 152 files on both Node 24.22.0 and Node 20.20.2.
    • CI's unforced step passes 105 of 105 on both.
    • CI's forced run passes every test it counted: 2,383 in npm test on Node 24.22.0, and all 2,412 on Node 20.20.2. A second forced run on Node 24, with TAP output, counted 2,408, all passing. --test-force-exit ended both Node 24 runs before every file had reported, so the per-file runs are the complete count.
The preload, and how to run the test with it
// f61-sync-delay.mjs: each framework's store.sync() takes F61_SYNC_DELAY_MS
// longer, as a synchronous wait (store.sync() is synchronous, so a slow fsync
// holds the event loop the same way), and is counted per framework.
// Use from the repository root after a build: node --import ./f61-sync-delay.mjs <file>
import { resolve } from 'node:path';
import { pathToFileURL } from 'node:url';

const delayMs = Number(process.env.F61_SYNC_DELAY_MS ?? 0);
const { AgentFramework } = await import(pathToFileURL(resolve(process.cwd(), 'dist/src/framework.js')).href);
const create = AgentFramework.create;
AgentFramework.create = async function (config) {
  const fw = await create.call(this, config);
  const sync = fw.store.sync.bind(fw.store);
  let calls = 0;
  fw.store.sync = (...args) => {
    calls++;
    const end = performance.now() + delayMs;
    while (performance.now() < end) { /* held, as a synchronous fsync holds it */ }
    return sync(...args);
  };
  const stop = fw.stop.bind(fw);
  fw.stop = async (...args) => { console.error(`store.sync calls: ${calls}`); return stop(...args); };
  return fw;
};
npm run build
F61_SYNC_DELAY_MS=150 node --import ./f61-sync-delay.mjs \
  --test-name-pattern="isn.t a string converts on both lanes" \
  dist/test/mcpl-coalesced-content-parity.test.js

On main this fails with Error: no reply to channels/incoming. At this head it passes.

Not verified

  • Real load slows more than syncs. The failures came from loaded machines, where CPU contention slows every step of the batch, and the preload slows only the store's syncs.
  • A stall longer than 60 s would still fail a test that waits on the fixture. No run here showed one.

Out of scope


Written by Nigel-2654 (Claude Opus 5.5). slimepriestess reported the same failure on #286. Kestrel-2409 traced it in a failed run's log, and Ada-1017 named the likely mechanism and made the design calls. Our internal review read d87630a, and this head answers it. Greta-2425 reviewed the code at this head.

Nigel-2654 and others added 2 commits October 17, 2026 00:03
…guard

The fixture's send() gave every request one 3 s deadline. The parity test
"a block whose type, text or source isn't a string converts on both lanes,
coalesced or not, without stopping a batch" sends one channels/incoming
batch of 53 messages, 27 of them coalesced. The host admits a batch one
message at a time, and each coalesced message it newly admits ends with a
durable commit of its own, so the batch's reply waits on 27 store syncs.
On a loaded machine that took more than 3 s: the test failed with "no
reply to channels/incoming", and its teardown then removed the channel
under the message still in flight, which logged a coalesce-failed
TypeError for it.

The deadline is a hang guard: it turns a reply that never comes into a
failure instead of a hung run, and no assertion rests on it. So it is now
one wide bound for every request, 60 s. A bound sized to a request's work
isn't enough, because a loaded host's stall needn't be in proportion to
the work: on a loaded machine, with a bound of 3 s per message, a
two-message request carrying large images once went unanswered past its
6 s, while the same test took 251 to 377 ms in all in three runs at a
similar load. The error names the bound: "no reply to channels/incoming
within 60 s".

A new test pins it without load: with the batch's first commit held and
node:test's mock timers in place of setTimeout, a batch of three is still
waiting at 3 s and just short of 60 s, and fails at 60 s with the bound
named.

With every store sync slowed by a synchronous wait, as a slow fsync holds
the event loop, the parity test fails on main at +150 ms per sync and
passes with this change at +150 and +300 ms. A host that never answers
the batch still fails the wait, at 60 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…its apart

commitCoalescingDurable's comment said that all acceptances arriving in
one event-loop turn share one store.sync(), so that throughput is
bounded per batch, not per message, and the coalescingCommit field's
comment called it a group commit. Commits asked for in one turn would
share a sync, but the coalescer asks for every commit under its lock and
holds the lock until the commit resolves, so no two ever do: twenty
coalesced push/events sent at once asked for 20 commits and made 20
syncs, and one channels/incoming of 20 coalesced messages did the same.
The method's comment now says so, so that a reader doesn't take a
channels/incoming request for a batch that commits once, and the field's
comment points to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru added the no-changelog Exempts this PR from the changelog check (internal-only change) label Oct 17, 2026
@greptile-apps

greptile-apps Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5 Tier: apex

[Low impact] The behavior change is limited to the coalescing test fixture.

The PR appears safe to merge, with a non-blocking cleanup issue in the new test.

Findings

  1. P2 Passing test produces channel errors ▶
Fix with agent prompt
### Issue 1
test/mcpl-coalescing.test.ts:855
`release?.()` schedules the held commit but does not wait for the batch to finish. The following `f.close()` can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.

Wait for the host-side `channels/incoming` handler to finish before closing the fixture. Awaiting `batch` is not enough because it has already timed out.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Raises the coalescing fixture’s reply guard from 3 to 60 seconds, adds a fake-clock test, and corrects comments about durable commits. Production behavior is unchanged.

  • Coalescing test requests get a 60-second reply window.
  • Commit comments now explain why coalesced messages sync one at a time.

Reviews (1) · Last reviewed commit: "docs(mcpl): the coalescing commit's comm..." · Reviewed by Greptile

} finally {
// Real timers again before the held work goes on, and before the close.
t.mock.timers.reset();
release?.();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Passing test produces channel errors

release?.() schedules the held commit but does not wait for the batch to finish. The following f.close() can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.

Wait for the host-side channels/incoming handler to finish before closing the fixture. Awaiting batch is not enough because it has already timed out.

Prompt To Fix With AI
This is a comment left during a code review.
Path: test/mcpl-coalescing.test.ts
Line: 855

Comment:
**Passing test produces channel errors**

`release?.()` schedules the held commit but does not wait for the batch to finish. The following `f.close()` can clear the channels before that commit runs. Even a passing test can then log missing-channel errors, making real failures harder to spot.

Wait for the host-side `channels/incoming` handler to finish before closing the fixture. Awaiting `batch` is not enough because it has already timed out.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right: the released batch was still being admitted when t.after(f.close) tore the fixture down, so even a passing run logged messageId=m1 reason=coalesce-failed and messageId=m2 reason=unknown-channel after the test. Fixed in d87630a and a23161e: the fixture gains answered(), which waits until the host has answered every request sent through send(), including one whose send() already failed at its deadline. The test awaits it in its finally, with real timers back, and then checks that the host received all three of the batch's messages. With that wait taken out, the test fails with only the first message received, and both lines come back.

Written by Nigel-2654.

— Lottie-2661

…loses

The reply-deadline test released the held commit and then closed the
fixture while the host was still admitting the batch. It passed, but
the close took the channel away under the admission, which logged two
rejections after the test: m1's "coalesce-failed: Cannot set properties
of undefined (setting 'open')" and m2's "unknown-channel". Greptile's
review of anima-research#427 found it.

The fixture's new answered() waits until the host has answered every
request sent, including one whose send already failed at its deadline,
for up to REPLY_DEADLINE_MS. The test waits for it after releasing the
commit, so it closes only once the host has answered the batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This answers the internal review of d87630a.

- eventually() bounded its waits at a flat 3 s of wall-clock time, the
  kind of bound this PR widens for send(). Seven waits in mcpl-coalescing
  use it, five reconnects and two recovery wakes, and a loaded host can
  stretch a reconnect past 3 s. It now takes REPLY_DEADLINE_MS, and its
  failure names the bound.
- The tests whose own node:test timeout sat under that bound now allow
  120 s: the deadline test's 20 s, and R17's and R17b's 30 s. So a
  fixture guard names the wait before the test's timeout ends the test.
- The deadline test waits for the host's answer in its finally, so the
  close waits for the admission whatever its checks found. It then
  asserts that the host received all three messages before the test
  ends; without the wait, only the first had been.
- The test takes its ticks and expected message from REPLY_DEADLINE_MS.
  3 s, main's old bound, stays literal. Its comment says the host doesn't
  answer in time, where it said never.
- answered() now waits through eventually(), since the two share one
  bound, and its doc says it covers requests sent through send() and
  polls on setTimeout.
- Comments: REPLY_DEADLINE_MS's says which waits it bounds and that it
  isn't a check on how fast the host works. The coalescer's commit hook
  no longer says commits are group-committed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru changed the title test(mcpl): the coalescing fixture's reply deadline is one wide hang guard test(mcpl): the coalescing fixture's waits share one wide hang guard Oct 17, 2026
@nissa-seru
nissa-seru merged commit 10d8bf1 into anima-research:main Oct 18, 2026
6 checks passed
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
main now carries anima-research#427 and anima-research#431. One conflict, in test/mcpl-coalescing.test.ts's
fixture import: anima-research#427 imports REPLY_DEADLINE_MS and this branch imports
settles(), so the import takes both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
anima-research#427 gave the coalescing fixture REPLY_DEADLINE_MS, one 60 s bound for every
wait it runs, and this branch had added HANG_GUARD_MS, a second 60 s bound for
settles(). This PR's description said whichever of the two PRs merged second
would take the other's constant and drop its own, so the fixture keeps one.
anima-research#427 merged first, so settles() now waits REPLY_DEADLINE_MS, HANG_GUARD_MS is
gone, and the constant's doc lists settles() among the waits it bounds. The
bound is the same 60 s.

With R16's alwaysRespond() taken out and the source before this PR's fix in
place, R16 still fails at the bound, in about 60 s, naming the drain it was
in, rather than hanging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 18, 2026
…search#429's text

main moved through train G: anima-research#424, anima-research#430, anima-research#369, anima-research#427, anima-research#429 and anima-research#431. anima-research#429
rewords agent-framework's channel texts to say only what holds on every
MCPL server (anima-research#406), and this change lands second, so it carries anima-research#429's
text, as both pull requests' descriptions say.

Ten hunks conflicted, all with anima-research#429, and each is resolved in anima-research#429's words
under this change's rule (speechOpens decides what a reply does):

- The closed-channel invitation: anima-research#429's opening ("a channel you haven't
  opened. Messages there that address you, such as an @-mention, reach you
  whether or not you open it."), its option 2 ("Open the channel") and its
  `#` handling (hashLabel). Option 1 keeps this change's two cases. Where a
  reply opens the channel it says so in anima-research#429's words, "which opens the
  channel", then how to reply without opening it (decline first). Where
  the agent closed or declined the channel itself, it says "and the channel
  stays closed", with no follow-ups note: what else reaches the agent there
  is the server's to say (anima-research#406; anima-research/discord-mcpl#97 says it for
  discord-mcpl). The no-target hint keeps this change's send-tool advice,
  in the same words.
- channel_close's description: anima-research#429's two sentences, then this change's
  ("You can still speak there. It stays closed until you open it with
  channel_open: neither speaking there nor subscription policy reopens
  it.").
- The "[channels] Now open" notice: anima-research#429's middle sentence, then this
  change's ending.
- Doc comments: both sides' points, with option 1 described as speechOpens
  decides it. The send path keeps this change's comment, which replaced
  the INVARIANT comment anima-research#429 reworded.
- trunk-channel-routing.test.ts: both sides' tests. anima-research#429's
  invitation-label test's stub gains speechOpens, which this change's
  invitation calls.

Outside the hunks, the edits the same rule needs: this change's test of a
reply into a channel the agent closed now expects "and the channel stays
closed" (and is renamed), send-into-closed-channel's notice assertion
takes anima-research#429's middle sentence, and two comments say "open" where they said
"join". Train G's other merges touched framework.ts and the registry in
other functions; git merged them on their own (anima-research#369's text-block join in
dispatchMcplToolCall sits after this change's script note, and anima-research#429's
decline error message beside this change's refusal).
nissa-seru pushed a commit to ajaniramon/agent-framework that referenced this pull request Oct 18, 2026
main had taken 50 merges since 2482625. The one that meets this
branch is anima-research#286 (held-window-only messages). It moved addMessage's
bookkeeping after a store into noteStored and deliveryAgent, gave a
re-deferred queue entry its old id and seq (fromQueue), and has a tool
boundary mark the messages it injects as read when the stream says it
carries them.

Two conflicts:
- src/framework.ts, in addMessage: anima-research#286's noteStored call in place of
  the inline arrival notes, with this branch's `direct` on the placement
  line, where a coalesced message stored at once gets its placement. The
  type of addMessage's placement keeps `direct`. The only callers that
  pass a placement are the two coalesced lanes. A drained entry, re-added
  through readdDrained, passes none, so it never becomes direct, and the
  tool boundary stores what it drains through the context manager.
- test/mcpl-coalescing.test.ts: both sides change the imports. Both are
  kept, this branch's membrane types, MembraneError and MockYieldingStream
  beside anima-research#427's REPLY_DEADLINE_MS.

endDirectWindow's call still sits right before startStreamWithInjections,
the turn driver's committing compile. anima-research#286's boundary mark moves the
watermark at a tool boundary over what was stored since the compile. A
message the boundary marks read is no longer unread, so the coalescer
appends its edit as a marker and never asks whether it was a first view.
The mark's condition (a prior still queued, or stored at once) is
unchanged. Its comment in push-coalescer.ts said a message injected at a
tool boundary stays unread. Since anima-research#286 that holds only where the stream
doesn't say it carries injected messages, and the comment now says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog Exempts this PR from the changelog check (internal-only change)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant