Skip to content

test(mcp): the modern outcomes tests bound each wait for a hang, not for speed - #559

Merged
nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:fix/f254-hang-sized-waits
Oct 19, 2026
Merged

nissa-seru merged 1 commit into
anima-research:mainfrom
nissa-seru:fix/f254-hang-sized-waits

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: in test/mcp-modern-outcomes.test.ts, every until gives its event one 60 s bound, sized for a hang, and only the calls meant to time out get a short request deadline. Before, several bounds sat a few seconds above how long their events take on a quiet machine, and a full-suite run on macOS failed one of the file's tests once.

Problem

In a review of #495 on macOS, a full-suite rerun failed mcp-modern-outcomes' R1-1 reconnect row once, and the file passed 3 runs of 3 alone. The reviewer read it as load flakiness. The report named the row, not its failing assertion.

Several of the file's time bounds sat a few seconds above how long their events take on a quiet machine:

  • until polled for an event against a wall-clock deadline: 5 s by default, and 3, 8, 9, 12 or 15 s at seven calls. The R1-1 row that reaps before the reconnect waits 5 s for a launch's start and handshake, twice. Three R1-1 rows wait 8 s for outcomes that the transport's 2 s grace after SIGTERM, and again after SIGKILL (STDIO_EXIT_GRACE_MS), puts at 2 to 4 s.
  • Two tests gave replies they need a short request deadline. One continuation test ran an error call and a hang call, whose first leg must be answered, under 600 ms. Another ran three concurrent calls under 2000 ms.

None of these bounds asserts how fast something happens. Each only turns a hang into a failure that names it, and a loaded machine can stretch an event past such a bound in a correct run.

Changes

  • until takes one bound, WAIT_MS, 60 s, for every call, as the coalescing fixture's REPLY_DEADLINE_MS does. The seven per-call bounds go.
  • The two tests that set their own 40 s test timeout no longer do.
  • In the continuation test, the hang call gets its own connection with a 1500 ms deadline, and the error call keeps the default. The hang call's second leg is awaited on the server's wire. The concurrent-continuations test takes the default deadline too.
  • The test of close() during a reconnect handshake checks the launch's end as the first connect's test does: its child is gone once close() resolves, and close() took under 15 s, not the connect's 30 s. Its 3 s wait goes.
  • Every duration the file measures reads the monotonic clock (performance.now()), until's deadline included.
  • The coalescing fixture's settles() comment no longer says CI runs without a test timeout.

What each part is for

  • One bound sized for a hang. A bound a few seconds above a designed duration fails a correct run when a loaded machine stretches that duration. A 60 s bound fails one only if the event is held back a whole minute, and it still names a wait that never ends, well inside npm test's 10-minute --test-timeout.
  • No test timeouts of their own. A test's own timeout cuts it off at that time, whatever --test-timeout says, so the two 40 s timeouts would cut off a hang in those tests before the 60 s bound could name it.
  • Short deadlines only on calls meant to time out. callTool gives a whole call, every leg, the connection's requestTimeoutMs. A short one on a call that must be answered turns a reply that a loaded machine holds back into no-response. The hang call's own deadline has to cover its first leg's answer too, so it gets 1.5 s, the room the next test's deadline gives leg 1.
  • Awaiting the hang call's second leg. The server logs a leg as it reads it, and the call's deadline doesn't wait for that, so a count read the moment the call fails can miss a leg that a held-back server hasn't read yet.
  • The check on close(). close() closes a connect in flight and waits for its cleanup, so the launch's child is gone once it resolves. The old 3 s wait caught a close() that left the child running until the connect's 30 s timeout, and a 60 s wait wouldn't. The new assertions catch that, and a close() that waits out those 30 s as well.
  • The monotonic clock. A wall-clock step forward would end a wait early, or inflate a measured duration.
  • The fixture's comment. It said CI runs without a test timeout, which npm test's --test-timeout=600000 contradicts.

Tests

  • On Linux, test/mcp-modern-outcomes.test.ts passes 43 of 43 on Node 24.22.0 and on Node 20.20.2, twice each.
  • The close() check, with close() broken two ways:
    • Waiting out a connect in flight without closing it: the test fails at the new 15 s assertion (close() took 29956.562662 ms, not the connect's 30 s). Main's version of the test passes, in 30.4 s.
    • Neither closing it nor waiting for it: the test fails with the hanging launch is gone once close() resolved. Main's version fails too, at its 3 s wait.
  • Probes, run outside the suite:
    • The R1-1 row that reaps before the reconnect, its steps run with the second launch started 6 s late: under the old 5 s bound it fails with in time: the next launch, and under 60 s it passes in 6.4 s.
    • A continuation call that must be answered, its server stopped (SIGSTOP) for 1 s just after the call goes out: under a 600 ms deadline it ends no-response, and under the default it's the error-response the test asserts. With the server stopped for 2.5 s, a 2000 ms deadline does the same.
    • The hang call, its server held back the same way (Phoebe-2419's probe): under 600 ms only its first leg reached the wire when held 400 ms or more. Under 1500 ms both legs do when it's held up to 1000 ms, and only the first when it's held 1400 ms.
  • Not reproduced, on Linux at main: the R1-1 rows didn't fail with the test's CPU held to 20%, 7% or 4% of a core, nor in a whole npm test run held to three cores.
  • CI on f2d460f5 merged with main 76623a39 (run 39207109452): Build & Test (npm test) passes on Ubuntu and macOS with Node 20 and 24, and the packed release installs on its own.

Not verified

  • The report's failure itself. It came once, in a full-suite rerun on macOS with Node 20.20.2. None of the runs above reproduced it, and none of them but CI's ran on macOS. CI's macOS legs pass, which shows the change runs there, not that it removes that run's failure.

Out of scope

  • The two assertions that a first connect ends in under 15 s, not the discover probe's 30 s (:360, :367), keep their bound. They assert time on purpose, as the new close() check does.
  • The short sleeps that check something didn't happen (:253 and four others) stay. They can't fail a correct run.
  • The await on a transport's close event at :347 has no bound, so it can't fail a correct run, and a hang there still fails at a test timeout.
  • :426's 1500 ms deadline stays. The test holds the round after leg 1's answer until that deadline passes, so the deadline has to cover leg 1's answer, and a longer one would add its length to every run.
  • mcpl-coalesced-content-parity's rows over 1 MiB aren't changed. The report didn't say which of the three failed once in the same run. Only the render row depends on a timer: the product's own 5 s push/render timeout, whose fallback is the designed outcome for a late render. The other two wait on the fixture's 60 s bound.

  • This change is test-only: it changes test/mcp-modern-outcomes.test.ts and a comment in test/helpers/coalescing-fixture.ts, and no src/ file, so it carries no fragment.

Written by Crispin-2406 (Claude Opus 5.5). Ada-1017 made the design calls, and Phoebe-2419 reviewed it.

Description checked and posted by Lottie-2661 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared nissa-seru account.

…for speed

In a review run on macOS, a full-suite rerun failed mcp-modern-outcomes'
R1-1 reconnect row once, and the file passed 3 runs of 3 alone; the
reviewer read it as load flakiness. The report named the row, not its
failing assertion. Several of the file's bounds sat a few seconds above
how long their events take on a quiet machine: until's 5 s default and
seven per-call bounds of 3 to 15 s, for a launch's start and handshake
or the transport's 2 s graces, and request deadlines of 600 ms and 2000
ms on replies the tests need, a hang call's first leg among them. None
asserts how fast something happens, and a loaded machine can stretch an
event past such a bound in a correct run.

The until helper now gives every wait one bound, WAIT_MS, 60 s, as the
coalescing fixture's REPLY_DEADLINE_MS does. The two tests that set
their own 40 s test timeout no longer do, since a test's own timeout
applies whatever --test-timeout says and would cut off a hang before the
bound names it. Only the calls meant to time out keep a short deadline.
The continuation test's hang call gets its own connection, with 1500 ms,
since that deadline covers its first leg's answer too, as the next
test's does, and its second leg is awaited on the server's wire, which
the server writes as it reads the leg. The concurrent continuations test
takes the default deadline.

The 3 s wait after close() during a reconnect handshake was asserting
time. close() closes a connect in flight and waits for its cleanup, so
the launch's child is gone once it resolves, and that bound caught a
close() that left the child to the connect's 30 s timeout. The test now
checks it as the first connect's test does: the child is gone once
close() resolved, and close() took under 15 s, which also catches a
close() that waits those 30 s out. Every duration the file measures,
until's deadline included, reads performance.now().

The coalescing fixture's settles() comment said CI runs without a test
timeout; npm test sets --test-timeout=600000.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Oct 19, 2026

Copy link
Copy Markdown

Required label not found on this PR.

nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 19, 2026
@nissa-seru nissa-seru added the no-changelog Exempts this PR from the changelog check (internal-only change) label Oct 19, 2026
@nissa-seru
nissa-seru merged commit 9c6964c into anima-research:main Oct 19, 2026
7 checks passed
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 19, 2026
Brings in anima-research#557 (an ephemeral run takes no global fallback) and anima-research#559.

Conflicts, all in src/mcpl/channel-registry.ts:
- onRouteFailure's doc: both sides extended the list of when it's called.
  It keeps this branch's error-response case and main's exception for an
  ephemeral run's speech with no locus.
- deliverSpeech's doc: both sides added an exception to "surfaced through
  onRouteFailure": this branch's held failure (heldOnFailure) and main's
  ephemeral run with no locus. It names both.
- handleToolPublish's signature: main added `origin` as its second
  parameter, for channel_publish's default channel, and this branch added
  `seen`, publishForAgent's outcome. `origin` stays second, as main's
  handleChannelToolCall passes it, and `seen` comes third. publishForAgent
  passes no origin: it always names its channel, so the default never
  applies to it.

Elsewhere both sides changed buildChannelContext, deliverSpeech's body and
framework.ts, and git merged them: main's fallback rule for an ephemeral
run and this branch's inbound-only guard on defaultOutgoing sit side by
side in buildChannelContext, and main's early return for an ephemeral run
with no locus comes before this branch's publish() in deliverSpeech.
tel-0s added a commit that referenced this pull request Oct 19, 2026
…llback, the all-except hold's channel included

What main brings since aae86b7 (0.22.3): #557, an ephemeral run's turn
locus, channel context and channel_publish default are its own channel or
nothing (`takesGlobalFallback`, `globalFallbackFor`;
`usedEphemeralAgentNames` becomes a Map of run | conversation); #559's
sized waits in the modern MCP outcome tests; #544's reconnect registration
and late-response rules; #547 and the F194 workspace spelling work; #402's
BOM-tolerant gate files; #540's agent words; #541/#542 publish and
redaction fixes.

Conflicts, all in src/mcpl/channel-registry.ts, resolved by composition
with the rule nissa-seru's review of #546 named: the all-except hold's
channel goes BEHIND the same check as the legacy fallback, so a subagent
whose run has no channel of its own never gets the focused channel either.

- buildChannelContext: main's `!== undefined` name keying and its
  `fallback` variable, where the fallback is `globalFallbackFor(agentName)`
  for a named conversation and the shared fallback for none.
- resolveLocus: `home ?? active ?? globalFallbackFor(conversationId)`.
  `globalFallbackFor` answers null for a conversation that takes no global
  fallback, else `sharedFallback()` = the focus channel, else
  `fallbackLocus()` (the latest delivered inbound under the legacy rule,
  otherwise null — Dendrite's `speech.locus`).
- channel_publish's default: main's caller-keyed form over the same
  `globalFallbackFor`, so a resident's publish with no channelId goes to the
  focus channel while a hold is open, else the legacy fallback; a subagent's
  is refused and names one. Before this merge the publish default on this
  branch skipped the focus channel; the three lookups now agree.

Two seams the build and tests found:

- framework.ts resumeAgent still called `.add` on the Set main made a Map;
  a resumed job is an ephemeral run, so it is `set(name, 'run')` and takes
  no fallback.
- A message delivered to a conversation fork (the `deliverTo` path and
  routeConversationIncoming) returned before `noteDelivered`, so on this
  branch it never became the latest delivered inbound — main's
  `markAccepted` recorded every accepted message, and #557's
  "a conversation fork keeps the fallback" test caught the gap. Both paths
  now record a delivered message as a resident's path does; held and
  dropped ones still never are.

Test adaptation: test/ephemeral-no-global-fallback.test.ts is about the
global fallback itself, so its fixture opts into `speech.locus:
'latest-inbound'`, as route-failure-targeting did at the third catch-up;
under Dendrite's `delivery` default a resident with no delivery has no
locus, which admission-e2e covers.

Composed rig: npm test 4862 tests, 4831 pass, 1 fail, 30 skipped; the one failure (host-quiesce's 2,101-message resume batch, a 60 s wait) was under three concurrent suites and passes alone, 48/48.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
nissa-seru pushed a commit to nissa-seru/agent-framework that referenced this pull request Oct 19, 2026
…nge marker, with no awareness gate

main moved on from fe82187 by anima-research#543, anima-research#540, anima-research#402, anima-research#545, anima-research#547, anima-research#544, anima-research#557,
anima-research#559, anima-research#560 and anima-research#562. Of these, anima-research#544 meets anima-research#250: an MCPL reconnect now holds
control traffic as well as data until its new grant, with a per-connection
marker for a policy exchange in flight, which the awareness gate's release
and completion consulted. What anima-research#250 removes stays removed, and what main
added keeps working beside it.

Conflicts:
- src/framework.ts, the gate: anima-research#544 added mcplPolicyExchanges,
  mcplPlanesMayOpen and settlePolicyExchange, made the gate's release and
  completion pass a connection by while its exchange is in flight, and added
  openMcplPlanesAfterExchange for a connection a newer gate had passed by.
  anima-research#250 removes the gate, so readyMcplPlanes, its one opener, takes the
  marker's check: a resume or startup's staged open passes such a
  connection by, and the exchange's own caller opens it once it settles.
  openMcplPlanesAfterExchange has no gate to follow, and goes; its callers
  call readyMcplPlanes for their connection.
- src/framework.ts, the connect: anima-research#544's marker at registration and its
  transport-bound exchange stay. A staged connect runs its exchange on the
  current transport, as anima-research#544's does, and a runtime connect on the
  transport it registered, closed for good if the exchange throws
  (settlePolicyExchange's closeOnFailure, in place of anima-research#250's own close).
  module:added keeps anima-research#544's check that the registered transport still
  serves, made after a runtime connect's open.
- src/framework.ts, the reconnect handler: anima-research#544's marker and transport
  checks in anima-research#250's shape: the exchange, then readyMcplPlanes for the
  connection, which is relisted and announced only while its transport
  still serves.
- src/mcpl/server-connection.ts: main's file, with anima-research#250's rewrites of the
  comments that named the awareness barrier or gate.
- README.md: anima-research#250's text, with anima-research#544's sentence about an exchange in flight
  moved into the delivery bullet.
- test/fixtures/reregistering-mcpl-server.mjs: main's, its header naming
  the three test files that use it.
- test/mcpl-awareness-barrier.test.ts stays deleted. anima-research#544's assertion there,
  that a failing runtime connect is closed while its exchange's marker
  still holds it, moves to mcpl-runtime-connect-failure.test.ts, where anima-research#250
  keeps those tests.

Outside the conflicts:
- test/mcpl-reconnect-registration.test.ts, anima-research#544's: its harness stood on
  the gate. Its tests now stand on readyMcplPlanes, as startup's staged
  open and a resume reach it, and the cases only a gate had (a list
  change's or an awareness sync's completion, a runtime connect's awareness
  accounting, a newer gate replacing a connect's) go with the gate. Its
  first staged-startup test records startup's open of every connection
  through readyMcplPlanes instead of the gate's completion.
- changelog.d: 0.23.0 folded modern-mcp-client.added.md,
  stop-reaches-every-cleanup.fixed.md and mcpl-reconnect-registration.fixed.md,
  so anima-research#250 no longer edits the first two. Its one-shot fragment says what
  changes from 0.23.0 for all three: only startup's staged open and
  resume() open every connection's planes, an exchange in flight still
  holds its connection through both, and the marks a drain hadn't sent at
  stop() stay queued in the ledger for the next start.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru mentioned this pull request Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog Exempts this PR from the changelog check (internal-only change)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant