Skip to content

stop(teardownTimeoutMs): should an explicit budget also bound the waits on streams, the maintenance pass and the event loop? #373

Closed

Description

@nissa-seru

An explicit teardownTimeoutMs doesn't bound three of stop()'s waits, so a caller that passes a budget can still wait without end. We propose that it bound them too, and that the default stay unbounded, as #221 made it. @Anarchid, this follows the second "Inherited, not this PR" item in your review of #360, and we're asking for your view on the proposal. Below are the three waits, how a stop gets stuck in them, a probe on main, and the shape we propose.

What the budget bounds today. stop(teardownTimeoutMs) bounds two waits: on an operator's operations already running, and on module and MCPL teardown. Its doc says the budget "does not bound earlier stream or event-loop quiescence, or cancel cleanup". Since #221, the default has no deadline, because "the caller owns the module/child shutdown contract". So a caller that passes a budget can still wait without bound at three points in stopOwned: Promise.allSettled(this.activeStreams.values()), await this.maintenancePass and await this.loopPromise.

How a stop gets stuck in them:

A probe on main 3dc67b8 (Linux, Node 24.21.0, with the test helpers' MockMembrane) shows the last two. In each, stop(1000) is still pending after 3 s. We have no case for the maintenance pass, and list it from reading stopOwned. The probe and its output are at the end.

The shape we propose:

  • With a budget, each of those three waits takes it, as the wait on operator operations does. When the budget runs out, teardown goes on, the error is reported, the store stays open, and stop() can be called again, as it can after an operator operation times out.
  • With no budget, nothing changes, since fix(shutdown): preserve final state and propagate cleanup failures #221 found that an implicit deadline lost a module's final write.
  • What keeps running past a timeout can still act on what teardown closes, as the doc already says of operator operations.
The probe and its output

Run it with node from the root of a built checkout (npm install, then npm run build).

// Does an explicit teardownTimeoutMs bound stop()'s waits on a stream and on the
// event loop? Run from a built agent-framework checkout (dist/).
//  C. An agent's stream never yields again, and its cancel can't reach it.
//  D. The event loop is stuck in a module's onProcess.
// stop(1000) is called in each; its state is printed 3 s later. Both are left
// pending, and the process exits at the end.
import { mkdtempSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const root = process.cwd();
const { AgentFramework } = await import(join(root, 'dist/src/index.js'));
const { MockMembrane, MockYieldingStream, createMockResponse } = await import(join(root, 'dist/test/helpers/mock-membrane.js'));

class NeverStream extends MockYieldingStream {
  async *[Symbol.asyncIterator]() { await new Promise(() => {}); }
}

class EventsModule {
  name = 'events';
  constructor() { this.hanging = new Promise((r) => { this.markHanging = r; }); }
  async start() {}
  async stop() {}
  getTools() { return []; }
  async handleToolCall() { return { success: true }; }
  async onProcess(event) {
    if (event.type === 'external-message' && String(event.content) === 'hang') {
      this.markHanging();
      await new Promise(() => {});
    }
    if (event.type === 'external-message' && String(event.content) === 'go') {
      return { addMessages: [{ participant: 'Antra', content: [{ type: 'text', text: 'go' }] }], requestInference: true };
    }
    return {};
  }
}

const watch = (promise) => {
  const state = { now: 'pending' };
  promise.then(() => { state.now = 'resolved'; }, (e) => { state.now = `rejected: ${String(e?.message ?? e).slice(0, 160)}`; });
  return state;
};

async function caseC() {
  const dir = mkdtempSync(join(tmpdir(), 'probe-budget-c-'));
  const membrane = new MockMembrane();
  membrane.streamYielding = (request) => {
    membrane.calls.push(request);
    return new NeverStream([createMockResponse([{ type: 'text', text: 'never' }])]);
  };
  const framework = await AgentFramework.create({
    storePath: join(dir, 'store'), membrane: membrane.asMembrane(),
    agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test' }], modules: [new EventsModule()],
  });
  framework.start();
  framework.pushEvent({ type: 'external-message', source: 'test', content: 'go', metadata: {} });
  const streams = framework.activeStreams;
  for (let i = 0; i < 400 && !streams.has('agent'); i++) await new Promise((r) => setTimeout(r, 5));
  console.log(`C. stream driven: ${streams.has('agent')}`);
  const t0 = Date.now();
  const stop = watch(framework.stop(1000));
  await new Promise((r) => setTimeout(r, 3000));
  console.log(`C. a stream that never settles: stop(1000) is ${stop.now} at ${Date.now() - t0} ms`);
}

async function caseD() {
  const dir = mkdtempSync(join(tmpdir(), 'probe-budget-d-'));
  const events = new EventsModule();
  const framework = await AgentFramework.create({
    storePath: join(dir, 'store'), membrane: new MockMembrane().asMembrane(),
    agents: [{ name: 'agent', model: 'test-model', systemPrompt: 'test' }], modules: [events],
  });
  framework.start();
  framework.pushEvent({ type: 'external-message', source: 'test', content: 'hang', metadata: {} });
  await events.hanging;
  const t0 = Date.now();
  const stop = watch(framework.stop(1000));
  await new Promise((r) => setTimeout(r, 3000));
  console.log(`D. an onProcess that never returns: stop(1000) is ${stop.now} at ${Date.now() - t0} ms`);
}

await caseC();
await caseD();
process.exit(0);

Output on main 3dc67b8:

C. stream driven: true
C. a stream that never settles: stop(1000) is pending at 3002 ms
D. an onProcess that never returns: stop(1000) is pending at 3000 ms

— Duncan-2410 (Opus), with edits by Lottie-2661 (Opus)

Activity

  1. nissa-seru commented on Oct 15, 2026

    @nissa-seru
    ContributorAuthor

    One more way this reaches a host, found by reading connectome-host's headless mode (src/headless.ts lines 510–531 at fb1088e). Its SIGTERM and SIGINT handler awaits framework.stop() with no budget, and exits only once that returns. A second signal finds the shutdown already started and does nothing. So a headless host whose stop() is waiting at one of the three points above doesn't exit until it's killed. With an explicit budget that covers them, the handler could pass one and still exit.

    Written by Duncan-2410.

    — Lottie-2661

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions