feat(inference)!: cut an AgentFramework model call that stops making progress, and give a deadline expiry one fresh call

README.md

@@ -594,6 +594,36 @@ static tools are listed like any server's.
 5. **Budget restart**: If input tokens exceed `maxStreamTokens`, the stream is cancelled, context is recompressed, and a fresh stream starts. An agent whose model binds its thinking (`thinkingBinding.binds`) with `physicalWindowTokens` set skips this, and the window projection restarts it instead before a request the window can't hold
 6. **Complete**: Final response saved to context, agent returns to idle
 
+### Inference bounds
+
+A model call that stops making progress is cut, so an agent isn't held by a provider that has gone quiet, or that keeps its connection open with keepalives (an Anthropic stream's pings, SSE comments) and never answers.
+
+- **A resident's streamed call**, and an MCPL server's streamed inference request, get membrane's progress bounds where the adapter's stream reports content (`AnthropicAdapter`, `BedrockAdapter`). Before content, a call ends 60 s after its send, or, once a thinking block has started, at the later of that and the block's allowance, its `max_tokens` at 87 tokens a second. Once content has begun, it ends after 150 s without a provider event, or after the block's allowance when the last event was a thinking block's start and the allowance is longer. Membrane's README gives the rules whole.
+- **Every streamed call the framework makes** also gets a flat bound from its last provider event: the larger of a setup term plus that allowance and 1,862 s (`FLAT_MINIMUM_MS`). The setup term is the larger of the before-content and between-events bounds, 150 s by default. So on an adapter with progress bounds the flat bound never runs out first. On one without, such as `OpenAICompatibleAdapter`, it's the only progress bound, and an SSE keepalive comment doesn't restart it, though a data frame that parses as an event does, a provider's own ping object included.
+- **A whole non-streamed call**, a compression's, the reply judge's, or an MCPL server's non-streamed inference request, ends the setup term plus its allowance after its send, or, when it names no `max_tokens`, at the larger of the setup term and the flat minimum, unless its caller passed a `progressTimeoutMs` of its own. Its caller gets the `timeout` error. The event gate's own 20 s judge timeout ends the reply judge's call first at the defaults. Context-manager takes a cut compression call as a transient failure: the lane pauses and tries again, so compression that never lands within its bound shows as a lane that keeps pausing, not as a failed turn, and is a reason to raise the bound. On an adapter whose `complete()` streams under the hood, such as membrane's OpenAI Responses adapter in subscription mode, membrane renews the bound at each parsed provider event, so there it runs from the last event, as a stream's flat bound does.
+- **A turn's call a bound cuts** fails as a `timeout`, and `DefaultErrorPolicy` gives it one fresh call, 1 s later, whatever other failures its turn had, except in a turn sent again after its retries ran out, before one of its rounds has run or had its tool calls cut off, whose failures get no retries. A fresh call that's cut too, with no other failure between, ends the turn, and its `[inference-failed]` note says the call timed out, that the call was made again and timed out again, and the reason. For a bound's cut the reason names the bound that ran out and the host's setting for it. A deadline expiry membrane throws (the Anthropic adapter's own watchdogs, the SDK's request timeout, a runtime's fetch timeout on a silent connection) counts the same toward the fresh call and the turn's end, and its reason is its own message, which names no setting. A bound's cut isn't `retryable`, so a host's own `errorPolicy` that retries only retryable errors gives it no fresh call; an expiry membrane throws stays `retryable`, as membrane made it.
+- **A stream that keeps sending content** isn't ended by any of these, since content is progress.
+
+These bounds don't replace the deadlines below them: a silent connection still meets the runtime's own fetch timeout (360 s on Bun 1.4.4) and, on `AnthropicAdapter`, the adapter's 600 s first-event and idle watchdogs, so on a silent connection a bound raised past those is met by them first.
+
+The phase bounds, and a reason that names the bound that ran out, need the `Membrane` a host passes in to come from membrane 0.8.0 or later. An older one ignores `progressBounds`, so only the flat bound acts, and a cut's reason names no bound.
+
+The flat and whole-call bounds read the request's `maxTokens`. A `max_tokens` set through `providerParams` or a hook isn't read, where membrane's own allowance reads what's sent, so a larger one set that way can let the flat bound run out before membrane's.
+
+Where the values come from: membrane's README gives its progress bounds' measurements, on a production router's streamed Anthropic calls to `claude-opus-5-5` and `claude-haiku-5-5`. The flat minimum is the longest silence between frames of any completed call in that router's OpenAI calls (the Responses API over a WebSocket), standing in for an OpenAI-compatible stream, which wasn't measured. The whole-call bound's setup term is five times the longest wait from the send to content or a thinking block's start in those calls (a wait that router cuts at 30 s and sends again, so a longer one is unseen), and its allowance is generation at 87 tokens a second, the 1st percentile of Opus's thinking speed over its thinking blocks of 30 seconds or more. No completed streamed call in that record ran past the whole-call bound. Bedrock, other models (`claude-opus-4-6` among them), OpenAI-compatible streams and calls that aren't streamed, compression's among them, weren't measured, so a host whose calls run slower raises the bounds with `inferenceBounds`. Membrane's README also reports the wait for content growing with the prompt, so a resident near a full context window is the likeliest to meet the before-content bound first.
+
+`inferenceBounds` sets them: `progressBounds` takes membrane's three values, from which the flat and whole-call bounds also derive, and `flatMinimumMs` the flat bound's least. `false` sets none. One setting covers every agent, so a host with one slower model or adapter raises them for all of its agents. A stream cut's reason is membrane's message, naming the bound that ran out, followed by the host setting for that bound, as in ``(the host's setting for this bound: `inferenceBounds.progressBounds.beforeContentMs`)``. A whole call's caller gets membrane's message alone. Each reason's opening and its setting:
+
+| The reason begins | The setting | Default |
+|---|---|---|
+| `No content within …s of the send` | `progressBounds.beforeContentMs` | 60,000 ms |
+| `No provider event within …s` | `progressBounds.betweenEventsMs` | 150,000 ms |
+| `No content within …s of a thinking block's start` (or `No provider event within …s of a thinking block's start`) | `progressBounds.thinkTokensPerSecond`: a lower rate allows a slower think | 87 |
+| `No provider progress within …ms`, on a stream | `flatMinimumMs`, or the setup term and allowance when they're larger | 1,862,000 ms |
+| `No provider progress within …ms`, on a call that isn't streamed | the setup term (`beforeContentMs`, `betweenEventsMs`) and the allowance (`thinkTokensPerSecond`, or the request's `maxTokens`); the larger of the setup term and `flatMinimumMs` when the request names no `maxTokens` | 150 s plus `maxTokens` at 87 a second; 1,862,000 ms with no `maxTokens` |
+
+Each cut of a turn's call is traced as `inference:failed`, with its reason as `error`, and logged with the turn's inference log as a failed call, whether or not its fresh call then answers, so a host can count cuts a resident never sees and raise a bound before turns start failing. An MCPL server's request that a bound cuts is traced as `mcpl:inference_fail` with the reason as `error`, and a compression or reply-judge call's cut reaches only its caller, as the `timeout` error.
+
 ### Abort / Cancellation
 
 ```typescript

changelog.d/378-inference-bounds.added.md

@@ -0,0 +1,9 @@
+- `FrameworkConfig.inferenceBounds` sets how long a model call may go without
+  progress: `progressBounds`, membrane's `beforeContentMs`, `betweenEventsMs`
+  and `thinkTokensPerSecond`, from which the flat and whole-call bounds also
+  derive, and `flatMinimumMs`, by default `FLAT_MINIMUM_MS` (exported). `false`
+  sets none. A value it can't use is refused at `create()`, naming the field
+  under `inferenceBounds`. One setting covers every agent, so a host with one
+  slower model raises the bounds for all of them.
+- Settings passed to `new DefaultErrorPolicy(settings)` can carry a
+  `deadline` schedule for deadline expiries.

changelog.d/378-inference-bounds.breaking.md

@@ -0,0 +1,38 @@
+- **Hosts:** a model call that stops making progress is now cut
+  (agent-framework#378). On an adapter whose stream reports content
+  (`AnthropicAdapter`, `BedrockAdapter`), a resident's streamed call, and an
+  MCPL server's, gets membrane's progress bounds: before content, 60 s from
+  its send, or, once a thinking block has started, the later of that and the
+  block's allowance (its `max_tokens` at 87 tokens a second); after content,
+  150 s from its last provider event, or the allowance when that event was a
+  thinking block's start and the allowance is longer. On every adapter a
+  streamed call also ends once the larger of a setup term (the larger of the
+  before-content and between-events bounds, 150 s) plus that allowance and
+  1,862 s passes with no provider event; an SSE keepalive comment isn't one.
+  A compression's, the reply judge's and an MCPL server's non-streamed call
+  ends the setup term plus its allowance after its send, or at the larger of
+  the setup term and 1,862 s when it names no `max_tokens`, unless its caller
+  passed a `progressTimeoutMs` of its own; on an adapter whose `complete()`
+  streams under the hood, such as membrane's OpenAI Responses adapter in
+  subscription mode, that bound runs from the last provider event instead.
+  Before, a call ended only at the Anthropic adapter's own 600 s watchdogs,
+  the SDK's request timeout or the runtime's fetch timeout, which still apply,
+  and a call kept open with keepalives didn't end. A call that would have run
+  past one of these bounds and then completed now fails as a `timeout`.
+  **Migration:** `inferenceBounds` raises them, and `inferenceBounds: false`
+  sets none of them. The phase bounds act only when the `Membrane` passed in
+  comes from membrane 0.8.0 or later; with an older one only the flat bound
+  acts, so a host that builds its own `Membrane` raises its membrane to
+  `^0.8.0` with this release.
+- **Hosts:** `DefaultErrorPolicy` gives a deadline expiry one fresh call,
+  1 s later. An expiry is a bound's cut, the Anthropic adapter's watchdogs,
+  the SDK's request timeout or a runtime's fetch timeout; the last three were
+  thrown as retryable timeouts and got the base schedule's three tries. For a
+  deadline expiry the framework passes an `errorPolicy` the count of that
+  call's consecutive expiries as `attempt`, in place of the turn's failures,
+  so each call a deadline cuts gets its fresh call; a host's own policy reads
+  that count too. A quiet call on an adapter without progress bounds now ends
+  after two of the runtime's fetch timeouts (on Bun, 360 s each) rather than
+  three. **Migration:** settings passed to `new DefaultErrorPolicy(settings)`
+  with no `deadline` schedule give an expiry their base schedule, counted the
+  same way.

changelog.d/378-inference-bounds.changed.md

@@ -0,0 +1,16 @@
+- A turn that ends on a timeout says so in its `[inference-failed]` note: the
+  model call timed out, or its last call did, with the reason, which for a
+  bound's cut names the bound that ran out. When that call was a fresh call made for an earlier cut and a
+  deadline ended it too, the note says so: "The call was made again, and it timed out again." The cause in parentheses
+  is left out when it's only `timeout`. Its closing line says that if a reply
+  is still owed from that turn, it can be given now. Other failures keep their
+  wording and advice.
+- An MCPL server's streamed inference request that ends early is answered
+  with the reason and, when membrane gives one, its message, such as the
+  bound that ran out: ``The inference stopped before it finished (timeout:
+  No content within 60s of the send: … (the host's setting for this bound:
+  `inferenceBounds.progressBounds.beforeContentMs`))``.
+- A stream cut's reason, in a turn's error and note and in an MCPL server's
+  answer, ends by naming the host setting for the bound it hit, such as
+  ``(the host's setting for this bound:
+  `inferenceBounds.progressBounds.beforeContentMs`)``.

src/agent.ts

@@ -2,6 +2,7 @@ import { NOTICE_TAG } from './notice-tags.js';
 import { heartbeatToolAlias } from './heartbeat/heartbeat.js';
 import type { Membrane, NormalizedMessage, NormalizedRequest, ContentBlock, ThinkingBindingEntry, YieldingStream } from '@animalabs/membrane';
 import { isAbortedResponse } from '@animalabs/membrane';
+import { streamBounds, type InferenceBounds } from './inference-bounds.js';
 import type { CacheWireReceipt, KvUnifiedRequestHooks } from './kv-unified-wire.js';
 import { ToolResultGuard } from './tool-result-guard.js';
 import { withoutTokenEstimates } from './thinking-token-stamp.js';
@@ -50,6 +51,9 @@ export interface ActivationRequestOptions {
    *  pass it (its errorPolicy is their retry layer); a direct caller of
    *  startStream keeps membrane's retries. */
   membraneRetries?: boolean;
+  /** The framework's inferenceBounds, resolved: the stream gets their phase
+   *  bounds and flat bound. None when left out. */
+  inferenceBounds?: InferenceBounds | null;
 }
 
 /** A compile's render stats, as context-manager's `getRenderStats()` types
@@ -2211,6 +2215,7 @@ export class Agent {
       // since membrane 0.7 its own schedules, ten minutes at a 529, would
       // otherwise run inside each of the caller's attempts, out of its sight.
       ...(options.membraneRetries === false ? { retry: false } : {}),
+      ...streamBounds(options.inferenceBounds ?? null, request.config.maxTokens),
       // Retry a content-policy refusal at the provider seam rather than at
       // the framework level: membrane replays the SAME request immediately
       // (cache-warm), where a framework-level requeue would recompile and