Long single LLM completions fail with 'Failed after 3 attempts. Last error: fetch failed' (node default 300s idle timeout on non-streaming requests) #38

Long single LLM completions fail with "Failed after 3 attempts. Last error: fetch failed"

What happens

runQuery and runMutation (packages/core/src/agent/agent.ts) use the Vercel AI SDK's generateText, i.e. non-streaming requests. The client POSTs to the configured OpenAI-compatible endpoint and then waits for a single JSON body. Node's built-in fetch (undici) enforces default idle timeouts of 300 s (headersTimeout/bodyTimeout) — on a non-streaming request, no bytes arrive until the entire completion is finished. So any single model completion taking longer than 5 minutes aborts the request with TypeError: fetch failed, which the AI SDK retries twice by default, and the run dies after ~16 min with:

answer: "Failed after 3 attempts. Last error: fetch failed"
outcome: "failed"

Reproduction (local model)

Stack: local Qwen3.8-27b behind a llama-swap gateway, reached through a small OpenAI-compatible proxy. Observed via the traces under <bundle>/.traces/:

The model was healthy throughout; runs #1/#2 lost entirely to the 300 s idle default, run #3 succeeded by luck of per-completion duration. Local models make single >5-min completions routine, so this effectively makes understory unreliable on slow hardware unless completions happen to be short.

Second symptom: no cancellation path for dead MCP clients

While a mutation runs, the tool call is held as a single pending JSON-RPC response. If the MCP client gives up (request timeout), nothing cancels the internal agent — LLM calls continue against the configured endpoint for minutes after the caller is gone, and the writes (concepts, log.md, trace) land only when the run finally ends. That makes failures hard to diagnose: the client reports an error, yet the bundle is silently updated 8–14 min later, and a live chat session on the same model gets starved by the zombie run.

Suggested fixes

1. Stream in runQuery/runMutation — use streamText (SSE) so bytes flow continuously. This makes idle timeouts moot at every layer including any proxy between understory and the model, and it is what a llama.cpp/llama-swap backend does best anyway.
2. Make the HTTP client timeouts configurable (e.g. LLM_TIMEOUT_MS / undici dispatcher with raised headersTimeout/bodyTimeout) so users on slow local models are not stuck at Node's 300 s default even if authN/authZ #1 is out of scope.
3. Honor MCP client disconnection: abort the internal agent loop (and the in-flight LLM call) when the requesting session/request closes, so a dead caller stops wasting model time.