bug(runtime): Mid-stream ECONNRESET after thinking output is terminal, while idle-watchdog timeout on the same state recovers

[chinawch007]

What happened

A long agent turn died on a transient mid-stream ECONNRESET even though the failure was classified retryable, because the plain retry gate counts streamed thinking as observable output — while the idle-watchdog recovery path allows exactly that state to recover.

Both paths in packages/runtime/src/ai-sdk-backend.ts guard the same invariant (my paraphrase, not a code comment): never splice or duplicate content the user has already seen. But they encode different judgments about what is safely recoverable:

• attemptCanRecoverFromIdleTimeout() (line 2228) notably omits attemptSawThinking: a thinking-only attempt may flush its partial thinking via flushStep() (sealed as its own message) and retry into a fresh message id. This behavior is pinned by the committed test retries one idle watchdog timeout after preserving partial thinking (commit 86bb419, fix(runtime): classify provider capacity errors #3365).
• attemptHasNoObservableOutput() (line 2222) includes !attemptSawThinking, and the gate (lines 2586–2592) requires it for the plain retryable branch — so the same thinking-only attempt is refused a retry when the failure arrives as a provider error instead of a local watchdog timeout.

Recovery safety depends on the attempt's state (what was emitted), not on which side detected the failure (local timer vs. remote error). The watchdog path proves thinking-only is safe to seal-and-retry; refusing the same state on the plain path protects nothing additional. The practical consequence: long streaming turns — precisely the ones most exposed to gateway resets — fail terminally on transient network errors the runtime already knows how to recover from.

Real-world incident (2026-08-28, dev build from source; records from that workspace's runtime.sqlite): the 13th model call of a code-review turn (kimi-k3 via an OpenAI-compatible connection) streamed thinking from TTFT 8.9s, then the connection went silent. The gateway's ECONNRESET arrived at 121.6s — seconds before the 120s idle watchdog (DEFAULT_STREAM_IDLE_TIMEOUT_MS, stream-watchdog.ts:21) would have fired. Result: failureClass: network, no retry, run_failed with partial thinking retained. Had the provider simply hung instead of resetting, the watchdog would have fired seconds later and the turn would have auto-recovered — the same network fault, opposite outcome, decided by which detector noticed it first.

How to reproduce

Unit-level (deterministic, reproduced on current main 8c491e64b): in packages/runtime/src/__tests__/ai-sdk-backend.test.ts, stream reasoning deltas, then fail the stream with an ECONNRESET-shaped error after the thinking has been consumed:

function connectionResetFailure(): Error {
  return Object.assign(new Error('Operation failed'), {
    cause: { code: 'ECONNRESET' },
  });
}

function midStreamFailureStream(
  chunks: readonly LanguageModelV4StreamPart[],
  failure: Error,
): { stream: ReadableStream<LanguageModelV4StreamPart>; fail: () => void } {
  let fail: () => void = () => {};
  const stream = new ReadableStream<LanguageModelV4StreamPart>({
    start(controller) {
      for (const chunk of chunks) controller.enqueue(chunk);
      fail = () => controller.error(failure);
    },
  });
  return { stream, fail: () => fail() };
}

test('does not retry a retryable network failure after partial thinking output', async () => {
  const durable = durableTurnHarness('turn-econnreset-thinking', 'review the commits');
  let failCurrentStream: (() => void) | undefined;
  let calls = 0;
  const model = new MockLanguageModelV4({
    doStream: async () => {
      calls += 1;
      if (calls > 1) {
        return {
          stream: simulateReadableStream({
            chunks: [
              { type: 'stream-start', warnings: [] },
              { type: 'text-start', id: 'text-1' },
              { type: 'text-delta', id: 'text-1', delta: 'recovered' },
              { type: 'text-end', id: 'text-1' },
              { type: 'finish', finishReason: { unified: 'stop', raw: 'stop' }, usage: emptyUsage() },
            ],
            initialDelayInMs: null,
            chunkDelayInMs: null,
          }),
        };
      }
      const failing = midStreamFailureStream(
        [
          { type: 'stream-start', warnings: [] },
          { type: 'reasoning-start', id: 'reasoning-1' },
          { type: 'reasoning-delta', id: 'reasoning-1', delta: 'partial thought' },
        ],
        connectionResetFailure(),
      );
      failCurrentStream = failing.fail;
      return { stream: failing.stream };
    },
  });
  const backend = createTestAiSdkBackend({
    sessionId: 'session-1',
    header: header(),
    appendMessage: async () => {},
    connection: connection(),
    apiKey: 'sk-test',
    modelId: 'mock-model-id',
    modelFactory: () => model,
    tools: [],
    loadTurnRuntimeEvents: durable.loadTurnRuntimeEvents,
    newId: idGenerator(),
    now: monotonicClock(),
    providerRetrySleep: async () => {},
  });
  const events: SessionEvent[] = [];
  for await (const event of backend.send(durable.input())) {
    durable.record(event);
    events.push(event);
    if (event.type === 'thinking_delta' && event.text === 'partial thought') failCurrentStream?.();
  }
  assert.equal(calls, 1); // ← no retry: the defect
  assert.equal(events.some((event) => event.type === 'provider_retry'), false);
  const error = events.find((event): event is Extract<SessionEvent, { type: 'error' }> => event.type === 'error');
  assert.equal(error?.reason, 'network');
});

The control case — the same error raised before any output — retries and completes, isolating the boundary to streamed thinking. (The stream must fail after the reasoning chunks are consumed: controller.error() discards queued-but-unread chunks.)

End-to-end (manual): point an OpenAI-compatible connection at a local mock that streams a few reasoning_content chunks, then either destroys the socket (network path) or stays silent past 120s so the idle watchdog fires (recovery path; the mock answers the retried request normally):

Mock behavior	Runtime outcome
reasoning chunks → socket destroyed	failed / network, no retry
reasoning chunks → 60s silence → destroy	failed / network, no retry (the incident shape)
reasoning chunks → silence ≥120s	watchdog timeout → flush + retry → completed

Verification gotcha: provider_retry is a live-only session event and is not persisted (agent-run.ts:608 returns before the runtime-event write), so durable verification should count provider_request_attempt_recorded rows per run in core_agent_run_events (1 = no retry, 2 = retried) together with the run's status.

Environment

• Maka commit: 8c491e64b (reproduced from source; the incident occurred on an Aug 25 build whose retry-gate predicates are byte-identical to this commit)
• OS: macOS 14 (darwin 23.2.0, x64)
• Surface: Runtime Host (packages/runtime)
• Node.js: v24.18.0

Logs, screenshots, or additional context

Incident records (2026-08-28, runtime.sqlite): final attempt of run 35cab2d6 — step 12, attempt 0, TTFT 8,944ms, latency 121,602ms, providerCode: ECONNRESET, run failureClass: network, turn_state: failed with partialOutputRetained: true. The turn had already completed 13 provider calls (~1.35M input tokens, 89% cache hit).

Suggested fix: let the plain retryable branch accept thinking-only attempts with the same predicate shape as attemptCanRecoverFromIdleTimeout(), reusing its existing flushStep() + fresh-message-id machinery and its own bounded budget. With the fix, the unit repro above flips to calls === 2 and the turn completes. Notably, no currently committed test pins the no-retry behavior (the repro above was written for this report), so the fix requires flipping no existing test — only adding coverage.

Scope note / open question: whether streamed text (answer) output should also become seal-and-retry is a separate product decision — two adjacent, divergent answers on the primary reading surface may be worse than a clean failure. Tool activity and provider continuation metadata should remain non-retryable regardless (side effects cannot be deduplicated; continuation identity cannot be replayed).

[chinawch007]

take