Skip to content

Detect and recover stalled session runtimes #63

Description

@ashwin-pc

Problem

A live session can remain isStreaming: true indefinitely after the underlying model stream stops making progress. During the incident that led to #62, session 019f827f-0a8b-7011-ad9e-4f62ef815f0d had:

  • runtimeStartedAt: 2026-07-24T21:45:57.351Z
  • runtimeLastActivityAt: 2026-07-24T21:48:51.666Z
  • isStreaming: true more than an hour later
  • transcript ending immediately after two sessions_spawn tool results, with no following assistant message or agent_end

Both workers subsequently completed and their orchestrator-watch-resolved entries were persisted, but wakeup delivery was blocked because the parent session never became idle. POST /api/abort immediately unblocked the parent; queued wakeups delivered and the turn completed normally.

This is separate from the client render-sync regression fixed by 1ffdf17 / #62. In this case the UI accurately reflected the server's stale runtime state.

Desired behavior

A runtime that is nominally running but has made no progress beyond a configurable threshold should be distinguishable from a healthy run and recoverable without requiring users to infer that it is hung.

Possible behavior:

  1. Track elapsed time since runtime.lastActivityAt while running.
  2. Surface a stalled state after a conservative threshold.
  3. Provide a clear abort/retry action; optionally auto-abort after a longer threshold.
  4. Ensure stalled orchestrator parents do not silently block completed worker wakeups forever.

The threshold and whether recovery is automatic need an explicit design decision. Ordinary long-running tools must not be mistaken for a stalled model stream if they continue reporting activity.

Acceptance criteria

  • A healthy active stream remains running.
  • A runtime with isRunning: true and stale lastActivityAt transitions to an explicit stalled condition after the configured threshold.
  • The stalled condition exposes a recovery action and does not masquerade indefinitely as normal activity.
  • Aborting/recovering clears the runtime and allows queued orchestrator wakeups to deliver.
  • Progress events reset the stale timer.

Test suggestion

The mock server already supports a "quiet runtime" prompt, which initializes an old startedAt and stale lastActivityAt. Extend it (or add a deterministic fake clock/runtime fixture) and assert:

  • below threshold: runtime is shown as running;
  • above threshold: stalled state/action is shown;
  • a progress event restores healthy running state;
  • abort clears the state and permits a queued wakeup to be processed.

Related: #62 (client state synchronization, fixed by 1ffdf17).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions