Skip to content

fix(runtime-host): Host recovery decodes every Session's full history before readiness, breaching the 45s election deadline #4032

Description

@me2seeks

What happened

Runtime Host startup recovery decodes the entire durable execution history of every recoverable Session before reporting readiness. On a long-lived workspace (~80 recoverable Sessions, ~500 runs, ~100k event/message rows in runtime.sqlite) recovery takes ~150s of mostly main-thread CPU, which breaches the client's default 45000ms election deadline: maka --resume <id> fails with "did not become ready before the startup deadline elapsed" (lastRegistration.state: "recovering", readyWaitFailed: 1) even though the Host is making progress. Retrying after the Host settles works.

This is the residual bottleneck after the artifact-store costs in #4027 are removed; CPU profile of a recovering Host (CDP Profiler on the main thread) shows:

  • ~32% readSqliteAgentRunEvents (packages/storage/dist/agent-run-store.js)
  • ~30% better-sqlite3 all() calls
  • decodeStoredRuntimeEvent, GC pressure

Root causes in packages/runtime-host/src/server/hosted-execution-recovery.ts (prepareHostedExecutionRecovery):

  1. Discarded per-run validation reads. For every Session, for every run, it awaits readEventsForRecovery + readRuntimeEvents and discards the results — a full decode of every event of every run, purely as a fail-closed consistency check.
  2. Messages are decoded twice. listForRecovery() (packages/storage/src/session-store.ts) already calls readMessagesForRecovery per header and discards the result; prepareHostedExecutionRecovery then re-reads the same messages per Session.
  3. The per-session loop is fully serial, though note better-sqlite3 all() and JSON decoding are synchronous main-thread work, so concurrency alone would not help — the fix has to reduce the amount of work, not interleave it.

Because admission/message repair decisions only involve Sessions with admissions, non-terminal runs, or pending closures, decoding the history of long-settled Sessions appears unnecessary for correctness — it is startup-time insurance with O(total history) cost paid on every Host start (upgrade, epoch cutover, crash recovery).

How to reproduce

  1. Use a workspace with a large accumulated history (many recoverable Sessions and runs; here ~80 Sessions / ~500 runs / ~100k rows).
  2. Restart the Runtime Host (upgrade or stop the previous one).
  3. Run maka --resume <session-id> within the first ~2 minutes.
  4. Observe the election deadline error while the Host is still recovering; Host main thread sits at ~50-70% CPU decoding SQLite rows.

Environment

  • Commit: d27c02af1 (main)
  • Node: 26.3.0
  • OS: Linux
  • Surface: Runtime Host / TUI

Logs, screenshots, or additional context

Election diagnostic from the client:

{"deadlineMs":45000,"elapsedMs":45008,"candidateLaunches":1,"sawEndpointConnected":true,
"observations":{"readyWaitFailed":1,"connected":1},
"lastRegistration":{"state":"recovering","lifecycleMode":"ephemeral"}}

Related: #4027 (artifact-store cold-start costs; fix in #4031).

Suggested directions (needs a design decision on the startup-validation contract):

  • Scope recovery to Sessions that actually need it (admissions, non-terminal runs, pending closures) instead of every recoverable Session; or make full-history validation lazy/on-demand.
  • Drop the discarded double read of Session messages (keep a single decode).
  • Optionally surface recovery progress so waiting clients can extend their deadline adaptively instead of failing at 45s.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions