What happened
Runtime Host startup recovery decodes the entire durable execution history of every recoverable Session before reporting readiness. On a long-lived workspace (~80 recoverable Sessions, ~500 runs, ~100k event/message rows in runtime.sqlite) recovery takes ~150s of mostly main-thread CPU, which breaches the client's default 45000ms election deadline: maka --resume <id> fails with "did not become ready before the startup deadline elapsed" (lastRegistration.state: "recovering", readyWaitFailed: 1) even though the Host is making progress. Retrying after the Host settles works.
This is the residual bottleneck after the artifact-store costs in #4027 are removed; CPU profile of a recovering Host (CDP Profiler on the main thread) shows:
- ~32%
readSqliteAgentRunEvents (packages/storage/dist/agent-run-store.js)
- ~30% better-sqlite3
all() calls
decodeStoredRuntimeEvent, GC pressure
Root causes in packages/runtime-host/src/server/hosted-execution-recovery.ts (prepareHostedExecutionRecovery):
- Discarded per-run validation reads. For every Session, for every run, it awaits
readEventsForRecovery + readRuntimeEvents and discards the results — a full decode of every event of every run, purely as a fail-closed consistency check.
- Messages are decoded twice.
listForRecovery() (packages/storage/src/session-store.ts) already calls readMessagesForRecovery per header and discards the result; prepareHostedExecutionRecovery then re-reads the same messages per Session.
- The per-session loop is fully serial, though note better-sqlite3
all() and JSON decoding are synchronous main-thread work, so concurrency alone would not help — the fix has to reduce the amount of work, not interleave it.
Because admission/message repair decisions only involve Sessions with admissions, non-terminal runs, or pending closures, decoding the history of long-settled Sessions appears unnecessary for correctness — it is startup-time insurance with O(total history) cost paid on every Host start (upgrade, epoch cutover, crash recovery).
How to reproduce
- Use a workspace with a large accumulated history (many recoverable Sessions and runs; here ~80 Sessions / ~500 runs / ~100k rows).
- Restart the Runtime Host (upgrade or stop the previous one).
- Run
maka --resume <session-id> within the first ~2 minutes.
- Observe the election deadline error while the Host is still recovering; Host main thread sits at ~50-70% CPU decoding SQLite rows.
Environment
- Commit:
d27c02af1 (main)
- Node: 26.3.0
- OS: Linux
- Surface: Runtime Host / TUI
Logs, screenshots, or additional context
Election diagnostic from the client:
{"deadlineMs":45000,"elapsedMs":45008,"candidateLaunches":1,"sawEndpointConnected":true,
"observations":{"readyWaitFailed":1,"connected":1},
"lastRegistration":{"state":"recovering","lifecycleMode":"ephemeral"}}
Related: #4027 (artifact-store cold-start costs; fix in #4031).
Suggested directions (needs a design decision on the startup-validation contract):
- Scope recovery to Sessions that actually need it (admissions, non-terminal runs, pending closures) instead of every recoverable Session; or make full-history validation lazy/on-demand.
- Drop the discarded double read of Session messages (keep a single decode).
- Optionally surface recovery progress so waiting clients can extend their deadline adaptively instead of failing at 45s.
What happened
Runtime Host startup recovery decodes the entire durable execution history of every recoverable Session before reporting readiness. On a long-lived workspace (~80 recoverable Sessions, ~500 runs, ~100k event/message rows in
runtime.sqlite) recovery takes ~150s of mostly main-thread CPU, which breaches the client's default 45000ms election deadline:maka --resume <id>fails with "did not become ready before the startup deadline elapsed" (lastRegistration.state: "recovering",readyWaitFailed: 1) even though the Host is making progress. Retrying after the Host settles works.This is the residual bottleneck after the artifact-store costs in #4027 are removed; CPU profile of a recovering Host (CDP Profiler on the main thread) shows:
readSqliteAgentRunEvents(packages/storage/dist/agent-run-store.js)all()callsdecodeStoredRuntimeEvent, GC pressureRoot causes in
packages/runtime-host/src/server/hosted-execution-recovery.ts(prepareHostedExecutionRecovery):readEventsForRecovery+readRuntimeEventsand discards the results — a full decode of every event of every run, purely as a fail-closed consistency check.listForRecovery()(packages/storage/src/session-store.ts) already callsreadMessagesForRecoveryper header and discards the result;prepareHostedExecutionRecoverythen re-reads the same messages per Session.all()and JSON decoding are synchronous main-thread work, so concurrency alone would not help — the fix has to reduce the amount of work, not interleave it.Because admission/message repair decisions only involve Sessions with admissions, non-terminal runs, or pending closures, decoding the history of long-settled Sessions appears unnecessary for correctness — it is startup-time insurance with O(total history) cost paid on every Host start (upgrade, epoch cutover, crash recovery).
How to reproduce
maka --resume <session-id>within the first ~2 minutes.Environment
d27c02af1(main)Logs, screenshots, or additional context
Election diagnostic from the client:
{"deadlineMs":45000,"elapsedMs":45008,"candidateLaunches":1,"sawEndpointConnected":true, "observations":{"readyWaitFailed":1,"connected":1}, "lastRegistration":{"state":"recovering","lifecycleMode":"ephemeral"}}Related: #4027 (artifact-store cold-start costs; fix in #4031).
Suggested directions (needs a design decision on the startup-validation contract):