rpc: in-process daemon serializes concurrent open_session, so a parent fan-out crosses the 30s open deadline

[code-yeongyu]

Summary

On the in-process shared daemon, concurrent open_session requests are processed one at a time. A burst of 32 opens on an idle machine completes in ~14 s with the fastest one taking 10.5 s — every open waits for nearly all the others. Under real load (a parent fanning out 16 children, twice) the serialized queue crosses the 30 s open deadline and every open behind the crossing point times out with an empty stderr.

This is distinct from #1719 (a 0 ms timer from a construction-time deadline — fixed in #1843) and from #1492 (a sync mkdir blocking the main thread). Nothing blocks synchronously here; the no-sync session-path audit is green. It is async per-open work on the single session loop that does not interleave across concurrent callers.

Reproduction

Start an in-process daemon with a mock provider, then from 32 independent RpcClients fire open_session (kind: "worker", retain_on_disconnect: true) simultaneously via Promise.allSettled.

Measured (idle sandbox, engine at the #1843 merge commit)

single open (extension via -e)            OPENED   710 ms
single open (extension via launch spec)   OPENED   738 ms
150 opens, SEQUENTIAL                     all opened, refusal null

32 opens, CONCURRENT:
  opened 32 / failed 0
  wall            13,973 ms
  ok_ms min       10,550 ms     <- if opens ran in parallel this would be ~700 ms
  ok_ms median    12,796 ms
  ok_ms max       13,971 ms

32 × ~430 ms of serialized loop work. Sequential opens never expose it (each waits only its own ~700 ms), which is why capacity tests at 150 sessions pass while burst topologies fail.

Actual (under load)

A live QA matrix — two parents × 16 children on a loaded host — reports per scenario:

childStart: { total: 14, running: 3, completed: 0, errored: 11 }
rootCause:  "Timeout waiting for response to open_session. Stderr: "

A few opens land before the 30 s deadline, most do not, and nothing crashed — so stderr is empty. Applying the #1719 fix changed nothing (identical counts, identical reason strings), which is what separated the two defects.

Expected

A machine-wide daemon whose purpose is one host serving several parents and their children must serve exactly that topology. One of these has to hold:

1. open_session work interleaves across concurrent callers, so burst latency is ~O(1) rather than O(n) per caller; or
2. the open deadline scales with observed queue depth instead of being a fixed 30 s wall clock; or
3. the protocol tells a client it is queued (position / ETA) so the client can wait or stagger instead of timing out blind.

(1) is the real fix. (2) and (3) are mitigations that keep the current serialization but stop it presenting as silent failure.

Acceptance

• 32 concurrent opens on an idle host: ok_ms min within ~2× of a single open, not 15×.
• The burst scenario above completes with errored: 0 on a loaded host.
• A queued open that will miss its deadline reports why (queue depth / position), never Stderr: with nothing after it.

Related

• shared RPC host: in-process unlimited sessions, session kind/context, retain-on-disconnect, capability compat + generation handoff, senpi host CLI #1782 — the shared-daemon tracking issue; this is a gap in its burst behaviour
• rpc: shared-host session worker open deadline is a hardcoded 30s wall clock, so attach silently falls back on a loaded machine #1719 / fix(rpc): budget each session open from its own send (#1719) #1843 — same error string, different defect (construction-time deadline)
• rpc multi-session host stops answering open_session/list_sessions (main thread blocked in mkdir; EINTR storm during session open) #1492 — same symptom family, different mechanism (sync mkdir on main thread)

Evidence held locally (per-open latency table, matrix logs, cleanup receipts).

[code-yeongyu]

Narrowed the site by reading the open path end to end. Each of these was checked and ruled out:

candidate	why not
session-registry.ts openSession awaits	only two, both conditional (closing-reservation settle, cross-generation claim) — neither runs on a fresh open
lifecycleMutex	per-entry, starts as Promise.resolve() per session; serializes ops on one session, not across sessions
runWithProviderScope	providerScopeStorage.run(scope, fn) — an AsyncLocalStorage context switch, no serialization

What remains is session-registry.ts:44-45:

entry.runtime = await runWithProviderScope(entry.scope, () =>
  createAgentSessionRuntime(this.options.createRuntime, {...}))

Everything around it is per-entry and non-blocking, so the ~430 ms per open is inside createAgentSessionRuntime. For 32 concurrent opens to cost 32 × 430 ms rather than ~430 ms, that work has to be mostly CPU-bound with few yield points — it holds the single session loop and the other 31 cannot interleave. That's consistent with the no-sync audit being green: nothing blocks, but a long synchronous stretch between awaits has the same effect on concurrency.

Next step is a CPU profile of one createAgentSessionRuntime call under the daemon to see which stretch holds the loop. I haven't done that; the claim here is only that the cost is in there, because everything else on the path has been read and eliminated.

[code-yeongyu]

Found the mechanism. The cost is not in the registry — it is that every open_session rebuilds the whole resource graph from scratch inside the daemon process.

Call chain, each step read:

session-registry.ts:44      entry.runtime = await createAgentSessionRuntime(this.options.createRuntime, ...)
agent-session-runtime.ts    -> const result = await createRuntime(options)     (thin wrapper)
main.ts                     -> createCliRuntimeFactory(...)                     (the injected factory)
                            -> await createAgentSessionServices({...})          (per session)
agent-session-services.ts   -> new DefaultResourceLoader({...})                 (per session)
                            -> await resourceLoader.reload(...)                 (per session)
                            -> await modelRuntime.refresh({ allowNetwork: false })
resource-loader.ts reload() -> settingsManager.reload()
                            -> packageManager.resolve()
                            -> packageManager.resolveExtensionSources(...)
                            -> (then loads / evaluates every extension module)

No process-level cache anywhere on that path. So each of the 32 concurrent opens re-reads settings, re-resolves packages, re-resolves extension sources, and re-evaluates every extension module — ., ./extensions/omo-member.js, plus the injected provider — in the same process. Module evaluation is CPU-bound with few yield points, which is why the work holds the single session loop and the other opens queue behind it: ~430 ms each, 32 × in series.

Why this is the wrong shape for a daemon specifically. The daemon's extension list comes from daemon-launch-spec.json and is fixed for the daemon's lifetime — the trust boundary in host-launch-spec.ts exists precisely so it cannot change after boot. Re-resolving an immutable set on every open is pure redundant work. The single-process CLI never noticed because it opens one session; the daemon opens hundreds.

Fix shape (real fix, per option 1 above): resolve and load the extension graph once per host at daemon boot — the launch spec already hands the daemon exactly that list — and have per-session createAgentSessionServices bind to the host-level loaded set instead of constructing a fresh DefaultResourceLoader. Per-session state that genuinely differs (cwd, project trust, session-scoped settings) stays per session; the immutable extension graph does not.

That turns burst cost from O(n × full-reload) into O(n × bind), which should bring ok_ms min for 32 concurrent opens down from 10.5 s to near the single-open figure — the first acceptance criterion.

Not done here: the implementation. It touches the resource-loader / session-services boundary and needs its own RED test (32 concurrent opens, assert min within ~2× of single) before any change.

[code-yeongyu]

Design update, with one approach rejected after reading the loader's mutation surface.

Rejected: pass a shared DefaultResourceLoader into per-session services. Two facts make it unsafe:

1. cwd is captured at construction (this.cwd = resolvePath(options.cwd)) with no way to rebind. A shared loader would resolve every session's project-scoped resources — AGENTS.md files, project skills, project trust — against the first session's cwd. Correctness bug, not a perf tradeoff.
2. The loader is a per-instance event bus (emitExtensionEvent / onExtensionEvent). Extensions subscribe on the loader they were loaded through, so one shared instance would deliver session A's extension events to session B's handlers. Cross-session leakage on the daemon — the exact class of bug per-session isolation exists to prevent.

(extendResources also mutates the instance post-construction.)

The right shape is a split, not a share. The expensive immutable part and the cheap per-session part are currently one object:

host-level (once per daemon, from the launch spec)	per-session (every open_session, cheap)
package resolution	cwd
extension source resolution	settings reload (already carved out via settingsAlreadyReloadedFor)
module evaluation of the launch-spec extension set	resolveProjectTrust
→ an immutable loaded extension graph	the event bus, extendResources, and binding the host graph into this session

Concretely: DefaultResourceLoader keeps its per-session shape but gains an optional preloaded extension graph input. When present, reload() skips packageManager.resolve / resolveExtensionSources / module evaluation and binds the preloaded modules instead. The daemon builds the graph once at boot; the CLI path passes nothing and is unchanged. This is settingsAlreadyReloadedFor extended one level deeper — the precedent already in the file.

Honest cost. This adds a path inside resource-loader.ts (~1.4k LOC). The fix is therefore (a) extract the extension-loading unit into its own module with characterization tests, (b) give it the preloaded-graph input, (c) wire daemon boot to build the graph once. A module boundary, not a parameter.

[code-yeongyu]

Built the "preloaded extension result" seam from the split design, took it RED→GREEN, then reverted it before committing — reading the wiring site showed it is unsafe for the daemon.

What worked. A preloadedExtensions constructor option; reload() binds it instead of calling loadFinalExtensionSet. RED (identity assertion fails on main), GREEN 2/2, existing loader suites 49/49 unchanged. The mechanism is fine.

Why it's wrong anyway. createCliRuntimeFactory passes sessionKind and sessionContext into the loader per session, and the comment there says why: "Per-session identity reaches the extensions this session loads and stops there." Following it down:

resource-loader.ts   this.extensionSession = { sharedHostEnabled, sessionKind, sessionContext }
extensions/loader.ts const api = { cwd, sharedHostEnabled, sessionKind, sessionContext, ... }
extensions/loader.ts await factory(load.api)          // once, at load time

The extension API object — carrying cwd, sessionKind, sessionContext — is built at load time and handed to each factory once; extensions capture it in closures. There is no rebind (runner.ts withSession handles session replacement, not identity). So a LoadExtensionsResult is not an immutable graph: it is a set of live instances each holding the identity of the session that loaded it. Binding A's result into B gives B's extensions A's cwd and A's context — the same cross-session leak as sharing the loader, arriving through the result object instead. My test couldn't see it because both loaders used one cwd.

The honest size of this fix. Evaluation and identity-binding are fused at loader.ts:816 (evaluate-then-invoke in one step). The real fix is a two-phase contract in the extension loader:

phase	when	produces
1 — evaluate	once per host, from the launch spec	factory functions (the ~430 ms of CPU work)
2 — instantiate	once per open_session, with that session's api	live extension instances

That is a change to the extension loading contract on the public extension API surface — a changes.md entry, docs, and its own characterization tests. A narrower shortcut exists (sessions with identical cwd + kind + context could share) but the daemon exists precisely for heterogeneous sessions, so it doesn't solve this issue.

Nothing from this attempt is committed. Recording it so the seam isn't re-implemented and the two-phase shape is the starting point.

[code-yeongyu]

One more constraint, found while going after acceptance criterion 3 ("a queued open reports why, never an empty Stderr:") as a smaller mitigation than the two-phase loader.

The queue depth already exists: RpcSessionState has "opening" (session-registry.ts:23), set when an open begins and flipped to "open" when it completes. So "count the opening entries" looked like a one-line addition to HostSessionCounts.

It can't be served the obvious way. Host status is built from a list_sessions probe (host-status.ts:110), and that probe is itself a request on the session loop. Under a burst it queues behind the opens it would be counting — the matrix's scenario C already showed it: Timeout waiting for response to list_sessions. A queue-depth field served through the queue is unreadable exactly when the queue is deep. Green in every test, blind in the one moment it exists for.

Also: the timeout error is minted client-side from a timer (rpc-client.ts:1143), which is why Stderr: is always empty — the host contributes nothing to it. And list_sessions rows carry no status field, so even the counting isn't free.

A real signal has to live off the session loop. Two shapes:

• (a) the supervisor (which already owns the public socket and idle-exit) keeps an in-flight-open counter — incremented at accept, decremented on the open_session reply — and answers status / get_protocol_info from its own side with that number.
• (b) the connection layer emits an immediate queued { position } record on receiving open_session, before handing it to the loop. The client learns its position even if the open never completes; rpc-client.ts:1143 can then put the last known position into the timeout message instead of an empty stderr.

(b) is the smaller and more useful one — it fixes the diagnosability gap directly. Both are protocol additions (docs/rpc.md + changes.md), smaller than the two-phase loader but not free. Not built here; recording so the mitigation isn't attempted through list_sessions.

[code-yeongyu]

Mitigation (b) — the queued { position } record — read through every consumer. It is smaller than I first sized it, with one trap that would have broken the desktop.

Where each piece already lives in senpi:

need	precedent
in-flight depth	session-command-router.ts:398 openWithBarrier registers every open in opensByConnection; depth = sum of set sizes
a record to a connection with no session	session-event-writer.ts:346 broadcastHostRecord — host_stalled already does this; queued is the same shape targeted at one connection
carrying it into the error	rpc-client.ts:1143 mints the timeout message; store the last position per request and include it

Consumers are tolerant — this is senpi-only. The desktop decodes socket lines as Schema.Unknown into Readonly<Record<string, unknown>> (omoRpcConnection.ts:56, :241). An unknown type parses and is ignored, not rejected. No desktop or omo change is required to survive the record; only to use it.

The trap. The desktop's settle() matches records to pending requests by the response-id field (OmoSharedProcess.ts:441-446). If queued put the open's request id there, the desktop would take it as the open's reply, settle open_session with a queued record, and log the real reply as a late reply — breaking every desktop open under load. So the record must carry the requester's id under a distinct key:

{ "type": "queued", "for_request": "<open's id>", "position": 12, "in_flight": 31 }

With for_request instead of the response-id field, omoResponseId() returns undefined and the record falls through to the notify path exactly as host_stalled does.

Emit point: in openWithBarrier, before return this.open(command, owner) — the record leaves on the connection's queue immediately, so the client knows its position even if the open never completes in time. Then rpc-client.ts:1143 can say queued at position 12 behind 31 opens instead of Stderr: .

Docs + changes.md + router/client tests. One senpi PR. Not built in this pass — but every consumer-side risk is now read rather than assumed.

[code-yeongyu]

Third acceptance criterion delivered in #1845 → 18c869cbab (25 checks green).

a queued open that will miss its deadline reports why, never Stderr: with nothing after it

On accepting an open_session the router now sends the opener, before the open enters the loop:

{ "type": "queued", "for_request": "open-7", "position": 12, "in_flight": 11 }

Two details that came out of measurement rather than preference:

• in_flight is host-wide, not per connection. Every session shares one loop, so the total is the wait a caller actually faces. Per-connection counting would have told two parents they were each first while both sat behind 32 opens — exactly the topology that exposed this.
• for_request, never the response-id field. A client settles pending requests by response id, so a queued record wearing the open's id would be taken as the open's reply and the real reply logged as late — worse than the timeout it replaces. Both tests assert queued.id is absent, so the trap is pinned by the suite.

Still open here: the serialization itself. This makes the queue visible; it does not shorten it. The two fix shapes and their real sizes are in the comments above — the two-phase extension loader (evaluate once per host, instantiate per session) remains the actual fix, and it is a public extension-API contract change.

[code-yeongyu]

The cost model, measured — this changes the fix

From source on main (18c869cbab), sandbox agent dir, 44 extensions, wall-clock brackets (reverted after):

Where an open's time goes (real RpcSessionRegistry.openSession):

cold #1: 462 ms = createAgentSessionServices 447 + createAgentSessionFromServices 10
warm #2: 128 ms =                             119 +                               9
warm #3: 118 ms =                             108 +                               9

session_start across 44 extensions is 9 ms. createAgentSessionServices is ~92% of every open. Inside it, warm: ModelRuntime.create ≈ 110 ms in parallel with reload ≈ 105 ms (joinStartupBranches), then refresh ≈ 4 ms. Total = max of the two branches, not the sum.

Why the matrix paid ~400 ms/open when a warm open is ~120 ms — 8 concurrent opens on a warm cache:

wall 1013 ms   min 1008   median 1011   max 1012   sum 8080

Every open waited for all eight. With start/finish stamps on 3 concurrent opens: all three enter at @0 (no lock), all three reload-done at @201 together, all three runtime-done at @215 together. The work is synchronous CPU on one loop, interleaved round-robin — so each open's wall time is N × its own cost. 8 × 126 = 1008. 32 × ~330 on a loaded box = the 10.5 s minimum I measured earlier. Not a lock, not a cache miss (the cwd-keyed factory cache is warm and working: cold 462 vs warm 118). It's redundant CPU, done N times for N sessions of a host whose extension set is fixed for its lifetime.

Consequence for the fix. A two-phase extension loader alone moves ~0 ms of the warm number, because ModelRuntime.create runs in parallel at the same cost and becomes the sole critical path. Both branches must be hoisted to host scope:

1. ModelRuntime per host — already injectable (options.modelRuntime), carries no session identity, keyed on agentDir. No public-API change. Removes ~100 ms of loop CPU per open.
2. Discovery + evaluation per host — the reload branch. Not injectable today: DefaultResourceLoader captures cwd, and the extension API binds sessionKind/sessionContext at load. The two-phase split's purpose is to hoist discovery + module evaluation out of the per-session path (the jiti import is already cached), while phase 2 keeps binding identity per session.

Neither alone helps; both together leave ~10-20 ms of per-open loop CPU, so 32 concurrent opens cost ~0.5 s instead of 10.5 s. Doing (1) first — small, no API change, and it moves the critical path wholly onto reload, which makes (2)'s effect measurable on its own.

My earlier comments here designed against "module evaluation is the cost". It's a fraction of it. Corrected.

[code-yeongyu]

Step (a) is up as #1846, measured live on the built daemon rather than from source — and the first version didn't work, which is the useful part:

Sharing the runtime alone changed nothing over the socket (base 450/458/505 vs fix 520/463/518 ms). The unconditional per-open refresh() recomposes every provider the shared instance has accumulated, serialized on it — 29-118 ms against ~5 ms on a private runtime. I'd swapped a parallel 100 ms for a serial 30-120. Scoping the refresh to what each open registered is what made it real:

9 rounds, alternating	base median	fix median
single warm open	277 ms	162 ms (~40%)
8 concurrent, wall	507 ms	439 ms (~15%)

The concurrent gain is bounded exactly as the cost model predicted: reload (~90 ms/open of loop CPU, interleaved N-way) is now the sole critical path. That's step (b), and it needs the two-phase loader.

[code-yeongyu]

Step (b1) is up as #1847 — package resolution memoized per host on its real inputs, stacked on #1846.

packageResolve 68 → 0-1 ms on warm opens. Live over the socket, 8 rounds against main: 8 concurrent opens 448 → 320 ms median, single open 222 → 168. Cumulative since before #1846: 507 → 320 on the burst (~37%).

Invalidation is by key (settings content is part of it, reloaded before the key is computed) and pinned with real loaders: a settings.json change re-resolves; three concurrent loaders share one in-flight resolution.

What's left, measured: reload is now ~53 ms warm-concurrent, of which ~48 is per-extension work spread across two code paths — 15 of 49 extensions go through loadExtension, the builtins through the inline route. A sync fast path on the cached-factory route measured as a no-op (49 → 48) and did not ship. Hoisting phase 1 for both routes is step (b2); after it, the per-open critical path is factory + AgentSession + session_start ≈ 30-40 ms, which is the shape criteria 1 and 3 need.