feat(server): give Codex turns worker-owned execution

[chuks-qua]

What

Each admitted Codex turn owns its execution in a fixed four-worker pool. One asynchronous SQLite writer commits ordered effects before publication. Bounded queues and execution fences isolate late results, crashes, Stop, and write failures.

The follow-up removes repeated history loads, reuses narrative queries, groups ready independent writes, copies only changed recovery records, reduces worker bundle size, and releases idle memory. Turn settings now persist in one atomic update.

Why

Seven active tasks previously caused 4.9–8.1 second server pauses. The first worker version improved control responsiveness but made completion and memory worse. These changes recover that performance while retaining durable ordering. Refs #1760.

Evidence

The final ten-run record contains every run and the measurement limits. Median completion fell from 72.9 s in the initial PR to 26.1 s. Peak event rate increased 39%, sampled memory fell 22%, and terminal and model requests improved. No main-server pause was logged in this default cohort. All runs passed event ordering, durable reload audits, and fixture cleanup.

Against the original implementation, completion is 21% faster, but peak event rate remains 11% lower and sampled memory 2.7% higher. Memory is the maximum of three working-set samples, not a continuous peak. The all-original-metrics target is not fully met.

Stop-one returned in 1.10 s, cancelled only its target, and left six peers to finish. Real Electron checks verified completion, cancellation, reload, task switching, models, and a PowerShell command. A separate traced run recorded an unattributed 842 ms startup pause, still tracked in #1767.

UI Changes

The existing captures below show the earlier lifecycle fix in this PR. A completed reply remains complete after reload, and Stop follows the durable terminal receipt. The final runtime repeated these checks; its timings and limitations are in the linked record.

electron-merged-stop-reload.mp4

Config Changes

No migration or new default configuration is required. MCODE_SERVER_WORK_TRACE=1 now records only the main server thread.

Review Notes

User-dispatched Codex turns use worker ownership. Claude, Cursor, and provider-originated resumes retain their existing route. Legacy event workers now start on demand. Claude and Cursor have focused adapter coverage, without new live performance measurements. Tests, lint, typecheck, build, and Linux canary packaging passed at bb1e241d. Follow-up fix 626f3e84 passed 109 focused tests, web typecheck, and targeted lint. CI passed through journal correction 81e6f25b; the latest browser correction is awaiting CI. This PR is not merged.

Follow-up regression correction

The user reported a second-turn spinner after the original verification. I reproduced it in the real Electron dev app with Codex GPT-5.6 Luna. The backend had completed, but the renderer discarded the new start because two empty file-effect IDs looked like a duplicate. The fix binds the new execution before applying completion events. Earlier first-turn and Stop checks missed this workflow.

The reproduction and verification record records five real successful turns in one task, switching away and back while the third turn streamed, another follow-up after navigation, 46 seconds remaining idle, and a successful follow-up after reload. All five replies were durable with distinct completed execution receipts. Owned tasks and runtime were cleaned up. This was Codex text-response proof, not new provider-wide or file-change Review coverage.

Mid-turn switching in the real Electron app:

mid-turn-switch.mp4

Event delivery audit update

Commit 81e6f25 gives each bounded task journal its own generation, preventing valid events from looking stale after journal eviction. Thirty focused server tests, server typecheck, and the rebuilt seven-task delivery proof pass. The live check delivered all 840 tool starts and 840 results with no sequence gaps or duplicates. CI tests, lint, typecheck, build, and canary packaging passed at 81e6f25.

Commit 419b537 corrects a second reproduced failure: rejected browser cursor writes dropped valid replies while the backend had completed. Failed writes now retain a cursor in the running renderer and suppress duplicates. The next successful write persists the latest cursor. Initially unreadable or malformed stored cursors keep their existing behavior.

A direct comparison corrected the earlier design rejection. Failed write, reload, hydration and old receipt republication leave the old and new implementations with the same persisted cursor and history. That replay gap already existed. The independent judge withdrew its objection to the narrow write-failure correction. This change does not claim complete recovery after reload while storage remains broken.

The clean-start Electron check used real Codex GPT-5.6 Luna with High reasoning and Full Access in the fixture repository. Five replies completed and persisted. During 340 injected cursor-write failures, the follow-up appeared, task switching preserved the active turn, and all 1,000 streamed integers appeared exactly once in order. Restoring storage persisted the cursor; reload and another follow-up passed. Five completed execution receipts and zero renderer errors were recorded. Owned task and runtime cleanup passed.

The browser correction passed 103 focused tests, web typecheck, targeted lint, and independent scoped review. Latest-head CI is pending. Other providers and broken-storage reload with old outbox republication were not live-tested in this correction. Investigation notes and detailed receipts remain under .dev at the user's request.

Browser cursor failure before and after: