Skip to content

rocky: background reads keep the shared database connection busy; ContextMD moments queue behind them #256

Closed
#257
Closed
rocky: background reads keep the shared database connection busy; ContextMD moments queue behind them#256
#257

Description

@lukemaj
lukemaj
opened on Oct 13, 2026

What: Rocky's backend pid 6379 on build 9b495751 stayed up from 12:59Z through 17:00Z, and RSS sat between about 893 MB and 911 MB for the last two hours.
Why: The crash loop was a cursorless boot replay of the event log. #258 and #267 are merged and live on that build. The higher baseline matches the data: 2,492 threads and a 7.6 GB database, against cavallo's 797 threads at about 346 MB RSS on the same build.
So what: Closed. The hourly watch is stopped. No GitHub sub-issues were open. Serving latency stays on #239 and the Cue deadline stays on #248. Provider updater ownership stays on #263, which has no assignee.

Evidence (rocky, 2026-10-13, Chromeria 916b0feb, read-only)

  • Connection occupancy: in a 57 s trace window (21:09Z), sql.execute and sql.transaction spans on chromeria-v2.sqlite covered 99% of wall time. There were 16,768 transactions (p50 0.9 ms, p99 23 ms, max 2,669 ms), 10 of them over 500 ms.
  • Event decoding: OrchestrationEventStore.rowToV2StoredEvent ran 127,398 times in 75 s (about 1,700 per second), 24 minutes after a restart, so this is steady state. These are root spans (no parent recorded). The query is SELECT sequence, event_id, command_id, ... FROM orchestration_events (readAgentEvents, paginated by EventStore.read / ws.ts replay).
  • Per-thread polling: in about a minute, SELECT payload_json FROM orchestration_v2_projection_threads WHERE thread_id = ? ran 28,676 times and ..._runtime_requests WHERE thread_id = ? 7,677 times. childThreads.readRetirementState and childThreads.reconcileRequests are the largest named parents.
  • Queueing, not work: a trivial SELECT provider FROM orchestration_v2_projection_provider_sessions WHERE status IN (...) logged 747 spans over 100 ms in 15 minutes. sql.execute spans include the wait for the connection.
  • Trace volume: about 92 MB of server.trace.ndjson per minute; the 11 rotated files hold about 70 s.
  • Effect on serving (21:12 to 21:27Z, 170 moments): 23 of 24 over_budget moments and 53 of 55 moments over 600 ms overlapped a database span over 100 ms (median overlap 409 ms and 196 ms). Only 32 of 91 moments under 600 ms did. The same serving queries run alone against the same file take p50 under 1 ms, max 17 ms.
  • Not the cause: the event loop answers /health at p50 9 ms, p99 82 ms. The HTTP span and the moment's own clock differ by p50 1 ms, max 9 ms. TCP connect takes under 1 ms.

Acceptance criteria

Non-goals

  • Changing the ContextMD budget or the hook cut-off.
  • Trace volume itself, except where the fix reduces it.

Blockers

None.

Required proof

  • The same occupancy and decode-rate measurements before and after on rocky, plus the serving log's over_budget rate for one day.

Elon record

  • Requirements and who asked: show where pre-tool time goes and what still makes calls slow (planner, 2026-10-13, contextmd serving: Cue misses its wait on long threads; pre-tool moments that start Cue now run out the hook deadline #248); file problems outside scope at once (planner).
  • Deleted: a shorter Cue window and faster scoring as the next lever for pre-tool time. Cue was mostly not_called overnight, and pre-tool still took p50 625 ms.
  • Bottleneck: the single shared database connection.
  • Checked myself: rocky's trace files over 15 minutes, the serving log, the serving queries run read-only against chromeria-v2.sqlite, and a /health latency probe.

Activity

  1. lukemaj
    added a commit that references this issue on Oct 13, 2026

    fix(childThreads): skip ancestor loads for children with no requests …

    7184184
  2. lukemaj
    closed this as completedin #257on Oct 13, 2026

  3. lukemaj commented on Oct 13, 2026

    @lukemaj
    lukemaj
    on Oct 13, 2026
    ContributorAuthor

    Planner, 2026-10-14 00:15 CEST. Rocky's backend server has been crash-looping since 2026-10-13 20:09Z. Each server process reaches about 4 GB and aborts in V8 OOMErrorHandler: 26 macOS crash reports, then restarts about every 80 to 90 s.

    Facts:

    Not yet known: which caller reads about 29,000 events per cycle. The candidates are the repeated completion deliveries to a39db5c0 and a thread subscription that streams a large thread from sequence 0. A heap snapshot near the limit would settle it.

  4. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Planner, 03:30 CEST: a heap sampling profile of a crashing backend (inspector over SIGUSR1, 25 s, on build 7184184). Script: ~/scratch/heap-sample-256.mjs on rocky.

    • 732 MB of the ~1 GB sampled was allocated in Effect Schema parseJson (JSON.parse inside fromJsonString decoding). The chunk is named claudeHistoryWorker-*.mjs, but that is only the bundler's shared chunk; the code is generic schema decoding.
    • Another 118 MB was parseUnionCandidates, from union decoding of the same payloads.
    • That matches the about 29,000 OrchestrationEventStore.rowToV2StoredEvent spans per trace file. Each server start decodes tens of thousands of stored events, and the payload JSON is parsed per row.
    • About 29,000 events is roughly the size of the planner thread mcp:14a2b264-c473-4ee3-b3e5-d717022a0871:fork (26,096 events, 51 MB of payload). That thread is the leading suspect for the full read on every boot. The caller is still not identified: the trace spans carry no thread id.
    • Ruled out: the desktop renderer (paused with SIGSTOP), and NODE_OPTIONS (the backend ignores it; no heap snapshot was written).

    Suggested next steps, in order:

    1. Log the stream id and sequence range on the event read path, and find which caller reads a whole thread at boot.
    2. Bound or stream that decode instead of materializing it.
    3. Until then, test the containment: stop that thread from being loaded at boot (archive it, or move the planner to a fresh thread).

  5. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Planner, 07:50 CEST, 2026-10-14. Containment test failed: I archived the old planner thread mcp:14a2b264-c473-4ee3-b3e5-d717022a0871:fork at about 07:31. The backend kept crash-looping. From 07:30 to 07:40, server pids 20399, 23241, 24705, 26196, 27576 and 28991 each grew to about 4.5 GB and died after 60 to 300 s; the desktop app also restarted at 07:39:26. So loading that thread is not the only cause, or archiving does not stop it from being read at boot. Latest trace file: 10,954 OrchestrationEventStore.rowToV2StoredEvent spans out of about 11,100. No pending or running rows in orchestration_v2_effect_outbox. Planning moves to a Claude Code session on bestia (tmux planner) while rocky is down; next step is finding the caller (log stream id and sequence range on the event read path).

  6. lukemaj
    reopened this on Oct 14, 2026

  7. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Canonical handoff

    What: Rocky's backend is the relaunched process on build 614c13f11120 (pid 14105 at 10:40 CEST, RSS 366384 kB). PRs #267 and #278 are in independent review and are not merged. This thread is not restarting rocky.
    Why: The cursorless boot replay is already fixed on the running build. #267 is the bounded-replay follow-up, now with the page-release proof commit. #278 fixes the Test Server 6 continuation-offer race.
    So what: Merge a PR only after review/independent-sol and review/independent-grok both succeed on its exact head and checks stay green. Do not deploy either PR. Any rocky relaunch goes through the planner, because another agent is changing Tailscale.

    Elon record

  8. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Findings: upstream

    What: No merged upstream fix surveyed covers the suspected delegated-completion redelivery loop; recommend a fork fix after the repro names the caller, rather than porting any of the four initial candidates.
    Why: Three candidates are still open PRs and one is an issue closed as not planned; the applicable merged V2 replay protections are already in 7184184.
    So what: The diagnosis worker should distinguish recovery delivery from a cursor-zero event subscriber and prove the smallest caller fix on the clientless repro before implementation is selected.

    Surveyed on 2026-10-14 against fork 7184184a5919f1f86296febca690e4b9fa389059 and upstream main 4d73e1d5cdd32dbfb470e9422421cc40475d5e99. This supports the Objective by making the Chromeria backend reliable for Promachos and delegated work. Read #256 and all its comments, including the canonical handoff. No checkout/worktree changes, server launch, or database access. The canonical handoff is unchanged.

    Four requested candidates

    Dates are UTC. An open PR's head SHA is not a merge SHA. Presence means the particular change, not merely an existing feature with a similar name.

    Candidate State and date Merge SHA or inspected head Present in fork? Actual diff / fork evidence Stops a clientless boot event-decode storm?
    #18856, double timeline-item validation Issue, closed not planned, 2026-10-12 23:52:42Z No merge SHA, no upstream PR diff No optimization. The reported double pass is present. packages/contracts/src/baseSchemas.ts:ForwardCompatibleUnionArray and packages/contracts/src/orchestrationV2.ts:projectedTurnItemArray still use ForwardCompatibleUnion followed by decodeTo(Array(toType(...))). The issue concerns snapshot/history arrays, not removing a stored-event reader. The maintainer explicitly closed it rather than merging a fix. No. Fewer validations would reduce per-item cost but neither prevent replay nor durably settle delivery.
    #18220, viewport snapshot Open, created 2026-10-11 No merge SHA; head 9e7f6b8510507491962a0c45a0f35e7681ce50a3 No. Adds acceptViewportWindow to ws.ts:subscribeOrchestrationV2Thread, viewportWindow to ThreadStream.ts:buildBoundedThreadStreamSnapshot, deferred historical support/checkpoint files, and StreamSerialization.ts. These additions are absent in the fork. Fork already has older bounded snapshots. Its event-store hunk optimizes append encoding/normalization, not rowToV2StoredEvent reads. No demonstrated coverage. It reduces client snapshot and wire work; it does not change EventSink.catchUp, cursor-zero background replay, or recovery delivery.
    #19226, thread-list reads Open, created 2026-10-14 No merge SHA; head febd5c2c3ca54bbae86127c987abd641014b7f15 (first commit 2d80bb24a6d9bf8dd0e62f383eb3d45178ba58c7) No. Adds packages/shared/src/nodeSqliteClient.ts:readOnly / readerWorker; opts in ShellStream snapshots and ThreadPullRequestService discovery. Fork persistence/Sqlite.ts:layerFromPath has no readerWorker, and the shared client has no readOnly routing. No. It moves selected SQL reads off the main connection. It does not reduce event decoding, retained replay, or recovery redelivery.
    #18649, exactly-once delegated results Open, created 2026-10-12 No merge SHA; head d5de3e23f79ef08b23a75e596256fb5051997282 (first commit 1afd87ce5c2bd82e9c055f72172fd48d6241c0c4) No. Adds resultRunId and task delivery generation to contracts:OrchestrationV2DelegatedCompletionTaskDelivery, rejects stale acknowledgments, and changes ProjectionStore.needsRecovery/getRecoveryThreadIds plus Orchestrator.finalizeAppOwnedSubagentTask to correlate each result run. Fork delivery schema has only state and observedByRunId; it has its own later-result implementation from Chromeria #213. Unknown for the delivery hypothesis; no coverage for the replay hypothesis. This repairs later-result suppression and stale ownership. It cannot by itself guarantee completion of a dispatch that never commits before process death; it does not bound EventStore.read.

    Merged work found by mechanism search

    These are merge SHAs, not PR branch heads. Where the original V1 source was removed, that is called out rather than calling an obsolete patch a missing V2 fix.

    Candidate Merged UTC Merge SHA Present in 7184184? Change and source evidence Clientless boot relevance
    #5147 2026-07-31 ca72e381c64f25d771236eecf70219f68e5f365b Yes, ancestor; V2 replacement present First surveyed owner of the thread catch-up cap: stale cursors fall back to a snapshot instead of globally replaying from zero; lightweight legacy HTTP/project snapshots avoid full-DB hydration. Fork ws.ts:subscribeOrchestrationV2Thread now preflights replay. Client subscription protection; not a stop for clientless background recovery.
    #8992 2026-09-02 7e460f429b740180cd72730418262a2df971ba54 Yes, ancestor; V2 replacement present Adds SQL payload-byte preflight before replay. Fork ThreadStream.ts:isThreadReplayRawPayloadSafe has a 1 MiB raw budget; ws.ts calls getAgentReplayStats before decoding. Stops oversized client replay, not every internal event-store consumer.
    #9521 2026-09-04 9c9ae3dc0e94a957d9c4a61bb211caf914828054 Yes, ancestor; equivalent V2 ordering present Starts subscription buffering before catch-up/snapshot so events cannot disappear during startup. Fork EventSink.ts:stream and LiveStreamBudget.ts:replayAndBufferProjectedLiveEvents retain that ordering. Prevents lost events, not excessive historical reads.
    #9715 2026-09-04 108f295cc3672716c3cb8291ce487846df5ce098 Yes, ancestor; V2 protection present Bounds slow-client retained buffers by count/bytes and resyncs on overflow. Fork LiveStreamBudget.ts is used by bounded EventSink.stream subscriptions. Client retention protection, not a cursor-zero internal replay stop.
    #9726 2026-09-04 50bfca43d76ced00f5d67cfbab8bc44c50eb0e53 Yes, ancestor; V2 SQL filtering present Reads/counts only the selected thread instead of decoding a global event range then filtering. Fork OrchestrationEventStore.ts:readApplicationRows filters stream_id, and ws.ts passes threadId. Prevents unrelated-thread subscription decoding; does not prevent a background reader asking for all history.
    #10777 2026-09-09 08463e2c401ce87858aaaebcb70ed86fb002fb5f Yes for its original change; not applied to V2 EventSink.catchUp Replaces recursive Stream.concat pagination in V1 readFromSequence/readAggregateRange with Stream.paginate so consumed pages can be released. Fork V2 OrchestrationEventStore.readAgentEvents also uses Stream.paginate, but its outer EventSink.catchUp still recursively concatenates collected 256-event pages, as upstream main does. Closest historical retention fix, but a literal port targets removed V1 methods. Applying the same principle to V2 would be new work requiring repro proof.
    #11499 2026-09-15 24476841df12112dea4cb37b34e5c52ad19a9f9e Yes, implementation present through V2 readAgentEvents pages until filtered range/total limit is exhausted instead of returning only one page. Fork OrchestrationEventStore.ts:readAgentEvents matches the pagination behavior; EventStore.ts:readByCommandId delegates to it. Completeness fix, not a replay stop; unlimited callers still read the whole range.
    #12846 2026-09-21 df6e17064e15372784d4a331c90654f3e43ecc7b No literal change; obsolete V1 path removed Replaces ProjectionPipeline.bootstrap attachment-cleanup replay with metadata-only SQL, avoiding unrelated payload decode. orchestration/Layers/ProjectionPipeline.ts is absent in fork and current upstream V2; fork ProjectionMaintenance explicitly does not replay events through a reducer. Would fix that V1 startup reader, which is not the named V2 path. Do not port the removed subsystem.
    #13340 2026-09-24 e4eb9977f0b02b29cb2ee361b0c433c685e87615 Yes, ancestor relay/AgentAwarenessRelay.ts:terminalWorkSinceStart suppresses historical terminal alerts when startup has no publish history. Stops old alert publication; does not bound V2 event-store reads or delegated completion.
    #5311 2026-08-05 99c957d26a3eab8b817ee02a14a8ada5dfe3826f Yes, evolved V2 mechanism via #2829 Introduces durable delegated delivery ownership/acknowledgment/disposal rather than redundant completion turns. Fork Orchestrator.ts:planDelegatedCompletionDelivery/finalizeDelegatedCompletionDelivery and delivery contracts contain these mechanisms. Its original two-delivery cap was subsequently removed by #13938. Delegated-delivery story only; no evidence it stops this never-committed-dispatch hypothesis.
    #13938 2026-09-27 b39e62dc92ced0224bbfe1319db8d1926e02532a Yes, through V2 Removes the two-delivery cap and batches pending siblings into successive wakes. Fork finalizeDelegatedCompletionDelivery reserves a successor from pending tasks; OrchestrationV2DelegatedCompletionCohort has no settledDeliveryCount. Ensures later child completions wake parents; not a decode or recovery-loop bound.
    #13619 2026-09-25 6c39f90e99ab4be831c5dca87d01709667599820 Yes, through V2 Recovers runless native subagent roots/items. Fork ProjectionStore.getRecoveryThreadIds includes node.run_id IS NULL / root_turn, and ProviderRuntimeRecoveryService cancels stale runless roots. Native stale-status recovery; not the app-owned delegated-completion replay mechanism.
    #2829 2026-10-02 de343914273eceb852a1d1d739cd1d38df7796ee Yes, ancestor Earliest merged owner found for the current V2 live-only fixes: Orchestrator.terminalEventsAfterSequence starts terminal reactions at the high water; streamDomainEvents captures latest sequence instead of replaying genesis. The merge source and fork both contain them. V2 also retains the 128-event/1 MiB subscription preflight and iterative readAgentEvents. Already prevents those specific clientless startup historical replays, but not other subscribers that explicitly/default to zero.
    #15323 2026-10-03 5108c978b1d9a8bd029e5ea96781d8cbc90b3877 Yes, ancestor Moves delegated recovery after runtime reconciliation and before effects; guards results while restart continuation is pending. Fork Orchestrator.recoverDelegatedTasks/childAwaitsRestartContinuation and serverRuntimeStartup implement this. Recovery correctness; still deliberately re-offers eligible persisted deliveries after restart.
    #15604 2026-10-04 253dea360ef0c743c5de8ba04f8412a7eb4694ac Yes, ancestor Stops restart continuation from waking settled roots merely because background work was lost. Fork RestartContinuation.ts:restartContinuationRun requires a running/prepared continuation. Different automatic wake path. Does not suppress a legitimate delegated delivery owed by a settled parent.
    #17028 2026-10-08 0bfd9dd19d0c9418a936190525ec8f4c4fdae09b Yes, ancestor Excludes held queued runs from delegated outstanding work and orders queue promotion before finalization. Fork SubagentProjection.delegatedTaskProgress and Orchestrator.handleTerminalRun match. Removes false pending work; not full-log replay or an uncommitted delivery bound.
    #17041 2026-10-08 9fba2093e50f208c52d9041c70a793296e311477 Yes, ancestor Forces command lookups to idx_orch_events_command_id rather than scanning all V2 events. Fork OrchestrationEventStore.ts:readApplicationRows contains that exact conditional INDEXED BY. Speeds durable command retry lookup; does not stop whole-thread reads or make a not-yet-committed dispatch committed.
    #17141 2026-10-08 1f31906c8e3003b2a9d7118c4da163544fa86586 Yes, ancestor Moves shell snapshot decoding outside the SQL transaction. Fork ProjectionStore.ts:readShellSnapshot returns the decode step for callers to run after commit. Reduces shared-connection occupancy, not historical event decoding.
    #18471 2026-10-13 fb8d0fc19c27ff860bdefb7dcfe69eae546f837c No Adds PRAGMA optimize = 0x10012 after migrations in persistence/Sqlite.ts:layerSetup; fork has no such call. Fixes #17955. Real missing query-planner improvement, but not a bound or stop for decoding requested events. Not a demonstrated #256 fix.
    #17696 2026-10-10 c572cb71d2711d8cdb4ff7e0ba88af5e9a23aa81 No Gives resumed follow-ups separate task ownership using Orchestrator.recordSubagentFollowup, delegatedTaskId, and finalizeAppOwnedSubagentTask; these are absent in fork. Chromeria #213 instead adds later-result handling to the original task. Different later-result ownership model; neither bounds replay nor proves recovery redelivery is stopped.
    #18778 2026-10-13 265fb7000f84d12a44c4e87c17426e1f29e0bb36 No Applies provider-confirmed native subagent outcomes in Orchestrator's stopped-work handler (command.subagent). Fork lacks that handler branch. Native subagent status repair, not app-owned task recovery or event replay.
    #19099 2026-10-13 05e4f888a0d8591c790d036140080092cf024d30 No Allows snooze with queued automatic wakes; cancels lost provider-buffered notifications on Stop/restart using NotificationMailbox.isBufferedProviderNotification, absent in fork. Its predicate explicitly excludes delegatedCompletion. Does not dispose delegated-completion messages or bound event reads.

    Keep the two causal stories separate

    1. A subscriber requests history from sequence zero. Fork ws.ts:subscribeOrchestrationV2Thread already filters by thread and checks 128 events / 1 MiB before decoding a persisted resume; without a cursor it sends a projection snapshot and tails from its snapshot sequence. The cited client PRs do not explain a clientless boot by themselves. Internal EventSink.ts:stream defaults afterSequence to zero; catchUp reads and collects 256 events repeatedly. The same recursive Stream.concat(current, loop(last)) is still present on surveyed upstream main. rowToV2StoredEvent still parses payload/metadata and then decodes the domain event and stored envelope on both heads.

      Concrete fork source lead, not a confirmed crash cause: apps/server/src/provider/providerAutoUpdate.ts:layer calls eventSink.stream() without a cursor, then filters WAKE_EVENT_TYPES after replay/decode. apps/server/src/server.ts loads ProviderAutoUpdate.layer into the boot runtime. Git attributes this fork module to 6e1653b671 (Chromeria feat(fork): carry Chromeria foundation onto upstream V2 #178); the path is absent upstream main. This is a clientless full-log reader worth correlating with the repro. Its existence does not prove it accounts for the observed whole-thread read counts or retained heap.

    2. Recovery keeps offering a delivery because its durable acceptance never commits. Fork recoverDelegatedTasks reads projection recovery candidates and calls offerDelegatedCompletionDelivery; the latter skips archived/deleted threads, non-open/empty cohorts, and already accepted messages unless they are undelivered mailbox steers. It does not directly invoke EventStore.read. Re-offering an outstanding cohort on the next boot is expected recovery behavior when acceptance is not durable. The assertion that the dispatch to the settled thread never commits is still a hypothesis: source review cannot establish the actual command receipt, accepted message, generation, or crash timing. #15323 and #15604 address restart ordering/continuation; #18649 addresses result-run/generation correlation. None is a verified stop for this exact hypothesis.

    Additional unresolved upstream records

    • #17433, open issue, and #17441, open PR created 2026-10-09, head c3078de425c57f00ba1fd2d7934d23192f15da46, no merge SHA, not in fork: fixes notification-started runs escaping bounded snapshot turn counts. Actual diff adds ProjectionStore SQL wake_anchors and threadHistoryPaging.threadHistoryTurnStarts; fork lacks both. This is projection snapshot loading on open/reconnect, not rowToV2StoredEvent replay at clientless boot.
    • #17512 and #18859 are open issues, not merged fixes. They report missing child finalization / queued completion notices, not evidence that the rocky: background reads keep the shared database connection busy; ContextMD moments queue behind them #256 recovery dispatch causes a decode/OOM loop.
    • #7538, merged 2026-08-27, SHA a6797b3b97dca6b6941574ff062d069c45c89b9a, ancestor present: makes V1 ProjectionPipeline.bootstrap replay all unapplied events rather than one page. That V1 path was removed by V2; this is a correctness expansion of required replay, not the desired loop stop.

    Evidence and limits

    Used gh pr view, actual gh pr diff source hunks, issue bodies/comments, upstream commit histories for V2 EventSink and Orchestrator, main source at the pinned SHA, git show 7184184a5919:<path>, git grep, and read-only git merge-base --is-ancestor. Searches covered rowToV2StoredEvent, readAgentEvents, EventStore.read, recoverDelegatedTasks, delegated completion/dispatch, subscribeThread, startup events, whole-thread reads and replay. The quoted delegated-completion:dispatch search failed GitHub's query parser; retrying with plain delegated-completion terms and reading the source supplied coverage.

    Some historical pre-V2 merge objects are not in the local object database; for #11499, #5311, #13619 and #13938, presence is established by matching the relevant implementation in the V2 merge/fork, not an ancestry claim. #12846 is a known non-ancestor and its obsolete path is absent. gh pr diff for #2829 returned an empty diff, so the relevant actual merged source was inspected through local git show instead. All other listed candidate diffs were retrieved.

    Not checked: no clientless runtime reproduction, heap-retention experiment, database receipts/projections, or before/after performance. The survey cannot confirm either suspected crash cause or claim a candidate stops OOM. The next decisive evidence is the repro caller/sequence range and durable delivery state around its dispatch.

    Elon record

    • Requirements and who asked: Planner/user requested an upstream survey for rocky: background reads keep the shared database connection busy; ContextMD moments queue behind them #256, concrete SHAs and fork presence, and one findings comment before choosing a port or fork fix.
    • Deleted: Treating open PRs or a closed performance issue as merged fixes; porting obsolete V1 subsystems; runtime/database/worktree work outside this research assignment.
    • Bottleneck: Identifying which caller requests historical events on the clientless repro, while separating that from uncommitted delegated delivery.
    • Checked myself: All four initial candidate states, their applicable source changes, relevant merged histories, pinned upstream main source, fork source/ancestry and clean unchanged main checkout.

    Research: gpt-6.1-sol, high reasoning effort, Codex harness in T3 Code.

  9. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Findings: caller

    What: The boot-time decode is ProviderAutoUpdate's cursorless eventSink.stream(), and a clientless boot of this repro dies in V8 OOM.
    Why: stream() with no cursor replays every agent event through rowToV2StoredEvent on every start. The startup path already wakes the updater once, so the replay has nothing to deliver.
    So what: Subscribe at latestSequence, the same live tail streamDomainEvents already uses. That microfix is clear-cut enough to ship before a bounded-read design. Deploy stays gated.

    Caller

    Every probed read is EventStore.read at apps/server/src/orchestration-v2/EventStore.ts:88, which calls readAgentEvents. The read input is {afterSequence, throughSequence, limit: 256} and, once, eventType. No thread id and no stream id.

    Build 7184184, probe in the disposable worktree. Run 2 is the OOM run (/home/luke/scratch/256-repro/run2.log). Run 1 (run1.log, killed at ~90s before OOM) has the same two stacks.

    Caller above EventStore.ts:88 Run 2 calls Run 2 rows Run 1 calls Run 1 rows
    loop EventSink.ts:763 ← anonymous EventSink.ts:781 3,328 851,712 3,330 852,480
    loop EventSink.ts:763 ← catchUp :785 ← replay :794 ← anonymous stream :829 10 257 10 257

    The anonymous frame at line 781 is the page continuation inside catchUp (Stream.concat(current, loop(last))). Those 3,328 calls are not a second caller. They are the later pages of one stream() started at line 829. One of the 10 starts asked for afterSequence: 0. Its first page ended at sequence 79,208, which is the 256th agent row, and the continuation walked that cursor to afterSequence 930,922 when the process aborted. throughSequence stayed 1,086,844. One in-flight read had no row line yet. Total rows decoded: 851,969.

    That single scan is 3,329 of 3,338 reads and 851,968 of 851,969 rows. Nothing else is close.

    The ten stream() starts:

    1. Orchestrator.ts:11074, eventType: "run.updated", cursor at the tip, 0 rows. Terminal-run subscription.
    2. Orchestrator.ts:11090, cursor at the tip, 0 rows. Request-wake subscription.
    3. providerAutoUpdate.ts:222, no cursor. This is the scan from sequence 0.
    4. Seven tip subscriptions, 1 row between them. They match RecoveryReactor.ts:98 (afterSequence captured from latestSequence) plus the six boot subscribers of streamDomainEvents: WightService.ts:86, AgentAwarenessRelay.ts:827, PullRequestSyncReactor.ts:428, ThreadPullRequestService.ts:393, ThreadSettlementService.ts:626, storageCleanup.ts:517.

    Orchestrator.streamStoredEvents (Orchestrator.ts:11315) is also eventSink.stream() with no cursor, but nothing in production pulls it. Only one from-0 start ran.

    recoverDelegatedTasks (Orchestrator.ts around 11108) is not on any read stack. The settled thread still gets a continuation dispatch at boot. That dispatch does not read this event range.

    Mechanism

    Observation. ProviderAutoUpdate.layer is in the desktop server layer. At boot it forks eventSink.stream() with no argument, then signals its wake queue once. stream() defaults afterSequence to 0, takes a pubsub subscription, then catchUp pages to the high-water mark with limit: 256. readAgentEvents keeps agent rows only (onlyAgentEvents). On the pristine snapshot that is 1,007,890 rows, sequences 78,953 through 1,086,844, and 2,687,752,625 bytes of payload_json. Each row is schema-decoded in rowToV2StoredEvent before the wake-type filter. The filter offers void into a sliding queue of 1.

    Inference. Historical wake events cannot change the updater beyond that one coalesced signal, and startup already signals. The replay does not checkpoint. The process aborts, the supervisor starts it again, and stream() begins at 0 again. That is a boot subscription, not a failed delivery retry.

    The live heap could not be freed. With --max-old-space-size=4096, mark-compact went 4089 MB → 4082 MB and then failed. A 256-row page cannot be that large: the biggest agent payload is 1.1 MB, and six payloads are over 1 MB. Decoded events from this scan were still reachable when the heap hit the cap. This pass did not name the retaining object.

    OOM

    Verified, clientless. No browser and no provider CLI. Restored chromeria-v2.sqlite from chromeria-v2.sqlite.pristine and removed the wal/shm sidecars. One server.

    • PID 1221656, exit 134 (abort)
    • Peak RSS 4,465,828 kB (4,361 MiB), about 84 s after start. Aborted about 102 s after start.
    • Log: /home/luke/scratch/256-repro/run2.log line 7074: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
    • RSS samples: /home/luke/scratch/256-repro/run2.rss
    • Run 2 stacks match run 1.

    Microfix

    In providerAutoUpdate.ts, capture eventSink.latestSequence() and pass stream({ afterSequence }). Leave the existing startup signal in place. Same shape as the comment on streamDomainEvents (Orchestrator.ts:11332): a cursorless stream() grinds the whole log, and this subscriber only needs new activity.

    What would falsify it: the same clientless boot after that change still climbs past about 1.5 GB. Then some other retainer remains. streamStoredEvents is still a cursorless stream if a future caller pulls it. That wants the bounded-read design. It is not what this boot ran.

    Elon record

    • Requirements and who asked: Name the caller of the boot event reads and say whether a clientless boot OOMs (diagnosis brief, this issue). Stop the rocky crash loop (issue).
    • Deleted: recoverDelegatedTasks as the reader. The archived planner thread as a required read. A bounded-read design as a precondition for stopping this replay.
    • Bottleneck: ProviderAutoUpdate's cursorless eventSink.stream(), which decodes the whole agent log on every boot.
    • Checked myself: run1.log stacks, pristine row counts, and clientless PID 1221656 (exit 134, peak RSS 4,361 MiB).

  10. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Design: bound whole-thread reads

    What: Every event-log subscription must name its cursor, and the catch-up replay inside EventSinkV2.stream reads its pages as a stream instead of holding all of them. Together these mean no boot, subscriber or thread read can keep the whole log in memory.
    Why: stream() treats a missing cursor as sequence 0. Its catchUp (EventSink.ts:753) then chains each 256-row page onto the next with a recursive Stream.concat, and a probe shows that chain keeps every decoded event reachable until the replay ends. One missing argument was enough to cause the rocky OOM.
    So what: Sol and Grok review this next. After revision it becomes its own PR, after the live-tail microfix. It is not on the deploy critical path: once the microfix lands, no production caller reaches this path today.

    Callers first

    // Live subscriber: unchanged in shape; the cursor is now required by the type.
const after = yield* eventSink.latestSequence();
yield* eventSink.stream({ afterSequence: after }).pipe(Stream.runForEach(onEvent));

// One whole thread, e.g. the 26,096-event planner fork. SQL filters by stream_id,
// so other threads are never decoded, and the replay holds about one 500-row page at a time.
yield* orchestrator.streamStoredEventsFrom({ threadId, afterSequence: 0 }).pipe(Stream.runForEach(onEvent));

// Client paging a thread: unchanged. ws.ts subscribeOrchestrationV2Thread serves a projection
// snapshot and tails from snapshotSequence. Resume replays only after the 128-event / 1 MiB preflight.

// Awkward case: no cursor.
eventSink.stream();                         // compile error: input and afterSequence are required
orchestrator.streamStoredEvents;            // compile error: removed
orchestrator.streamStoredEventsFrom();      // compile error: input required
eventSink.stream({ afterSequence: 0 });     // allowed, and visible in review: a deliberate whole-log replay.
                                            // Memory stays at about one page; the cost is decode time.

    Chosen shape

    At one owner, EventSinkV2.stream, the method every subscriber already goes through:

    1. No default cursor. Make afterSequence required everywhere a "from genesis" default exists today:

      // EventSinkV2Shape
readonly stream: (input: {
  readonly afterSequence: number;
  readonly threadId?: ThreadId;
  readonly eventType?: OrchestrationV2DomainEvent["type"];
  readonly bounded?: boolean;
}) => Stream.Stream<OrchestrationV2StoredEvent, EventSinkV2Error>;

// OrchestratorV2Shape: streamStoredEvents is deleted
readonly streamStoredEventsFrom: (input: {
  readonly afterSequence: number;
  readonly threadId?: ThreadId;
  readonly eventType?: OrchestrationV2DomainEvent["type"];
}) => Stream.Stream<OrchestrationV2StoredEvent, OrchestratorV2Error>;

// OrchestrationEventStore: the same `?? 0` default, removed
readonly streamApplicationEvents: (input: { readonly afterSequence: number }) => ...;
readonly streamProjectedApplicationEvents: <A>(input: { readonly afterSequence: number; readonly project: ... }) => ...;

      This makes the unbounded read impossible to write by accident, and it costs no runtime policy.

    2. Stream the catch-up instead of retaining it. Delete catchUp and replay through the read that already paginates without holding pages (readAgentEvents, Stream.paginate, 500 rows):

      const replay = (throughSequence: number) =>
  eventStore
    .read({ afterSequence, throughSequence, threadId, eventType })   // no limit: paginate to throughSequence
    .pipe(Stream.filter(matches));

      The rest of stream stays the same: subscribe first, then read the high-water mark, then replay through it, then follow live events above max(highWater, afterSequence), with the bounded path charging one item at a time. This applies to V2 the principle upstream #10777 applied to the V1 readers it removed. Upstream main still has the same recursive catchUp (per the survey comment), so this part can go upstream.

    Evidence for the retention claim. This was a disposable probe outside the repo (/tmp/256-design-probe/probe2.mjs), not a run of the server. It used effect@4.0.1 with integrity matching pnpm-lock.yaml, on Node 26.8.2. It copied catchUp's structure: an inner Stream.paginate of 500 rows, an Effect.fn decode for each row, runCollect of each 256-event page, and recursive Stream.concat. Over 51,200 synthetic events (about 200 MiB of payload), heap after forced GC grew steadily: 55, 109, 164 and 218 MiB above baseline at pages 50, 100, 150 and 200. With the direct paginated read it stayed at 0 to 2 MiB throughout. That matches the diagnosis finding that decoded events from the scan were still reachable at the heap cap.

    Alternatives rejected

    • A runtime replay cap, such as failing with a specific error past N events or bytes, like the client's 128 / 1 MiB. Once the replay streams, memory is about one page whatever the range, so a cap would limit time, not memory. It would also need a threshold nobody can defend. ProjectionMaintenance legitimately reads the whole log. And internal workers must not lose their subscription under load (EventSink.ts:185). Client paths already have their own preflight.
    • Fixing only catchUp and keeping the default. Memory would be safe, but a cursorless subscriber would still decode about 1e6 events (2.5 GiB of JSON.parse) on every boot, and the bug would stay one omitted argument away.
    • Requiring the cursor only, and keeping the recursive catchUp. This stops accidental reads from genesis, but any deliberate replay still holds every decoded event until it ends. That includes the planner fork from 0, and a global subscriber resuming from an old cursor, which would still OOM. Replacing catchUp is a net deletion, so keeping it saves nothing.
    • An afterSequence: number | "latest" variant that would let the owner take the live-tail cursor. Only two callers want "live only" (streamDomainEvents and the microfixed ProviderAutoUpdate). The others (Orchestrator terminal and request-wake, RecoveryReactor, the thread and task waits) take their cursor earlier on purpose, to cover a window. This would be a second way to spell the same tail. Revisit if more live-only callers appear.
    • A new whole-thread read API, a cache, or a decode rewrite. No consumer needs a separate thread-read API: clients read projections, and internal thread reads already pass threadId. With the replay streamed, one large thread costs about one page of decoded events at a time, which is less than its payload. So this issue does not justify the double decode in rowToV2StoredEvent (#18856) as a follow-on.
    • Porting upstream. Nothing upstream has merged for this (see the survey comment).

    Consumers

    Must change (all are compile-driven):

    • EventSink.ts: the stream signature, and catchUp deleted.
    • Orchestrator.ts: streamStoredEvents deleted from the shape (:375), the implementation (:11315) and the stub (:11462). streamStoredEventsFrom input becomes required.
    • ThreadManagementService.ts:385,951: the streamStoredEvents passthrough removed. At 7184184 it has no production puller; only tests use it.
    • OrchestrationEventStore.ts: afterSequence required on streamApplicationEvents and streamProjectedApplicationEvents. The only production caller (ws.ts:1054) already passes it.
    • Tests that read from genesis on small databases switch to an explicit afterSequence: 0 on the unbounded EventSinkV2.stream, keeping today's unbounded semantics. They are testkit/OrchestratorScenario.ts:274, ProviderSwitch.integration.test.ts (11 sites, plus :2158), ThreadLaunchService.test.ts:2384,2550, FoundationPersistence.test.ts:624, runtimeLayer.test.ts:4693, and the AgentAwarenessRelay.test.ts:219 stub.

    Must stay equivalent: the same events, the same ascending order, no gap between subscribing and the high-water mark, and the same error types. This covers the thread subscription (ws.ts:785, :808), the archived-shell tail (ws.ts:1795), the shell stream, the MCP task wait (OrchestratorMcpService.ts:1404), the thread run wait (ThreadManagementService.ts:718), the Orchestrator terminal and request-wake subscriptions (:11074, :11090), RecoveryReactor.ts:98, the streamDomainEvents subscribers, ProjectionMaintenance (it uses EventStoreV2.read, untouched) and Spectrum.

    One change is observable: today a decode error inside a page means none of that page is emitted. With the streamed replay, the rows before the bad one are emitted, then the stream fails with the same EventSinkStreamError.

    Found while mapping, outside the bound: spectrum/LaunchService.ts:223 starts its durable cursor at 0, and Controller.ts:259 reads 256 global events per step only to advance that cursor (batch is used for nothing else). It is memory-bounded, but each new Spectrum decodes old, unrelated events. The likely fix is to seed the cursor from latestSequence(). I have not measured its cost; it should be a separate issue.

    What the in-flight ProviderAutoUpdate tail covers

    The microfix (latestSequence() then stream({ afterSequence })) satisfies the new required cursor and compiles unchanged. It removes the one boot subscriber that read from genesis.

    It does not cover:

    • The default itself: the next cursorless stream() brings the OOM back.
    • streamStoredEvents: still cursorless, and exported through ThreadManagementService, so a new caller or MCP tool would read the whole log.
    • The retention: any legitimate long replay through stream() still holds every decoded event until it ends. That means a whole-thread read of the planner fork, or a global subscriber with an old cursor.

    The design holds after the microfix because it changes only how the owner treats a missing cursor and how it replays. It does not touch any caller's tail.

    Proof that would falsify this

    1. Equivalence (focused test). Interleave two threads across more than three 500-row pages. stream({ threadId, afterSequence: 0 }), both bounded and unbounded, emits exactly the old sequence list, then live events above the high-water mark, with no duplicates or gaps. A failure means the replay changed semantics.
    2. One large thread (disposable repro, pristine rocky snapshot copy, clientless). Drain streamStoredEventsFrom({ threadId: <planner fork>, afterSequence: 0 }) (26,096 events) while sampling heapUsed after GC. If heap grows with the number of events drained, instead of staying within a few MiB of one page, the retention is somewhere else (for example tracer spans) and this design is wrong.
    3. The original OOM shape. Explicitly run stream({ afterSequence: 0 }) with no threadId over the 1,007,890-event log. It must finish with flat RSS, well under the microfix's 1.5 GB bar. If it still climbs, the probe's mechanism is not the one that fails in production.
    4. Type check. stream(), stream({}) and streamStoredEventsFrom() fail to compile.

    Load-bearing assumption: the retention seen on rocky is the catchUp chain and not another retainer. Proofs 2 and 3 test exactly that.

    Elon record

    • Requirements and who asked: Planner brief on this issue: no boot, subscriber or thread read may hold the whole log in memory. One large thread (the 26,096-event planner fork) stays readable without retaining the other ~1e6 events. An unbounded read must be impossible to write or a specific error, never a silent cap. Project AGENTS.md adds: performance, the event-sourced server, no version bump, upstream first.
    • Deleted: The runtime replay cap. The "latest" cursor variant. A new thread-read API, a cache, and any decode or double-decode rewrite. Orchestrator.streamStoredEvents and its passthrough. EventSink.catchUp (replaced by the existing paginated read).
    • Bottleneck: catchUp's recursive Stream.concat, which retains decoded pages and is reached through the ?? 0 default. After the microfix, no production caller reaches it, so this closes a class of bug that is now latent, not the live crash.
    • Checked myself: Every production caller of EventSinkV2.stream, streamStoredEvents(From), EventStoreV2.read, readAgentEvents and streamApplicationEvents at 7184184. That streamStoredEvents has no production puller. The retention probe above, which is synthetic data, not the rocky snapshot.

    Design: claude-opus-5-5, Claude Code harness in T3 Code. No repository edits.

  11. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Findings: microfix is in PR #258, HEAD ae1a67e; all 15 focused tests pass.
    Why: ProviderAutoUpdate now subscribes after latestSequence while retaining its startup signal and later event wakes.
    So what: The pristine-copy clientless boot stayed alive for 180 seconds, PID 1255930, peak RSS 521,116 KiB (508.90 MiB), flat after warmup; that PID was stopped and its listener verified gone. Independent review and merge remain later steps.

    Upstream search: the matching component path is absent on pingdotgg/t3code main (404); open PR searches for provider auto update and providerAutoUpdate, and open Issue searches for event replay OOM and providerAutoUpdate, returned no matches. Proof and the Elon record are in the PR.

  12. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Roast: Sol

    What: Accept with required changes: deleting recursive catch-up is justified, but the design overstates the memory guarantee and its proof does not cover boot decode work.
    Why: The decoded-page retention is real Effect 4.0.1 behavior; a required cursor prevents omission, but explicit zero and the unchanged unbounded live queue remain separate concerns.
    So what: Revise the guarantee, proof, and caller inventory before implementing this design. The ProviderAutoUpdate live-tail microfix remains independent and should proceed.

    Reviewed the complete design against clean main at 7184184a5919f1f86296febca690e4b9fa389059. Lenses: behavior, lifetime, design, delivery. The goal is to read one large thread without materializing history from unrelated threads, and to prevent accidental historical replay at boot. Recommend the existing paginated reader plus an explicit cursor at the subscription owner, propagated through its facade; no new reader, cache, or arbitrary replay cap.

    Required changes

    1. The promise exceeds the mechanism. At EventSink.ts:223, live events use PubSub.unbounded. At :823-829, an unbounded subscriber takes its subscription before replay, but does not consume live events until replay finishes. Trigger: a long replay or slow consumer while other threads publish. Effect: live events accumulate without a bound; with only threadId filtering, unrelated-thread events also wait in that subscription until the later filter runs. Replacing catchUp fixes historical-page retention, not this queue. Rewrite “no boot, subscriber or thread read can keep the whole log in memory” as the specific guarantee actually established: consumed historical pages are released, and SQL filters a thread before decoding. State the retained live-tail limitation. If the intended acceptance instead demands bounded total subscriber memory under arbitrary concurrent writes, the current shape does not meet it and needs an owner-level live policy. Do not silently drop internal worker events to make that claim true. Add a replay/live concurrency case to proof; a clientless, quiescent synthetic scan cannot establish total subscriber memory bounds.

    2. Flat heap is not a boot-work bound. At EventSink.ts:789, the proposed explicit afterSequence: 0 still requests all matching historical events. Removing recursion makes that historical scan page-bounded, but does not remove its SQL, JSON/schema decoding, or per-event subscriber effects. Trigger: migrating a boot consumer by adding zero rather than its correct earlier/high-water cursor. Effect: roughly one million decodes and delayed useful live handling remain, even if the heap plateaus. A genuinely omitted TypeScript call cannot do this after the required signature is implemented; explicit zero still can. Keep the signature guard, but make proof also count historical decodes during normal boot after the microfix, and record time to replay completion/first useful live handling for the intentional large replay. No synthetic elapsed time here establishes rocky's post-change boot time. Also, proofs 2 and 3 cannot simply “finish” streamStoredEventsFrom/stream: they tail indefinitely. End their historical measurement at a known high-water milestone or known count, cancel the subscription, and test the live continuation separately. Do not use runCollect over the whole range as the memory oracle.

    3. Fix the migration inventory and reachability claim. Three cursorless callers were omitted: ThreadLaunchService.test.ts:1342, :1602, and :2262 call streamStoredEventsFrom({ threadId }). They need an explicit intentional cursor too. ProviderSwitch.integration.test.ts has 10 streamStoredEvents consumers, plus the separate cursorless sink call at :2158, rather than 11 property consumers. “After the microfix, no production caller reaches [catchUp]” is false: all sink subscriptions still traverse replay, normally over an empty or small range. The supported statement is that no identified production subscriber defaults to a genesis replay after the microfix. The real reachable old-cursor consumers are the thread/task waits and WebSocket tails, so these remain relevant equivalence cases.

    Claims cleared against code and dependency behavior

    • Retention: confirmed, not an external-array artifact. catchUp reads at most 256 decoded events, collects that page, creates Stream.fromIterable(events), then recursively concatenates the continuation. The probe's only external collection is numeric/string samples, not events. I read and reran the existing probe without modifying it. On Node v26.8.2, effect@4.0.1, 51,200 synthetic events with 4 KiB text, forced-GC heap growth was 55/109/164/218 MiB at 50/100/150/200 pages, versus 2/2/2/0 MiB for direct pagination; after completion it fell to 3/0 MiB. Commands: node --expose-gc /tmp/256-design-probe/probe2.mjs recursive and the same with direct. Probe SHA-256: be62b1530b7bd462a85376ed6b81611a32b8e46f0180a01c18a636da226b82b5. Its package integrity matches pnpm-lock.yaml; the repository's Effect patch changes MCP/HTTP/RPC code, not these stream primitives. Effect 4.0.1 Stream source implements concat through an iterator over the operand streams; the recursive unwrap boundaries retain the nested replay structure while it is active. Holding arrays inside that structure is the production mechanism, not a reason to dismiss the probe. This corroborates a retainer; it does not independently identify every rocky heap root.
    • 256 versus 500: both are correct at different layers. OrchestrationEventStore.ts:182 sets READ_PAGE_SIZE = 500. readAgentEvents at :497-552 queries min(remaining, 500). Today's catchUp passes total limit: 256, making each inner read one 256-row page. The proposed read without a total limit queries successive 500-row raw pages, decodes/emits each row, and carries only cursor/remaining state forward. Its working set depends on page bytes and decoding overhead, not merely 500 decoded events; a few MiB or flat RSS is empirical, not guaranteed by row count alone.
    • Thread isolation: confirmed. EventStoreV2.read forwards threadId and eventType; readApplicationRows at :384 applies stream_id = threadId in SQL before rowToV2StoredEvent. One thread's historical read does not decode the other approximately one million events. streamStoredEventsFrom sets bounded: true, but its budget charges/releases replay items incrementally, so the 1,000-item live limit is not a total historical-event cap. The 26,096-event thread is not rejected just for its count.
    • No missed production puller of streamStoredEvents. It is defined in Orchestrator, exposed through ThreadManagementService, and consumed by testkit/tests. ws.ts:785,1795, ThreadManagementService.ts:718,723, and OrchestratorMcpService.ts:1404 use streamStoredEventsFrom with cursors. Removing the unused cursorless property is justified.
    • Projection rebuild: preserved. ProjectionMaintenance.ts:145 explicitly loops over eventStore.read({ afterSequence: lastSequence, throughSequence, limit: 500 }), collecting and applying one page at a time. It does not use catchUp or either removed/changed facade. The design does not break it or leave it with this recursive decoded-history retainer. It remains a deliberate whole-log operation with proportional work; this inspection is not a general proof against every possible rebuild OOM.
    • Cursor omissions today: production providerAutoUpdate.ts:222; the unused property construction Orchestrator.ts:11315; forwarding facade :11324 accepts optional input; tests FoundationPersistence.test.ts:624, ProviderSwitch.integration.test.ts:2158, the three ThreadLaunchService calls above, and runtimeLayer.test.ts:4693 (streamApplicationEvents()). The property consumers are the 10 ProviderSwitch sites, OrchestratorScenario.ts:274, and ThreadLaunchService.test.ts:2384,2550; AgentAwarenessRelay.test.ts:219 is a stub property, not a replay. No production application-stream caller omits its cursor: ws.ts:1054 supplies it.

    Optional design/delivery notes

    The cursor requirement touches upstream-shared interfaces and several tests, so it is more rebase surface than the fork-local microfix. That does not by itself violate the Mission: the safety rule belongs at the existing subscriber owner, and the replay deletion is suitable for upstream. Deleting only the public cursorless property leaves the original EventSink.stream() boot failure possible. Fixing only recursion leaves accidental full-decode work possible. Defaulting a missing cursor to latest is a smaller signature change, but silently changes an existing historical-read call into a live-only call; compile errors expose that decision more reliably. I favor the explicit cursor at EventSink and its facade. Requiring cursors on both application-stream methods is adjacent hardening, not necessary to remove this recursive V2 retainer; keep it only with that distinct rationale. Leave EventStoreV2.read available for finite intentional rebuilds.

    The design correctly discloses the changed partial-page error behavior. Preserve that in a focused malformed-row case alongside ordering, replay/live handoff, cancellation, and error wrapping. Keep raw stored-event test consumers on the unbounded sink rather than inadvertently migrating them to the wire-projecting facade.

    No repository, design comment, canonical handoff, microfix worktree, or ~/scratch/256-repro edits. No server, browser, live database, or rocky snapshot run. The probe is synthetic; proposed implementation and production timing remain unverified. No Spectrum issue filed.

    Elon record

    Requirements and who asked: User's read-only roast of #256: large-thread isolation, replay lifetime, boot work, actual callers, and minimal upstream-shared surface; leave the live-tail microfix independent. Reliability supports the Objective's Promachos/delegated work.
    Deleted: A runtime cap or new thread-read API without a demonstrated need; treating explicit cursors as a runtime work bound; assuming the page probe proves total subscriber memory or rocky timing.
    Bottleneck: Establish the exact historical-retention guarantee and faithful acceptance proof before implementing the separate bound; the crash-loop microfix does not wait on this design.
    Checked myself: Pinned source, reachable consumers, SQL filtering and paging, projection rebuild, Effect package integrity and patch scope, original probe source and both reruns, clean unchanged main checkout.

    Review: gpt-6.1-sol, high reasoning effort, Codex harness in T3 Code.

  13. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Roast: Grok

    What: Accept with required changes. Replacing catchUp with the existing paginated read is the right fix, and a required cursor belongs on EventSinkV2.stream and the facades that forward it.
    Why: Recursive Stream.concat really does keep every decoded page alive until replay ends. The writeup then promises more than that mechanism delivers, and its proof and caller list are wrong in specific places.
    So what: Revise the guarantee, the two drain proofs, and the caller inventory before implementation. Leave the ProviderAutoUpdate live-tail microfix alone.

    Reviewed the design comment against clean main at 7184184a5919f1f86296febca690e4b9fa389059. Lenses: behavior, lifetime, design, delivery. No repo, design-comment, handoff, microfix-worktree, or ~/scratch/256-repro edits.

    Required changes

    1. The memory sentence is larger than the code. EventSink.ts:223 builds liveEvents with PubSub.unbounded. On the unbounded path (:823-829) the subscription is taken, then replay runs, and only then does Stream.concat pull the live side. Effect 4.0.1 documents that an unbounded pubsub keeps every value published while a subscriber is active until that subscriber takes it (PubSub.ts makeAtomicUnbounded). Stream.filter(matches) runs after the take, so a thread filter does not keep other threads out of that queue. Trigger: an unbounded stream({ afterSequence: 0 }) or any other long replay, while commits continue. Effect: historical pages can be released and the shared pubsub can still retain every event published during the replay. The bounded facade (streamStoredEventsFrom, bounded: true) is different: replayAndBufferProjectedLiveEvents drains during replay into a 1,000-item / 8 MiB budget (LiveStreamBudget.ts:17-18). Rewrite the guarantee as: consumed historical pages are released, and a threadId read seeks that stream before rowToV2StoredEvent. State the unbounded live queue as a limit. Do not drop internal worker events to make the stronger sentence true (EventSink.ts:185 already forbids that). A quiescent synthetic scan cannot prove subscriber memory under concurrent publishes.

    2. Flat heap is not a bound on boot work, and proofs 2 and 3 do not finish. After catchUp is gone, stream({ afterSequence: 0 }) still asks readAgentEvents for every matching row. Trigger: a caller writes 0 instead of a high-water cursor. Effect: about 1,007,890 decodes and 2.5 GiB of payload_json still run. The caller note on this issue already recorded one such scan decoding 851,969 rows and aborting about 102 s after start, while the retainer was also holding those rows. I did not re-time a flat scan, so I am not claiming a post-fix duration. Omitting the argument becomes a compile error once afterSequence is required. Explicit 0 stays legal and still does the work. Proofs 2 and 3 tell you to drain stream / streamStoredEventsFrom until they finish. Both concat a live tail and do not finish. Stop the historical measurement at a known high-water count, cancel, and check the live continuation separately. Count decodes or time to that mark. Stream.runCollect over the open stream holds the events and hides the page release.

    3. The caller inventory and the reachability sentence are wrong. ThreadLaunchService.test.ts:1342, :1602, and :2262 call streamStoredEventsFrom({ threadId }) with no cursor. They are compile failures under a required afterSequence, and they are absent from the "must change" list. ProviderSwitch.integration.test.ts has 10 streamStoredEvents call sites, plus the separate cursorless eventSink.stream({ threadId }) at :2158, which is 10 property consumers rather than 11. "After the microfix, no production caller reaches catchUp" is false. Every stream() still replays from its cursor to the high-water mark. The accurate sentence is: after the microfix, no production caller starts that replay at genesis. Thread waits, the MCP task wait, and the WebSocket tails still enter replay over a short or empty range, and they are the equivalence cases that matter.

    Shape

    Keep the three decisions that match the requirement.

    • Delete catchUp. Replay through eventStore.read / readAgentEvents, which already pages.
    • Require afterSequence on EventSinkV2.stream. That is the owner of input?.afterSequence ?? 0 (EventSink.ts:789). Facades that forward the input (streamStoredEventsFrom on the orchestrator and on ThreadManagementService) have to require it too, or they reintroduce the default.
    • Delete streamStoredEvents. Nothing in production pulls it. ThreadManagementService.ts:385 and :951 only pass it through. Tests and the unavailable stub are the other references.

    A smaller owner does not meet the requirement:

    • Deleting only streamStoredEvents leaves eventSink.stream() legal. That is the call ProviderAutoUpdate makes today (providerAutoUpdate.ts:222). The microfix patches that one site. The default remains for the next one.
    • Fixing only the recursion leaves a missing cursor as a full decode. Memory stays near one page. The work is still the boot scan that abort took about 102 s to die inside.
    • Defaulting a missing cursor to latestSequence() is a smaller diff and a silent behavior change. FoundationPersistence.test.ts:624, ProviderSwitch.integration.test.ts:2158, and the three ThreadLaunchService calls above omit the cursor in order to see history on a small database. A compile error makes that choice visible. A latest default turns those reads into live tails.

    This signature edit is on upstream-shared files. It does not fight the fork mission. The mission puts fork features behind small hooks. This is a bug in the shared subscriber, and the owner of the ?? 0 default is the right place to close it. The catchUp deletion is an internal change upstream can take. Requiring the cursor on streamApplicationEvents and streamProjectedApplicationEvents is a second file and a second replay (catchUpApplicationEvents, OrchestrationEventStore.ts:651), which already uses Stream.paginate rather than recursive concat. The only production caller, ws.ts:1054, already passes afterSequence. Leave that signature alone in this change.

    ProjectionMaintenance.rebuild (ProjectionMaintenance.ts:145-173) loops eventStore.read with limit: 500, applies one page, and discards it. It does not call catchUp or either facade this design removes. The design does not break it. It remains a deliberate whole-log read with whole-log work. This review does not prove the rebuild can never OOM for some other reason.

    Claims cleared

    • Retention is real Effect 4.0.1 behavior. catchUp (EventSink.ts:753-785) collects one page with Stream.runCollect, wraps that array in Stream.fromIterable, and Stream.concats the next loop. Stream.concat stores both operands on the result (concatParts) and the iterator closure captures them for the life of that stream. Stream.unwrap holds the produced stream until it completes, so each page's concat stays reachable from the previous one until replay ends. The probe's consumer is Stream.runForEach of a discarded event, plus a few sample strings. I reran /tmp/256-design-probe/probe2.mjs on Node v26.8.2 and effect@4.0.1: recursive growth was 55 / 109 / 164 / 218 MiB at 50 / 100 / 150 / 200 pages, and 3 MiB after completion; the direct paginated read stayed at 2 / 2 / 2 / 0 MiB and 0 after. The pages are held by the concat chain, and they are released when the chain ends. patches/effect@4.0.1.patch touches MCP, HTTP, and RPC, not Stream or PubSub.
    • 256 and 500 are different layers. catchUp passes limit: 256. READ_PAGE_SIZE is 500 (OrchestrationEventStore.ts:182). With a total limit of 256, readAgentEvents issues one query of 256 rows (min(remaining, 500)). The proposed read omits limit, so each query is 500 raw rows and the paginate closure keeps the cursor, not prior pages. "One 500-row page" describes the replacement. It does not describe today's catchUp.
    • One thread does not decode the others. readApplicationRows puts stream_id = threadId in the SQL, and rowToV2StoredEvent runs on returned rows. A local EXPLAIN on this schema with 200,000 rows (not the rocky database) seeks idx_orch_events_stream_sequence on aggregate_kind and stream_id. Other streams are not visited. The 26,096-event thread stays a legal read. streamStoredEventsFrom is bounded, and the budget charges one replay item at a time, so 26,096 events are not rejected for count.
    • No missed production puller of streamStoredEvents. Production cursors that already exist: ws.ts:785 and :1795, ThreadManagementService.ts:718 and :723, OrchestratorMcpService.ts:1404, orchestrator terminal and request-wake at :11074 and :11090, RecoveryReactor.ts:98, and streamDomainEvents (Orchestrator.ts:11336), which already takes latestSequence() before stream. The cursorless production call is providerAutoUpdate.ts:222, which the microfix owns.
    • Explicit afterSequence: 0 after the deletion spends decode time and holds about one page of history. It does not, by itself, hold the historical log. It also does not make that scan cheap. See required change 2.

    Optional

    The design is right that a decode error mid-page changes behavior: runCollect fails the page before any row is emitted; a streamed mapEffect emits the earlier rows of that page and then fails with the same EventSinkStreamError. Keep that as a focused case next to order, the replay/live handoff, and cancellation. The Spectrum cursor note stays out of scope. I did not file an issue for it.

    Elon record

    • Requirements and who asked: This roast's brief: no boot, subscriber, or thread read may materialize the whole log; one thread read must not decode the other ~1e6 events; judge the required-cursor edit against the fork mission; leave the live-tail microfix independent. The rocky crash loop is the issue's.
    • Deleted: A runtime cap, a new thread API, and a pubsub redesign as part of accepting this shape. Treating a required cursor as a bound on explicit-zero work. Treating the synthetic probe as a rocky boot time.
    • Bottleneck: The recursive concat that retains decoded pages, reached from the ?? 0 default. The microfix removes the one production caller that uses that default. This design closes the retainer and the default.
    • Checked myself: EventSink.catchUp, readAgentEvents, both application-event replays, projection rebuild, the caller searches above, Effect 4.0.1 Stream.concat / Stream.paginate / PubSub.unbounded, the effect patch file list, a local query plan, and both probe variants rerun.

    Review: Grok, read-only, against 7184184a5919.

  14. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Design revision: bound whole-thread reads

    Supersedes the first design, which stays in place. It applies the binding adjudication of the Sol and Grok roasts.

    What: EventSinkV2.stream replays history through the existing paged read instead of the recursive catchUp. It also requires a cursor, and Orchestrator.streamStoredEvents is deleted.
    Why: catchUp's recursive Stream.concat keeps every decoded page alive until the replay ends: 218 MiB at 51,200 events in the probe, released only afterwards. The ?? 0 default meant one omitted argument started that replay at genesis.
    So what: This is the guarantee for historical replay only. The live queue of unbounded subscribers stays as it is and is stated as a limit below. Implementation is a separate PR after #258, the ProviderAutoUpdate live tail, which this design does not touch.

    Guarantee, narrowed

    What the change establishes:

    • Historical replay pages are released as they are consumed. A replay through stream holds about one raw 500-row page (READ_PAGE_SIZE) and the rows decoding from it, whatever the range length. Actual bytes depend on payload sizes, so a flat heap is measured, not derived from the row count.
    • A threadId read filters in SQL (stream_id = ? in readApplicationRows) before rowToV2StoredEvent. Reading one thread, including the 26,096-event planner fork, never decodes the other ~1e6 events.
    • A missing cursor is a compile error.

    What it does not bound:

    • Work. afterSequence: 0 stays legal and still decodes the whole matched range: about 1,007,890 rows and 2.5 GiB of payload_json with no threadId. The required cursor makes that choice visible in review. It does not make the scan cheap.
    • The live queue of an unbounded stream. liveEvents is PubSub.unbounded (EventSink.ts:223). The unbounded path subscribes, reads the high-water mark, replays, and only then pulls the live side (:823-829). Every event published during the replay waits in that subscription until it is taken. Stream.filter(matches) runs after the take, so a threadId filter does not keep other threads' events out of the queue. An eventType filter does, because it routes to a per-type pubsub. A long replay on an unbounded subscription while commits continue therefore retains whatever was published during the replay. This design does not drop internal worker events to hide that, because EventSink.ts:185 requires workers to keep their subscription. The bounded facade (streamStoredEventsFrom) already drains into a 1,000-item / 8 MiB budget during replay and fails with LiveStreamBufferError on overflow.

    Reachability: after #258, no identified production subscriber starts a replay at genesis. Every stream still replays from its cursor to the high-water mark. The thread and task waits, the MCP task wait and the WebSocket tails replay short or empty ranges, and they are the equivalence cases.

    Revised signatures

    // EventSinkV2Shape (EventSink.ts)
readonly stream: (input: {
  readonly afterSequence: number;          // required; 0 is a deliberate full replay of the matched range
  readonly threadId?: ThreadId;
  readonly eventType?: OrchestrationV2DomainEvent["type"];
  readonly bounded?: boolean;
}) => Stream.Stream<OrchestrationV2StoredEvent, EventSinkV2Error>;

// inside stream: catchUp is deleted
const replay = (throughSequence: number) =>
  eventStore
    .read({ afterSequence: input.afterSequence, throughSequence, threadId: input.threadId, eventType: input.eventType })
    .pipe(Stream.filter(matches));          // no total limit: readAgentEvents paginates 500 rows to throughSequence

// OrchestratorV2Shape (Orchestrator.ts): streamStoredEvents is deleted
readonly streamStoredEventsFrom: (input: {
  readonly afterSequence: number;
  readonly threadId?: ThreadId;
  readonly eventType?: OrchestrationV2DomainEvent["type"];
}) => Stream.Stream<OrchestrationV2StoredEvent, OrchestratorV2Error>;

// ThreadManagementService.ts:386 already reuses the Orchestrator type, so it follows.

    Everything else in stream stays the same: subscribe first, then the high-water mark, then replay through it, then live events above max(highWater, afterSequence). The bounded path still charges replay one item at a time. streamApplicationEvents and streamProjectedApplicationEvents are unchanged: that replay already uses Stream.paginate, and ws.ts:1054 passes a cursor.

    Alternatives rejected

    • Defaulting a missing cursor to latestSequence(). It would silently turn the history reads that tests make on small databases into live tails. The compile error forces the decision instead.
    • A runtime replay cap. It would limit time, not memory. Nobody can defend a threshold for it, and ProjectionMaintenance legitimately reads the whole log.
    • Dropping or capping internal worker live events, or an owner-level live policy for unbounded subscribers. Workers must keep their subscription. Nothing in this issue shows live-queue growth on rocky, because boot subscribers replay near-empty ranges. Reopen if proof 4 or production shows that growth.
    • Fixing only the recursion. A missing cursor would remain a full decode on boot.
    • Requiring the cursor only. Any deliberate long replay would still retain every decoded page until it ends.
    • Also requiring a cursor on the application-event streams. That replay is not affected.
    • A new thread-read API, a cache, or a decode rewrite. None is needed. With the replay streamed, one large thread does not cost more than its payload, so the double decode (#18856) is not a follow-on.
    • Porting upstream. Nothing upstream has merged for this. The catchUp deletion can be offered upstream.
    • Spectrum's cursor stays out of scope.

    Consumers

    Must change:

    • EventSink.ts: the stream signature, catchUp deleted, and replay through eventStore.read.
    • Orchestrator.ts: streamStoredEvents removed from the shape (:375), the implementation (:11315) and the unavailable stub (:11462). streamStoredEventsFrom input becomes required (:376).
    • ThreadManagementService.ts:385,951: the streamStoredEvents passthrough removed.
    • providerAutoUpdate.ts:222 is owned by fix(server): provider auto-update tails the event log #258, whose latestSequence() then stream({ afterSequence }) already conforms. Land this design after fix(server): provider auto-update tails the event log #258, or it will not compile.
    • Tests that read raw stored events from genesis keep the unbounded sink with an explicit afterSequence: 0. Moving to streamStoredEventsFrom is acceptable only where the wire-projected event is enough. The sites:
      • ProviderSwitch.integration.test.ts: 10 streamStoredEvents sites, plus the cursorless eventSink.stream({ threadId }) at :2158.
      • ThreadLaunchService.test.ts: streamStoredEvents at :2384 and :2550, and cursorless streamStoredEventsFrom({ threadId }) at :1342, :1602 and :2262.
      • testkit/OrchestratorScenario.ts:274 (streamStoredEvents).
      • FoundationPersistence.test.ts:624: cursorless sink.stream({ threadId, eventType }).
      • AgentAwarenessRelay.test.ts:219: the stub property is removed. It is not a replay.

    Must stay equivalent: the same events, the same ascending order, no gap between subscribing and the high-water mark, the same error types, and the same cancellation. This covers:

    • the thread subscription (ws.ts:785, :808) and the archived-shell tail (ws.ts:1795)
    • the MCP task wait (OrchestratorMcpService.ts:1404) and the thread run wait (ThreadManagementService.ts:718,723)
    • the Orchestrator terminal and request-wake subscriptions (:11074, :11090), RecoveryReactor.ts:98, and streamDomainEvents and its subscribers
    • ProjectionMaintenance.rebuild, which uses EventStoreV2.read with limit: 500 and is untouched
    • the shell stream

    Disclosed change: today a decode error inside a page means none of that page is emitted. With the streamed replay, the rows before the bad one are emitted, then the stream fails with the same EventSinkStreamError.

    Proof that would falsify this

    Every measurement over stream or streamStoredEventsFrom stops at a known historical count with Stream.take(N) and runForEach, then cancels. Both streams follow a live tail and never finish, and runCollect would hold the events and hide whether pages are released.

    1. Equivalence (focused tests). Two threads interleaved across more than three 500-row pages. On both the bounded and unbounded paths, take(N) of stream({ threadId, afterSequence: 0 }) returns exactly the expected sequences. A separate case commits one event after the high-water mark and checks the live continuation delivers it once, with no gap or duplicate. Also cover cancellation mid-replay, and a malformed row mid-page (earlier rows emitted, then EventSinkStreamError).
    2. One large thread (disposable copy of the pristine rocky snapshot, clientless). Run streamStoredEventsFrom({ threadId: <planner fork>, afterSequence: 0 }), take(26,096). Sample heapUsed after forced GC every 2,000 events. Record decoded rows and elapsed time to the 26,096th event. Falsifier: heap grows with the number of events consumed.
    3. The original shape, explicit. Run stream({ afterSequence: 0 }) with no threadId, take the agent-row count up to the high-water mark (1,007,890 on the pristine copy), with the same sampling. Record decoded rows and time to that mark. Falsifier: heap grows with events consumed. This run is quiescent, so it proves historical-page release only.
    4. Replay/live concurrency. Repeat a smaller version of 3 while committing events on other threads at a fixed rate. Expected: historical pages are still released, retained heap tracks the number of events published during the replay, and it drops once the live side is taken. This documents the stated limit. It does not assert a bound.
    5. Boot work after fix(server): provider auto-update tails the event log #258. On a normal clientless boot of the pristine copy with fix(server): provider auto-update tails the event log #258 and this design, count rowToV2StoredEvent decodes from start to ready. Expected: only the tip subscriptions' small ranges (the diagnosis saw 1 row across 7 of them). Falsifier: any subscriber decoding a historical range at boot.
    6. Type check. stream(), stream({ threadId }), streamStoredEventsFrom({ threadId }) and streamStoredEvents fail to compile.

    Load-bearing assumption: the retainer on rocky was the catchUp chain. Proofs 2 and 3 test it. The synthetic probe cannot settle rocky's boot time, and no timing is claimed here.

    Elon record

    • Requirements and who asked: Planner brief and binding adjudication on this issue. Historical replay through a boot, subscriber or thread read must not retain the log. One thread, the 26,096-event planner fork, stays readable without decoding the others. A missing cursor is a compile error and never defaults to latest. streamStoredEvents is deleted. fix(server): provider auto-update tails the event log #258 is left alone. The guarantee is narrowed to historical pages, with the live queue stated as a limit. Project AGENTS.md adds: performance, the event-sourced server, no version bump, upstream first.
    • Deleted: A runtime cap. The "latest" default and variant. Dropping or capping worker live events. A live-queue policy. A cursor on the application-event streams. A new thread-read API, a cache, and a decode rewrite. EventSink.catchUp, Orchestrator.streamStoredEvents and its passthrough. The first design's claim that "no boot, subscriber or thread read can keep the whole log in memory".
    • Bottleneck: The recursive catchUp concat, reached from the ?? 0 default. fix(server): provider auto-update tails the event log #258 removes the one production caller that used that default, so this design closes the retainer and the default for future callers, off the deploy critical path.
    • Checked myself: At 7184184, PubSub.unbounded at EventSink.ts:223. The ThreadManagementService.streamStoredEventsFrom type reuse (:386). Every cursorless streamStoredEventsFrom site: only ThreadLaunchService.test.ts:1342,1602,2262; all four OrchestratorMcpToolkit.integration.test.ts sites pass a cursor. The 10 ProviderSwitch streamStoredEvents sites and the :2158 sink call. The remaining streamStoredEvents references. Both roasts reran the probe with the same numbers.

    Design: claude-opus-5-5, Claude Code harness in T3 Code. No repository edits.

  15. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Planner, 09:03 CEST: verified live on rocky after the #258 deploy.

    • The running app is 614c13f11120 (t3codeCommitHash in app.asar), and codesign --verify --deep --strict passes.
    • Backend pid 65563 has stayed up since 08:55:39. RSS was flat at 415 to 456 MB across 24 samples from 08:56:43 to 09:02:30; before this deploy the backend climbed to about 4.5 GB and died every 60 to 300 s. No new crash report (still 46).
    • OrchestrationEventStore.rowToV2StoredEvent is absent from the current server.trace.ndjson.
    • One thing to watch, not a failure: the current trace has 3,724 ThreadSettlementServiceV2.settleThread spans, the most common span. If that persists past startup it is worth a look.

  16. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Planner, 09:15 CEST, follow-up on the settleThread count: checked, and it's not a problem. ThreadSettlementServiceV2.sweep runs once a minute by design (Schedule.spaced("1 minute") in apps/server/src/orchestration-v2/ThreadSettlementService.ts). Each sweep checks about 1,863 candidate threads, and each check takes about 4 µs (p50; p99 9 µs). Across 13 sweeps from 06:56Z to 07:08Z, a sweep took 129 to 248 ms, with three slower ones (1.25 s, 1.62 s, and 5.27 s on the first sweep after boot). That's well under 1% of wall time, and RSS stayed flat.

  17. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    Implementation PR: #267

    EventSink.stream now requires afterSequence and replays through the paged eventStore.read. catchUp and streamStoredEvents are removed. Not merged.

  18. lukemaj
    added a commit that references this issue on Oct 14, 2026

    fix(childThreads): skip ancestor loads for children with no requests …

    64e0241

  19. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: The authorized rocky rollout is staging merged main 8f939c24ea898e7600c9c93892cb3ad63ad88686, including #243, before cavallo.
    Why: ContextMD #39 needs the serving command matcher to stop denying prose mentions.
    So what: Watch owner task chromeria-256-orchestrator-r2-2026-10-14 should expect a new backend process; its PID and start time will follow after activation.

    Old rocky backend PID 64309, start 2026-10-14T11:09:08Z; still running at 12:11:48Z. Fresh pre-check found zero active runs, unfinished tools and background tasks. Signed staged app.asar SHA-256 is 47e8a2c75f2838342683e5523fb8cb9280dfe23d71152c74177675ca07aaef85, identical on both Macs. Read-only online database backups passed integrity checks. The restart uses the established restart-and-resume path with a task-only copy to apply #292's two explicitly authorized settings while stopped; installed tooling stays unchanged. Bestia is excluded.

    Requirements and who asked: User requested merged main on rocky then cavallo, local restart/resume, rollback apps, ten-minute process/health proof, #243 deny/mention proof, and runbook/Issue records; the planner also authorized #292's two settings during the stopped interval.
    Deleted: Bestia restart, another Mac build, version changes, new services, browser use and feature work.
    Bottleneck: Signed artifact promotion to rocky, then the required ten-minute observation.
    Checked myself: Exact main SHA, CI test/build/type/lint job results, clean owned build worktree, signatures and shared asar digest, backups/integrity, running-work checks, old backend identity, and #292's unapplied candidate/live settings.

  20. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky now runs 8f939c24ea89; backend PID 92991 replaced 64309.
    Why: The authorized #243 rollout required one restart; #292's two HTTPS fields were applied while stopped in that same restart.
    So what: Watch owner chromeria-256-orchestrator-r2-2026-10-14 should restart its process observation on PID 92991, start 2026-10-14T12:12:51Z.

    Old backend 64309 started 2026-10-14T11:09:08Z. Graceful stop began 12:12:47Z, setting application and swap at 12:12:50Z, health and exactly-once callback delivery at 12:13:18Z. Main PID 92964; local health/discovery 200; remote HTTPS 8446 discovery 200 and anonymous /api/orchestration/shell 401. Pre-check had zero cut running turns/tools/background work; post-check has no missing continuation or held queue. Rollback app is bcc4db86b85a at Chromeria.app.prev. The rollout's separate ten-minute observation is running; longer #256 acceptance remains the watch owner's responsibility.

  21. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky's ten-minute proof passed on 8f939c24ea89, backend PID 92991, and its runbook record is pushed.
    Why: This rollout replaced watched backend 64309 and needed fresh process/health evidence.
    So what: #256's owner continues its longer watch from 92991, start 2026-10-14T12:12:51Z; cavallo is activating after rocky passed.

    21 read-only samples from 2026-10-14T12:13:52.123732Z to 12:23:54.568222Z, 602.43 seconds: PID/start identity unchanged, every health and discovery response 200, every embedded hash 8f939c24ea89. RSS 420,032 to 592,400 KiB. The observation proves that interval, not indefinite stability or the rest of #256's acceptance. No missing continuations or held queues; watchdog probation/failure/pending-resume/restart-lock clear.

    Pushed runbook commit 8958552, owned new path rocky/CHROMERIA-2026-10-14-8f939c24.md. Pulled with rebase immediately before commit, staged only that path and pushed normally. The record includes artifact/backup digests, process handoff, harmless installed-selector/deny proof and #292 activation/rollback/endpoints. Task-only helpers retired; installed tooling unchanged. Bestia was not restarted.

  22. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky's Chromeria was restarted onto fork main 9b495751dc20 (PR #290); the watch must follow the new backend process.
    Why: Rollout for #164 replaced the 8f939c24ea89 process the rocky watch was observing.
    So what: The #256 watch owner moves the rocky watch to backend PID 6379.

    • Old: backend PID 92991, started 2026-10-14T12:12:51Z; main PID 92964; build 8f939c24ea89.
    • New: backend PID 6379, started 2026-10-14T12:59:42Z (14:59:42 CEST); main PID 6353; build 9b495751dc20.
    • Timeline: restart-and-resume scheduled 12:59:08Z, quitting 12:59:38Z, swapped 12:59:41Z, healthy 13:00:07Z, resume delivered once 13:00:08Z, build-healthy 13:00:15Z.
    • Checks: pre-check found zero active runs, unfinished tools and background tasks; post-check found no cut turn without continuation and no held queue. /Applications/Chromeria.app.prev holds 8f939c24ea89.
    • Backups: online backups with integrity_check ok are in ~/.local/state/chromeria-backup/2026-10-14-pre-9b495751dc20/.
    • Reachability: https://rocky.napoleon-sun.ts.net/.well-known/t3/environment returns 200 from bestia.

  23. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky's backend pid 6379 on 9b495751 has stayed up for 4 hours with no crash, and RSS has sat around 893 to 911 MB for the last two hours, so #256 is closed.
    Why: The process that used to reach about 4 GB was replaying the event log from the start. #258 and #267 are on this build, and the higher RSS matches a 7.6 GB database with 2,492 threads.
    So what: The hourly watch is stopped. No open sub-issue was left to close. #239, #248, and #263 stay open.

    Pid 6379 started 2026-10-14T12:59:42Z (main pid 6353, build 9b495751dc20). Close sample at about 17:00Z: etime 04:00:35, RSS 917856 KiB. Same pid the whole window. No staged app on the hourly samples.

    RSS KiB after the swap: 767376 at 58 minutes; ten-minute samples 14:58Z to 15:58Z were 893392, 806912, 910784, 923248, 774096, 912016, 923568; then 913136, 920560, 933280, and 917856. The middle hour dipped to about 756 MB. From about 15:02Z the samples stayed between 893 MB and 911 MB. The planner accepted that plateau.

    Read-only check at close: chromeria-v2.sqlite is 7632191488 bytes and orchestration_v2_projection_threads has 2492 rows. Cavallo on the same build: 797 threads, backend RSS 354208 KiB (pid 24290, etime 03:57:41), database 2714898432 bytes.

    No GitHub sub-issues. #259 was already closed. #257, #258, and #267 are merged. Left open, with no assignee on any of them: #239 owns serving latency, #248 owns the Cue deadline, and #263 owns provider self-updaters that do not update the binary Chromeria runs. The two original measurements (database occupancy over a busy hour, and over_budget under 2% for a day) were not repeated. They were not the close gate.

    Requirements and who asked: Planner, 2026-10-14: close #256 once pid 6379 has run for a few hours with flat memory and no crash, and stop the watch.
    Deleted: another sample hour, a heap snapshot, a leak issue, and a restart of rocky.
    Bottleneck: waiting for the same pid to stay up.
    Checked myself: the close ps sample, the sqlite counts, and cavallo's process and database size.

  24. lukemaj
    closed this as completedon Oct 14, 2026

  25. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: The authorized rocky rollout is activating d74c023f7816 through chromeria-restart-and-resume.
    Why: This replaces watched build 9b495751dc20 with merged #279, #288, #303 and #304.
    So what: The #256 watch owner should expect a new backend PID; the process handoff and ten-minute result will follow.

    Old backend PID 6379; current live identity and fresh empty running-work pre-check retained in the rollout evidence. Signed staged app.asar SHA-256 5ed53ccf9748c5391ef223c6062ace3152b3bfc03f36a82b9b2cfadb1dcc4b07, identical on both Macs. Read-only VACUUM INTO backups passed integrity checks. Rollback app will retain 9b495751dc20.

  26. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky now runs d74c023f7816; backend PID 60366 replaced 6379.
    Why: The authorized rollout activated merged #279, #288, #303 and #304 through the established restart-and-resume path.
    So what: The #256 watch owner should follow backend PID 60366, started 2026-10-14 19:07:37 CEST; this task is collecting its ten-minute interval.

    Old backend 6379 started 2026-10-14 14:59:42 CEST. Quit at 17:07:34Z, swap 17:07:36Z, healthy and resume delivered 17:08:04Z. Fresh pre-check had no active runs, unfinished tools or background tasks. Health 200; t3 environment list lists bestia and cavallo reachable with full access. Rollback app retains 9b495751dc20; read-only VACUUM INTO backups passed integrity checks. Signed installed app.asar SHA-256 5ed53ccf9748c5391ef223c6062ace3152b3bfc03f36a82b9b2cfadb1dcc4b07.

  27. lukemaj commented on Oct 14, 2026

    @lukemaj
    lukemaj
    on Oct 14, 2026
    ContributorAuthor

    What: Rocky's ten-minute rollout proof passed on d74c023f7816, backend PID 60366.
    Why: The new rollout replaced watched PID 6379 and required a fresh observation.
    So what: The #256 watch owner can continue from 60366, started 2026-10-14 19:07:37 CEST; the per-machine runbook record is pushed.

    21 samples from 2026-10-14T17:08:21.889994+00:00 to 2026-10-14T17:18:23.764558+00:00, 601.94 seconds: same backend PID/start identity 60366 Wed Oct 14 19:07:37 2026, health and discovery 200, installed hash d74c023f7816 in every sample. This proves that interval, not indefinite stability.

    Peer links to both other machines remain reachable. Post-check has no cut turn missing continuation or held queue. Runbook bb39de6, owned path rocky/CHROMERIA-2026-10-14-d74c023f.md; pulled with rebase immediately before commit, staged only that file and pushed normally.

  28. lukemaj
    added a commit that references this issue on Oct 29, 2026

    fix(childThreads): skip ancestor loads for children with no requests …

    f3c6f7f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions