Skip to content

fix(events): snapshot cadence follow-ups (history doubling, open counts, deferred checks) - #1

Merged
mossaab merged 4 commits into
mossaab:fix/event-log-saturationfrom
vv7r:fix/event-log-saturation-followups
Oct 6, 2026
Merged

mossaab merged 4 commits into
mossaab:fix/event-log-saturationfrom
vv7r:fix/event-log-saturation-followups

Conversation

@vv7r

@vv7r vv7r commented Oct 4, 2026

Copy link
Copy Markdown

Follow-ups to co-l#401, as discussed in the thread: fixes 1–3, one commit each, each with its own test (written first, failing before the fix). co-l#4 comes later, once the sub-agent compaction fix lands.

Commit Problem Fix
keep one history entry per compaction across snapshots takeSnapshot() builds from combineEventsWithSnapshot(), whose stubs replay the previous snapshot's contextWindows / formatRetries just before it; the fold reseeds from the snapshot on top of them, so each snapshot doubles the history (1 → 2 → 4 → 8 over four flush()es at a662baed; seen live: 7 compactions as 2.3M entries, a 433 MB snapshot) When the fold seeds from a snapshot, keep one entry per (windows or attempt, timestamp), the only fields the stubs carry. This also shrinks a snapshot already inflated this way at the next snapshot
count open messages and pending tool calls as sets A successful compaction starts its summary message twice and closes it once: starts minus dones left one message open for good, so the cadence never snapshotted again in that turn Count the ids started and not done in the tail (and tool calls without a result), as you suggested. A done or result whose start is before the tail can no longer hide another open message or pending call. Same covering index (session_id, event_type, seq): 30 ms on a 48 MB tail, 64 ms on 100 MB, 269 ms on 400 MB, mostly the byte sum
snapshot at the next quiet point once a check was deferred A check that found the turn busy still reset lastCheckAt, so a sub-agent that is nearly always busy usually missed the next 30 s check too Once deferred, the next quiet point (message.done / tool.result) checks again, at most every 5 s

Verified: lint, typecheck, format, duplicate and the full suite (unit and e2e) pass on each commit, through the pre-commit hook.

AI-Enhanced Development

  • AI Models: Claude Opus 5.5 (Claude Code); live tests with a local Qwen3-based model on llama.cpp

Cache Impact

  • No. Snapshot persistence and the event log only; the requests sent to the model are unchanged.

🤖 Generated with Claude Code

vv7r and others added 3 commits October 4, 2026 23:17
A snapshot is built from the previous one plus the events since
(combineEventsWithSnapshot). That stream also replays the previous
snapshot's compactions and format retries as synthetic events, just
before it, for the consumers that count them; the fold then seeded its
history from the snapshot on top of them, so each snapshot doubled the
history and every later one doubled it again. With the mid-turn cadence
snapshotting often, it exploded: seen live, 7 compactions stored as 2.3
million entries in a 433 MB snapshot, rewritten at every turn end (8 s
per write, a prompt took that long to show) until the disk was full.

The fold keeps one entry per compaction or retry (same windows or
attempt, same timestamp) when it seeds from a snapshot. This also
shrinks a snapshot already inflated that way on the next snapshot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A successful compaction starts its summary attempt's message again with
the summary as content, then closes it once. The tail counted starts
minus dones, so every compaction left one message open for good: the
snapshot cadence then found the turn busy at every check and never took
a mid-turn snapshot for the rest of the turn (main turn or sub-agent).

The tail now counts the ids started and not done in it (and tool calls
without a result), instead of subtracting counts: a message started
twice counts once, and a done or result whose start is before the tail
can no longer hide another open message or pending call. The counts read
only their own event types through the session/type/seq index; the
check measured 30 ms on a 48 MB tail (the default budget), 64 ms on
100 MB, 269 ms on 400 MB, dominated by summing the tail's payload bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The cadence measures the tail at most every 30 s, on whatever event comes
in, and only snapshots at a quiet point (no message streaming, no tool
round in flight). A deferred check still restarted the 30 s period, and a
sub-agent is nearly always streaming or running a tool: the check kept
landing on a busy moment and the snapshot was put off to the end of the
turn. Seen live on 7 and 12 minute sub-agent runs, the tail over its
budget and no mid-turn snapshot (together with the open-message miscount
fixed in the next commit).

A check that finds the tail over budget while the turn is busy now marks
the cadence deferred, and the next quiet point (a message.done or a
tool.result) checks again without waiting for the period, at most every
5 s: each check scans the tail (~0.1 s at 46 MB), and parallel tools or
sub-agents can keep the turn busy over many quiet points in a row. The
budget is evaluated before the in-flight test, so only an over-budget tail
defers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mossaab

mossaab commented Oct 5, 2026

Copy link
Copy Markdown
Owner

One defensive nit on the new set-based counts: the NOT IN (SELECT ...) subquery is NULL-fragile. If a single message.done (or tool.result) in the tail had a NULL id, x NOT IN (..., NULL, ...) evaluates to NULL for every row, so openMessages / pendingToolCalls would silently read 0 — and the cadence would then treat the turn as at rest and snapshot mid-stream. I checked my session DB and found no NULL ids, so it's purely theoretical, but an IS NOT NULL in the subquery (or NOT EXISTS) removes the footgun for one line.

`x NOT IN (subquery)` is never true when the subquery returns a NULL: a
single message.done or tool.result without an id in the tail would read
openMessages and pendingToolCalls as 0, and the cadence would snapshot
mid-message. The subqueries now skip NULL ids.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vv7r

vv7r commented Oct 5, 2026

Copy link
Copy Markdown
Author

Good catch, fixed in 72d614e: the NOT IN subqueries now skip NULL ids, with a test where an id-less message.done and tool.result used to read both counts as 0.

@mossaab

mossaab commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Reviewed 72d614eb and verified it independently: with store.ts reverted to 542a8330 and your new test in place, it fails exactly as you described (expected +0 to be 1 for both counts). The four commits are merged fast-forward onto fix/event-log-saturation — linear history, no merge commit, your authorship and your one-commit-per-fix split intact. GitHub marked this PR as merged with 72d614eb as the merge commit.

Full suite on the resulting tip, on my side: 443 files / 5,994 unit tests and 49 files / 357 e2e passing, plus tsc --noEmit, ESLint, Prettier and jscpd clean.

One nit, same class as the one I reported and purely theoretical: the IS NOT NULL guard sits only on the done/result subquery, so a message.start with a NULL id can never be counted — NULL NOT IN (…) is not true. The typed append() API cannot produce one (you had to cast as never to force the case in the test), but a symmetric test would close the class rather than the instance.

co-l#4 is tracked in #2. Thanks again — still the most useful review this PR got.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants