perf: bound the daemon's warm mirror to recent history (#167) - #172
Merged
Merged
Conversation
The daemon mirrored the entire bus — every key AND every body — and refreshed it on every poll. On a snapshot-capable backend that is one round trip, but it read every blob in the store each time: pending mail, the roster, the whole 30-day archive, and every telemetry batch. The cost of a poll therefore grew with all history, forever, on a bus where nothing but the last few messages is hot. The mirror now holds what is worth holding. `agents/` is re-read every poll (records mutate); `inbox/` is always warm; `read/` and `events/` are warm only inside a window (AGENTCOMM_MIRROR_HISTORY_MS, default 7 days). Older history keeps its KEY, so list, log, purge and channel discovery see the whole store exactly as before, and a cold body loads on demand through the `get` passthrough. Message blobs are immutable, so a body already held is never re-read either: steady state is "the roster plus whatever is new". `Snapshottable.snapshot` grew a body filter and now reports the keys it saw alongside the bodies it read — listing names is cheap, reading blobs is not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #167. (Supersedes #168, auto-closed by GitHub when its base branch merged; same commit, rebased onto
main.)The daemon mirrored the entire bus — every key and every body — and refreshed it on every poll. That read every blob in the store each time: pending mail, the roster, the whole 30-day archive, and every telemetry batch. The cost of a poll grew with all history, forever, on a bus where nothing but the last few messages is hot.
What changed
src/daemon.ts— a warm-window policy:agents/re-read every poll (records mutate),inbox/always warm,read/andevents/warm only insideAGENTCOMM_MIRROR_HISTORY_MS(default 7 days). Aged-out history keeps its key, solist,log,purgeand channel discovery see the whole store exactly as before; the body loads on demand through the existinggetpassthrough. Immutable blobs already held are never re-read, so steady state is "the roster plus whatever is new".src/types.ts/src/backends/git.ts—Snapshottable.snapshot(prefix, {bodies})now takes a body filter and returns{keys, bodies}. Listing names (ls-tree) is cheap;cat-fileover every object is not.Tests
test/git.e2e.test.ts: a snapshot reads only the requested bodies while still reporting every key, and a skipped body is still onegetaway.test/daemon.e2e.test.ts: a month-old archived message is outside the window, yetlogstill returns its body andpurge --dry-runstill counts it.