[nissa-seru]

For #251. A MessageStore re-materialized its whole message slot at its next read after a sibling MessageStore on the same store appended to it. With this change, a read after nothing but appends fetches the new records by point lookup and adds them to its cache. Any other write still costs the reload it did. The change relies on the slot being written only through MessageStore, as the cache's revalidation already does, and "Changes" says what a write outside it can now do.

Problem

getAllInternal checks the branch and the write version the store's MessageStores share before anything else. Every append moves that version (bumpWriteVersion), so after a sibling's append the check fails, and the read loads the whole slot from chronicle (getStateJson). The issue places the miss at the revalidation by count and last record. That revalidation sits inside the version check and runs only while the version is unchanged, so a sibling's append never reaches it. With CM_CACHE_DIAG=1, each such read logs a line such as getAllInternal MISS (writeVersion 2000→2001) stateId=messages rebuilt 2001 in 30ms.

At 20,000 short messages, with chronicle 0.4.0 under Bun 1.4.3 on a shared Linux host (load average 5–9), the reader's getAllInternal after one sibling append took 202–272 ms on main. Its getAll() with no write between took 0.09–0.12 ms. Those are the medians of three runs of 11 rounds each.

A second cost sits in chronicle. After any write to a slot, a point lookup of the newest record is cheap, and a point lookup of any older record rebuilds chronicle's per-item cache, which then serves cheaply until the next write. On chronicle 0.4.0 at 20,000 messages, after two appends, the newest record read in 0.014 ms, the second-newest in 87 ms, and the third-newest just after in 0.063 ms (medians of 7). getStateSlice of the last three records after three appends took 69 ms, and of the last four just after, 0.033 ms. So the check the issue suggests, a point lookup of the cached last record, costs that rebuild at every read after a sibling's append.

Changes

• getAllInternal reads the appended records instead of reloading the slot, when the branch is the same and the slot grew by exactly the write versions moved since the cache was built. An append moves the version by one and adds one record, and every other mutation moves it and adds none. So the equality holds only where nothing but appends happened, through any MessageStore on the store. bumpWriteVersion's doc now says so. appendedSince reads the new records by point lookup, and getAllInternal pushes them onto the cache and stamps it with the current version and sequence. Anything else reloads as before:
  • a sibling's edit or removal;
  • a branch switch;
  • a slot whose length the versions don't account for;
  • a record that doesn't read;
  • a chronicle without point lookup;
  • a failed revalidation with no version moved.
• The cached records aren't looked up to confirm them. On chronicle 0.4.0 that lookup would cost the per-item rebuild at every read after a sibling's append.
• append keeps a cache that wasn't current, for the next read to extend or reload, where it used to drop it. By our reading of agent-framework, residents that share a slot each append to it, so after another resident's append, each one's next read now extends its cache rather than reloading the slot. The append's CM_CACHE_DIAG line now says append write-through deferred (cache wasn't current before the append), and test/manager-diagnostics.test.ts follows it.
• The contract this relies on. Like the revalidation by count and last record, the change relies on the slot being written only through MessageStore. A write outside it moves no version. So one that adds a record beside a sibling's edit or removal leaves the slot's length just what the versions predict. The reader then extends a stale cache, and an edit or remove through its index can reach another record, where main's version check reloaded. Nothing writes the slot otherwise today:
  • context-manager's src writes it only through MessageStore;
  • agent-framework e01a2337 and connectome-host c7137862 write no message slot directly;
  • chronicle 0.4.0 refuses a second open of a store, in the same process or another ("Store is locked by another process"), so the version, kept per store object, counts every write through MessageStore.
• A changelog fragment, which states the contract too.

This meets #238. Merged with #238 at bce0c8b2, this change conflicts in append's write-through, beside the seven places in src/message-store.ts where #238 already conflicts with main. getAllInternal merges with no conflict marker. There, #238's slotRevision takes the write version's place, so the merged code passes the difference of two slotRevision values to appendedSince as the count of appends. #238 drops the edit version and keeps sharedWriteVersions as slotRevision's fallback for a chronicle without stateHeadOffset, so on chronicle 0.4.0, which has none, that difference is still the count. Against a chronicle with stateHeadOffset (chronicle #40), slotRevision is the log offset of the slot's newest record, which every write to the slot moves, and so do the snapshots the store adds after them. The difference is then a log distance rather than a count of writes. A branch-bound view is also a JS object of its own. The check would stay exact under stateHeadOffset if chronicle also reported the slot's newest non-append offset. While that offset is unchanged, only appends happened, through any handle.

What each part is for

• The length check against the versions moved. It is the test for "only appends since" through MessageStore, in O(1). A version count alone can't tell an edit from an append, and a record lookup to confirm the cache costs what this change avoids.
• The point lookups of the appended records. They read only what's new. After two or more appends, the first lookup pays chronicle's per-item rebuild, and the rest are cheap.
• The branch check. A cache built on one branch is never extended with another branch's records, even where the lengths happen to agree.
• Running only when the version moved. With no version moved, a failed revalidation by count and last record reloads, as on main, rather than reading "nothing appended" as a current cache.
• appendedSince's refusals. A record that doesn't read, or a chronicle without getStateItemJson, reloads. Without the second, getInternal would fall back to getAllInternal and recurse.
• Restamping the version and sequence after extending. The next read with no new write then returns the cache without a lookup.
• Keeping a stale cache in append. Without it, a reader's own append after a sibling's drops the cache, and its next read reloads the slot whatever getAllInternal does.
• The diagnostics. CM_CACHE_DIAG's miss line now says when a cache wasn't extended (not appends only, with the counts), and the append's line says it kept the cache. The append's failure line loses two reasons that could no longer print.
• The tests' helpers. siblings() builds the reader and its sibling on one store, and texts() reads a listing's text. pointLookupsDuring() records which records a read looks up, which is what pins the point lookups above.
• The fragment says what hosts will notice, the contract included.
• Cut. A separate edit-version check beside the length check. Through MessageStore, an edit or removal moves the write version without adding a record, so the length check refuses it, and taking the separate check out failed no test. Beside a write outside MessageStore, it would narrow the stale case without closing it, since an outside removal and an outside append beside a sibling's append still match.

Tests

At this head on main 7f963c75, on Linux with Node 24.22.0 and chronicle 0.4.0 from the lockfile (npm ci):

• The suite. npm run build is clean, and npm test passes 2,214 of 2,214, nine of them new.
• Red-first. With main's src/message-store.ts under these tests, tests 1 and 2 below fail. The others pass there too, since they pin reloads main already does.
• Each piece taken out, one at a time, fails a test:
  • no tail fill: tests 1 and 2;
  • no branch check: test 9;
  • the length checked only for shrinking: tests 3 to 5, and three earlier tests in the file;
  • an unreadable record kept: test 6;
  • no point-lookup check: test 7;
  • the fill run with no version moved: test 8;
  • the version kept after a fill: tests 1 and 2;
  • the sequence kept after a fill: test 1;
  • append dropping a stale cache: test 2;
  • the cached last record looked up before the fill, as the issue suggests: test 1.
• Composed with the open pull requests that merge cleanly. As of 13:38Z on October 19, fix(views): a merged view serves one listing while its sources are unchanged (#45) #228 (d41178e2), feat(kv-unified): latent demand measures the candidate set in one joint solve (#45, #131) #229 (38922c2e), feat(kv-unified): read ages in geometric bands so a leaf's weight moves only at band edges (#45) #230 (80dedf58), feat(kv-unified): subtree-memoized engine, the default on every forest (#45) #231 (00aaedd8), fix(kv-unified): hold a level pin at the deepest level its chain supplies until the pinned level exists (#193) #213 (26a97abe), chore(deprecate): ContextInjection and compile(injections) #140 (cc1e7e71), feat(identity): witnessedIdentityReminder: inherited and lived history get their own reminders #129 (427ceb91) and fix(merge): merge around quarantined sources; add operator merge holds #117 (5a3a6b6b) merged cleanly with this change and with each other, and with all of them in, npm test passed 2,304 of 2,304. Every other open pull request conflicted with main already, in the same files with and without this change. feat(derive): a context manager derived from a live one — shared reads, separate writes, the rendering frontier reused (Dendrite) #238's conflict is the one above.

The new tests, in test/message-store-sibling-append.test.ts, count whole-slot reads (getStateJson) and point lookups:

1. A sibling's two appends are read with no reload, in order, and the read looks up only those two records. The next read neither reloads nor looks anything up, and a lookup by id finds an appended record with no reload.
2. A reader's own append after a sibling's, its own edit and its own removal, each followed by a sibling's append, cost no reload.
3. A sibling's edit, then its append, costs one reload, and the reader sees the edit.
4. A sibling's removal, then its append, costs one reload, and the reader sees the removal.
5. A slot whose length a write outside MessageStore cut, with a sibling's append, costs one reload and reads as the slot does.
6. An appended record that doesn't read by point lookup costs one reload, not a short cache.
7. A chronicle without point lookup reloads without recursing.
8. With no version moved, a last record that isn't the cached one costs one reload, as the revalidation by count and last record gives it.
9. A cache built on one branch isn't extended on another branch, where a write outside MessageStore replaced a record and a sibling appended. The length agrees, and the read still costs one reload.

Timing. Two MessageStores on one store, at 20,000 short messages, under Bun 1.4.3 with chronicle 0.4.0, in three runs that alternate this change's code and main's (medians of 11 rounds each):

• One sibling append between reads. getAllInternal took 0.012–0.022 ms with the change, against 202–272 ms on main. getAll() after it took 0.37–0.62 ms with the change, and 4.3–6.2 ms on main.
• Three sibling appends between reads. getAllInternal took 74–116 ms with the change, against 136–242 ms on main. In each run's pair the change cost 25–69% less.

A read after one append costs a point lookup of the newest record. A read after two or more pays chronicle's per-item rebuild once, plus a cheap lookup per appended record, and saves the parse of the whole slot. A second probe, under Node 24.22.0 on Linux, found the same direction. At 10 to 6,000 appends onto 20,000 messages, the change cost 29–68% less than main. At 15,000 appends onto a reader warmed at 200 messages, it cost about what the reload costs (72 and 85 ms, against 90 and 81 ms).

In agent-framework, the issue's example reader is the subconscious. It reads the residents' shared slot through its own MessageStore (auxiliaryMessageViews), and its tune-out wakes it, not each message, so several appends usually land between its reads. That's from reading agent-framework 8225e4bf, not from a run. The one-append figure comes from these probes, not from a host.

Not verified

• A host's own read pattern. The timings come from probes of two MessageStores on one store, not from agent-framework or connectome-host. Nobody measured how many appends land between a resident's reads, and nothing was compiled against a real store.
• Size. Only short messages written by the probes, at most 30,111 in a slot.
• The timed code. The Bun timings ran on this change's logic at an earlier round, whose message-store.ts differs from this head's only in comments and diagnostic text.

Out of scope

• Chronicle's per-item rebuild after a write, which a read after two or more appends still pays. Chronicle's open release context-manager 0.5.10 #40 describes its per-item cache "advanced by a write instead of dropped".
• The cost the API docs give for getWindow, getMessageWindow and the timestamp query, O(window) and the like. On chronicle 0.4.0, their first read of an older record after a write pays that rebuild too, with or without this change.
• A bound on how many appended records a read looks up. One lookup per record cost less than the reload, or about the same, at every count measured above, so a cut-over to a reload would add a branch for no measured gain.
• The id index. A lookup by id after a sibling's write still rebuilds its map from the extended cache, O(slot) in map entries, with no reload.
• getAll()'s mapping. It still loops over every message after a sibling's write, reusing each earlier mapped object.
• Guarding edit and remove against a stale index, by checking the record at the index has the id before writing. Under the contract no path reaches a stale index, and the check would cost a point lookup per mutation.
• Typing getAll()/getTail() as readonly, which the issue also suggests. It isn't a defect, and callers would meet the change.

---

• Changelog fragment added: changelog.d/251-sibling-append-tail.fixed.md

— Hilda-2580 (Claude Opus 5.5). Oscar-1005 (Claude Opus 5.5) placed the issue with me and made the design calls. Ramona-2416 (Claude Opus 5.5) read the change, found the case the contract now names, and ran the timing probe under Node. An internal review read it as well. Checked and opened by Fiona-2757 (Claude Opus 5.5) through the shared nissa-seru account.

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling.

[greptile-apps]

Required label not found on this PR.