[nissa-seru]

With this change, removing the messages behind most of AutobiographicalStrategy's chunk records, or hiding them with a viewFilter, no longer stops compression of the messages that follow. While the chain-break guard does stop it, getCompressionDebt() reads critical and says why. The guard now counts a chunk record against the store only when the store can't show it ever held any of the record's message ids. It does that with a new MessageStore.held(id). Records naming ids the store never held, the case the guard exists for, still stop compression.

Fixes #421.

@antra-tess, we're asking you to review this before it's merged, since it changes whether a resident's memory goes on compressing after a removal or under a viewFilter, and what getCompressionDebt() reports while the guard stops it.

Problem

rebuildChunks' guard (src/strategies/autobiographical.ts:17901–17920 at fb4cf1046) counted a chunk record as orphaned when none of its messages was in the strategy's view. Once at least three records existed and more than half were orphaned, no new chunk formed. A removal (removeMessage, removeMessages) or a viewFilter hiding old history gives that count from ids the store did hold. #421 shows it by a test file it carries, at fb4cf1046, on Linux with Node 24.22.2. These are its lines after a removal of the messages behind 5 of 8 records, and after a reopen under a filter hiding them:

remove-5-of-8 after 40 more: records 8 (compressed 8); L1s 8; calls 9; queue 0; orphaned true; isReady true; debt healthy pending 0; messages 58; fail-closed errors 1
remove-5-of-8 reopened, after 40 more: records 8 (compressed 8); L1s 8; calls 9; queue 0; orphaned true; isReady true; debt healthy pending 0; messages 98; fail-closed errors 2
hide-5-of-8-then-show reopened: records 8 (compressed 8); L1s 8; calls 9; queue 0; orphaned true; isReady true; debt healthy pending 0; messages 38; fail-closed errors 1
hide-5-of-8-then-show after 40 more: records 8 (compressed 8); L1s 8; calls 9; queue 0; orphaned true; isReady true; debt healthy pending 0; messages 78; fail-closed errors 1

Changes

• MessageStore.held(id) (src/message-store.ts:1175) answers whether the store ever held a message under an id, on any of its branches. It answers true for a live message and for one removed since, and false for an id that names no Append to this store's messages state, including one chronicle refuses as a record id. MessageStoreView gains it as the optional held? (src/types/strategy.ts:36). filterMessageStoreView passes it through, so a filtered view answers for the messages it hides (src/message-view.ts:62). mergeMessageStoreViews answers when every source can (:152).
• The guard counts a record with no message in the view only when the view can't show that the store held any of the record's ids (src/strategies/autobiographical.ts:17848, :17925). A view without held counts as before. The answer is kept per record for the load, except a negative over a record number the store hasn't used yet, which a later write can give a message (:17857).
• getCompressionDebt(), while the guard holds, reads critical, with failure: 'chunk-records', an addition to failure's union, and an error saying what happened, with Cause:, Effect: and Fix: lines (:237, :8549). When the quarantine ledger can't be read at the same time, error holds both messages.
• The FAIL-CLOSED log line and the guard's comments now say the records name ids the store can't show it ever held.
• Tests: test/removal-guard.test.ts is new, and one comment in test/chunk-persistence.test.ts changes.
• Changelog: two fragments, changelog.d/removal-keeps-compressing.fixed.md and changelog.d/removal-keeps-compressing.added.md.

In all, 8 files, +413 −14. Merged at 19:04 UTC on October 30 with each of the 13 open pull requests' heads, this head conflicts where main does, except with #350 (below).

With #350

@tel-0s, by git merge-tree at 19:04 UTC on October 30, your #350 at 4d04b425ae merges into main fb4cf1046 cleanly and conflicts with this head in src/types/strategy.ts, where each adds an optional method to MessageStoreView after sequenceOf: this head held? and yours positionOf?. Both can stand.

Tests

On Linux with Node 24.22.2, after npm ci from the lockfile, which pins @animalabs/membrane 0.9.0 and @animalabs/chronicle 0.5.1. Each run is in a clean checkout of the commit named, with a file added where one is named.

• At this head, c1dcc0fd0: npm run build exits 0. test/removal-guard.test.ts passes 11 of 11, and test/chunk-persistence.test.ts 6 of 6, its chain-break test unchanged.
• Red first: with fb4cf1046's src and this head's tests, npm ci and npm run build each exit 2 on 14 type errors, all at the tests' calls of held, and the tests then run from what tsc emitted. All 11 of removal-guard's tests fail. Five removal and filter cases fail at "an L1 covers messages added after it". The numbers-given-later case and the two never-held cases fail at the debt report, which reads healthy where they expect critical, and the ledger case where it reads unknown. held's two tests fail where fb4cf1046 has no held.
• The issue's test file at this head: after the removal and 40 more messages, and after the reopen under the filter, compression goes on, with no FAIL-CLOSED line:

remove-5-of-8 after 40 more: records 18 (compressed 18); L1s 18; calls 21; queue 0; orphaned false; isReady true; debt healthy pending 0; messages 58; fail-closed errors 0
remove-5-of-8 reopened, after 40 more: records 28 (compressed 28); L1s 28; calls 32; queue 0; orphaned false; isReady true; debt healthy pending 0; messages 98; fail-closed errors 0
hide-5-of-8-then-show reopened: records 8 (compressed 8); L1s 8; calls 9; queue 0; orphaned false; isReady true; debt healthy pending 0; messages 38; fail-closed errors 0
hide-5-of-8-then-show after 40 more: records 18 (compressed 18); L1s 18; calls 21; queue 0; orphaned false; isReady true; debt healthy pending 0; messages 78; fail-closed errors 0

• What removal-guard pins:
  1. removing the messages of 3 of 5 records one by one, or of both of 2 as a range, leaves later messages compressing;
  2. a store reopened under a viewFilter hiding the messages of 5 of 8 records compresses later messages;
  3. a store whose records a removal emptied before it closed compresses later messages after it reopens;
  4. records each naming a removed message beside ids the store never held leave later messages compressing, and records naming record numbers the store gives messages later in the same load stop halting compression once it has;
  5. records naming ids the store never held, of another form or record numbers it never assigned, still stop compression, with the debt report critical, carrying the ledger's error too when the ledger can't be read;
  6. held answers as above, through filtered and merged views, after compactState, after a reopen and on a fork.

The whole suite runs in this pull request's CI.

Not verified

• A real store. Every run used synthetic stores with a stubbed summarizer, and none compiled against a host's store.
• What held costs on a large store. An emptied record's check decodes one removed message's record per load. No timing of it here ran on an unloaded machine.
• A store rebuilt in a new chronicle. No test here copies chunk records into a store whose new message ids collide with the ids the records name.
• With feat(view): blocks with an audience, and the reader a manager compiles for #364 merged. At its head, 6dfbd9c83c, strategyMessageView wraps the strategy's view in audienceMessageStoreView (src/context-manager.ts:600), whose view has no held (src/message-view.ts:168). Merged with it, an earlier version of this change, never pushed, failed its five removal and filter tests at "an L1 covers messages added after it". This head differs from that version in two tests and one rule for what the guard keeps per record, and wasn't run merged with feat(view): blocks with an audience, and the reader a manager compiles for #364's head.
• With feat(presentation): a window per manager, which the strategy view reads (#291 step 1) #350 merged: whether its window view passes held through. No run here merged them.

---

• Changelog fragments added: changelog.d/removal-keeps-compressing.fixed.md and changelog.d/removal-keeps-compressing.added.md.

Fern-2426 found the defect. Cedric-2315 made the change and measured where the guard trips, Alfred-2570 checked that each part is needed, and Oscar-1005 placed it. Lottie-2661 ran the tests at this head and wrote this description.