fix(compression): honor raw pins in L1 recall

[ian-de-marcellus]

Problem

A raw (level-0) pin means the pinned span renders raw in the live view. The live selector honors it. The canonical L1 compression prompt did not: it still recalled the frontier summary covering the pinned span, so the compressor saw a summary where the resident sees raw text.

Fix

• Mixed frontier. A frontier summary that covers a force-raw message is descended rather than recalled: its children take its place, recursively, so only the branch containing the pin opens. The raw added to the prompt is just the pinned L1 chunks; sibling branches stay summarized. This mirrors the live selector (kv-control keeps protected raw leaves beside their group's recall).
• Empty-content frontier summaries (which can't be emitted as recall pairs) are descended the same way instead of exposing their whole span raw.
• The covered set is derived from the mixed frontier, so the pinned span renders raw instead of disappearing.
• Pin precedence matches the kv-stable selector (isForceRawPinBound: a fixed level wins over maxLevel).

Not covered

• Merge recall, L3+ merge inputs, transition summaries and the source-only L1 don't consult pins.
• A size bound / admission check on the canonical L1 request. It isn't pin-specific, so it's proposed as a follow-up PR.
• Summaries whose DAG children are missing (pre-existing; the frontier descends only into children that exist).

Tests

• test/l1-recall-raw-pins-frontier.test.ts: an L2+ frontier with one pinned message. Old messages raw in the next L1 prompt: 185 on the previous head, 9 with this change (matching the live view). Siblings stay as recall pairs; the pinned chunk is raw. Also a precedence table (classic, level: 0, maxLevel: 0, overlapping pins). Fails on the previous head and on main.
• test/summary-reasoning-roundtrip.test.ts: the direct-L1 case.
• Full suite: 855/855 (node 22).

🤖 Generated with Claude Code

[Anarchid]

Verdict: changes requested. One pinned message can make every later L1 prompt carry an entire L3 raw

Reviewed at 08b4314 against main eca70da (0.11.0).

The goal is right: the compressor shouldn't recall a summary over a span the resident sees raw. But the fix drops the whole covering frontier summary and replays every leaf under it through the raw-middle section, and that section has no budget. The live selector doesn't do this; it keeps sibling branches folded. So the PR swaps a small mismatch for a large one that can overflow the context window.

Blocker

1. A pin inside a merged summary expands the entire ancestor raw, with no bound. The new filter (autobiographical.ts:5337-5350) removes any frontier summary whose transitive leaves include a force-raw message. Coverage is now computed from the retained frontier only (:5368-5370), so every leaf of the dropped summary is uncovered, and section 3 "Raw middle" (:5485-5492) emits all of them. compressionRecallBudgetTokens caps recall pairs only. Nothing caps the raw middle, and the canonical L1 request goes out before context admission (:5909-5915); admission applies only to the fallback variants.

Repro: 240 messages, hierarchical, mergeThreshold: 6, adaptive → frontier {L1: 4, L2: 1, L3: 1}. pinRange() on one old message, then compress the next chunk:

	L1 prompt messages	bytes	old messages raw	live view (compile(4000)): old raw
main	19	3,388	5	9
main + this PR	199	43,129	187	9

The live view keeps 9 raw and folds the rest. The compressor now gets 187. This repeats on every L1 while the pin exists. At production scale an L3/L4 spans thousands of messages, so the result is a context_length rejection on every L1 and a stalled pyramid. The trigger is an ordinary operator action: connectome-host's WebUI exposes pinRange and markDocument (panel-data.ts:633-636), and a document mark is itself a pin (markDocument pushes kind: 'document' into this.pins, and pinnedPositions doesn't filter by kind).

Fix: build a pin-aware mixed frontier. Descend only the branches that contain a raw pin, keep untouched child summaries as recall pairs, and emit raw only for the pinned leaves (plus any unavoidable gap). That's what live kv-control already does: it allows protected raw leaves beside their group's recall (kv-control.ts:868-875). Separately, the canonical request needs a size bound or admission check before dispatch.

repro116.mjs (run against a built CM: node repro116.mjs <cm-root> <tmp-store-dir>)

import { rmSync } from 'node:fs';
const [root, store] = process.argv.slice(2);
const { ContextManager, AutobiographicalStrategy } = await import(`${root}/dist/src/index.js`);
rmSync(store, { recursive: true, force: true });
const requests = [];
let calls = 0;
const membrane = { complete: async (req) => { requests.push(JSON.parse(JSON.stringify(req))); calls++;
  return { stopReason: 'end_turn', content: [{ type: 'text', text: `Summary #${calls}: ordinary events.` }], usage: { inputTokens: 500, outputTokens: 100 } }; } };
const strategy = new AutobiographicalStrategy({ compressionModel: 'm', targetChunkTokens: 200, headWindowTokens: 0, recentWindowTokens: 0,
  autoTickOnNewMessage: false, minChunkCharsForLLM: 0, summaryParticipant: 'Claude', hierarchical: true, mergeThreshold: 6, adaptiveResolution: true });
const cm = await ContextManager.open({ path: store, strategy, membrane });
const filler = (n) => 'word '.repeat(n);
const ids = [];
for (let i = 0; i < 240; i++) ids.push(cm.addMessage(i % 2 ? 'Claude' : 'User', [{ type: 'text', text: `OLD-${i} ${filler(30)}` }]));
for (let r = 0; r < 200; r++) { await cm.compile(); await cm.tick(); if (calls > 150) break; }
const lv = {}; for (const s of strategy.summaries) if (!s.mergedInto) lv[s.level] = (lv[s.level] ?? 0) + 1;
console.log('frontier by level', JSON.stringify(lv));
cm.pinRange(ids[5], ids[5], { name: 'one-message' });   // one message deep inside the oldest top-level summary
const live = JSON.stringify(await cm.compile(4000));
console.log('live after pin: OLD raw =', (live.match(/OLD-\d+ /g) || []).length);
requests.length = 0;
for (let i = 0; i < 20; i++) cm.addMessage(i % 2 ? 'Claude' : 'User', [{ type: 'text', text: `NEW-${i} ${filler(30)}` }]);
await cm.compile(); await cm.tick();
const l1 = requests[0];
const texts = l1.messages.flatMap((m) => m.content).filter((b) => typeof b.text === 'string').map((b) => b.text);
console.log(`first L1 prompt after pin: ${l1.messages.length} messages, ${JSON.stringify(l1).length} bytes, ` +
  `OLD raw = ${texts.filter((t) => t.startsWith('OLD-')).length}, recall pairs = ${texts.filter((t) => t.startsWith('[CM] Recall memory')).length}`);
await cm.close();

Minor

2. The force-raw predicate differs from the live selectors for some pin combinations. (Sol) pinLevelBounds can store both level and maxLevel from overlapping pins (:2418-2427). This PR treats {level: 2, maxLevel: 0} as raw, while kv-stable checks pinLevel first and fixes it at L2 (kv-stable.ts:104-117). The flat and oldest-first folders honour only c.pinned (greedy-fold.ts:122), so pinAtLevel(…, 0) isn't live-raw there despite pinAtLevel's doc. That's narrow, and partly pre-existing: the live selectors already disagree with each other. Fix: resolve pin precedence once and use that result in compression and in every selector.
3. The claim is broader than the scope. (Sol) The comment at :5321 and the PR body say "a raw pin is a promise about every model-facing view". But merge recalls (:6954-6967), L3+ merge inputs (:7152-7166), transition summaries (:9927-9943) and source-only L1 are all untouched. Either apply the same rule there or narrow the comment to the canonical L1 prompt.
4. Missing DAG children fail open. (Sol) expandSummaryToLeafMessageIds silently skips absent children (:9129-9144). Reopen can drop an empty L1 and leave its L2 parent's sourceIds pointing at nothing (:1936-1974), so a pin under that missing child isn't detected. The resulting double representation (L2 recalled, orphan leaves raw) already happens on main for that legacy shape; the new filter just doesn't catch it.
5. Empty-content frontier summaries now surface raw as well. With the all-L1 expansion gone, a frontier summary skipped for empty content no longer hides its leaves, so they reach the same unbounded raw middle as #1. That's arguably more faithful, but it needs the same bound.
6. The test can't see #1. It pins an entire L1's span with mergeThreshold: 1_000 (no L2+), so the excluded span always equals the pinned span. Fix: add an L2+ frontier with a single-message pin, and assert that unpinned sibling branches stay summarized. Add table cases for classic, level: 0, maxLevel: 0 and overlapping pins.

Verified fine

• On a healthy forest there's no raw/summary duplication: only retained frontier roots populate coverage, and merged children are never recalled.
• The new L1's sourceIds/sourceRange still come only from chunk.messages (:6484-6505), so recall and raw context never leak into provenance.
• Transitive expansion is cycle-safe and linear in covered DAG size, since frontier roots are leaf-disjoint.
• The new test is red on main and non-vacuous for the direct-L1 case.
• main + #114–#118 merged together builds and passes 852/852 (node 22).

Method

Two reviewers in parallel. Sol (Codex gpt-5.6-sol, xhigh, read-only, no network; ~13 min, 90 commands, 4.1M input tokens (3.8M cached), 33k output) read the PR worktree against main and the sibling PRs. Claude built main and the five-PR merge and ran the repro above. It also checked every Sol citation. Both found #1 independently. Claude reproduced it at runtime; Sol traced the missing admission check. Sol only: #2, #3, #4. Claude only: #5, the WebUI/document-mark trigger path. Both: #6. Sol rated #2–#4 major; they were lowered because they're narrow or partly pre-existing.

[ian-de-marcellus]

Thanks. The blocker was real and reproduced exactly as described. Addressed in a577fbc, with the branch rebased onto current main (d1d17b0).

1. A mixed frontier, not a dropped one. A frontier summary that covers a force-raw pin (or has empty content) is now descended: its children take its place, recursively, so only the branch containing the pin opens up, and the raw added is just the pinned L1 chunk. On your repro (240 messages, one pinned message deep in the oldest top-level summary), old messages raw in the next L1 prompt: 185 on the previous head, 9 now, the same 9 your live compile(4000) shows. Unpinned siblings stay as recall pairs.
The canonical size bound: agreed it's needed, but it isn't specific to pins (a large head or chunk can oversize the canonical request too), and it touches the fallback-plan and trace machinery. With the mixed frontier, pins no longer cause the blow-up. I'd like to do canonical admission as its own follow-up PR rather than fold it in here. Say if you'd rather have it in this one.

2. Precedence: isForceRawPinBound resolves it the way kv-stable does: a fixed level wins over maxLevel, so {level: 2, maxLevel: 0} stays at L2. I haven't unified the flat/oldest-first folders (partly pre-existing, as you note); happy to follow up.
3. Scope: the comment, changelog and PR body now say "the canonical L1 prompt", and name the paths that don't consult pins.
4. Missing DAG children: pre-existing and unchanged. The mixed frontier descends only into children that exist. Worth its own issue.
5. Empty-content summaries: descended the same way, so their span is represented by their children rather than exposed raw.
6. Tests: l1-recall-raw-pins-frontier.test.ts builds an L2+ frontier and pins a single message. It asserts that the raw added is at most the pinned chunk, that siblings stay summarized, and that the pinned chunk is raw. Plus a precedence table (classic, level: 0, maxLevel: 0, overlapping). It fails on the previous head (185 raw) and on main (the pin is ignored).

Full suite: 855/855 (node 22).

🤖 Generated with Claude Code

[greptile-apps]

Confidence Score: 2/5

[Medium risk] Changes how compression recalls pinned messages.

Not safe to merge until pinned pre-transition messages remain present and the canonical L1 request preserves chronology and fits the compression model.

Findings

1.  Pinned history disappears ▶
2.  Pinned turns appear out of order ▶
3.  Wide pins exceed context ▶
4.  Missing children lose coverage ▶
5.  Cyclic summaries overflow descent ▶

Fix with agent prompt

### Issue 1
src/strategies/autobiographical.ts:5399-5402
When a raw pin covers a summarized message before a reset head-window start, the new frontier removes its recall pair. The head loop begins after that message, and the raw-middle loop begins after the head, so neither emits it. The pinned history disappears from the canonical L1 prompt instead of appearing raw.

### Issue 2
src/strategies/autobiographical.ts:5372-5374
When an old L1 is opened by a pin, its raw turns are emitted after all retained recall pairs, including pairs for later events. The compressor then sees those later events before the older conversation, rather than the chronological order used by the live view. This can distort the next memory it writes.

### Issue 3
src/strategies/autobiographical.ts:5373-5374
When a pin spans many compressed L1s, this change replaces each touched summary with its full raw chunk. The recall budget caps only the summaries that remain, and the canonical request is sent without the context-budget check used for fallback variants. If those raw chunks exceed the compression model's window, the next L1 repeatedly fails with a context-length error instead of compressing.

### Issue 4
src/strategies/autobiographical.ts:5375-5377
Loading legacy state can remove an empty child summary while leaving its ID in a higher-level parent's `sourceIds`. If a pin on a surviving sibling opens that parent, this loop silently skips the missing child. Its span loses recall coverage and may be replayed raw, needlessly enlarging later L1 prompts. Please handle an incomplete branch explicitly.

### Issue 5
src/strategies/autobiographical.ts:5372-5377
The existing leaf-expansion walk guards against cycles in stored summary graphs, but this new descent does not. If a raw pin makes an ancestor in a cyclic graph non-recallable, descent revisits it until the stack overflows and compression aborts. A visited guard would keep a damaged store from causing that failure.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The PR makes canonical L1 recall descend summary branches that contain raw pins or empty content, then derives raw-message coverage from that mixed frontier.

• Adds tests for a pin inside a merged frontier, a direct-L1 pin, and pin-bound precedence.
• The new prompt construction needs attention at head-window resets, chronological replay, and request-size boundaries.

_{Reviews (1) · Last reviewed commit: "fix(compression): pin-aware mixed fronti..."}

[greptile-apps]

Pinned history disappears When a raw pin covers a summarized message before a reset head-window start, the new frontier removes its recall pair. The head loop begins after that message, and the raw-middle loop begins after the head, so neither emits it. The pinned history disappears from the canonical L1 prompt instead of appearing raw.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 5399-5402

Comment:
**Pinned history disappears** When a raw pin covers a summarized message before a reset head-window start, the new frontier removes its recall pair. The head loop begins after that message, and the raw-middle loop begins after the head, so neither emits it. The pinned history disappears from the canonical L1 prompt instead of appearing raw.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Pinned turns appear out of order When an old L1 is opened by a pin, its raw turns are emitted after all retained recall pairs, including pairs for later events. The compressor then sees those later events before the older conversation, rather than the chronological order used by the live view. This can distort the next memory it writes.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 5372-5374

Comment:
**Pinned turns appear out of order** When an old L1 is opened by a pin, its raw turns are emitted after all retained recall pairs, including pairs for later events. The compressor then sees those later events before the older conversation, rather than the chronological order used by the live view. This can distort the next memory it writes.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Wide pins exceed context When a pin spans many compressed L1s, this change replaces each touched summary with its full raw chunk. The recall budget caps only the summaries that remain, and the canonical request is sent without the context-budget check used for fallback variants. If those raw chunks exceed the compression model's window, the next L1 repeatedly fails with a context-length error instead of compressing.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 5373-5374

Comment:
**Wide pins exceed context** When a pin spans many compressed L1s, this change replaces each touched summary with its full raw chunk. The recall budget caps only the summaries that remain, and the canonical request is sent without the context-budget check used for fallback variants. If those raw chunks exceed the compression model's window, the next L1 repeatedly fails with a context-length error instead of compressing.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Missing children lose coverage Loading legacy state can remove an empty child summary while leaving its ID in a higher-level parent's sourceIds. If a pin on a surviving sibling opens that parent, this loop silently skips the missing child. Its span loses recall coverage and may be replayed raw, needlessly enlarging later L1 prompts. Please handle an incomplete branch explicitly.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 5375-5377

Comment:
**Missing children lose coverage** Loading legacy state can remove an empty child summary while leaving its ID in a higher-level parent's `sourceIds`. If a pin on a surviving sibling opens that parent, this loop silently skips the missing child. Its span loses recall coverage and may be replayed raw, needlessly enlarging later L1 prompts. Please handle an incomplete branch explicitly.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Cyclic summaries overflow descent The existing leaf-expansion walk guards against cycles in stored summary graphs, but this new descent does not. If a raw pin makes an ancestor in a cyclic graph non-recallable, descent revisits it until the stack overflows and compression aborts. A visited guard would keep a damaged store from causing that failure.

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 5372-5377

Comment:
**Cyclic summaries overflow descent** The existing leaf-expansion walk guards against cycles in stored summary graphs, but this new descent does not. If a raw pin makes an ancestor in a cyclic graph non-recallable, descent revisits it until the stack overflows and compression aborts. A visited guard would keep a damaged store from causing that failure.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.