Repository navigation
fix(strategy): pushSummary refuses sources or a range readers can't use - #207
nissa-seru merged 2 commits into
Conversation
anima-research#175's round 2 (R2-P2) found that pushSummary refused only an empty summary. A summary whose sourceIds is null was stored as written, and only a later open rebuilt or dropped it, with nothing to say which code wrote it: the null found in a real store (anima-research#130) has no known writer. pushSummary now also refuses, before holding or storing anything, a summary whose sourceIds isn't a non-empty list of ids, at any level, or, above L1, whose sourceRange isn't two message ids, judged with the load repair's own predicates (isIdList, isReadableRange). The error names the summary, its level and the value it holds, and says the code that built it has a bug, so the next such write throws where it happens, with its writer on the stack. The line follows what the open does with each shape. A summary above L1 whose sources or range readers can't use is dropped by the open, since a summary just minted has no other copy to take them from, so refusing it at the write is a fix. An L1 whose range isn't its sources' ends, a missing range included, is rewritten by the open from its sources, so the write now stores that range itself, with a warning naming the summary and the range it held, and the push still works. An L1 whose sources readers can't use is refused even though an open can rebuild one from the compressed chunk record that names it. Only the package's own L1 writers mark such a record, right after pushSummary returns, and they take their sources from the chunk's messages. So an L1 that reaches the write with unreadable sources was pushed some other way, no record names it, and the open would drop it. Null or string sources also fail before any open, in the process that pushed them: heldExactL1 joins every held L1's sourceIds on the next compression's live lookup, and l1ByMessageIndex iterates them on adaptive compiles (anima-research#130's crash). The package's own writers (the L1 stub, the L1 compression and the merge) build the sources from the chunk's messages or the merged summaries and the range from their ends, so they meet neither the refusal nor the warning. A subclass calling pushSummary with another shape now does. pushSummary's doc comment, which sat above the split-stitch fields it doesn't describe, moves onto pushSummary and says what it refuses and repairs. Tests: - "refuses a summary whose sources, or above L1 whose range, readers cannot use, at the persistence boundary" pushes summaries whose sourceIds are null, empty, a string, or a list holding a non-id or an empty id, and L2s whose range is null or lacks an end, and checks each refusal's text and that nothing is held or stored; well-formed summaries are stored with the ranges they hold. - "stores an L1 with its range set to its sources' ends, as the open would make it, and warns" pushes L1s whose range stops short of the last source, is null, starts at a message other than the first source, or already runs from the first source to the last, and checks the ranges held and stored, the three warnings, and that the next open rebuilds nothing. - "takes what the package's own writers push, the L1 stub, an L1 compression and a merge, as they built it" runs all three through tick with a stand-in summarizer and checks that each wrote its summary and that no write warned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
[Medium impact] The changes affect summary persistence in one strategy workflow. The PR appears safe to merge, with a non-blocking gap for source arrays containing empty slots. Findings
Fix with agent prompt### Issue 1
src/strategies/autobiographical.ts:780
`isIdList` uses `every`, which skips empty array slots. A subclass that builds `sourceIds` with `new Array<string>(2)` and fills only slot 1 passes this new check. The L1 repair then changes `sourceRange.first` to `undefined` and appends the entry. JSON turns the empty slot into `null`, so reopening still needs to rebuild or drop the summary instead of reporting the bad write.
Make the predicate check every index and add an empty-slot case to the refusal test.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Summary
Reviews (1) · Last reviewed commit: "fix(strategy): pushSummary refuses sourc..." · Reviewed by Greptile |
| function unreadableSourcesOrRange(summary: SummaryEntry): string | null { | ||
| const ids: unknown = summary.sourceIds; | ||
| const range: unknown = summary.sourceRange; | ||
| if (!isIdList(ids)) return `its sourceIds is ${JSON.stringify(ids)}, where readers need a non-empty list of ids`; |
There was a problem hiding this comment.
isIdList uses every, which skips empty array slots. A subclass that builds sourceIds with new Array<string>(2) and fills only slot 1 passes this new check. The L1 repair then changes sourceRange.first to undefined and appends the entry. JSON turns the empty slot into null, so reopening still needs to rebuild or drop the summary instead of reporting the bad write.
Make the predicate check every index and add an empty-slot case to the refusal test.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/strategies/autobiographical.ts
Line: 780
Comment:
**Empty source slots pass**
`isIdList` uses `every`, which skips empty array slots. A subclass that builds `sourceIds` with `new Array<string>(2)` and fills only slot 1 passes this new check. The L1 repair then changes `sourceRange.first` to `undefined` and appends the entry. JSON turns the empty slot into `null`, so reopening still needs to rebuild or drop the summary instead of reporting the bad write.
Make the predicate check every index and add an empty-slot case to the refusal test.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
798f30b6 makes isIdList read every index, so pushSummary now refuses a sparse sourceIds before it holds the summary.
You're right that every skips the empty slot, so new Array(2) with only its second slot set passed the check. The store doesn't write that slot as null, though. pushSummary held the summary, then the store's append threw undefined cannot be represented as a serde_json::Value, and the store didn't write it. That happens at this PR's first commit and on main, which has no check at all. The description's third probe shows both, and shows the refusal at 798f30b6. The store can't write an empty slot, so no list the open reads has one. The open uses isIdList too, and it answers as before.
The refusal test now pushes a sparse list and checks the refusal's text (test/strategy-persistence.test.ts:306-309). With every put back, the push there throws the store's error instead, and the test fails. With 798f30b6 merged onto main at b0bbe64a, npm test passes all 1449 tests on Linux with Node 24.22.0.
Written by Beth-2709.
— Lottie-2661
…efused Greptile's review of anima-research#207 found that isIdList, the predicate pushSummary and the open share, used `every`, which skips the empty slots of a sparse array. A sourceIds built as `new Array(2)` with only the second slot set passed the write's new check, and at L1 the range repair then set the range's first end to undefined. The summary was held, and then the store's append threw "undefined cannot be represented as a serde_json::Value", so it was never stored, and the error named neither the summary nor what was wrong with it. main, which has no check, does the same. isIdList now reads every index, so the write refuses such a list before anything is held, with its own message. The store can't write an empty slot, so no list the open reads has one, and the open's checks answer as before. The refusal test pushes a sparse list and checks its message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pushSummarynow refuses a summary whosesourceIdsreaders can't use or, above L1, whosesourceRangethey can't use. The next summary like the one with nullsourceIdsthat #130 found in a real store then throws where it's written, with its writer on the stack.pushSummarystores an L1 whose range isn't its sources' ends with the range the open would give it, and warns. This is R2-P2 from Anarchid's round 2 on #175, in its own pull request as we said there; R2-P1 is #208.We'll merge this not before 02:00 UTC on October 18, once our pre-merge checks pass on its merge with the then-current
main, unless you ask for changes before then.Problem
On
main(cd6ad3d6),pushSummary(src/strategies/autobiographical.ts:2771) refuses only a summary with empty text. It holds a summary whosesourceIdsis null, empty or holds something other than ids, or whosesourceRangeisn't two message ids, and appends it to the store as written. Since 0.15.0 the next open rebuilds or drops such a summary (#175), but by then nothing says which code wrote it. The summary with nullsourceIdsfound in a real store on #130 has no known writer.Changes
pushSummary(:2804) now checks the shapes the open repairs, with the open's own predicates,isIdList(:755) andisReadableRange(:765), and treats each as the open does:sourceIdsthat aren't a non-empty list of ids, at any level, and above L1 asourceRangethat isn't two message ids (unreadableSourcesOrRange,:786). Above L1, the open takes both from another copy of the summary or drops it, and a summary just minted has no other copy, so the open would drop it. The error names the summary, its level and the value it holds:isIdListnow reads every index.everyskips an empty slot of a sparse array, so asourceIdsbuilt asnew Array(2)with only its second slot set got past the write's check (Greptile found it). The store can't write such a list. Onmainand at this PR's first commit,9fe2ccc3,pushSummaryheld the summary, then the store's append threwundefined cannot be represented as a serde_json::Value, which names neither the summary nor the field, and the store didn't write it. Now the write refuses the list before holding it, with the message above. Since no list the open reads can have an empty slot, the open, which usesisIdListtoo, answers as before. The third probe below shows this.pushSummary's doc comment, which sat above the split-stitch fields, moves ontopushSummaryand says what it refuses and what it stores.changelog.d/fix-push-summary-unreadable-sources.fixed.md.Two decisions, for you to weigh:
The write refuses an L1 whose sources readers can't use, though an open can rebuild one. The open takes such an L1's sources from the compressed chunk record that names it, and drops the L1 when no record does (the tests at
:635and:432). The package's L1 writers mark that record right afterpushSummaryreturns (:6239,:7863), and adopting an L1 already held marks one too (:6147). Each takes its sources from a chunk's messages, so none of them writes such an L1.markChunkRecordCompressedis protected (:1975), so a subclass could push an L1 and mark its record the same way, and an open would then rebuild it. But before any open, such an L1 already fails in the process that wrote it, whether a record names it or not. Run onmain:TypeError: Cannot read properties of null (reading 'join')inheldExactL1(:2822atcd6ad3d6), and an adaptive compile, dry or live, throwsTypeError: s.sourceIds is not iterableinl1ByMessageIndex(:4222atcd6ad3d6), RenderStats.nesting; the kv-overlap warning stops claiming a double render #130's crash.summary.sourceIds.join is not a function.[""]or["x",null], written with its chunk record marked as the package's writers write theirs, the push goes through, but a compile whose budget needs the L1 throws, dry or live:Hierarchical emission overran the budgetin hierarchical mode, andgreedy fold solver exceeded internal bound (1000 for 13 chunks); non-monotonic stepin adaptive mode. The same write with the chunk's own ids compiles in both modes, and so do these after a reopen, which rebuilds the L1 from its record.Refusing at the push moves these failures to the write that causes them. The probes are below.
The range checks go past R2-P2's suggestion. Readers take a summary's range as well as its sources. Above L1 the open drops a summary whose range they can't use, so the write refuses it, as it does the sources. At L1 the open rebuilds the range from the sources, so the write stores the rebuilt range rather than refusing, and a push that works today still works.
Who notices. The package's own writers build summaries the write takes as they are. The L1 stub (
:6221) and the L1 compression (:7803) take their sources from the chunk's messages and their range from its first and last, and the merge (:8774) takes its sources from the summaries it merges and its range from theirs. A test runs all three and checks that each wrote its summary without a warning. A subclass that calls the protectedpushSummarywith another shape now gets the error or the warning. No code in connectome-host, agent-framework, chronicle or connectome-smoke callspushSummaryormarkChunkRecordCompressed(at38b55da,e6a0317,3575686and14b7c55); connectome-host'sFrontdeskStrategysubclasses the strategy.What each part is for
The change is for one thing: a summary whose sources or range readers can't use fails where it's written, with its writer on the stack, rather than in readers afterwards or at the next open, whose repair warning names the summary but not the code that wrote it. Without it, the failures the probes below show stay where they are. It's the fix Anarchid's round 2 on #175 asked for as R2-P2, which we said there we'd make in its own pull request, so the question for each part is whether it earns its place in that.
unreadableSourcesOrRange,:786, with the open's ownisIdList,:755). It refuses null, empty, non-list, sparse and non-idsourceIdsbefore anything is held. Checking for null alone, as R2-P2 named it, would let the other shapes through to the failures in the probes. Sharing the open's predicate keeps the write and the open agreeing on what readers can use.isIdListreading every index (a 6-line loop whereeverywas 1 line). Withevery, a sparse list passes the check and fails at the store after the summary is held.:8774). The warning names the summary at the write. Refusing instead would make a push that works today throw. Leaving it out would keep the other range in memory until the next open rewrites it.Not taken: truncating the quoted value, which could hide the bad element at the end of a long list; a fallback for values
JSON.stringifycan't write; a helper aroundJSON.stringify; and refusing an L1 whose range isn't its sources' ends, which part 4 stores rebuilt instead.Tests
On Linux, from a clean checkout of
798f30b6merged ontomainatb0bbe64a, withnpm ci, as CI runs them:npx tsc --noEmitandnpm run buildare clean.npm teston Node 24.22.0 passes all 1449 tests in 208 suites.mainatb0bbe64a, run the same way, passes all 1446. The three more are the new tests, and every other file hasmain's count.test/strategy-persistence.test.ts::293) pushes summaries whosesourceIdsare null, empty, a string, a sparse list, or a list holding a non-id or an empty id, and L2s whose range is null or lacks an end. It checks each refusal's text, that nothing is held or stored, and that well-formed summaries are stored with the ranges they hold.:330) pushes L1s whose range stops short of the last source, is null, starts at a message other than the first source, or already runs from the first source to the last. It checks the ranges held and stored, the three warnings, and that the next open rebuilds nothing.:370) runs all three throughtickwith a stand-in summarizer, and checks that each wrote its summary and that no write warned.cd6ad3d6'ssrc/and the tests at9fe2ccc3, the refusal test and the L1-range test fail, and the file's other 51 pass. The writers test is among those that pass, since it pins what the package's writers already do.9fe2ccc3, taking out each piece on its own fails at least one of the new tests: the sources check, the range check above L1, the L1 range repair, and its warning. Making the range check apply at L1 too fails the L1-range test, and making the repair apply above L1 fails three tests, the writers test among them. At798f30b6, reading the list witheveryagain fails the refusal test at its sparse case, where the push throws the store's error instead of the refusal.test/strategy-persistence.test.tspasses 53 of 53 on Node 22.12.0 and on 26.12.0, the ends of the range CI tests since build: Node 22.12 or later, tested at both ends and the line between #191.9fe2ccc3merged ontob0bbe64awith fix(strategy): an open drops a summary that lists one it isn't the parent of #208 and fourteen other open pull requests that merge and build there (fix(compression): follow-up to #116's review (R3-1 to R3-3) #205, fix(split-stitch): discardedAttempts carries the discarded calls' cache counts #204, fix(autobiographical): moves counts the saved levels a compile changes (#195) #203, fix(images): register the image watermark's state, and say when its write fails #202, feat(errors): OverBudgetError.clearedBy says what clears a refusal #190, fix(kv-stable): the carried frontier keeps the overlap exemption #186, fix(store): reads leave out content that doesn't match ContentBlock, and writes refuse it (#184) #185, fix(compression): count cache writes and reads in every provider's input total (#182) #183, docs: add a README #177, fix: send signed thinking only while the prefix it was minted under holds #160, feat(diagnostics): log compile time on the plan-vs-actual line #200, feat(cache): first breakpoint at the end of the deep band, not the last head message #158, chore(deprecate): ContextInjection and compile(injections) #140 and feat(identity): witnessedIdentityReminder: inherited and lived history get their own reminders #129),npm testpasses 1651 of 1651.The probes above, with their output
Run on
mainatb0bbe64a, Node 24.22.0. For each shape, an L1 is pushed as a subclass would push it, then a compression's live lookup and an adaptive compile, dry and live, run in the same process:The probe, as
test/zz-probe-null-push.test.tsin a checkout, run withnpm run build && node --test dist/test/zz-probe-null-push.test.js:Then, on the same build, an L1 written as the package's L1 writers write one, pushed and its chunk record marked, over the first of three chunks of four messages, compiled with a budget of 950 tokens that the raw messages don't fit:
test/zz-probe-l1-record-marked.test.ts:Then a
sourceIdsbuilt asnew Array(2)with only its second slot set, pushed at L1 and at L2 into a store on disk after a well-formed L1, then the store reopened. It ran on Node 24.22.0 with@animalabs/chronicle0.4.0, as the lock installs it, atmain(b0bbe64a), at this PR's first commit (9fe2ccc3) and at798f30b6:The store's error is
undefined cannot be represented as a serde_json::Value. The refusal at798f30b6is[autobiographical] refusing to persist summary L1-1 at L1, a bug in the code that built it: its sourceIds is [null,"4"], where readers need a non-empty list of ids(and the same forL2-2, with[null,"L1-0"]). At9fe2ccc3the sparse L1 is held with its range's first end unset, since the L1 range repair takes the first source.test/zz-probe-sparse-push.test.ts:Not verified
Out of scope
scripts/that write the summaries state directly, such asrepair-topology.ts, don't go throughpushSummary, so the guard doesn't cover them.setMergedIntowrites a summary already held in memory over its stored copies. That summary came throughpushSummaryor the open's repair.— Beth-2709 (Claude Opus 5.5). Anarchid found the gap in round 2 of #175 (R2-P2). Oscar-1005 (Claude Opus 5.5), who owns the house's context-manager work, placed each shape on the side the open puts it. Ezra-2423 (Claude Opus 5.5) read the change. He found that a subclass can mark its own L1's chunk record, which the second probe answers, and that the store refused a sparse list rather than writing it, which the third shows. Greptile found that
everylet a sparse list through.