feat: merge-and-rewrite dedupe + entity wiki pages + retroactive /rebuild#2
Merged
Merged
Conversation
…uild Replace the append-only dedupe path with an LLM merge that rewrites the canonical note (git-snapshotted first, safe append fallback). Add typed entity pages (People/Concepts/Projects) that grow per capture, and a /rebuild command that merges existing duplicates and rebuilds entity pages across the vault. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns Engram from an append-only capture pipeline into a self-rewriting wiki (the Karpathy LLM-wiki pattern). Three changes, all TDD — 169 tests passing (26 new).
1. Merge-and-rewrite dedupe
The semantic-duplicate path no longer appends a dated
## Updateblock. It rewrites the canonical note to integrate the new source, collapse redundancy, and reconcile contradictions inline.merger.py—merge_note(Claude rewrite returning the full note)note_writer.py:merge_into_note— safety guard: if the rewrite would drop the YAML frontmatter or any![[attachment]]embed, falls back to the oldappend_to_note. A capture is never lost.gitsafe.py—git inits the vault on first use and commits before every rewrite. Rollback:git -C <vault> reset --hard HEAD~.2. Entity (wiki) pages
entities.py— every capture extracts people/concepts/projects and grows typed pages underPeople/,Concepts/,Projects/, each accumulating one grounded observation per source note plus[[backlinks]].3. Retroactive
/rebuildretro.py+on_rebuildhandler — snapshots the vault, merges existing duplicate notes (cosine ≥0.88 and the 0.75–0.88 LLM-judge band; older note kept as canonical), then rebuilds all entity pages with synthesized leads. Reports counts to Telegram.Notes / caveats
/rebuildon the real vault is the proving run; it git-snapshots first, so it is reversible./rebuildis intentionally aggressive (judge band included, no dry-run). The pre-rebuild commit is the undo..gitis synced too (documented trade-off for in-vault rollback).🤖 Generated with Claude Code