feat: merge-and-rewrite dedupe + entity wiki pages + retroactive /rebuild

README.md

@@ -20,8 +20,10 @@ Built as a personal second-brain pipeline; published in case it's useful to anyo
 - **Capture anything** — text, URLs, photos, voice notes, PDFs, Word docs (`.docx`/`.doc`), plain-text/code files, forwarded posts, YouTube links. Media groups (multi-photo posts) are debounced and stitched into a single note.
 - **Read the contents, not just the message** — Claude vision OCR for photos, OpenAI Whisper for voice, PyPDF/python-docx for documents, YouTube Transcript API for videos, plain HTTP fetcher for web pages.
 - **AI routing** — Claude (Sonnet) picks a folder, writes a title, summarises the body, generates tags, and proposes up to 5 related notes from your existing vault. Falls back to an "Other" bucket and an inbox queue when confidence is low.
-- **Smart dedupe** — incoming notes that match by URL, title, or semantic similarity (cosine on OpenAI embeddings) get *appended* to the existing note instead of creating a duplicate.
+- **Merge-and-rewrite dedupe** — incoming notes that match by URL, title, or semantic similarity (cosine on OpenAI embeddings) are **merged into the canonical note**: Claude rewrites the page to integrate the new source, collapse redundancy, and reconcile contradictions inline (instead of stacking dated append blocks). The vault is git-snapshotted before every rewrite; if a rewrite would drop the frontmatter or an attachment it safely falls back to a plain append, so a capture is never lost.
+- **Entity (wiki) pages** — every capture also grows typed backbone pages under `People/`, `Concepts/`, and `Projects/`. Each entity page accumulates one grounded observation per source note plus `[[backlinks]]`, turning the chronological capture stream into a navigable wiki.
 - **Vault-grounded Q&A** — `/ask` runs a hybrid keyword + embedding retrieval over your notes and answers with Claude, with multi-turn follow-ups via Telegram reply threads.
+- **Retroactive rebuild** — `/rebuild` git-snapshots the vault, merges existing duplicate notes, and rebuilds all entity pages from scratch. One command turns an existing note pile into the wiki.
 - **Manual override** — every capture shows an inline-keyboard folder picker; misroutes are one tap away. `/redo`, `/edit`, `/undo`, and `/relink` cover the rest.
 - **Single-tenant by design** — an `ALLOWED_USER_IDS` allowlist gates every handler. Nobody else who finds your bot can use it.
 
@@ -37,9 +39,14 @@ Telegram message
      ↓
 Claude enrichment: title · summary · tags · folder · related notes · confidence
      ↓
-Dedupe check (URL → title → semantic) → append to existing OR create new
+Dedupe check (URL → title → semantic)
+     ├─ match    → git snapshot → Claude merge-rewrites the canonical note
+     └─ no match → create new note
+     ↓
+Entity pass: extract people/concepts/projects → grow typed wiki pages
      ↓
 <vault>/<Category>/<Title>.md   with YAML frontmatter + [[backlinks]] + attachments/
+<vault>/{People,Concepts,Projects}/<Entity>.md   accumulating observations + backlinks
 ```
 
 ## Requirements
@@ -109,11 +116,23 @@ All config is via environment variables (loaded from `.env` if present). See [.e
 | `/inbox` | List notes flagged for review (low-confidence routing) |
 | `/review` | Walk pending notes one at a time with move / mark-reviewed / delete buttons |
 | `/relink [folder]` | Refresh related-note backlinks. No arg = last capture; with arg = entire folder |
+| `/rebuild` | Git-snapshot the vault, merge existing duplicate notes, and rebuild all `People`/`Concepts`/`Projects` entity pages from scratch. Destructive but reversible (see below) |
 | `/redo` | Reply with `/redo` to regenerate a capture using the higher-quality Opus model |
 | `/edit <text>` | Replace the source of the last capture and re-enrich |
 | `/undo` | Delete the last capture in this chat |
 | `/refresh` | Rescan the vault index (also runs automatically every 10 minutes) |
 
+### Rolling back a merge or rebuild
+
+Every merge and every `/rebuild` commits the vault to a git repo (auto-initialised at `BASE_DIR` on first use) **before** writing. To undo the most recent rewrite:
+
+```bash
+git -C "$BASE_DIR" log --oneline      # find the engram: pre-* commit
+git -C "$BASE_DIR" reset --hard HEAD~ # discard the last rewrite
+```
+
+> **iCloud note:** if your vault lives in an iCloud-synced folder, the `.git` directory is synced too. This works fine but can cause occasional sync churn; that is the documented trade-off for in-vault rollback.
+
 The plain message path: send a message → tap a folder button → done. Send a photo without a caption and the bot OCRs it first so it can route by content.
 
 ## What a note looks like
@@ -157,11 +176,12 @@ uv sync
 uv run pytest -v
 ```
 
-15 test modules cover the bot handlers, vault indexing, embeddings, dedupe, link enrichment, vision/whisper/youtube/pdf adapters, and the inbox/review flow. `pytest-asyncio` is in `auto` mode. CI runs on push and PR against Python 3.11 / 3.12 / 3.13.
+The test suite covers the bot handlers, vault indexing, embeddings, dedupe, link enrichment, the merge-and-rewrite path, entity extraction/pages, the retroactive `/rebuild` flow, git snapshotting, vision/whisper/youtube/pdf adapters, and the inbox/review flow. `pytest-asyncio` is in `auto` mode. CI runs on push and PR against Python 3.11 / 3.12 / 3.13.
 
 ## Roadmap
 
 - **Local-model support** — swap Claude / OpenAI for Ollama or llama.cpp so the bot can run end-to-end without paid API keys. Embeddings first (cheapest win), then enrichment. Tracked in [#1](https://github.com/mishablank/Engram/issues/1) — help welcome.
+- **Scheduled maintenance** — a nightly pass that re-synthesises entity leads, reconciles contradictions across notes, and DMs a digest of what changed in the vault overnight.
 
 ## Security model
 

src/engram/bot.py

@@ -24,17 +24,24 @@
     filters,
 )
 
-from . import pending_store
-from .config import DEFAULT_CATEGORY, Config, load_config
+from . import gitsafe, pending_store
+from .config import DEFAULT_CATEGORY, ENTITY_TYPE_FOLDERS, Config, load_config
 from .dedupe import find_semantic_duplicate
 from .embeddings import OpenAIEmbedder, SemanticIndex, hybrid_search
+from .entities import extract_entities, rebuild_entity_pages, upsert_entity_page
 from .fetcher import fetch_urls
 from .inbox import clear_pending, find_pending, move_note, preview
 from .linker import answer_from_vault, enrich_note
-from .note_writer import CapturedMessage, append_to_note, extract_urls, write_note
+from .note_writer import (
+    CapturedMessage,
+    append_to_note,
+    extract_urls,
+    merge_into_note,
+    write_note,
+)
 from .pdf import extract_pdf_text
 from .relinker import relink_note
-from .vault import VaultIndex, scan_vault, search_vault
+from .vault import VaultIndex, load_note_body, scan_vault, search_vault
 from .vision import ocr_image
 from .whisper import transcribe
 
@@ -441,8 +448,20 @@ async def _process_messages(
             original_message_id=fwd_msg_id,
             review_pending=review_pending,
         )
-        path = append_to_note(duplicate, captured)
-        return path, f"{path.parent.name} (appended)"
+        # Karpathy-wiki merge: rewrite the canonical note to integrate the new
+        # source instead of appending a dated block. Snapshot first so a bad
+        # rewrite is one `git revert` away; merge_into_note falls back to a safe
+        # append if the LLM rewrite would drop frontmatter or an attachment.
+        gitsafe.snapshot(
+            state.config.base_dir, f"engram: pre-merge {duplicate.stem}"
+        )
+        path, merged = await asyncio.to_thread(
+            merge_into_note, duplicate, captured, state.anthropic
+        )
+        state.invalidate_vault_cache()
+        await asyncio.to_thread(_update_entities, state, path.stem, body_text)
+        verb = "merged" if merged else "appended"
+        return path, f"{path.parent.name} ({verb})"
 
     target_dir = state.config.base_dir / folder
     attachments_dir = target_dir / "attachments"
@@ -467,9 +486,24 @@ async def _process_messages(
     )
     path = write_note(captured, target_dir, related=related, tags=tags)
     state.invalidate_vault_cache()
+    await asyncio.to_thread(_update_entities, state, path.stem, body_text)
     return path, folder
 
 
+def _update_entities(state: BotState, source_title: str, body: str) -> None:
+    """Extract entities from a captured note and grow their typed wiki pages.
+
+    Best-effort: never raises into the capture path.
+    """
+    if not body.strip():
+        return
+    try:
+        for ent in extract_entities(state.anthropic, source_title, body):
+            upsert_entity_page(state.config.base_dir, ent, source_title)
+    except Exception:
+        log.exception("Entity update failed for %s", source_title)
+
+
 def _move_pending_files(srcs: list[Path], target_dir: Path) -> list[str]:
     if not srcs:
         return []
@@ -1228,14 +1262,67 @@ async def on_relink(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
             f"Done. Updated {changed_count}/{len(notes)} note(s)."
         )
 
+    async def on_rebuild(update: Update, context: ContextTypes.DEFAULT_TYPE) -> None:
+        if not _is_authorized(state, update):
+            return
+        base_dir = state.config.base_dir
+        await update.effective_chat.send_message(
+            "Rebuilding vault: snapshotting to git, merging duplicates, "
+            "rebuilding entity pages… this may take a minute."
+        )
+        committed = await asyncio.to_thread(
+            gitsafe.snapshot, base_dir, "engram: pre-rebuild snapshot"
+        )
+
+        merges = 0
+        if state._semantic_index.enabled:
+            await asyncio.to_thread(state._semantic_index.refresh)
+            note_paths = _vault_note_paths(base_dir)
+            merges = await asyncio.to_thread(
+                merge_duplicate_notes,
+                base_dir,
+                state._semantic_index,
+                state.anthropic,
+                note_paths,
+            )
+
+        notes = [
+            (p.stem, load_note_body(p)) for p in _vault_note_paths(base_dir)
+        ]
+        entity_pages = await asyncio.to_thread(
+            rebuild_entity_pages, base_dir, state.anthropic, notes
+        )
+        state.invalidate_vault_cache()
+        await asyncio.to_thread(
+            gitsafe.snapshot, base_dir, "engram: post-rebuild"
+        )
+        snap = "snapshot saved" if committed else "no snapshot (git unavailable or clean)"
+        await update.effective_chat.send_message(
+            f"Rebuild done. Merged {merges} duplicate note(s); "
+            f"wrote {entity_pages} entity page(s). Pre-rebuild {snap}; "
+            "revert with `git -C <vault> reset --hard HEAD~` if it looks wrong."
+        )
+
     return (
         on_message, on_start, on_refresh, on_redo, on_folder_choice,
         on_search, on_ask, on_undo, on_edit,
         on_inbox, on_review, on_review_choice,
-        on_relink,
+        on_relink, on_rebuild,
     )
 
 
+def _vault_note_paths(base_dir: Path) -> list[Path]:
+    """All capture notes, excluding attachments and the entity (wiki) folders."""
+    skip = {"attachments", *ENTITY_TYPE_FOLDERS.values()}
+    out: list[Path] = []
+    for p in base_dir.rglob("*.md"):
+        if any(part in skip for part in p.relative_to(base_dir).parts):
+            continue
+        out.append(p)
+    # Deterministic order so rebuilds and their git diffs are reproducible.
+    return sorted(out)
+
+
 def main() -> None:
     config = load_config()
     log_path = Path(os.environ.get("LOG_FILE", DEFAULT_LOG_PATH)).expanduser()
@@ -1251,7 +1338,7 @@ def main() -> None:
         on_message, on_start, on_refresh, on_redo, on_folder_choice,
         on_search, on_ask, on_undo, on_edit,
         on_inbox, on_review, on_review_choice,
-        on_relink,
+        on_relink, on_rebuild,
     ) = make_handlers(state)
 
     app.add_handler(CommandHandler("start", on_start))
@@ -1264,6 +1351,7 @@ def main() -> None:
     app.add_handler(CommandHandler("inbox", on_inbox))
     app.add_handler(CommandHandler("review", on_review))
     app.add_handler(CommandHandler("relink", on_relink))
+    app.add_handler(CommandHandler("rebuild", on_rebuild))
     app.add_handler(CallbackQueryHandler(on_folder_choice, pattern=r"^f\|"))
     app.add_handler(CallbackQueryHandler(on_review_choice, pattern=r"^r\|"))
     app.add_handler(

src/engram/config.py

@@ -20,9 +20,18 @@
 SUMMARY_MODEL = "claude-sonnet-4-6"
 REDO_MODEL = "claude-opus-4-7"
 VISION_MODEL = "claude-haiku-4-5"
+MERGE_MODEL = "claude-sonnet-4-6"
+ENTITY_MODEL = "claude-sonnet-4-6"
 EMBEDDING_MODEL = "text-embedding-3-small"
 EMBEDDING_DIM = 1536
 
+# Entity (wiki backbone) pages live in typed top-level folders.
+ENTITY_TYPE_FOLDERS: dict[str, str] = {
+    "person": "People",
+    "concept": "Concepts",
+    "project": "Projects",
+}
+
 
 @dataclass
 class Config: