Feat/transcription platform

README.md

@@ -1,12 +1,17 @@
-# Unseen
+<p align="center">
+  <img src="banner.png" alt="Unseen — your photographic memory" width="100%" />
+</p>
 
-**An open-source, on-screen AI copilot for live conversations.** Unseen listens to your meeting, transcribes it in real time, and surfaces answers, notes, and talking points in a floating panel — one that (optionally) never shows up in your screen share.
+# Unseen — your photographic memory
+
+**Unseen is the part of your memory that never forgets.** An open-source, local-first transcription engine with a memory that compounds: dictate into any app, capture every meeting, ingest your notes — and it quietly distills all of it into a private, structured knowledge base that you *and* your AI can draw on.
 
 Bring your own AI: **Anthropic Claude, OpenAI, Google Gemini, Ollama (fully local), or any OpenAI-compatible endpoint** (LM Studio, Groq, OpenRouter, vLLM, …).
 
-> The copilot only you can see — your notes, answers, and prompts stay out of your screen share and on your machine.
+> Everything stays on your machine. Talk to it all day; forget nothing.
+
 <table> <tr> <td width="110" valign="top">
-<img width="98" height="98" alt="Screenshot 2026-06-07 at 4 11 50 PM" src="https://github.com/user-attachments/assets/58c6a45f-7ce8-4c5c-8cdb-3f77603d855a" />
+<img width="98" height="98" alt="Screenshot 2026-06-07 at 4 11 50 PM" src="https://github.com/user-attachments/assets/58c6a45f-7ce8-4c5c-8cdb-3f77603d855a" />
 </td> <td valign="top">
 Built with ❤️ using Repowise
 
@@ -17,17 +22,33 @@ Beyond documentation, Repowise continuously monitors **code health**, highlighti
 ⭐ If you're building AI-powered software, check out Repowise: https://github.com/repowise-dev/repowise
 </td> </tr> </table>
 
-## What it does
+## Three surfaces, one memory
+
+Unseen is one STT + LLM + storage core wearing three hats:
+
+1. **🎙 Dictation** — tap a hotkey in *any* app, talk into *any* text field, get your words inserted at the cursor. Optionally, a fast LLM pass strips fillers, false starts, and stray punctuation first.
+2. **💬 Meeting copilot** — the floating, capture-invisible overlay that transcribes live and surfaces answers, notes, and talking points on demand.
+3. **🧠 Memory** — every dictation, meeting, and note feeds a per-day log that a distillation job turns into an accumulating, structured knowledge base — split into **personal** and **work**, combinable on demand. That memory is injected back into your prompts, so the more you use Unseen, the more context your AI has.
+
+```
+dictation ┐
+meetings  ├─→ daily log (append-only JSONL) ─→ distill ─→ personal / work facts
+notes     ┘                                                      │
+                                                                 ▼
+                                              injected as context into your prompts
+```
+
+## How the copilot works
 
 ```
 mic → streaming STT (Deepgram nova-3, diarized)
     → rolling transcript with speaker turns
     → trigger engine (questions, requests, code asks, keywords — per profile)
-    → your chosen LLM (streaming, prompt-cached)
+    → your chosen LLM (streaming, prompt-cached, with your distilled memory)
     → live answer feed in a floating, capture-invisible overlay
 ```
 
-Behavior is driven by **profiles** — plain YAML files that define the prompt, when the copilot speaks up, and how it answers:
+Behavior is driven by **profiles** — plain YAML files that define the prompt, when the copilot speaks up, how it answers, and which memory namespaces it can see:
 
 | Profile | What it does |
 |---|---|
@@ -41,6 +62,31 @@ Behavior is driven by **profiles** — plain YAML files that define the prompt,
 
 Adding your own use case = writing one YAML file. No code.
 
+## Dictation
+
+Press **⌃Space** (Control+Space) in any app to start, talk, and press it again to stop. A small HUD shows your words live; on stop the text is pasted at your cursor — clipboard contents restored afterward. If a paste ever fails, the text is left on your clipboard so you can paste it manually.
+
+- **Optional AI cleanup** (Settings → Memory): off = your words inserted verbatim, instantly; on = a fast model tidies fillers/punctuation first (streamed into the HUD).
+- Works in any text field (uses clipboard-paste, not fragile keystroke synthesis).
+- macOS needs **Accessibility** permission once (the first-run wizard walks you through it).
+- Per-app exclude list: name apps where dictation should never paste or log.
+- The hotkey is remappable in Settings.
+
+## Memory that writes itself
+
+Everything transcribed accrues into a private knowledge base — no filing, no manual notes:
+
+- **Daily log** — dictations, meeting finals, and ingested notes are appended to `memory/log/<date>.jsonl`.
+- **Distillation** — on demand ("Distill now" in Settings → Memory) or on a schedule, an LLM turns the day's log into deduped, structured facts (people, projects, decisions, preferences, recurring terms). Re-running never duplicates.
+- **Namespaces** — facts are split into **personal** and **work**; a profile picks which to inject (`memory.namespaces: [work]`, or `[personal, work]` to combine).
+- **Watched sources** — point a namespace at markdown files or a folder (work docs, an Obsidian vault) and they're read straight in.
+- **Apple Notes** *(macOS)* — typed notes are ingested via the official scripting API; handwriting is read from Apple's pre-rendered images and OCR'd locally and offline (swappable engine; see [docs/ocr-sidecar.md](docs/ocr-sidecar.md)).
+- **Background sync** *(macOS)* — Unseen can install its own LaunchAgent to run ingestion + distillation on a schedule, even when the window is closed. No manual setup.
+
+## Local-first, with optional iCloud sync
+
+Your data lives on your machine by default. Flip Settings → Memory → **Store data in iCloud** and Unseen moves `memory/`, `knowledge/`, and `sessions/` into your iCloud Drive — so a second Mac sees the same logs and memory after cloning. Append-only logs merge cleanly across devices and a per-day lock keeps two Macs from distilling the same day twice. API keys and settings always stay local. The on-disk format is frozen — see [docs/data-layout.md](docs/data-layout.md).
+
 ## Install
 
 **Download an installer** from [Releases](https://github.com/repowise-dev/unseen/releases) — dmg (macOS), exe (Windows), AppImage/deb (Linux).
@@ -62,11 +108,13 @@ First run: open Settings (⚙ in the overlay) → Providers → pick your LLM, p
 - a **Deepgram** API key for transcription (generous free tier), and
 - an LLM: an **Anthropic / OpenAI / Gemini** key — or **no key at all** with [Ollama](https://ollama.com) running locally.
 
+On macOS, dictation also asks for **Accessibility** permission so it can paste for you.
 
 ## Hotkeys
 
 | Default | Action |
 |---|---|
+| ⌃Space | start / stop dictation (talk into any app) |
 | ⌘⇧\\ | show / hide the overlay |
 | ⌘⇧Space | answer the latest thing said ("Ask now") |
 | ⌘⇧P | pause / resume listening |
@@ -81,8 +129,10 @@ The overlay is excluded from OS-level screen capture (`NSWindow.sharingType = .n
 
 ## Privacy & data
 
-- Transcripts and answers stay **on your machine**. No telemetry, ever.
-- API keys are stored **encrypted in your OS keychain** (Electron `safeStorage`).
+- Transcripts, answers, and your distilled memory stay **on your machine** (or your own iCloud). No telemetry, ever.
+- API keys are stored **encrypted in your OS keychain** (Electron `safeStorage`) and **never** sync to iCloud.
+- Handwriting OCR runs **fully offline** — no cloud, no Claude OCR.
+- Pause and a per-app exclude list let you keep specific apps and moments out of memory entirely.
 - Recording-consent laws vary by jurisdiction — **get consent before transcribing other people**. See [docs/privacy-and-consent.md](docs/privacy-and-consent.md).
 
 ## Extending
@@ -93,6 +143,7 @@ The overlay is excluded from OS-level screen capture (`NSWindow.sharingType = .n
 | An LLM vendor | one `LlmProvider` implementation | [docs/extending/llm-provider.md](docs/extending/llm-provider.md) |
 | An STT vendor | one `SttProvider` + one parser | [docs/extending/stt-provider.md](docs/extending/stt-provider.md) |
 | A trigger detector | one pure function | [docs/extending/detector.md](docs/extending/detector.md) |
+| An OCR engine | one stdin/path → `{ text }` command | [docs/ocr-sidecar.md](docs/ocr-sidecar.md) |
 
 Architecture overview: [docs/architecture.md](docs/architecture.md).
 

docs/data-layout.md

@@ -0,0 +1,51 @@
+# On-disk data layout (frozen contract)
+
+This layout is **frozen** as of Phase 4 so the future native iOS client (Phase
+5) can read/write the same files without chasing a moving target.
+
+```
+<dataDir>/
+  memory/
+    log/<YYYY-MM-DD>.jsonl          # append-only daily event log
+    facts/personal/<YYYY-MM-DD>.json
+    facts/work/<YYYY-MM-DD>.json
+    facts/<ns>/<YYYY-MM-DD>.lock    # transient single-distiller lock
+  knowledge/                       # hand-imported + pinned markdown
+  sessions/                        # meeting JSONL (existing)
+```
+
+`<dataDir>` is configurable (Settings → Memory → Data location):
+
+- **Local (default):** the OS userData dir
+  (`~/Library/Application Support/Unseen` on macOS).
+- **iCloud:** `~/Library/Mobile Documents/com~apple~CloudDocs/Unseen/`. A second
+  Mac that signs into the same iCloud sees the same data after sync.
+
+## What does NOT sync (stays in local userData)
+
+- `settings.json` — read at startup to learn `dataDir`, so it can't itself live
+  under `dataDir` (chicken-and-egg). Each device keeps its own settings.
+- `secrets.json` — API keys, keychain-encrypted. **Never** leaves the device.
+
+> This is a deliberate, documented deviation from the roadmap's §7.3 sketch
+> (which placed `settings.json` in the synced dir): the bootstrap ordering and
+> the secrets-stay-local rule make local settings the correct call.
+
+## Log event shape
+
+```jsonc
+{ "t": 1750000000000, "kind": "dictation|meeting|note",
+  "ns": "personal|work", "app": "Slack", "text": "...", "sessionId": "..." }
+```
+
+## Sync mechanics
+
+- **Append-only JSONL** → two devices appending the same day rarely truly
+  conflict. The merge is `sort by t` + `dedupe by (t, kind, text)`
+  (`mergeLogEvents`), applied on every read.
+- **Facts** are produced by an idempotent merge (`mergeFacts`, stable key), so
+  re-distilling a day — or distilling on either device — never duplicates.
+- **Single-distiller lock:** `facts/<ns>/<day>.lock` carries a device id +
+  timestamp. A peer skips a day another device is actively distilling; a stale
+  lock (>1h, e.g. a crashed run) is reclaimed automatically.
+- Consistency is **eventual**, not real-time — iCloud Drive does the syncing.

docs/ocr-sidecar.md

@@ -0,0 +1,53 @@
+# OCR sidecar (handwriting → text)
+
+Apple Notes handwriting is captured by reading Apple's **pre-rendered drawing
+PNGs** from the Notes group container's `Media/` folders (we never decode the
+gzipped protobuf — decision #6). Those PNGs are turned into text by a **vendored,
+fully-offline OSS OCR process** — no cloud, no Claude OCR (decision #7).
+
+## Contract
+
+The engine is swappable without touching any caller. The contract is:
+
+```
+<command> <png-path>   →   stdout: JSON { "text": "..." }   (plain text also accepted)
+```
+
+`src/main/services/notes/ocr-sidecar.ts` resolves the command and shells out to
+it. When no engine is configured, `ocrImage()` returns `""` and Notes ingestion
+simply skips handwriting (typed notes still ingest) — the count of skipped PNGs
+is surfaced in Settings → Memory.
+
+## Configuring an engine
+
+Today the command is resolved from the `UNSEEN_OCR_CMD` environment variable:
+
+```bash
+export UNSEEN_OCR_CMD=/path/to/unseen-ocr     # called as: unseen-ocr <png>
+```
+
+A reference engine should:
+
+- Default to **PaddleOCR-VL-1.5** (runs on Apple Metal).
+- Fall back to **TrOCR-large-handwritten** for messy handwriting.
+- Print `{"text": "..."}` to stdout and exit 0.
+
+## Vendoring (packaging)
+
+The runtime + model weights are **hundreds of MB** and are intentionally **not**
+committed to this repo. At package time they are bundled as an
+`extraResources` entry (alongside `profiles/`) and `ocr-sidecar.ts` is extended
+to prefer that bundled path over `UNSEEN_OCR_CMD`. Document the exact footprint
+in the release notes when the engine is added.
+
+> Footprint note: budget ~300–700 MB for the PaddleOCR-VL weights + a minimal
+> Python/Metal runtime, or a self-contained compiled binary.
+
+## Why this shape
+
+- **Offline + update-proof:** depends on nothing in macOS, so an OS update can't
+  break it.
+- **Swappable:** Paddle ↔ TrOCR (or anything else) is a command change, no code
+  change in callers.
+- **Tunable:** it's one writer (the user), so the engine can be tuned to their
+  handwriting.