feat(vector): add contextual chat and transcript embeddings

[salmonumbrella]

What changed

• Add a voyage-contextual embedding API format for Voyage Context 4 while keeping the existing OpenAI-compatible path as the default.
• Embed chats as gap-, day-, and size-bounded windows, and embed meeting notes and speaker turns as contextual documents with exact source-span ownership.
• Add durable change journals, document publication ledgers, and atomic publication/CAS behavior for SQLite and PostgreSQL so edits, moves, deletes, retries, and crashes converge safely.
• Wire contextual rebuild, incremental scheduling, activation, hybrid search excerpts, and a synthetic retrieval evaluator through the production paths.

Why

The current embedding path treats each message or chunk in isolation. That removes the conversation or speaker context needed to retrieve short and ambiguous answers. This adds grouped contextual embeddings without changing the legacy default. On the fixed 20k evaluation, retrieval quality and the ANN, latency, memory, index-size, token, error, and append-amplification gates passed. Full rebuild time was 2.28x the current path against the 1.5x target, so rebuild performance remains an explicit pre-merge review boundary.

Usage

Set VOYAGE_API_KEY, then configure:

[vector]
enabled = true
backend = "sqlite-vec"

[vector.embeddings]
api_format = "voyage-contextual"
endpoint = "https://api.voyageai.com/v1"
api_key_env = "VOYAGE_API_KEY"
model = "voyage-context-4"
dimension = 1024

api_format = "voyage-contextual" selects the contextual document endpoint. If api_format is omitted, msgvault keeps using the regular OpenAI-compatible embeddings path.

Build and atomically activate a new generation:

msgvault embeddings build --full-rebuild --yes

Refs #550

[salmonumbrella]

@wesm I am not sure about this one. Goal is better retrieval of chat messages—needs testing to see if it's worth.

cc: @danshapiro @jesserobbins @mariusvniekerk would love your input. This Voyage blog post served as inspiration.

[roborev-ci]

roborev: Combined Review (93fefbd)

Verdict: Changes requested—one high-severity scope bypass and two medium-severity journal consistency/performance issues.

High

• Contextual embeddings bypass the configured message-type scope
  Locations: cmd/msgvault/cmd/serve_vector.go:156, internal/vector/embed/context_worker.go:234
ContextWorker is constructed without vectorCfg.Embed.Scope.BuildScope() and performs unscoped discovery, journal processing, and reconciliation. Excluded emails, chats, or transcripts may therefore be indexed and sent to the external Voyage endpoint, increasing exposure and storage/API costs.
  Fix: Pass the configured BuildScope into ContextWorker and enforce it throughout discovery, journal processing, and reconciliation. Records moving out of scope should have existing vectors removed without submitting their content. Add a behavioral test confirming excluded message bodies never reach the embedding client.

Medium

• Journal triggers do not track all timestamps used for contextual grouping
  Locations: internal/store/dialect_sqlite.go:639, internal/store/dialect_pg.go:816
  Documents group messages using COALESCE(sent_at, received_at, internal_date), while journal triggers detect only sent_at changes. Messages without sent_at can remain embedded in an obsolete day or window when fallback timestamps change or records move.
  Fix: Trigger on received_at and internal_date changes and journal both old and new canonical timestamps using the same COALESCE expression as document assembly.

• New generations replay the entire append-only journal before reconciliation
  Location: internal/vector/embed/context_worker.go:200
  Starting from sequence zero causes fresh generations to repeatedly assemble and embed historical versions of scopes. Because entries are never pruned and processing occurs in small batches, rebuild costs can grow indefinitely with import history.
  Fix: Start new generations at a pinned current source sequence and reconcile that snapshot before processing later changes, or compact the journal safely using retained-generation watermarks.

---

Reviewers: 2 done | Synthesis: codex, 16s | Total: 17m17s

[roborev-ci]

roborev: Combined Review (79f010e)

Code has two medium-severity correctness issues that may leave contextual vectors stale.

Medium

• Stale vector snapshots can overwrite newer publications — internal/vector/pgvector/documents.go:80, internal/vector/sqlitevec/documents.go:88

  Publication is serialized but does not reject stale SourceSequence values. Concurrent workers may publish sequences 10 and 20 in reverse completion order, leaving sequence-10 vectors installed while the watermark remains at 20.

  Fix: Persist the latest applied sequence per scope and atomically ignore older publications, including unchanged and empty-scope processing.

• Fallback participant-name changes are not journaled — internal/store/dialect_pg.go:998, internal/store/dialect_sqlite.go:884

  Journaling observes only display_name, while chat assembly renders COALESCE(display_name, email_address, phone_number). Changes to a fallback email address or phone number can therefore alter embedding input without generating a journal event.

  Fix: Monitor all three rendered-name fields and enqueue affected conversations whenever their effective display value changes.

---

Reviewers: 2 done | Synthesis: codex, 24s | Total: 12m44s

[roborev-ci]

roborev: Combined Review (20a9d88)

The review found two medium-severity reliability issues in contextual embedding updates; no security vulnerabilities were identified.

Medium

• Fallback participant changes can leave embeddings stale — internal/store/dialect_sqlite.go:884, internal/store/dialect_pg.go:998
  Contextual documents fall back to participant email or phone when no display name exists, but change-journal triggers watch only display_name. Updating a fallback email or phone can change rendered text without scheduling re-embedding.
  Fix: Journal changes to display_name, email_address, and phone_number, ideally by comparing the effective rendered participant label.

• Metadata fan-out is unbounded and may prevent cursor progress — internal/vector/embed/context_worker.go:899
  A single metadata event expands across every day of every affected conversation, accumulating documents, inputs, embeddings, and publications before one transaction. Large archives may exhaust memory or repeatedly time out without advancing the journal cursor.
  Fix: Process metadata fan-out in bounded, resumable pages and cap the scopes handled by each publication transaction.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 12m52s

[roborev-ci]

roborev: Combined Review (f251e85)

The change is functionally substantial and security-clean, but has three Medium-severity correctness and performance issues to address.

Medium

• Repeated reconciliation of chat-day scopes — internal/vector/embed/context_worker.go:737
  Reconciliation paginates by message ID but deduplicates chat-day scopes only within each page. A chat day spanning multiple pages is fully reassembled and coverage-stamped once per page, causing quadratic reads and repeated writes during builds and backstops.
  Fix: Enumerate or persist unique scope keys across the entire reconciliation pass so each chat-day scope is processed once.

• One oversized document suppresses valid sibling documents — internal/vector/embed/context_worker.go:471, internal/vector/embed/context_worker.go:496
  An oversized document marks its entire scope as failed, so publication contains no documents or chunks. In a chat-day scope with several sessions, one rejected session tombstones valid sibling sessions and stamps all members as covered, allowing an incomplete generation to converge.
  Fix: Track failures per document, publish successful siblings, and quarantine only the oversized document and its members.

• Idle contextual runs repeatedly scan the full messages table — internal/vector/embed/context_worker.go:281, internal/vector/embed/context_worker.go:828
  Every contextual RunOnce begins missing-message discovery at ID zero, and convergence repeats the scan. Because idx_messages_embed_gen is removed, even an idle scheduled run traverses the complete messages table twice.
  Fix: Persist a normal-pass watermark as the ordinary worker does, reserving zero-based scans for RunBackstop, or add an index suitable for the contextual discovery query.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 13m43s

[roborev-ci]

roborev: Combined Review (956c358)

One high-severity and two medium-severity issues require attention.

High

• Remote chat metadata changes can trigger unbounded paid re-embedding — internal/vector/embed/context_worker.go:953

  A conversation_title or membership change invalidates every historical day of the conversation because each document revision includes the title and participants. A remote group administrator could repeatedly change this metadata, causing the entire conversation history to be uploaded to Voyage again. Batch request limits do not cap total work, creating potentially substantial API costs and prolonged embedding jobs.

  Fix: Apply persistent per-run token/byte budgets and rate limits, or exclude mutable remote metadata from historical document revisions. Make archive-wide metadata re-indexing explicit or process it through a capped, resumable queue.

Medium

• Ordinary message changes may remain unembedded — internal/vector/embed/context_worker.go:152

  After reconciliation reaches done:, normal runs skip ordinary discovery. Since ordinary inserts and body edits are not added to the contextual mutation journal, they remain unembedded until the periodic backstop runs—or indefinitely when backstop discovery is disabled.

  Fix: Run incremental discovery for ordinary messages on every execution, or journal every mutation that invalidates embeddings. Test that changing an ordinary message after initial convergence causes the next normal run to re-embed it.

• Oversized documents are incorrectly counted as embedded — internal/vector/embed/context_worker.go:494

  Documents rejected for excessive size are not published as vectors, but their source messages still receive the current embed_gen. This permits convergence and activation despite missing vectors and prevents future retries.

  Fix: Stamp only successfully published documents. If oversized documents require permanent quarantine, track that state explicitly and report it separately from successful embedding coverage.

---

Reviewers: 2 done | Synthesis: codex, 17s | Total: 14m3s

[roborev-ci]

roborev: Combined Review (674b322)

One medium-severity convergence issue should be fixed before merge.

Medium

• internal/vector/embed/context_worker.go:617 — Journal processing does not stamp a live message when its updated content produces no document. For example, editing an indexed body to empty resets embed_gen; the empty scope is published, but the early return skips coverage before the journal watermark advances. Because reconciliation remains done:, normal later runs skip discovery, leaving the generation non-converged until a backstop runs.
  • Fix: Preserve live source versions for scope members that yield no document and CAS-stamp them as terminal blanks before advancing the journal or reconciliation cursor. Add coverage for a nonempty-to-empty update after reconciliation, including contextual message types.

---

Reviewers: 2 done | Synthesis: codex, 10s | Total: 16m43s

[roborev-ci]

roborev: Combined Review (17a971e)

Three medium-severity reliability issues could stall indexing, permit stale generation activation, or cause unbounded fanout work.

Medium

• Permanently rejected documents block indexing progress — internal/vector/embed/context_worker.go:179
  An oversized document returns errContextDocumentRejected without advancing the journal watermark or discovery position. Subsequent runs repeatedly retry the same batch, preventing later messages and journal events from being indexed.
  Fix: Leave the rejected scope uncovered while advancing durable progress, and retry quarantined documents separately during backstop processing.

• Context mutations can race with generation activation — internal/scheduler/embed_job.go:231
  Contextual convergence is checked before activation, but PostgreSQL activation only rechecks embed_gen. A title, membership, or participant-name mutation committed between those steps can advance the contextual journal without clearing embed_gen, allowing a stale generation to activate and retire the valid generation.
  Fix: Add the expected journal sequence and completed reconciliation state to the transactional activation predicate, or serialize convergence checking with activation.

• Large metadata fanouts repeatedly restart from the beginning — internal/vector/embed/context_worker.go:313
  Fanout is eagerly expanded across all affected conversation/day scopes, while its watermark advances only after completion. If the byte budget interrupts processing, the next run recreates and reprocesses the entire fanout, risking unbounded memory usage and quadratic database work.
  Fix: Keyset-page fanout expansion and persist a subcursor for the current journal event so later runs resume from the next scope.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 22m26s

[roborev-ci]

roborev: Combined Review (2b14829)

One high-severity activation regression and one medium-severity stale-publication race must be fixed before merge.

High

• OpenAI-format builds cannot activate
  • Location: cmd/msgvault/cmd/embed_vector.go:183, internal/scheduler/embed_job.go:267
  • Problem: Completed OpenAI-format builds now call ActivateGenerationIfConverged, but the OpenAI worker never creates embedding_document_progress. Real backends therefore reject activation due to missing contextual progress: CLI builds fail to activate, while daemon builds retry indefinitely.
  • Fix: Use ActivateGeneration(..., false) for OpenAI builds. Reserve sequence-bound activation for contextual builds.

Medium

• Unchanged revisions leave the source-sequence fence stale
  • Location: internal/vector/embed/context_worker.go:556
  • Problem: Skipping publication for unchanged document revisions does not advance the scope’s source-sequence fence. A delayed worker holding an older snapshot can then publish stale contextual vectors even though the journal watermark has advanced beyond the newer snapshot.
  • Fix: Atomically advance each unchanged scope’s source sequence without replacing its vectors, and add a behavioral test covering delayed publication interleaved with a reverted source.

---

Reviewers: 2 done | Synthesis: codex, 12s | Total: 22m8s

[roborev-ci]

roborev: Combined Review (6272dd3)

Code is not ready to merge due to a regression-gate correctness issue.

Medium

• scripts/contextual-retrieval-eval/bootstrap.go:495 — safeRatio returns 0 when the baseline is zero and the candidate is positive. Since regression gates accept low ratios, an unmeasurable zero baseline—such as a sub-millisecond build rounded to zero—can make a slower candidate appear optimal and incorrectly pass. Treat a positive candidate over a zero baseline as unavailable or worst-case, and add a behavioral test confirming the associated gate cannot pass.

---

Reviewers: 2 done | Synthesis: codex, 11s | Total: 25m16s

[roborev-ci]

roborev: Combined Review (c573309)

Code review found three medium-severity issues; no critical, high-severity, or security findings were identified.

Medium

• Concurrent mutation can make a newly activated generation stale — internal/vector/pgvector/backend.go:325
  Sequence-bound activation locks the journal clock, but most PostgreSQL journal triggers acquire that lock only in AFTER triggers. A concurrent update can modify an uncommitted source row, block while appending its journal entry, and remain invisible to activation’s coverage check, allowing activation to commit first.
  Fix: Acquire the same lock before source-row mutations, such as through BEFORE triggers or a shared transaction-level advisory lock.

• Large full rebuilds can exit successfully before completion — cmd/msgvault/cmd/embed_vector.go:137
  The CLI invokes the contextual worker only once, while the worker silently stops successfully after processing 1 MB of input (internal/vector/embed/context_worker.go:29). A large embeddings build --full-rebuild may therefore finish without embedding the entire corpus or activating the generation.
  Fix: Remove the per-tick budget for direct CLI builds, or repeatedly invoke the worker until durable convergence or a non-progress error occurs. Retain the bounded budget for scheduler ticks.

• Journal processing ignores the configured batch size — internal/vector/embed/context_worker.go:305
  Journal draining fetches only one event despite ChangeBatchSize, causing independent ordinary messages to be assembled and embedded in separate provider requests and database transactions.
  Fix: Batch consecutive non-fanout events, deduplicate their scopes, publish them together, and advance the watermark only after the entire batch succeeds. Keep metadata-fanout events individually resumable.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 19m16s

[salmonumbrella]

@wesm all you. this one needs some thought.

[roborev-ci]

roborev: Combined Review (13e1ad9)

One-line verdict: Three medium-severity issues could cause stale embeddings, excessive embedding costs, and denial-of-service through metadata fanout.

Medium

• Unchanged sibling documents are re-embedded after cache loss — internal/vector/embed/context_worker.go:573
  Reuse depends on a bounded in-memory vector cache. After restart or eviction, changing one chat window re-embeds every unchanged window in that day’s scope, substantially increasing latency and provider costs. Preserve unchanged documents during publication or reload persisted vectors, and test tail updates after restart/cache eviction.

• Bodyless ordinary-message changes can leave embeddings missing or stale — internal/store/dialect_sqlite.go:642, internal/store/dialect_pg.go:826, internal/vector/embed/context_worker.go:1337
  Messages may be embedded from their subject without a message_bodies row, while journaling and reconciliation require that row. Subject-only inserts, edits, deletions, and type transitions can therefore remain unsynchronized indefinitely when backstops are disabled. Journal ordinary-message lifecycle and subject changes independently of body existence, and include bodyless ordinary messages in reconciliation.

• Metadata changes cause unbounded full-history fanout — internal/vector/embed/context_worker.go:322, internal/store/dialect_sqlite.go:816, internal/store/dialect_pg.go:967
  Each title or membership event expands across every historical day in a conversation. Complete membership replacement also emits delete-and-insert events for every participant—even when unchanged—and journal rows are never pruned. This enables repeated O(participants × conversation history) work, persistent journal growth, indexing delays, and potentially full-history API usage. Apply membership snapshots as diffs, coalesce events before fanout, budget scope assembly, and compact consumed journal rows.

---

Reviewers: 2 done | Synthesis: codex, 13s | Total: 23m58s

[roborev-ci]

roborev: Combined Review (5d3c24c)

Contextual retrieval has four medium-severity correctness and resource-exhaustion issues.

Medium

• Raw timestamp comparisons can omit messages from contextual documents — internal/vector/embed/assemble.go:195
  SQLite compares timestamp text against UTC-formatted bounds. Timestamps containing non-UTC offsets may be excluded or misordered when crossing a UTC day boundary. Normalize timestamps in range and ordering expressions or when persisted, and add coverage for offset timestamps crossing UTC midnight.

• N-c4 evaluation may score a partially built index — scripts/contextual-retrieval-eval/arms.go:1141
  The evaluation calls ContextWorker.RunOnce only once, despite the worker’s 1 MB per-run limit. The default 20,000-distractor corpus exceeds that limit. Run until Contextual.Converged, fail if a pass makes no progress, and verify expected source coverage before scoring.

• Batch boundaries can drop chunks in singleton-arm indexes — scripts/contextual-retrieval-eval/arms.go:1241
  Documents are batched independently even when they contain chunks belonging to the same message. A later Backend.Upsert can delete chunks inserted by an earlier batch, leaving incomplete O-c4/S-c4 indexes. Keep all chunks for a message in one upsert or use chunk-preserving insertion; test a two-chunk message with batch size one.

• Membership journal events amplify into repeated full-history assembly — internal/store/dialect_sqlite.go:816, internal/store/dialect_pg.go:1044, internal/vector/embed/context_worker.go:321
  Complete syncs delete and reinsert every participant, producing per-row journal events that each fan out across every historical conversation day. Even unchanged documents incur assembly work without consuming the embedding-byte budget, allowing participant churn to cause sustained database/CPU load, journal growth, and failure to converge. Reconcile memberships by set difference, emit at most one metadata event per conversation transaction, coalesce same-conversation events before fan-out, and budget source rows or assembled bytes regardless of provider calls.

---

Reviewers: 2 done | Synthesis: codex, 15s | Total: 23m20s