feat: enhancement wave — eight 2026-07-26 specs, Jugalbandi-resolved

CHANGELOG.md

@@ -7,8 +7,113 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Changed
+
+- **MCP 2026-07-28.** The server speaks the final "stateless MCP" revision on
+  the v2 SDK line (`@modelcontextprotocol/server` 2.0). `daftari serve` is
+  stateless per the spec's Decision 1: the initialize handshake, the
+  `Mcp-Session-Id` header, and the session table are gone — identity is
+  resolved from the bearer on every request against the same config-declared
+  map, and 2025-era traffic is refused (no dual-stacking). stdio serves both
+  eras from one factory, so lagging clients use stdio; the RBAC,
+  existence-disclosure, and process-lock invariants carry over unchanged.
+  Design record:
+  `docs/superpowers/specs/2026-07-26-mcp-2026-07-28-readiness-design.md`.
+- `vault_ratify` called without a `decision` now answers with a stateless
+  form-mode elicitation (`input_required`): approve/reject with reject
+  preselected, plus HMAC-signed opaque state carrying the action id, the
+  vault HEAD at proposal time, and the deciding user. A declined form applies
+  nothing and leaves the action pending; a direct call with the decision
+  inline keeps working — including a batch `ids` call, which always requires
+  an explicit decision. The server proposes, the human disposes — now on the
+  wire itself (Decision 5).
+- The maintenance passes (`sleep`, `consolidate`, `audit`, `eval`) remain
+  CLI-only: the spec's Decision 4 kill condition fired — the final revision
+  moved Tasks to a standalone extension (removing `tasks/list`) and the
+  TypeScript SDK ships no tasks runtime yet.
+
 ### Added
 
+- **Contextual chunking + optional local reranker.** `chunkDocument`
+  (replacing `chunkText`) splits document bodies at ATX headings (never
+  packing across a heading boundary) and gives every chunk a one-line
+  breadcrumb context (`{collection} › {title} › {headings} · tags: a, b, c`)
+  that is hashed and embedded together with the chunk's body text, and
+  stored as a second `chunks_fts` column — contextual embeddings and
+  contextual BM25 from one string-prefix change, no LLM call. Displayed
+  snippets are read from the body column only; the synthesized breadcrumb
+  never appears in served content. Schema bump `SCHEMA_VERSION` 10 → 11:
+  every chunk's hash input changed, so the first post-upgrade reindex is a
+  one-time full re-embed of the whole corpus (~25 min local-minilm / ~2 min
+  ~$0.10 openai-3-small for the 44k-chunk reference vault). Retitling a
+  document, moving it between collections, or changing its tag *set*
+  re-embeds all of that document's chunks (the breadcrumb is part of the
+  hash); tag *reorder* is a no-op (tags are sorted before hashing).
+  Also lands an optional local cross-encoder reranker: `rerank.provider:
+  local-bge-m3` (default `none`) reorders the top-50 RBAC-filtered
+  `vault_search` hits with a ~600MB ONNX q8 model
+  (`onnx-community/bge-reranker-v2-m3-ONNX` via `@huggingface/transformers`,
+  already a dependency — zero new deps), between the RBAC filter and the
+  slice to `limit`. A 1.5s per-search timeout, a not-warm skip that fires a
+  background warm instead of blocking the call, and a provider `Result.err`
+  all degrade to the fused order identically. `vault_search`'s result gains
+  `rerankUsed: boolean`, the honest twin of `vectorUsed`. Ships opt-in; the
+  default flip to `local-bge-m3` is gated on a post-merge recall
+  measurement, same playbook as chunk-level BM25's v1.29.0 default flip.
+  Design record:
+  `docs/superpowers/specs/2026-07-26-contextual-chunking-reranker-design.md`.
+
+- **Two new local embedding providers + int8 vec-index quantization
+  (opt-in; default embedder unchanged).** `embeddings.provider` gains
+  `local-embeddinggemma` (`google/embeddinggemma-300m`, 768d native,
+  Matryoshka-truncatable to 512/768) and `local-qwen3-0.6b`
+  (`Qwen/Qwen3-Embedding-0.6B`, up to 768d exposed), both fully local via
+  the existing `@huggingface/transformers` dependency, both with
+  asymmetric document/query prompt prefixing. The durable `embeddings`
+  cache now stores the full NATIVE-dim vector under a dim-free cache id
+  (e.g. `local-embeddinggemma#p1`); a new `embeddings.dim` config key picks
+  the INDEX-time truncation. A new `embeddings.quantize: int8 | none`
+  config key (default `int8` for the two new providers, `none` — today's
+  exact behavior — for `local-minilm`/`openai-3-small`) stores the
+  sqlite-vec mirror as `int8[dim]` with scan-then-rescore: candidates are
+  selected by quantized distance, then rescored with exact float32 cosine
+  against the durable cache, so quantization never becomes the reported
+  score. Switching `provider`, `dim`, or `quantize` between server runs is
+  a config change plus a background reindex; a `dim`/`quantize` flip alone
+  needs no re-embed (`isIndexFresh` detects the change and the reindex is
+  all cache hits — a vec-mirror rebuild, not a cold re-embed). `local-
+  minilm` remains the default for `loadConfig`'s programmatic fallback;
+  the vault-init template default flip and any measured cold-reindex /
+  query-latency / RSS numbers are gated on the governing spec's Phase 0
+  smoke spike and Phase 5 recall-bench, neither of which has run against a
+  real model download as of this entry — see `docs/architecture.md`'s
+  "Vec-index quantization" section and the verification-honesty notes on
+  each new provider file.
+  Design record:
+  `docs/superpowers/specs/2026-07-26-embedding-refresh-quantization-design.md`.
+
+- **MCP `content`-channel summaries for every remaining tool.** Every
+  registered tool now has a `summarize` (compact, model-facing text) and,
+  where it names documents, a `docLinks` (`resource_link` entries) — closing
+  the gap Decision 3 (#302) left on everything but search and lint.
+  `vault_read`'s body now rides the wire exactly once: `content[0].text`
+  carries it verbatim, `structuredContent` omits it (a `wireValue`
+  projection), with the doc resource (`daftari://doc/{path}`) as the
+  programmatic alternative. The CallTool bridge hardens presentation: a
+  throwing `summarize`/`docLinks` falls back to the pre-Decision-3
+  `JSON.stringify` behavior and logs to stderr instead of turning a
+  successful tool call into an error response.
+  Design record:
+  `docs/superpowers/specs/2026-07-26-mcp-2026-07-28-readiness-design.md`.
+
+### Fixed
+
+- `vault_tier2_queue`'s `field_changes` could report a field with no `before`
+  key at all (dropped by JSON serialization on a document's first write),
+  violating its own declared output shape. `before` now normalizes to `null`
+  when the log has no prior value, matching the schema's documented
+  "`null` means no prior value" contract.
+
 - **Bi-temporal validity.** Two optional built-in frontmatter fields,
   `valid_from` and `valid_until`, recording when a document's claim was true
   *in the world* — as distinct from when the vault recorded it, which git

CLAUDE.md

@@ -18,12 +18,14 @@ Concepts and layer boundaries: docs/architecture.md. Where things live:
 - `src/witness/` — per-principal track records
 - `src/asof/` — belief archaeology over git history (`daftari asof`)
 - `src/audit/` — doc-to-code coherence audit (`daftari audit`)
+- `src/anchors/` — citation-anchor pin grammar and the shared JIT classifier (used by vault_read, `daftari audit`, and vault_lint)
 - `src/eval/` — vault quality eval: question generation + LLM judging (`daftari eval`)
 - `src/interview/` — principal interview: question sheet from tensions/staleness/open questions, verbatim transcript (`daftari interview`)
 - `src/backfill/`, `src/import/`, `src/okf/` — adoption paths: metadata backfill, foreign-vault import, OKF export/import
 - `src/serve/` — server mode over Streamable HTTP (`daftari serve`); `src/sync/` — push/restore against storage backends
 - `src/hooks/` — vault-supplied hook module loading
 - `src/themes/` — clustering primitives for vault_themes
+- `src/context/` — context-pack assembly for vault_context: deterministic selection, budget-cut, markdown templating over enriched candidates (`src/context/assemble.ts`), chars/4 token estimation (`src/context/estimate.ts`). No I/O, no LLM call — see the standing constraints below.
 - `src/utils/` — config.yaml loading, git plumbing, paths, hashing
 
 Entrypoints: `src/index.ts` (stdio MCP entry), `src/server.ts` (MCP server wiring), `src/cli.ts` (CLI).
@@ -43,7 +45,11 @@ Entrypoints: `src/index.ts` (stdio MCP entry), `src/server.ts` (MCP server wirin
 - Tension/edge visibility: omission over redaction, no existence leak. Doc lists never name docs in unreadable collections; hidden-blast remainders are reported coarsened (none/some/many), never as exact counts — small cells disclose linked existence. Vault-global lint aggregates stay unfiltered by design. See docs/superpowers/specs/2026-07-14-edge-graph-existence-disclosure-design.md.
 - The Tension Court is an operator-only surface. Court/docket code never takes an access context. Exposing any court surface via MCP requires revisiting the 2026-07-14 edge-graph spec first.
 - Storage backends (#6) are dumb sync targets — `get/put/list/delete` over opaque keys. The local git working copy is canonical; backends never understand markdown, git, or locks; index/locks stay local and never sync. See spec 2026-07-20 Decision 3.
+- MCP: the server speaks the 2026-07-28 stateless revision (v2 SDK, `@modelcontextprotocol/server`). `daftari serve` resolves identity per request from the bearer — no sessions — and refuses 2025-era traffic; stdio serves both eras. `vault_ratify` without a decision elicits an approve/reject form (reject preselected) with HMAC-signed opaque state — single-`id` calls only; a batch `ids` call always requires an explicit decision. The maintenance passes stay CLI-only until the Tasks extension has a TS SDK runtime. See docs/superpowers/specs/2026-07-26-mcp-2026-07-28-readiness-design.md.
 - Only one daftari process may hold a vault at a time. `.daftari/process.lock` is the per-vault process lock, and it records the holder's mode (stdio or serve). Live-holder precedence favors the durable tenant (2026-07-20 spec, Decision 4): stdio finding a live stdio holder SIGTERMs it and waits up to 3 seconds before taking over (the original single-user convenience — the only implicit live takeover); stdio finding a live `daftari serve` REFUSES to start; a new serve refuses against ANY live holder unless started with `--takeover`. Stale locks (dead PID, or PID recycled) are overwritten silently in every mode. The lockfile is ephemeral — never check it in.
+- The staged-action queue's risk score is derived/ordinal, computed fresh on every `vault_lint` read, never stored as a `risk` field or column — the same posture `derives_from` strength takes. `vault_ratify`'s `ids` batch is an explicit, capped id list (`BATCH_RATIFY_MAX` = 20); the parameter shape has no threshold and no "all pending" sentinel, so auto-approval-by-score cannot be expressed. Proposal records carry the authenticated stager (`staged_by_principal`, from `access.user`) alongside the unauthenticated, caller-claimed `proposed_by` display string — the witness and the risk score's W term key on the former when present. Each decision record additionally carries a non-authoritative `risk_at_decision` snapshot (JSONL-only, never read for ordering) so the risk score's own predictive-power kill condition is evaluable later. See docs/superpowers/specs/2026-07-26-risk-triaged-ratification-design.md.
+- Citation-anchor pins (`describes` entries suffixed `#L<start>-<end>@<sha>`) are advisory, annotate-only — a `moved`/`missing` pin never invalidates, demotes, or rewrites a doc, and an intact pin softens decay's banner copy but never extends the TTL clock. The read-path check is batched git plumbing only (one `hash-object` invocation per referenced repo per read, not per pin) and silent on failure (a classifier error drops that entry, never fails the read). Configuring `code_repos` makes blob-level facts about those repos (path existence, blob-match, relocated line numbers) visible to every reader whose role carries the `code_repo_visibility` grant (default off) — see docs/superpowers/specs/2026-07-26-citation-anchors-jit-verification-design.md.
+- `vault_tools` is the always-advertised in-band catalog: index mode (`{name, oneLine}` per tool) or expand mode (full schemas for named tools). It reads the FULL registry minus the vault's `exclude` list — exclude always wins (#104), but tier and `include` never affect it, since making tiered-out tools discoverable in-band is its entire purpose. `vault_context(task, budget?)` assembles a token-budgeted markdown brief: hybrid search → RBAC filter (before any budgeting) → supersession dedup (a collapsed chain's flags are ALL keyed on the head's own path, never a stale member's) → greedy budget cut at `budget * 0.9` estimated tokens (chars/4, no tokenizer) → render. No LLM call anywhere in the handler — pure selection and templating; a tension always renders both claims, never a blended verdict. `hidden_remainder` is a lower-bound signal over OBSERVABLE withholding (RBAC-dropped BM25-side pool candidates, dropped coverage additions, restricted supersession hops) — "none" means "no withholding observed," never "nothing withheld," because the vector half of retrieval is already RBAC-pushdown-scrubbed and structurally invisible to this count. Both tools join `CORE_TOOLS`; the default `tools.tier` stays `full` this wave — see docs/superpowers/specs/2026-07-26-context-packs-progressive-disclosure-design.md.
 
 ## Labeling
 - [DATA] for values read from files or the index

README.md

@@ -414,6 +414,43 @@ SDK's standard environment chain, never from vault config. GCS is reached via
 its S3-interoperability endpoint. Restore refuses non-empty directories and
 reindexes when done.
 
+## Embedding providers
+
+`vault_search`'s vector half is a swappable, config-driven `EmbeddingProvider`.
+Four ship:
+
+```yaml
+# .daftari/config.yaml
+embeddings:
+  provider: local-minilm          # local-minilm | openai-3-small | local-embeddinggemma | local-qwen3-0.6b
+  # dim: 512                      # local-embeddinggemma / local-qwen3-0.6b only: 512 (default) | 768
+  # quantize: int8                # int8 (default for the two new local providers) | none
+```
+
+- **`local-minilm`** (programmatic default) — `all-MiniLM-L6-v2`, 384d,
+  fully local via `@huggingface/transformers`, free, no API key. ~100MB
+  model footprint.
+- **`local-embeddinggemma`** — `google/embeddinggemma-300m`, 768d native,
+  Matryoshka-truncatable to 512 (default) or 768, fully local. A
+  model-generation upgrade over `local-minilm`, currently opt-in — see
+  `docs/architecture.md` for the verification-honesty caveat before relying
+  on it in production. ~600MB-class footprint.
+- **`local-qwen3-0.6b`** — `Qwen/Qwen3-Embedding-0.6B`, exposes up to 768d
+  (native 1024d not yet offered), fully local, larger footprint
+  (~1.5GB-class) than EmbeddingGemma. The documented alternative, not a
+  default candidate.
+- **`openai-3-small`** — OpenAI `text-embedding-3-small`, 1536d, paid,
+  requires `OPENAI_API_KEY`. Untouched by the two entries above.
+
+`warm_embeddings: false` (default `true`) skips the background model
+warm-up at startup — the escape hatch that matters more once the default
+footprint options grow past `local-minilm`'s. Switching `provider`, `dim`,
+or `quantize` between server runs never loses data: the durable embeddings
+cache is keyed by `(content_hash, model)` and stores every provider's
+vectors side by side, so switching back is free and a `dim`/`quantize`
+change alone needs no re-embed at all (see `docs/architecture.md`,
+"Vec-index quantization").
+
 ## How it compares
 
 |                    |AGENTS.md        |RAG                          |Daftari                              |
@@ -437,9 +474,14 @@ Deliberately deferred to keep the surface tight:
   advisory boundary warnings shipped in the meantime: `vault_lint`'s
   `domainLeaks` check and write-time `domain_warnings`)
 
-(LLM reranking, deferred here originally, has since shipped as the opt-in
-agent-as-judge `rerank_candidates` on `vault_search`: the server prepares the
-fused candidate pool and the protocol; the calling agent is the judge.)
+(LLM reranking, deferred here originally, has since shipped two ways. The
+agent-as-judge `rerank_candidates` on `vault_search` is opt-in and free: the
+server prepares the fused candidate pool and the protocol; the calling agent
+is the judge. A second, local cross-encoder reranker (`rerank.provider:
+local-bge-m3` in `.daftari/config.yaml`) also ships, opt-in and default off —
+it reorders the top-50 RBAC-filtered hits with a local ONNX model before the
+`vault_search` response is sliced to `limit`; see `docs/architecture.md` for
+the config block and degradation behavior.)
 
 Each is a clean increment on a surface that already works.