feat: enhancement wave — eight 2026-07-26 specs, Jugalbandi-resolved - #310
Open
mavaali wants to merge 46 commits into
Open
feat: enhancement wave — eight 2026-07-26 specs, Jugalbandi-resolved#310mavaali wants to merge 46 commits into
mavaali wants to merge 46 commits into
Conversation
…dge hardening
Part 1/3 of Decision 3's remaining gap closure (spec 2026-07-26, final plan
in .jugalbandi/mcp-readiness-remainder/final-plan.md, PR 1).
- src/tools/summary.ts: clip() and the shared SUMMARY_DETAIL_CHARS /
SUMMARY_MAX_ROWS constants, extracted from vault_lint's summarizer so
every new summarizer clips detail text the same way.
- ToolDefinition gains wireValue?: (value) => Record<string, unknown> — a
per-tool projection of the ok-value onto structuredContent, distinct from
the full value summarize/docLinks still see. vault_read uses it to drop
`content` (the body) from structuredContent: the body now ships exactly
once, in content[0].text, instead of doubled onto the wire (C11) — the doc
resource (daftari://doc/{path}) is the programmatic alternative.
- vault_read, vault_index, vault_status, vault_reindex get summarize
(vault_read also gets docLinks, naming itself plus every visible upstream
unit).
- server.ts: the CallTool bridge's presentation step is extracted into
formatSuccessResult (a pure-ish function of a ToolDefinition and its
ok-value) so it can be unit-tested directly against hand-built stub tools.
summarize/docLinks each run in their own try/catch — a throw falls back to
the pre-Decision-3 JSON.stringify (or an empty link list) and logs to
stderr, so a summarizer bug can never turn a successful tool call into an
error response (C5). docLinks output is filtered to non-empty strings
before becoming resource_link entries. allRegisteredTools() is exported as
a read-only test seam.
- packages/router/test/integration.test.ts: updated for vault_read's
content[0].text no longer being bare JSON — asserts against
structuredContent instead, which is what the router's own fanout.ts
already preferred.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tools Part 2/3 of Decision 3's remaining gap closure (final plan PR 1). - write.ts: one shared summarizeWrite/docLinksWrite pair for all eight write tools (WriteResult is the common shape) — verb, path, commit short-sha, plus staged/hint/domain-warning notes when present. WriteResult gains an optional `sources` field, populated only by vault_merge with the two canonical source paths it superseded: docLinks only ever names paths literally present in the result value, and merge's sources otherwise live nowhere in that value. - staged-actions.ts: vault_stage_action / vault_ratify get one-line summaries (id/expiry/conflicts, and decision/applied-or-shadow respectively). Neither gets docLinks — the target document isn't part of either result value, and docLinks must never re-derive a path from args. - curation.ts: clip()/its constant move to src/tools/summary.ts (curation.ts re-imports); vault_tension_log/resolve, vault_tension_clusters, vault_tension_blast, and vault_provenance get summarize, with docLinks restricted to sourceA/sourceB or the value's own downstream/cluster paths — coarsened buckets (hidden_downstream: none/some/many) are repeated verbatim in the summary, never sharpened into a count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eceipt, tier1/2, and staleness tools Part 3/3 of Decision 3's remaining gap closure (final plan PR 1). - edges.ts: vault_edge_observe/contest/edges — one line per edge (from → to, k/strength/status), docLinks naming both endpoints (deduped, capped at SUMMARY_MAX_ROWS for the list tool). - consumes.ts: vault_consumes — count + per-edge lines, docLinks over the visible artifact/unit paths. - themes.ts: vault_themes — theme count + one line per theme (label, size, top exemplar), docLinks naming one exemplar per theme (a theme with no retained primary member contributes nothing rather than guessing). - witness.ts: vault_witness — principal + track-record counts for both result shapes (full report and single-principal). No docLinks: a principal is an identity string, not a vault document path. - receipt.ts: vault_receipt — verdict (flags, or "clean") plus per-source status/confidence lines, docLinks over every cited path. - tier1.ts: vault_tier1 — dispatch outcome one-liner, docLinks naming the anchor unit. - tier2.ts: vault_tier2_queue/verdict — queue depth + first-10 items (a tool-specific row cap, denser per-row than the shared default), verdict one-liner; docLinks over the judged artifact/unit pair. - edge-staleness.ts: vault_staleness — renders the per-artifact counts (or the vault-global broken-read-rate report) without writing new prose over what summarizeUpstream already computed; the hidden_pending coarse bucket is repeated verbatim, never sharpened. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt key
accumulateFieldChanges built { before: diff.before, after: diff.after }
verbatim from the provenance log's frontmatter_diff. On a document's first
write (a `create` action), frontmatterDiff has no prior frontmatter to read
`before` from, so it sets `before: undefined` — and JSON.stringify drops
undefined-valued keys entirely when the entry is appended to the log. Read
back, `diff.before` is `undefined` (the key is simply absent), which
propagated into the FieldChange the tool returns.
tier2WorkItemSchema requires both `before` and `after` on every field_changes
entry, documenting `null` as "no prior value" — an absent key was a shape the
contract never promised. Caught by the new ajv output-schema test added
alongside vault_tier2_queue's summarizer (a real vault_tier2_queue call
against a freshly-created dependent failed schema validation on this exact
field). `before` now normalizes to `null` via `?? null` when the log has no
prior value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Final plan PR 1 §1.3: per-tool output-schema validation with the jugalbandi-challenge-C6 resolution (strict compilation, not strict: false), plus the summarizer degenerate-value coverage C5 asked for. - ajv added as a devDependency (it already rode in transitively via the SDK) — production code never validates outputs at runtime; this is test-only. - test/helpers/output-schema.ts: compileToolSchema/expectMatchesOutputSchema wrap one Ajv2020 instance built with strict: true. allowUnionTypes: true is the one deliberate relaxation (the registry's schemas use `type: [X, "null"]` throughout for "nothing to say" fields — standard 2020-12, not laxness); an empty, documented ALLOWED_VOCABULARY allow-list is the seam for any future non-standard keyword, reviewed one at a time. Misspelled keywords fail compilation, which was the whole point of C6's challenge to a strict: false version of this helper. - test/server.test.ts: a registry-wide test that every tool's outputSchema compiles under strict 2020-12, plus formatSuccessResult bridge tests (no-summarize JSON fallback, a throwing summarize/docLinks never producing isError, docLinks→resource_link round-tripping through docUri, non-empty string filtering, wireValue projecting structuredContent while summarize/docLinks still see the full value) and a vault_read pin that structuredContent carries no `content` field. - test/tools/summarizers.test.ts: every summarizer-bearing tool driven against the smallest legal value its Result type allows — zero counts, empty arrays, null banners, "none"/"many" coarsened buckets — asserting summarize/docLinks never throw and a coarsened bucket is never sharpened into a number. This is what caught the tier2 field_changes bug fixed separately. - test/tools/summary.test.ts: unit tests for the new clip()/constants module. - Existing test/tools/*.test.ts files (read, write, staged-actions, curation, edges, consumes, themes, witness, receipt, tier1, tier2, edge-staleness, search): one expectMatchesOutputSchema pin added to a representative happy-path assertion per tool, so a real handler result is checked against its own declared schema, not just a hand-built one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PRs 2-4 The spec's status line still read "proposed — implementation not started" despite Decision 2/3-core/6 having shipped in 8a1db89 (#302); updates it to reflect that plus this wave's PR 1 (Decision 3's remaining gap). Records the preflight-gate evidence for why PRs 2-4 (stateless serve, Tasks, form-mode elicitation) did not proceed: @modelcontextprotocol/sdk 1.30.0 (latest, 2026-07-27) does not implement the 2026-07-28 revision — stable LATEST_PROTOCOL_VERSION is still 2025-11-25, the matching schema exists only under a spec.types.d.ts pulled from a branch literally named DRAFT-2026-v1, Tasks are experimental with a "may change without notice" warning, and no InputRequiredResult type exists anywhere in the package. CHANGELOG entry for the new summarize/docLinks coverage and the vault_read wire-projection change, plus the tier2 field_changes fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a "rrf" FusionMode to hybrid.ts next to today's "weighted" min-normalize + weighted-sum path. RRF fuses via scaled reciprocal rank (RRF_K=60, contribution (k+1)/(k+rank) so rank 1 = 1.0 and the fused score keeps its top≈1 scale for downstream consumers). hybridSearch defaults to "weighted" (DEFAULT_FUSION, flipped in a future PR gated on the fusion bench); relatedSearch keeps its own always-weighted default (no bench coverage yet for its different fusion shape). Both accept an explicit `fusion` option. Also adds a deterministic tie-break (score desc, then path asc) to the final sort in both modes — RRF makes exact fused-score ties common, and the old SQL-row-order tie-break had no cross-run guarantee. bm25Score/vectorScore output-schema descriptions become fusion-neutral text, true under both modes. Per docs/superpowers/specs/2026-07-26-retrieval-fusion-overhaul-design.md and the resolved final plan (Decisions 1 & 4 dispositions C4, C10). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n 2)
Each quoted span of >= 2 usable tokens in a query now ALSO emits an FTS5
phrase branch ("tok1 tok2") alongside the existing per-token prefix-OR
branches, instead of replacing them. Strictly recall-non-shrinking (a
superset MATCH query): a document containing the exact phrase now
additionally satisfies the phrase branch and BM25 scores it above
token-scatter documents, which is what quoting a phrase is supposed to mean.
This is what makes the router's forthcoming "quoted phrase" signal
(classifying a query extreme-lexical, {bm25: 1, vector: 0}) sound: without
phrase emission, the extreme route would disable the semantic ranker at
exactly the moment the lexical engine couldn't distinguish an exact phrase
from its scattered tokens.
Per the resolved final plan, disposition C5.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…(Decision 2)
New src/search/router.ts: pure functions, no I/O except the injected df
lookup. classifyQuery(rawQuery) evaluates five signals on the raw query
string and its whitespace-split tokens — quoted phrase, path-like token,
CamelCase, snake_case, digit-heavy — plus an optional rare-term signal (df
in [1, DF_RARE_FLOOR=2], disabled entirely below MIN_DOCS_FOR_RARE=100
documents) via an injected df lookup. Precedence: any extreme signal wins;
else any lexical signal; else balanced. Every fired signal is reported, not
just the ones that decided the class.
routeWeights maps extreme-lexical -> {bm25:1, vector:0} (skips query
embedding entirely downstream), lexical -> {bm25:0.8, vector:0.2}, balanced
-> DEFAULT_WEIGHTS.
makeDfLookup(db) prepares `SELECT count(*) FROM documents_fts WHERE
documents_fts MATCH ?` and double-quotes the token so FTS5's porter
tokenizer stems the query side too — document-frequency is stem-aware by
construction, no fts5vocab table and no index-db.ts schema change.
Per the resolved final plan, dispositions C8 (fts5vocab dropped, small-vault
guard) and C3 (df lookup reused for the bench's identifier category).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New `search` config block, following the `tools` block's validation pattern: `routing: boolean`, defaulting to false. Accepts both YAML booleans and the bare strings "on"/"off" — js-yaml 4's YAML-1.2 core schema loads bare `off`/`on` as strings, not booleans, and the spec's own example writes `search.routing: off`. Unknown child keys fail loud via the shared rejectUnknownKeys helper. DaftariConfig.search is always populated (routing: false when the block is absent). Per the resolved final plan, section 2.3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vaultSearch now resolves weights with an explicit precedence: a valid
explicit `weights` arg always wins; an INVALID `weights` arg gets static
DEFAULT_WEIGHTS (expressed intent to control weights must never silently
fall through to router-driven ranking); a genuinely absent `weights` arg
consults the router only when the vault's `search.routing` config is on; a
config-load failure degrades to static defaults rather than failing the
search. vault_search_related is untouched — it has no user query to
classify.
The result gains an optional `routed: { class, signals }` field, present
ONLY when the router chose the weights — added to outputSchema, summarize,
and the ajv-checked output-schema tests. This makes a routed lexical-only
result (`routed` present, vectorUsed: false) distinguishable from an
embedding-provider degrade (`routed` absent), and surfaces the router's
signals diagnostic instead of discarding it.
parseWeights is replaced by parseExplicitWeights (absent vs invalid vs
valid), with a staticWeightsFallback helper preserving vault_search_related's
unchanged behaviour.
Per the resolved final plan, section 2.4 and disposition C9.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New integrations/recall-bench/fusion-runner.mjs, cloned from
chunkbm25-runner.mjs's pattern (top-of-file constants, dist/ dynamic
imports, --smoke, JSON outputs). Four arms over the day vault plus a
restricted-role split vault:
A: hybridSearch(db, q, { limit: 50 }) — weighted fusion (today)
B: A + fusion: "rrf" — RRF fusion
C: B + weights: routeWeights(classifyQuery(q).class) — RRF + router
D: C's config on an 8-collection split vault, post-filtered (D-post,
simulating the pre-a2ec361 starvation bug) vs pushed-down into the KNN
(D-push, the shipped fix) — measures the ACL pushdown under the new
fusion.
Three question categories: `paraphrase` (the machine-local
ea-180d-partial-2026-06-21 fixture, uncommitted per the shipped chunkbm25
convention), and two synthetic categories built from doc bodies with a
committed seeded PRNG — `phrase` (a corpus-unique quoted 2-3 token run,
bench mass for the extreme-lexical route) and `identifier` (a stem-aware
df===1 token, bench mass for the rare-term signal), replacing the rejected
title-based identifier set.
Per-arm, expectation-shaped vectorUsed assertions (a routed extreme-lexical
{bm25:1,vector:0} result legitimately reports vectorUsed:false and must not
abort the run; any other mismatch is a real embedding failure and aborts).
Rank-identity/no-leak comparison on the split vault collects mismatches
across the full run instead of aborting mid-flight, and separates genuine
mismatches from "boundary ties" (docs sitting at the K=64 vector-KNN
boundary, where sqlite-vec gives no ordering guarantee). Restricted-arm
recall excludes unreadable relevant days from the denominator.
gates.rrfFlip, gates.routingFlip, and gates.noLeak are computed and printed,
not eyeballed. Provenance (sha256 of the questions file and the day vault's
document listing) is recorded in the summary JSON.
prep-vault.mjs gains an --out <dir> flag (default unchanged at
/tmp/cov-recall/vault for the chunkbm25 bench) so the fusion bench preps
into its own /tmp/fusion-recall/vault without clobbering the sibling
bench's fixture — no forked prep script. ROOT is now derived from
import.meta.url in both scripts instead of a hardcoded absolute path, so
they run correctly from any checkout/worktree.
Verified with a real end-to-end smoke run against a synthetic 180-doc
corpus and a synthetic paraphrase question set (not committed — the real
Stevenic/recall corpus and ea-180d questions fixture are machine-local, per
the shipped chunkbm25 precedent): all four arms, the split-vault build and
its collection-count assertions, the router wiring, the no-leak comparison,
and all three gates ran clean with zero unexpected aborts.
Per the resolved final plan, sections 1.4 and 2.5, and dispositions C1, C2,
C4, C7.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace chunkText with chunkDocument({title, collection, tags, body}),
implementing spec 2026-07-26-contextual-chunking-reranker-design.md
Decisions 1-2. Line-scans the body tracking fenced code blocks (a `#`
line inside a fence is never a heading) and the open ATX heading stack
(levels 1-4; #####/setext degrade to plain text). A heading boundary
always starts a new chunk — no packing across sections, ever — reusing
today's paragraph-packing loop (extracted as packParagraphs) within each
section.
Every chunk carries a one-line breadcrumb context ({collection} ›
{title} › {headings} · tags: a, b, c), built by buildContext with a
deterministic truncation order (collapse middle headings, then
tail-truncate the innermost heading, then drop tags, then tail-truncate
the title — collection and title always survive as components). Tags
are sorted lexicographically before the 5-tag cap so tag reorder can
never perturb the hash (plan C7).
embeddingInput(chunk) = context + "\n\n" + text is the single source of
truth for both the content hash and the embedding text, so they can
never drift.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SCHEMA_VERSION 10 -> 11: chunks gains a context TEXT NOT NULL DEFAULT '' column, and chunks_fts becomes a two-column FTS5 table (context, text) so bm25(chunks_fts) spans both — contextual BM25, spec Decision 2. All three chunk triggers (chunks_ai/ad/au) carry both columns. Every chunk's hash input changes, so this bump forces a full re-embed by design (see the SCHEMA_VERSION comment for the Phase 0 cross-spec numbering contract with the embedding-refresh work implemented after this one). ChunkRowInput/insertChunkRow/IndexedChunk/rowToChunk/getAllChunks/ getChunksForPath carry the new column; context is optional on ChunkRowInput (defaults to '') so low-level test call sites unrelated to chunking don't all need a real breadcrumb. Adds three read helpers Part B's reranker will use to resolve passage text without joining the whole candidate set: getChunkTextsByRowids (batched, by chunks.rowid), getChunkByPathAndHash, and getFirstChunk (the terminal fallback, backed by chunkDocument's >=1-chunk guarantee). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
stageOne hoists the resolved collection so the document row and its
chunk breadcrumbs agree, calls chunkDocument({title, collection, tags,
body}), and hashes each chunk over embeddingInput(chunk) instead of raw
text. StagedDocument.chunks is now DocumentChunk[]; both the full and
incremental reindex paths embed embeddingInput(chunk) and write
chunk.context through to insertChunkRow.
Consequence (spec Decision 3 / plan C7), now pinned by tests: identical
body text under identical title/collection/tags still shares one
embedding row (same-pass miss-dedupe); identical body text under
different titles produces two rows, not a cache hit — a stale-vector
hit would silently serve pre-edit semantics, which is worse than the
recompute. This replaces the old "moved paragraph re-embeds zero" test,
whose premise (chunk hash depends on text alone) is exactly what
Decision 2 changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chunkFtsRanking's served-snippet query now targets chunks_fts column 1 (text) explicitly, so a query that matches only in the synthesized context column can never leak breadcrumb prose into a snippet (spec Decision 4). Documents the tier-semantics shift contextual chunking causes (plan C1): title/collection/tag tokens now flow into every chunk's context column, so a title- or tag-only query enters tieredLexical's UPPER band via a genuine chunk match, not just the lower title/tag fallback tier — the strict "body outranks title-only" guarantee now holds only for docs with no context-column match at all. New tests pin this as a tested decision: a title-only match clears TIER_SPLIT, a pure-body match and a context-only match co-rank by bm25 (both clear the band), and the title/tag fallback still fires for a document whose chunks are somehow absent from chunks_fts. Also lands the plumbing Part B's reranker needs without paying for passage-text resolution over the whole over-fetched candidate set (plan C2/C4): vecRanking returns each path's best-KNN content_hash, chunkFtsRanking returns its winner rowids, and rankDocuments can attach a cheap PassageRef per hit (capturePassageRefs option) choosing between the lexical winner and the vector winner by whichever signal's normalized score is higher for that path. HybridSearchResult gains optional rerankUsed/passageRefs fields, set/consumed by the tool handler in a later commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New src/search/rerank-provider.ts, mirroring embedding-provider.ts / vector.ts's selection block (spec 2026-07-26-contextual-chunking- reranker-design.md Decision 5): setRerankProvider/getRerankProvider memoise one provider per process, "none" maps to null so callers branch on presence instead of a null-object provider, warmRerankModel() is a no-op ok() when nothing is configured, and setRerankProviderForTests/ resetRerankProviderForTests give tests a fast, deterministic seam. New src/search/providers/local-bge-m3.ts, sibling of local-minilm.ts: BAAI/bge-reranker-v2-m3 (onnx-community/bge-reranker-v2-m3-ONNX), ONNX q8, via @huggingface/transformers — already a dependency, zero new deps. Score = sigmoid of the single (query, passage) logit, scored in fixed sub-batches of 8 (same peak-memory argument as local-minilm's EMBED_BATCH_SIZE). The model loads lazily and is memoised; isReady() lets the search path check readiness without ever triggering a synchronous load inside a tool call (Decision 8 / plan C5). A 600MB model download is unreasonable for this environment, so this file is implemented against the documented transformers.js API surface but not exercised against the real model here — every test in this PR uses the RerankProvider seam with a fake, and a real-model smoke test is gated behind DAFTARI_BGE_SMOKE in a later commit so default `npm test` never downloads it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… var describe.skipIf(!process.env.DAFTARI_BGE_SMOKE) — sanity-checks that the real ~600MB q8 model scores an obviously-relevant passage above an obviously-irrelevant one. Skipped by default so `npm test` never downloads it; run explicitly with DAFTARI_BGE_SMOKE=1 when a real model verification is wanted. This is the §3.2 spike's ordering-sanity half; the measured-latency half is a throwaway scratchpad script per the plan, not a committed test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DaftariConfig.rerankProvider (default "none"), parsed the same way as embeddings.provider: mapping-shape checks, unknown-id hard error — a typo that meant to enable reranking must never silently no-op. No env-var check: local-bge-m3 is a fully local model, no API key to validate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
setRerankProvider(config.value.rerankProvider) installs alongside setProvider in the same fail-loud block, in both src/index.ts (stdio) and src/serve/index.ts (serve) — mirroring the embedding provider's startup wiring exactly. runBackgroundWarm (shared by both entry points via startVaultServices) warms the reranker after the embedder, gated by the existing warm_embeddings flag — no new config knob, since "pay model cold-starts at startup, not on the first query" already covers either model (spec Decision 8). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wires the Part B cross-encoder into vaultSearch, between the RBAC/ validity filter and the slice to `limit` (spec Decision 7): a fixed RERANK_POOL (50) of the fused hits is scored against the query and reordered when a reranker is configured and warm; a 1500ms timeout (withTimeout, racing the provider call) and a provider Result.err both degrade to the fused order identically, logged once per process. A not-ready provider skips reranking for the current search and fires warmRerankModel() in the background rather than ever blocking a tool call on a cold model load (C5). resolvePassages resolves passage TEXT for exactly the top-50 permitted hits via the storage-layer lookups landed in an earlier commit (getChunkTextsByRowids batched for lexical refs, getChunkByPathAndHash for vector refs, getFirstChunk as the terminal fallback), never for the full over-fetched candidate set (C2). hybridSearch is called with capturePassageRefs only when a reranker is actually configured. The #3 agent-as-judge rerank_candidates pool now draws from the reranked order (finalRanked), not the pre-rerank fused order — when both features are on, the agent judges the already-reranked pool. vault_search's result gains rerankUsed (added to outputSchema as a required field) and internal passageRefs is stripped before returning. Tests: nine tool-level scenarios against a fake RerankProvider (RBAC- before-rerank ordering, promotion past the default page, Result.err and timeout degradation, not-ready background warm, provider none, rerank_candidates sourcing, pool-boundary stability, and passage- resolution scope) plus two hybrid.ts-level tests pinning the C4 provenance choice (a hit whose vector signal outscores its lexical signal presents its KNN chunk) with hand-built index rows so both signals are independently controllable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
architecture.md: chunkDocument's heading-aware, breadcrumb-contextual splitting; the search-pipeline ordering table gains the optional rerank stage; the honest-assessment section names the 1.33.0 schema-bump re-embed cost and the ongoing metadata-edit re-embed cost (retitle / re-collection / tag-set change), plus a new paragraph on the reranker as a second, opt-in cost lever; a new subsection documents the RerankProvider seam with its config block. CHANGELOG.md: an [Unreleased] entry covering both parts. README.md: the "what's not in v1" LLM-reranking note now distinguishes the free agent-as-judge rerank_candidates path from the new local cross-encoder option and points at architecture.md for the config block and degradation behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mma, Qwen3) Adds the provider-layer half of the embedding-refresh-quantization spec (Phase 1): a shared local-transformers.ts factory generalizing local-minilm.ts's shape (memoised lazy extractor, index-state markers, Result-typed embed, fixed sub-batches) to cover two new local models — EmbeddingGemma-300M (mean pooling) and Qwen3-Embedding-0.6B (last-token pooling) — both Matryoshka-truncatable and asymmetric-prompt-prefixed. EmbeddingProvider gains nativeDim/embedQuery/isLoaded. The durable cache now stores the FULL native-dim vector under a dim-free cache id (`local-embeddinggemma#p1`); toIndexDim (vector.ts) is the single choke point where a native-dim vector is truncated + re-normalized to the configured index dim. l2Normalize moves from openai-3-small.ts to vector.ts as a shared primitive. Prompt-prefix strings are [TRAINING] hypotheses from the governing spec, unverified against the Phase 0 spike (not run in this environment — no model download). The #pN cache-id suffix means a future prefix correction behaves like a provider switch (cache miss, re-embed), not a silent stale read. New provider tests mock @huggingface/transformers entirely (no real model download in default `npm test`); real-model smoke tests are added gated behind DAFTARI_EMBEDDINGGEMMA_SMOKE / DAFTARI_QWEN3_SMOKE, mirroring the existing DAFTARI_BGE_SMOKE pattern, and were not run in this session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extends EMBEDDING_PROVIDERS with local-embeddinggemma and local-qwen3-0.6b, and adds two new embeddings.* keys (spec 2026-07-26 embedding-refresh-quantization, Phase 2): - dim: Matryoshka truncation target, validated per-provider against EMBEDDING_DIMS (null for local-minilm/openai-3-small — a hard error if set on a fixed-dim provider). Defaults to each new provider's first allowed dim (512) when the provider supports it but dim is unset. - quantize: "int8" | "none", the vec-index representation. Defaults to "int8" for the two new providers, "none" for the existing ones (existing vaults stay bit-identical unless the operator opts in). Accepted for any provider. loadConfig's programmatic fallback stays local-minilm — this is not the default flip (Phase 6), which is gated on the spec's Phase 5 recall-bench and has not run against real models in this environment. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e-dim cache
Phase 3 of the embedding-refresh-quantization spec:
- storage/index-db.ts: VecKind ("float32" | "int8"), VEC_KIND_META_KEY
twin of VEC_DIM_META_KEY, quantizeInt8 (round(x*127) clamped to
[-127, 127]), createVecTable(dim, kind). openIndexDb's third parameter
expectedVecKind is now REQUIRED (disposition C2) — the same posture as
the existing required-dim precedent, closing the silent-destructive-
drop hazard a defaulted kind would open the moment quantization is
used anywhere in a vault. All 9 production call sites (reindex.ts,
watcher.ts, tools/search.ts, tools/read.ts, curation/staged-actions.ts,
curation/edges.ts x4) now pass getQuantize(); every test caller passes
"float32" or "int8" explicitly (mechanical update, ~230 call sites
across test/).
- insertEmbeddingVec / rebuildEmbeddingsVec: int8 writes go through
sqlite-vec's `vec_int8(?)` SQL wrapper, not a raw blob — confirmed
empirically against the pinned sqlite-vec build (a raw int8-byte blob
bound directly is rejected: "expected type int8, but a float32 vector
was provided"). This is the "working insert/query binding form" the
spec's Phase 0 gate 3 asks the (unrun) spike to record.
- hybrid.ts vecRanking: scan-then-rescore on the int8 path (disposition
C3). The KNN scan over the quantized column selects VEC_KNN_K x
RESCORE_MULTIPLIER(4) candidates by quantized distance ONLY; each is
then rescored with exact float32 cosine against the durable cache,
joined in the same statement. A candidate whose cache row is missing
(a gc race) is DROPPED, never approximated by the quantized distance —
there is no distance-to-score conversion on the int8 path at all. The
float32 path is byte-for-byte unchanged.
- reindex.ts: the durable `embeddings` cache now stores the FULL
NATIVE-dim vector (disposition C9) instead of the configured index
dim — insertEmbedding's dim guard checks nativeDim. toIndexDim
truncates at the single choke point where a cached vector meets the
vec mirror (rebuildEmbeddingsVec, the indexDocument incremental
mirror). A dim change is now a pure vec-mirror rebuild from cache (all
cache hits), not a cold re-embed. Same fix applied to relatedSearch's
meanEmbedding inputs and vault_themes' chunk-vector loader (hybrid.ts,
tools/themes.ts) — both read the same native-dim cache.
- isIndexFresh gains a vec-coherence check (disposition C1 — this is
what makes "config change + background reindex" an actually-triggered
migration): (a) stored embedding_model meta must equal the active
provider's cache id, (b) embeddings_vec must be non-empty whenever
chunks is non-empty. Either failing routes through the normal
reindex, which for a dim/quantize flip is all cache hits (a mirror
rebuild in minutes) and for a provider switch is the intended cold
re-embed. This also closes a pre-existing hole for provider switches
under the old code (a provider swap could report "fresh" with an
emptied vec mirror).
Tests: quantizeInt8 rounding/clamping, kind-mismatch drop-recreate with
VEC_KIND_META_KEY persisted, float32<->int8 round trip through a real
KNN query; hybrid.test.ts covers rescored ordering vs raw quantized
distance and the orphan-drop + score-range invariant; reindex.test.ts
covers the C1 freshness check and C9's all-cache-hits dim flip using a
fake network-free provider.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 5 of the embedding-refresh-quantization spec: a recall-bench runner
modeled on chunkbm25-runner.mjs / fusion-runner.mjs — one vault,
provider-switched per arm via setProvider() + reindexVault() (the actual
migration path Phase 3d makes real), Stage-A recall@{10,20,50} metrics,
hybrid and vector-only variants, plus the doc-doc mean-embedding overlap
smoke (disposition C6).
Arms: A local-minilm (baseline), B local-embeddinggemma@512 float32,
C local-embeddinggemma@512 int8+rescore (proposed default), D
local-qwen3-0.6b@512 (vector-only, always run, gates selectability only).
Gates computed and reported per the final plan: C>=A on recall@10 (ship
gate), |C-B|<=1pp at every K (quantize bug detector), B-vs-A ungated
headline, D-not-pathologically-worse-than-A.
Verified structurally only: all imports resolve against dist/ with correct
function signatures, and the script fails at exactly the expected point
(missing machine-local QFILE fixture) when run. NOT executed end-to-end —
arms B/C/D need real ONNX model downloads (EmbeddingGemma ~600MB,
Qwen3 ~1.5GB) this environment did not fetch, and the QFILE corpus
fixture is machine-local and uncommitted, same convention as the other
runners in this directory. No measurement numbers exist yet; do not flip
any default off a run of this script without a human reviewing the
results doc it feeds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
architecture.md, README.md, CHANGELOG.md updated for the two new local embedding providers (local-embeddinggemma, local-qwen3-0.6b), the embeddings.dim / embeddings.quantize config keys, and the int8 scan-then-rescore vec-index representation. Deliberately does NOT claim measured footprint, cold-reindex, query- latency, or RSS numbers for the new providers, and does not update the vault-init template default — both are gated on the governing spec's Phase 0 spike and Phase 5 recall-bench, neither of which has run against a real model in this environment. Each new-provider description carries an explicit verification-honesty note pointing at the corresponding [TRAINING]/[HYPOTHESIS] labels in the source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d ratification Adds the 2026-07-26 risk-triaged-ratification spec's Decision 3 storage layer: decision_kind, reason_category, amended_diff, and staged_by_principal follow the decided_by_principal/run_id precedent exactly (JSONL + row type only, no sqlite DDL). staged_by_principal is the authenticated access.user at stage time, closing the C4 laundering/poisoning hole where the witness and the future risk score's W term keyed only on the unauthenticated, caller- claimed proposed_by string. Per Mihir's 2026-07-27 decision resolving the spec's Decision-1-vs- kill-condition-#1 contradiction, decision records also gain a non- authoritative risk_at_decision snapshot (JSONL-only, never mirrored to sqlite, never read for queue ordering). Adds the shared proposalTallies implementation (total/ratified/rejected/ expired/pending/edited/byCategory, keyed on stagedByPrincipal ?? proposedBy) so the witness and the risk scorer's W term can never drift apart. Deletes pendingLintItems/listPendingForLint — superseded by the risk-ranked queue item in src/curation/risk.ts (added next). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New src/curation/risk.ts (2026-07-26 spec, Decision 1; final plan Phase 4). rankPendingActions computes a [0,1] risk score per pending staged action as a weighted sum of six deterministic terms (K action-kind, D diff-size, B blast radius, T open tension, C conflict/retry, W proposer track record) — pure, no I/O, recomputed on every call, never stored. Final-plan dispositions folded in directly: - C3: diffBucket (small/medium/large) is defined over raw serialized bytes, decoupled from D's log-scaled formula — D<0.25 is unreachable in bytes that small, so a D-based bucket made 'small' mathematically unreachable. - C1: C's clause (b) drops 'expired' (an expiry is reviewer capacity, not a human declining — W already prices it) and adds a later-ratified-clears rule for the same (actionType, targetPath) pair. - C5: tension endpoints are canonicalized via the vault's existing link-resolution before the T-term comparison, with raw-string fallback for unresolvable endpoints. - C4: W reads proposalTallies, keyed on the authenticated stager. - B probes direct (distance-1) inbound only — no BFS — and is computed per vantage, with a hidden remainder bumping the term by a coarse notch (Decision 4) instead of ever surfacing an exact count. riskForAction resolves one action's current score — used by vault_ratify next for the non-authoritative risk_at_decision snapshot. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implements Decisions 2 + 3 of the 2026-07-26 risk-triaged-ratification spec
in vault_ratify:
- Batch: an 'ids' array alternative to 'id', capped at BATCH_RATIFY_MAX (20),
no duplicates, no empty array — an explicit list, never a threshold or an
"all pending" sentinel. Each id runs the full per-action path (pending
check, tier-0 gates, dispatch) independently; a gate-blocked or failing id
leaves that action pending and the batch continues with per-id outcomes,
never a rollback. The staged-actions log is collapsed once per call; docs
load once and reload only after an applied dispatch mutates the vault
(invalidate-on-write) — the C2 cost hoists. Re-issuing an interrupted batch
is the documented recovery path: already-landed ids report a 'not pending'
outcome and the remainder applies.
- reason_category (closed 8-value enum) is now REQUIRED on reject — an
intentional, spec-mandated break from the prior optional contract for
reject callers only; approve-path callers are untouched (C6 disposition).
The error enumerates the categories so a caller self-corrects in one
round trip.
- edit-then-approve: an optional amended_diff on a single-id approve
dispatches the amendment instead of the staged diff — the tier-0 gates run
against it too — and the decision record keeps both what was proposed and
what landed. Under shadow_mode, amended_diff errors instead of silently
discarding the amendment (C7): shadow mode records no decisions of any
kind, so there is nowhere honest to put it.
- vault_stage_action now stamps the authenticated access.user as
staged_by_principal (C4); proposed_by stays claimed-agent display metadata.
- Every decision additionally gets a non-authoritative risk_at_decision
snapshot from src/curation/risk.ts, computed once per call over the
hoisted pre-decision queue state.
The single-action approve/reject path is extracted into
approveOneAction/rejectOneAction so a single 'id' call and a batch share one
implementation — a single-id call keeps today's RatifyResult shape (plus an
optional decision_kind); a batch returns {decision, results, succeeded,
failed}. Both outputSchema shapes ride one anyOf (MCP requires
type:'object' at the schema root, which both branches already satisfy).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sal records buildWitness's hand-rolled proposal tally loop is replaced by proposalTallies (src/curation/staged-actions.ts) — one implementation shared with the risk scorer's W term, so the two can never drift apart. Each principal's proposals record gains edited (a subset of ratified, not additional to it) and byCategory. Tallies key on stagedByPrincipal ?? proposedBy (C4), closing the laundering/poisoning hole named in the 2026-07-26 risk-triaged-ratification spec: rotating the unauthenticated proposed_by string no longer resets a principal's track record to the Laplace midpoint, and junk staged under a rival's claimed name counts against the actual (authenticated) stager. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… disclosure gap Implements Decisions 1/2/4 of the 2026-07-26 risk-triaged-ratification spec in runLint/vault_lint. The staged-actions section is now risk descending with soonest-to-expire as the tiebreak (inverting the prior expiry-only sort) via rankPendingActions (src/curation/risk.ts). Each item grows risk, proposerTrackRecord, diffBucket, blast, openTension, and conflict. Closes a live disclosure gap named by Decision 4: runLint previously passed the staged-actions list through UNFILTERED — pathVisible was applied to every other lint surface but not this one, so vault_lint could name target paths of pending actions in collections the caller could not read. The queue listing is now filtered to the caller's vantage like every other finding; the hidden remainder is reported coarsened via the new hiddenStagedActions field (none/some/many), never an exact count. The B and T terms are also computed per vantage internally (direct inbound counts and tension-endpoint matches respect pathVisible), consistent with the rest of the B'/#217 posture. tensions.md is now read once in runLint and threaded into both computeTensionHealth (signature change: takes the list instead of loading it — vault-global, unfiltered by design) and rankPendingActions (which applies pathVisible internally) — preserving the "each log is read ONCE" discipline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, amend per review Flips the 2026-07-26 risk-triaged-ratification spec's status line to implemented (2026-07-28), and applies the final plan's Phase 6 spec-text amendments in place (the spec was still 'proposed' when implementation started, so these are corrections, not silent deviations): - Decision 1 gains the narrow risk_at_decision carve-out (Mihir's 2026-07-27 decision resolving the Decision-1-vs-kill-condition-#1 contradiction) and clarifies B probes direct inbound only, no BFS. - The D bullet notes the diff-size display bucket is byte-based, decoupled from D's own formula (C3). - The C bullet drops `expired` and gains the later-ratify-clears rule (C1). - The T bullet notes endpoint canonicalization (C5). - The W bullet notes staged_by_principal keying (C4). - Kill condition #1 notes it is now evaluable via risk_at_decision. docs/architecture.md's Staged actions section documents batch ratify, decision_kind/reason_category, edit-then-approve, and the derived/ never-stored/per-vantage risk score with its risk_at_decision snapshot; the Honest assessment section cites the spec's own three kill conditions instead of only the pre-existing advisory-wager prose. Project CLAUDE.md gets one Key decisions bullet summarizing the same. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Independence-aware promotion, Decisions 1-2 (2026-07-26 spec). Each edge
observe may now carry an fp {inputs, principal, model, prompt}; inputs is a
sha256 over the (path, content-hash) set actually read, principal is the
authenticated identity, model/prompt are caller-attested. Store-side:
observeEdge validates and records fp; collapse() partitions COUNTED votes
into evidence-class equivalence classes (agree on inputs+principal+model;
prompt excluded per the decorrelation verdict) and computes a discounted
k_eff = Sigma_classes Sigma_j rho^(j-1) at rho=0.5, alongside the existing raw
k_survived. deriveEdge/deriveEdgeFromRow expose kEff and strengthIndependent
(agedStrength over k_eff) on DerivesFromEdge; live strength/status are
untouched (shadow posture, Decision 4).
derives_from_edges gains a k_eff REAL column (schema 11 -> 12, drop-and-
rebuild, index.db is ephemeral). New read helpers edgeEvidenceClasses and
independenceCalibrationView collapse the jsonl on demand for the revision
loop and the future lint calibration section.
The consolidation loop's writers (revision.ts, birth.ts) now record a real
fp on every surviving observe: inputs hashed over the exact truncated prompt
bodies, principal CONSOLIDATE_AGENT, model the panel's model id, prompt the
template/mode id ("revision/<axis>" | "birth/foundational") — replacing the
round-robin store axis's semantic claim (the axis field itself stays; it
still drives the existing replay guard).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (PR-2) Independence-aware promotion, Decisions 3-4 (2026-07-26 spec, amended). New module src/consolidate/independence.ts: independenceVerdict is pure math over evidence-class counts (marginal k_eff gain vs EDGE_NEEDS_REVIEW_MIN_GAIN, strict); appendIndependenceShadow / listIndependenceShadow manage .daftari/independence-shadow.jsonl (its own per-collapse journal, distinct from shadow-actions.jsonl); needsReviewTensionInput renders the correlated-only-survival tension body as a guaranteed single-line, human-readable class breakdown (structured descriptors, never raw newline-joined class keys, so tensions.md's line-oriented round-trip holds). revision.ts: fetches the edge's pre-panel evidence classes once at panel start; on an admitted majority-survives decision, computes the panel's marginal k_eff gain against them and always journals one shadow row (including tie/gated/fails/no-vote, wouldDecision null). Shipped default (independenceGraduated: false) leaves the two-way decision and writes untouched — the verdict feeds only the journal and the trace. Graduated + correlated-only flips the decision to "needs-review": no observes, no further envelope consult, a tension instead. A failed pre-panel classes read degrades to shadow-off for that panel only and never changes the live decision. index.ts (CLI wiring): independence_graduated config flag (default false, opt-in, no explicit-declaration tripwire); runRevisionLoop parks a due edge under an OPEN needs-review tension before Phase-1 elicitation (C1) — zero LLM calls, no trace row, panelsSkippedNeedsReview counted; exit report gains needs_review_emitted (suffixed "shadowed — would-be" pre-graduation) and panels_skipped_needs_review (graduated only). vault-gitignore.ts (C7): adds independence-shadow.jsonl and the previously- ungitignored revision-trace.jsonl; ensureVaultGitignore upgrades from block-marker idempotence to per-line reconciliation, so a vault whose .gitignore predates a later VAULT_GITIGNORE addition picks up exactly the missing lines instead of never retrofitting. Amends the spec's Decision 3 to state the marginal-gain threshold as the operative rule (a second vote in a count-1 class gains exactly 0.5 and accrues — "one half-fresh vote"), recasts "opens a new equivalence class" as the intuition, and adds the parking sentence: decay continues while a needs-review tension is open, by design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (PR-3) Independence-aware promotion surfaces (2026-07-26 spec, PR-3). vault_edge_observe gains evidence_paths / model / prompt_id: fp.inputs is server-computed (sha256 over the named paths' CURRENT full bytes, each path validated to exist), fp.principal comes only from the access context (never from args or the free-text observed_by), fp.model/fp.prompt are recorded verbatim as caller-attested — the tool description states the trust split explicitly and names this as the human resolution path for a needs-review tension. derivesFromEdgeSchema (shared by all three edge tools) gains required kEff/strengthIndependent, described as shadow calibration values. New src/curation/independence-calibration.ts: independenceCalibrationSummaryOf is a pure aggregation over independenceCalibrationView + the shadow journal — k-vs-k_eff distribution, would-drop-below-trigger counts (split legacy-only vs signal), the would-be needs-review rate with an "informative panel" denominator (C5: excludes a legacy edge's first-ever fingerprinted panel, which reads would_accrue by construction and would otherwise degenerate the rate to ~0), the legacy-unfingerprinted fraction, and total operator-attested (non-loop-principal) counted votes (C3). Wired into vault_lint's independenceCalibration section — vault-global counts only, no paths, error-tolerant on both underlying reads (lint stays advisory). docs/architecture.md gains a calibration section covering the shadow stream, the graduation criteria with the warm-up rule, both kill conditions and where each would be executed, the fingerprint's byte-stability caveat (C4: any endpoint edit mints a fresh class, so the flagged population skews toward byte-frozen docs), and the attested/computed trust split (C3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…IT verification
Implements Phases 1-5 of the citation-anchors-jit final plan (spec
2026-07-26-citation-anchors-jit-verification-design.md, Decisions 1-2 and 4):
- src/anchors/pin.ts: the #L<start>-<end>@<sha> pin grammar (splitPin,
end-anchored, byte-identical passthrough for unpinned entries) and a
tightened looksLikeMalformedPin heuristic (C11).
- src/anchors/classify.ts: the 4-step classifier, hardened per C5 (realpath
symlink confinement, lifted to src/utils/paths.ts as symlinkSafeExistsWithin
and re-exported from src/audit/collect.ts) and C7 (CRLF normalization,
16-char trivial-content floor, first-occurrence relocation).
- src/utils/git.ts: hashObjects/hashObject/blobAtHead/blobSize/catBlob —
hashObjects batches one `git hash-object` invocation per repo (C1), so the
all-intact read path costs one subprocess per referenced repo, not one per
pin.
- src/utils/config.ts: `code_repos` / `jit_anchors` config blocks, and a new
per-role `code_repo_visibility` grant (default off) gating the returned
`anchors` annotation per the 2026-07-27 decision from Mihir.
- src/tools/read.ts: vault_read gains the role-gated `anchors` field (null
when there's nothing to say, same contract as decay/structural) and
Decision 4's intact-pin decay-banner softening; computeDecay itself stays
pure. Telemetry (anchors_moved/missing/errored) is recorded to the read log
unfiltered by role, matching the broken_upstream precedent (C8's `errored`
count keeps the softening from quantifying over a censored sample).
Tests: test/anchors/{pin,classify}.test.ts, git.test.ts and config.test.ts
additions, and test/tools/read-anchors.test.ts covering the cap, gating,
telemetry, and softening behavior end to end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t --pin
Implements Phase 6-7 of the citation-anchors-jit final plan (Decision 3
batch audit integration, Decision 5 backfill, plan resolutions C2/C6/C10/C11):
- src/audit/types.ts: DescribesEdge.pin, PinFinding/PinState, RegistryMismatch,
and pin totals on AuditReport.
- src/audit/describes.ts: classifyDescribesEdges strips pins via
src/anchors/pin.ts before parseDescribesEntry; targetPath stays pin-free so
checkDescribesRefs/runSemanticCheck are unaffected.
- src/audit/checks/pins.ts: checkPins batches classification per target repo
(same C1 primitive as the read path).
- src/audit/docs-repo.ts: resolveSingleDocsRepo, generalized from the former
resolveTensionVault so --pin --apply can share it (C10).
- src/audit/pin.ts: daftari audit --pin (plan, default) / --pin --apply.
Plan batch-hashes the working tree and skips any file that differs from
HEAD (C6 — a dirty pin would classify `moved` on arrival); apply refuses
against any live daftari process holding the docs vault's process.lock,
naming its pid/mode, no override flag in v1 (C10).
- src/audit/index.ts: registry cross-check warns (never fails) when a pinned
repo name resolves in exactly one of {audit registry, docs repo's own
code_repos} or to different realpaths in both (C2); moved-first stable
partition ahead of the --max-semantic slice; missing-pin auto-tension,
deduplicated by title across runs, never fired by a bare `moved` (C11);
the "--auto-tension has no effect" warning now fires only when there are
also zero pinned bindings.
- src/audit/report.ts: pin verification table, registry-mismatch notes, and
renderPinPlan/renderPinApplyResult formatters.
Tests: test/audit/pins.test.ts (checkPins unit coverage), test/audit/pin-cli.test.ts
(plan/apply CLI, dirty-skip, live-holder refusal via a real spawned process,
registry mismatch in all three directions, missing-pin dedupe, moved-first
ordering), plus report.test.ts and describes.test.ts updates for the new
DescribesEdge/AuditReport shape.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tening Implements Phase 8 of the citation-anchors-jit final plan: - src/curation/lint.ts: new malformedPins check (pure string scan, LINT_CHECKS appended per house style); Decision-4 softened stale copy for docs whose pinned bindings are all intact, batched per the C1 primitive (one hashObjects call per referenced repo for the WHOLE lint run, not per doc) and budgeted at LINT_PIN_STEP3_BUDGET (200) step-3 classifications per run (C3) — verdicts are memoised by full pin identity so a triple recurring across docs is classified once, and docs beyond the budget simply don't get the softened copy (advisory degradation, never a lint failure). - LintReport/VaultLintResult/vault_lint's outputSchema gain `pinsClassified` — the budget-spend counter that makes a slowdown attributable. Tests: test/curation/lint-anchors.test.ts covers malformedPins, softening present/absent (moved pin, fresh doc, jit_anchors: false), step-3 via a range pin, cross-doc verdict memoisation, and the 200-classification budget boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tripwire
Implements Phase 9 of the citation-anchors-jit final plan:
- docs/architecture.md: new "Citation anchors" subsection under Doc-to-code
coherence — the pin grammar, the JIT read-path check, the end-anchored-path
ambiguity ("the pin wins"; a real collision surfaces as broken_describes at
audit time), and the code_repos visibility posture gated by
code_repo_visibility.
- CLAUDE.md: src/anchors/ added to the code map; a Key Decisions line stating
pins are advisory/annotate-only, the read-path check is batched git
plumbing with silence on failure, and code_repos visibility is
role-gated.
- docs/superpowers/specs/2026-07-26-citation-anchors-jit-verification-design.md:
flipped to implemented; corrected the "no new disclosure surface" claim
(C4) with the actual posture (gated per the 2026-07-27 decision from
Mihir); amended the kill-condition paragraph so condition (b) is stated as
measuring only the observable subset, with absence alone insufficient to
kill the check (C9); documented the C1 batching redesign in the
"cheap by construction" paragraph; updated the annotation shape to include
`errored`.
- test/anchors/perf.test.ts: the spec's CI tripwire — 24 intact pins across
2 repos classify under 150ms — runs unconditionally (not RUN_PERF-gated)
since it's fast and deterministic.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t task briefs
Implements Decisions 1-3 of docs/superpowers/specs/2026-07-26-context-packs-
progressive-disclosure-design.md (final plan Phases 1-2), as revised by
.jugalbandi/context-packs-disclosure/final-plan.md.
Phase 1 (registry + vault_tools):
- Extract the tool registry from server.ts into src/tools/registry.ts to
break the import cycle vault_tools' self-closure otherwise creates. The
module exports allTools/registeredToolNames/allRegisteredTools (re-exported
from server.ts for back-compat) and serializeToolDefinition, the single
wire-shape serializer BOTH the ListTools handler and vault_tools' expand
mode use, so they cannot drift.
- New required `oneLine` field on ToolDefinition, written for all 33 prior
tool definitions plus vault_tools/vault_context.
- vault_tools: index mode ({name, oneLine} per tool) or expand mode (full
schemas for named names). Reads the registry minus the vault's config
`exclude` list — exclude always wins (#104, C2) — but tier/include never
narrow it, since discoverability of tiered-out tools is the point. An
excluded or unregistered expand name lands in `unknown`, never a batch
error.
- advertisedSurfaceCost(tools) + a startup log line in src/index.ts and
src/serve/index.ts reporting the measured cost of the tier-resolved
exposed set. Default tools.tier stays 'full' this wave — the log is the
notice, not the flip.
- vault_tools and vault_context join CORE_TOOLS.
Phase 2 (vault_context):
- src/context/estimate.ts (chars/4 token estimator) and
src/context/assemble.ts (pure, deterministic selection + greedy budget-cut
+ markdown templating over an already-enriched PackEntry[] — no I/O, no
LLM, no database). Decision 3's refusal is structurally enforced: a
tension flag always renders both claims from claimSelf/claimOther, never a
blended sentence.
- src/tools/context.ts: the assembly pipeline — hybrid retrieval, RBAC
filter before any budgeting, supersession dedup (a collapsed chain's
flags are ALL keyed on the head's own index row, never a stale member's —
C3), then assembleContextPack. hidden_remainder is a lower-bound signal
over OBSERVABLE withholding only (RBAC-dropped BM25-side pool candidates,
dropped coverage additions, restricted supersession hops) — never a
completeness claim (C4), since the vector half is already RBAC-pushdown-
scrubbed and structurally invisible to the count.
- Budget parsing (C9): absent/non-finite defaults to 4000; below 500 is an
error (never silently delivers less than stated); above 20000 clamps down
silently. Zero-hit and zero-entry-fits-budget are distinct ok() outcomes
with distinct body text.
- src/tools/search.ts: split annotateAndLogServedHits into
annotateUpstreamHits (buckets + pending log entries, writes nothing) and
logServedHits (the batch append) — C1. vault_search/vault_search_related
call both back-to-back, behavior unchanged; vault_context logs only the
entries that survive its budget cut, keyed on the rendered (head) paths.
- src/storage/index-db.ts: documents.updated_by column (schema 12 -> 13,
full-rebuild migration, no in-place statement needed) backing the pack's
provenance flag; populated in src/search/reindex.ts's stageOne.
Tests: test/tools/registry.test.ts, test/context/{estimate,assemble}.test.ts,
test/tools/context.test.ts, test/context/no-court-import.test.ts (the
court-import tripwire), plus extensions to test/server.test.ts and
test/storage/index-db.test.ts for the schema bump. CLAUDE.md and
docs/architecture.md updated with the new module and both tools.
Deferred: Phase 3 (eval --condition pack) lands in the next commit; Phase 4
(the measurement gate and the default-tier flip) is explicitly out of scope
for this wave per the plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ameters
Implements Decision 4 of docs/superpowers/specs/2026-07-26-context-packs-
progressive-disclosure-design.md (final plan Phase 3), as revised by
.jugalbandi/context-packs-disclosure/final-plan.md.
- src/eval/pack-condition.ts: runPackAnswerer builds a vault_context brief
per (question, k) and hands it to the answerer via a single llm.complete
call — no tools. access is undefined, the same posture tool-surface.ts
already takes; vault_context is deliberately NOT added to the eval
tool-loop surface, so the baseline `tools` condition stays a clean
control. Same persist/--resume shape as runAnswerer. New
PACK_ANSWERER_SYSTEM_PROMPT in prompts.ts — a new prompt for a new
condition, not an edit to an existing one, so PROMPT_VERSION is untouched.
- C5 (maxToolCalls is not a true cap as specified — parallel tool_use blocks
can overshoot a naive check-then-execute guard): src/eval/llm.ts and the
OpenRouter twin now execute only the first `maxToolCalls - used` calls in
a round, in block order; every excess id still gets a tool_result (the API
requires one per id) carrying TOOL_BUDGET_EXHAUSTED_MESSAGE and is never
counted; the next request omits `tools` entirely, forcing a final answer.
Realized call count is now a true, enforced upper bound.
- C8 (run parameters were not persisted, so --resume could blend a capped
run into an uncapped one): EvalRun/Score/HistoryEntry gain
condition?/pack_budget?/max_tool_calls? (all absent on a legacy artifact,
which loads and scores as an uncapped `tools` run). Capped tools runs mint
a `-tools-c{N}` id segment, pack runs `-pack-b{budget}`, symmetric with
each other. --resume compares the persisted condition/budget/cap against
the CLI flags and refuses (exit 2) on any mismatch — never a silent
override. TierScore gains mean_tokens (the pack-condition twin of
trace_efficiency); the score CLI's printed header is self-describing
(condition/budget/cap) and each tier line now prints tokens/correct
alongside tool-call efficiency.
- src/eval/index.ts: --condition tools|pack (default tools, invalid value
exits 2), --budget (default 4000, delegated to vault_context's own
validation), --max-tool-calls (default uncapped). HELP text updated.
Tests: test/eval/pack-condition.test.ts (zero-tool-call traces with pack
metadata, resume-skips-completed, the brief handed to the LLM is byte-
identical to vault_context's own return); maxToolCalls overshoot coverage in
test/eval/{llm,llm-openrouter}.test.ts for both transports; CLI-level id
minting, resume-mismatch (all three fields), and legacy-artifact-still-
scores coverage in test/eval/index.test.ts; mean_tokens aggregation in
test/eval/score.test.ts; condition/max_tool_calls stamping in
test/eval/run.test.ts.
Deferred: Phase 4 (running the gate at n=60/k=2 against scratch vault
copies, publishing the results artifact, and the gated default-tier flip)
is explicitly out of scope for this wave — it requires a real LLM spend
under Mihir's OpenRouter budget authority and is not something to fabricate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CP clients test/e2e/server.e2e.test.ts (the built-artifact stdio e2e suite) caught it: the MCP SDK's client-side Zod validator requires every tool's outputSchema to declare type:"object" at the root. vault_tools' outputSchema used a bare oneOf with no sibling type, which validates fine under ajv (the project's own strict-compile test in test/server.test.ts) but fails the real MCP client handshake — a gap between "compiles as JSON Schema" and "is a valid MCP Tool.outputSchema" that only an over-the-wire e2e test exercises. Add type:"object" alongside the existing oneOf; the discriminated-on-mode shape is unchanged. Only caught after `npm run build` (this suite runs against dist/, not tsc --noEmit) — a reminder that the definition of done for this branch is the built artifact, not just the type checker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both lineages claimed SCHEMA_VERSION 11 (#309's valid-time missed-bump fix on main; contextual chunking on this branch). The wave chain is renumbered 12-14 so no value is claimed twice and every pre-merge index, either lineage, rebuilds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… migration test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| @@ -0,0 +1,146 @@ | |||
| // local-bge-m3 — BAAI/bge-reranker-v2-m3 cross-encoder, ONNX q8, run locally | |||
Contributor
There was a problem hiding this comment.
Test coverage gap: this new 146-line module has real logic (sub-batch looping at RERANK_BATCH_SIZE, sigmoid, the memoised-promise reset-on-failure pattern in getModel) but no mocked unit test exercises any of it.
test/search/local-bge-m3-smoke.test.tsisdescribe.skipIf(!process.env.DAFTARI_BGE_SMOKE)— skipped by default, so it never runs innpm test.test/search/rerank-provider.test.tsexplicitly says (line 5) it "never touches the real local-bge-m3 model" — it only checks registry wiring (setRerankProvider('local-bge-m3')returns an object withid === 'local-bge-m3'), never calling.warm()/.rerank()on the real provider.
So the PR's "2921 passed" run never actually executes this file's batching/sigmoid/error-path logic. Its sibling providers (local-embeddinggemma.ts, local-qwen3.ts) both get real mocked unit tests under test/search/providers/ that exercise their logic with a mocked @huggingface/transformers — this file would benefit from the same treatment (per CLAUDE.md: "Tests mirror src/ structure").
Contributor
Review summaryReviewed the full diff locally (git diff 2fce607 HEAD, 162 files / +20225 -902) against CLAUDE.md's hard invariants: frontmatter-only metadata, derived/ephemeral SQLite index, git as version layer, advisory-only curation, existence-disclosure rules, Result<T, Error> (no throws from tool handlers), no classes. Invariant checks -- all clean:
Correctness bugs found (could not anchor as inline comments -- see why below):
Test coverage gap (posted inline on No other correctness, simplification, or efficiency issues surfaced with enough confidence to report -- the diff is large (8 specs, 45 commits) but consistently disciplined about the project's conventions (Result types, provisional/shadow flags, HYPOTHESIS-labeled uncertainty, existence-disclosure reuse). Generated with Claude Code |
Owner
Author
|
@claude resolve conflicts |
Contributor
|
Claude finished @mavaali's task in 22m 43s —— View job Resolving merge conflicts
What was going on: How it was resolved (
Pushed as Caveat — please have CI confirm this, and update |
… landing (#312) origin/main gained one commit past this branch's merge-base: PR #312, "MCP 2026-07-28: stateless serve (Decision 1), ratify elicitation (Decision 5), Decision 4 killed" — landing exactly the work this branch's own MCP-readiness spec had deferred pending an SDK gate that failed as of 2026-07-27. The gate evidently passed by the time #312 shipped (2026-07-28), so main is now the newer, authoritative state for that surface. Reconciled by hand (git merge/fetch/reset/checkout were unavailable in this sandbox — no destructive git ops, no network for npm install): - src/server.ts, src/serve/index.ts, src/index.ts: adopted main's v2 SDK (@modelcontextprotocol/server/node/client) migration — stateless per-request serve, serveStdio dual-era stdio — layered under this branch's own tools/registry.ts extraction, CORE_TOOLS additions (vault_tools, vault_context), and formatSuccessResult presentation helper. - src/tools/staged-actions.ts: added describeRatifyElicitation (Decision 5) alongside this branch's batch `ids` / reason_category / amended_diff / risk_at_decision vaultRatify rewrite. Elicitation is single-`id` only; a batch `ids` call always requires an explicit `decision` (server.ts guards on args.ids === undefined before entering the elicitation round). - docs spec doc, CHANGELOG.md, CLAUDE.md: adopted main's landed-state narrative (superseding this branch's "gate failed, deferred" framing) and merged in this branch's own unrelated Key-decisions bullets. - package.json/package-lock.json: adopted main's SDK dependency swap, kept this branch's `ajv` devDependency (already resolved in main's lockfile at the same version, so no new resolution needed). - test/ratify-elicitation.test.ts (new from main), test/serve/oauth.test.ts, test/serve/serve.test.ts: adopted main's versions verbatim — untouched by this branch. Also discarded a stray pre-existing uncommitted edit to CLAUDE.md (unrelated to this request) that had dropped several of this branch's own Key-decisions bullets. NOT independently verified: npm install (no network egress in this sandbox) and tsc/npm test (both blocked by the sandbox's command allowlist) could not be run here. CI should be treated as the first real build/test signal for this commit. Co-authored-by: Mihir Wagle <40620108+mavaali@users.noreply.github.com>
Owner
Author
|
@claude resolve conflicts |
Contributor
|
Claude finished @mavaali's task in 4m 16s —— View job Resolving merge conflicts
Finding: no new conflicts to resolve. This branch's conflicts with
I checked this commit against the branch:
Caveat on verification method: this sandbox has no network egress, same limitation as the prior run — Also checked CI: the only run recorded against this PR's head commit ( No commits pushed this round — the branch was already conflict-free relative to current |
mavaali
enabled auto-merge (squash)
August 7, 2026 17:14
This was referenced Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Sequential implementation of the eight remaining 2026-07-26 enhancement specs, each first run through the Jugalbandi protocol (Proposer → Challenger → Resolver in isolated contexts; 75 challenges, 71 accepted, 1 rejected, 3 escalated to Mihir and decided). The resolved per-spec plans — the review documents for this PR — are the eight
final-plan.mdfiles referenced below (run artifacts in.jugalbandi/, gitignored; ask if you want them attached).Per-spec commit map
9559846…e46b920c8d58a8…4fed657e96b5cd…c13a0b1none3dae327…02f0451vec_int8()wrapper bug found empirically2453d41…b860ddarisk_at_decisionsnapshot per Mihir's decisioned2c182,aa29601,b99c989independence_graduated42bdce2…a244ff3code_repo_visibilityrole gate from day one per Mihir's decision3f56357,1a7af0f,20dc919Verification
npm run buildclean;npm test2921 passed / 0 failed (262 files); lint clean of new warnings. Three latent bugs caught and fixed by the new test rigor: tier2field_changes.beforedrop, sqlite-vec int8 insert shape,vault_toolsschema passing ajv but failing the MCP handshake.Deliberately not done (no fabricated measurements)
Bench/eval-gated default flips (fusion, embeddings, tool tier), real-model spikes, independence graduation, and the SDK-gated MCP work all remain gated on real runs.
🤖 Generated with Claude Code