perf(mcp): build the cold-scan trigram index in parallel (~4.5x on the build phase)

docs/mcp.md

@@ -261,7 +261,35 @@ The installer does this for you.
 
 ---
 
-## 7. Going deeper
+## 7. Response verbosity — lean vs rich (token cost)
+
+Every MCP tool result carries the data block the model consumes
+(`audience: assistant`). Interactive clients also get two `audience: user`
+blocks: a colored one-line summary and a follow-up hint. Well-behaved clients
+render those in a preview pane and keep them out of the model context — but
+many forward everything to the model, where they cost output tokens for output
+the model can't render (they add ~34% to a small result like `codedb_symbol`).
+
+codedb decides per session, from the `clientInfo.name` sent at `initialize`:
+
+- **Agent harnesses default lean** (data block only) — `claude-code`, `codex`,
+  and any client not on the rich allowlist.
+- **Human-facing GUI clients get the rich blocks** — currently `claude-ai`
+  (Claude Desktop).
+
+Override the default:
+
+| Env var | Effect |
+|---|---|
+| `CODEDB_MCP_LEAN=1` | Force lean for every client (data block only). |
+| `CODEDB_MCP_RICH=1` | Force rich for every client. |
+| `CODEDB_MCP_RICH_CLIENTS=name1,name2` | Add clients (by `clientInfo.name`, case-insensitive) to the rich allowlist. |
+
+`CODEDB_MCP_LEAN` takes precedence over `CODEDB_MCP_RICH`.
+
+---
+
+## 8. Going deeper
 
 - [Architecture](architecture.md) — engine internals, index layout
 - [CLI reference](cli.md) — every command, every flag

experiments/ranking/def-first-eval.md

@@ -0,0 +1,91 @@
+# Def-first ranking eval — methodology & result
+
+Harness: [`scripts/rank-eval.py`](../../scripts/rank-eval.py). Change it validated:
+**PR #665** (`codedb_search` now ranks the *defining* file first).
+
+## What it measures
+
+For a set of symbol-shaped queries, does the file that actually **defines** the
+symbol rank at the top of `codedb_search`? Two metrics over the gold set:
+
+- **def-file-#1** — the def file is the first result.
+- **def-file-top3** — the def file is in the top 3.
+
+Gold cases are `(query, expected_def_file)` pairs, deliberately weighted toward
+symbols whose def file has a basename that does *not* match the query (e.g.
+`searchContentRanked` → `src/explore.zig`) — the case plain hit-count + basename
+ranking gets wrong.
+
+## The confound this harness exists to kill: snapshot drift
+
+codedb indexes the current project and persists a snapshot under `$HOME/.codedb`.
+If you eval against the **codedb repo itself**, editing the ranking code
+re-indexes the corpus *mid-measurement*, so the score moves for reasons that have
+nothing to do with your change. The first def-first attempt looked like an 8→7
+**regression** — and reproduced 8→7 with the code **reverted**. That was 100%
+drift, not the code.
+
+The fix (what "pinned" means):
+
+1. **Frozen corpus** — a representative slice of the repo (`src/` + `CHANGELOG.md`
+   + `experiments/`) copied *out* of the repo into a temp dir. Never edited, so
+   the index is stable across builds.
+2. **Isolated `$HOME`** — a throwaway home dir, wiped per binary, so each build
+   re-indexes the *identical bytes* from scratch. The only variable across an A/B
+   is the binary.
+
+Result: **deterministic** — repeated runs of the same binary are byte-identical,
+so any delta is attributable to the code.
+
+## Gotchas (encoded in the harness)
+
+- **Temp-root guard** — the corpus lives under a temp dir, which codedb refuses to
+  index by default, so the child gets `CODEDB_ALLOW_TEMP=1`.
+- **Async snapshot load** — a `tools/call` fired immediately after `initialize`
+  hits `state=loading_snapshot` and returns 0 results. The harness spaces them in
+  the pipe (`printf init; sleep 6; printf query; sleep 2`) and pre-warms once with
+  `codedb . index`.
+- **Debug is fine** — result *order* is version-independent; only indexing speed
+  differs. Build once with `zig build`; no ReleaseFast needed for a ranking A/B.
+- **`CODEDB_NO_CLI_DAEMON=1`** — no shared warm daemon, so each run is
+  self-contained and can't inherit another run's state.
+
+## Run it
+
+```sh
+zig build                                   # Debug build is fine for ranking
+python3 scripts/rank-eval.py                # self-freezes a corpus, evals zig-out/bin/codedb
+
+# A/B two builds against ONE frozen corpus:
+python3 scripts/rank-eval.py --binary /tmp/codedb.baseline --corpus /tmp/frozen
+python3 scripts/rank-eval.py --binary /tmp/codedb.head     --corpus /tmp/frozen
+```
+
+Custom gold set: `--cases mycases.json` where the file is a JSON list of
+`[query, expected_def_file]` pairs.
+
+## Result (PR #665)
+
+Same frozen corpus, Debug builds, only the binary differs:
+
+| binary   | def-file-#1 | def-file-top3 |
+|----------|-------------|---------------|
+| baseline | 8/10        | 9/10          |
+| def-first| **9/10**    | 9/10          |
+
+`searchContentRanked` moved pos 3 → **#1**; **zero regressions**. The change:
+tier0 gets a `defines` flag (via `fileDefinesSymbol`) and `+20` to the rank prior
+for defining files — same result *set*, better *order*. Regression test:
+`def-first: renderPlainSearch ranks the defining file above higher-count mentions`
+in `src/test_search.zig` (fails on baseline, passes with the fix).
+
+## Known holdout / next pass
+
+`renderPlainSearch` as a query stays at pos 4: it bails from the tier0 fast path
+to `searchContentAuto`, a **different** path where docs aren't demoted
+(`CHANGELOG.md` / `experiments/**/*.md` outrank the def) and this boost doesn't
+apply. Two follow-ups for the next def-first pass:
+
+1. Apply def-first + doc demotion in the `searchContentAuto` fallback too.
+2. Surface the def *line* within the file (today the def file floats up but the
+   render still shows the file's first hit line, often a mention, not the def).

experiments/ranking/todo.md

@@ -19,6 +19,17 @@ signals in isolation — 3 experiments proved that flat-lines; the win is the
       lexically-relevant) competitors in lockstep. See research.md "P0 result". Keep
       `line_count` as a P4 feature; it is inert alone.
 
+- [x] **Def-first: rank the defining file first in codedb_search** — **DONE → +1, SHIPPED (PR #665).**
+      `codedb_search`'s fast path (`renderPlainSearchUncached`) ordered tier0 by hit
+      count + basename prior only; the def-aware ranking lived in `searchContentRanked`,
+      which `codedb_search` never calls. Added a `defines` flag (via `fileDefinesSymbol`)
+      + `+20` rank prior for defining files. Measured on a **pinned** eval
+      (`scripts/rank-eval.py`, methodology in `def-first-eval.md`): def-file-#1 8/10→9/10,
+      zero regressions. The pinned corpus + isolated `$HOME` is what makes a ranking A/B
+      trustworthy — an earlier unpinned run showed a phantom 8→7 that reproduced with the
+      code reverted (snapshot drift). Holdout: `renderPlainSearch`-as-query bails to
+      `searchContentAuto` (docs not demoted there) — next pass.
+
 ## P1 — biggest signal gap (the real IRFL win)
 - [ ] **Git version-history signals** (AmaLgam / Rahman / BLIA). Mine `git log`
       once at index time:

install/install.sh

@@ -224,21 +224,74 @@ hooks_dir = os.path.join(home, ".claude", "hooks")
 settings_path = os.path.join(home, ".claude", "settings.json")
 os.makedirs(hooks_dir, exist_ok=True)
 
+# Opt-out: CODEDB_NO_HOOKS=1 at install time, or a persisted removal marker,
+# skips (re-)registering the PreToolUse block-legacy hook. Without this, any
+# reinstall/update silently re-adds a hook the user deliberately removed.
+no_hooks_marker = os.path.join(home, ".codedb", "no-hooks")
+skip_pretooluse = bool(os.environ.get("CODEDB_NO_HOOKS")) or os.path.exists(no_hooks_marker)
+if os.environ.get("CODEDB_NO_HOOKS") and not os.path.exists(no_hooks_marker):
+    os.makedirs(os.path.dirname(no_hooks_marker), exist_ok=True)
+    with open(no_hooks_marker, "w") as f:
+        f.write("")
+
 scripts = {
     "codedb-block-legacy.sh": r'''#!/bin/bash
+# codedb PreToolUse guard. Nudges agents from native file tools to codedb —
+# but ONLY inside a codedb-indexed repo, and never for paths outside it.
+# Fail-open by design (a nudge, not a wall). Disable entirely: CODEDB_NO_HOOKS=1.
+[ -n "$CODEDB_NO_HOOKS" ] && exit 0
 command -v jq >/dev/null 2>&1 || exit 0
 command -v codedb >/dev/null 2>&1 || exit 0
+
 INPUT=$(cat)
 CMD=$(echo "$INPUT" | jq -r '.tool_input.command // empty')
+[ -z "$CMD" ] && exit 0
+
 STRIPPED=$(echo "$CMD" | sed -E 's/^[[:space:]]*(env|sudo|command|builtin|exec|nohup)[[:space:]]+//')
 STRIPPED=$(echo "$STRIPPED" | sed -E 's/^[[:space:]]*[A-Za-z_][A-Za-z0-9_]*=[^[:space:]]*[[:space:]]+//')
 FIRST=$(echo "$STRIPPED" | awk '{print $1}')
+
+case "$FIRST" in
+  grep|rg|egrep|fgrep|cat|head|tail|sed|awk|find) ;;
+  *) exit 0 ;;
+esac
+
+# Scope 1: cwd must sit at/under a codedb-indexed root. Otherwise codedb has
+# nothing to offer here — allow the native tool (no blocking outside repos, on
+# unindexed dirs, or in ~/.claude).
+PWD_ABS=$(pwd -P)
+REPO_ROOT=""
+for pt in "$HOME"/.codedb/projects/*/project.txt; do
+  [ -f "$pt" ] || continue
+  root=$(head -1 "$pt" 2>/dev/null)
+  [ -z "$root" ] && continue
+  [ "$root" = "/" ] && continue        # degenerate root matches everything
+  [ "$root" = "$HOME" ] && continue    # home indexed as a project would block all of ~
+  if [ "$PWD_ABS" = "$root" ] || [ "${PWD_ABS#"$root"/}" != "$PWD_ABS" ]; then
+    REPO_ROOT="$root"; break
+  fi
+done
+[ -z "$REPO_ROOT" ] && exit 0
+
+# Scope 2: if the command names an ABSOLUTE path outside this repo (cat /etc/hosts,
+# grep x /tmp/f), codedb can't read it — allow. Relative paths are assumed
+# in-repo (cwd is in-repo). Fail-open: any out-of-repo absolute arg -> allow.
+set -f
+for tok in $STRIPPED; do
+  case "$tok" in
+    /*)
+      if [ "$tok" = "$REPO_ROOT" ] || [ "${tok#"$REPO_ROOT"/}" != "$tok" ]; then :; else set +f; exit 0; fi
+      ;;
+  esac
+done
+set +f
+
 case "$FIRST" in
-  grep|rg|egrep|fgrep) echo "BLOCKED: Use mcp__codedb__codedb_search instead of $FIRST. If codedb MCP is not connected, use Bash directly." >&2; exit 2 ;;
-  cat) echo "BLOCKED: Use mcp__codedb__codedb_read instead of cat. If codedb MCP is not connected, use Bash directly." >&2; exit 2 ;;
-  head|tail) echo "BLOCKED: Use mcp__codedb__codedb_read with line_start/line_end instead of $FIRST. If codedb MCP is not connected, use Bash directly." >&2; exit 2 ;;
-  sed|awk) echo "BLOCKED: Use mcp__codedb__codedb_edit instead of $FIRST. If codedb MCP is not connected, use Bash directly." >&2; exit 2 ;;
-  find) echo "BLOCKED: Use mcp__codedb__codedb_find or mcp__codedb__codedb_glob instead of find. If codedb MCP is not connected, use Bash directly." >&2; exit 2 ;;
+  grep|rg|egrep|fgrep) echo "BLOCKED in indexed repo ($REPO_ROOT): use codedb_search \"<text>\" (codedb_word for an exact identifier, codedb_callers for call sites) instead of $FIRST — ranked + fewer tokens. Native $FIRST is allowed outside this repo or with CODEDB_NO_HOOKS=1." >&2; exit 2 ;;
+  cat) echo "BLOCKED in indexed repo ($REPO_ROOT): use codedb_read path=<file> (codedb_outline first for a map) instead of cat. CODEDB_NO_HOOKS=1 to disable." >&2; exit 2 ;;
+  head|tail) echo "BLOCKED in indexed repo ($REPO_ROOT): use codedb_read path=<file> line_start=.. line_end=.. instead of $FIRST. CODEDB_NO_HOOKS=1 to disable." >&2; exit 2 ;;
+  sed|awk) echo "BLOCKED in indexed repo ($REPO_ROOT): use codedb_edit (op=str_replace) instead of $FIRST for edits. CODEDB_NO_HOOKS=1 to disable." >&2; exit 2 ;;
+  find) echo "BLOCKED in indexed repo ($REPO_ROOT): use codedb_find (fuzzy names) or codedb_glob (patterns) instead of find. CODEDB_NO_HOOKS=1 to disable." >&2; exit 2 ;;
 esac
 exit 0
 ''',
@@ -296,7 +349,8 @@ def merge_hook(event, new_entry):
         existing.append(new_entry)
     hooks[event] = existing
 
-merge_hook("PreToolUse", {"matcher": "Bash", "hooks": [{"type": "command", "command": "$HOME/.claude/hooks/codedb-block-legacy.sh"}]})
+if not skip_pretooluse:
+    merge_hook("PreToolUse", {"matcher": "Bash", "hooks": [{"type": "command", "command": "$HOME/.claude/hooks/codedb-block-legacy.sh"}]})
 merge_hook("SessionStart", {"matcher": "", "hooks": [{"type": "command", "command": "$HOME/.claude/hooks/codedb-warmup.sh"}]})
 
 # Auto-allow codedb's own MCP tools so callers aren't prompted for every