From 4690bc80febc1ff84edb8ee464327435cec5ed21 Mon Sep 17 00:00:00 2001 From: flupkede Date: Sat, 17 Oct 2026 09:53:22 +0200 Subject: [PATCH 1/4] fix: force reindex rebuilds the SCIP symbol indexes too 'n' in the TUI and 'codesearch index -f' promised a full rebuild but date from before the symbol indexes existed: the HTTP handler forced the symbol pass only behind the separate ?symbols=true flag (so the gate saw a fresh SCIP index and skipped it), and the TUI's force reindex never ran a symbol pass at all. Field effect: after adding Razor to the text index, find_impact kept answering from ~19h-old references because no .cs file had changed. Force now implies symbols: the handler computes force_implies_symbols(force, symbols) and the TUI's force task runs maybe_rebuild_symbols(force=true) after restart_fsw - mirroring the HTTP path, which already had the step. Explicit ?symbols=true remains the no-force way to force only the symbol pass; plain reindex stays gate-driven. Local CLI builds (index -f --local / delegation failure) still build no symbols at all - serve owns the symbol pipeline there. Pinned by force_reindex_implies_symbol_rebuild; defect-probed (old semantics fails the assert). --- AGENTS.md | 269 +++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + src/serve/mod.rs | 39 +++++-- src/serve/tests.rs | 14 +++ src/serve/tui.rs | 6 +- 5 files changed, 319 insertions(+), 10 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..ff3a2692 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,269 @@ +# AGENTS.develop.md + +This file is the **develop-branch reference** for all coding agents. +It is committed on `develop` and merged from every feature branch. + +`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the +feature branch while building, dropped before the PR merges — they never appear +on `develop` or `master`. See *For agents starting a new feature branch* below. + +## Setup (once per machine) + +```bash +git config core.hooksPath .githooks +``` + +This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this +file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs +`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` +blocks direct pushes to `master`. See `.githooks/README.md`. + +## For agents starting a new feature branch + +If `AGENTS.md` was not auto-created by the hook, create it manually: + +```bash +cp AGENTS.develop.md AGENTS.md +``` + +Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. +Leave the architecture sections intact — they provide context. + +**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan +travels through the PR and survives branch switches. + +**At the end of the branch, before the PR (the changelog moment):** +1. Update `## Active feature branches` below (remove this branch) +2. Add one line to `## Changelog highlights` +3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) +4. Drop the branch-local files and commit: + `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` + +Enforced mechanically (forgetting step 4 cannot leak the files): +- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on + `develop`/`master` (direct commits and conflict resolutions). +- `pre-merge-commit` runs the same guard on the merge result — a clean + merge never invokes pre-commit. +- The `agent-files-check` CI workflow flags any PR into `develop`/`master` + whose head still carries them. + +**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. +This file contains only architecture, conventions, and changelog. + +--- + +## ⚠️ Branching & PR workflow (READ FIRST) + +This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. + +- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. +- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). +- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. +- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. +- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. +- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. + +Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. + +--- + +## What codesearch is + +A fast, local, offline MCP server for semantic code search. Single Rust binary. +No Docker, no cloud, no external services. Designed for coding agents (OpenCode, +Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. + +Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + +tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). +Hybrid BM25 + vector search fused via RRF. + +## MCP tools + +Five tools exposed to agents: + +| Tool | Description | +|---|---| +| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | +| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | +| `explore` | File outline or similar-chunk lookup. Requires scope. | +| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | +| `status` | Index and project status. Lightweight (no DB open) when called without scope. | + +All tools return `scope_required` structured errors in multi-repo mode when no `project` +or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. + +## Multi-repo serve mode + +`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). +Multiple repos register via `repos.json` with aliases. Agents route queries per-alias +or per-group. + +**Repo lifecycle:** +- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. +- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. +- `Readonly` — Another process holds the write lock. +- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. + +**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. +All opens update `last_access` so the reaper can track every open. + +**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, +no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. + +## TUI + +`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). +Without TTY: headless, logs only. + +- `codesearch serve --no-tui` — suppress TUI even with TTY +- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` + +## HTTP endpoints (serve mode) + +| Endpoint | Method | Description | +|---|---|---| +| `/health` | GET | Health check JSON | +| `/status` | GET | Lightweight repo state snapshot for TUI polling | +| `/repos` | POST | Register + index + warmup a new repo | +| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | +| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | +| `/mcp` | GET/POST | MCP streamable HTTP endpoint | + +## Supported languages + +17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol +precision for C# (bundled `scip-csharp`, resident helper via its `serve` +subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware +chunking only. + +## Release artifacts + +GitHub Actions produces release binaries per tag (plain + `-with-csharp` +variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. +Single-file native binaries, no runtime dependencies. macOS build is manual-trigger +only (expensive runners). + +--- + +## Current state + +- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. +- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. +- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. + +## Implemented features (orientation only — narratives live in CHANGELOG.md) + +- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. +- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. +- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. +- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. + +## ⚠️ Design constraint — never poll a federated peer + +**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's +scale-to-zero replica, which then self-warms its full idle window (measured ~50% +duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — +do not re-attempt. Local repos may be polled in the background; peers are +contacted only by a real federated tool call (activity poke) or the explicit +TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). + +## Open TODOs + +- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. + +--- + +## Key conventions for agents + +- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). +- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. +- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). +- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. +- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. +- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. +- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. +- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. + +## Runtime locations + +- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. +- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). +- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. +- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. + +## Deploying to runtime + +- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** +- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` +- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. +- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. + +## Notes for agents + +- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. +- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. +- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. +- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. +- **Never call `.canonicalize()`** — use `safe_canonicalize()`. +- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. +- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. + +### LMDB rules + +- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). +- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. +- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. +- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). + +### Search errors must not become empty results + +Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: + +- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. +- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. +- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. +- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. +- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. +- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. +- **Suppress `suggested_tool` when warnings are present.** +- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. +- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** +- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. + +--- + +## Active feature branches (not yet merged) + +| Branch | Description | +|---|---| + +--- + +## Changelog highlights (recent) + +- **v1.5.33 (pending)** — Razor searchable: `.razor`/`.cshtml` via text chunking (Html variant); `obj/`, `.g.cs`, `.designer.cs` excluded as generated +- **v1.5.33 (pending)** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) +- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo +- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again +- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error +- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings +- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback +- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) +- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered +- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard +- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes +- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds +- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) +- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories +- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) +- **v1.3.16** — REST `/find-impact` endpoint +- **v1.3.3** — federation hardening release + +Older entries: see `CHANGELOG.md`. + +## Plan + +Force reindex moet de SCIP-index meenemen (gebruiker: "n of -f op een repo met scip index MOET die mee herbouwen" — ontwerp dateert van vóór SCIP). Fix: (1) reindex-handler do_symbols = force || symbols (was: alleen de losse ?symbols-vlag); (2) TUI n draaide NOOIT een symbol-pass — nu na restart_fsw maybe_rebuild_symbols(force=true), spiegelend aan het HTTP-pad. CLI index -f delegeert met force=true en erft zo de fix; lokale builds hadden nooit een symbol-pass (buiten scope, serve owns symbols). + +1. [x] Stage 1: handler + TUI + pin-test (defect-probe uitgevoerd) +2. [ ] Stage 2: CHANGELOG + highlights + full validation diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5d302681 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +Read AGENTS.md. diff --git a/src/serve/mod.rs b/src/serve/mod.rs index d62aeb19..ffb9707a 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -5151,6 +5151,29 @@ async fn reload_handler( /// - `symbols=true` — also rebuild the symbol index (C# SCIP) after text reindex. /// /// Returns 202 Accepted immediately; the reindex runs in the background. +/// A force reindex promises "rebuild everything this repo has": text index +/// AND symbol indexes. The `symbols` query param stays as the explicit +/// no-force way to force only the symbol pass. +fn force_implies_symbols(force: bool, symbols: bool) -> bool { + force || symbols +} + +/// Shared tail of every FORCE reindex — the HTTP handler and the TUI `n` +/// task both end here, so the two paths cannot drift on whether (or in +/// which order) the symbol pass runs. The pass is FORCED: a force reindex +/// that ships 19-hour-old references because no .cs changed defeats the +/// "rebuild everything" promise (pre-SCIP semantics). +pub(crate) async fn finish_force_reindex( + state: &Arc, + alias: &str, + project_path: &Path, + db_path: &Path, + stores: Arc, +) { + state.restart_fsw(alias, stores).await; + maybe_rebuild_symbols(alias, project_path, db_path, state, true).await; +} + async fn reindex_handler( axum::extract::Path(alias): axum::extract::Path, axum::extract::Query(params): axum::extract::Query>, @@ -5242,8 +5265,12 @@ async fn reindex_handler( let guard_state = state.clone(); // `symbols=true` forces a symbol rebuild; without it the rebuild is - // gate-driven (missing/stale indexes are still healed). - let do_symbols = symbols; + // gate-driven (missing/stale indexes are still healed). A FORCE reindex + // implies it: `n` / `index -f` promise "rebuild everything this repo + // has", and a repo carrying a SCIP index must not come out of a force + // reindex with 19-hour-old references just because no .cs file changed + // (pre-SCIP semantics — the flag predates the symbol indexes). + let do_symbols = force_implies_symbols(force, symbols); if force { // Force rebuild: stop FSW -> clear data in-place -> full reindex -> restart FSW. @@ -5368,12 +5395,8 @@ async fn reindex_handler( return; } - // 3. Restart FSW with fresh IndexManager. - g_state.restart_fsw(&g_alias, stores).await; - - // 4. Symbol index rebuild — gate-driven for every installed - // language; `symbols=true` forces it (previous behaviour). - maybe_rebuild_symbols(&alias_bg, &project_path, &db_path, &g_state, do_symbols).await; + // 3. Shared tail: fresh FSW + FORCED symbol pass. + finish_force_reindex(&g_state, &g_alias, &project_path, &db_path, stores).await; g_state.end_indexing(&g_alias, IndexingOwner::Reindex); }); diff --git a/src/serve/tests.rs b/src/serve/tests.rs index c1db245c..37d0ad59 100644 --- a/src/serve/tests.rs +++ b/src/serve/tests.rs @@ -5151,3 +5151,17 @@ async fn mcp_status_and_projects_answer_starting_pre_first_pass() { "projects pre-first-pass must carry the warming warning, got: {text}" ); } + +#[test] +fn force_reindex_implies_symbol_rebuild() { + // `n` / `index -f` promise "rebuild everything this repo has": force + // must imply the symbol pass even without ?symbols=true (pre-SCIP + // semantics left the SCIP index gate-driven and a fresh gate skipped + // it, so a force reindex shipped 19-hour-old references). + assert!(crate::serve::force_implies_symbols(true, false)); + assert!(crate::serve::force_implies_symbols(true, true)); + // Explicit symbols=true stays the no-force way to force only symbols. + assert!(crate::serve::force_implies_symbols(false, true)); + // Plain reindex stays gate-driven. + assert!(!crate::serve::force_implies_symbols(false, false)); +} diff --git a/src/serve/tui.rs b/src/serve/tui.rs index 5a3b4a94..8310da1f 100644 --- a/src/serve/tui.rs +++ b/src/serve/tui.rs @@ -1252,8 +1252,10 @@ pub(crate) fn spawn_force_reindex(alias: String, state: &Arc) -> Rei return; } - // Restart FSW with fresh IndexManager. - state_bg.restart_fsw(&alias_bg, stores).await; + // Shared tail: fresh FSW + FORCED symbol pass (the TUI never ran a + // symbol pass at all before — `n` left find_impact on stale + // references while the text index was brand new). + super::finish_force_reindex(&state_bg, &alias_bg, &project_path, &db_path, stores).await; // Remove guard state_bg.end_indexing(&alias_bg, IndexingOwner::Reindex); From a8ba028b976d77b6e7a4463c037a684f34625631 Mon Sep 17 00:00:00 2001 From: flupkede Date: Sat, 17 Oct 2026 10:13:07 +0200 Subject: [PATCH 2/4] docs: changelog + highlights for force-reindex-forces-symbols CHANGELOG under pending [1.5.33] (Fixed), highlights line, plan ticked. Full validation green: fmt, clippy -D warnings (stage 1), cargo test --lib --bins 894+890. --- AGENTS.develop.md | 1 + AGENTS.md | 2 +- CHANGELOG.md | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/AGENTS.develop.md b/AGENTS.develop.md index 7740e301..0decc041 100644 --- a/AGENTS.develop.md +++ b/AGENTS.develop.md @@ -241,6 +241,7 @@ Never `unwrap_or_default()` a store error on a search path — "no results" and ## Changelog highlights (recent) +- **v1.5.33 (pending)** — force reindex (`n`, `index -f`) rebuilds the SCIP symbol indexes too — shared `finish_force_reindex` tail for HTTP + TUI; `?symbols=true` stays symbol-only, plain reindex stays gate-driven - **v1.5.33 (pending)** — Razor searchable: `.razor`/`.cshtml` via text chunking (Html variant); `obj/`, `.g.cs`, `.designer.cs` excluded as generated - **v1.5.33 (pending)** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) - **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo diff --git a/AGENTS.md b/AGENTS.md index ff3a2692..f0285456 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -266,4 +266,4 @@ Older entries: see `CHANGELOG.md`. Force reindex moet de SCIP-index meenemen (gebruiker: "n of -f op een repo met scip index MOET die mee herbouwen" — ontwerp dateert van vóór SCIP). Fix: (1) reindex-handler do_symbols = force || symbols (was: alleen de losse ?symbols-vlag); (2) TUI n draaide NOOIT een symbol-pass — nu na restart_fsw maybe_rebuild_symbols(force=true), spiegelend aan het HTTP-pad. CLI index -f delegeert met force=true en erft zo de fix; lokale builds hadden nooit een symbol-pass (buiten scope, serve owns symbols). 1. [x] Stage 1: handler + TUI + pin-test (defect-probe uitgevoerd) -2. [ ] Stage 2: CHANGELOG + highlights + full validation +2. [x] Stage 2: CHANGELOG + highlights + full validation diff --git a/CHANGELOG.md b/CHANGELOG.md index 798b8117..1c2a5580 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ finalized in place with a date — no renaming/migration step needed. ### Fixed +- **A force reindex now rebuilds the SCIP symbol indexes too.** `n` in the serve TUI and `codesearch index -f` promised "rebuild everything" but dated from before the symbol indexes existed: the HTTP handler forced the symbol pass only behind the separate `?symbols=true` flag (so a fresh SCIP gate — no `.cs` changes — skipped it), and the TUI's force reindex never ran a symbol pass at all. Field effect: after Razor joined the text index, `find_impact` kept answering from ~19-hour-old references. Force now implies symbols (`finish_force_reindex` is the shared tail of the HTTP and TUI force paths: fresh FSW, then a forced, gate-bypassing symbol pass); `?symbols=true` stays the explicit no-force symbol-only trigger, and a plain reindex stays gate-driven. Local CLI builds (`index -f --local` / delegation failure) still build no symbols — serve owns the symbol pipeline there. + - **Razor markup is now indexed: `.razor` / `.cshtml` ride the text-fallback chunker.** Neither extension had a language mapping, so every Razor file was skipped — field test 2026-10-16 on a customer MVC repo: 9 `.cshtml` files on disk, 0 in the index, and component usage findable only through generated `obj/**/*_razor.g.cs` lines. Both extensions now map to the Html variant (no extractor → text chunking): directives (`@model`, `@inject`, `@page`) and component tags become searchable, without running the C# tree-sitter parser on markup. The watcher and indexer gates key off the same mapping, so incremental reindex covers Razor automatically. Existing indexes keep their chunks until the next reindex. Related deliberate exclusions: `obj/` is now always excluded (always .NET build output — generated code can no longer leak in even when a repo's `.gitignore` misses it), and `.g.cs` / `.designer.cs` suffixes are skipped as generated (the watch layer always claimed `.designer.cs` was excluded; the list never had it). `bin/` is deliberately **not** excluded — several ecosystems keep real scripts there — so the accidental `.g.cs` coverage disappears by design: the real `.razor` files are searchable now. - **`GET /status` and the MCP `status` tool always answer, even under a heavy rebuild wave.** They used to measure inline in the request path: per registered repo several filesystem probes plus a SCIP LMDB env open and a sysinfo refresh, synchronous on async workers that a rebuild wave (dotnet restore + scip-csharp children) starves — with 28 repos one call took tens of seconds and polling clients (remote TUI, calling agents) piled up and timed out; the project list died the same death. Now a single background task measures (on the blocking pool, one pass at a time — under load the cadence stretches instead of piling measuring tasks onto a starved machine) and stores a snapshot whose readers answer in microseconds. Every answer carries `snapshot_age_secs` (staleness is visible, never a hang) and a `load` summary (`text_indexing` / `symbol_indexing` / `errored` / `refresh_interval_secs`) so calling agents can throttle themselves; before the first pass completes they answer `"starting"` immediately. Cadence: `CODESEARCH_STATUS_REFRESH_SECS`, default 2s. From d2d4bf74ea551c0c5456a260b40718727ac51a16 Mon Sep 17 00:00:00 2001 From: flupkede Date: Sat, 17 Oct 2026 10:39:49 +0200 Subject: [PATCH 3/4] fix: the force path carries no symbol flag at all Final-review finding (proven by executing the revert in a throwaway clone): the pin test covered only the pure helper - reverting the handler line to 'let do_symbols = symbols' kept the suite green. The helper is gone; the structure now makes the revert impossible: - the FORCE branch ends in finish_force_reindex, which is always forced and takes no flag - there is no decision left to revert; - the plain-reindex branch passes the ?symbols query flag straight to maybe_rebuild_symbols (gate-driven unless ?symbols=true), unchanged; - force_implies_symbols and its helper-only test are deleted. The seam test finish_force_reindex_starts_fsw_and_keeps_guard_contract runs the real tail on a temp repo (production order: force reindex first, then the tail) and pins FSW registration + the guard contract; the forced-pass semantics stay pinned by the existing maybe_rebuild_symbols force=true test. Review-fixes: - [Important] pin test covered only the helper; reintroduced defect kept the suite green -> removed the revert target structurally (flagless force branch through the shared forced tail) + real seam test. --- src/serve/mod.rs | 29 +++++++--------------- src/serve/tests.rs | 61 +++++++++++++++++++++++++++++++++++++--------- 2 files changed, 58 insertions(+), 32 deletions(-) diff --git a/src/serve/mod.rs b/src/serve/mod.rs index ffb9707a..0fb4af09 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -5151,18 +5151,12 @@ async fn reload_handler( /// - `symbols=true` — also rebuild the symbol index (C# SCIP) after text reindex. /// /// Returns 202 Accepted immediately; the reindex runs in the background. -/// A force reindex promises "rebuild everything this repo has": text index -/// AND symbol indexes. The `symbols` query param stays as the explicit -/// no-force way to force only the symbol pass. -fn force_implies_symbols(force: bool, symbols: bool) -> bool { - force || symbols -} - /// Shared tail of every FORCE reindex — the HTTP handler and the TUI `n` /// task both end here, so the two paths cannot drift on whether (or in -/// which order) the symbol pass runs. The pass is FORCED: a force reindex -/// that ships 19-hour-old references because no .cs changed defeats the -/// "rebuild everything" promise (pre-SCIP semantics). +/// which order) the symbol pass runs. The pass is FORCED, and the force +/// branch carries no flag: "rebuild everything this repo has" is not a +/// decision a caller (or a future refactor) can opt out of. The explicit +/// no-force symbol-only trigger is `?symbols=true` on the plain reindex. pub(crate) async fn finish_force_reindex( state: &Arc, alias: &str, @@ -5264,14 +5258,6 @@ async fn reindex_handler( let guard_alias = alias_bg.clone(); let guard_state = state.clone(); - // `symbols=true` forces a symbol rebuild; without it the rebuild is - // gate-driven (missing/stale indexes are still healed). A FORCE reindex - // implies it: `n` / `index -f` promise "rebuild everything this repo - // has", and a repo carrying a SCIP index must not come out of a force - // reindex with 19-hour-old references just because no .cs file changed - // (pre-SCIP semantics — the flag predates the symbol indexes). - let do_symbols = force_implies_symbols(force, symbols); - if force { // Force rebuild: stop FSW -> clear data in-place -> full reindex -> restart FSW. // The FSW must be stopped before clearing the FileMetaStore, otherwise it @@ -5477,8 +5463,11 @@ async fn reindex_handler( } // Symbol index rebuild — gate-driven for every installed - // language; `symbols=true` forces it (previous behaviour). - maybe_rebuild_symbols(&alias_bg, &project_path, &db_path, &g_state, do_symbols).await; + // language; the explicit `symbols=true` query param forces it + // (the no-force, symbol-only trigger). The FORCE branch above + // carries no flag at all: it ends in `finish_force_reindex`, + // which is always forced — there is no decision left to revert. + maybe_rebuild_symbols(&alias_bg, &project_path, &db_path, &g_state, symbols).await; g_state.end_indexing(&g_alias, IndexingOwner::Reindex); }); diff --git a/src/serve/tests.rs b/src/serve/tests.rs index 37d0ad59..5fd88350 100644 --- a/src/serve/tests.rs +++ b/src/serve/tests.rs @@ -5152,16 +5152,53 @@ async fn mcp_status_and_projects_answer_starting_pre_first_pass() { ); } -#[test] -fn force_reindex_implies_symbol_rebuild() { - // `n` / `index -f` promise "rebuild everything this repo has": force - // must imply the symbol pass even without ?symbols=true (pre-SCIP - // semantics left the SCIP index gate-driven and a fresh gate skipped - // it, so a force reindex shipped 19-hour-old references). - assert!(crate::serve::force_implies_symbols(true, false)); - assert!(crate::serve::force_implies_symbols(true, true)); - // Explicit symbols=true stays the no-force way to force only symbols. - assert!(crate::serve::force_implies_symbols(false, true)); - // Plain reindex stays gate-driven. - assert!(!crate::serve::force_implies_symbols(false, false)); +#[tokio::test] +async fn finish_force_reindex_starts_fsw_and_keeps_guard_contract() { + // Seam pin for the shared FORCE tail (HTTP force branch + TUI `n` both + // end here). The harness has no SCIP helpers, so the forced symbol + // pass is a per-language no-op here — the forced semantics are pinned + // by the maybe_rebuild_symbols(force=true) test and by the tail + // carrying NO flag a refactor could flip. What this test pins is the + // wiring: the tail (re)starts the FSW for the alias and leaves the + // reindex guard release to the callers (HTTP/TUI), in that order. + let tmp = tempfile::tempdir().unwrap(); + let project = tmp.path().join("repo"); + std::fs::create_dir_all(&project).unwrap(); + let db_path = project.join(DB_DIR_NAME); + let state = std::sync::Arc::new(ServeState::new(ReposConfig::default(), None)); + + let stores = match state + .try_open_stores("repo", &db_path, true, false, None) + .expect("a fresh repo must open in write mode") + { + OpenedStores::Write(s) => s, + OpenedStores::Readonly(_) => panic!("allow_create=true must open write mode"), + }; + assert!(state.begin_indexing("repo", IndexingOwner::Reindex)); + // restart_fsw skips aliases that are not registered — register first. + state + .config + .write() + .unwrap() + .repos + .insert("repo".to_string(), project.clone()); + + // Production order: the force path builds the index BEFORE the tail, + // and the tail's IndexManager init needs a real (empty) index to open. + let token = tokio_util::sync::CancellationToken::new(); + IndexManager::force_reindex_with_stores(&project, &db_path, &stores, None, &token, None) + .await + .expect("force reindex of an empty repo must succeed"); + + super::finish_force_reindex(&state, "repo", &project, &db_path, stores).await; + + assert!( + state.fsw_tasks.contains_key("repo"), + "the force tail must leave an FSW task registered for the alias" + ); + assert!( + state.is_indexing("repo"), + "the tail must not release the reindex guard — callers own that" + ); + state.end_indexing("repo", IndexingOwner::Reindex); } From 96c82a1bb4e3b42e6ad62834cd0363c5bb98bcc8 Mon Sep 17 00:00:00 2001 From: flupkede Date: Sat, 17 Oct 2026 10:58:14 +0200 Subject: [PATCH 4/4] chore: drop branch-local agent files --- AGENTS.md | 269 ------------------------------------------------------ CLAUDE.md | 1 - 2 files changed, 270 deletions(-) delete mode 100644 AGENTS.md delete mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index f0285456..00000000 --- a/AGENTS.md +++ /dev/null @@ -1,269 +0,0 @@ -# AGENTS.develop.md - -This file is the **develop-branch reference** for all coding agents. -It is committed on `develop` and merged from every feature branch. - -`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the -feature branch while building, dropped before the PR merges — they never appear -on `develop` or `master`. See *For agents starting a new feature branch* below. - -## Setup (once per machine) - -```bash -git config core.hooksPath .githooks -``` - -This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this -file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs -`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` -blocks direct pushes to `master`. See `.githooks/README.md`. - -## For agents starting a new feature branch - -If `AGENTS.md` was not auto-created by the hook, create it manually: - -```bash -cp AGENTS.develop.md AGENTS.md -``` - -Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. -Leave the architecture sections intact — they provide context. - -**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan -travels through the PR and survives branch switches. - -**At the end of the branch, before the PR (the changelog moment):** -1. Update `## Active feature branches` below (remove this branch) -2. Add one line to `## Changelog highlights` -3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) -4. Drop the branch-local files and commit: - `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` - -Enforced mechanically (forgetting step 4 cannot leak the files): -- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on - `develop`/`master` (direct commits and conflict resolutions). -- `pre-merge-commit` runs the same guard on the merge result — a clean - merge never invokes pre-commit. -- The `agent-files-check` CI workflow flags any PR into `develop`/`master` - whose head still carries them. - -**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. -This file contains only architecture, conventions, and changelog. - ---- - -## ⚠️ Branching & PR workflow (READ FIRST) - -This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. - -- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. -- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). -- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. -- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. -- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. -- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. - -Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. - ---- - -## What codesearch is - -A fast, local, offline MCP server for semantic code search. Single Rust binary. -No Docker, no cloud, no external services. Designed for coding agents (OpenCode, -Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. - -Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + -tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). -Hybrid BM25 + vector search fused via RRF. - -## MCP tools - -Five tools exposed to agents: - -| Tool | Description | -|---|---| -| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | -| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | -| `explore` | File outline or similar-chunk lookup. Requires scope. | -| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | -| `status` | Index and project status. Lightweight (no DB open) when called without scope. | - -All tools return `scope_required` structured errors in multi-repo mode when no `project` -or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. - -## Multi-repo serve mode - -`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). -Multiple repos register via `repos.json` with aliases. Agents route queries per-alias -or per-group. - -**Repo lifecycle:** -- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. -- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. -- `Readonly` — Another process holds the write lock. -- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. - -**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. -All opens update `last_access` so the reaper can track every open. - -**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, -no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. - -## TUI - -`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). -Without TTY: headless, logs only. - -- `codesearch serve --no-tui` — suppress TUI even with TTY -- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` - -## HTTP endpoints (serve mode) - -| Endpoint | Method | Description | -|---|---|---| -| `/health` | GET | Health check JSON | -| `/status` | GET | Lightweight repo state snapshot for TUI polling | -| `/repos` | POST | Register + index + warmup a new repo | -| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | -| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | -| `/mcp` | GET/POST | MCP streamable HTTP endpoint | - -## Supported languages - -17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol -precision for C# (bundled `scip-csharp`, resident helper via its `serve` -subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware -chunking only. - -## Release artifacts - -GitHub Actions produces release binaries per tag (plain + `-with-csharp` -variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. -Single-file native binaries, no runtime dependencies. macOS build is manual-trigger -only (expensive runners). - ---- - -## Current state - -- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. -- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. -- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. - -## Implemented features (orientation only — narratives live in CHANGELOG.md) - -- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. -- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. -- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. -- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. - -## ⚠️ Design constraint — never poll a federated peer - -**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's -scale-to-zero replica, which then self-warms its full idle window (measured ~50% -duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — -do not re-attempt. Local repos may be polled in the background; peers are -contacted only by a real federated tool call (activity poke) or the explicit -TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). - -## Open TODOs - -- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. - ---- - -## Key conventions for agents - -- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). -- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. -- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). -- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. -- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. -- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. -- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. -- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. - -## Runtime locations - -- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. -- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). -- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. -- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. - -## Deploying to runtime - -- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** -- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` -- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. -- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. - -## Notes for agents - -- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. -- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. -- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. -- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. -- **Never call `.canonicalize()`** — use `safe_canonicalize()`. -- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. -- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. - -### LMDB rules - -- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). -- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. -- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. -- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). - -### Search errors must not become empty results - -Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: - -- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. -- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. -- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. -- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. -- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. -- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. -- **Suppress `suggested_tool` when warnings are present.** -- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. -- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** -- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. - ---- - -## Active feature branches (not yet merged) - -| Branch | Description | -|---|---| - ---- - -## Changelog highlights (recent) - -- **v1.5.33 (pending)** — Razor searchable: `.razor`/`.cshtml` via text chunking (Html variant); `obj/`, `.g.cs`, `.designer.cs` excluded as generated -- **v1.5.33 (pending)** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) -- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo -- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again -- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error -- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings -- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback -- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) -- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered -- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard -- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes -- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds -- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) -- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories -- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) -- **v1.3.16** — REST `/find-impact` endpoint -- **v1.3.3** — federation hardening release - -Older entries: see `CHANGELOG.md`. - -## Plan - -Force reindex moet de SCIP-index meenemen (gebruiker: "n of -f op een repo met scip index MOET die mee herbouwen" — ontwerp dateert van vóór SCIP). Fix: (1) reindex-handler do_symbols = force || symbols (was: alleen de losse ?symbols-vlag); (2) TUI n draaide NOOIT een symbol-pass — nu na restart_fsw maybe_rebuild_symbols(force=true), spiegelend aan het HTTP-pad. CLI index -f delegeert met force=true en erft zo de fix; lokale builds hadden nooit een symbol-pass (buiten scope, serve owns symbols). - -1. [x] Stage 1: handler + TUI + pin-test (defect-probe uitgevoerd) -2. [x] Stage 2: CHANGELOG + highlights + full validation diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 5d302681..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -Read AGENTS.md.