From 2f64544306fadf9d8ba3f8946bb887649da31fca Mon Sep 17 00:00:00 2001 From: flupkede Date: Wed, 28 Oct 2026 16:26:42 +0100 Subject: [PATCH 1/4] [worker] fix: reconcile unregisters vanished repos, snapshot pass heals the registry A worktree deleted while the hub keeps running ghosted status kind=projects forever: boot warmup prunes once, reconcile_all_paths only relocated and logged 'run index prune'. Unrelocatable missing paths now unregister (prune_stale), evict in-memory state (reload_if_changed eviction shape) and persist; compute_status_snapshot runs the reconcile first, so the 2s background pass is the periodic heal. Test pins the removal: fails against the old keep-and-log behavior (verified by stash-run). Review-fixes: - [Important] the pinning test never executed the eviction branch -> test now seeds a RepoState::Write entry (real SharedStores + CancellationToken) for the vanished alias and asserts the map entry is gone and the token is cancelled --- AGENTS.md | 287 +++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + src/serve/mod.rs | 50 ++++---- src/serve/tests.rs | 63 ++++++++++ 4 files changed, 381 insertions(+), 20 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..034df5aa --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,287 @@ +# AGENTS.develop.md + +This file is the **develop-branch reference** for all coding agents. +It is committed on `develop` and merged from every feature branch. + +`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the +feature branch while building, dropped before the PR merges — they never appear +on `develop` or `master`. See *For agents starting a new feature branch* below. + +## Setup (once per machine) + +```bash +git config core.hooksPath .githooks +``` + +This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this +file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs +`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` +blocks direct pushes to `master`. See `.githooks/README.md`. + +## For agents starting a new feature branch + +If `AGENTS.md` was not auto-created by the hook, create it manually: + +```bash +cp AGENTS.develop.md AGENTS.md +``` + +Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. +Leave the architecture sections intact — they provide context. + +**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan +travels through the PR and survives branch switches. + +**At the end of the branch, before the PR (the changelog moment):** +1. Update `## Active feature branches` below (remove this branch) +2. Add one line to `## Changelog highlights` +3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) +4. Drop the branch-local files and commit: + `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` + +Enforced mechanically (forgetting step 4 cannot leak the files): +- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on + `develop`/`master` (direct commits and conflict resolutions). +- `pre-merge-commit` runs the same guard on the merge result — a clean + merge never invokes pre-commit. +- The `agent-files-check` CI workflow flags any PR into `develop`/`master` + whose head still carries them. + +**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. +This file contains only architecture, conventions, and changelog. + +--- + +## ⚠️ Branching & PR workflow (READ FIRST) + +This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. + +- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. +- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). +- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. +- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. +- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. +- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. + +Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. + +--- + +## What codesearch is + +A fast, local, offline MCP server for semantic code search. Single Rust binary. +No Docker, no cloud, no external services. Designed for coding agents (OpenCode, +Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. + +Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + +tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). +Hybrid BM25 + vector search fused via RRF. + +## MCP tools + +Five tools exposed to agents: + +| Tool | Description | +|---|---| +| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | +| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | +| `explore` | File outline or similar-chunk lookup. Requires scope. | +| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | +| `status` | Index and project status. Lightweight (no DB open) when called without scope. | + +All tools return `scope_required` structured errors in multi-repo mode when no `project` +or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. + +## Multi-repo serve mode + +`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). +Multiple repos register via `repos.json` with aliases. Agents route queries per-alias +or per-group. + +**Repo lifecycle:** +- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. +- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. +- `Readonly` — Another process holds the write lock. +- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. + +**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. +All opens update `last_access` so the reaper can track every open. + +**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, +no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. + +## TUI + +`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). +Without TTY: headless, logs only. + +- `codesearch serve --no-tui` — suppress TUI even with TTY +- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` + +## HTTP endpoints (serve mode) + +| Endpoint | Method | Description | +|---|---|---| +| `/health` | GET | Health check JSON | +| `/status` | GET | Lightweight repo state snapshot for TUI polling | +| `/repos` | POST | Register + index + warmup a new repo | +| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | +| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | +| `/mcp` | GET/POST | MCP streamable HTTP endpoint | + +## Supported languages + +17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol +precision for C# (bundled `scip-csharp`, resident helper via its `serve` +subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware +chunking only. + +## Release artifacts + +GitHub Actions produces release binaries per tag (plain + `-with-csharp` +variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. +Single-file native binaries, no runtime dependencies. macOS build is manual-trigger +only (expensive runners). + +--- + +## Current state + +- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. +- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. +- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. + +## Implemented features (orientation only — narratives live in CHANGELOG.md) + +- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. +- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. +- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. +- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. + +## ⚠️ Design constraint — never poll a federated peer + +**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's +scale-to-zero replica, which then self-warms its full idle window (measured ~50% +duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — +do not re-attempt. Local repos may be polled in the background; peers are +contacted only by a real federated tool call (activity poke) or the explicit +TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). + +## Open TODOs + +- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. + +--- + +## Key conventions for agents + +- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). +- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. +- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). +- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. +- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. +- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. +- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. +- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. + +## Runtime locations + +- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. +- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). +- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. +- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. + +## Deploying to runtime + +- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** +- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` +- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. +- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. + +## Notes for agents + +- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. +- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. +- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. +- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. +- **Never call `.canonicalize()`** — use `safe_canonicalize()`. +- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. +- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. + +### LMDB rules + +- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). +- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. +- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. +- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). + +### Search errors must not become empty results + +Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: + +- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. +- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. +- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. +- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. +- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. +- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. +- **Suppress `suggested_tool` when warnings are present.** +- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. +- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** +- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. + +--- + +## Active feature branches (not yet merged) + +| Branch | Description | +|---|---| +| fix/reconcile-prune-stale-registry | serve auto-unregisters stale registry entries (periodic reconcile) | + +--- + +## Changelog highlights (recent) + +- **v1.6.0** — status CPU = whole serve process tree (incl. scip/dotnet children) instead of the serve process alone +- **v1.6.0** — Razor refs mapped to source: helper emits `GetMappedLineSpan()` paths/lines (#line → .razor/.cshtml), SCIP stamp split (`SCIP_INDEX_BUILDER_VERSION`) so symbol rebuilds skip the text re-embed wave +- **v1.6.0** — FTS commit-failure hardening: per-file transient redrive, crash-safe vector rollback, mixed-generation sweep in refresh, stale note on get_chunk +- **v1.6.0** — `.codesearchinclude` whitelist: when present (repo root or `~/.codesearch/`), only matching files are indexed; ignore rules still win, absent list = historical behavior +- **v1.6.0** — MCP tools declare `readOnlyHint` annotations: Claude Code plan mode auto-allows all six tools instead of prompting every call +- **v1.6.0** — force reindex (`n`, `index -f`) rebuilds the SCIP symbol indexes too — shared `finish_force_reindex` tail for HTTP + TUI; `?symbols=true` stays symbol-only, plain reindex stays gate-driven +- **v1.6.0** — Razor searchable: `.razor`/`.cshtml` via text chunking (Html variant); `obj/`, `.g.cs`, `.designer.cs` excluded as generated +- **v1.6.0** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) +- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo +- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again +- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error +- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings +- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback +- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) +- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered +- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard +- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes +- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds +- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) +- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories +- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) +- **v1.3.16** — REST `/find-impact` endpoint +- **v1.3.3** — federation hardening release + +Older entries: see `CHANGELOG.md`. + +## Plan + +Reconcile stale registry entries automatically — a worktree deleted while the +hub keeps running ghosted `status kind=projects` forever (boot warmup prunes +once, `reconcile_all_paths` only relocated and logged). + +1. `ServeState::reconcile_all_paths` (serve/mod.rs): unrelocatable missing + paths now unregister (`prune_stale`), evict in-memory state (reload_if_changed + eviction shape), and persist; signature `&Arc` → `&self`; doc comment + updated (no more "run index prune" advice). +2. `compute_status_snapshot` calls `reconcile_all_paths()` first — the 2s + background pass (already on the blocking pool) is the periodic heal; cheap + when nothing moved (N × exists()). +3. Test in `serve/tests.rs`: vanished dir unregisters from config + override + file, live dir survives. Proof: fails against the old keep-and-log behavior. +4. Docs: README `index prune` section, CHANGELOG [1.7.3] entry, changelog + highlight. Branch-local files dropped before the merge to develop. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5d302681 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +Read AGENTS.md. diff --git a/src/serve/mod.rs b/src/serve/mod.rs index 8e95cfca..c8ff65d1 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -1745,14 +1745,16 @@ impl ServeState { ); } - /// Reconcile registered repo paths against the filesystem before warmup. + /// Reconcile registered repo paths against the filesystem. /// - /// For each alias whose stored path no longer exists (folder renamed/moved), - /// attempt a best-effort git-identity relocation and rewrite `repos.json`. - /// When relocation fails the entry is left in place and merely logged — it is - /// skipped safely at warmup and never crashes serve. Explicit cleanup of - /// unrecoverable entries is available via `codesearch index prune`. - pub(crate) fn reconcile_all_paths(self: &Arc) { + /// For each alias whose stored path no longer exists (folder renamed/moved, + /// or a worktree deleted while the hub keeps running), attempt a best-effort + /// git-identity relocation. What cannot be relocated is unregistered, its + /// in-memory state evicted, and `repos.json` rewritten. Boot warmup has its + /// own prune, but a long-lived serve never reboots — without this pass a + /// deleted worktree ghosted `status kind=projects` forever. Explicit manual + /// cleanup remains available via `codesearch index prune`. + pub(crate) fn reconcile_all_paths(&self) { let aliases = self.aliases(); if aliases.is_empty() { return; @@ -1766,27 +1768,30 @@ impl ServeState { } }; - let (relocated, unresolved) = config.relocate_missing(); + let (relocated, removed) = config.prune_stale(); for (alias, new_path) in &relocated { info!("reconcile: relocated '{}' → {}", alias, new_path.display()); } - for alias in &unresolved { - let missing = config - .repos - .get(alias) - .map(|p| p.display().to_string()) - .unwrap_or_default(); - warn!( - "reconcile: '{}' path missing ({}); skipping — \ - run `codesearch index prune` to remove it", - alias, missing + for alias in &removed { + info!( + "reconcile: unregistered stale '{}' (path no longer exists)", + alias ); + // Same eviction shape as reload_if_changed: fire the Write task's + // cancel token while removing its RepoState, then detach the FSW + // handle so a draining task never races the config swap. The DB + // dir lived inside the vanished tree — nothing left to delete. + if let Some((_, RepoState::Write { cancel_token, .. })) = self.repos.remove(alias) { + cancel_token.cancel(); + } + self.last_access.remove(alias); + self.fsw_tasks.remove(alias); } - if !relocated.is_empty() { + if !relocated.is_empty() || !removed.is_empty() { if let Err(e) = self.persist_config(&config) { - warn!("reconcile: failed to persist relocated paths: {}", e); + warn!("reconcile: failed to persist reconciled registry: {}", e); } } } @@ -4056,6 +4061,11 @@ impl ServeState { /// read endpoints answer from the stored result and must never call /// this on the request path. pub(crate) fn compute_status_snapshot(&self) -> status_snapshot::StatusSnapshot { + // Periodic registry heal: this pass is the only recurring + // reconciliation in a long-lived serve (boot warmup runs once), and + // it already runs on the blocking pool. Cheap when nothing moved — + // one exists() probe per registered alias. + self.reconcile_all_paths(); let repos = self.repo_statuses_lightweight(); let active_sessions = self.active_session_count(); diff --git a/src/serve/tests.rs b/src/serve/tests.rs index 6cd34bd1..51a48f0e 100644 --- a/src/serve/tests.rs +++ b/src/serve/tests.rs @@ -1581,6 +1581,69 @@ fn persist_config_honors_override_path() { ); } +/// A worktree deleted while serve runs must vanish from the registry on the +/// next reconcile pass. Boot warmup prunes once, so in a long-lived hub the +/// dead alias used to ghost `status kind=projects` forever (the old reconcile +/// kept the entry and only logged "run index prune"). +#[test] +fn reconcile_all_paths_unregisters_vanished_repos() { + let tmp = tempfile::tempdir().unwrap(); + let config_file = tmp.path().join("repos.json"); + let gone = tmp.path().join("gone-repo"); + let alive = tmp.path().join("alive-repo"); + std::fs::create_dir(&gone).unwrap(); + std::fs::create_dir(&alive).unwrap(); + + let state = std::sync::Arc::new(ServeState::new( + ReposConfig::default(), + Some(config_file.clone()), + )); + { + let mut cfg = state.config.write().unwrap(); + cfg.register_with_alias(gone.clone(), Some("gone-repo".to_string())) + .unwrap(); + cfg.register_with_alias(alive.clone(), Some("alive-repo".to_string())) + .unwrap(); + state.persist_config(&cfg).unwrap(); + } + std::fs::remove_dir(&gone).unwrap(); + + // Seed in-memory Write state for the vanished alias so the eviction + // branch (token cancel + map removal + FSW handle detach) is executed, + // not just the config unregister. + let gone_db = tmp.path().join("gone-db"); + let stores = std::sync::Arc::new(SharedStores::new(&gone_db, 2).expect("fixture stores")); + let token = CancellationToken::new(); + state.repos.insert( + "gone-repo".to_string(), + RepoState::Write { + stores, + index_manager: None, + cancel_token: token.clone(), + }, + ); + + state.reconcile_all_paths(); + + let on_disk = ReposConfig::load_from(&config_file).unwrap(); + assert!( + !on_disk.repos.contains_key("gone-repo"), + "vanished repo must be unregistered and persisted" + ); + assert!( + on_disk.repos.contains_key("alive-repo"), + "live repo must survive the reconcile pass" + ); + assert!( + state.repos.get("gone-repo").is_none(), + "vanished alias must be evicted from memory" + ); + assert!( + token.is_cancelled(), + "the Write task's cancel token must be cancelled" + ); +} + #[test] fn config_reload_picks_up_new_alias() { let tmp = tempfile::tempdir().unwrap(); From 097022269095306d8985280c9b22717354d499ba Mon Sep 17 00:00:00 2001 From: flupkede Date: Wed, 28 Oct 2026 16:38:06 +0100 Subject: [PATCH 2/4] =?UTF-8?q?[worker]=20stage=202/2:=20docs=20=E2=80=94?= =?UTF-8?q?=20registry=20self-heal=20in=20README,=20changelog,=20branch=20?= =?UTF-8?q?bookkeeping?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README prune section: serve unregisters unrelocatable paths itself (startup + status refresh); index prune stays as manual trigger. CHANGELOG [1.7.3] Fixed entry + AGENTS.develop.md highlight. --- AGENTS.develop.md | 1 + CHANGELOG.md | 2 ++ README.md | 15 +++++++++++---- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/AGENTS.develop.md b/AGENTS.develop.md index ffd4be04..20ec70d9 100644 --- a/AGENTS.develop.md +++ b/AGENTS.develop.md @@ -241,6 +241,7 @@ Never `unwrap_or_default()` a store error on a search path — "no results" and ## Changelog highlights (recent) +- **v1.7.3 (pending)** — serve self-heals its registry: a repo whose path is gone and unrelocatable is unregistered at startup and on every status refresh, no more manual `codesearch index prune` - **v1.6.0** — status CPU = whole serve process tree (incl. scip/dotnet children) instead of the serve process alone - **v1.6.0** — Razor refs mapped to source: helper emits `GetMappedLineSpan()` paths/lines (#line → .razor/.cshtml), SCIP stamp split (`SCIP_INDEX_BUILDER_VERSION`) so symbol rebuilds skip the text re-embed wave - **v1.6.0** — FTS commit-failure hardening: per-file transient redrive, crash-safe vector rollback, mixed-generation sweep in refresh, stale note on get_chunk diff --git a/CHANGELOG.md b/CHANGELOG.md index 17f1ca0a..0212e073 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,8 @@ finalized in place with a date — no renaming/migration step needed. - **Repos registered before identity capture now join their worktree family after a serve restart.** Startup backfills the missing git common-dir identities for every registered repo (previously only group members were backfilled during group resolution), so the TUI family tree groups all worktrees under their real main even when the main predates identity capture. +- **Stale registrations now disappear on their own.** Serve unregisters a repo whose stored path is gone and unrelocatable — at startup, and again on every background status refresh — instead of keeping it listed forever as a `not indexed` ghost until someone ran `codesearch index prune` by hand. Moved repos are still relocated first via their git-remote identity, and the manual verb stays available for the same sweep while serve is not running. + - **The serve TUI no longer decays into unreadable columns, and worktree families render as a tree.** The family note ("worktree of …", up to ~40 chars) was budgeted into the alias column, so on busy fleets the alias column overflowed and ratatui clipped every column from the left. The note now renders in its own dim "Worktree" column (capped at 30 chars with ellipsis), long aliases middle-ellipsis instead of clipping, and the column-width budget accounts for the lock column and ratatui's inter-column spacing. Worktrees now group under their main tree with a `|` indent (families anchor at the main's position; orphans stay indented at the tail), the note text is the bare main alias, and `groups list` renders `alias (main)`. Also fixes main-tree election when the main repo was registered before identity capture existed (identity-less mains now win the exact `.git`-path match over a shortest-path worktree). ## [1.7.2] - 2026-10-27 diff --git a/README.md b/README.md index 1ab580d8..4f80e863 100644 --- a/README.md +++ b/README.md @@ -410,13 +410,14 @@ codesearch index rm /path/to/my-project # List registered repos codesearch index list -# Clean up stale entries (relocates moved repos, drops the rest) +# Clean up stale entries (relocates moved repos, drops the rest; +# serve also does this itself at startup and on every status refresh) codesearch index prune ``` Serve reads `~/.codesearch/repos.json` on startup and manages all registered repos. -#### Moved or renamed repositories +#### Moved, renamed or deleted repositories If you rename or move a registered folder, serve does **not** crash. On startup it tries to **relocate** each missing repo automatically: it captures every @@ -424,8 +425,14 @@ repo's git remote (`remote.origin.url`) at registration, and on a missing path it scans nearby folders (bounded depth, override with `CODESEARCH_RELOCATE_MAX_DEPTH`, default `3`) for a git checkout with the same remote. A single unambiguous match is rewritten into `repos.json`; otherwise the -entry is logged and skipped (never indexed against a dead path). Run -`codesearch index prune` to relocate what can be relocated and drop the rest. +entry is logged and skipped (never indexed against a dead path). + +When a path is unrecoverable, serve now **unregisters it by itself** — at +startup, and again on every background status refresh — so a deleted repo or +worktree stops showing up as a `not indexed` ghost on its own (worktree deletion +fires no git hook, and `rm -rf` bypasses every observer, so serve reconciles +its own registry instead). `codesearch index prune` stays available as the +manual trigger for the same sweep, e.g. while serve is not running. A hand-edited `repos.json` is also tolerated: empty entries, orphaned metadata, and group references to unknown repos are cleaned up on load rather than From fa12a546cb6dd03b197223f71dd557ddaad2c60a Mon Sep 17 00:00:00 2001 From: flupkede Date: Wed, 28 Oct 2026 16:39:31 +0100 Subject: [PATCH 3/4] chore: drop branch-local agent files --- AGENTS.md | 287 ------------------------------------------------------ CLAUDE.md | 1 - 2 files changed, 288 deletions(-) delete mode 100644 AGENTS.md delete mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 034df5aa..00000000 --- a/AGENTS.md +++ /dev/null @@ -1,287 +0,0 @@ -# AGENTS.develop.md - -This file is the **develop-branch reference** for all coding agents. -It is committed on `develop` and merged from every feature branch. - -`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the -feature branch while building, dropped before the PR merges — they never appear -on `develop` or `master`. See *For agents starting a new feature branch* below. - -## Setup (once per machine) - -```bash -git config core.hooksPath .githooks -``` - -This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this -file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs -`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` -blocks direct pushes to `master`. See `.githooks/README.md`. - -## For agents starting a new feature branch - -If `AGENTS.md` was not auto-created by the hook, create it manually: - -```bash -cp AGENTS.develop.md AGENTS.md -``` - -Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. -Leave the architecture sections intact — they provide context. - -**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan -travels through the PR and survives branch switches. - -**At the end of the branch, before the PR (the changelog moment):** -1. Update `## Active feature branches` below (remove this branch) -2. Add one line to `## Changelog highlights` -3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) -4. Drop the branch-local files and commit: - `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` - -Enforced mechanically (forgetting step 4 cannot leak the files): -- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on - `develop`/`master` (direct commits and conflict resolutions). -- `pre-merge-commit` runs the same guard on the merge result — a clean - merge never invokes pre-commit. -- The `agent-files-check` CI workflow flags any PR into `develop`/`master` - whose head still carries them. - -**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. -This file contains only architecture, conventions, and changelog. - ---- - -## ⚠️ Branching & PR workflow (READ FIRST) - -This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. - -- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. -- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). -- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. -- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. -- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. -- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. - -Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. - ---- - -## What codesearch is - -A fast, local, offline MCP server for semantic code search. Single Rust binary. -No Docker, no cloud, no external services. Designed for coding agents (OpenCode, -Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. - -Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + -tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). -Hybrid BM25 + vector search fused via RRF. - -## MCP tools - -Five tools exposed to agents: - -| Tool | Description | -|---|---| -| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | -| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | -| `explore` | File outline or similar-chunk lookup. Requires scope. | -| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | -| `status` | Index and project status. Lightweight (no DB open) when called without scope. | - -All tools return `scope_required` structured errors in multi-repo mode when no `project` -or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. - -## Multi-repo serve mode - -`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). -Multiple repos register via `repos.json` with aliases. Agents route queries per-alias -or per-group. - -**Repo lifecycle:** -- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. -- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. -- `Readonly` — Another process holds the write lock. -- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. - -**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. -All opens update `last_access` so the reaper can track every open. - -**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, -no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. - -## TUI - -`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). -Without TTY: headless, logs only. - -- `codesearch serve --no-tui` — suppress TUI even with TTY -- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` - -## HTTP endpoints (serve mode) - -| Endpoint | Method | Description | -|---|---|---| -| `/health` | GET | Health check JSON | -| `/status` | GET | Lightweight repo state snapshot for TUI polling | -| `/repos` | POST | Register + index + warmup a new repo | -| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | -| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | -| `/mcp` | GET/POST | MCP streamable HTTP endpoint | - -## Supported languages - -17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol -precision for C# (bundled `scip-csharp`, resident helper via its `serve` -subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware -chunking only. - -## Release artifacts - -GitHub Actions produces release binaries per tag (plain + `-with-csharp` -variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. -Single-file native binaries, no runtime dependencies. macOS build is manual-trigger -only (expensive runners). - ---- - -## Current state - -- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. -- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. -- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. - -## Implemented features (orientation only — narratives live in CHANGELOG.md) - -- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. -- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. -- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. -- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. - -## ⚠️ Design constraint — never poll a federated peer - -**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's -scale-to-zero replica, which then self-warms its full idle window (measured ~50% -duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — -do not re-attempt. Local repos may be polled in the background; peers are -contacted only by a real federated tool call (activity poke) or the explicit -TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). - -## Open TODOs - -- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. - ---- - -## Key conventions for agents - -- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). -- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. -- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). -- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. -- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. -- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. -- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. -- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. - -## Runtime locations - -- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. -- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). -- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. -- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. - -## Deploying to runtime - -- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** -- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` -- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. -- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. - -## Notes for agents - -- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. -- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. -- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. -- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. -- **Never call `.canonicalize()`** — use `safe_canonicalize()`. -- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. -- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. - -### LMDB rules - -- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). -- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. -- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. -- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). - -### Search errors must not become empty results - -Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: - -- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. -- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. -- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. -- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. -- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. -- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. -- **Suppress `suggested_tool` when warnings are present.** -- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. -- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** -- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. - ---- - -## Active feature branches (not yet merged) - -| Branch | Description | -|---|---| -| fix/reconcile-prune-stale-registry | serve auto-unregisters stale registry entries (periodic reconcile) | - ---- - -## Changelog highlights (recent) - -- **v1.6.0** — status CPU = whole serve process tree (incl. scip/dotnet children) instead of the serve process alone -- **v1.6.0** — Razor refs mapped to source: helper emits `GetMappedLineSpan()` paths/lines (#line → .razor/.cshtml), SCIP stamp split (`SCIP_INDEX_BUILDER_VERSION`) so symbol rebuilds skip the text re-embed wave -- **v1.6.0** — FTS commit-failure hardening: per-file transient redrive, crash-safe vector rollback, mixed-generation sweep in refresh, stale note on get_chunk -- **v1.6.0** — `.codesearchinclude` whitelist: when present (repo root or `~/.codesearch/`), only matching files are indexed; ignore rules still win, absent list = historical behavior -- **v1.6.0** — MCP tools declare `readOnlyHint` annotations: Claude Code plan mode auto-allows all six tools instead of prompting every call -- **v1.6.0** — force reindex (`n`, `index -f`) rebuilds the SCIP symbol indexes too — shared `finish_force_reindex` tail for HTTP + TUI; `?symbols=true` stays symbol-only, plain reindex stays gate-driven -- **v1.6.0** — Razor searchable: `.razor`/`.cshtml` via text chunking (Html variant); `obj/`, `.g.cs`, `.designer.cs` excluded as generated -- **v1.6.0** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) -- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo -- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again -- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error -- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings -- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback -- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) -- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered -- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard -- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes -- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds -- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) -- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories -- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) -- **v1.3.16** — REST `/find-impact` endpoint -- **v1.3.3** — federation hardening release - -Older entries: see `CHANGELOG.md`. - -## Plan - -Reconcile stale registry entries automatically — a worktree deleted while the -hub keeps running ghosted `status kind=projects` forever (boot warmup prunes -once, `reconcile_all_paths` only relocated and logged). - -1. `ServeState::reconcile_all_paths` (serve/mod.rs): unrelocatable missing - paths now unregister (`prune_stale`), evict in-memory state (reload_if_changed - eviction shape), and persist; signature `&Arc` → `&self`; doc comment - updated (no more "run index prune" advice). -2. `compute_status_snapshot` calls `reconcile_all_paths()` first — the 2s - background pass (already on the blocking pool) is the periodic heal; cheap - when nothing moved (N × exists()). -3. Test in `serve/tests.rs`: vanished dir unregisters from config + override - file, live dir survives. Proof: fails against the old keep-and-log behavior. -4. Docs: README `index prune` section, CHANGELOG [1.7.3] entry, changelog - highlight. Branch-local files dropped before the merge to develop. diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 5d302681..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -Read AGENTS.md. From 07e0f4e08d71a5864ff2450ce8ddd191f715be37 Mon Sep 17 00:00:00 2001 From: flupkede Date: Wed, 28 Oct 2026 17:03:00 +0100 Subject: [PATCH 4/4] fix: reconcile after the status pass in compute_status_snapshot MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit reconcile's aliases() can trigger a config reload (override-less state + existing repos.json loads the real registry); the statuses must measure the config as-was, the order the read endpoints implicitly rely on — the empty-fleet test pinned exactly that ordering. Heal still runs every pass; a healed alias leaves the list one 2s pass later, harmless. --- src/serve/mod.rs | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/serve/mod.rs b/src/serve/mod.rs index c8ff65d1..74391d67 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -4061,13 +4061,17 @@ impl ServeState { /// read endpoints answer from the stored result and must never call /// this on the request path. pub(crate) fn compute_status_snapshot(&self) -> status_snapshot::StatusSnapshot { + let repos = self.repo_statuses_lightweight(); + let active_sessions = self.active_session_count(); + // Periodic registry heal: this pass is the only recurring // reconciliation in a long-lived serve (boot warmup runs once), and - // it already runs on the blocking pool. Cheap when nothing moved — - // one exists() probe per registered alias. + // it already runs on the blocking pool. It runs AFTER the status + // pass on purpose: reconcile's aliases() may trigger a config + // reload, and the statuses above must measure the config as-was — + // the order the read endpoints implicitly rely on. Cheap when + // nothing moved: one exists() probe per registered alias. self.reconcile_all_paths(); - let repos = self.repo_statuses_lightweight(); - let active_sessions = self.active_session_count(); // CPU usage — serve process PLUS all its descendants. The SCIP // helpers and their dotnet children do the heavy lifting during a