From 6e88dfb7a685f96226db0da4912ecd0e150b1526 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 14:33:07 +0200 Subject: [PATCH 1/5] feat: /status answers from a background snapshot, never measures inline GET /status ran repo_statuses_lightweight() + a sysinfo refresh in the request path: across 28 repos that is dozens of filesystem probes and a SCIP LMDB env open, synchronous on async workers that a heavy rebuild wave (dotnet restore + scip-csharp children) OS-starves. Field case 2026-10-16: /status and the project list timed out for minutes while symbol indexing ran - the one endpoint that must always answer was the one that could not. Now a single background task measures (spawn_blocking, one pass at a time - under load the cadence stretches instead of piling measuring tasks onto a starved machine) and stores a StatusSnapshot in a slot whose lock is only ever held to swap an Arc. The handler is O(1): it answers the stored snapshot with snapshot_age_secs for staleness and a load summary (text_indexing / symbol_indexing / errored / refresh_interval_secs) so calling agents can throttle themselves. Before the first pass completes it answers "starting" immediately. Cadence: CODESEARCH_STATUS_REFRESH_SECS, default 2s. --- AGENTS.md | 268 +++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + src/serve/mod.rs | 182 +++++++++++++++--------- src/serve/status_snapshot.rs | 104 ++++++++++++++ 4 files changed, 487 insertions(+), 68 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md create mode 100644 src/serve/status_snapshot.rs diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..741fec8d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,268 @@ +# AGENTS.develop.md + +This file is the **develop-branch reference** for all coding agents. +It is committed on `develop` and merged from every feature branch. + +`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the +feature branch while building, dropped before the PR merges — they never appear +on `develop` or `master`. See *For agents starting a new feature branch* below. + +## Setup (once per machine) + +```bash +git config core.hooksPath .githooks +``` + +This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this +file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs +`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` +blocks direct pushes to `master`. See `.githooks/README.md`. + +## For agents starting a new feature branch + +If `AGENTS.md` was not auto-created by the hook, create it manually: + +```bash +cp AGENTS.develop.md AGENTS.md +``` + +Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. +Leave the architecture sections intact — they provide context. + +**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan +travels through the PR and survives branch switches. + +**At the end of the branch, before the PR (the changelog moment):** +1. Update `## Active feature branches` below (remove this branch) +2. Add one line to `## Changelog highlights` +3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) +4. Drop the branch-local files and commit: + `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` + +Enforced mechanically (forgetting step 4 cannot leak the files): +- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on + `develop`/`master` (direct commits and conflict resolutions). +- `pre-merge-commit` runs the same guard on the merge result — a clean + merge never invokes pre-commit. +- The `agent-files-check` CI workflow flags any PR into `develop`/`master` + whose head still carries them. + +**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. +This file contains only architecture, conventions, and changelog. + +--- + +## ⚠️ Branching & PR workflow (READ FIRST) + +This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. + +- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. +- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). +- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. +- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. +- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. +- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. + +Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. + +--- + +## What codesearch is + +A fast, local, offline MCP server for semantic code search. Single Rust binary. +No Docker, no cloud, no external services. Designed for coding agents (OpenCode, +Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. + +Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + +tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). +Hybrid BM25 + vector search fused via RRF. + +## MCP tools + +Five tools exposed to agents: + +| Tool | Description | +|---|---| +| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | +| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | +| `explore` | File outline or similar-chunk lookup. Requires scope. | +| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | +| `status` | Index and project status. Lightweight (no DB open) when called without scope. | + +All tools return `scope_required` structured errors in multi-repo mode when no `project` +or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. + +## Multi-repo serve mode + +`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). +Multiple repos register via `repos.json` with aliases. Agents route queries per-alias +or per-group. + +**Repo lifecycle:** +- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. +- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. +- `Readonly` — Another process holds the write lock. +- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. + +**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. +All opens update `last_access` so the reaper can track every open. + +**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, +no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. + +## TUI + +`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). +Without TTY: headless, logs only. + +- `codesearch serve --no-tui` — suppress TUI even with TTY +- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` + +## HTTP endpoints (serve mode) + +| Endpoint | Method | Description | +|---|---|---| +| `/health` | GET | Health check JSON | +| `/status` | GET | Lightweight repo state snapshot for TUI polling | +| `/repos` | POST | Register + index + warmup a new repo | +| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | +| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | +| `/mcp` | GET/POST | MCP streamable HTTP endpoint | + +## Supported languages + +17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol +precision for C# (bundled `scip-csharp`, resident helper via its `serve` +subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware +chunking only. + +## Release artifacts + +GitHub Actions produces release binaries per tag (plain + `-with-csharp` +variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. +Single-file native binaries, no runtime dependencies. macOS build is manual-trigger +only (expensive runners). + +--- + +## Current state + +- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. +- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. +- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. + +## Implemented features (orientation only — narratives live in CHANGELOG.md) + +- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. +- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. +- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. +- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. + +## ⚠️ Design constraint — never poll a federated peer + +**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's +scale-to-zero replica, which then self-warms its full idle window (measured ~50% +duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — +do not re-attempt. Local repos may be polled in the background; peers are +contacted only by a real federated tool call (activity poke) or the explicit +TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). + +## Open TODOs + +- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. + +--- + +## Key conventions for agents + +- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). +- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. +- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). +- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. +- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. +- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. +- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. +- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. + +## Runtime locations + +- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. +- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). +- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. +- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. + +## Deploying to runtime + +- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** +- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` +- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. +- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. + +## Notes for agents + +- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. +- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. +- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. +- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. +- **Never call `.canonicalize()`** — use `safe_canonicalize()`. +- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. +- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. + +### LMDB rules + +- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). +- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. +- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. +- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). + +### Search errors must not become empty results + +Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: + +- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. +- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. +- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. +- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. +- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. +- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. +- **Suppress `suggested_tool` when warnings are present.** +- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. +- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** +- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. + +--- + +## Active feature branches (not yet merged) + +| Branch | Description | +|---|---| + +--- + +## Changelog highlights (recent) + +- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo +- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again +- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error +- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings +- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback +- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) +- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered +- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard +- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes +- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds +- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) +- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories +- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) +- **v1.3.16** — REST `/find-impact` endpoint +- **v1.3.3** — federation hardening release + +Older entries: see `CHANGELOG.md`. + +## Plan + +/status must always answer: serve used to measure inline in the handler (28 repos x FS probes + SCIP LMDB opens + sysinfo refresh on async workers OS-starved by rebuild children -> tens of seconds, client timeouts). Now a background task (spawn_blocking, serial passes, CODESEARCH_STATUS_REFRESH_SECS default 2s) computes a StatusSnapshot into a swap-only RwLock slot; /status answers O(1) with snapshot_age_secs + a load summary (text_indexing / symbol_indexing / errored / refresh_interval_secs); before the first pass it answers "starting" immediately. Stage 2: MCP status tool via snapshot. Stage 3: tests + CHANGELOG + validation. + +1. [x] Stage 1: status_snapshot.rs + ServeState field + compute_status_snapshot + new status_handler + spawn at startup +2. [ ] Stage 2: MCP status (kind=index) reads the snapshot +3. [ ] Stage 3: tests + CHANGELOG + full validation diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5d302681 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +Read AGENTS.md. diff --git a/src/serve/mod.rs b/src/serve/mod.rs index 65d77c26..8d1ccc5d 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -10,6 +10,7 @@ //! Holds a `DashMap>` keyed by repo alias. //! Lazy-opens stores on first query. Conflicted repos are isolated. +mod status_snapshot; mod tui; mod tui_common; mod tui_remote; @@ -464,6 +465,12 @@ pub(crate) struct ServeState { reload_count: std::sync::atomic::AtomicUsize, /// Instant when ServeState was created — used to compute uptime for TUI header. started_at: std::time::Instant, + /// Precomputed status snapshot the read endpoints answer from: a + /// background task measures (filesystem probes, sysinfo), handlers + /// never do. Under a heavy rebuild wave the measuring workers are + /// OS-starved; inline measurement made /status itself take tens of + /// seconds across the fleet (the "status must always answer" fix). + status_snapshot: Arc, } impl std::fmt::Debug for ServeState { @@ -548,6 +555,7 @@ impl ServeState { #[cfg(test)] reload_count: std::sync::atomic::AtomicUsize::new(0), started_at: std::time::Instant::now(), + status_snapshot: Arc::new(status_snapshot::StatusSnapshotHandle::new()), } } @@ -3807,6 +3815,49 @@ impl ServeState { self.end_indexing(alias, IndexingOwner::Reindex); } + /// Compute one full status pass for the background snapshot task — + /// the ONLY place that measures (filesystem probes, sysinfo refresh). + /// Runs on the blocking pool via [`status_snapshot::spawn`]; the + /// read endpoints answer from the stored result and must never call + /// this on the request path. + pub(crate) fn compute_status_snapshot(&self) -> status_snapshot::StatusSnapshot { + let repos = self.repo_statuses_lightweight(); + let active_sessions = self.active_session_count(); + + // CPU usage — reuse shared System instance so cpu_usage() can compute delta + let cpu = (|| -> Option { + use sysinfo::ProcessesToUpdate; + let pid = sysinfo::get_current_pid().ok()?; + let mut sys = self.sysinfo_system.lock().ok()?; + sys.refresh_processes(ProcessesToUpdate::Some(&[pid]), true); + let proc = sys.process(pid)?; + let num_cpus = sys.cpus().len().max(1) as f32; + Some(format!("{:.0}%", proc.cpu_usage() / num_cpus)) + })() + .unwrap_or_else(|| "—".to_string()); + + let csharp_helper = self + .symbol_registry + .get(LANG_CSHARP) + .map(|i| i.is_available()) + .unwrap_or(false); + + let ts_helper = self + .symbol_registry + .get(LANG_TYPESCRIPT) + .map(|i| i.is_available()) + .unwrap_or(false); + + status_snapshot::StatusSnapshot { + repos, + active_sessions, + cpu_percent: cpu, + csharp_helper, + ts_helper, + generated_at: std::time::Instant::now(), + } + } + /// Get lightweight repo statuses WITHOUT opening any databases. /// Returns a list of (alias, status_info) where status is derived from DashMap state only. pub(crate) fn repo_statuses_lightweight(&self) -> Vec<(String, RepoStatusInfo)> { @@ -4356,15 +4407,33 @@ async fn indexing_handler( /// Status handler: GET /status /// -/// Returns a JSON snapshot of all repo states, active sessions, and CPU usage. -/// Used by the standalone TUI (`codesearch serve tui`) to poll server state. +/// Answers from the precomputed [`status_snapshot::StatusSnapshot`] — the +/// measuring runs on a background task, so this handler is O(1) no matter +/// how loaded the machine is (the old inline version took tens of seconds +/// across the fleet during a rebuild wave; clients timed out). First ticks +/// after startup get an immediate minimal answer with `"status": "starting"`. +/// Every answer carries `snapshot_age_secs`; the `load` object counts the +/// busy repos so calling agents can throttle themselves. async fn status_handler( axum::extract::State(state): axum::extract::State>, ) -> AxumJson { - let repos = state.repo_statuses_lightweight(); - let active_sessions = state.active_session_count(); + let uptime_secs = state.started_at().elapsed().as_secs(); + let version = env!("CARGO_PKG_VERSION"); + + let Some(snap) = state.status_snapshot.get() else { + return AxumJson(json!({ + "version": version, + "status": "starting", + "repos": [], + "active_sessions": 0, + "cpu_percent": "—", + "snapshot_age_secs": serde_json::Value::Null, + "uptime_secs": uptime_secs, + })); + }; - let repo_json: Vec = repos + let repo_json: Vec = snap + .repos .iter() .map(|(alias, info)| { let status_str = match info.status { @@ -4409,73 +4478,44 @@ async fn status_handler( }) .collect(); - let uptime_secs = state.started_at().elapsed().as_secs(); - - // Serve-wide default model for newly created indexes (`serve --model`). - // `null` means the built-in default. - let default_model = state.default_model().map(|m| m.short_name()); - - // CPU usage — reuse shared System instance so cpu_usage() can compute delta - let cpu = { - use sysinfo::ProcessesToUpdate; - let pid = match sysinfo::get_current_pid() { - Ok(p) => p, - Err(_) => { - return AxumJson(json!({ - "version": env!("CARGO_PKG_VERSION"), - "repos": repo_json, - "active_sessions": active_sessions, - "default_model": default_model, - "cpu_percent": "—", - "uptime_secs": uptime_secs, - })); - } - }; - let mut sys = match state.sysinfo_system.lock() { - Ok(s) => s, - Err(_) => { - return AxumJson(json!({ - "version": env!("CARGO_PKG_VERSION"), - "repos": repo_json, - "active_sessions": active_sessions, - "default_model": default_model, - "cpu_percent": "—", - "uptime_secs": uptime_secs, - })); - } - }; - sys.refresh_processes(ProcessesToUpdate::Some(&[pid]), true); - match sys.process(pid) { - Some(proc) => { - let num_cpus = sys.cpus().len().max(1) as f32; - let pct = proc.cpu_usage() / num_cpus; - format!("{:.0}%", pct) - } - None => "—".to_string(), - } - }; - - let csharp_helper = state - .symbol_registry - .get(LANG_CSHARP) - .map(|i| i.is_available()) - .unwrap_or(false); - - let ts_helper = state - .symbol_registry - .get(LANG_TYPESCRIPT) - .map(|i| i.is_available()) - .unwrap_or(false); + // Load summary for calling agents: how busy is this serve right now. + // Symbol columns count separately from the text label — a pure SCIP + // wave never lights "indexing" at the repo level (v1.5.31 semantics). + let text_indexing = snap + .repos + .iter() + .filter(|(_, i)| i.status == RepoStateLabel::Indexing) + .count(); + let symbol_indexing = snap + .repos + .iter() + .filter(|(_, i)| { + matches!(i.csharp_index, CSharpIndexStatus::Indexing) + || matches!(i.typescript_index, CSharpIndexStatus::Indexing) + }) + .count(); + let errored = snap + .repos + .iter() + .filter(|(_, i)| i.status == RepoStateLabel::Error) + .count(); AxumJson(json!({ - "version": env!("CARGO_PKG_VERSION"), + "version": version, "repos": repo_json, - "active_sessions": active_sessions, - "default_model": default_model, - "cpu_percent": cpu, - "csharp_helper": csharp_helper, - "ts_helper": ts_helper, + "active_sessions": snap.active_sessions, + "default_model": state.default_model().map(|m| m.short_name()), + "cpu_percent": snap.cpu_percent, + "csharp_helper": snap.csharp_helper, + "ts_helper": snap.ts_helper, "uptime_secs": uptime_secs, + "snapshot_age_secs": snap.age_secs(), + "load": { + "text_indexing": text_indexing, + "symbol_indexing": symbol_indexing, + "errored": errored, + "refresh_interval_secs": status_snapshot::refresh_interval().as_secs(), + }, })) } @@ -6592,6 +6632,12 @@ pub async fn run_serve( // — it never polls a federated peer on a timer at all. let serve_state = Arc::new(ServeState::new(config, None).with_default_model(default_model)); + // The status snapshot refresher: only this task measures. Read + // endpoints (GET /status, MCP status) answer from its stored result, + // so they respond in microseconds even while a rebuild wave starves + // the machine — the failure mode that made /status itself time out. + status_snapshot::spawn(serve_state.clone()); + // One line that answers "why is it slower / less parallel than before?" // without grepping the code: the effective heavy-work budget. info!( diff --git a/src/serve/status_snapshot.rs b/src/serve/status_snapshot.rs new file mode 100644 index 00000000..a1c8c439 --- /dev/null +++ b/src/serve/status_snapshot.rs @@ -0,0 +1,104 @@ +//! Always-answering status snapshot for serve mode. +//! +//! GET /status and the MCP `status` tool used to run their probes inline: +//! per registered repo several filesystem stats, a SCIP LMDB env open on +//! cache miss, and a sysinfo refresh — all synchronous on async workers. +//! Under a heavy rebuild wave those workers are OS-starved by the dotnet +//! and scip-csharp children, and one "lightweight" call took tens of +//! seconds across 28 repos; every polling client piled on and timed out. +//! +//! The fix is architectural: only a background task measures; handlers +//! read a precomputed [`StatusSnapshot`] out of a lock-held-only-to-swap +//! slot and answer in microseconds, no matter how loaded the machine is. +//! `snapshot_age_secs` in every answer tells the caller exactly how stale +//! what it is reading is — degraded freshness is visible, never a hang. + +use std::sync::{Arc, RwLock}; +use std::time::Duration; + +use super::{RepoStatusInfo, ServeState}; + +/// One computed status pass. Cheap to clone (handlers get an `Arc`). +pub(crate) struct StatusSnapshot { + pub(crate) repos: Vec<(String, RepoStatusInfo)>, + pub(crate) active_sessions: u64, + pub(crate) cpu_percent: String, + pub(crate) csharp_helper: bool, + pub(crate) ts_helper: bool, + pub(crate) generated_at: std::time::Instant, +} + +impl StatusSnapshot { + /// Age of this snapshot in whole seconds. + pub(crate) fn age_secs(&self) -> u64 { + self.generated_at.elapsed().as_secs() + } +} + +/// The slot handlers read from. The inner RwLock is only ever held to swap +/// an `Arc` — never across computation — so reader latency is bound by the +/// swap itself, not by how long the machine takes to measure. +#[derive(Default)] +pub(crate) struct StatusSnapshotHandle { + slot: RwLock>>, +} + +impl StatusSnapshotHandle { + pub(crate) fn new() -> Self { + Self::default() + } + + /// The latest snapshot, or `None` before the first background pass + /// completes. Handlers must answer something immediately in that case, + /// never fall back to measuring inline. + pub(crate) fn get(&self) -> Option> { + self.slot.read().ok().and_then(|s| s.clone()) + } + + fn store(&self, snapshot: StatusSnapshot) { + if let Ok(mut slot) = self.slot.write() { + *slot = Some(Arc::new(snapshot)); + } + } +} + +/// Refresh cadence: `CODESEARCH_STATUS_REFRESH_SECS` when a positive +/// integer, otherwise 2s. Unparseable values fall back to the default. +pub(crate) fn refresh_interval() -> Duration { + match std::env::var("CODESEARCH_STATUS_REFRESH_SECS") + .ok() + .and_then(|v| v.parse::().ok()) + { + Some(secs) if secs > 0 => Duration::from_secs(secs), + _ => Duration::from_secs(2), + } +} + +/// Spawn the background refresher. One pass runs at a time: the sleep +/// starts only after the previous pass completed, so under heavy load the +/// cadence stretches (visible as a growing `snapshot_age_secs`) instead of +/// piling measuring tasks onto a starved machine. The computation runs on +/// the blocking pool — it is filesystem work, not async work. +pub(crate) fn spawn(state: Arc) { + let handle = state.status_snapshot.clone(); + let interval = refresh_interval(); + tokio::spawn(async move { + loop { + let snap = tokio::task::spawn_blocking({ + let state = state.clone(); + move || state.compute_status_snapshot() + }) + .await; + match snap { + Ok(snapshot) => handle.store(snapshot), + Err(err) => { + // Join failure means the runtime is shutting down; the + // next loop iteration would also fail, but sleeping + // keeps shutdown semantics simple. + tracing::debug!("status snapshot pass failed: {err}"); + } + } + tokio::time::sleep(interval).await; + } + }); +} From e5a5069341bea59d315daf426720fc97ca362a36 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 14:38:52 +0200 Subject: [PATCH 2/5] feat: MCP status (scope-free) answers from the snapshot The scope-free serve branch of index_status called repo_statuses_lightweight() inline - the same measuring-on-request-path problem that made GET /status time out under a rebuild wave. It now reads the background snapshot via ServeState::latest_status_snapshot() and answers "starting" before the first pass completes, so the MCP status tool responds in microseconds at all times. --- src/mcp/status.rs | 27 ++++++++++++++++++++++++++- src/serve/mod.rs | 7 +++++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/src/mcp/status.rs b/src/mcp/status.rs index 18e2efca..4e9aa891 100644 --- a/src/mcp/status.rs +++ b/src/mcp/status.rs @@ -78,7 +78,32 @@ impl CodesearchService { // Count the virtual "all" group when repos are registered, so the // summary doesn't read "0 group(s)" while `all` is actually available. let group_count = config.groups.len() + if config.repos.is_empty() { 0 } else { 1 }; - let statuses = serve_state.repo_statuses_lightweight(); + // From the background snapshot, never an inline measure: under a + // rebuild wave the snapshot task is the ONLY thing allowed to + // touch the filesystem (the GET /status timeout fix) — and this + // tool must always answer too. Pre-first-pass answers "starting". + let Some(snap) = serve_state.latest_status_snapshot() else { + let response = IndexStatusResponse { + indexed: repo_count > 0, + status: "starting".to_string(), + status_message: + "serve is computing its first status snapshot; retry in a few seconds" + .to_string(), + total_chunks: 0, + total_files: 0, + model: self.model_type.short_name().to_string(), + dimensions: 0, + max_chunk_id: 0, + db_path: format!("({} repos)", repo_count), + project_path: format!("serve mode — {} repo(s)", repo_count), + error_message: None, + mode: self.mcp_mode(), + }; + let json = + serde_json::to_string(&response).unwrap_or_else(|_| "{}".to_string()); + return Ok(CallToolResult::success(vec![ContentBlock::text(json)])); + }; + let statuses = &snap.repos; let open_count = statuses .iter() .filter(|(_, r)| matches!(r.status, crate::serve::RepoStateLabel::Open)) diff --git a/src/serve/mod.rs b/src/serve/mod.rs index 8d1ccc5d..bdb9c409 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -714,6 +714,13 @@ impl ServeState { .is_none() } + /// The latest precomputed status snapshot, or `None` before the first + /// background pass completes. Read endpoints answer from this — never + /// by measuring inline (the /status timeout fix). + pub(crate) fn latest_status_snapshot(&self) -> Option> { + self.status_snapshot.get() + } + /// Return the instant when serve started, used to compute uptime. pub(crate) fn started_at(&self) -> std::time::Instant { self.started_at From c870f2f6d3fb68f5c79121b5235b7aa1a3d89d93 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 14:53:40 +0200 Subject: [PATCH 3/5] docs+test: status snapshot tests, changelog, env const Three unit tests over the snapshot logic: handle None-before-first-pass + roundtrip + age, compute on an empty fleet (incl. latest_status_snapshot being None pre-store), and the CODESEARCH_STATUS_REFRESH_SECS cadence env (7s override; unparseable/zero/empty all fall back to the 2s default - zero would busy-loop the measuring task). Env var name lives in constants.rs (STATUS_REFRESH_SECS_ENV). CHANGELOG under pending [1.5.33] + highlights line; plan ticked. Validation: cargo fmt --check clean, clippy --all-targets -D warnings clean (stage 2), cargo test --lib --bins 889+885 all green. --- AGENTS.develop.md | 1 + AGENTS.md | 4 +-- CHANGELOG.md | 6 ++++ src/constants.rs | 6 ++++ src/serve/status_snapshot.rs | 4 +-- src/serve/tests.rs | 63 ++++++++++++++++++++++++++++++++++++ 6 files changed, 80 insertions(+), 4 deletions(-) diff --git a/AGENTS.develop.md b/AGENTS.develop.md index b68a6d40..b3b9bf18 100644 --- a/AGENTS.develop.md +++ b/AGENTS.develop.md @@ -241,6 +241,7 @@ Never `unwrap_or_default()` a store error on a search path — "no results" and ## Changelog highlights (recent) +- **v1.5.33 (pending)** — /status + MCP status answer from a background snapshot (never measure inline): always responsive under rebuild waves, with `snapshot_age_secs` + a `load` summary for calling agents (`CODESEARCH_STATUS_REFRESH_SECS`, default 2s) - **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo - **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again - **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error diff --git a/AGENTS.md b/AGENTS.md index 741fec8d..f1608f20 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -264,5 +264,5 @@ Older entries: see `CHANGELOG.md`. /status must always answer: serve used to measure inline in the handler (28 repos x FS probes + SCIP LMDB opens + sysinfo refresh on async workers OS-starved by rebuild children -> tens of seconds, client timeouts). Now a background task (spawn_blocking, serial passes, CODESEARCH_STATUS_REFRESH_SECS default 2s) computes a StatusSnapshot into a swap-only RwLock slot; /status answers O(1) with snapshot_age_secs + a load summary (text_indexing / symbol_indexing / errored / refresh_interval_secs); before the first pass it answers "starting" immediately. Stage 2: MCP status tool via snapshot. Stage 3: tests + CHANGELOG + validation. 1. [x] Stage 1: status_snapshot.rs + ServeState field + compute_status_snapshot + new status_handler + spawn at startup -2. [ ] Stage 2: MCP status (kind=index) reads the snapshot -3. [ ] Stage 3: tests + CHANGELOG + full validation +2. [x] Stage 2: MCP status (kind=index) reads the snapshot +3. [x] Stage 3: tests + CHANGELOG + full validation diff --git a/CHANGELOG.md b/CHANGELOG.md index 7f1d1f1d..6930d7a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,12 @@ more PRs land; when the release is actually tagged, the same section is finalized in place with a date — no renaming/migration step needed. --> +## [1.5.33] + +### Fixed + +- **`GET /status` and the MCP `status` tool always answer, even under a heavy rebuild wave.** They used to measure inline in the request path: per registered repo several filesystem probes plus a SCIP LMDB env open and a sysinfo refresh, synchronous on async workers that a rebuild wave (dotnet restore + scip-csharp children) starves — with 28 repos one call took tens of seconds and polling clients (remote TUI, calling agents) piled up and timed out; the project list died the same death. Now a single background task measures (on the blocking pool, one pass at a time — under load the cadence stretches instead of piling measuring tasks onto a starved machine) and stores a snapshot whose readers answer in microseconds. Every answer carries `snapshot_age_secs` (staleness is visible, never a hang) and a `load` summary (`text_indexing` / `symbol_indexing` / `errored` / `refresh_interval_secs`) so calling agents can throttle themselves; before the first pass completes they answer `"starting"` immediately. Cadence: `CODESEARCH_STATUS_REFRESH_SECS`, default 2s. + ## [1.5.31] ### Changed diff --git a/src/constants.rs b/src/constants.rs index 5f7dbc34..197609a1 100644 --- a/src/constants.rs +++ b/src/constants.rs @@ -616,6 +616,12 @@ pub const REMOTE_PEER_RETRY_BACKOFF_MS: &[u64] = &[3000, 8000]; /// single millisecond value (test hook so retry tests don't sleep for real). pub const REMOTE_PEER_RETRY_BACKOFF_ENV: &str = "CODESEARCH_REMOTE_RETRY_BACKOFF_MS"; +/// Cadence of the background status-snapshot refresher (seconds). Read +/// endpoints answer from the stored snapshot, so this only trades staleness +/// (`snapshot_age_secs`) for probing cost — under load the cadence +/// stretches on its own; this knob only sets the idle pace. +pub const STATUS_REFRESH_SECS_ENV: &str = "CODESEARCH_STATUS_REFRESH_SECS"; + /// Cooperative join window (seconds) for `await_index_task` and /// `await_fsw_shutdown`: how long a background indexing / file-watcher task /// is given to observe its `CancellationToken` and exit on its own before it diff --git a/src/serve/status_snapshot.rs b/src/serve/status_snapshot.rs index a1c8c439..41deee5d 100644 --- a/src/serve/status_snapshot.rs +++ b/src/serve/status_snapshot.rs @@ -55,7 +55,7 @@ impl StatusSnapshotHandle { self.slot.read().ok().and_then(|s| s.clone()) } - fn store(&self, snapshot: StatusSnapshot) { + pub(crate) fn store(&self, snapshot: StatusSnapshot) { if let Ok(mut slot) = self.slot.write() { *slot = Some(Arc::new(snapshot)); } @@ -65,7 +65,7 @@ impl StatusSnapshotHandle { /// Refresh cadence: `CODESEARCH_STATUS_REFRESH_SECS` when a positive /// integer, otherwise 2s. Unparseable values fall back to the default. pub(crate) fn refresh_interval() -> Duration { - match std::env::var("CODESEARCH_STATUS_REFRESH_SECS") + match std::env::var(crate::constants::STATUS_REFRESH_SECS_ENV) .ok() .and_then(|v| v.parse::().ok()) { diff --git a/src/serve/tests.rs b/src/serve/tests.rs index 44e9897e..a2125347 100644 --- a/src/serve/tests.rs +++ b/src/serve/tests.rs @@ -5010,3 +5010,66 @@ async fn sweeper_reasserts_cancellation_for_a_removed_live_task() { } } } + +// ── status snapshot ────────────────────────────────────────────────────── + +#[test] +fn status_snapshot_handle_starts_empty_and_roundtrips() { + let handle = status_snapshot::StatusSnapshotHandle::new(); + // Before the first background pass handlers must see None and answer + // their "starting" shape — never measure inline. + assert!(handle.get().is_none()); + + let snap = status_snapshot::StatusSnapshot { + repos: Vec::new(), + active_sessions: 0, + cpu_percent: "0%".to_string(), + csharp_helper: false, + ts_helper: false, + generated_at: std::time::Instant::now(), + }; + handle.store(snap); + + let got = handle.get().expect("snapshot readable after store"); + assert_eq!(got.cpu_percent, "0%"); + assert!(got.age_secs() <= 1, "fresh snapshot must have ~0 age"); +} + +#[test] +fn compute_status_snapshot_reports_empty_fleet_without_state() { + let state = ServeState::new(ReposConfig::default(), None); + assert!(state.latest_status_snapshot().is_none()); + + let snap = state.compute_status_snapshot(); + assert!(snap.repos.is_empty()); + assert_eq!(snap.active_sessions, 0); + assert!(!snap.cpu_percent.is_empty(), "cpu is always a string"); + assert!(snap.age_secs() <= 1); +} + +#[test] +#[serial] +fn status_refresh_interval_env_overrides() { + { + let _env = + crate::testing::EnvRestore::set(&[(crate::constants::STATUS_REFRESH_SECS_ENV, "7")]); + assert_eq!(status_snapshot::refresh_interval(), Duration::from_secs(7)); + } + { + // Unparseable falls back to the default rather than failing. + let _env = + crate::testing::EnvRestore::set(&[(crate::constants::STATUS_REFRESH_SECS_ENV, "abc")]); + assert_eq!(status_snapshot::refresh_interval(), Duration::from_secs(2)); + } + { + // Zero would busy-loop the measuring task; the default floor applies. + let _env = + crate::testing::EnvRestore::set(&[(crate::constants::STATUS_REFRESH_SECS_ENV, "0")]); + assert_eq!(status_snapshot::refresh_interval(), Duration::from_secs(2)); + } + { + let _env = + crate::testing::EnvRestore::set(&[(crate::constants::STATUS_REFRESH_SECS_ENV, "")]); + assert_eq!(status_snapshot::refresh_interval(), Duration::from_secs(2)); + } +} From 52b0082201cb057f87789a9d140aced87d4a2543 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 15:12:54 +0200 Subject: [PATCH 4/5] fix: projects list answers from the snapshot + handler tests Final-review findings: (1) list_projects (MCP status kind=projects) was the last read endpoint still measuring inline - metadata reads, try_read stats and lock probes per repo on the request path, the exact second timeout the user reported on 2026-10-16. The per-repo probe pass moved into compute_projects_snapshot() (the only measurer, called from the background task); list_projects clones the stored payload and answers immediately; pre-first-pass it answers an empty list with an explicit warming warning (channel terminates - never a silent empty answer). RepoInfo gains Clone for the slot-out clone. (2) Handler-contract tests: status_handler starting-shape + full stored pass (legacy fields, load counters, numeric age) and MCP status/projects pre-first-pass answers. Review-fixes: - [Important][Test] changed handler behavior had no test -> direct async calls of status_handler and the status tool pin the starting shapes, legacy JSON fields and the load counters. - [Design] list_projects still measured inline -> snapshot-ized via compute_projects_snapshot; user's second timeout path now covered. --- src/mcp/mod.rs | 2 + src/mcp/status.rs | 197 ++++++++++++++++++++--------------- src/mcp/types.rs | 4 +- src/serve/mod.rs | 5 + src/serve/status_snapshot.rs | 8 ++ src/serve/tests.rs | 78 ++++++++++++++ 6 files changed, 209 insertions(+), 85 deletions(-) diff --git a/src/mcp/mod.rs b/src/mcp/mod.rs index ac0a38a7..530e1b1e 100644 --- a/src/mcp/mod.rs +++ b/src/mcp/mod.rs @@ -121,6 +121,8 @@ pub(crate) use helpers::*; pub(crate) use instructions::*; pub(crate) use responses::*; pub use runtime::*; +/// The projects-measure pass for the serve status-snapshot background task. +pub(crate) use status::compute_projects_snapshot; /// Re-order lexical `usages` hits so source-code paths come before everything /// else (docs, configs, markdown). Stable: score order is preserved within diff --git a/src/mcp/status.rs b/src/mcp/status.rs index 4e9aa891..29dba823 100644 --- a/src/mcp/status.rs +++ b/src/mcp/status.rs @@ -16,6 +16,102 @@ const LOCK_STATUS_HELD_IN_PROCESS: &str = "held-in-process"; const LOCK_STATUS_LOCKED_EXTERNALLY: &str = "locked-externally"; const LOCK_STATUS_AVAILABLE: &str = "available"; +/// One full projects-measure pass for the background status-snapshot task +/// — the ONLY place these per-repo probes (metadata reads, try_read stats, +/// lock detection) run. `list_projects` answers from the stored result and +/// must never call this on the request path. +pub(crate) fn compute_projects_snapshot( + serve_state: &crate::serve::ServeState, +) -> (Vec, Vec) { + let config = serve_state.config_snapshot(); + let project_groups = config.project_groups(); + let mut repos_info = Vec::new(); + let mut list_warnings: Vec = Vec::new(); + + for (alias, path) in &config.repos { + let db_path = path.join(crate::constants::DB_DIR_NAME); + + let (total_chunks, total_files, model, lock_status, error) = if db_path.exists() { + let (model_name, _dims) = read_model_metadata(&db_path); + + // For repos already opened in DashMap, use the live SharedStores for stats + // WITHOUT opening a new VectorStore connection. + // For unopened repos, just report metadata — do NOT open the DB. + let opened = serve_state.get_opened_stores(alias); + // Never wait on the store lock: an indexing run holds it for + // whole batches, and status must answer immediately. + let stats_result = opened + .as_ref() + .and_then(|stores| stores.vector_store.try_read().ok().map(|vs| vs.stats())); + if let Some(stats_result) = stats_result { + // `0 chunks` alone reads exactly like "not indexed yet" — the + // repo may in fact be full and simply failing to answer (the + // read-only-incident shape this branch exists for). Attribute + // the failure to THIS repo rather than a top-level channel: + // list_projects returns one entry per repo, so per-item is the + // shape that actually matches the fan-out. + // `repo_stats_from_result` carries only the part of this + // decision that varies by Ok/Err — see its doc comment. + // `record_stats_or_warn` wraps it so this call site cannot + // silently drop the warning half without also breaking the + // counts it returns — see its own doc comment. + let (total_chunks, total_files, error) = + record_stats_or_warn(stats_result, alias, &mut list_warnings); + ( + total_chunks, + total_files, + model_name, + serve_state + .repo_lock_status(alias) + .unwrap_or("unknown") + .to_string(), + error, + ) + } else { + // Not opened, or its store is busy — persisted stats from + // metadata.json; do NOT open the DB. Our own holders are + // checked before the file-lock probe, which cannot tell + // this process apart from another one. + let (md_chunks, md_files) = read_metadata_stats(&db_path); + let lock_status = if serve_state.is_indexing(alias) { + LOCK_STATUS_INDEXING + } else if opened.is_some() { + LOCK_STATUS_BUSY + } else if serve_state.is_held_in_process(alias, &db_path) { + LOCK_STATUS_HELD_IN_PROCESS + } else if crate::index::is_database_locked(&db_path) { + LOCK_STATUS_LOCKED_EXTERNALLY + } else { + LOCK_STATUS_AVAILABLE + }; + ( + md_chunks, + md_files, + model_name, + lock_status.to_string(), + None, + ) + } + } else { + (0, 0, "not indexed".to_string(), "unknown".to_string(), None) + }; + + repos_info.push(RepoInfo { + alias: alias.clone(), + project_path: path.display().to_string(), + database_path: db_path.display().to_string(), + total_chunks, + total_files, + model, + lock_status, + groups: project_groups.get(alias).cloned().unwrap_or_default(), + error, + }); + } + + (repos_info, list_warnings) +} + #[tool_router(router = status_router, vis = "pub(crate)")] impl CodesearchService { /// Unified status tool — dispatches based on `kind`. @@ -376,90 +472,23 @@ impl CodesearchService { // When serve is active, use ServeState as source of truth for lock status if let Some(ref serve_state) = self.serve_state { let config = serve_state.config_snapshot(); - let project_groups = config.project_groups(); - let mut repos_info = Vec::new(); - let mut list_warnings: Vec = Vec::new(); - - for (alias, path) in &config.repos { - let db_path = path.join(crate::constants::DB_DIR_NAME); - - let (total_chunks, total_files, model, lock_status, error) = if db_path.exists() { - let (model_name, _dims) = read_model_metadata(&db_path); - - // For repos already opened in DashMap, use the live SharedStores for stats - // WITHOUT opening a new VectorStore connection. - // For unopened repos, just report metadata — do NOT open the DB. - let opened = serve_state.get_opened_stores(alias); - // Never wait on the store lock: an indexing run holds it for - // whole batches, and status must answer immediately. - let stats_result = opened.as_ref().and_then(|stores| { - stores.vector_store.try_read().ok().map(|vs| vs.stats()) - }); - if let Some(stats_result) = stats_result { - // `0 chunks` alone reads exactly like "not indexed yet" — the - // repo may in fact be full and simply failing to answer (the - // read-only-incident shape this branch exists for). Attribute - // the failure to THIS repo rather than a top-level channel: - // list_projects returns one entry per repo, so per-item is the - // shape that actually matches the fan-out. - // `repo_stats_from_result` carries only the part of this - // decision that varies by Ok/Err — see its doc comment. - // `record_stats_or_warn` wraps it so this call site cannot - // silently drop the warning half without also breaking the - // counts it returns — see its own doc comment. - let (total_chunks, total_files, error) = - record_stats_or_warn(stats_result, alias, &mut list_warnings); - ( - total_chunks, - total_files, - model_name, - serve_state - .repo_lock_status(alias) - .unwrap_or("unknown") - .to_string(), - error, - ) - } else { - // Not opened, or its store is busy — persisted stats from - // metadata.json; do NOT open the DB. Our own holders are - // checked before the file-lock probe, which cannot tell - // this process apart from another one. - let (md_chunks, md_files) = read_metadata_stats(&db_path); - let lock_status = if serve_state.is_indexing(alias) { - LOCK_STATUS_INDEXING - } else if opened.is_some() { - LOCK_STATUS_BUSY - } else if serve_state.is_held_in_process(alias, &db_path) { - LOCK_STATUS_HELD_IN_PROCESS - } else if crate::index::is_database_locked(&db_path) { - LOCK_STATUS_LOCKED_EXTERNALLY - } else { - LOCK_STATUS_AVAILABLE - }; - ( - md_chunks, - md_files, - model_name, - lock_status.to_string(), - None, - ) - } - } else { - (0, 0, "not indexed".to_string(), "unknown".to_string(), None) - }; - - repos_info.push(RepoInfo { - alias: alias.clone(), - project_path: path.display().to_string(), - database_path: db_path.display().to_string(), - total_chunks, - total_files, - model, - lock_status, - groups: project_groups.get(alias).cloned().unwrap_or_default(), - error, - }); - } + // From the background snapshot, never an inline measure — the same + // contract as /status and status(kind=index): under a rebuild wave + // the measuring task is the only thing that touches the filesystem, + // and the project list must still answer (it was the second timeout + // in the 2026-10-16 field case). Pre-first-pass answers an empty + // list with an explicit warming warning instead of hanging. + let (repos_info, list_warnings) = match serve_state.latest_status_snapshot() { + Some(snap) => (snap.projects.clone(), snap.projects_warnings.clone()), + None => ( + Vec::new(), + vec![ + "status snapshot still warming up — the project list is empty for a few \ + seconds after serve startup; retry shortly" + .to_string(), + ], + ), + }; let response = ListProjectsResponse { repos: repos_info, diff --git a/src/mcp/types.rs b/src/mcp/types.rs index e113eff8..67df7f68 100644 --- a/src/mcp/types.rs +++ b/src/mcp/types.rs @@ -526,7 +526,9 @@ pub struct RemoteProjectInfo { } /// Information about a single registered project/repo. -#[derive(Debug, Serialize)] +/// `Clone`: the snapshot stores the computed list and every projects answer +/// clones it out (the slot lock is never held across response building). +#[derive(Debug, Clone, Serialize)] pub struct RepoInfo { pub alias: String, pub project_path: String, diff --git a/src/serve/mod.rs b/src/serve/mod.rs index bdb9c409..d62aeb19 100644 --- a/src/serve/mod.rs +++ b/src/serve/mod.rs @@ -3855,8 +3855,13 @@ impl ServeState { .map(|i| i.is_available()) .unwrap_or(false); + let (snapshot_projects, snapshot_projects_warnings) = + crate::mcp::compute_projects_snapshot(self); + status_snapshot::StatusSnapshot { repos, + projects: snapshot_projects, + projects_warnings: snapshot_projects_warnings, active_sessions, cpu_percent: cpu, csharp_helper, diff --git a/src/serve/status_snapshot.rs b/src/serve/status_snapshot.rs index 41deee5d..99ff02b0 100644 --- a/src/serve/status_snapshot.rs +++ b/src/serve/status_snapshot.rs @@ -21,6 +21,14 @@ use super::{RepoStatusInfo, ServeState}; /// One computed status pass. Cheap to clone (handlers get an `Arc`). pub(crate) struct StatusSnapshot { pub(crate) repos: Vec<(String, RepoStatusInfo)>, + /// Precomputed per-repo project-list payload (metadata reads, store + /// stats via try_read, lock probes) — list_projects answers from this + /// for the same reason /status does: the request path never measures. + pub(crate) projects: Vec, + /// Warnings captured while computing `projects` (per-repo stats + /// failures). Re-emitted verbatim on each projects answer so a repo + /// that failed to report stays visible. + pub(crate) projects_warnings: Vec, pub(crate) active_sessions: u64, pub(crate) cpu_percent: String, pub(crate) csharp_helper: bool, diff --git a/src/serve/tests.rs b/src/serve/tests.rs index a2125347..c1db245c 100644 --- a/src/serve/tests.rs +++ b/src/serve/tests.rs @@ -5022,6 +5022,8 @@ fn status_snapshot_handle_starts_empty_and_roundtrips() { let snap = status_snapshot::StatusSnapshot { repos: Vec::new(), + projects: Vec::new(), + projects_warnings: Vec::new(), active_sessions: 0, cpu_percent: "0%".to_string(), csharp_helper: false, @@ -5073,3 +5075,79 @@ fn status_refresh_interval_env_overrides() { assert_eq!(status_snapshot::refresh_interval(), Duration::from_secs(2)); } } + +#[tokio::test] +async fn status_handler_answers_starting_then_snapshot_with_load() { + let state = Arc::new(ServeState::new(ReposConfig::default(), None)); + + // Before the first background pass: immediate "starting" answer, no + // measurement, null age. + let res = status_handler(axum::extract::State(state.clone())).await; + assert_eq!(res.0["status"], "starting"); + assert!(res.0["snapshot_age_secs"].is_null()); + assert!(res.0["repos"].is_array()); + + // After a pass is stored: the full contract callers depend on — legacy + // fields preserved, load counters present, age numeric. + state.status_snapshot.store(state.compute_status_snapshot()); + let res = status_handler(axum::extract::State(state)).await; + let v = res.0; + assert!( + v["status"].is_null(), + "stored pass must not read 'starting'" + ); + assert_eq!(v["load"]["text_indexing"], 0); + assert_eq!(v["load"]["symbol_indexing"], 0); + assert_eq!(v["load"]["errored"], 0); + assert!(v["load"]["refresh_interval_secs"].is_u64()); + assert!(v["snapshot_age_secs"].is_u64()); + assert!(v["version"].is_string()); + assert!(v["uptime_secs"].is_u64()); +} + +#[tokio::test] +async fn mcp_status_and_projects_answer_starting_pre_first_pass() { + let state = Arc::new(ServeState::new(ReposConfig::default(), None)); + let service = crate::mcp::CodesearchService::new_for_serve(state).unwrap(); + + // kind=index, no snapshot yet: structured "starting" answer. + let res = service + .status(rmcp::handler::server::wrapper::Parameters( + crate::mcp::types::StatusRequest { + kind: Some("index".to_string()), + project: None, + group: None, + }, + )) + .await + .expect("status must not error"); + let text = match res.content.first() { + Some(rmcp::model::ContentBlock::Text(t)) => t.text.clone(), + other => panic!("expected text content, got {other:?}"), + }; + assert!( + text.contains("\"starting\""), + "scope-free index status must answer starting pre-first-pass, got: {text}" + ); + + // kind=projects, no snapshot yet: empty list + explicit warming warning + // (the warnings channel terminates — never a silent empty answer). + let res = service + .status(rmcp::handler::server::wrapper::Parameters( + crate::mcp::types::StatusRequest { + kind: Some("projects".to_string()), + project: None, + group: None, + }, + )) + .await + .expect("projects must not error"); + let text = match res.content.first() { + Some(rmcp::model::ContentBlock::Text(t)) => t.text.clone(), + other => panic!("expected text content, got {other:?}"), + }; + assert!( + text.contains("warming up"), + "projects pre-first-pass must carry the warming warning, got: {text}" + ); +} From ae0159c75bb0abfc2824d67211d86718a3a126d3 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 15:47:07 +0200 Subject: [PATCH 5/5] chore: drop branch-local agent files --- AGENTS.md | 268 ------------------------------------------------------ CLAUDE.md | 1 - 2 files changed, 269 deletions(-) delete mode 100644 AGENTS.md delete mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index f1608f20..00000000 --- a/AGENTS.md +++ /dev/null @@ -1,268 +0,0 @@ -# AGENTS.develop.md - -This file is the **develop-branch reference** for all coding agents. -It is committed on `develop` and merged from every feature branch. - -`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the -feature branch while building, dropped before the PR merges — they never appear -on `develop` or `master`. See *For agents starting a new feature branch* below. - -## Setup (once per machine) - -```bash -git config core.hooksPath .githooks -``` - -This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this -file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs -`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` -blocks direct pushes to `master`. See `.githooks/README.md`. - -## For agents starting a new feature branch - -If `AGENTS.md` was not auto-created by the hook, create it manually: - -```bash -cp AGENTS.develop.md AGENTS.md -``` - -Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. -Leave the architecture sections intact — they provide context. - -**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan -travels through the PR and survives branch switches. - -**At the end of the branch, before the PR (the changelog moment):** -1. Update `## Active feature branches` below (remove this branch) -2. Add one line to `## Changelog highlights` -3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) -4. Drop the branch-local files and commit: - `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` - -Enforced mechanically (forgetting step 4 cannot leak the files): -- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on - `develop`/`master` (direct commits and conflict resolutions). -- `pre-merge-commit` runs the same guard on the merge result — a clean - merge never invokes pre-commit. -- The `agent-files-check` CI workflow flags any PR into `develop`/`master` - whose head still carries them. - -**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. -This file contains only architecture, conventions, and changelog. - ---- - -## ⚠️ Branching & PR workflow (READ FIRST) - -This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. - -- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. -- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). -- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. -- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. -- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. -- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. - -Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. - ---- - -## What codesearch is - -A fast, local, offline MCP server for semantic code search. Single Rust binary. -No Docker, no cloud, no external services. Designed for coding agents (OpenCode, -Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. - -Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + -tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). -Hybrid BM25 + vector search fused via RRF. - -## MCP tools - -Five tools exposed to agents: - -| Tool | Description | -|---|---| -| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | -| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | -| `explore` | File outline or similar-chunk lookup. Requires scope. | -| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | -| `status` | Index and project status. Lightweight (no DB open) when called without scope. | - -All tools return `scope_required` structured errors in multi-repo mode when no `project` -or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. - -## Multi-repo serve mode - -`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). -Multiple repos register via `repos.json` with aliases. Agents route queries per-alias -or per-group. - -**Repo lifecycle:** -- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. -- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. -- `Readonly` — Another process holds the write lock. -- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. - -**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. -All opens update `last_access` so the reaper can track every open. - -**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, -no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. - -## TUI - -`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). -Without TTY: headless, logs only. - -- `codesearch serve --no-tui` — suppress TUI even with TTY -- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` - -## HTTP endpoints (serve mode) - -| Endpoint | Method | Description | -|---|---|---| -| `/health` | GET | Health check JSON | -| `/status` | GET | Lightweight repo state snapshot for TUI polling | -| `/repos` | POST | Register + index + warmup a new repo | -| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | -| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | -| `/mcp` | GET/POST | MCP streamable HTTP endpoint | - -## Supported languages - -17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol -precision for C# (bundled `scip-csharp`, resident helper via its `serve` -subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware -chunking only. - -## Release artifacts - -GitHub Actions produces release binaries per tag (plain + `-with-csharp` -variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. -Single-file native binaries, no runtime dependencies. macOS build is manual-trigger -only (expensive runners). - ---- - -## Current state - -- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. -- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. -- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. - -## Implemented features (orientation only — narratives live in CHANGELOG.md) - -- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. -- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. -- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. -- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. - -## ⚠️ Design constraint — never poll a federated peer - -**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's -scale-to-zero replica, which then self-warms its full idle window (measured ~50% -duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — -do not re-attempt. Local repos may be polled in the background; peers are -contacted only by a real federated tool call (activity poke) or the explicit -TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). - -## Open TODOs - -- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. - ---- - -## Key conventions for agents - -- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). -- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. -- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). -- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. -- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. -- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. -- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. -- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. - -## Runtime locations - -- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. -- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). -- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. -- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. - -## Deploying to runtime - -- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** -- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` -- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. -- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. - -## Notes for agents - -- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. -- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. -- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. -- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. -- **Never call `.canonicalize()`** — use `safe_canonicalize()`. -- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. -- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. - -### LMDB rules - -- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). -- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. -- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. -- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). - -### Search errors must not become empty results - -Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: - -- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. -- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. -- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. -- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. -- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. -- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. -- **Suppress `suggested_tool` when warnings are present.** -- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. -- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** -- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. - ---- - -## Active feature branches (not yet merged) - -| Branch | Description | -|---|---| - ---- - -## Changelog highlights (recent) - -- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo -- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again -- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error -- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings -- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback -- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) -- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered -- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard -- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes -- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds -- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) -- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories -- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) -- **v1.3.16** — REST `/find-impact` endpoint -- **v1.3.3** — federation hardening release - -Older entries: see `CHANGELOG.md`. - -## Plan - -/status must always answer: serve used to measure inline in the handler (28 repos x FS probes + SCIP LMDB opens + sysinfo refresh on async workers OS-starved by rebuild children -> tens of seconds, client timeouts). Now a background task (spawn_blocking, serial passes, CODESEARCH_STATUS_REFRESH_SECS default 2s) computes a StatusSnapshot into a swap-only RwLock slot; /status answers O(1) with snapshot_age_secs + a load summary (text_indexing / symbol_indexing / errored / refresh_interval_secs); before the first pass it answers "starting" immediately. Stage 2: MCP status tool via snapshot. Stage 3: tests + CHANGELOG + validation. - -1. [x] Stage 1: status_snapshot.rs + ServeState field + compute_status_snapshot + new status_handler + spawn at startup -2. [x] Stage 2: MCP status (kind=index) reads the snapshot -3. [x] Stage 3: tests + CHANGELOG + full validation diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 5d302681..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -Read AGENTS.md.