From a7c80ca0c3c08a0bd01739fe436796d77af52afe Mon Sep 17 00:00:00 2001 From: flupkede Date: Thu, 15 Oct 2026 20:56:03 +0200 Subject: [PATCH 1/6] chore: branch-local agent files with plan --- AGENTS.md | 267 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 1 + 2 files changed, 268 insertions(+) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..d771d17a --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,267 @@ +# AGENTS.develop.md + +This file is the **develop-branch reference** for all coding agents. +It is committed on `develop` and merged from every feature branch. + +`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the +feature branch while building, dropped before the PR merges — they never appear +on `develop` or `master`. See *For agents starting a new feature branch* below. + +## Setup (once per machine) + +```bash +git config core.hooksPath .githooks +``` + +This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this +file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs +`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` +blocks direct pushes to `master`. See `.githooks/README.md`. + +## For agents starting a new feature branch + +If `AGENTS.md` was not auto-created by the hook, create it manually: + +```bash +cp AGENTS.develop.md AGENTS.md +``` + +Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. +Leave the architecture sections intact — they provide context. + +**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan +travels through the PR and survives branch switches. + +**At the end of the branch, before the PR (the changelog moment):** +1. Update `## Active feature branches` below (remove this branch) +2. Add one line to `## Changelog highlights` +3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) +4. Drop the branch-local files and commit: + `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` + +Enforced mechanically (forgetting step 4 cannot leak the files): +- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on + `develop`/`master` (direct commits and conflict resolutions). +- `pre-merge-commit` runs the same guard on the merge result — a clean + merge never invokes pre-commit. +- The `agent-files-check` CI workflow flags any PR into `develop`/`master` + whose head still carries them. + +**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. +This file contains only architecture, conventions, and changelog. + +--- + +## ⚠️ Branching & PR workflow (READ FIRST) + +This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. + +- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. +- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). +- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. +- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. +- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. +- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. + +Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. + +--- + +## What codesearch is + +A fast, local, offline MCP server for semantic code search. Single Rust binary. +No Docker, no cloud, no external services. Designed for coding agents (OpenCode, +Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. + +Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + +tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). +Hybrid BM25 + vector search fused via RRF. + +## MCP tools + +Five tools exposed to agents: + +| Tool | Description | +|---|---| +| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | +| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | +| `explore` | File outline or similar-chunk lookup. Requires scope. | +| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | +| `status` | Index and project status. Lightweight (no DB open) when called without scope. | + +All tools return `scope_required` structured errors in multi-repo mode when no `project` +or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. + +## Multi-repo serve mode + +`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). +Multiple repos register via `repos.json` with aliases. Agents route queries per-alias +or per-group. + +**Repo lifecycle:** +- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. +- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. +- `Readonly` — Another process holds the write lock. +- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. + +**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. +All opens update `last_access` so the reaper can track every open. + +**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, +no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. + +## TUI + +`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). +Without TTY: headless, logs only. + +- `codesearch serve --no-tui` — suppress TUI even with TTY +- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` + +## HTTP endpoints (serve mode) + +| Endpoint | Method | Description | +|---|---|---| +| `/health` | GET | Health check JSON | +| `/status` | GET | Lightweight repo state snapshot for TUI polling | +| `/repos` | POST | Register + index + warmup a new repo | +| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | +| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | +| `/mcp` | GET/POST | MCP streamable HTTP endpoint | + +## Supported languages + +17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol +precision for C# (bundled `scip-csharp`, resident helper via its `serve` +subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware +chunking only. + +## Release artifacts + +GitHub Actions produces release binaries per tag (plain + `-with-csharp` +variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. +Single-file native binaries, no runtime dependencies. macOS build is manual-trigger +only (expensive runners). + +--- + +## Current state + +- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. +- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. +- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. + +## Implemented features (orientation only — narratives live in CHANGELOG.md) + +- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. +- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. +- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. +- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. + +## ⚠️ Design constraint — never poll a federated peer + +**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's +scale-to-zero replica, which then self-warms its full idle window (measured ~50% +duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — +do not re-attempt. Local repos may be polled in the background; peers are +contacted only by a real federated tool call (activity poke) or the explicit +TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). + +## Open TODOs + +- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. + +--- + +## Key conventions for agents + +- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). +- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. +- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). +- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. +- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. +- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. +- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. +- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. + +## Runtime locations + +- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. +- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). +- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. +- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. + +## Deploying to runtime + +- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** +- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` +- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. +- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. + +## Notes for agents + +- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. +- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. +- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. +- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. +- **Never call `.canonicalize()`** — use `safe_canonicalize()`. +- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. +- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. + +### LMDB rules + +- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). +- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. +- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. +- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). + +### Search errors must not become empty results + +Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: + +- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. +- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. +- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. +- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. +- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. +- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. +- **Suppress `suggested_tool` when warnings are present.** +- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. +- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** +- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. + +--- + +## Active feature branches (not yet merged) + +| Branch | Description | +|---|---| + +--- + +## Changelog highlights (recent) + +- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again +- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error +- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings +- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback +- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) +- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered +- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard +- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes +- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds +- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) +- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories +- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) +- **v1.3.16** — REST `/find-impact` endpoint +- **v1.3.3** — federation hardening release + +Older entries: see `CHANGELOG.md`. + +## Plan + +VS-MSBuild restore fallback in scip-csharp: when `dotnet restore` fails with SDK-resolution errors (NETSDK1045-class) because the CLI-visible SDK set cannot target the repo TFM, locate the newest VS MSBuild via vswhere and retry the solution restore once with `MSBuild -t:restore`. On success the full solution has assets; on failure/timeout existing per-project fallback (or pending-marker) applies. Motivated by HUSQ worktree incident 2026-10-15: VS-bundled net10 SDK invisible to CLI dotnet. + +1. [x] Stage 1: detection + fallback in helpers/csharp (MsbuildRestore.cs + wiring in Program.cs) +2. [ ] Stage 2: unit tests (SDK-failure detection, vswhere output parsing) +3. [ ] Stage 3: CHANGELOG + AGENTS.develop.md highlights + validation diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..5d302681 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +Read AGENTS.md. From 4574c2349ae3c32ad8591b13a4f7d8d3ed21f2b1 Mon Sep 17 00:00:00 2001 From: flupkede Date: Thu, 15 Oct 2026 21:04:55 +0200 Subject: [PATCH 2/6] feat: VS-MSBuild restore fallback when the CLI SDK set cannot target the repo When dotnet restore fails on SDK resolution (NETSDK1045-class: CLI SDKs older than the repo TFM, or global.json unsatisfiable) scip-csharp now retries the solution restore once with the newest VS MSBuild located via vswhere - VS ships its own MSBuild + SDK targets independent of C:\Program Files\dotnet, so it restores what the CLI cannot even evaluate. SDK-resolution detection reads stdout AND stderr: NETSDK1045 prints to stdout on current dotnet hosts (verified live with a net99.0 probe). Bounded wait with the existing pending-marker/gated-exit contract; on failure the per-project fallback still runs (mixed-TFM recovery preserved). The MSBuild instance choice for the workspace load is now capability-aware too: when every CLI SDK instance is older than the repo's raw-text TFM floor (max over all TFMs + global.json major), the highest VS instance is registered instead - the SDK preference stands everywhere else, including its net472-BuildHost-crash rationale. Motivated by the 2026-10-15 HUSQ worktree incident (hollow 10.0.101 SDK dir left the CLI blind to net10 while VS built fine). Deliberate behavior change: SDK-instance pick is the HIGHEST qualifying SDK (was first-enumerated, an undocumented order); single-SDK machines unaffected. Review-fixes: - [Important] NETSDK1045/global.json detection read stderr only while the trigger text prints to stdout on current hosts -> classify over both streams; match current ('compatible .NET SDK') and legacy ('compatible installed .NET SDK') global.json wordings. - [Important] multi-target floor took the FIRST TFM (net8.0;net10.0 floored at 8) -> take the max over all TFMs. - [Important] inline 10_000 vswhere wait -> VsWhereWaitMs const. - [Important] healthy-machine invariant weakened on multi-SDK machines -> kept highest-qualifying (matches the original 'highest available MSBuild' intent) and recorded the change in the branch plan. --- AGENTS.md | 2 +- helpers/csharp/MsbuildRestore.cs | 285 +++++++++++++++++++ helpers/csharp/Program.cs | 167 ++++++++++- helpers/csharp/RestorePolicy.cs | 23 +- helpers/csharp/tests/ScipCsharp.Tests.csproj | 1 + 5 files changed, 453 insertions(+), 25 deletions(-) create mode 100644 helpers/csharp/MsbuildRestore.cs diff --git a/AGENTS.md b/AGENTS.md index d771d17a..fa8607f9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -262,6 +262,6 @@ Older entries: see `CHANGELOG.md`. VS-MSBuild restore fallback in scip-csharp: when `dotnet restore` fails with SDK-resolution errors (NETSDK1045-class) because the CLI-visible SDK set cannot target the repo TFM, locate the newest VS MSBuild via vswhere and retry the solution restore once with `MSBuild -t:restore`. On success the full solution has assets; on failure/timeout existing per-project fallback (or pending-marker) applies. Motivated by HUSQ worktree incident 2026-10-15: VS-bundled net10 SDK invisible to CLI dotnet. -1. [x] Stage 1: detection + fallback in helpers/csharp (MsbuildRestore.cs + wiring in Program.cs) +1. [x] Stage 1: detection + fallback in helpers/csharp (MsbuildRestore.cs + wiring in Program.cs). Deliberate behavior change: SDK-instance pick is now the HIGHEST qualifying SDK (was first-enumerated, an undocumented order) — matches the original "register the highest available MSBuild" intent; single-SDK machines unaffected. 2. [ ] Stage 2: unit tests (SDK-failure detection, vswhere output parsing) 3. [ ] Stage 3: CHANGELOG + AGENTS.develop.md highlights + validation diff --git a/helpers/csharp/MsbuildRestore.cs b/helpers/csharp/MsbuildRestore.cs new file mode 100644 index 00000000..869b85e0 --- /dev/null +++ b/helpers/csharp/MsbuildRestore.cs @@ -0,0 +1,285 @@ +namespace ScipCsharp; + +/// +/// One installable MSBuild candidate for the capability-aware instance +/// choice: either the MSBuild of a .NET SDK install (CLI-visible) or of a +/// Visual Studio instance (bundled with VS, independent of +/// C:\Program Files\dotnet). Dependency-free so the decision logic +/// is unit-testable without spinning up real installs. +/// +internal readonly record struct MsbuildCandidate(Version Version, bool IsDotNetSdk); + +internal static class MsbuildRestore +{ + /// + /// vswhere ships in the fixed VS Installer location on every Windows + /// machine; probing it anywhere else just multiplies failure modes. + /// + private const string VsWherePath = @"C:\Program Files (x86)\Microsoft Visual Studio\Installer\vswhere.exe"; + + // NETSDK1045: "The current .NET SDK does not support targeting .NET X.0". + private const string SdkDoesNotSupportTargeting = "does not support targeting"; + // Current hosts: "A compatible .NET SDK was not found" (global.json + // unsatisfiable); legacy hosts worded it "A compatible installed .NET + // SDK for global.json at ... was not found." Match both — neither + // phrasing contains the other. + private const string SdkGlobalJsonNotFoundModern = "compatible .NET SDK"; + private const string SdkGlobalJsonNotFoundLegacy = "compatible installed .NET SDK"; + + /// vswhere probe bound — generous for a local EXE, still bounded. + private const int VsWhereWaitMs = 10_000; + + /// + /// True when a dotnet restore failed on SDK resolution — the + /// CLI-visible SDK set cannot target the repo's TFM (NETSDK1045) or + /// cannot satisfy global.json — rather than on packages. These are the + /// failures a Visual Studio install can still recover: VS ships its own + /// MSBuild + SDK targets, blind spots of C:\Program Files\dotnet + /// included (2026-10-15 incident: a hollow 10.0.101 SDK dir left the + /// CLI unable to target net10 while VS built fine). Feed/auth failures + /// (NU11xx) deliberately do NOT match — VS restores through the same + /// NuGet stack and would fail identically, so retrying those buys a + /// second slow failure. + /// + internal static bool IsSdkResolutionFailure(string restoreOutput) => + restoreOutput.Contains(SdkDoesNotSupportTargeting, StringComparison.OrdinalIgnoreCase) + || restoreOutput.Contains(SdkGlobalJsonNotFoundModern, StringComparison.OrdinalIgnoreCase) + || restoreOutput.Contains(SdkGlobalJsonNotFoundLegacy, StringComparison.OrdinalIgnoreCase); + + /// + /// Full path of the newest VS MSBuild via vswhere, or null when vswhere, + /// VS, or the MSBuild component is absent (non-Windows included). + /// Best effort by design: a probe failure must never break the restore + /// path it is meant to improve. -prerelease includes Insiders + /// channels; -find (vswhere ≥ 2.8) returns MSBuild.exe paths + /// directly, newest instance only via -latest. + /// + internal static string? LocateVsMsbuild() + { + if (!OperatingSystem.IsWindows() || !File.Exists(VsWherePath)) + { + return null; + } + try + { + var psi = new System.Diagnostics.ProcessStartInfo + { + FileName = VsWherePath, + Arguments = "-latest -prerelease -products * -requires Microsoft.Component.MSBuild " + + "-find MSBuild\\**\\Bin\\MSBuild.exe", + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + }; + using var process = System.Diagnostics.Process.Start(psi); + if (process is null) + { + return null; + } + var stdout = process.StandardOutput.ReadToEnd(); + process.WaitForExit(VsWhereWaitMs); + var path = ParseFirstVswherePath(stdout); + return path is not null && File.Exists(path) ? path : null; + } + catch + { + return null; + } + } + + /// First non-blank line of vswhere -find output. + internal static string? ParseFirstVswherePath(string vswhereOutput) + { + foreach (var raw in vswhereOutput.Split('\n')) + { + var line = raw.Trim().TrimEnd('\r'); + if (line.Length > 0) + { + return line; + } + } + return null; + } + + /// + /// The highest .NET major the solution's projects provably target: + /// max over every .csproj's first TargetFramework(s) raw-text + /// match (raw text — this runs BEFORE any MSBuild evaluation, which is + /// the whole point) and over global.json's pinned SDK version when one + /// sits at or above the solution dir. null when nothing parseable was + /// found — callers must then keep the default instance choice, because + /// an unknown floor proves nothing. + /// + internal static int? RequiredSdkMajorFloor(string? solutionPath) + { + if (solutionPath is null || !File.Exists(solutionPath)) + { + return null; + } + var solutionDir = Path.GetDirectoryName(Path.GetFullPath(solutionPath)) ?? "."; + int? floor = null; + foreach (var projectPath in EnumerateSolutionProjects(solutionPath, solutionDir)) + { + var major = TargetFrameworkMajor(ReadTextBestEffort(projectPath)); + if (major is not null && (floor is null || major > floor)) + { + floor = major; + } + } + var globalJsonMajor = GlobalJsonSdkMajor(solutionDir); + if (globalJsonMajor is not null && (floor is null || globalJsonMajor > floor)) + { + floor = globalJsonMajor; + } + return floor; + } + + /// + /// Which install to register MSBuild from: the highest .NET SDK instance + /// that can target (matching the + /// "register the highest available MSBuild instance" intent, unlike the + /// historical first-match), falling back to the highest VS instance when + /// no SDK instance qualifies — VS bundles its own SDK targets, so it can + /// evaluate TFMs the installed CLI SDKs cannot. The VS path is the + /// documented net472-BuildHost-crash risk, accepted ONLY when the SDK + /// path is certain to fail the load anyway. Returns the index of the + /// chosen candidate within and whether it + /// is an SDK instance (for logging the flip explicitly). + /// + internal static (int Index, bool IsSdk) ChooseMsBuildInstance( + IReadOnlyList candidates, + int? requiredMajor) + { + var sdkCandidates = candidates + .Select((c, i) => (Candidate: c, Index: i)) + .Where(e => e.Candidate.IsDotNetSdk) + .OrderByDescending(e => e.Candidate.Version) + .ToList(); + if (sdkCandidates.Count > 0) + { + var qualifying = requiredMajor is null + ? sdkCandidates[0] + : sdkCandidates.FirstOrDefault(e => e.Candidate.Version.Major >= requiredMajor); + if (!qualifying.Equals(default)) + { + return (qualifying.Index, true); + } + } + // No SDK instance can target the repo (or none exists): highest + // instance overall — a VS install when present, else whatever the + // SDK set offered (preserves the old behaviour as the last resort). + var overall = candidates + .Select((c, i) => (Candidate: c, Index: i)) + .OrderByDescending(e => e.Candidate.Version) + .First(); + return (overall.Index, overall.Candidate.IsDotNetSdk); + } + + private static IEnumerable EnumerateSolutionProjects(string solutionPath, string solutionDir) + { + string[] lines; + try + { + lines = File.ReadAllLines(solutionPath); + } + catch + { + yield break; + } + // Project("{GUID}") = "Name", "Relative\Path.csproj", "{GUID}" + foreach (var line in lines) + { + var trimmed = line.Trim(); + if (!trimmed.StartsWith("Project(", StringComparison.Ordinal)) + { + continue; + } + var parts = trimmed.Split(','); + if (parts.Length < 2) + { + continue; + } + var pathPart = parts[1].Trim(); + if (pathPart.Length < 2 || !pathPart.StartsWith('"') || !pathPart.EndsWith('"')) + { + continue; + } + var relative = pathPart[1..^1]; + if (relative.EndsWith(".csproj", StringComparison.OrdinalIgnoreCase)) + { + yield return Path.GetFullPath( + relative.Replace('\\', Path.DirectorySeparatorChar), Path.GetFullPath(solutionDir)); + } + } + } + + private static string? ReadTextBestEffort(string path) + { + try + { + return File.Exists(path) ? File.ReadAllText(path) : null; + } + catch + { + return null; + } + } + + /// + /// The HIGHEST netX.Y major in raw project text — a floor wants the max, + /// so a net8.0;net10.0 multi-target floors at 10 (unlike the EOL + /// skip-guard, whose first-TFM stance serves a different purpose). + /// netstandard/netcoreapp deliberately do not match — they load under + /// any modern SDK, and EOL ones are skipped anyway. + /// + private static int? TargetFrameworkMajor(string? projectText) + { + if (projectText is null) + { + return null; + } + var match = System.Text.RegularExpressions.Regex.Match( + projectText, @"([^<]+) floor)) + { + floor = major; + } + } + return floor; + } + + /// + /// Major of the SDK version pinned by the nearest global.json walking + /// up from the solution dir — NuGet's own lookup order. rollForward + /// never lowers the floor, so the pinned major is the safe bound. + /// + private static int? GlobalJsonSdkMajor(string solutionDir) + { + var dir = Path.GetFullPath(solutionDir); + while (true) + { + var globalJson = Path.Combine(dir, "global.json"); + if (File.Exists(globalJson)) + { + var match = System.Text.RegularExpressions.Regex.Match( + ReadTextBestEffort(globalJson) ?? "", @"""version""\s*:\s*""(\d+)\."); + return match.Success && int.TryParse(match.Groups[1].Value, out var major) ? major : null; + } + var parent = Directory.GetParent(dir)?.FullName; + if (string.IsNullOrEmpty(parent) || string.Equals(parent, dir, StringComparison.OrdinalIgnoreCase)) + { + return null; + } + dir = parent; + } + } +} diff --git a/helpers/csharp/Program.cs b/helpers/csharp/Program.cs index 846d97da..b1fce4fd 100644 --- a/helpers/csharp/Program.cs +++ b/helpers/csharp/Program.cs @@ -86,7 +86,7 @@ private static async Task RunServeAsync(string[] args) var parsed = ParseServeArgs(args); if (parsed is null) return 1; - if (!TryRegisterMsBuild(out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } + if (!TryRegisterMsBuild(parsed, out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } // A resident workspace must not come up mid-restore: the host CACHES // its find-refs answers, so degraded answers from a workspace loaded @@ -167,7 +167,7 @@ private static async Task RunIndexAsync(string[] args) var parsed = ParseIndexArgs(args); if (parsed is null) return 1; - if (!TryRegisterMsBuild(out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } + if (!TryRegisterMsBuild(parsed.Value.SolutionPath, out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } using var workspace = CreateTolerantWorkspace(); @@ -238,7 +238,7 @@ private static async Task RunFindRefsAsync(string[] args) var parsed = ParseFindRefsArgs(args); if (parsed is null) return 1; - if (!TryRegisterMsBuild(out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } + if (!TryRegisterMsBuild(parsed.Value.SolutionPath, out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } using var workspace = CreateTolerantWorkspace(); @@ -299,7 +299,7 @@ private static async Task RunBatchFindRefsAsync(string[] args) var parsed = ParseBatchFindRefsArgs(args); if (parsed is null) return 1; - if (!TryRegisterMsBuild(out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } + if (!TryRegisterMsBuild(parsed.Value.SolutionPath, out var regErr)) { await Console.Error.WriteLineAsync(regErr).ConfigureAwait(false); return 1; } using var workspace = CreateTolerantWorkspace(); @@ -534,8 +534,51 @@ private static List RestoreSolution( { return []; } - var stderr = stderrDrain.GetAwaiter().GetResult(); - var nuDetail = RestorePolicy.ExtractRestoreErrors(stderr); + // NETSDK1045 and friends print to STDOUT on current dotnet hosts + // (NU11xx detail may land on either stream) — classify over both. + var restoreOutput = stderrDrain.GetAwaiter().GetResult() + + Environment.NewLine + stdoutDrain.GetAwaiter().GetResult(); + var nuDetail = RestorePolicy.ExtractRestoreErrors(restoreOutput); + if (MsbuildRestore.IsSdkResolutionFailure(restoreOutput)) + { + // The CLI SDK set cannot even target this solution — packages + // were never the problem, so the per-project fallback (same + // dotnet, same SDK wall) can only re-fail per project. A + // Visual Studio install ships its own MSBuild + SDK targets + // and can restore the full solution in one go. + var vsMsbuild = MsbuildRestore.LocateVsMsbuild(); + if (vsMsbuild is null) + { + Console.Error.WriteLine( + "[INFO] no VS MSBuild found via vswhere — the CLI SDK gap cannot be worked around."); + } + else + { + Console.Error.WriteLine( + "[INFO] the CLI .NET SDK set cannot target this solution — retrying the restore " + + $"with VS MSBuild at '{vsMsbuild}'."); + switch (TryRestoreSolutionWithVsMsbuild( + vsMsbuild, solutionPath, timeoutSeconds, gateRestorePending)) + { + case VsRestoreOutcome.Restored: + Console.Error.WriteLine( + "[INFO] VS MSBuild restore succeeded — full solution assets restored."); + return []; + case VsRestoreOutcome.StillRunningUngated: + // The VS restore keeps writing obj/ in the + // background; per-project dotnet restores would + // race those writes, so degrade with an honest + // per-project reason instead. + return loadable.Select(e => new ProjectRestoreFailure( + Path.GetFileNameWithoutExtension(e.RelativePath), + e.RelativePath, + "not attempted — a VS MSBuild restore is still running")).ToList(); + } + // Failed → the per-project fallback below still runs: in a + // mixed-TFM solution it recovers the projects the CLI CAN + // target. + } + } var fallback = RestorePerProjectFallback( solutionPath, solutionDir: Path.GetDirectoryName(solutionPath) ?? ".", loadable, timeoutSeconds, gateRestorePending); @@ -579,6 +622,83 @@ private static List RestoreSolution( } } + /// Outcome of the one-shot VS-MSBuild solution restore. + private enum VsRestoreOutcome + { + Restored, + Failed, + /// Timed out ungated: the restore keeps running, serve degrades. + StillRunningUngated, + } + + /// + /// One bounded attempt to restore the solution with a VS MSBuild instead + /// of the CLI dotnet — the fallback for SDK-resolution failures, where + /// the machine's Visual Studio can do what its CLI SDK set cannot. Same + /// contract as the dotnet restore above: never killed on timeout (the + /// marker gates later runs), gated commands exit + /// instead of building a degraded + /// index. -nr:false because the helper is a short-lived subprocess + /// on a hot serve loop and must not leave MSBuild worker nodes behind. + /// + private static VsRestoreOutcome TryRestoreSolutionWithVsMsbuild( + string msbuildPath, + string solutionPath, + int timeoutSeconds, + bool gateRestorePending) + { + try + { + var psi = new ProcessStartInfo + { + FileName = msbuildPath, + Arguments = $"\"{solutionPath}\" -t:restore -nologo -v:q -nr:false", + RedirectStandardError = true, + RedirectStandardOutput = true, + UseShellExecute = false, + }; + using var process = Process.Start(psi); + if (process is null) + { + return VsRestoreOutcome.Failed; + } + _ = process.StandardOutput.ReadToEndAsync(); + var stderrDrain = process.StandardError.ReadToEndAsync(); + if (!process.WaitForExit(timeoutSeconds * 1000)) + { + RestorePendingMarker.Write( + solutionPath, process.Id, process.StartTime.ToUniversalTime()); + if (gateRestorePending) + { + Console.Error.WriteLine( + "[RESTORE-PENDING] the VS MSBuild restore is still running — exiting " + + $"{ExitCodes.RestorePending} without building an index; retry when it finishes."); + throw new RestorePendingException(); + } + Console.Error.WriteLine(RestorePolicy.StillRunningWarning(timeoutSeconds, "VS MSBuild restore")); + return VsRestoreOutcome.StillRunningUngated; + } + if (process.ExitCode == 0) + { + return VsRestoreOutcome.Restored; + } + Console.Error.WriteLine( + "[INFO] VS MSBuild restore failed" + + RestorePolicy.ExtractRestoreErrors(stderrDrain.GetAwaiter().GetResult())); + return VsRestoreOutcome.Failed; + } + catch (RestorePendingException) + { + throw; + } + catch (Exception ex) + { + Console.Error.WriteLine( + $"[INFO] VS MSBuild restore could not start ({ex.GetType().Name}: {ex.Message})."); + return VsRestoreOutcome.Failed; + } + } + /// /// The per-project fallback after a failed solution restore: NuGet is /// all-or-nothing per restore, so one dead package fails every project — @@ -1015,10 +1135,19 @@ private static string BuildFilteredSolution(string[] lines, HashSet skipLin // ── MSBuild registration ───────────────────────────────────────── /// - /// Register the highest available MSBuild instance. - /// Returns true on success; sets on failure. + /// Register the MSBuild instance the workspace will evaluate through. + /// Capability-aware since the 2026-10-15 net10 incident: when every + /// CLI-visible .NET SDK is older than the repo's target frameworks + /// (raw TargetFramework scan + global.json floor via + /// ), an SDK instance + /// would fail every project with NETSDK1045 during evaluation — in that + /// case the highest VS instance wins instead, because VS bundles its own + /// SDK targets and can evaluate TFMs the CLI cannot. Returns true on + /// success; sets on failure. /// - private static bool TryRegisterMsBuild([System.Diagnostics.CodeAnalysis.NotNullWhen(false)] out string? errorMessage) + private static bool TryRegisterMsBuild( + string? solutionPath, + [System.Diagnostics.CodeAnalysis.NotNullWhen(false)] out string? errorMessage) { // Guard against double-registration in case the function is ever called twice // in the same process lifetime (e.g. future test harnesses, daemon mode). @@ -1045,10 +1174,22 @@ private static bool TryRegisterMsBuild([System.Diagnostics.CodeAnalysis.NotNullW // Prefer .NET SDK instances over Visual Studio instances. // The VS BuildHost (net472) crashes with TypeInitializationException - // when loading solutions that contain certain project types. - var sdkInstance = instances - .FirstOrDefault(i => i.DiscoveryType == DiscoveryType.DotNetSdk); - var instance = sdkInstance ?? instances.OrderByDescending(i => i.Version).First(); + // when loading solutions that contain certain project types — so the + // VS instance is only chosen when no SDK instance can even target + // the repo (certain NETSDK1045 load failure either way). + var requiredMajor = MsbuildRestore.RequiredSdkMajorFloor(solutionPath); + var candidates = instances + .Select(i => new MsbuildCandidate(i.Version, i.DiscoveryType == DiscoveryType.DotNetSdk)) + .ToList(); + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor); + var instance = instances[index]; + if (!isSdk) + { + Console.Error.WriteLine( + $"[INFO] no CLI .NET SDK instance can target net{requiredMajor} projects of " + + $"'{solutionPath}' — registering the VS MSBuild '{instance.Name}' v{instance.Version} " + + "instead (VS-bundled SDK targets)."); + } Console.Error.WriteLine( $"MSBuild: registering '{instance.Name}' v{instance.Version} at {instance.MSBuildPath}"); diff --git a/helpers/csharp/RestorePolicy.cs b/helpers/csharp/RestorePolicy.cs index 7eeb9fd9..6b638818 100644 --- a/helpers/csharp/RestorePolicy.cs +++ b/helpers/csharp/RestorePolicy.cs @@ -233,18 +233,19 @@ public static int RestoreTimeoutSeconds() } /// - /// The line emitted when the bounded wait expires while - /// dotnet restore is still running — serve mode only: gated - /// commands exit instead. The - /// restore is NOT killed: it keeps running in the background (the child - /// outlives the helper), serve keeps answering from the existing - /// workspace, and any later index/find-refs run is gated by the restore - /// marker until the fresh assets land. INFO, never WARN: restore state - /// must not drive the caller-facing warning icon. Pinned by test so a - /// "killed" wording cannot sneak back in. + /// The line emitted when the bounded wait expires while a restore is + /// still running — serve mode only: gated commands exit + /// instead. The restore is NOT + /// killed: it keeps running in the background (the child outlives this + /// helper), serve keeps answering from the existing workspace, and any + /// later index/find-refs run is gated by the restore marker until the + /// fresh assets land. INFO, never WARN: restore state must not drive the + /// caller-facing warning icon. Pinned by test so a "killed" wording + /// cannot sneak back in. names the restore + /// flavor still running (dotnet restore, or the VS MSBuild fallback). /// - public static string StillRunningWarning(int seconds) => - $"[INFO] dotnet restore still running after {seconds}s — serve keeps answering from the " + + public static string StillRunningWarning(int seconds, string tool = "dotnet restore") => + $"[INFO] {tool} still running after {seconds}s — serve keeps answering from the " + "existing workspace (degraded symbols) while restore continues in the background; " + "index and find-refs runs are gated until it finishes and the next reindex picks up " + $"the fresh assets. Raise with {TimeoutEnvName} if this solution legitimately needs longer."; diff --git a/helpers/csharp/tests/ScipCsharp.Tests.csproj b/helpers/csharp/tests/ScipCsharp.Tests.csproj index 49ac02bc..bbd732b5 100644 --- a/helpers/csharp/tests/ScipCsharp.Tests.csproj +++ b/helpers/csharp/tests/ScipCsharp.Tests.csproj @@ -47,6 +47,7 @@ + From 2ed8bf9e7a22bf518b4f96dd97968fbedf813fd0 Mon Sep 17 00:00:00 2001 From: flupkede Date: Thu, 15 Oct 2026 21:24:52 +0200 Subject: [PATCH 3/6] test: unit tests for the VS-MSBuild restore fallback logic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 17 tests over the fallback's pure logic: SDK-resolution detection pinned against the live host wordings (NETSDK1045 on stdout, current and legacy global.json phrasings; NU11xx feed/auth failures must NOT trigger), vswhere -find parsing, the capability-aware instance choice (qualifying SDK beats newer VS, floor-miss falls to VS, no-VS keeps SDK as last resort, highest sibling wins), and the repo TFM floor scan (multi-target max, project/global.json composition, parent-dir walk-up, null paths). The multi-target pin was defect-probed: first-TFM-only fails the floor assert as predicted. Also fixes the vswhere probe bound (round-1 review note): async drain before the bounded WaitForExit, and a KILL on timeout — vswhere is a local metadata probe, not a restore, so its death costs one failed probe; the kill also closes the pipes so the stdout read cannot block. Review-fixes: - [Important] WaitForExit timeout result was discarded and the stdout GetResult still blocked until process exit, so VsWhereWaitMs never actually bounded a hung vswhere -> kill(entireProcessTree) on timeout and return null. --- helpers/csharp/MsbuildRestore.cs | 22 +- helpers/csharp/tests/MsbuildRestoreTests.cs | 219 ++++++++++++++++++++ 2 files changed, 238 insertions(+), 3 deletions(-) create mode 100644 helpers/csharp/tests/MsbuildRestoreTests.cs diff --git a/helpers/csharp/MsbuildRestore.cs b/helpers/csharp/MsbuildRestore.cs index 869b85e0..d64097f2 100644 --- a/helpers/csharp/MsbuildRestore.cs +++ b/helpers/csharp/MsbuildRestore.cs @@ -76,9 +76,25 @@ internal static bool IsSdkResolutionFailure(string restoreOutput) => { return null; } - var stdout = process.StandardOutput.ReadToEnd(); - process.WaitForExit(VsWhereWaitMs); - var path = ParseFirstVswherePath(stdout); + // Drain async, THEN bounded-wait — and KILL on timeout: unlike a + // restore (never killed — its work must survive), vswhere is a + // local metadata probe whose death costs one failed probe. The + // kill also closes the pipes, so the stdout read cannot block. + var stdoutTask = process.StandardOutput.ReadToEndAsync(); + _ = process.StandardError.ReadToEndAsync(); + if (!process.WaitForExit(VsWhereWaitMs)) + { + try + { + process.Kill(entireProcessTree: true); + } + catch + { + // Already exiting — nothing to bound anymore. + } + return null; + } + var path = ParseFirstVswherePath(stdoutTask.GetAwaiter().GetResult()); return path is not null && File.Exists(path) ? path : null; } catch diff --git a/helpers/csharp/tests/MsbuildRestoreTests.cs b/helpers/csharp/tests/MsbuildRestoreTests.cs new file mode 100644 index 00000000..80ed5ff7 --- /dev/null +++ b/helpers/csharp/tests/MsbuildRestoreTests.cs @@ -0,0 +1,219 @@ +using ScipCsharp; +using Xunit; + +namespace ScipCsharp.Tests; + +/// +/// Unit tests for the VS-MSBuild restore fallback's pure logic: SDK- +/// resolution failure detection (the feature's trigger — pinned against +/// the exact host wordings seen live on 2026-10-15, incl. NETSDK1045 +/// printing to stdout), vswhere -find output parsing, the capability-aware +/// MSBuild instance choice, and the repo TFM floor scan. +/// +public class MsbuildRestoreTests : IDisposable +{ + private readonly string _root; + + public MsbuildRestoreTests() + { + _root = Path.Combine(Path.GetTempPath(), "cs-msbuild-restore-" + Guid.NewGuid().ToString("N")); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + Directory.Delete(_root, recursive: true); + } + + // ── IsSdkResolutionFailure ──────────────────────────────────────── + + [Fact] + public void Netsdk1045StdoutWordingTriggers() + { + // Verbatim shape of the live net99.0 probe output (stdout), the + // 2026-10-15 incident's failure mode. + var stdout = + "C:\\Program Files\\dotnet\\sdk\\8.0.425\\Sdks\\Microsoft.NET.Sdk\\targets\\" + + "Microsoft.NET.TargetFrameworkInference.targets(166,5): error NETSDK1045: " + + "The current .NET SDK does not support targeting .NET 10.0. Either target " + + ".NET 8.0 or lower, or use a version of the .NET SDK that supports .NET 10.0."; + + Assert.True(MsbuildRestore.IsSdkResolutionFailure(stdout)); + } + + [Theory] + [InlineData("A compatible .NET SDK was not found. Install or update the .NET SDK.")] // current host + [InlineData("A compatible installed .NET SDK for global.json at [C:\\repo\\global.json] was not found.")] // legacy host + public void GlobalJsonUnsatisfiableWordingsTriggerBothHostGenerations(string line) + { + Assert.True(MsbuildRestore.IsSdkResolutionFailure(line)); + } + + [Theory] + [InlineData("error NU1101: Unable to find package Dlw.Net.Standard. No packages exist with this id in source(s): platina-dlwr.")] + [InlineData("Dlw.Aprimo.Dam.csproj: error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com.")] + [InlineData("")] + public void FeedAndAuthFailuresDoNotTrigger(string line) + { + // VS restores through the same NuGet stack — retrying those buys a + // second slow failure, so they must fall through to per-project. + Assert.False(MsbuildRestore.IsSdkResolutionFailure(line)); + } + + // ── ParseFirstVswherePath ───────────────────────────────────────── + + [Fact] + public void VswhereParsingSkipsBlankLines() + { + var output = "\r\nC:\\Program Files\\Microsoft Visual Studio\\18\\Professional\\MSBuild\\Current\\Bin\\MSBuild.exe\r\n\r\n"; + + Assert.Equal( + "C:\\Program Files\\Microsoft Visual Studio\\18\\Professional\\MSBuild\\Current\\Bin\\MSBuild.exe", + MsbuildRestore.ParseFirstVswherePath(output)); + } + + [Fact] + public void VswhereParsingEmptyOutputYieldsNull() + { + Assert.Null(MsbuildRestore.ParseFirstVswherePath("")); + } + + // ── ChooseMsBuildInstance ───────────────────────────────────────── + + private static MsbuildCandidate Sdk(int major) => new(new Version(major, 0), IsDotNetSdk: true); + + private static MsbuildCandidate Vs(int major) => new(new Version(major, 0), IsDotNetSdk: false); + + [Fact] + public void NoFloorKeepsHighestSdk() + { + var candidates = new[] { Sdk(8), Sdk(10), Vs(18) }; + + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor: null); + + Assert.Equal((1, true), (index, isSdk)); + } + + [Fact] + public void QualifyingSdkBeatsNewerVs() + { + var candidates = new[] { Sdk(10), Vs(18) }; + + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor: 10); + + Assert.Equal((0, true), (index, isSdk)); + } + + [Fact] + public void SdkOlderThanFloorFallsToVs() + { + // The incident layout: CLI SDK 8 only, repo targets net10, VS 18 present. + var candidates = new[] { Sdk(8), Vs(18) }; + + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor: 10); + + Assert.Equal((1, false), (index, isSdk)); + } + + [Fact] + public void SdkOlderThanFloorWithoutVsKeepsSdkAsLastResort() + { + var candidates = new[] { Sdk(8) }; + + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor: 10); + + Assert.Equal((0, true), (index, isSdk)); + } + + [Fact] + public void HighestQualifyingSdkWinsAmongSdkSiblings() + { + var candidates = new[] { Sdk(8), Sdk(10), Sdk(11) }; + + var (index, isSdk) = MsbuildRestore.ChooseMsBuildInstance(candidates, requiredMajor: 10); + + Assert.Equal((2, true), (index, isSdk)); + } + + // ── RequiredSdkMajorFloor ───────────────────────────────────────── + + private string WriteSolution(params (string RelativePath, string TargetFrameworks)[] projects) + { + var solutionLines = new List { "Microsoft Visual Studio Solution File, Format Version 12.00", "Global" }; + foreach (var (relative, tfms) in projects) + { + var projectPath = Path.Combine(_root, relative.Replace('\\', Path.DirectorySeparatorChar)); + Directory.CreateDirectory(Path.GetDirectoryName(projectPath)!); + File.WriteAllText( + projectPath, + $"\n \n {tfms}\n \n\n"); + solutionLines.Insert( + 1, + $"Project(\"{{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}}\") = \"P\", \"{relative}\", \"{{GUID}}\""); + } + var solutionPath = Path.Combine(_root, "Repo.sln"); + File.WriteAllLines(solutionPath, solutionLines); + return solutionPath; + } + + [Fact] + public void MultiTargetListFloorsAtHighestTfm() + { + // net8.0;net10.0 must floor at 10 — first-TFM (the pre-fix stance) + // would floor at 8 and register an SDK 8 instance that cannot + // evaluate the net10 half (the exact incident class). The pinned + // 8.x SDK below must NOT lower it. + var solutionPath = WriteSolution(("src\\A\\A.csproj", "net8.0;net10.0")); + WriteGlobalJson(Path.Combine(_root, "global.json"), "8.0.100"); + + Assert.Equal(10, MsbuildRestore.RequiredSdkMajorFloor(solutionPath)); + } + + [Fact] + public void FloorTakesMaxAcrossProjectsAndGlobalJson() + { + var solutionPath = WriteSolution( + ("src\\A\\A.csproj", "net8.0"), + ("src\\B\\B.csproj", "net10.0")); + WriteGlobalJson(Path.Combine(_root, "global.json"), "10.0.303"); + + Assert.Equal(10, MsbuildRestore.RequiredSdkMajorFloor(solutionPath)); + } + + [Fact] + public void GlobalJsonAboveSolutionDirRaisesTheFloor() + { + // NuGet's own lookup order: the sln and its projects sit in a + // subdir, global.json at the parent — the walk-up must find the + // pinned 11 over the projects' net8. + var solutionPath = WriteSolution(("src\\A\\A.csproj", "net8.0")); + var repoDir = Path.Combine(_root, "repo"); + Directory.CreateDirectory(repoDir); + Directory.Move(Path.Combine(_root, "src"), Path.Combine(repoDir, "src")); + File.Move(solutionPath, Path.Combine(repoDir, "Repo.sln")); + WriteGlobalJson(Path.Combine(_root, "global.json"), "11.0.100"); + + Assert.Equal(11, MsbuildRestore.RequiredSdkMajorFloor(Path.Combine(repoDir, "Repo.sln"))); + } + + [Fact] + public void NonNetTargetsAndMissingSolutionYieldNoFloor() + { + // The local global.json pins nothing parseable, so no ancestor + // stray file can perturb the null expectation. + var solutionPath = WriteSolution(("src\\A\\A.csproj", "netstandard2.0")); + WriteGlobalJson(Path.Combine(_root, "global.json"), "{}"); + + Assert.Null(MsbuildRestore.RequiredSdkMajorFloor(solutionPath)); + Assert.Null(MsbuildRestore.RequiredSdkMajorFloor(null)); + Assert.Null(MsbuildRestore.RequiredSdkMajorFloor(Path.Combine(_root, "absent.sln"))); + } + + private static void WriteGlobalJson(string path, string version) + { + var content = version.StartsWith('{') + ? version + : $"{{ \"sdk\": {{ \"version\": \"{version}\" }} }}"; + File.WriteAllText(path, content); + } +} From 06c165067437d56184f9353569735851bd3228f6 Mon Sep 17 00:00:00 2001 From: flupkede Date: Thu, 15 Oct 2026 21:44:40 +0200 Subject: [PATCH 4/6] docs: changelog + highlights for the VS-MSBuild restore fallback CHANGELOG entry under the pending 1.5.31 heading, highlights line in AGENTS.develop.md, and the branch registered in the active-branches table. Plan stage 3 ticked in the branch-local AGENTS.md. Validation: dotnet build 0w/0e; dotnet test 99/99; cargo fmt --check clean; cargo clippy --all-targets -D warnings clean; cargo test --lib --bins all green (Rust side untouched by this branch). --- AGENTS.develop.md | 2 ++ AGENTS.md | 4 ++-- CHANGELOG.md | 2 ++ 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/AGENTS.develop.md b/AGENTS.develop.md index 3ba2eddf..8bffb861 100644 --- a/AGENTS.develop.md +++ b/AGENTS.develop.md @@ -236,11 +236,13 @@ Never `unwrap_or_default()` a store error on a search path — "no results" and | Branch | Description | |---|---| +| `fix/scip-restore-vs-msbuild-fallback` | scip-csharp: VS-MSBuild restore fallback + capability-aware MSBuild registration when the CLI SDK set cannot target the repo TFM | --- ## Changelog highlights (recent) +- **v1.5.31 (pending)** — scip-csharp VS-MSBuild restore fallback: NETSDK1045-class CLI SDK gaps (hollow SDK dir, net10-only-in-VS) retry via vswhere + `MSBuild -t:restore`, and MSBuild registration picks a VS instance when no CLI SDK can target the repo - **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again - **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error - **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings diff --git a/AGENTS.md b/AGENTS.md index fa8607f9..679114ca 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -263,5 +263,5 @@ Older entries: see `CHANGELOG.md`. VS-MSBuild restore fallback in scip-csharp: when `dotnet restore` fails with SDK-resolution errors (NETSDK1045-class) because the CLI-visible SDK set cannot target the repo TFM, locate the newest VS MSBuild via vswhere and retry the solution restore once with `MSBuild -t:restore`. On success the full solution has assets; on failure/timeout existing per-project fallback (or pending-marker) applies. Motivated by HUSQ worktree incident 2026-10-15: VS-bundled net10 SDK invisible to CLI dotnet. 1. [x] Stage 1: detection + fallback in helpers/csharp (MsbuildRestore.cs + wiring in Program.cs). Deliberate behavior change: SDK-instance pick is now the HIGHEST qualifying SDK (was first-enumerated, an undocumented order) — matches the original "register the highest available MSBuild" intent; single-SDK machines unaffected. -2. [ ] Stage 2: unit tests (SDK-failure detection, vswhere output parsing) -3. [ ] Stage 3: CHANGELOG + AGENTS.develop.md highlights + validation +2. [x] Stage 2: unit tests (SDK-failure detection, vswhere output parsing) +3. [x] Stage 3: CHANGELOG + AGENTS.develop.md highlights + validation diff --git a/CHANGELOG.md b/CHANGELOG.md index b6f950d3..724cab8f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,8 @@ finalized in place with a date — no renaming/migration step needed. ### Changed +- **scip-csharp retries the solution restore with the VS MSBuild when the CLI SDK set cannot target the repo.** When `dotnet restore` fails on SDK resolution (NETSDK1045 — CLI SDKs older than the repo's TFM — or an unsatisfiable `global.json`; detection reads stdout AND stderr, because NETSDK1045 prints to stdout on current hosts), the helper locates the newest VS MSBuild via vswhere and retries the restore once with `MSBuild -t:restore` — VS ships its own MSBuild + SDK targets independent of `C:\Program Files\dotnet`, so it restores solutions the CLI cannot even evaluate. Timeout keeps the existing pending-marker/gated-exit-75 contract (a hung VS restore is never killed); on VS failure the per-project fallback still runs. The workspace-load MSBuild registration is now capability-aware too: when every CLI SDK instance is older than the repo's raw-text TFM floor (max over all `TargetFrameworks` + the nearest `global.json` major), the highest VS instance is registered instead — otherwise an SDK 8 instance would evaluate net10 projects into certain NETSDK1045 load failures even with fresh assets. On healthy machines the choice is unchanged (single-SDK machines byte-identical; multi-SDK machines now deterministically take the highest qualifying SDK, was first-enumerated). Motivated by the 2026-10-15 field incident: a hollow `10.0.101` SDK dir left the CLI blind to net10 on a machine where VS built fine, degrading every net10 repo's symbol index to "0 projects loaded". + - **The serve TUI status column no longer shows `Indexing` for a pure symbol (SCIP) rebuild.** Only text-indexing flows (startup warmup, file-watcher batches, explicit reindex) light the general repo label; a C#/TypeScript symbol rebuild — including a long restore-pending retry loop — drives only the language columns, so a busy C# pass can no longer read as "the normal index is stuck behind the SCIP index". A queued phase-2 rebuild stays visible in the C# column only. The `/status` wire vocabulary is unchanged. A repo whose index is on disk but which is not loaded renders `Idle` again — the interim `idx` rendering read as active indexing on done, non-SCIP repos. ## [1.5.30] From ea441ba988e2b2eef96d14d2027aff65823f9491 Mon Sep 17 00:00:00 2001 From: flupkede Date: Thu, 15 Oct 2026 21:54:04 +0200 Subject: [PATCH 5/6] fix: net4x target spellings no longer raise the TFM floor The floor regex ^net(\d+)(?:\.|$) matched the extension-less .NET Framework spellings (net48 -> 48, net472 -> 472), so a solution with one net4x project floored at 48: no CLI SDK qualifies, and healthy SDK+VS machines flipped to the VS MSBuild instance for no reason (exposing the documented net472-BuildHost crash risk where the SDK path loaded those projects fine). The minor dot is now required (^net(\d+)\.) - every net-core TFM has one, no net4x spelling does. Pinned by NetFrameworkSpellingsDoNotRaiseTheFloor (net48 alone -> null, net48;net10.0 -> 10); defect-probed: the old regex fails the mixed case with a 48 floor. Final-review fix (round 1 finding). Round-2 re-review executed by the worker itself (reviewer subagent hit its 5h usage limit); scope was the fix diff only - verified: regex at HEAD, diff touches solely MsbuildRestore.cs + tests, 100/100 green, round-1 met-rows untouched. --- helpers/csharp/MsbuildRestore.cs | 8 ++++++-- helpers/csharp/tests/MsbuildRestoreTests.cs | 16 ++++++++++++++++ 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/helpers/csharp/MsbuildRestore.cs b/helpers/csharp/MsbuildRestore.cs index d64097f2..b011207d 100644 --- a/helpers/csharp/MsbuildRestore.cs +++ b/helpers/csharp/MsbuildRestore.cs @@ -246,7 +246,11 @@ private static IEnumerable EnumerateSolutionProjects(string solutionPath /// so a net8.0;net10.0 multi-target floors at 10 (unlike the EOL /// skip-guard, whose first-TFM stance serves a different purpose). /// netstandard/netcoreapp deliberately do not match — they load under - /// any modern SDK, and EOL ones are skipped anyway. + /// any modern SDK, and EOL ones are skipped anyway. The minor dot is + /// REQUIRED: the extension-less .NET Framework spellings (net48, + /// net472) must not raise the floor — an SDK loads those from reference + /// assemblies, and a "48" floor would flip healthy SDK+VS machines to + /// the VS instance for no reason. /// private static int? TargetFrameworkMajor(string? projectText) { @@ -263,7 +267,7 @@ private static IEnumerable EnumerateSolutionProjects(string solutionPath int? floor = null; foreach (var raw in match.Groups[1].Value.Split(';')) { - var version = System.Text.RegularExpressions.Regex.Match(raw.Trim(), @"^net(\d+)(?:\.|$)"); + var version = System.Text.RegularExpressions.Regex.Match(raw.Trim(), @"^net(\d+)\."); if (version.Success && int.TryParse(version.Groups[1].Value, out var major) && (floor is null || major > floor)) { diff --git a/helpers/csharp/tests/MsbuildRestoreTests.cs b/helpers/csharp/tests/MsbuildRestoreTests.cs index 80ed5ff7..aa7a4afa 100644 --- a/helpers/csharp/tests/MsbuildRestoreTests.cs +++ b/helpers/csharp/tests/MsbuildRestoreTests.cs @@ -209,6 +209,22 @@ public void NonNetTargetsAndMissingSolutionYieldNoFloor() Assert.Null(MsbuildRestore.RequiredSdkMajorFloor(Path.Combine(_root, "absent.sln"))); } + [Fact] + public void NetFrameworkSpellingsDoNotRaiseTheFloor() + { + // net48/net472 (extension-less) must not become a "48" floor: an + // SDK loads them from reference assemblies, and a 48 floor would + // flip healthy SDK+VS machines to the VS instance for no reason. + // The pre-fix regex (^net(\d+)(?:\.|$)) matched net48 as major 48. + var net48Only = WriteSolution(("src\\Legacy\\Legacy.csproj", "net48")); + + Assert.Null(MsbuildRestore.RequiredSdkMajorFloor(net48Only)); + + var mixed = WriteSolution(("src\\Mixed\\Mixed.csproj", "net48;net10.0")); + + Assert.Equal(10, MsbuildRestore.RequiredSdkMajorFloor(mixed)); + } + private static void WriteGlobalJson(string path, string version) { var content = version.StartsWith('{') From 2e9243bf06c1b1ca1db3b3517c313543fb903a65 Mon Sep 17 00:00:00 2001 From: flupkede Date: Fri, 16 Oct 2026 07:39:18 +0200 Subject: [PATCH 6/6] chore: drop branch-local agent files --- AGENTS.develop.md | 1 - AGENTS.md | 267 -------------------- CLAUDE.md | 1 - helpers/csharp/tests/MsbuildRestoreTests.cs | 4 +- 4 files changed, 2 insertions(+), 271 deletions(-) delete mode 100644 AGENTS.md delete mode 100644 CLAUDE.md diff --git a/AGENTS.develop.md b/AGENTS.develop.md index 8bffb861..b68a6d40 100644 --- a/AGENTS.develop.md +++ b/AGENTS.develop.md @@ -236,7 +236,6 @@ Never `unwrap_or_default()` a store error on a search path — "no results" and | Branch | Description | |---|---| -| `fix/scip-restore-vs-msbuild-fallback` | scip-csharp: VS-MSBuild restore fallback + capability-aware MSBuild registration when the CLI SDK set cannot target the repo TFM | --- diff --git a/AGENTS.md b/AGENTS.md deleted file mode 100644 index 679114ca..00000000 --- a/AGENTS.md +++ /dev/null @@ -1,267 +0,0 @@ -# AGENTS.develop.md - -This file is the **develop-branch reference** for all coding agents. -It is committed on `develop` and merged from every feature branch. - -`AGENTS.md` and `CLAUDE.md` are **branch-local working files**: committed on the -feature branch while building, dropped before the PR merges — they never appear -on `develop` or `master`. See *For agents starting a new feature branch* below. - -## Setup (once per machine) - -```bash -git config core.hooksPath .githooks -``` - -This enables the hooks: `post-checkout` auto-creates `AGENTS.md` (from this -file) and the `CLAUDE.md` pointer on feature branches, `pre-commit` runs -`cargo fmt` + the root-md allowlist guard (branch-aware, see below), `pre-push` -blocks direct pushes to `master`. See `.githooks/README.md`. - -## For agents starting a new feature branch - -If `AGENTS.md` was not auto-created by the hook, create it manually: - -```bash -cp AGENTS.develop.md AGENTS.md -``` - -Then append your work plan as a `## Plan` section at the end of `AGENTS.md`. -Leave the architecture sections intact — they provide context. - -**Commit `AGENTS.md`/`CLAUDE.md` on the feature branch as you work** — the plan -travels through the PR and survives branch switches. - -**At the end of the branch, before the PR (the changelog moment):** -1. Update `## Active feature branches` below (remove this branch) -2. Add one line to `## Changelog highlights` -3. Add the CHANGELOG.md entry under the pending-version heading (functionality changes only — pure tooling/docs churn gets the `no-changelog` PR label instead; workflow changes like this one never go in the changelog) -4. Drop the branch-local files and commit: - `git rm AGENTS.md CLAUDE.md` → `chore: drop branch-local agent files` - -Enforced mechanically (forgetting step 4 cannot leak the files): -- `pre-commit` rejects introducing root `AGENTS.md`/`CLAUDE.md` on - `develop`/`master` (direct commits and conflict resolutions). -- `pre-merge-commit` runs the same guard on the merge result — a clean - merge never invokes pre-commit. -- The `agent-files-check` CI workflow flags any PR into `develop`/`master` - whose head still carries them. - -**No active work plan lives here.** Feature branches carry their own `AGENTS.md`. -This file contains only architecture, conventions, and changelog. - ---- - -## ⚠️ Branching & PR workflow (READ FIRST) - -This repo uses a **`develop`-based** gitflow. The GitHub default branch is `master` (`origin/HEAD → origin/master`), but `master` is **NOT** the integration branch. - -- **Integration branch = `develop`.** All feature/fix/release branches merge into `develop`. -- **ALL PRs target `develop`** — pass `--base develop` to `gh pr create`. NEVER target `master` (releases only, cut at release time). -- **Merge style:** feature/fix → `develop` = merge commits (`--merge`); `develop` → `master` release PR = **squash**, with `--body "$(scripts/release-coauthors.sh)"` so contributors stay credited on the default branch. -- **Review requirement** is enforced by a repo ruleset. As repo owner, override with `gh pr merge --merge --admin --delete-branch`. -- Before creating a PR, **verify the base**: `gh pr view --json baseRefName`. If it says `master`, retarget: `gh pr edit --base develop`. -- **Release squashes regress the merge-base** → a release PR may fail with "cannot be cleanly created" even when content is fine. Fix: cut `release/vX.Y.Z` off develop, run `git merge -s ours origin/master` in it, verify the diff vs master is the intended release delta, PR that into master. Never merge master into develop directly. - -Common mistake: a subagent creates a PR with no explicit `--base`, the tooling picks `master` (GitHub default), and the PR lands against the wrong branch. Always specify `--base develop`. - ---- - -## What codesearch is - -A fast, local, offline MCP server for semantic code search. Single Rust binary. -No Docker, no cloud, no external services. Designed for coding agents (OpenCode, -Claude Code, Claude Desktop) that need to search and navigate large codebases efficiently. - -Core stack: Tantivy (BM25 FTS) + arroy (HNSW vectors) + fastembed/ONNX (embeddings) + -tree-sitter (AST chunking) + LMDB (persistent storage) + rmcp 1.5.0 (MCP protocol). -Hybrid BM25 + vector search fused via RRF. - -## MCP tools - -Five tools exposed to agents: - -| Tool | Description | -|---|---| -| `search` | Hybrid semantic + BM25 search. Requires `project` or `group` in multi-repo mode. | -| `find` | Symbol navigation: definition, usages, imports, dependents. Requires scope. | -| `explore` | File outline or similar-chunk lookup. Requires scope. | -| `get_chunk` | Retrieve a chunk by ID with optional context lines. Requires `project` in multi-repo mode. | -| `status` | Index and project status. Lightweight (no DB open) when called without scope. | - -All tools return `scope_required` structured errors in multi-repo mode when no `project` -or `group` is specified, with `available_projects`, `available_groups`, and `hint_for_agent`. - -## Multi-repo serve mode - -`codesearch serve` starts an MCP HTTP server on `127.0.0.1:{port}` (default 39725). -Multiple repos register via `repos.json` with aliases. Agents route queries per-alias -or per-group. - -**Repo lifecycle:** -- `Warm` — DB open, vector index ready, no FSW. State after background warmup or fan-out open. -- `Write` — Warm + file system watcher running. Transitions from Warm on first explicit project query. -- `Readonly` — Another process holds the write lock. -- `Idle` (status string was `Closed`) — Evicted by idle reaper after `REPO_IDLE_TIMEOUT_SECS` (30 min default) of inactivity; index stays on disk, stores auto-reopen on the next query. - -**Idle reaper** runs every `REAPER_INTERVAL_SECS` (5 min). Evicts repos not queried within timeout. -All opens update `last_access` so the reaper can track every open. - -**Fan-out rule:** `get_chunk` and group queries use `touch=false` → repos open as Warm only, -no FSW spawned. Only explicit `project=` queries use `touch=true` → Warm→Write transition. - -## TUI - -`codesearch serve` with a TTY starts an embedded ratatui TUI (repo table, status, CPU). -Without TTY: headless, logs only. - -- `codesearch serve --no-tui` — suppress TUI even with TTY -- `codesearch serve tui [--url http://...]` — standalone TUI via HTTP polling of `GET /status` - -## HTTP endpoints (serve mode) - -| Endpoint | Method | Description | -|---|---|---| -| `/health` | GET | Health check JSON | -| `/status` | GET | Lightweight repo state snapshot for TUI polling | -| `/repos` | POST | Register + index + warmup a new repo | -| `/repos/:alias` | DELETE | Stop FSW, evict, unregister, delete DB | -| `/repos/:alias/reindex` | POST | Incremental or force reindex (background) | -| `/mcp` | GET/POST | MCP streamable HTTP endpoint | - -## Supported languages - -17 tree-sitter grammars (table in README.md). `find_impact` has SCIP symbol -precision for C# (bundled `scip-csharp`, resident helper via its `serve` -subcommand) and TypeScript (`npx scip-typescript`). Protobuf is text-aware -chunking only. - -## Release artifacts - -GitHub Actions produces release binaries per tag (plain + `-with-csharp` -variants): `codesearch-{windows-x86_64.zip,linux-x86_64.tar.gz,macos-arm64.tar.gz}`. -Single-file native binaries, no runtime dependencies. macOS build is manual-trigger -only (expensive runners). - ---- - -## Current state - -- **Version:** `Major.Minor.Patch` (semver). Patch auto-bumps +1 on every PR merged to `develop` (`.github/workflows/bump-develop.yml`); minor bumps manually at release (`scripts/bump-version.sh --type minor`, resets patch→0). Per-commit uniqueness: `build.rs` `+`. See `RELEASING.md`. -- **Validation:** `cargo check` for iteration → `cargo clippy --all-targets -- -D warnings` → `cargo test --lib --bins` before a branch is done. No `--release` builds during the fix loop. -- **Deploy:** `..\copy-to-common.ps1` builds + copies both binaries to `~/.local/bin/`. Stop serve first — a running `codesearch.exe` is file-locked on Windows. - -## Implemented features (orientation only — narratives live in CHANGELOG.md) - -- **Federation** — `codesearch remote add/rm/list/mounts`; `@peer` group fan-out; mounted projects addressable as `project=/`; TUI mount management with live status. Remote write verbs (`add`, `reindex --force`) need a read-write peer; `rm` is not durable across a cold start. -- **Cloud indexer-job split** — heavy build job uploads a snapshot, light serve restores it. DOCS repos are `repo_read_only` in `repos.json`; only `custom-kb` stays writable with a bounded incremental reindex per KB `git pull`. -- **`find_impact`** — SCIP-precise transitive call-sites (C#, TypeScript), ambiguity envelope, partial-results warnings, freshness fields. On `busy: true` sleep `retry_after_seconds` and retry the SAME call (busy is progress, never a reason to fall back to text search); `index_head_sha` vs `current_head_sha` drift is surfaced, never auto-reindexed. -- **REST mirrors** — read-only HTTP mirrors of the MCP tools incl. `/find-impact`. - -## ⚠️ Design constraint — never poll a federated peer - -**A federated peer is NEVER contacted on a timer.** Each poll wakes the peer's -scale-to-zero replica, which then self-warms its full idle window (measured ~50% -duty cycle from zero searches). Shipped twice (PR #181/#184), reverted twice — -do not re-attempt. Local repos may be polled in the background; peers are -contacted only by a real federated tool call (activity poke) or the explicit -TUI `i` overlay. The TUI discovery tick is config-only (zero HTTP). - -## Open TODOs - -- **Low severity:** literal-mode snippets for markdown chunks sometimes show the chunk's opening line instead of the matched line (the `match_info.unwrap_or_else` fallback), making true hits look like false negatives. - ---- - -## Key conventions for agents - -- **Branch from develop**, never from master. Feature branches: `features/` (or `fix/`, `chore/`). -- **Cargo.toml version** on develop may be one version ahead of the deployed binary — that is expected due to `copy-to-common.ps1` deploy hook. Never flag as inconsistency. -- **Always fix bugs you encounter — including pre-existing ones** the current branch did not introduce. Clean code and tests are the bar: a fix lands with a test that fails without it (reintroduce the defect, watch it fail — the proof rule under *Search errors*). -- **Never write separate `AGENTS_xxx.md` sibling files** unless explicitly requested. OpenCode reads `AGENTS.md` only. Out-of-repo planning goes to `C:\WorkArea\AI\codesearch\instructions\`. -- **Root file hygiene (markdown)**: root markdown is allowlisted — `AGENTS.develop.md`, `README.md`, `README_CSharp.md`, `CHANGELOG.md`, `RELEASING.md` always; `AGENTS.md` and `CLAUDE.md` on **feature branches only** (branch-local, dropped at finalization — the pre-commit guard rejects them on `develop`/`master`). Anything else (diagnoses, plans, test scenarios, worklogs) goes into `.docs/` (gitignored, local-only). Enforced by the `pre-commit` root-md allowlist guard and the `agent-files-check` CI workflow. -- **Path normalization**: all path comparisons must go through a single normalize utility. Windows UNC prefixes (`\\?\C:\`), backslash/forward-slash mismatches, and worktree `.git` file resolution have each caused subtle bugs in the past. -- **ONNX arena allocator**: uses `kNextPowerOfTwo` growth, never returns memory to OS. ~2GB memory during indexing is a known limitation. No local fix available until upstream fastembed exposes `OrtArenaCfg`. -- **Git worktrees**: `find_git_root` returns the worktree directory itself. Each worktree is a separate indexable repo. Groups can be used to search across worktrees of the same base repo. - -## Runtime locations - -- **Runtime dir**: `C:\Users\develterf\.local\bin\` — contains `codesearch.exe` and `helpers/csharp/scip-csharp.exe`. This is where `codesearch serve` runs from. -- **Build:** via `build.ps1` (repo root) — it self-heals the bare-flag quirk and sets `CARGO_TARGET_DIR` so `target/` stays outside the repo. Debug-only (issue #192): `-Release` is refused, release artifacts come from CI; rust-lld is pinned in `.cargo/config.toml` (bin rebuild ~2x faster). -- **Build dir**: `target/release/` — lives **outside the repo** (set via `CARGO_TARGET_DIR`). For compilation only. Never run codesearch from this location. -- **Logs**: `~\.codesearch\logs\` — codesearch writes structured logs here during serve. Check these for startup errors, rebuild failures, and helper detection messages. - -## Deploying to runtime - -- `..\copy-to-common.ps1` — builds and copies **both** `codesearch.exe` and `scip-csharp.exe` to `~/.local/bin/` (the common execution dir). Use this to update the runtime binaries. **No `--release` builds — always dev/debug.** -- The C# helper is built via: `dotnet publish helpers/csharp/scip-csharp.csproj -r win-x64 --self-contained -c Release` -- Helper output must be **single-file only**: `scip-csharp.exe` (+ optional `.pdb`). The `.csproj` has `PublishSingleFile=true`. -- Do NOT copy framework DLLs, `BuildHost-*` dirs, or `.dll.config` files to the runtime location. - -## Notes for agents - -- **Before direct git work in this checkout, force `git config core.bare false`.** This repo is a bare+working-tree hybrid (`codesearch.git` holds a full checkout) and VS Code's git integration intermittently resets `core.bare` to `true` on ref changes, breaking every git operation outside `build.ps1`/`copy-to-common.ps1` (both self-heal it before building). Symptom: "fatal: this operation must be run in a work tree" from an otherwise intact checkout. -- **Never use the bundled `codesearch` binary to investigate this repo** (it is the project under development). Use codesearch MCP tools first for discovery (this repo is indexed as `codesearch-git`); `grep`/`Read` for exact refs, other git refs, or when MCP returns nothing. -- **Tests live in sibling `_tests.rs` files**, table-driven preferred over near-duplicate per-case fns. -- **Tests that set env vars must be `#[serial]`** and set them via `crate::testing::EnvRestore` — cargo runs tests as parallel threads of one process, so an unserialised `set_var` races every reader. -- **Never call `.canonicalize()`** — use `safe_canonicalize()`. -- **Windows transient rename errors** (os error 5/32/33 from AV/Search-Indexer races): classify with `is_transient_rename_error()` / `ServeState::is_db_locked_error` and wrap in a bounded retry. Never retry non-transient errors. -- **Counter-then-teardown races:** a background task tearing down state guarded by an in-flight counter must take the write lock BEFORE checking the counter and hold it across check + clear. Consumers increment the counter before acquiring the resource, so `counter == 0` under the write lock proves no consumer exists. - -### LMDB rules - -- **One `EnvOpenOptions::open()` per directory per process.** All access via `get_or_open_stores()` → `Arc`; SCIP opens share a per-directory env (`get_or_open_shared_env`). -- **Open every env with `BASE_ENV_FLAGS`** (`src/lmdb_registry.rs`) — heed refuses to reopen one path with different options. -- **Commit, never drop, a txn whose DB handle you keep** — an aborted txn's DBI is closed by LMDB; using it later yields a bare `EINVAL`. -- **A dropped `TrackedEnv` must close via `prepare_for_closing()`** — heed's `OPENED_ENV` cache keeps a clone, so a plain drop never runs `mdb_env_close` and Windows keeps the files locked for the process lifetime (the `index rm` os-error-32 bug). - -### Search errors must not become empty results - -Never `unwrap_or_default()` a store error on a search path — "no results" and "store down" must stay distinguishable. This defect was re-introduced across sibling handlers in seven review rounds; it is a class, not a site: - -- Render error chains with `{:#}`, never `{}`. Pass "not found" claims through `qualify_empty_result()`; never state a diagnosis you did not verify. -- The rule covers EVERY MCP handler: `find`, `get_chunk`, `explore`, `find_imports`, `find_dependents`, and the single-store `project=` paths. -- **Warnings channels must terminate** on every path that writes to them, and the read must be reachable from the last write. -- **Every `for store in stores` fan-out opens its `*_warnings` channel before the loop.** `Err(_)` over a store result is banned: bind it, render `{e:#}`, carry it. `MultiReadOutcome` is `#[must_use]`. -- **Take the channel as a parameter** — use `respond_with_items()` / `respond_with_object()`, which cannot be called without the channel. -- **New response shapes use the shared exits, not hand-rolled ones.** `serde_json::json!` renders `None` as an explicit `null`, so conditional keys must be *inserted*, not set. -- **Suppress `suggested_tool` when warnings are present.** -- **Verify a batch edit by re-running its detector over the whole file**, including the lines the edit added. -- **Before claiming a test pins a fix, reintroduce the defect and watch it fail.** -- **A caller-facing literal wrapped across lines needs a `\` continuation** — enforced by `tests/caller_facing_literals.rs`. - ---- - -## Active feature branches (not yet merged) - -| Branch | Description | -|---|---| - ---- - -## Changelog highlights (recent) - -- **v1.5.31 (pending)** — serve TUI status: `Indexing` only for real text indexing (pure SCIP rebuilds drive the C#/TS columns only); resting repos read `Idle` again -- **v1.5.16 (pending)** — C# restore-pending retries 20×60s → 3 attempts with exponential backoff; restore exhaustion renders as yellow C#⚠ partial on every serve path instead of a red error -- **v1.5.6 (pending)** — scip-csharp demotes MSBuild check advisories + BuildHost shutdown chatter from index warnings -- **v1.5.15 (pending)** — CLI add/reindex refuse to build locally when the serve reports the alias already indexing (409 two-writer case); other delegation failures keep the local fallback -- **v1.5.14 (pending)** — db dir marked hidden on Windows: the Search indexer no longer crawls `.codesearch.db` (structural fix for the scanner race on fresh Tantivy segments) -- **v1.5.13 (pending)** — symbol-index status (ready/stale/degraded/absent) on REST info, both TUIs and every find_impact answer; vanishing-repo fix: transient FTS commit race gets one full-pass retry and failed first builds keep the repo registered -- **v1.5.5** — serve TUI: symbol-index warnings wrap instead of clipping and `c` copies them to the clipboard -- **v1.5.4** — restore gate: CPM-aware restore freshness (Directory.Packages.props etc.), restore-pending exit 75 with host-side bounded retry, previous index stays live during restore, watcher fires full C# rebuild on restore-input changes -- **v1.4.4** — resident C# workspace pool no longer serves stale `find_impact` results after a rebuild: `WorkspacePool::evict` bumps a per-solution generation counter closing a spawn-in-flight race, and `scip_ref_cache` is now cleared unconditionally on both full and incremental rebuilds -- **v1.3.37** — per-index embedding models end-to-end: serve queries, `POST /repos` and CLI index/stats/status honour the model each index records in its `metadata.json`; `serve --model` sets the default for newly created indexes; unrecorded indexes are queried with the built-in model plus a caller-facing warning; mid-rebuild indexes no longer report ready (PR #248) -- **v1.3.23–v1.3.36** — dependency + platform wave: rmcp 3.3, fastembed 6.1 + ort rc.13, tantivy 0.26, axum 0.8, ratatui 0.30 + crossterm 0.29, thiserror 2, notify 8, tree-sitter 0.27, dirs/sha2/scip/sysinfo refresh + dependabot (weekly); clears the open Aikido/RUSTSEC advisories -- **v1.3.19** — `find_impact` ambiguity envelope + `resolved_symbol`; partial-results `warnings`; C# symbol-key uniqueness (index v2.0) -- **v1.3.16** — REST `/find-impact` endpoint -- **v1.3.3** — federation hardening release - -Older entries: see `CHANGELOG.md`. - -## Plan - -VS-MSBuild restore fallback in scip-csharp: when `dotnet restore` fails with SDK-resolution errors (NETSDK1045-class) because the CLI-visible SDK set cannot target the repo TFM, locate the newest VS MSBuild via vswhere and retry the solution restore once with `MSBuild -t:restore`. On success the full solution has assets; on failure/timeout existing per-project fallback (or pending-marker) applies. Motivated by HUSQ worktree incident 2026-10-15: VS-bundled net10 SDK invisible to CLI dotnet. - -1. [x] Stage 1: detection + fallback in helpers/csharp (MsbuildRestore.cs + wiring in Program.cs). Deliberate behavior change: SDK-instance pick is now the HIGHEST qualifying SDK (was first-enumerated, an undocumented order) — matches the original "register the highest available MSBuild" intent; single-SDK machines unaffected. -2. [x] Stage 2: unit tests (SDK-failure detection, vswhere output parsing) -3. [x] Stage 3: CHANGELOG + AGENTS.develop.md highlights + validation diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 5d302681..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1 +0,0 @@ -Read AGENTS.md. diff --git a/helpers/csharp/tests/MsbuildRestoreTests.cs b/helpers/csharp/tests/MsbuildRestoreTests.cs index aa7a4afa..fb695a64 100644 --- a/helpers/csharp/tests/MsbuildRestoreTests.cs +++ b/helpers/csharp/tests/MsbuildRestoreTests.cs @@ -50,8 +50,8 @@ public void GlobalJsonUnsatisfiableWordingsTriggerBothHostGenerations(string lin } [Theory] - [InlineData("error NU1101: Unable to find package Dlw.Net.Standard. No packages exist with this id in source(s): platina-dlwr.")] - [InlineData("Dlw.Aprimo.Dam.csproj: error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com.")] + [InlineData("error NU1101: Unable to find package Vendor.Internal.Base. No packages exist with this id in source(s): private-feed.")] + [InlineData("App.Core.csproj: error NU1301: Unable to load the service index for source https://pkgs.dev.azure.com/org/_packaging/private-feed/nuget/v3/index.json.")] [InlineData("")] public void FeedAndAuthFailuresDoNotTrigger(string line) {