[ENHANCEMENT] Make codebase indexing reliable, controllable, and easy to diagnose

[WebMad]

This consolidates reported symptoms; their causes and applicability to Zoo-Code have not yet been confirmed.

Problem

Indexing may fail to start, stall, complete only partially, or report success while search remains broken. Users need a predictable process: a correct index, actionable errors, and recovery without restarting the editor or reindexing the entire workspace.

Context

Roo-Code reports cover Windows/macOS/Linux, local and remote workspaces, large projects, multiple languages, Ollama/OpenAI-compatible/Gemini providers, and local/cloud Qdrant deployments.

Reported symptom groups — with references for validation:

• Startup and settings: start/save buttons do not work; settings reset or export incorrectly; users report problems saving keys and selecting custom models — #4919, #5091, #5035, #12366.
• Progress and recovery: stalls at zero or partway through, progress percentages decrease, false success states, reindexing fails after clearing the index, and no convenient resume after failure — #12159, #5819, #11214, #4582, #4875, #5803.
• Compatibility: available Ollama models are rejected; Qdrant URL/port issues, HTTP 400/405 errors, batch/token limits, and HTTP 429 quota errors; vector dimensions mismatch the collection, including after switching models — #10863, #11428, #4092, #5088, #5123, #5713, #10991, #5616.
• Index contents and quality: excluded files enter the index, exclusion rules are lost after clearing it; documentation/.NET code is skipped; Python code is chunked incorrectly, Go snippets are duplicated; search fails even after successful indexing — #11797, #5656, #4660, #5238, #5247, #5367, #5089.
• Stability: UI freezes, out-of-memory failures, extension host crashes; specific reports concern Flutter/Dart and repeated index-update errors — #4188, #5711, #5642, #5516.
• Isolation and coverage: per-project settings are requested; users report keys being shared across profiles. Related limitations include multi-root workspaces, symlinks, file search in large projects, and searching JAR dependencies — #11947, #5442, #8041, #4654, #5721, #5183.

Desired behavior

Indexing runs in the background, respects exclusions and project settings, reports outcomes accurately, explains failures, and allows users to stop/resume processing. Search returns up-to-date, permitted, meaningful snippets.

Constraints / preferences

Keep the editor responsive; bound memory and API usage; do not expose secrets in diagnostics; avoid unnecessary full reindexing. Agree separately on the scope of related coverage and isolation requests.

Request checklist

• Complete the duplicate search: issues have been partially reviewed; Discussions have not been checked.
• Confirm symptoms in Zoo-Code and agree on the enhancement scope.

Zoo Code Task Links

No public task link available.

Acceptance criteria

• Settings persist and are restored; the selected model and dimensions are used consistently.
• Progress distinguishes active processing, partial success, completion, and failure; the failure reason and next step are visible.
• Stopping, restarting, clearing, and resuming after failure work without restarting VS Code.
• Exclusions are respected during initial indexing, updates, clearing/reindexing, and search; deleted/modified files are accurately reflected in the index.
• Provider limits are handled without endless retries; large projects do not freeze the UI or crash the extension host.
• Index completeness, snippet quality, and search functionality are verified for an agreed set of languages.

Proposed approach

First reproduce the reports and assess them against the current Zoo-Code version; then split confirmed problems into subtasks and add regression checks. Use this request as an umbrella tracker.

Trade-offs / risks

The scope is too broad for a single PR; some reports are closed or concern external service configuration. Index format fixes may require rebuilding the index; reducing concurrency improves stability but slows processing. None of the referenced reports has yet been confirmed as a current Zoo-Code bug.

[WebMad]

Existing Zoo-Code issues and overlap

Follow-up duplicate search in Zoo-Code-Org/Zoo-Code: no full duplicate of this umbrella tracker was found, but several parts of its scope are already tracked. The initial references in the issue body came from Roo-Code; the links below are Zoo-Code's own reports.

Direct overlaps — currently open

Issue	Scope covered
#518	Semble search failures are hidden as empty results while the UI reports readiness. Direct overlap with accurate status and actionable diagnostics. The report describes an applied fix, but the issue remains open; merge/release status was not verified.
#549	Independent per-workspace indexing controls, persistence across restarts, and stopping active scans without globally disabling indexing.
#564	Remote OpenAI-compatible embedder initialization times out, leaving search and index clearing unavailable; saving settings can trigger recovery.
#593	OpenAI-compatible embedder validation fails with a llama.cpp base URL but succeeds with the full embeddings endpoint.
#335	Provider batch limits cause partial indexing: 60 items sent where only 32 are allowed.
#340	HTTP 400 responses when writing embeddings to local Qdrant under WSL.
#1370	Initial scanning fails when connecting to Qdrant. The author attributes this to client/server version compatibility; that diagnosis has not been independently verified here.
#751	Missing coverage for Dart, Docker-related files, and shell scripts. Compare with closed #940 before defining remaining work.

Related requests / additional scope — currently open

• [ENHANCEMENT] #534 — Reuse a shared index across devices to avoid repeated embedding costs. Related to avoiding unnecessary reindexing, but a separate feature requiring explicit scope agreement.
• [BUG] Raise Semble download archive size limit #1220 — Increase the Semble download archive size limit. Adds a local-indexer installation failure scenario not covered by the original Roo-Code references.

Closed reports to consider during validation

• [ENHANCEMENT] Add Dart support to codebase indexing #940 — Add Dart support to codebase indexing.
• [BUG] Codebase indexing silently skips .txt files #931 — Codebase indexing silently skips text files.
• [BUG] Codebase indexing Error - OpenAI Compatible Provider: Failed during initial scan: fetch failed #654 — OpenAI-compatible indexing fails during the initial scan with a connection error.

Tracking recommendation

Keep #1592 as an umbrella tracker, not a replacement for these issues. Reuse existing issues for covered work; create new subtasks only for reproduced gaps that are not already tracked. Include Semble explicitly when agreeing on scope, since the original Roo-Code-based inventory did not cover it.

This review covered open and closed issues using indexing, embeddings, Qdrant, Semble, and code-search queries. Discussions and linked PR fixes were not separately reviewed. Reported symptoms and proposed root causes have not been independently reproduced in the current Zoo-Code version, and a closed issue alone does not establish that every related symptom is resolved.