Repository navigation
fix: FTS commit-failure hardening — no more mixed generations - #352
Merged
Merged
Conversation
The per-file incremental paths (batch loop, refresh Phase 4) now redrive on the transient Windows FTS commit race via retry_on_transient_fts — the same protection force_reindex already had. Backoffs promoted to a shared associated const so the three sites cannot drift. hide_db_dir_on_windows also marks the lazily-created fts/ subdir hidden (it does not inherit the parent's attribute). Root cause context: Windows indexer/AV holds freshly created tantivy segment files -> os error 5 on the commit write -> writer discarded by design (recommit of a half-joined pipeline is unsafe). serve.log shows this firing daily, concentrated on worktrees under source/repos.
…blish failure index_single_file inserted new chunks into the vector store BEFORE the FTS commit with no rollback: when the FTS commit failed (the transient Windows segment-file race), the just-inserted ids never reached file_meta, so no later pass could find and clean them — orphaned vector chunks next to lingering old FTS chunks = mixed generations (search and get_chunk disagreeing, the HUSQ stale-index symptom). The FTS insert + commit + file_meta update now live in publish_file_chunks; on any error the caller best-effort deletes the fresh chunk_ids from both stores and returns Err. file_meta stays old (it still names the previous generation), so the next pass — redrive or refresh — heals cleanly with no orphans.
Phase 3.6 in refresh_index_with_stores: a failed FTS commit used to leave the failed attempt's vector chunks untracked (never in file_meta) while the tracked generation's ids were already deleted from the vector store — search (FTS) and get_chunk (vector) then disagreed (the HUSQ stale-index symptom). The sweep deletes untracked vector ids and queues tracked files whose ids are missing from the vector store for the Phase 4 re-index loop, so existing corrupted indexes heal on the next refresh without a force reindex. FileMetaStore::file_chunk_ids added as the non-destructive per-file lookup (remove_file clears the entry as a side effect).
…he index get_chunk now stats the chunk's source file against file_meta.json's mtime (the timestamp of the last successful indexing write — per-file saves). A file modified after it is pending re-index or its last pass failed, so the indexed lines may not match disk; the response's existing note field says so. Chosen over a ServeState last_indexed_unix field + indexing callbacks: file_meta.json's mtime already IS that timestamp, needs no new state, no callback wiring across the serve-side indexing sites, and works in stdio mode too.
…e stale note - rollback_vector_insert_removes_fresh_ids_from_both_stores: a failed publish leaves no orphans in either store. - refresh_heals_mixed_generations_on_live_files: the Phase 3.6 sweep deletes untracked vector chunks of live files (Phase 3.5's ghost sweep must not claim them) and queues tracked files missing their ids for re-index; without a real embedding model the re-index attempt fails gracefully (pinned), the real heal rides the HUSQ remediation reindex. The sweep moved BEFORE the up-to-date early-exit: a quiet corrupted repo must heal too, not only one with fresh changes. - get_chunk_notes_stale_source_newer_than_last_indexing_write: a source file newer than file_meta.json carries the stale note.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (proven from serve.log + code): on Windows an indexer/AV filter holding a freshly created tantivy segment file makes the FTS commit fail (os error 5) and the writer is discarded BY DESIGN (recommitting a half-joined pipeline is unsafe). The per-file incremental paths inserted the new chunks into the vector store first with no rollback, so every failure left the failed attempt's ids orphaned (never in file_meta) while the tracked generation was already deleted — search (FTS) and get_chunk (vector) disagreed and stale lines kept coming back (the HUSQ.Aprimo-209169-import-progress-table symptom, firing daily in serve.log).
Three layers:
retry_on_transient_fts(backoffs 5s/20s), the same protection force reindex already had; backoffs promoted to a shared associated const.publish_file_chunks; on any error the fresh chunk ids are rolled back out of both stores (rollback_vector_insert). file_meta stays old, so the next pass heals clean with no orphans.Plus: get_chunk warns (existing
notefield) when the source file is newer than file_meta.json's mtime (the last successful indexing write — chosen over ServeState plumbing: zero new state, works in stdio too); the fts/ subdir is marked hidden on Windows like its parent.Ops-side (user action, biggest lever): Defender process exclusions for codesearch.exe + scip-csharp.exe — the affected repos sit under
C:/Users/develterf/source/repos(likely outside the current path exclusions), and hidden ≠ AV-excluded.Tests: rollback removes fresh ids from both stores; the sweep deletes untracked chunks + queues missing-id files (graceful failure pinned without a model); the stale note. Validation: fmt, clippy
--all-targets -D warnings,cargo test --lib --binsAND the fullcargo test --test '*'integration suite — all green.