Skip to content

fix(serve): adopt an existing index on POST /repos instead of force-rebuilding - #357

Merged
flupkede merged 1 commit into
developfrom
fix/add-repo-adopts-existing-index
Oct 20, 2026
Merged

flupkede merged 1 commit into
developfrom
fix/add-repo-adopts-existing-index

Conversation

@donbowman

Copy link
Copy Markdown
Collaborator

Problem

Registering a repo through a running serve (POST /repos, which is what codesearch index add delegates to) unconditionally ran the destructive force reindex. If the repo already had a valid .codesearch.db — for example one built directly with codesearch index <path> before it was registered — the registration cleared the vector store, FTS index and file metadata, then re-embedded every file.

For a batch of registrations this is expensive and confusing: each add looks like it starts another background job, and the heavy-job gate (CODESEARCH_INDEX_JOBS) serializes the rebuilds, so the visible result is a long period with no committed progress while already-indexed work is thrown away and redone. The local codesearch index add path already reuses an existing index; the serve path did not.

Fix

POST /repos now adopts a usable index already on disk instead of force-rebuilding it:

  • Adopt when metadata.json records a model and the request is not changing it. The background task runs the gated incremental refresh, so unchanged files stay in the index and only real changes are re-chunked/re-embedded.
  • Force-rebuild when there is no recorded model (new index, or a pre-model legacy index) or when an explicit model differs from the recorded one. A model change is the one case that genuinely invalidates every stored vector (dimensions and vector space change).

The adoption decision is a small helper (should_adopt_existing_index) next to the existing resolve_add_repo_model precedence rule.

Tests

  • should_adopt_existing_index_precedence: decision table — no recorded model → build; recorded model with no/matching override → adopt; differing override → build.
  • add_repo_handler_adopts_existing_index_instead_of_rebuilding: end-to-end fixture with a tracked unchanged file and one seeded chunk. The add returns 202 and the chunk/file survive the background refresh; the old force path clears the store first, so the assertion fails without the fix.
  • Existing add_repo_handler_* tests (brand-new repo, serve-default model, recorded model kept) still pass, as does the full QC gate.

Notes

The add task still spawns one background task per registration (that is the "N jobs" in the log), but with adoption each task is an incremental refresh that normally returns "index is up to date" almost immediately rather than a full rebuild.

A separately observed "no committed progress" symptom on an old binary is the pre-v1.6.0 file-metadata persistence issue fixed by 241e641 ("persist file metadata per index batch, atomically"); this change addresses the registration path that discarded existing indexes in the first place.

…ebuilding

Registering a repo through the running serve (codesearch index add, the
git-hook path, or a script adding many repos) unconditionally ran the
destructive force reindex. A repo that already had a valid .codesearch.db -
for example one indexed directly before being registered - lost its index and
had to re-embed every file, serialized behind CODESEARCH_INDEX_JOBS. With a
batch of registrations that looked like no progress at all for a long time.

The local 'codesearch index add' path already reuses an existing index; the
serve path now matches it. Adopt when metadata.json records a model and the
request is not changing it, and run the gated incremental refresh instead of
the clear-and-rebuild. A new index (no recorded model) and an explicit model
change still take the force path, because changing the model invalidates every
stored vector.

Tests: decision table for the adopt rule, plus an end-to-end add that seeds an
unchanged tracked file and asserts its chunk survives (the old force path
clears it).
@donbowman
donbowman requested a review from flupkede as a code owner October 18, 2026 22:16

@flupkede flupkede left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed in depth (see session notes): adoption decision table correct, force path preserved for legacy/no-model/mismatch, tests pin both paths, changelog follows convention. Safe to merge.

@flupkede
flupkede merged commit 4694da6 into develop Oct 20, 2026
7 checks passed
@flupkede
flupkede deleted the fix/add-repo-adopts-existing-index branch October 22, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants