proposal(serve): bounded candidate expansion for post-filtered search #178

Reproduced candidate-window limitation

At upstream b4fe76301465eaee3d8ac75e0bec54664e53c072, /search with n_docs=2 and articles_only=true requests 20 ranked candidates, then removes meta pages. In a deterministic TestClient regression with precomputed query embeddings and a ranked fake backend, ranks 1–20 are Portal:* pages and ranks 21–22 are eligible articles. The endpoint returns zero hits; the regression expecting the two eligible hits fails.

This demonstrates a candidate-window limitation, not its production incidence or an unconditional guarantee that every request should return n_docs hits. The relevant code is the initial search and post-filtering.

Related: #14 introduced articles_only; #115 introduced department pre-filtering; #122 and #154 bound request sizes; #164 bounds on-demand renders. This proposal does not implement or depend on hybrid search in #97. I checked open/closed issues and PRs and related comments; I did not find an existing bounded-expansion proposal. Discussions are disabled for this repository.

Local prototype — budget proposed for discussion

The implementation and tests are currently local and unpublished. I am seeking agreement on the policy before opening an implementation PR.

• Preserve the initial window: 10 * n_docs with articles_only, 5 * n_docs with height-only filtering, otherwise n_docs.
• Retry only queries that have too few eligible hits and received a full backend candidate window. A short result stops expansion for that search configuration; it does not establish that the entire corpus is exhausted.
• Allow at most two additional batch calls. Each retry requests up to twice the preceding window, sharing the remaining allowance equally across pending queries.
• For B initial requested candidate slots across all queries, allow at most min(3B, 10,000) additional slots across the request, counting repeated prefixes. Stop when the allowance cannot buy a larger window for all pending queries.
• Preserve the final backend window's ranking and deduplicate vector IDs. Reuse the query embeddings and keep department pre-filters on every call. Unfiltered requests and requests satisfied by their initial batch still search once.
• Select final hits before any image hydration. Retain the existing request-wide render budget. FAISS receives request-local nprobe; configured max_codes is preserved, with no shared-setting mutation or probe broadening.

The conservative total bound is B + min(3B, 10,000) requested slots, at most 330,000 under current request limits, and three raw_search batch calls. This does not bound internal ANN scans or wall-clock latency. For one query starting at 20, the prototype requests 40 next and stops: the remaining 20 slots cannot buy a larger window. Batched queries can reach a second retry when others finish and release their share of the remaining allowance.

Validation and measured tradeoff

The original regression failed with [] rather than vector IDs [20, 21]. The prototype's selected API/backend/neighboring suites pass 77 tests, zero skips, including real small FAISS flat/IVF/PQ and in-memory Qdrant fixtures. Coverage includes no matches, late matches, partial results at the cap, ranking, duplicates, combined filters, department pre-filters, batch budgeting, backend errors, settings preservation, and no repeated hydration. Ruff lint/format and diff whitespace checks pass. The full browser/ML suite was not run. No models or paid APIs were used, and production was not touched.

A local benchmark compares the original API at the above commit with the prototype using shared real flat-FAISS/in-memory-Qdrant backends, 80 synthetic vectors, fixed query embeddings, n_docs=2, and no image hydration. Median endpoint latency after 10 warmups over 100 requests on macOS arm64 / Python 3.12.13:

Counts match across both backends; returned candidate counts equal requested slots in this fixture, including repeated prefixes. These measurements show the extra-work tradeoff, including cases with no gain. They are not production latency or semantic-recall claims; small differences on unchanged paths are timing noise. Approximate backends may change candidate membership/order at larger windows, so additional eligible hits are not guaranteed.

Questions before an implementation PR

1. Is a two-retry limit with min(3B, 10,000) extra request-wide slots an acceptable starting budget, or should the limits differ?
2. Is equal allocation among pending queries preferable to spending leftover budget on a subset? The current policy avoids favoring query order but can leave budget unused, and batching can change an individual query's reachable window.
3. Should expansion require an explicit request opt-in, a server-side setting, or apply by default to filtered searches? The local prototype currently applies it automatically, which explicitly increases worst-case filtered-search work.
4. What representative workload and latency measurements would you want before adopting the policy?