Repository navigation
feat(retrieval): add Qdrant-native lexical and hybrid recall - #447
Leeminjing wants to merge 9 commits into
Conversation
Wait for the worker-thread lease-close completion instead of counting event-loop turns. Verify the watcher import outcome before expecting the next automatic trigger to be a no-op, so a watcher-first execution cannot spuriously fail the permission contract.
|
To reduce the review burden, I performed a self-review of this PR from first principles. The core problem is candidate coverage: reranking cannot recover a relevant memory that never entered the dense candidate pool. This PR addresses that gap with an opt-in Qdrant-native path that independently retrieves lexical candidates from content and tags, then combines them with dense candidates using backend RRF for both public search and write-time reconciliation. Correctness depends on applying scope filters before each branch’s candidate limit, resolving evolution chains to valid representatives, keeping RRF ranking scores separate from cosine evidence used for graph relationships, and distinguishing required retrieval failures from healthy empty results so failures cannot trigger unintended ADD operations. Validation included 92 passing tests covering contracts and real Qdrant integration, plus an additional experiment with 70 higher-cosine distractors where both lexical targets were absent from dense top60 but still reached the actual reconciliation prompt. |
Change
Memories containing an error code, identifier, path or Chinese keyword can now enter public search results and the actual write-time reconcile prompt even when they are outside the dense candidate pool.
reader="native"with dense, keyword and hybrid modes, plus the independentrecall.reconcile_retriever="native"write-side option.contentandtags[], a KEYWORD index ontags, and Query API nested RRF. Retrieval/scoring/fusion stay in Qdrant; there is no sidecar index or Python-side fusion.Related to #432. This PR retains its recall goal: memories outside the dense candidate pool can be recalled through lexical evidence in public search and write-time reconcile. It replaces the sidecar retrieval approach with Qdrant-native capabilities and supports Qdrant Server only. Qdrant Local, Chroma, FAISS, and MemHub integration are outside this PR's scope. The content-only update / stored-vector preservation fix is outside this PR's scope.
User impact
The new path is opt-in, supports Qdrant Server/client 1.20.0 and requires Python >=3.10. It does not require FastEmbed. Existing default retrieval remains unchanged.
Native retrieval rejects unsupported scenes/options/backends and missing or incompatible indexes without falling back. Public native search preserves raw scores and backend order and does not schedule dedup/SHADOW/access-count writes.
The unified lock also selects client 1.20.0 for the existing qdrant/all extras on Python >=3.10; Python 3.9 retains client 1.16.1. A two-line postponed-annotations compatibility change restores SDK import on real Python 3.9 so native can explicitly report unsupported.
Validation
b591b40): preserve all legacyReadRequestpositional arguments and timestamp-ordered native history. Before-fix regressions reproduced both reported symptoms. Local checks: 273 passed, 4 Neo4j skips; real Qdrant Server integration 26 passed without FastEmbed; changed modules on Linux/Python 3.12 51 passed; minimal Python 3.9 compatibility 3 passed. Details and commands are in the execution record below.b591b40passed: SDK, MemHub and open-source scan checks all succeeded (CI run 43228937929). The detailed earlier CI results below remain attributed to head6f18457.>to>=makes T08 fail; restoring>passes.git diff --check,uv lock --check --offlineand locked native sync.6f18457.Historical local Windows SDK full-suite results and isolated upstream comparisons are retained in the execution record. An additional local WSL2 watcher check produced 28 passed/1 failed with auto-polling; selecting the native watcher backend without a diagnostic plugin passed all 29 adjacent tests. That local mode sensitivity was not hidden or changed through a product patch. The minimal Python 3.9 checks do not certify a complete Python 3.9 extra installation.
The full run commands, environment/image digest, acceptance mapping, baseline failure comparison and limitations are in the execution record. The three original handoff materials remain unchanged local untracked files and are not part of this PR.
Privacy and security
Rollback
Select
reader="legacy"/"hybrid"andrecall.reconcile_retriever="default". Existing search_text/sparse maintenance continues. Rollback does not automatically drop or rebuild native indexes.This remains one SDK PR against
main; upstreama2a1858was merged to reproduce the CI merge context. It does not reopen #432.