Skip to content

feat(retrieval): add Qdrant-native lexical and hybrid recall - #447

Open
Leeminjing wants to merge 9 commits into
Tencent-Hunyuan:mainfrom
Leeminjing:codex/qdrant-native-retrieval
Open

Leeminjing wants to merge 9 commits into
Tencent-Hunyuan:mainfrom
Leeminjing:codex/qdrant-native-retrieval

Conversation

@Leeminjing

@Leeminjing Leeminjing commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Change

Memories containing an error code, identifier, path or Chinese keyword can now enter public search results and the actual write-time reconcile prompt even when they are outside the dense candidate pool.

  • Add an opt-in reader="native" with dense, keyword and hybrid modes, plus the independent recall.reconcile_retriever="native" write-side option.
  • Use Qdrant Server scored TEXT indexes on content and tags[], a KEYWORD index on tags, and Query API nested RRF. Retrieval/scoring/fusion stay in Qdrant; there is no sidecar index or Python-side fusion.
  • Add explicit, idempotent preparation, strict schema/version checks, shared scope/evolution resolution, raw typed score thresholds and fail-closed required retrieval.
  • Preserve existing readers and compatibility writes. Reconcile graph evidence uses real backend cosine rather than RRF.
  • Repair three CI test contracts: Python 3.12 timeout exception conversion, worker-thread lease-close completion, and watcher-first synchronization. Assertions retain diagnostic, closure-order, import, permission and freeze behavior; no MemHub production change or integration is added.
  • Add native tests, dependency locking, SDK CI integration and usage documentation.

Related to #432. This PR retains its recall goal: memories outside the dense candidate pool can be recalled through lexical evidence in public search and write-time reconcile. It replaces the sidecar retrieval approach with Qdrant-native capabilities and supports Qdrant Server only. Qdrant Local, Chroma, FAISS, and MemHub integration are outside this PR's scope. The content-only update / stored-vector preservation fix is outside this PR's scope.

User impact

The new path is opt-in, supports Qdrant Server/client 1.20.0 and requires Python >=3.10. It does not require FastEmbed. Existing default retrieval remains unchanged.

Native retrieval rejects unsupported scenes/options/backends and missing or incompatible indexes without falling back. Public native search preserves raw scores and backend order and does not schedule dedup/SHADOW/access-count writes.

The unified lock also selects client 1.20.0 for the existing qdrant/all extras on Python >=3.10; Python 3.9 retains client 1.16.1. A two-line postponed-annotations compatibility change restores SDK import on real Python 3.9 so native can explicitly report unsupported.

Validation

  • B1/B2 follow-up (b591b40): preserve all legacy ReadRequest positional arguments and timestamp-ordered native history. Before-fix regressions reproduced both reported symptoms. Local checks: 273 passed, 4 Neo4j skips; real Qdrant Server integration 26 passed without FastEmbed; changed modules on Linux/Python 3.12 51 passed; minimal Python 3.9 compatibility 3 passed. Details and commands are in the execution record below.
  • Current-head CI for b591b40 passed: SDK, MemHub and open-source scan checks all succeeded (CI run 43228937929). The detailed earlier CI results below remain attributed to head 6f18457.
  • Acceptance coverage added for T01–T18, including 35 dense distractors, actual reconcile prompts, strict dual-tag indexing, scope/evolution failures, update/delete and preparation recovery.
  • Previous-head Ubuntu/Python 3.12 CI: 67 native tests passed (41 contracts + 26 real Qdrant Server integration), with Server/client 1.20.0 and FastEmbed absent. No native integration skip is counted as a pass.
  • Earlier local focused SDK compatibility/graph/reconcile suite: 221 passed, 4 skipped (unconfigured Neo4j).
  • Real Python 3.9.12 SDK import/native rejection and focused legacy/default/version checks: 7 passed, using minimal test dependencies; this does not certify a complete Python 3.9 extra installation.
  • Affected end-to-end paths: public sync/async search, actual reconcile prompt, real Qdrant storage updates, and writer graph projection into a recording graph sink. Embeddings/LLM are deterministic stubs; Qdrant is real.
  • Mutation: changing the exclusive score comparison from > to >= makes T08 fail; restoring > passes.
  • Retrieval-path Ruff checks, compileall, git diff --check, uv lock --check --offline and locked native sync.
  • Previous-head complete SDK CI suite: 2039 passed, 30 skipped, 71 deselected, 1 warning in 57.25s. The separate Chroma local/real-HTTP regression step also passed 53 tests.
  • Python 3.12 native installation, contracts and real Server integration passed in CI. Earlier local Windows download/disk failures remain historical records, not successful installation claims.
  • MemHub CI passed, including Node checks and the complete Core PR suite: 4799 passed, 19 skipped, 13 deselected, 1 xfailed, 8 warnings in 405.48s (0:06:45). Two existing Core tests received timing-only repairs; no MemHub production implementation changed.
  • MemHub focused Linux/Python 3.12 modules: 13 passed; the delayed lease-close regression passed 20/20 repetitions, and the repaired permission case passed twice. Both original failure conditions were reproduced before repair.
  • Previous-head SDK, MemHub and open-source scan checks all succeeded: CI run 43224711091, head 6f18457.

Historical local Windows SDK full-suite results and isolated upstream comparisons are retained in the execution record. An additional local WSL2 watcher check produced 28 passed/1 failed with auto-polling; selecting the native watcher backend without a diagnostic plugin passed all 29 adjacent tests. That local mode sensitivity was not hidden or changed through a product patch. The minimal Python 3.9 checks do not certify a complete Python 3.9 extra installation.

The full run commands, environment/image digest, acceptance mapping, baseline failure comparison and limitations are in the execution record. The three original handoff materials remain unchanged local untracked files and are not part of this PR.

Privacy and security

  • No screen, AXTree, OCR, keyboard/mouse, clipboard or background-audio collection added.
  • No credentials, private service addresses or real user data included; validation used synthetic inputs and a dedicated disposable test Server.
  • Dependency license recorded: qdrant-client is Apache-2.0. Other transitive dependency versions are unchanged.

Rollback

Select reader="legacy"/"hybrid" and recall.reconcile_retriever="default". Existing search_text/sparse maintenance continues. Rollback does not automatically drop or rebuild native indexes.

This remains one SDK PR against main; upstream a2a1858 was merged to reproduce the CI merge context. It does not reopen #432.

@Leeminjing
Leeminjing marked this pull request as ready for review November 12, 2026 13:55
@Leeminjing

Copy link
Copy Markdown
Contributor Author

To reduce the review burden, I performed a self-review of this PR from first principles. The core problem is candidate coverage: reranking cannot recover a relevant memory that never entered the dense candidate pool. This PR addresses that gap with an opt-in Qdrant-native path that independently retrieves lexical candidates from content and tags, then combines them with dense candidates using backend RRF for both public search and write-time reconciliation. Correctness depends on applying scope filters before each branch’s candidate limit, resolving evolution chains to valid representatives, keeping RRF ranking scores separate from cosine evidence used for graph relationships, and distinguishing required retrieval failures from healthy empty results so failures cannot trigger unintended ADD operations. Validation included 92 passing tests covering contracts and real Qdrant integration, plus an additional experiment with 70 higher-cosine distractors where both lexical targets were absent from dense top60 but still reached the actual reconciliation prompt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant