Skip to content

fix(qdrant): add explicit BM25 sparse upgrade - #452

Merged
kristinhe1111 merged 5 commits into
mainfrom
fix/qdrant-bm25-sparse-upgrade
Nov 15, 2026
Merged

kristinhe1111 merged 5 commits into
mainfrom
fix/qdrant-bm25-sparse-upgrade

Conversation

@kristinhe1111

@kristinhe1111 kristinhe1111 commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

  • add read-only plan_qdrant_bm25_upgrade() and explicit
    upgrade_qdrant_bm25(writers_stopped=True) SDK APIs
  • use Qdrant Server's native named sparse-vector schema API on Server 1.18+
  • keep indexing and scoring in Qdrant: FastEmbed produces the sparse vectors,
    and query_points(using="bm25") performs native sparse retrieval
  • backfill only the derived bm25 vector in bounded batches
  • preserve point IDs, dense vectors and user payload fields; skip L1_RAW
  • mark empty or zero-token documents with private Qdrant payload state
    _hy_memory_bm25_state=empty, without searchable sentinels or sidecar state
  • require the IDF modifier used by FastEmbed Qdrant/bm25
  • keep keyword recall disabled when schema or point coverage is incomplete
  • report old servers, missing dependencies, empty text, encoding failures and
    partial remote failures as structured states
  • prepare Hy Memory 2.0.17 release metadata

Safety

  • maintenance-only; every writer must be stopped
  • no collection deletion
  • no dense-vector or user payload-field rewrite
  • repeated execution is idempotent
  • interrupted/partial backfills remain fail-closed and can be resumed
  • Qdrant Local stays on its owner thread

Verification

  • uv lock --check
  • focused Ruff check: passed
  • 29 focused Qdrant/BM25 tests passed locally; 2 real-server tests are
    environment-gated and run in CI against Qdrant Server 1.18
  • real local Qdrant dense-only collection upgraded in place
  • real local Qdrant non-IDF sparse schema upgraded in place with the owner
    thread preserved
  • real Qdrant Server 1.18 CI covers dense-only upgrade, partial failure/resume,
    non-IDF delete/recreate, empty-document coverage and native keyword retrieval
  • dense vector, payload and point ID preserved
  • post-upgrade native keyword_search returned the unique marker

Not in this PR

  • MemHub Runtime FastEmbed packaging and offline model cache
  • MemHub SDK pin/lock update
  • MemHub maintenance CLI
  • quarantine handling
  • Chroma, FAISS or PostgreSQL retrieval/packaging changes
  • Qdrant Server older than 1.18 (the maintenance API refuses the upgrade)

Official Qdrant contract

Qdrant 1.18+ supports adding named sparse vectors to an existing collection;
existing points must then be populated:
https://qdrant.tech/documentation/manage-data/collections/#update-vector-schema

kristinhe1111 and others added 5 commits November 15, 2026 13:22
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@kristinhe1111

Copy link
Copy Markdown
Collaborator Author

Validation evidence for final head 104dd35d875281fb97ec35e9ccea921955b97d3a:

  • Local: uv lock --check; focused Ruff; uv run pytest -q tests/test_qdrant_bm25_upgrade.py tests/test_bm25_chinese.py → 31 passed, 2 real-server tests skipped because Docker is unavailable locally.
  • SDK CI: uv run pytest tests/ -q -o log_cli=false passed: https://github.com/Tencent-Hunyuan/Hy-Memory/actions/runs/43686204317/job/132728120431
    • tests/test_qdrant_bm25_upgrade.py starts qdrant/qdrant:v1.18.0 under CI and covers dense-only migration, non-IDF delete/create, partial failure/resume, native keyword retrieval, the empty-SparseVector 422 counterexample, and indexed/empty payload coverage invariants.
  • MemHub CI rerun on the same head passed all Node and full Core suites: https://github.com/Tencent-Hunyuan/Hy-Memory/actions/runs/43686204317/job/132728119673
    • The first attempt had one unrelated desktop cleanup race (ENOTEMPTY .../.one-memory/logs); no MemHub files changed.
  • CodeQL, Analyze (actions/python/javascript-typescript), and open-source scan all passed.

Coverage contract: non-raw points are ready only when they are exactly one of (HasVector(bm25) AND state=indexed) or (no bm25 vector AND state=empty). Encoder unavailable/runtime errors write neither state and keep the reader fail-closed.

@kristinhe1111
kristinhe1111 requested a review from txfei November 15, 2026 06:55
@kristinhe1111
kristinhe1111 merged commit 23cb604 into main Nov 15, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant