feat(retrieval): BM25 lexical retriever and dense hybrid fusion with RRF (#233)

[shivamm-gupta]

Description

Fixes #233.

This PR introduces an in-memory Okapi BM25 lexical retriever and a hybrid fusion retriever combining dense ANN vector search (S3 Vectors) with sparse BM25 lexical search using Reciprocal Rank Fusion (RRF).

Key Additions & Features

1. Pure-Python BM25 Index (dynavec.bm25):

   • Zero third-party runtime dependencies (uses Python standard library).
   • Lucene / Robertson-Spärck Jones smoothed IDF prevents negative weights on common terms.
   • Standard Okapi BM25 term frequency saturation with document length normalization ($k_1$, $b$).
   • default_tokenize: Preserves exact compound identifiers (e.g. SKU-892-XZ, XPS-13-9310, ConnectionResetError, 10.0.0.1) both as whole tokens and sub-tokens for maximal recall on both exact codes and partial queries, with configurable stopword filtering.
   • Supports metadata filtering (equality, $in, $nin, $gt, $gte, $lt, $lte, $ne).

2. BM25Retriever (dynavec.retrievers):

   • Bound to a Dynavec client or NamespaceView.
   • Supports search(), async asearch(), index_documents(), and populate_from_store().

3. BM25HybridRetriever (dynavec.retrievers):

   • Executes dense ANN vector search and sparse BM25 lexical search concurrently using thread pool execution.
   • Fuses ranked lists using reciprocal_rank_fusion with configurable weights (dense_weight and sparse_weight).
   • Directly compatible with RRFWeightFitter learned weights or FitResult instances.
   • Provides both sync search() and async asearch().

4. Client & Namespace Ergonomics:

   • Dynavec.as_bm25_retriever()
   • Dynavec.as_hybrid_retriever()
   • Dynavec.hybrid_search()
   • NamespaceView.as_bm25_retriever()
   • NamespaceView.as_hybrid_retriever()
   • NamespaceView.hybrid_search()

5. Exports:

   • BM25Index, BM25Retriever, and BM25HybridRetriever exported in top-level dynavec.

Verification

• Added comprehensive test suite tests/test_bm25_retriever.py (18 unit tests):
  • Tokenization & compound identifier extraction
  • BM25 index CRUD, scoring, clearing, and metadata operator filtering
  • BM25 retriever sync and async execution
  • Dense + BM25 hybrid fusion with lexical boost for exact identifiers
  • RRFWeightFitter / FitResult weight passing and override
  • Dynavec and NamespaceView ergonomics
• Full repository test suite passed (638 tests passed).
• Linting verified via ruff check (0 errors).

[codeforstartups]

Excellent — this is the sparse/BM25 hybrid from the roadmap. A pure-Python Okapi BM25 index (zero deps, Lucene-style IDF + TF saturation) plus BM25Retriever and BM25HybridRetriever that fuses lexical + dense results via RRF. Big quality win for keyword-heavy queries where pure vector search underperforms. Verified: ruff clean, 18 tests pass, CI green incl. typecheck. Merging — thanks @shivamm-gupta! 🙌