Repository navigation

feat: CJK bigram tokenizer (ports @hulinming's registry from #2) - #4

Merged

IvenKooLab merged 2 commits into

mainfrom

feat/bm25-cjk-bigram

Oct 26, 2026

Merged

feat: CJK bigram tokenizer (ports @hulinming's registry from #2)#4
IvenKooLab merged 2 commits into
mainfrom
feat/bm25-cjk-bigram

Commits on Oct 26, 2026
- feat: pluggable BM25 tokenizers with jieba Chinese word segmentation
  
  hulinming
  authored and
  IvenKooLab
  committed
  355de30
- feat: CJK bigram tokenizer, so Chinese phrase search stops trading precision for recall
  
  IvenKooLab
  and
  hulinming
  committed
  5808ebb