What problem does this solve?
https://github.com/tobi/qmd
This repo is a good reference for hybrid search, we should try to implement BM25 in LEANN and make it to increase accuracy too!
We have #211 before, but have not provide document about how to use and how to combine together with dense search, it only provide a simple API now, also it seems that PR use a self written version of BM25(in python)
Proposed solution
Can we use similiar method like
https://github.com/tobi/qmd?tab=readme-ov-file#search-backends
SQLite FTS5 BM25
to make it more robust
Example usage
search(query, dense-score-ratio=0.6, full-text-score-ration=0.4)
Let's make the first version of the PR and set default ratio ull text search to be 0, and when it is stable we can change that.
Note: fulltextsearch == BM25 == sparse search here
What problem does this solve?
https://github.com/tobi/qmd
This repo is a good reference for hybrid search, we should try to implement BM25 in LEANN and make it to increase accuracy too!
We have #211 before, but have not provide document about how to use and how to combine together with dense search, it only provide a simple API now, also it seems that PR use a self written version of BM25(in python)
Proposed solution
Can we use similiar method like
https://github.com/tobi/qmd?tab=readme-ov-file#search-backends
SQLite FTS5 BM25
to make it more robust
Example usage