[hulinming]

Summary

A batch of enhancements focused on Chinese-language retrieval and usability. All changes are backward compatible — the original dependency-free single-char CJK tokenizer stays the default, and every addition ships as an optional extra.

1. Pluggable BM25 tokenizers with jieba Chinese word segmentation

• src/loci/tokenizers.py: tiny registry (default / jieba); jieba is imported lazily and falls back to default with a one-time install hint (fail-open).
• Config: [bm25] tokenizer = "default" | "jieba"; unknown names warn and fall back to default.
• Optional extra: pip install 'loci-rag[zh]' — no new runtime dependency.
• Effect: the default tokenizer splits Chinese into single characters, so phrase-level BM25 recall is weak (A/B on a real index: a distractor doc outranked the correct one 0.416 vs 0.325); with jieba the correct doc scores 0.595 and the distractor 0.000.

2. Per-document diversity cap

• Retriever keeps at most max_per_doc chunks (default 2) per source document in one result list, applied after RRF fusion/rerank/filters so surplus chunks give way to other relevant documents.
• Config: [retrieval] max_per_doc (0 = no cap, preserves the original behavior).
• Effect: top-5 results spread across 3 documents instead of being dominated by one long file.

3. Gradio Web UI (optional)

• New loci webui command: chat pane + per-answer evidence panel that reuses build() / make_retriever() (same pipeline as the CLI and HTTP API); evidence shows document name > section.
• Optional extra: pip install 'loci-rag[ui]'.

4. Smaller items

• SYSTEM_PROMPT rewritten as strict grounding rules: excerpts only, explicit "not in KB" answer instead of guessing, bullet structure, mandatory Sources: citations.
• search / ask / bench accept --top_k as a long-form alias of -k (same args.k, no behavior change).
• Missing optional deps (pdf / docx / ocr / jieba) now warn once per run with the exact pip install command instead of skipping files silently.
• .gitignore: added .venv/.

Tests

• 207 passed, 1 skipped — all offline (FakeEmbedder; gradio/jieba tests are importorskip-guarded), +24 new tests in this branch.

[IvenKooLab]

Taking the tokenizer half of this in #4, with your commit intact and co-authored — the registry, config validation, bench --tokenizer and the fail-open behaviour are all unchanged.

One thing I changed, with numbers. jieba segments 全文检索 into a single token, so a 检索 query matches nothing:

correct = "全文检索是知识库的核心能力"
decoy   = "定期检查索引状态并重建"
query   = "检索"

default   → decoy 0.379 > correct 0.351   (the bug you found — confirmed)
jieba     → correct 0.000, no match
bigrams   → correct 0.666, decoy absent

So cjk_bigram is what ships as the recommended option: overlapping bigrams, no dictionary, no dependency, and your 混合检索 case still passes. jieba stays in the registry as an opt-in, and your two tests came over untouched; I added a recall case (检索 inside 全文检索) so the trade-off is pinned by a test instead of by a comment.

Also fixed the same bug in the latin half: default now indexes the joined form of hyphenated compounds, so wifi finds Wi-Fi.

Three things I still want your read on, from the rest of this PR:

• max_per_doc = 2 as a default changes results for every existing user. Defaulting to 0 plus a loci bench before/after would make it an easy yes.
• the SYSTEM_PROMPT rewrite changes the answer format for everyone — gating it behind [llm] strict_grounding feels safer.
• the Gradio UI: I would rather decide the console's future deliberately than add a second one here.

Thanks for the careful commit messages — they are why this took an hour instead of a day.

---

分词器这部分我接手了，开在 #4，你的 commit 原样保留、co-author 也是你（注册表、配置校验、bench --tokenizer、fail-open 全都没动）。只换了中文的实现：jieba 把「全文检索」切成一个词，查「检索」就一条都不命中，所以推荐项改成重叠双字（不用词典、不用额外依赖），你那个「混合检索」的用例照样成立。jieba 仍在注册表里作为可选项，你的测试原样保留，我补了一个召回用例。缺依赖提示、--top_k、还有那 374 行测试都很好。上面那三个默认值的问题，想听你的意见。

[IvenKooLab]

Landed in #4 — your commit is on main as 7650a46, authored by you with the message untouched, and the bigram tokenizer sits on top as c034cb7. So the registry, [bm25] tokenizer config validation, bench --tokenizer and the fail-open behaviour are all in, with your tests.

Closing this one as superseded, not rejected. The reason is purely mechanical: the branch's base commit (0365ec6d) no longer exists on main and the two histories never shared an ancestor, so GitHub can only show this PR as a 3,777-line deletion of the Web console and the guides. Nothing about the code in it was wrong.

Three things from the rest of this PR are still worth doing, and I would rather keep them than lose them:

• max_per_doc — default it to 0 and add a loci bench before/after in the PR, then it is an easy yes
• SYSTEM_PROMPT — gate the strict rules behind [llm] strict_grounding
• Gradio webui — needs a decision on the console's future first; two UIs is a real cost

I am happy to open those three, or to hand you the branches if you would rather write them yourself — say which and I will set it up.

Thanks for the care in those commit messages. A PR that sat for three weeks and still ported in an hour is a PR written by someone who thought about the reviewer.

---

同样的话用中文再说一遍：#4 已合并，你那个 commit 以你的名义在 main 上（7650a46），bigram 是在它上面的一层。这个 PR 关了，但关的原因是「被取代」，不是「被拒」——分支的 base commit 在 main 上已经不存在，GitHub 只能把它显示成删掉 3,777 行。

max_per_doc、SYSTEM_PROMPT、Gradio UI 这三块我留着，想单独开 PR 做。你要自己写的话我先把分支搭好，说一声就行。谢谢你每个 commit 写清楚原因——这种 PR 放三周还能一小时合进去，是因为作者替 reviewer 想过。