Repository navigation
[PR] fix: cover every Han block in plan search term extraction (planTokenize) - #4325
ranxianglei wants to merge 3 commits into
Conversation
#4324) The CJK character class only matched U+4E00-9FFF, so Extension A/B-H, compatibility ideographs and compatibility-supplement characters produced zero plan terms and were silently dropped from the index — an extended-Han-only plan even yielded null plan state and disabled plan-aware re-ranking entirely. Match \p{Script=Han} at both sites; single-char term semantics for Han are unchanged.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.5137-pr.4325.90258cbThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr4325.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 关联项: #4324(本 PR 要修的问题)、#4315/#4321(kernel 估算器的汉字区块覆盖——同属「汉字块覆盖缺口」家族但机制不同: token 估算 vs 词条提取, #4324 正文已注明, 不重复立项)。无其他在途 PR 触碰 标签: bug + scope/wiring + p3(缺陷类; host 侧 search_context/decompress 车道; 边缘场景 UX, 无可用性影响) 审核中(识别为 agent PR, 先核对代码与 CI, 不重做):
|
The job ran npm install --ignore-scripts then invoked node --test directly, but the contract suite's import chain reaches acp-kernel dist (src/config.ts), which is untracked since the in-repo kernel move (#2092); the lane failed ERR_MODULE_NOT_FOUND from its first run. Same pretest build step as the main test lanes. Found while reviewing #4325.
(ai: vllm-qwen/qwen3.8-27b) 审查进展:PR 本体改动已核过,顺手修掉了挡在合并前的两个 master 存量 CI 红。 PR 本体(planTokenize 全汉字覆盖)
发现的两个 CI 红(均与 PR 无关,master 存量问题;查重:open+closed issue/PR 均无对应条目)PR head 的真实全矩阵运行里 6 个 test shard 全部秒死 + mysql-contract 失败,根因各一:
两个 commit 均已推到本 PR 分支( 一个提醒:当前 PR 页面上的绿标是假的15:52 那次「success」是标签同步触发的 labeled 事件运行——plan job 对非 |
(ai: vllm-qwen/qwen3.8-27b) 审查完成,最终结论。CI 真实全矩阵已跑完(push 触发的非 skip 运行),本 PR 代码可合;唯一剩余红是第三个 master 存量问题,已立项 #4338。 八问
CI 最终状态(run 39812533386 @ head 861aec4)
第三个红 → 已立 #4338windows shard 1 上两个 corrupt-index.db 测试(#2771、#3040)确定性失败(首跑 + rerun 两次同签名,临时目录名不同):断言全过,死在 finally 清理 处理依据政策:不熟悉子系统 + 本地(Linux)无法复现 Windows 锁语义,不当场盲修——已开 #4338(来源标记指向本 PR),正文含完整证据链(run/job 号、错误原文、时间点推理、bili 侧持有排除)、最小 repro 方案与修法方向;标签 bug/p2/scope/storage。查重:无(#2771/#2781/#3040/#2908 为相关历史但机制不同,已在正文说明)。 判定
|
(ai: vllm-qwen/qwen3.8-27b)
Fixes #4324 (来源: #4315 分析时发现,详见该 issue 正文与评论)
What / why
The decompress tool's plan-aware search (
search_contextre-ranking / steering) extracts its term index from the latest plan snapshot viaplanTokenize(src/decompress-shared.ts). Its CJK character class only matchedU+4E00–9FFF(basic-block Han):Extension A/B–H, compatibility ideographs (U+F900–FAFF) and compatibility-supplement characters therefore produced zero tokens and were silently dropped from the index. An extended-Han-only plan even yielded
terms.size === 0→extractPlanStatereturnednull→ plan-aware re-ranking was disabled entirely for that session.Both sites now match
\p{Script=Han}(with theuflag). Single-char term semantics for Han are unchanged; ASCII/digit tokens are untouched.Behavior change (intended, disclosed)
extractPlanState→ null (re-ranking off)Additive only: existing terms are never removed or renamed; previously-dropped chars gain ranking signal. No wire/protocol change, no config surface change, host-side (
src/decompress-shared.ts) so no kernel gate applies.Verification
tests/search-plan-aware.test.ts: mixed basic/ext-A/compat/supplementary TodoWrite snapshot asserts each character present inst.terms; pure-extended-Han user turn yields non-null state.not ok 4, 13/14); with it,tests/search-plan-aware.test.tsis 14/14 green.npm run typecheckclean.八问
planTokenize的正则字符类写死U+4E00–9FFF,与 kernel 估算器同类覆盖缺口,但机制独立(词条提取 vs token 估算)。\p{Script=Han}(u 标志)匹配全部汉字区块,单字成词语义不变。extractPlanState/scoreWithPlan/steering 既有行为清单——基本区汉字、ASCII 数字词条、null-state 回退路径全部保持;变更仅「新增此前被丢弃的词条」,纯增量信号。现有 13 个用例原值不变。备注:假名/韩文同样不在该分词器覆盖内(kernel 估算器有覆盖)——是否让计划词条也覆盖假名/韩文是另一个取舍,未纳入本 PR,已在 #4324 正文留注待决定。