Skip to content

[PR] fix: cover every Han block in plan search term extraction (planTokenize) - #4325

Open
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-22_plan-tokenize-han-coverage
Open

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-22_plan-tokenize-han-coverage

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Fixes #4324 (来源: #4315 分析时发现,详见该 issue 正文与评论)

What / why

The decompress tool's plan-aware search (search_context re-ranking / steering) extracts its term index from the latest plan snapshot via planTokenize (src/decompress-shared.ts). Its CJK character class only matched U+4E00–9FFF (basic-block Han):

const re = /[a-z0-9]+|[\u4e00-\u9fff]/g;
...
const cjk = /[\u4e00-\u9fff]/.test(t);

Extension A/B–H, compatibility ideographs (U+F900–FAFF) and compatibility-supplement characters therefore produced zero tokens and were silently dropped from the index. An extended-Han-only plan even yielded terms.size === 0 → extractPlanState returned null → plan-aware re-ranking was disabled entirely for that session.

Both sites now match \p{Script=Han} (with the u flag). Single-char term semantics for Han are unchanged; ASCII/digit tokens are untouched.

Behavior change (intended, disclosed)

Input in plan snapshot / user turns Before After
Basic-block Han (U+4E00–9FFF) indexed, 1 char = 1 term unchanged
Extension A/B–H, compat ideographs, compat supplement silently dropped (no terms) indexed, 1 char = 1 term
Extended-Han-only plan extractPlanState → null (re-ranking off) non-null state, re-ranking active

Additive only: existing terms are never removed or renamed; previously-dropped chars gain ranking signal. No wire/protocol change, no config surface change, host-side (src/decompress-shared.ts) so no kernel gate applies.

Verification

  • New regression test in tests/search-plan-aware.test.ts: mixed basic/ext-A/compat/supplementary TodoWrite snapshot asserts each character present in st.terms; pure-extended-Han user turn yields non-null state.
  • Negative control: with the source change stashed, that test fails (not ok 4, 13/14); with it, tests/search-plan-aware.test.ts is 14/14 green.
  • npm run typecheck clean.
  • Full-suite verdict: remote CI on this head.

八问

  1. 是什么问题:计划/待办感知的搜索词条提取只认基本区汉字,扩展/兼容汉字被静默丢出索引;纯扩展汉字的计划会让重排整体失效。
  2. 什么原因导致的:planTokenize 的正则字符类写死 U+4E00–9FFF,与 kernel 估算器同类覆盖缺口,但机制独立(词条提取 vs token 估算)。
  3. 影响范围与严重程度:p3 边缘场景 UX——计划涉及生僻/扩展汉字时搜索重排丢失信号;无数据丢失、无可用性影响。
  4. 怎么修复的:两处正则改以 Unicode 脚本属性 \p{Script=Han}(u 标志)匹配全部汉字区块,单字成词语义不变。
  5. 是否有回归问题:审过 extractPlanState/scoreWithPlan/steering 既有行为清单——基本区汉字、ASCII 数字词条、null-state 回退路径全部保持;变更仅「新增此前被丢弃的词条」,纯增量信号。现有 13 个用例原值不变。
  6. 是否破坏缓存:不涉及(搜索时计算,不缓存、不改持久化)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

备注:假名/韩文同样不在该分词器覆盖内(kernel 估算器有覆盖)——是否让计划词条也覆盖假名/韩文是另一个取舍,未纳入本 PR,已在 #4324 正文留注待决定。

#4324)

The CJK character class only matched U+4E00-9FFF, so Extension A/B-H, compatibility ideographs and compatibility-supplement characters produced zero plan terms and were silently dropped from the index — an extended-Han-only plan even yielded null plan state and disabled plan-aware re-ranking entirely. Match \p{Script=Han} at both sites; single-char term semantics for Han are unchanged.
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_plan-tokenize-han-coverage (861aec4)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5137-pr.4325.90258cb

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4325.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 关联项: #4324(本 PR 要修的问题)、#4315/#4321(kernel 估算器的汉字区块覆盖——同属「汉字块覆盖缺口」家族但机制不同: token 估算 vs 词条提取, #4324 正文已注明, 不重复立项)。无其他在途 PR 触碰 planTokenize。

标签: bug + scope/wiring + p3(缺陷类; host 侧 search_context/decompress 车道; 边缘场景 UX, 无可用性影响)

审核中(识别为 agent PR, 先核对代码与 CI, 不重做):

  • 代码: 2 文件 +23/−2。diff 已通读——src/decompress-shared.ts 的 planTokenize 两处正则由写死 U+4E00–9FFF 改为 \p{Script=Han}(u 标志), 新增一个回归用例(tests/search-plan-aware.test.ts)。方向正确、单字成词语义不变, 纯增量信号。
  • CI: 本 head 上 6 个 test shard 全红, 但失败签名是 scripts/run-ci-tests.mjs 启动即报 tests/storage-durability.test.ts 未登记进 LIVE_FILES——master 本身同样全红(master head 各 shard 同一错误), 与本 PR 改动无关。静态门禁(kernel-guard / pi-subagents-guard / version-guard / lock-sync / unused-exports / config-surface-guard / docs-coverage 等)全绿。
  • 接下来: ①本地定向跑 tests/search-plan-aware.test.ts(PR 分支); ②核对 mysql-contract 失败签名是否同源; ③查 master 红是否已有在途条目, 有则补充信息不开新项。完成后按八问回复结论。

@ranxianglei ranxianglei added bug Something isn't working p3 scope/wiring labels Oct 22, 2026
awork added 2 commits October 23, 2026 00:15
#4177 added a process-spawning test without registering it, so the run-ci-tests.mjs tripwire dies every shard of every PR whose merge tree includes it (red since aa198fe). Concurrency reviewed: unique mkdtempSync dirs, stdio:ignore child, no port binding. Found while reviewing #4325.
The job ran npm install --ignore-scripts then invoked node --test directly, but the contract suite's import chain reaches acp-kernel dist (src/config.ts), which is untracked since the in-repo kernel move (#2092); the lane failed ERR_MODULE_NOT_FOUND from its first run. Same pretest build step as the main test lanes. Found while reviewing #4325.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审查进展:PR 本体改动已核过,顺手修掉了挡在合并前的两个 master 存量 CI 红。

PR 本体(planTokenize 全汉字覆盖)

  • diff 逐行读过:两处正则 [\u4e00-\u9fff] → \p{Script=Han}(u 标志),单字成词语义、ASCII/数字词条路径、null-state 回退全部保持,纯增量信号,host 侧改动无 kernel gate。
  • 本地定向验证(worktree @ PR head + pretest 构建):tests/search-plan-aware.test.ts 14/14 通过(含新增的扩展区/兼容区/补充区断言)。
  • 回归清点:基本区汉字、ASCII 数字词条、stopword 过滤、extractPlanState null 回退——既有行为清单逐项保持,无漂移。

发现的两个 CI 红(均与 PR 无关,master 存量问题;查重:open+closed issue/PR 均无对应条目)

PR head 的真实全矩阵运行里 6 个 test shard 全部秒死 + mysql-contract 失败,根因各一:

  1. test shard 全灭:[PR] storage: durability trio — open-time quick_check + online VACUUM INTO backup + second-process lockfile (#4175) #4177 新增 tests/storage-durability.test.ts(会 spawn 子进程)但没登记进 scripts/live-files.mjs 的 LIVE_FILES,run-ci-tests.mjs 的 tripwire 在每个 shard 启动即死。PR 的 CI checkout 是 merge commit(含 master),所以每个 PR 都中招。
    → 已在本 PR 分支独立 commit 18cb81fe0 修掉:登记进 LIVE_FILES。并发假设按 tripwire 要求审过:独立 mkdtempSync 临时目录、stdio:ignore 子进程、不绑端口,安全。
  2. mysql-contract 失败:该 job 只做 npm install --ignore-scripts 就直接跑测试,但契约测试的 import 链会到 acp-kernel 的 dist(src/config.ts),而 in-repo kernel 迁移(acp-kernel 并入billion-context并保持相对独立 #2092)后 kernel/dist 不进 git——这个 lane 从引入第一天起就是 ERR_MODULE_NOT_FOUND。
    → 已在独立 commit 861aec448 修掉:给该 job 补上与主测试 lane 相同的 npm run pretest 构建步骤。本地已验证补构建后 tests/storage-backend-knob.test.ts 5/5 通过。

两个 commit 均已推到本 PR 分支(d74187ad9..861aec448),各自独立标注,可单独 revert。tripwire 扫描逻辑我在本地按原样复刻跑过:登记后顶层未登记 spawner 为零。

一个提醒:当前 PR 页面上的绿标是假的

15:52 那次「success」是标签同步触发的 labeled 事件运行——plan job 对非 ci:full 标签事件输出 scope=skip,下游全部跳过,空转成 success。真实判决看 push 触发的全矩阵运行,我现在盯着它,跑完给最终八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审查完成,最终结论。CI 真实全矩阵已跑完(push 触发的非 skip 运行),本 PR 代码可合;唯一剩余红是第三个 master 存量问题,已立项 #4338。

八问

  1. 是什么问题:计划感知的搜索词条提取只认基本区汉字(U+4E00–9FFF),扩展 A/B–H、兼容区、补充区汉字被静默丢出索引;纯扩展汉字的计划会让 extractPlanState 返回 null,重排整体失效。
  2. 什么原因导致的:planTokenize(src/decompress-shared.ts)两处正则字符类写死基本区区间,与 kernel 估算器同类覆盖缺口但机制独立。
  3. 影响范围与严重程度:p3 边缘场景 UX——计划涉及生僻/扩展汉字时搜索重排丢信号;无数据丢失、无可用性影响。
  4. 怎么修复的:两处正则改 \p{Script=Han}(u 标志),单字成词语义不变,纯增量信号。
  5. 是否有回归问题:无。既有行为清单逐项核过——基本区汉字、ASCII/数字词条、stopword 过滤、null-state 回退全部保持,现有 13 个用例原值不变;新增回归测试本地复跑 14/14。我追加的两个 CI commit 只动 scripts/live-files.mjs(登记一行)和 .github/workflows/ci.yml(mysql job 加一步构建),不碰产品代码路径。
  6. 是否破坏缓存:不涉及(搜索时计算)。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

CI 最终状态(run 39812533386 @ head 861aec4)

Lane 结果 说明
全部静态 gate(kernel-guard、version-guard、lock-sync 等 12 项) ✅
mysql-contract ✅ 我的 pretest 修复生效(此前从引入起必挂)
test shards 5/6 ✅ tripwire 解除后首次真正跑完全套
test (windows-latest, 22, 1) ❌ 与本 PR 无关的第三个存量红,见下

第三个红 → 已立 #4338

windows shard 1 上两个 corrupt-index.db 测试(#2771、#3040)确定性失败(首跑 + rerun 两次同签名,临时目录名不同):断言全过,死在 finally 清理 rmrf 的 EBUSY unlink ...index.db——Windows 上降级路径有句柄未释放,tmp-rm.ts 约 10s 重试梯全部耗尽。POSIX 上 unlink 对打开文件照样成功,所以只有 Windows 显形。与 #2781 同族但泄漏点在更晚的窗口(#2771 测试里 loadSync 后的 rename-guard 是通过的,句柄在 writeNow 阶段才被拿到)。

处理依据政策:不熟悉子系统 + 本地(Linux)无法复现 Windows 锁语义,不当场盲修——已开 #4338(来源标记指向本 PR),正文含完整证据链(run/job 号、错误原文、时间点推理、bili 侧持有排除)、最小 repro 方案与修法方向;标签 bug/p2/scope/storage。查重:无(#2771/#2781/#3040/#2908 为相关历史但机制不同,已在正文说明)。

判定

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p3 scope/wiring

Projects

None yet

1 participant