Skip to content

[PR] fix: defaultCountTokens counts every Han ideograph and subtracts the UTF-16 unit footprint (#4315) - #4321

Open
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-22_default-count-tokens-han-coverage
Open

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-22_default-count-tokens-han-coverage

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

kernel-change: defaultCountTokens (the kernel's default token estimator) undercounts Extended/Compatibility/supplementary-plane Han — its CJK regex only covered U+4E00–9FFF plus kana/Hangul, so Extension A/B–H and compatibility ideographs fell into the chars/4 remainder, and supplementary-plane characters (two UTF-16 units each) were double-counted in the remainder even where matched. The estimator is consumed directly across host and kernel (message sizing, range/budget decisions, cold-start budget); only the producer of these bytes — this function — can fix it, and no host-side knob reaches it.

Closes #4315

What

defaultCountTokens() now bills EVERY Han ideograph at ~1 token via \p{Script=Han} (u flag) alongside the existing kana (U+3040–30FF) and Hangul-syllable (U+AC00–D7AF) ranges, and subtracts the matches' UTF-16 CODE-UNIT footprint from text.length (not their character count) before estimating the non-CJK remainder with /4.

Behavior change (disclosed)

Per 100 chars:

input before after
Basic Han U+4E00–9FFF 100 100 (unchanged)
Extension A U+3400–4DBF 25 100
Compatibility Han U+F900–FAFF 25 100
Supplementary plane (Ext B U+20000+, C/D/E/F/G, compat supplement) 50 100
Kana / Hangul syllables unchanged unchanged
Pure ASCII / digits / other non-CJK unchanged unchanged

Extended-Han-heavy estimates move UP toward the function's own stated heuristic ("CJK ~1 token per char"); nothing previously counted as CJK changes value. Estimation-caliber fix only: no config surface, persistence format, wire shape, or tool schema touched. Usage calibration (#2366/#2369) still overrides on calibrated paths.

Verification

  • Issue's exact repro script: all four cases → 100 (was 100/25/25/50); mixed basic+supplementary 75→100; ASCII/kana/Hangul samples keep their exact pre-fix values.
  • New regression tests in kernel/tests/tokenizer.test.ts: extension-block table (basic, Ext A/B/C/D/G, compatibility, compatibility supplement), the supplementary-unit-leak case, ASCII-unchanged pins. Kernel tokenizer suite 12/12.
  • tokenize.ts is Tier-A guarded by the verif module: tests/verif-subset-guard.test.ts 2/2 — the new syntax (script-property regex, String.join, optional chaining) stays inside the translatable subset.
  • Host suites importing defaultCountTokens (preflight-fork-estimate, side-request-isolation, issue3713, issue2407-thinking-caliber, issue1320-thinking-mass, issue532-panel-system-tokens, web-blocks-tree, issue3645-calibrated-futility): green locally; full-suite verdict = CI on this head.
  • npm run typecheck clean locally.

Incidental test fixes (separate labeled commits)

  1. tests/preflight-fork-estimate.test.ts — the mock upstream carried a copy-paste of this very formula; it now imports defaultCountTokens so it cannot drift (fixtures are ASCII-only → values identical).
  2. tests/side-request-isolation.test.ts — pre-existing master red: since [BUG] acp-proxy upstream HTTP 400: Request with 1,198,269 tokens exceeds the model's 1,048,576 token limit, consistently | [BUG] acp-proxy upstream HTTP 400:请求 1,198,269 tokens 超过模型上限 1,048,576,一直复现——是插件压缩未生效,还是 DSH 的问题? #3900 the guard reserves max_tokens headroom on every wire including anthropic, but the "within 15% tolerance → forward" row never carried the reservation (declared limit = floor(est/1.10) → effective −100 → blocked=true). The row now passes floor(est/1.10)+100, same shape as the OpenAI case below it. Verified the failure reproduces on clean master (git stash + rebuilt kernel dist).

八问

  1. 是什么问题:默认估算器只认基本区汉字——扩展区/兼容汉字被按 /4 低估(100 字算成 25),补充平面汉字还因 UTF-16 双码元在余数里重复计入(算成 50)。
  2. 什么原因导致的:正则区间写死在 U+4E00–9FFF,没有跟随 Unicode 扩展区演进;且「CJK 按字计、其余 /4」的减法用字符数而非 UTF-16 码元数,两个缺陷叠加。
  3. 影响范围与严重程度:所有直接消费 defaultCountTokens 的消息大小/区间/冷启动预算判断,遇扩展区或补充平面汉字时低估(最多 4×);基本区汉字与非 CJK 文本完全不受影响。精度瑕疵级——usage 校准(长会话中token用量被系统性低估(CJK 文本),导致agent误判上下文压力、进入「小收益折叠」循环 #2366/fix: expose billed input in acp_status + two-way estimator calibration (#2366) #2369)兜住部分路径,报告者亦未复现超窗或会话卡死。
  4. 怎么修复的:以 Unicode 脚本属性 \p{Script=Han} 覆盖全部汉字区块(新扩展区落地自动跟上),保留既有假名/韩文区间;余数计算改为减去匹配字符的码元占用,消除双计数。
  5. 是否有回归问题:BMP 内既有行为逐值不变(英文/数字/假名/韩文样例原值钉死在回归测试里);唯一变化面是原本就被低估的扩展区/兼容/补充平面汉字,向上修正到函数自身声明的启发式。wire、配置、持久化、工具面均未触碰;副作用是扩展汉字密集的会话触发压缩的时机略提前(其真实体量确实更大)——方向即本修复目的。
  6. 是否破坏缓存:缓存键/前缀不消费该估算器;校准路径逻辑未动。仅压缩触发时机对扩展汉字密集会话略提前(见上),不失效任何缓存。
  7. 是否改了设计方向:不涉及——仍是「CJK 按字计、其余 /4」启发式,只是把「CJK」的定义补全为整个 Unicode Han 属性。
  8. 是否引入了新的配置:无。

awork added 3 commits October 22, 2026 23:27
…int in defaultCountTokens (#4315)

The CJK match only covered U+4E00-9FFF plus kana/Hangul ranges, so Extension A/B-H, compatibility ideographs and compatibility-supplement characters fell into the chars/4 remainder, and supplementary-plane characters (two UTF-16 units each) were double-counted even where matched. Match \p{Script=Han} (u flag) alongside the existing kana/Hangul ranges, and subtract the matches' code-unit footprint from text.length instead of their character count.
…stead of copying its formula (#4315)

The mock upstream's window enforcement carried a copy-paste of defaultCountTokens' old regex+formula; a future estimator change would silently drift from it. Import the real function.
Since #3900 sideRequestGuard reserves max_tokens headroom on every wire including anthropic, the declared window must include that reservation on top of the 15% tolerance band for the payload to forward (same shape as the OpenAI case). The row failed on master without it.
@ranxianglei ranxianglei changed the title fix: defaultCountTokens counts every Han ideograph and subtracts the UTF-16 unit footprint (#4315) [PR] fix: defaultCountTokens counts every Han ideograph and subtracts the UTF-16 unit footprint (#4315) Oct 22, 2026
Resolve kernel version conflict: master advanced to 0.0.153, so this PR's
tokenize.ts change bumps 0.0.153 -> 0.0.154 (Kernel Boundary rule 2). No other
content changes.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-22_default-count-tokens-han-coverage (8d01517)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.5133-pr.4321.7008999

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4321.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复 —— 本 PR 专治 #4315 的 defaultCountTokens Han 覆盖缺口,无其他在途 issue/PR 触及该估算器。

结论先行:代码正确、测试有牙、可合(待 CI 绿)。 以下为独立审核(未照抄 PR 侧八问),并已顺手把分支 rebase 到当前 master、解掉版本冲突推回。

结构化八问

  1. 是什么问题:默认估算器只认基本区汉字 U+4E00–9FFF,扩展 A / 兼容汉字被按 /4 低估(100 字算成 25),补充平面汉字还因 UTF-16 双码元在余数里重复计入(算成 50)。
  2. 什么原因导致的:正则区间写死在 U+4E00–9FFF,没跟随 Unicode 扩展区演进;且「CJK 按字计、其余 /4」的减法用字符数而非 UTF-16 码元数,两个缺陷叠加。
  3. 影响范围与严重程度:所有直接消费 defaultCountTokens 的消息体量 / 区间 / 冷启动预算判断,遇扩展区或补充平面汉字时低估(最多 4×);基本区汉字与非 CJK 文本完全不受影响。精度瑕疵级——usage 校准(长会话中token用量被系统性低估(CJK 文本),导致agent误判上下文压力、进入「小收益折叠」循环 #2366/fix: expose billed input in acp_status + two-way estimator calibration (#2366) #2369)兜住部分路径。
  4. 怎么修复的:用 \p{Script=Han}(u flag)覆盖全部汉字区块(新扩展区落地自动跟上),保留既有假名 / 韩文区间;余数改为减去匹配字符的 UTF-16 码元占用(join("").length)以消除双计数。
  5. 是否有回归问题:本 PR 无任何 src 变更。BMP 内新增命中恰为扩展 A(U+3400–4DBF)+ 兼容(U+F900–FAFF),都是本修复目标,无意外过度匹配;基本区 / ASCII / 假名 / 韩文逐值不变(回归测试把原值钉死)。wire / 配置 / 持久化 / 工具面均未触碰;副作用是扩展汉字密集会话触发压缩略提前(其真实体量确实更大)——方向即本修复目的。
  6. 是否破坏缓存:缓存键 / 前缀不消费该估算器,不失效任何缓存;仅压缩触发时机对扩展汉字密集会话略提前。
  7. 是否改了设计方向:不涉及——仍是「CJK 按字计、其余 /4」启发式,只是把「CJK」的定义补全为整个 Unicode Han 属性。
  8. 是否引入新配置:无。

Kernel Boundary 合规

  • body 带 kernel-change: 声明 ✓
  • tokenize.ts(Tier-A)版本随 master 推进解到 0.0.153 → 0.0.154 ✓(Kernel Boundary 规则 2)
  • tests/verif-subset-guard.test.ts 2/2 通过——新语法(script-property regex / String.join / 可选链)仍在可翻译子集内 ✓

本地定向验证(在已 rebase 到 master 的树上跑,非全量)

  • kernel/tests/tokenizer.test.ts 12/12(含扩展区块表 basic/ExtA/ExtB/C/G/compat/compat-supplement、FA6D 尾边界、supplementary-leak 用例、ASCII 原值钉死)
  • tests/verif-subset-guard.test.ts 2/2 · tests/preflight-fork-estimate.test.ts 2/2 · tests/side-request-isolation.test.ts 33/33
  • npm run typecheck 干净
  • 全量回归 = 远端 CI 在本 head 上的 test lane(本地不预跑全量,§7.2)

我已做的动作

分支落后 master 30 个提交,唯一冲突是 kernel 版本行(master 已到 0.0.153)。我把当前 master 合入分支、把版本解到 0.0.154,fast-forward 推回(无 force-push)。现在 PR diff vs master 恰好是预期的 6 个文件,mergeable=true(blocked = 等 CI 检查项,不是冲突)。

一处需澄清的小发现(非阻塞)

tests/side-request-isolation.test.ts 那笔「补 #3900 输出预留」的顺手改动(anthropic 容差行窗口 +100)在当前 master 上是 no-op。我在当前 src 下实测:est=2007,floor(est/1.10)=1824 本身就 blocked=false(放行),加不加 +100 都放行;PR body 声称的「clean master 上复现的红」在当前 master + 估算器下复现不出来(该输入纯 ASCII,est 新旧估算器一致,差异只在预留逻辑,而预留逻辑属 src、当前 master 已含 #3900)。改动本身无害(该测试其余行仍有牙),但理由表述不准——建议要么撤掉这个 +100,要么改正确 commit / PR 里的说明。不影响合并判定。

合并判定

CI 绿即可合(人为合并,我不代合)。代码与测试质量达标;上面 side-request-isolation 的理由 nit 建议顺手处理,但不拦。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant