Repository navigation
[PR] fix: defaultCountTokens counts every Han ideograph and subtracts the UTF-16 unit footprint (#4315) - #4321
ranxianglei wants to merge 4 commits into
Conversation
…int in defaultCountTokens (#4315) The CJK match only covered U+4E00-9FFF plus kana/Hangul ranges, so Extension A/B-H, compatibility ideographs and compatibility-supplement characters fell into the chars/4 remainder, and supplementary-plane characters (two UTF-16 units each) were double-counted even where matched. Match \p{Script=Han} (u flag) alongside the existing kana/Hangul ranges, and subtract the matches' code-unit footprint from text.length instead of their character count.
…stead of copying its formula (#4315) The mock upstream's window enforcement carried a copy-paste of defaultCountTokens' old regex+formula; a future estimator change would silently drift from it. Import the real function.
Since #3900 sideRequestGuard reserves max_tokens headroom on every wire including anthropic, the declared window must include that reservation on top of the 15% tolerance band for the payload to forward (same shape as the OpenAI case). The row failed on master without it.
Resolve kernel version conflict: master advanced to 0.0.153, so this PR's tokenize.ts change bumps 0.0.153 -> 0.0.154 (Kernel Boundary rule 2). No other content changes.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.5133-pr.4321.7008999This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr4321.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重:不重复 —— 本 PR 专治 #4315 的 结论先行:代码正确、测试有牙、可合(待 CI 绿)。 以下为独立审核(未照抄 PR 侧八问),并已顺手把分支 rebase 到当前 master、解掉版本冲突推回。 结构化八问
Kernel Boundary 合规
本地定向验证(在已 rebase 到 master 的树上跑,非全量)
我已做的动作分支落后 master 30 个提交,唯一冲突是 kernel 版本行(master 已到 0.0.153)。我把当前 master 合入分支、把版本解到 0.0.154,fast-forward 推回(无 force-push)。现在 PR diff vs master 恰好是预期的 6 个文件, 一处需澄清的小发现(非阻塞)
合并判定CI 绿即可合(人为合并,我不代合)。代码与测试质量达标;上面 side-request-isolation 的理由 nit 建议顺手处理,但不拦。 |
kernel-change: defaultCountTokens (the kernel's default token estimator) undercounts Extended/Compatibility/supplementary-plane Han — its CJK regex only covered U+4E00–9FFF plus kana/Hangul, so Extension A/B–H and compatibility ideographs fell into the chars/4 remainder, and supplementary-plane characters (two UTF-16 units each) were double-counted in the remainder even where matched. The estimator is consumed directly across host and kernel (message sizing, range/budget decisions, cold-start budget); only the producer of these bytes — this function — can fix it, and no host-side knob reaches it.
Closes #4315
What
defaultCountTokens()now bills EVERY Han ideograph at ~1 token via\p{Script=Han}(u flag) alongside the existing kana (U+3040–30FF) and Hangul-syllable (U+AC00–D7AF) ranges, and subtracts the matches' UTF-16 CODE-UNIT footprint fromtext.length(not their character count) before estimating the non-CJK remainder with /4.Behavior change (disclosed)
Per 100 chars:
Extended-Han-heavy estimates move UP toward the function's own stated heuristic ("CJK ~1 token per char"); nothing previously counted as CJK changes value. Estimation-caliber fix only: no config surface, persistence format, wire shape, or tool schema touched. Usage calibration (#2366/#2369) still overrides on calibrated paths.
Verification
npm run typecheckclean locally.Incidental test fixes (separate labeled commits)
defaultCountTokensso it cannot drift (fixtures are ASCII-only → values identical).八问