Skip to content

[PR] fix: validate token memo against exact counted text - #7211

Open
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_validate-token-memo-text
Open

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-18_validate-token-memo-text

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #7208.

Problem

Kernel IDs describe logical message identity. Tool arguments serialized with Unicode escapes and literal CJK can therefore have the same ID and text length while the CJK-aware token counter returns different values. The memo currently treats that pair as identical, so unrelated earlier requests can change the output budget for the same current request.

A real OpenAI codec fixture gives 100-character arguments with equal IDs but counts of 25 and 29; the Responses codec has the same behavior. With 800 completed tool calls through the HTTP proxy, the same Unicode request receives max_tokens=51271 after an escaped-JSON warmup but 47687 from a cold memo.

Change

Require the cached entry's exact text to match as well as the existing ID, length and thinking-token key. A live message anchors its immutable text/count entry through a WeakMap, and the global memo holds only WeakRefs. Cloned histories can still reuse live entries, while expired entries are safely recounted. Resetting the test memo also resets those anchors.

The kernel, wire serialization, compression modes and budget formulas are unchanged. Both plugin and proxy paths use the same corrected estimator; this does not change their different summary carriers.

Validation

  • 17 new regressions: 15 fail / 2 pass on the unmodified base. The fix passes 61 related tests, including both codecs, CJK variants, both warmup orders, in-place rewrites, clones, thinking changes and real HTTP forwarding.
  • The HTTP fixture now forwards 47687 for both warm and cold Unicode requests. Each returns HTTP 200 with the expected answer, one upstream request and all 800 original tool arguments preserved.
  • npm run typecheck, npm run build, npm run check:test-import-graph and git diff --check pass.
  • Source-verified fresh-process cost checks make no speedup claim: hot estimation of 2,000 unchanged messages measured about 0.53–0.55 ms before and 0.68 ms after. Clone-plus-JSON-parse timings were noisy. After releasing a 1,000-message history and collecting across event-loop turns, candidate heap growth fell from about 32.5 MB to 0.38 MB and recounting stayed correct. Live messages still retain their entries; GC timing and total live-history memory are not bounded by the index cap.
  • E2E_CHECK=1 finds the Codex CLI and built dist, but the configured local upstream at 127.0.0.1:8199 is unreachable. The required real-model four-phase Codex suite has not run and remains a pre-merge gate. Full-suite validation is left to remote CI.

Prior work

Compared all 272 live fork branches and 920 upstream file-history commits (171 complete versions). The memo introduced by #5131 does not verify counted text; the actual patches in #6460 and #6474 address different calibration/billing behavior.

@ranxianglei ranxianglei changed the title fix: validate token memo against exact counted text [PR] fix: validate token memo against exact counted text Nov 18, 2026
@ranxianglei ranxianglei added bug Something isn't working p1 scope/kernel labels Nov 18, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review of PR #7211 — fix: validate token memo against exact counted text

Duplicate screening: 查重: 不重复. Searched open+closed issues/PRs for the same fix (token memo / estimateCoreMessages / stale Unicode count) — only #7211 fixes #7208. The PR's own prior-work check (272 fork branches, 920 file-history commits) independently confirms no prior fix.

Bug verification: Confirmed the #7208 bug is real. The memo key (id, textLength, thinkingTokens) omits the exact counted text. Two messages with the same kernel id, same length, but different text (Unicode-escaped JSON vs literal CJK) have different CJK-aware counts (25 vs 29 in the repro) yet share one memo entry — the first count is reused for the second. I reproduced it with the real codecs (openaiToCore, responsesToCore) across three CJK character sets.

Fix correctness: The fix adds an exact-text gate to the memo hit (cached.text === m.text). I verified countMessageTokens (kernel/src/tokenize.ts:34) depends ONLY on message.text + message.thinkingTokens — both already in the memo key — so a hit is provably safe. The change is a strict tightening (fewer hits, never more false hits), so it cannot introduce a new miscount; it only converts the stale hits into safe recounts.

Tests: Verified the 17 new tests are real regressions — 15 fail / 2 pass on the unmodified base (I swapped in the base preflight.ts and re-ran). The 2 that pass on base test already-correct behavior (clone/thinking reuse; length / non-kernel-id / reset). The HTTP end-to-end test reproduces the #7208 budget skew (warm 51,271 vs cold 47,687) and asserts warm==cold after the fix. All 23 tests pass on the PR head.

CI: Green — 16 success, 12 skipped, 0 failed, 2 pending (mysql-contract, docs-sync). typecheck and check:test-import-graph pass locally. The blocked merge state is the external-PR review gate plus those 2 in-flight checks.

Memory: The fix trades Map<string, number> for Map<string, WeakRef> + WeakMap<CoreMessage, entry>. The owners WeakMap holds one small entry per live message (the text is a shared reference, not a copy). This is a modest, session-proportional increase, not a leak; the PR correctly notes live-history memory is not bounded by the 200K index cap.

Behavior change (disclosed): A warm Unicode request now receives its own correct budget (47,687) instead of inheriting the stale memo's budget (51,271). This is the intended fix.

Labels: Applied bug + p1 + scope/kernel (stale cached decision input in the context-management estimator).

Verdict: Recommend merge — correct root-cause fix, real regression tests, green CI, clean diff, behavior change disclosed. Pending the 2 in-flight CI checks and owner approval.


结构化八问(中文,供人工复核):

  1. 是什么问题: token 估算 memo 用 (id, 文本长度, thinking) 作键,但同 id 同长度的两种 Unicode 序列化(转义 JSON vs 字面 CJK)的 CJK 感知计数不同(复现里 25 vs 29),memo 却把先到的计数复用到后者——同一请求的输出预算被无关的前置流量改变(800 调用场景偏差 3,584 token)。

  2. 什么原因导致的: memo 键只含逻辑身份(id + 长度 + thinking),不含被计数的确切文本。kernel 的 id 是逻辑身份而非文本摘要,所以同 id 同长度的消息可能文本不同、计数不同。根因在 memo 键设计(缺文本维度),不在序列化、不在预算公式。

  3. 影响范围与严重程度: 影响所有走 estimateCoreMessages 的路径(preflight 尺寸、窗口门、输出 clamp、relay、k̂ 校准);plugin 与 proxy 两种模式共用同一估算器,都受影响。触发条件:同一逻辑消息以两种「同长度、不同文本」的序列化先后出现(转义 vs 字面 CJK)。严重程度 p1:输出预算被无关流量改变,可影响压缩决策或无谓削减输出;但会话仍可用(HTTP 200),属「能用但决策输入被污染」,非完全不可用。

  4. 怎么修复的: memo 命中必须匹配确切文本(cached.text === m.text),保留现有 id/长度/thinking 键。活消息经 WeakMap 锚定其不可变「文本/计数」条目,全局 memo 只持 WeakRef——克隆历史仍可复用活条目,过期条目安全重计。修在 memo 键(源头补文本维度),不是过滤文本或禁用 memo(治标)。

  5. 是否有回归问题: 逐项审过既有行为——①稳态复用(同消息重估):保持(同文本→命中);②克隆复用(同文本克隆):保持;③碰撞(同 id 同长不同文本):从「错误复用」改为「正确重计」(即修复本身);④D 车道 / renderTags 原地改写:保持(文本变→重计);⑤memo 上限 200K:保持。高危模式逐项:无 fail-fast 变吞错、无阈值/时序漂移、无输出格式变化、无缓存键失效。唯一行为变更是 warm Unicode 请求预算 51,271→47,687(有意修复,已披露)。

  6. 是否破坏缓存: 不破坏 prompt cache(出站前缀字节不变;memo 是进程内估算缓存,不是 prompt cache)。memo 命中更严格(更多重计),不影响上游前缀缓存命中策略。

  7. 是否改了设计方向: 没改架构/边界/数据流。只是把 memo 键从「逻辑身份」加强为「逻辑身份 + 确切文本」,并把值从强引用改为弱引用(WeakRef + WeakMap 锚定)以不延长历史存活。属 bug 修复,非设计转向。

  8. 是否引入了新的配置: 无新配置/env/默认值。memo 上限 200K 不变,无迁移。

收工判定: 建议合并——根因修复正确、回归测试真实(base 上 15 红 / 2 绿)、CI 绿、diff 干净、行为变更已披露。待 2 个在跑 CI 检查通过 + owner 批准。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Token-estimate memo reuses stale counts for equivalent same-length Unicode tool arguments

2 participants