Skip to content

[PR] fix: cache-attribution root cause (OpenAI omitted cache fields + cache-diff byte offsets) - #6517

Closed
ranxianglei wants to merge 0 commit into
masterfrom
2026-11-13_cache-attribution-root-cause
Closed

ranxianglei wants to merge 0 commit into
masterfrom
2026-11-13_cache-attribution-root-cause

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Root-cause fix for the cache-attribution batch: two defects that both corrupt cache hit-rate attribution, fixed in one PR.

What

1. OpenAI routing: omitted cache fields are now a measured zero, not "unmeasured" (#6329)

sglang returns prompt_tokens_details: null (and OpenAI omits the field when cached_tokens = 0). The OpenAI adapter (src/loop/adapter-openai.ts) treated a missing field as "no data" (undefined → unmeasured), so those requests were excluded from hit-rate attribution entirely. Result: the UI session hit rate was pulled down but the attribution panel could not explain the gap (a 12.6pp unexplained miss in the reported case), and TTL/eviction cost was systematically underestimated.

Now: prompt_tokens present + both cache fields absent ⇒ cached = 0 (a measured miss, counted in the denominator). A usage object missing prompt_tokens stays unmeasured.

2. bili acp-cache diff: byte offsets now read in original-text coordinates (#6495)

acp-cache-diff compared a UTF-8 byte mismatch offset against UTF-16 JSON span indices and sliced evidence at that byte index. A long CJK/emoji prefix inflated the byte offset past the UTF-16 end of the last shared element, so a pre-array field rewrite (e.g. a system-prompt change) was misclassified as a healthy append, and the divergence messageIndex/evidence snippet were lost. textOffsetAtByte converts the mismatch byte to an original-text position (anchoring inside a multibyte code point to its start) and is used only for the span comparisons and evidence slicing; public byte offsets and byte-based attribution are unchanged.

Provenance

Both fixes were already prepared as separate PRs by external contributors; this PR cherry-picks them (original authorship preserved) so the root-cause batch closes in one merge:

Behavior change (intended)

  • OpenAI routing: a request whose upstream omits the cache fields now counts as cached = 0 in the hit-rate denominator (was: excluded as unmeasured). The headline hit-rate number is unchanged; the attribution panel's "unattributed backfill" now accounts for these requests (e.g. −0.4% → ~−12.9% in the reported case), and TTL/eviction seam detection now covers them.
  • acp-cache diff (offline diagnostic only): for multibyte-prefixed dumps, a genuinely pre-array divergence is now classified diverge (was: misreported append), preArray is restored, and divergence.messageIndex/evidence resolve correctly. Public byteOffset values are unchanged. No live compression/cache path is touched.

Verification

Targeted (local) — the full-suite verdict is the remote CI test lanes:

Fixes #6329
Fixes #6495

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-13_cache-attribution-root-cause (4fc5dfb)

Latest build — install this exact version

npm install -g billion-context@0.0.9517-pr.6517.5556042

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
4fc5dfb billion-context@0.0.9517-pr.6517.5556042 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6517.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei added bug Something isn't working p2 scope/wiring labels Nov 13, 2026
@ranxianglei ranxianglei changed the title fix: cache-attribution root cause (OpenAI omitted cache fields + cache-diff byte offsets) [PR] fix: cache-attribution root cause (OpenAI omitted cache fields + cache-diff byte offsets) Nov 13, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Progress update — my worker process stalled (~34 min, likely a hanging dependency install) and was restarted. Here's where the review of PR #6517 (mirror #6518) stands:

Code review — both fixes verified correct by reading the surrounding code:

Live state (just checked): PR head 4fc5dfb372 is behind current master; CI shows 12 guard/lint lanes green (kernel-guard, version-guard, config-surface-guard, …) and the 18 test/e2e/static lanes skipped (consistent with a behind-base branch).

Next: I'll do a bounded local run of the targeted suites (loop-core, acp-cache-diff, acp-cache-diff-unicode, golden-storage-render) and then post the full 8-question review. The authoritative full-suite verdict is the remote CI test lanes once the branch is current with master.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #6517 review — both fixes verified correct; rebase needed for full-suite green

I reviewed the code, ran the targeted suites locally (bounded), and verified the regression + golden claims.

Code review — both fixes correct:

Verification (local, bounded):

CI (live-checked): 12 guard/lint lanes green (kernel-guard, version-guard, config-surface-guard, docs-sync, unused-exports, pi-subagents-guard, test-import-graph, lifecycle-ledger-guard, plan, lane-guard, docs-coverage, lock-sync); the 18 test/e2e/static lanes are skipped because the branch is 18 commits behind current master (a0079746). 0 failed, 0 pending.

Verdict: code is correct and the fixes are sound. The branch must be rebased to current master so the CI test lanes run for the authoritative full-suite green before merge — the test lanes are currently skipped due to the behind-base state.


8问审核(中文,供人工 reviewer):

  1. 是什么问题:两个缺陷都破坏缓存命中率归因——① OpenAI Routing: Requests where the upstream omits cache fields are misrecorded as "unmeasured" and excluded from attribu | OpenAI 路由:上游省略缓存字段的请求被误记为"未测量"并剔除出归因,命中率与归因面板差 12.6pp 无法解释 #6329:OpenAI 路由中上游省略缓存字段(sglang 返回 prompt_tokens_details: null / OpenAI 在 cached_tokens=0 时省字段)时,适配器把"字段缺失"当"无数据"(unmeasured),这些请求被排除在命中率分母外,UI 命中率被拉低但归因面板解释不了缺口(报告案例 12.6pp 未解释 miss),TTL/驱逐成本被系统性低估;② cache-diff misclassifies Unicode-prefixed message rewrites as healthy appends #6495:acp-cache diff 拿 UTF-8 字节偏移量去比 UTF-16 span 索引并切证据,长 CJK/emoji 前缀把字节偏移膨胀到最后一个共享元素 UTF-16 末尾之后,数组前字段重写(如改 system-prompt)被误判成健康 append,diverge 的 messageIndex/证据丢失。

  2. 什么原因导致的:① openaiCachedTokens 原来只在缓存字段存在时返回数值、否则一律 null,没区分"上游明确报 0 缓存"和"上游没报缓存数据",两者都落到 unmeasured;② compareWires/classifyPair 直接用 UTF-8 字节偏移 O 做 span 比较和证据切分,但 JSON span 是 UTF-16 坐标,多字节字符字节数 > UTF-16 码元数导致偏移被膨胀。

  3. 影响范围与严重程度:① OpenAI Routing: Requests where the upstream omits cache fields are misrecorded as "unmeasured" and excluded from attribu | OpenAI 路由:上游省略缓存字段的请求被误记为"未测量"并剔除出归因,命中率与归因面板差 12.6pp 无法解释 #6329 影响所有走 OpenAI 路由且上游省缓存字段的请求——归因数据不完整(命中率低估、TTL/驱逐成本低估),但不影响压缩/缓存的实际行为,纯观测/归因层,p2(能用但归因面板不可信);② cache-diff misclassifies Unicode-prefixed message rewrites as healthy appends #6495 只影响 acp-cache diff 离线诊断在多字节前缀 dump 上的分类,不在任何在线压缩/缓存路径上,p3(离线诊断、边缘场景)。整体 p2(与 PR 标签一致)。

  4. 怎么修复的:① openaiCachedTokens 加一条判定——prompt_tokens 存在 + 两个缓存字段都缺 ⇒ 返回 0(measured miss,计入分母);prompt_tokens 缺 ⇒ 仍 null(unmeasured)。下游 recordUsage 用 typeof cached === "number" 归一化,0→measured zero、null/undefined→unmeasured。② 新增 textOffsetAtByte(body, offset) 把不匹配字节偏移转成原文坐标(锚定到多字节码元起始),只用于 span 比较与证据切分;公开字节偏移与字节归因不变。

  5. 是否有回归问题:审过的既有行为——OpenAI 路由:缓存字段存在时行为不变(仍返回字段值)、prompt_tokens 缺失时仍 unmeasured,只有"字段省略"从 unmeasured 变 measured zero(有意变更,PR 已披露);acp-cache-diff:ASCII 路径不受影响(字节==UTF-16)、健康 append 路径不变([Enhancement] bili acp-cache diff <dump-dir>: prefix-diff attribution over ACP_DUMP_BODY dumps (from #1254 ①) #1266 测试钉住)、公开 byteOffset 不变。高危模式逐项过:无 fail-fast 变吞错、无阈值/时序/输出格式漂移。回归测试 OpenAI Routing: Requests where the upstream omits cache fields are misrecorded as "unmeasured" and excluded from attribu | OpenAI 路由:上游省略缓存字段的请求被误记为"未测量"并剔除出归因,命中率与归因面板差 12.6pp 无法解释 #6329 无修复时失败(cacheSamples 0 vs 1)、有修复时通过;黄金 chat.json 变化已验证为纯 cachedTokens(state/blockContents digest 不变,无存储变更)。

  6. 是否破坏缓存:不涉及。两个修复都不触碰缓存/重试/增量游标/去重键的失效策略——OpenAI Routing: Requests where the upstream omits cache fields are misrecorded as "unmeasured" and excluded from attribu | OpenAI 路由:上游省略缓存字段的请求被误记为"未测量"并剔除出归因,命中率与归因面板差 12.6pp 无法解释 #6329 只影响归因观测(命中率计算),cache-diff misclassifies Unicode-prefixed message rewrites as healthy appends #6495 是离线诊断工具,均不在在线缓存路径上。

  7. 是否改了设计方向:不涉及。两个都是缺陷修复,不改架构/边界/数据流;OpenAI Routing: Requests where the upstream omits cache fields are misrecorded as "unmeasured" and excluded from attribu | OpenAI 路由:上游省略缓存字段的请求被误记为"未测量"并剔除出归因,命中率与归因面板差 12.6pp 无法解释 #6329 遵循既有 measured/unmeasured 归因契约(recordUsage 的 typeof 归一化),cache-diff misclassifies Unicode-prefixed message rewrites as healthy appends #6495 遵循"公开字节偏移量不变"的既有契约。

  8. 是否引入了新的配置:不涉及。无新增 env/config/默认值。

合并判定:代码正确、回归测试有效、黄金变更已验证、typecheck 干净、4 个定向套件全绿(63 测试 0 失败)、CI guard 车道全绿。唯一阻碍:分支落后 master 18 个提交,CI 的 18 个 test/e2e/static 车道被 skip——需 rebase 到当前 master 让 CI test 车道跑全量回归后才能给出权威 green 判定。代码可合(pending rebase + CI 全量绿)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Resolved the merge conflict. Key finding: this PR is fully redundant — every line of it is already in master, so after rebasing it had zero net-new content and GitHub closed it as a consequence (head became identical to base, nothing to merge). No code was changed by me.

What happened:

七问:

  1. 是什么问题:PR [PR] fix: cache-attribution root cause (OpenAI omitted cache fields + cache-diff byte offsets) #6517 与 master 冲突(merge state: dirty),CI 自动修复车道被唤醒解冲突。
  2. 什么原因导致的:该 PR 是「cache-attribution root-cause 批量」,只 cherry-pick 了两个早已单独合入 master 的 PR([PR] fix: OpenAI cache attribution — treat omitted cache fields as measured zero #6340 openai 省略缓存字段→记 0、[PR] fix: interpret cache-diff byte offsets in original text coordinates #6496 cache-diff 字节偏移)。两者的代码、回归测试、storage-render golden 重新生成(master 7402629)都已在 master,PR 相对 master 零 net-new 内容;rebase 时唯一文本冲突是 golden chat.json(PR 侧 digest 是旧 base 243709b 的过期值)。
  3. 影响范围与严重程度:无用户影响——PR 无任何新代码/新测试/新配置,全部改动都已在 master 生效。严重程度:无(纯冗余 PR)。
  4. 怎么修复的:PR 分支 rebase 到 master;rebase 自动跳过 4fc5dfb([PR] fix: interpret cache-diff byte offsets in original text coordinates #6496 已在 master),f0ba6e15c8 的代码改动也已在 master,唯一冲突 chat.json 取 master 侧值(master 已在 7402629 为同一改动重新生成)。解完分支与 master 完全一致,force-push 后 GitHub 因 head=base 无内容可合自动关闭。
  5. 是否有回归问题:无。未改任何代码(改动都已在 master);golden 取 master 现值,与 master CI 一致。
  6. 是否破坏缓存:不涉及(无代码/缓存/重试/游标逻辑变更)。
  7. 是否改了设计方向 / 引入新配置:都不涉及。

说明:force-push 是任务明确授权的冲突 rebase 操作;PR 的关闭是 head 被推到与 base 一致后 GitHub 的自然结果(无内容可合),我没有执行任何显式 close 动作。该 PR 的内容全部已在 master,无需后续处理,建议 owner 确认关闭即可。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

1 participant