Skip to content

[PR] fix: pin client-managed cache marks so mid-history stays byte-stable (#6861) - #6869

Closed
sx4im wants to merge 1 commit into
ranxianglei:masterfrom
sx4im:fix-6861-claude-mid-history-rewrite
Closed

sx4im wants to merge 1 commit into
ranxianglei:masterfrom
sx4im:fix-6861-claude-mid-history-rewrite

Conversation

@sx4im

@sx4im sx4im commented Nov 15, 2026

Copy link
Copy Markdown
Contributor

Claude Code stamps a cache_control breakpoint on its last message and moves it as the conversation grows. bili harvested those marks per request and re-emitted only the current ones, so the message that carried the breakpoint last turn lost it and its bytes rewrote mid-history on every request. That is the seam the issue reports: prefix cache hit down to 8.7% on one request, 119K tokens re-billed.

What changed: client-managed breakpoints are now pinned cumulatively, the same discipline as bili's own marks. A marked message stays marked until it folds away, inside the 4-breakpoint budget. Proxy mode only; in plugin mode the host owns the rolling marks, so pinning there is skipped.

How verified: new regression test tests/issue6861-client-cache-marks-pinned.test.ts, 5/5 green (unit tests for pinning, fold retirement, budget eviction, and the no-cache-control knob, plus a two-turn rig test proving turn-1 messages serialize byte-identically on turn 2). Neighbor suites anthropic-cache-control and opencode-rolling-cache-marks pass 20/20. Typecheck clean in the changed files. Full suite verdict is remote CI.

Fixes #6861

…anxianglei#6861)

Claude Code stamps a cache_control breakpoint on its last message and
moves it as the conversation grows. anthropicToCore harvests those marks
into a per-request map that coreToAnthropic re-emits verbatim, so the
message that carried the breakpoint last turn loses it and its bytes
rewrite mid-history on every request, collapsing the prefix cache.

Pin client-managed breakpoints cumulatively, the same discipline as
bili's own marks: a marked message stays marked until it folds away,
within the 4-breakpoint budget. Proxy mode only; in plugin mode the host
owns the rolling marks.
@ranxianglei ranxianglei changed the title fix: pin client-managed cache marks so mid-history stays byte-stable (#6861) [PR] fix: pin client-managed cache marks so mid-history stays byte-stable (#6861) Nov 15, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Dedup: not a duplicate. This is the only open PR that pins client-managed message breakpoints. Related open PRs #4139/#4391 fix seam-detector classification (reading marker migration as append rather than mid-history) and #4234/#4232 are kernel system-message mark fixes — different mechanism and surface from this byte-stability fix.

PR #6869 review — pin client-managed cache marks (fixes #6861)

I pulled the head (753d7a3adf, single commit on current master f648217cb8), read the full diff + surrounding pipeline, and ran the suites locally.

Root cause (verified, independent of the issue's framing): real and correctly located. anthropicToCore harvests the client's cache_control into a per-request map keyed by canonical content-derived core ids; coreToAnthropic re-emits exactly that map. Claude Code moves its breakpoints (last message + last assistant) as the conversation grows (#2188), so the message that carried the mark last turn loses it and its bytes rewrite mid-history — the prefix breaks exactly there. That matches the issue's evidence (12/12 pairs diverged mid-history, one request at 8.7%, no fold/switch attribution). The fix targets the producer of the rewrite (cumulative pinning, the same discipline as bili's own marks, #1637) — a root-cause fix, not a mask.

Regression review (existing behavior on the touched path):

  • bili's own marks (non-client-managed sessions): untouched — anthropicCacheMarks ?? clientCacheControls still resolves to bili's marks.
  • Plugin mode: untouched — gated by !pluginMode; the host owns the rolling marks (opencode retouch "no accumulation" contract). opencode-rolling-cache-marks passes.
  • no-cache-control knob: pinClientMessageMarks clears stored pins and returns only current marks — consistent with computeAnthropicMessageMarks.
  • 4-breakpoint budget (WC-010): in a client-managed session bili does NOT stamp the system (stampAnthropicSystemCacheControl(systemOut, ours=false)) and bili-injected tools carry no cache_control (verified: 0 occurrences in compress-tool.ts/inject.ts), so messageCap = 4 - clientSystem - clientTools is exact. For any valid client request clientControls.size ≤ cap, and the eviction loop terminates with merged.size ≤ cap. No new upstream 400 introduced.
  • Round-2 rebuild ([bug] anthropic 线 proxy 模式:压缩回合的 round-2 重建丢失客户端 cache_control,折叠后下一条请求只能命中 system 前缀 #2499): consumes the same pinned map via anthropicClientCacheControls → rebuild stays byte-consistent with the steady path.
  • New session.metadata["anthropicClientCacheMarks"] key is additive (mirrors the existing anthropicCacheMarkIds pattern); old sessions simply lack it.

Code + tests: clean, well-named, comments explain the why. 4 unit tests (pinning, fold retirement, budget eviction order, knob) + 1 real two-turn rig test asserting turn-1 messages serialize byte-identically on turn 2 and mark indices [0,2] — the e2e has teeth (removing pinning flips it red). Diff is 3 files, +354 −7, every line on-purpose, no lockfile/whitespace churn, no version touch.

Local verification: typecheck clean, build OK, new suite 5/5, neighbors anthropic-cache-control + opencode-rolling-cache-marks 20/20. CI (live): all 54 check runs success/skipped, none failing; mergeable=true, merge_state=blocked (awaiting human review, not a CI failure).

Known boundary (non-blocking): sidecar messages (image blocks, redacted_thinking, tool_result with non-text content, unknown block types) ride rawAnthropicBlock and coreToAnthropic re-emits them verbatim without applying cc(id) — so a pinned mark on such a message is silently dropped and the seam persists there. This is pre-existing kernel behavior (not a regression from this PR); fixing it means a kernel/ change (human-gated, out of scope here). The reported repro (reading text files) is unaffected.

结构化八问(中文):

  1. 是什么问题: Claude Code 把 cache_control 断点盖在最后一条消息(和最后一条 assistant 消息)上,会话增长时断点会移动;bili 每个请求只采集并重发"当前"这批断点,导致上一轮带断点的那条消息在下一轮丢掉断点、字节被重写,前缀缓存恰好在那里断裂(单请求命中掉到 8.7%,119K token 重复计费)。
  2. 什么原因导致的: 根因是"客户端管理的断点按请求重发"这个机制本身——anthropicToCore 把客户端 cache_control 采集成每请求 map,coreToAnthropic 只重发这个 map;断点移动后旧消息字节被改写。这是字节重写的生产者,不是表面现象。
  3. 影响范围与严重程度: 仅 proxy 模式下客户端自管缓存的 Anthropic 会话(典型 bili claude);触发条件是客户端在消息上盖 cache_control 且断点随会话移动。严重度:前缀缓存命中率大幅下降、token 重复计费(成本问题),但功能可用、不丢数据。
  4. 怎么修复的: 把客户端管理的断点改成"累积固定"(与 bili 自己的断点同一纪律 [Bug] Anthropic lane never emits cache_control breakpoints — prompt cache can only ever cover the system head (~11K), the whole history re-bills at full price every request #1637)——一旦某条消息被客户端盖过断点,后续请求持续重发,直到该消息被折叠移出活视图;在 Anthropic 4 断点预算内(系统+工具+消息合计),超预算时优先驱逐"最新的仅固定"断点,保证接缝落在尾部而非早期元素。仅 proxy 模式;plugin 模式宿主自管滚动断点,跳过固定。
  5. 是否有回归问题: 逐项审过——①bili 自管断点路径不变;②plugin 模式不变(有 !pluginMode 门);③no-cache-control 旋钮:清除已存固定、只发当前断点,与 computeAnthropicMessageMarks 一致;④4 断点预算:客户端自管会话里 bili 不盖系统断点(ours=false)、注入工具不带 cache_control(已验证),messageCap=4-系统-工具 精确,不引入新 400;⑤round-2 重建器([bug] anthropic 线 proxy 模式:压缩回合的 round-2 重建丢失客户端 cache_control,折叠后下一条请求只能命中 system 前缀 #2499)消费同一份固定 map,与稳态路径字节一致。未发现回归。
  6. 是否破坏缓存: 正向修复——出站历史变为 append-stable,前缀缓存命中恢复;不破坏任何既有缓存/重试/去重策略。新增 session.metadata 键 anthropicClientCacheMarks(增量,旧会话无此键不受影响)。
  7. 是否改了设计方向: 否。沿用既有"累积断点"纪律([Bug] Anthropic lane never emits cache_control breakpoints — prompt cache can only ever cover the system head (~11K), the whole history re-bills at full price every request #1637)扩展到客户端自管断点,不改变架构/边界/数据流。
  8. 是否引入了新的配置: 否。复用既有 BILI_NO_CACHE_CONTROL / compat.noCacheControl 旋钮,无新 env/config/默认值。

合并判定: 方向正确(修根因非堵症状)、无回归、CI 全绿(54 checks)、本地 typecheck/build/新测试 5/5/邻接套件 20/20 通过、diff 干净(3 文件 +354 −7,每行相关)。属 wire 消息形状变更(出站 cache_control 落点),按 §7.4 必须人工合并、不可自动合并。建议:可合并(需人工)。 侧车消息断点被丢弃的边界建议作为后续 kernel 跟进(需 owner 批准 kernel 变更),不阻塞本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner

此 PR 的内容已整体进入批量 PR #7107(逐 commit 原样 cherry-pick,作者与 commit message 保留,已合并)。

This PR's changes have been merged verbatim (commit-by-commit, authorship preserved) via the batch PR #7107. Closing this to avoid a duplicate merge. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants