Skip to content

[Cache miss] 0% prompt-cache hit on plain conversational turns — 1.27M tok stable-prefix re-bill with no attributable cause (v0.1.204, #1195) #4397

Open

Description

@raeglt

Summary

A brand-new session whose visible content was a single greeting ("你好你好,大聪明") plus one assistant reply was billed 36,155 input tokens on turn 3, and across the first 20 turns the prompt cache hit rate was 0.0% — every request re-billed the entire stable prefix even when the real new content was a few hundred tokens.

acp_cache reports the grand ledger for that session (session-46f79a03-b861-4bda-88c0-0ad1d6f221f0, 34 requests, bili 0.1.204):

metric value
total input 2.06M tok
total cached 614.5K tok (hit 29.9%)
new content 127.1K tok
compress re-pay 47.5K tok
upstream-ttl-or-client-rewrite (unattributed) 1.27M tok

CACHE INVALIDATION attributes 0 tok to every switch cause it can fingerprint (model / effort / key / prompt rewrite / projection / wire / upstream / restart-refork), i.e. 100% of the stable-prefix re-bill is unattributed. The report concludes it is "expected provider-side behaviour (cache TTL expiry / eviction / relay rotation), NOT a bili bug" and points at #1195 coverage-mismatch.

BODY STABILITY is the part that argues against a provider-eviction-only explanation: 33 pairs, 0 byte-identical, 33 diverged — 2 × mid-history and 31 × tail(params/notes). For the tail divergences bili's own message list is byte-stable ("shared history bytes untouched"), so the provider re-partitioned its cache on a trailing field alone.

Two upstream pointers from the same report:

ABORT-CORRELATED flags 33 of 33 missed samples within 30s of a client mid-stream abort, but hit samples carry the same abort, so that correlation is not discriminating in this data.

Environment

  • bili 0.1.204 (commit c87185a0), running as the DSH billion-context host plugin.
  • OS: Windows; config resolved from ~/.config/billion-context/billion-context.json (XDG vars unset).
  • Upstream: forward POST → http://<private-host>:63882/v1/chat/completions — a local relay, constant host and port across the whole session, so not a relay-rotation signature at the TCP level.
  • Model: workbuddy/hy4-preview-f (reasoningEffort: high), no model switches observed.
  • The session also saw [tag-echo] stripped model-emitted render tag (plugin passthrough): <acp tokens="18" type="text"> — the model itself is emitting ACP-shaped tags, which is adjacent to the render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 class.

What made it better

Writing {"providers":{},"diagnostics":{"renderNone":true}} took effect in 77 ms (first processTurn: … renderTags=none right after the config mtime) and the hit rate recovered:

# time input cached hit%
1–20 16:20–16:36 — — 0% (except #18 79.0%)
27 16:41:13 63.5K 61.3K 96.6%
29 16:43:00 66.6K 63.4K 95.2%
30 16:43:31 66.8K 66.6K 99.6%
32 16:44:52 69.8K 66.8K 95.7%
33 16:46:13 70.2K 69.8K 99.4%
34 16:47:05 73.3K 70.1K 95.8%

So render tags were a real contributor, and the mitigation works — but residual 0% turns remain (#24, #28, #31), and #24 is reported as a CACHE SEAM:

#24 16:38:48 hit 0.0% · input 47.9K · divergence ≥23.7KB at message[1] of 15→17
  1 sample(s) · 47.8K tok re-billed with no fold/switch/restart attribution

i.e. a genuine mid-history rewrite that no fingerprint cause explains, suspected #1548-family round-2/steady render seam.

Why this still costs real money

Even with the mitigation, one 0% turn at ~70K input is the whole prefix re-billed. During the 0% stretch, acp_status showed 1.4K visible text / 0 tool against 36,155 billed input — the bill was ~25× the visible conversation, and ~72K tok of that stretch was pure re-pay.

The base size is legitimate (DSH injects system prompt + ~90 tool JSON schemas, 30K+ tok of tool definitions alone); the defect is that this base is re-billed every turn.

Request

  1. Make the tail-params divergence non-cache-breaking: if only trailing top-level params / tail notes change, confirm the provider actually re-partitions and, if so, have bili keep them out of the cacheable prefix (or pin them).
  2. Ship per-route render-tags control so renderNone is not a global all-or-nothing switch (it currently disables ref-based compression, opencode 插件模式下 render tag 注入消息正文,模型回显漏网后原样渲染进 TUI(元数据与正文同层问题) #933/render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483).
  3. Explain / eliminate the residual 0% turns and the #1548-family seam at message[1] with no attribution.
  4. Consider making CACHE INVALIDATION able to distinguish "provider eviction" from "bili rewrote a byte it does not account for" — right now 1.27M tok lands in one unattributed bucket and the report self-certifies "not a bili bug".

Notes for reproduction

diagnostics.dumpReq=true is already on and the raw dumps for this session exist under <state dir>/raw, so the exact outbound bytes for the 0% turns (#24, #28, #31) and the seam are available if you want them attached.

Related: #1195, #2483, #3803, #1548, #2867, #2350, #933.

Activity

  1. ranxianglei commented on Oct 23, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重:不重复。 open+closed issue/PR 全检,无条目覆盖本组合。最近的邻居都已关闭且机制各异:

    Triage 三步结论(按 v0.1.204 tag 代码核对,非 master):

    1. 复现/查证:数据自洽——BODY STABILITY(31×tail / 2×mid / 0 identical) + renderNone 77ms 生效即恢复 95–99% + 残余 3 个 0% 回合,互相印证;成本影响真实(~25× 计费)。缺陷成立。
    2. 层次判断:主因是已知机制而非 bili 改字节。你的 local relay 属 render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 类:relay 逐字校验客户端回传历史 vs 自己存的 transcript,bili 加在 assistant 文本上的 render tags 触发校验失败 → relay 每轮当新会话 → 0%。证据链:① bili 自己的消息列表跨轮字节稳定(BODY STABILITY 证明——若 bili 改字节,分叉应在 mid-history 而不是 tail);② 翻转后尾部字段(trailing user note / nudge marker)照常每轮换,命中却回到 95–99% → 该 relay 不因尾部字段单独重分区;③ 翻转前后唯一系统性差异就是 render tags。
    3. 方案评估:四个请求方向基本对,但分属四层,分开处置(见下)。

    标签:bug / dsh / scope/wiring / p1 —— 20 轮持续 0% 命中 = 成本爆炸(owner p1 口径「缓存命中崩坏导致成本爆炸」);面在 wire 渲染+归因报告;宿主车道 DSH。

    分 facet 结论:

    请求 1(tail params 非破坏化):你自己的数据已给出答案——此 relay 不吃这一口。 翻转前 31/33 对只分叉在 tail(params/notes),翻转后同样的尾部字段照常每轮变化、命中却回到 95–99%,说明该 relay 的 cache key 不含尾部字段,bili 侧无需 pinning。我会把这条实证补进 #2483 缓解说明。想要直接证据的话,翻转后跑一次 /acp-cache detail:"full" 看 tail 类配对数即可。

    请求 3(残余 #24/#28/#31):需要你提供的 dump 才能定案。 #24 签名(divergence ≥23.7KB @ message[1] of 15→17)是真·mid-history rewrite,两个候选来源:

    请求 4(归因拆分):部分已存在,剩一个真缺口,现在修(无决策依赖)。 v0.1.204 的 SEAM 段其实已分 providerSide/client/TTL/seam-suspect 桶(你引用的 #2483 advisory 就打印在那里),但 grand-total 行把所有东西并进 upstream-ttl-or-client-rewrite (unattributed) 一行——所以标题读起来「100% unattributed」而明细段其实解释了一大半。修法:grand-total 行按子桶拆出各自 token 数;另当 renderTags≠none ∧ body 字节稳定 ∧ 全前缀 miss(#2483 签名)时给一个命名计数「render-tag relay suspect」。纯观测层、零出站字节变化、双向测试。PR 出来我按八问回复带链接。

    请求 2(per-route render-tags control):新配置面,等 owner 拍板。 现状 renderNone 全局二值(env ACP_RENDER_NONE > diagnostics.renderNone > false),一开全进程失去 ref 渲染。两个选项:

    A:providers 表加字段(推荐) B:按上游地址匹配
    形状 现有 providers[<name>] 条目加 renderNone?: boolean(先例:#1469 compactionOptIn 落 providers 表) 新增 diagnostics.renderNoneRoutes: ["<host>:<port>", ...] 模式列表
    解析序 env(全局急停,保留)> providers[name].renderNone > diagnostics.renderNone > false env > route 匹配 > diagnostics.renderNone > false
    影响面(Q3) 只影响路由到该 provider 条目的流量;默认关、未配置零行为变化;粒度=provider 条目,正对「我的 relay 是 stateful、别家不是」场景 同语义 + 模式匹配面;typo 永不命中 = J 类静默配置失效风险;host:port 本身不稳定(relay 换端口配置即废)
    回归风险(Q5) 混模式会话切 provider 时一次 replay(与今天全局翻转的一次 replay 同性质,已文档化) 同 A + 模式匹配边界 case
    迁移(Q8) 无(新增可选字段,默认 false) 无(默认空列表)

    推荐 A:映射到既有 providers 表、最小面(一个布尔)、语义与今天的全局开关同构只是作用域收窄;B 若将来确需更细粒度可在 A 之上扩展,不必现在发明新寻址方案。未获你签批前不动手(配置面纪律)。

    状态:请求 4 的 PR 进行中;请求 3 等 dump;请求 2 等拍板(荐 A);请求 1 已定论无需代码。

    另:平台侧小记——我在打标签时建了一个探测用 label zzz-probe-label(id 62),该 fork 的 API 无删除端点(DELETE/PATCH 均 404),未挂任何 issue,请 owner/ops 手动清一下。

  2. ranxianglei commented on Oct 23, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    请求 4 已实现 — PR #4446:#4446 跑完前状态以远端为准)

    改动全部在宿主观测层,零出站字节变化、零配置面、kernel 未动:

    1. CACHE INVALIDATION 对账行:未归因残差不再是一口黑箱——新增子行把它按 body forensics 分桶列出(provider-side miss / provider TTL expiry / client mid-history rewrite / history rewind / orphan cleanup sweep / suspected mid-history break),并显式标出低于取证门槛或无 body 捕获的余量。各桶按构造是残差的子集(具名 cause 的样本根本到不了 detectSeam),所以这行永远能对账。你的 1.27M 在下次报告里就会拆成「provider 侧 N tok · 疑似中缝 M tok · …」而不是一个桶。
    2. 警告措辞精化:当「疑似中缝」占残差 ≥10% 时,原来那句一刀切的 NOT a bili bug 换成拆分表述(指向 CACHE SEAM 段 + 附 raw dump 的请求);无嫌疑时措辞与旧版逐字节一致(cache-ledger 归因账本:同一残差重复计入多个重叠桶、switch flag 只比紧邻上一条、未归因残差占绝对主导、unknown-cache 样本被画成 0% 平台 #1847 测试继续钉住旧文案)。
    3. render-tags 实锤计数(render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 签名):两个发送 chokepoint 现在记录该 body 发出时 render tags 是否激活(稀疏 rt 标志,seam slot → ledger line);带此标志的 provider-side miss 计入新的 taggedSide 子集,在 PROVIDER-SIDE MISS 段显示「of which N sample(s) · M tok were sent WITH bili render tags active」。你这种「字节稳定却 0% 命中、翻 renderNone 后恢复」的场景从此是测出来的数据,不再是怀疑。

    行为变更披露(仅报告文本):新行只在对应桶非空时出现,无取证的会话输出逐字节不变;警告措辞仅在嫌疑实质存在时改变;新增两个持久化 agg 键,旧账本加载时归一化为 0(有测试)。

    测试:新 tests/issue4397-invalidation-reconciliation.test.ts 6 例(#3617 双向断言:三个桶形状各自正例 + tag 标记正反 + 警告精化正反 + 旧文案钉住 + 旧账本兼容);cache/seam 全族定向回归本地 213/213 绿,typecheck+build 干净,全量判定交给 CI。


    仍在等的两件事:

  3. ranxianglei commented on Oct 23, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    R4(PR #4446)CI 全绿,收工判定如下。 PR 状态:open / mergeable / behind(behind 的原因仅是 master 期间合入了 v0.1.205 发版提交——git diff fd973ca7d..origin/master 只有 package.json/package-lock.json 两行版本号,与本 PR 零交集,故不 rebase、不重跑全量矩阵)。


    结构化八问(整单)

    1. 是什么问题:DSH 会话里,出站 body 逐字节稳定的情况下,稳定前缀每轮被重新计费(前 ~20 轮 0% prompt-cache 命中;全账本 1.27M tok 落进「unattributed」一个黑箱桶,且账本警告在没有任何可观测原因时直接宣布「NOT a bili bug」)。
    2. 什么原因导致的:主因是有状态的本地 relay(claude-code 系,如 magpie)按自己的 transcript 校验回声历史;bili 的 render tags 出现在 assistant 文本上时校验失败 → 整历史缓存 miss(render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 家族,已有 closed 先例)。证据链:BODY STABILITY 33/33 仅尾部(params/notes)分歧 + 翻 renderNone 后尾部照样逐轮变化却拿到 95–99% 命中(证明 relay 缓存键不含尾字段,也证明触发物只能是渲染差异)+ 日志里 model 自身回显 ACP tag。次因(本次已修):账本把 body forensics 的全部结果揉进一个未归因桶,且警告一刀切——嫌疑中缝实质存在时仍说「不是 bili bug」。
    3. 影响范围与严重程度:① 有状态 relay + 默认 render tags 的用户面:稳定前缀每轮全量重计费 = 成本爆炸(owner 口径 p1「缓存命中崩坏导致成本爆炸」);绕行 = 全局 diagnostics.renderNone=true(代价:ref 式压缩渲染整体关闭)。② 账本缺陷面:所有读 CACHE INVALIDATION 的人被误导归因,体验/诊断瑕疵级。
    4. 怎么修复的:R4 走观测层拆分(PR [PR] fix: split unattributed cache-miss residual over forensics buckets and tag render-tag provider misses (#4397) #4446,纯报告文本,零出站字节):a) 未归因残差按 forensics 分桶对账(provider-side / TTL / client rewrite / rewind / orphan-sweep / suspect + 低于门槛余量);b) 嫌疑中缝占残差 ≥10% 时警告改为拆分表述并指向 raw-dump 请求;c) provider-side miss 增加「发送时 render tags 是否激活」实锤计数(taggedSide,render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 签名从怀疑变数据)。R1(尾字段打破缓存):结论是伪问题——你这条 relay 上翻 renderNone 后尾字段仍逐轮变却高命中,证明其缓存键排除尾字段,无需改码(实证会补进 render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 缓解文档语境)。R2/R3 见下。
    5. 是否有回归问题:无。出站字节零改动(两个 chokepoint 只多记一个稀疏标志位,不进 wire);报告新行全部条件出现,无取证的会话输出逐字节不变;旧警告措辞在无嫌疑时逐字节保留(cache-ledger 归因账本:同一残差重复计入多个重叠桶、switch flag 只比紧邻上一条、未归因残差占绝对主导、unknown-cache 样本被画成 0% 平台 #1847 测试继续钉住);新增两个持久化 agg 键对旧账本加载归一化为 0(有测试)。高危模式逐项过:无语义漂移、无 fail-fast→吞错、无阈值/时序/输出格式漂移(除上述已披露的报告文本)、缓存策略本身未动。
    6. 是否破坏缓存:不涉及——不改任何出站内容、不动重试/游标/去重;本地账本文件只增两个键(additive)。
    7. 是否改了设计方向:不涉及——完全沿用现有 seam forensics 架构,只加子集计数与报告行;kernel 未动(kernel 侧 grand-ledger 行文案属 human-gated 区,如需同步措辞另行走 kernel 流程)。
    8. 是否引入了新的配置:本 PR 无。R2 提案(待拍板)是 Option A:providers[<name>].renderNone?: boolean,解析序 env > providers[name].renderNone > diagnostics.renderNone > false,默认不变、纯 opt-in,先例 One-shot: named providers as real lanes — bind + compactionOptIn + agent-side switch, landing in a single PR (replaces #1466 staging) #1469 compactionOptIn;拍板后我开独立小 PR(含 CONFIGURATION.md en+zh 与双向测试)。

    当前状态与等待项

    issue 保持打开,直到 R2 落地 + R3 定性完成。

  4. raeglt commented on Oct 24, 2026

    @raeglt
    Author

    确认一下身份:我是 reporter,账号 raeglt,R2 我没有拍板权,只从使用侧表态。

    R2 我倾向 Option A:providers[].renderNone?: boolean,解析序 env > providers[name].renderNone > diagnostics.renderNone > false。理由是 opt-in 且默认不变,对存量用户零风险,跟 #1469 的 compactionOptIn 一个路子。Option B 的 typo 静默失效(J 类)和 host:port 不稳定你都已经识别到了,A 没什么可犹豫的。

    R3 我来准备 dump。 顺便提醒一句:dumpBody 默认 false 必须显式开,而且 maskHeadersForLog 只覆盖 header 和 url,请求体是裸的——我这边的日志里出现过凭据原文。这点建议写进 issue 的取证步骤,免得别人照着踩。

    另外先给一条不用等 dump 的证据,来自同一份 bili.log:

    代码块

    [fold-reconcile] resent history drifted: 250 covered id(s) missing —
    reanchored 250 (0 positional, 9 order break(s) resynced, 0 by toolCallId,
    0 by normalized identity, 235 by assistant-turn key) onto churned bytes (~56699 tok unfolded in churn region)
    注意 235/250 是靠 assistant-turn key 重新锚定的,positional / toolCallId / normalized identity 三条路全部为 0。这说明 drift 不是零散漂移,而是成片的、集中在 assistant 回合。这个分布更像你的 H2(bili rebuild 路径的稳态缝,#1548 家族),不太像 H1(宿主侧重渲染大块静态 blob,#3799 同类)——如果 H1 成立,按说 normalized identity 应该能捞回一部分。

    还有一条配套的:同一次会话里 system prompt changed (11360 → 11361 chars),只差 1 个字符,fold state 未受影响。1 字符的系统提示词变动也会被算进 churn,这个量级值得单独看一眼。

  5. ranxianglei commented on Oct 24, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    三件事:先一条时效性安全提醒,然后你给的两条证据我都对着 master 代码核实了——解读比你的判断还要再进一步:H1 和经典 H2 都被这条日志排除,指向第三个假设。

    ⚠️ 贴 dump 之前请先清洗(时效性)

    你的凭据观察在代码层面确认了:开 ACP_DUMP_BODY=1/diagnostics.dumpBody=true 时,raw/*-REQ.txt(src/server/relay.ts:534-541)与 raw/*-INCOMING.txt(src/server/handle.ts:2545-2553)对 header 和 URL 脱敏,但请求体原文直接落盘、零处理;同类还有 dumps/req-*.json 与 req-*-REREQUEST.json。这些文件里已经出现过凭据原文的话——贴到本线程之前请先 grep 掉 token/key 模式,或轮换已暴露的凭据,否则附件一贴凭据就公开了。已单独立项 #4482(dump 文件头 banner + CONFIGURATION.md en/zh 警告 + 取证流程固定清洗步;不建议自动 redaction,会毁掉 byte-diff 的取证价值)。你说的「写进取证步骤」并入了那条。

    fold-reconcile 证据:核实完毕,解读再进一步

    日志行出自 src/fold-reconcile.ts:2015,每条重锚路径我逐个对了实现:

    • byTurn 不是结构兜底,是严格内容验证(:1008-1012):整回合文本(reasoning+text 拼接)归一化后精确相等,带长度守卫,从不模糊。所以 235/250 意味着这 235 个 assistant 回合的全文在 NFC/空白归一化后原样存活——内容没变,变的是「形状」。
    • 因此 H1(宿主侧重渲染大块静态 blob,[cache][dsh] latest catalog loses BLOCKED protection and retroactively gains ACP tag, rewriting cached history #3799 家族)出局:blob 重渲染改的是内容(数字标签漂移能穿过归一化),不会产生「整回合文本精确匹配」的分布。你的直觉对,且理由比「normalized identity 应该捞回一部分」更硬。
    • 同时经典 H2(bili 改写 assistant 字节)也被削弱:若 bili 在改写字节(比如剥 tag),归一化后的整回合文本同样对不上,byTurn 也会是 0。
    • 这个分布(positional=0 ∧ toolCallId=0 ∧ byNorm=0 ∧ byTurn=235 ∧ 9 处 order break resynced)实际刻画的是:单核身份(role/contentType/toolCallId/单核文本)全失败、而整回合文本全匹配 → 同一对话、同样的 assistant 文本,两次发送的消息分解与 id 不同。这是**宿主侧 transcript 重建/重投影(H3)**的签名:例如 DSH session resume/replay 从盘上重建历史,id 全部重铸、thinking+text 分块被重新分解(split↔merge)。plugin 模式下 bili 对客户端历史逐字透传,锚点本身也源自客户端自己早前的重发——分歧源头在客户端侧,不在 bili。
    • 与持续 0% 的衔接:如果这个会话经历过一次宿主侧重放(resume),那么从第一个请求起回声形式 ≠ relay 存储的 live 形式 → stateful relay 校验(render tags on echoed assistant text break session continuity on stateful upstreams (magpie / Claude Code), causing full-history cache misses #2483 家族)每轮失败 → 持续 0%,与你「前 20 轮全 0」吻合;renderNone 翻回恢复则提示重投影过程恰好动了 render tag 字节——这点要 dump 才能钉死。另外 ~56.7K tok unfolded 是真实成本:drift 区里本轮未被 claim(重)覆盖的质量(含未被折叠覆盖的变更消息),即这一趟全量重发的量。

    细化后的 dump 请求(turns #24/#28/#31 + 各自相邻上一轮):

    1. 每个 assistant 回合在相邻 INCOMING dump 间的核心分解表(role/contentType/id 序列)——直接检验 H3;
    2. ACP tag 在 assistant 文本中的存在性/位置,drift episode 起点前后对比;
    3. drift episode 起点时间戳 vs bili.log 里 DSH 进程重启/session resume 标记;
    4. 1 字符 system 变动发生在第几轮 + 该轮 INCOMING(客户端原文)vs REQ(实际出站)的 system 字段 diff。

    1 字符 system prompt 变动

    那行是设计内的可观测注记(#1921,src/fold-reconcile.ts:2062-2076):system 块不参与消息身份,fold state 按构造不受影响(你的读法对)。缓存影响:index 0 变化只使紧接着的一个请求的前缀缓存失效——一次性事件 ≈ 一轮重计费,解释不了持续 0%(翻 renderNone 后命中恢复也支持它是一次性的)。来源有两个候选,判别手段代码里已有(#2867):宿主重写 vs bili 自身 mode/projection 切换(self-heal degrade-to-proxy 会改变转发投影)——diff 该轮 INCOMING(客户端原文)与 REQ(出站)的 system 字段,并在该时间戳附近找 self-heal/degrade 日志行:客户端原文就变了 1 字符 → 宿主侧(很可能是 DSH 合成 system 里的计数器/时间戳位数);客户端原文不变、只有出站变了 → bili 投影侧,我来追。

    R2

    你对 Option A 的支持我记录在案(与我的推荐一致)。拍板权确实在 owner——选项表维持上一条评论,批准后我立刻实现。

  6. 7 remaining items

  7. lanxingyun commented on Nov 15, 2026

    @lanxingyun

    Follow-up on the byte-level evidence request from #6543 (comment 6875760851, issue subsequently closed as completed before this could be discussed), filed here because this is the open thread whose title matches the shape exactly: a stable-prefix re-bill with no attributable cause.

    Two byte-level samples, answering "top-level param or transport-level re-partition?"

    Method: diagnostics.dumpBody=true writes the full forwarded request body to dumps/req-*.json; I diffed every adjacent pair (829 files / 446 MB) and joined against [acp-usage] rows. Detector: previous request's messages array preserved element-wise by JSON.stringify equality, uncapped; plus ledger cached dropping while input grows.

    ts re-billed hit% prefix bytes message prefix top-level params
    2026-11-14T15:46:40.687 175,744 tok 2.2% 559,562 B byte-identical all identical
    2026-11-14T16:21:33.621 119,040 tok 14.7% 411,414 B byte-identical only output_config.effort default->high

    Sample 2 is the EFFORT SWITCH shape from #6543/#3803: the only difference is a trailing top-level param, message prefix untouched — that mechanism is now byte-confirmed.

    Sample 1 is the unattributed one. No top-level parameter change, no route change, no model/effort/key switch, message list byte-identical, yet the provider re-billed 175,744 tokens of a prefix it had just cached (98.5% -> 2.2%). On this evidence the answer is transport-level re-partition / eviction, not a top-level parameter.

    Client-side ordering (correlation, not causation): the request immediately before sample 1's was aborted client-side 2.3 s earlier (client disconnected mid-stream / client aborted mid-stream at 15:46:26.175/.180); sample 1's own stream was aborted 17 ms after its usage row. The next request recovered to 93.5%. Since the aborted predecessor itself had hit 98.5%, "aborted => re-bill" is not universal, but "a cut response never commits the new prefix to the provider cache" would explain a same-param, same-route re-bill that no switch explains. Worth an instrumented repro if that is testable on your side.

    Artifacts (hashes only; no conversation body attached)

    The four dumps are still on disk. SHA256 for independent verification:

    req-1794642362324-...json 661,083 B 8ACCD0C17F7CEFFFD101E53330A8BF77BD50A957B6DE18746376C5B1C070F275 (sample 1, previous)
    req-1794642388482-...json 695,132 B A394D6B25EDD416344F37D3D19942A5202CF4A8401F05C1AC68268F758672BCD (sample 1, re-billed)
    req-1794644470974-...json 527,746 B 8CE5F3E660CCC38D46DE619EC16A60D9295D8BE72B9749362599E4ED43FB6AAB (sample 2, previous)
    req-1794644485538-...json 530,120 B 6659BD8875C21851E77CE479146A399F2CE443EB19A0F16E12F270D0B9AB6B47 (sample 2, re-billed)
    

    Machine-readable evidence (hashes, byte counts, ledger rows, param diffs — no body text) exists as two JSON files on this host; say the word and I will attach them.

    Two tooling notes from collecting this: bili acp-cache diff looks for *-REQ.txt/*-INCOMING.txt and reports "no ACP_DUMP_BODY dumps found" against dumps/req-*.json (the bodies it wants are in raw/, retention ~6.8 min at this capture rate); the full-body retention that made this possible is dumps/req-*.json (oldest ~5 h), so byte evidence for a provider-side miss is only available for a few hours — likely why the earlier 4 samples could not be backfilled.

    Config unchanged: dumpBody=true, dumpReq=true, maxTotalBytes=1073741824, maxAgeDays=2; renderNone remains off.

    Questions: (1) does the "aborted response never commits the prefix" ordering survive a targeted repro/test on your side, or is there a store-side signal that already proves/disproves it? (2) if it does not reproduce, is transport-level re-partition the accepted explanation and this can stay a note rather than an action item?

  8. ranxianglei commented on Nov 15, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Thanks — both samples land cleanly. Answers to your two questions, then the tooling note:

    Sample 2 (effort switch) — byte-confirmed, and it is a named bucket now. The #3803 effort dimension (PR #3809, shipped in v0.1.204) fingerprints the top-level body fields output_config, reasoning_effort, and reasoning.effort — wire-agnostic, so an output_config.effort default→high flip is exactly what it catches. On any build carrying that dimension this sample should attribute to effort, not sit in the unattributed residual. If your ledger still shows it unattributed, send the version + that ledger line and I'll check coverage.

    Sample 1 (the unattributed one) — Q1: no, the commit semantics are not testable from our side; the store-side signal already exists, and it's correlation by design. Whether a provider/relay commits a cache entry for a response that gets cut mid-stream is inside the provider — nothing in bili can instrument or reproduce that behavior. What the ledger already records is the ordering you observed: samples settled within 30 s of a client mid-stream abort get the abortedNear flag and are reported in the seam section as "abort/retry churn … correlation, not causation". Your 2.3 s abort→re-bill gap is precisely the shape that flag captures.

    One technical observation on the numbers, though: a near-total miss (98.5% → 2.2%) is not what "the aborted write never committed the new prefix" predicts on its own. Prefix matching is longest-prefix — if only P's incremental write were missing, S would still hit the previously committed entry for most of the prefix and land high, not at 2.2%. A 2.2% hit means the entire cached namespace was unavailable to that request — transport-level discontinuity: relay session/node rotation, eviction, or (for stateful relays, the #2483 class) the abort killing the relay session that carries the cache context, with S landing on a fresh session. Which of those it is is provider-side and not distinguishable from the outside; what is testable on our side is only that the ledger attributes this shape correctly (provider-side miss + abort-correlated), which is already the case.

    Q2: yes — transport-level re-partition/eviction is the accepted classification for this shape (the ledger's "provider-side miss: eviction / relay node rotation — not a bili rebuild seam" bucket). Sample 1 stays a note, not an action item.

    Your tooling note is a confirmed real gap — filed as #6787. Verified against current master: acp-cache diff's collector only matches *-INCOMING.txt / *-REQ.txt filenames, and its parser additionally requires the request-line + headers + blank line + body layout — so dumps/req-*.json (banner + pure JSON body, no headers) is invisible to it both by name and by format. That is exactly why the only dump format with hours of retention is the one the offline forensics tool cannot consume, and why your earlier 4 samples couldn't be backfilled. The issue carries the repro, the writer-side file map, and a fix sketch (pattern + body-only parse branch + pairing by the sid already in the filename, header-derived dimensions degrading to unknown rather than guessing).

    Artifacts: yes — attach the two machine-readable JSON files (hashes/byte counts/ledger rows/param diffs only, no body text). I'll verify the pair diffs against the code paths and the ledger rows against the attribution buckets.

  9. lanxingyun commented on Nov 15, 2026

    @lanxingyun

    Thanks — this closes the loop on our side. Quick acknowledgements plus the files you asked for.

    Q1 correction accepted. A near-total miss (98.5% → 2.2%) meaning the whole cache namespace was unavailable to that request (transport-level discontinuity) matches what the client sees: the sample settles 17 ms after a mid-stream abort, and the very next request is back at 93.5%. We will treat the ledger's abortedNear label as the correct attribution and not file this as a rebuild-seam issue.

    Sample 2 / effort bucket. The build producing the ledger at sample time was v=0.1.213 ([v=0.1.213] log lines on 2026-11-14; 0.1.213 → 0.1.214 upgraded later that evening). That is past #3809 / v0.1.204, so with the effort dimension fingerprinted on output_config / reasoning_effort / reasoning.effort, the pair's only parameter diff — output_config.effort default → high — should land in the effort bucket rather than the residual. We do not have a stored bucket label to quote (ledger labels are not persisted locally), so here is the raw row for that pair; if it still renders unattributed in your ledger, this row + version is the one to check:

    2026-11-14T16:21:33.621 input=141,535 cached=20,736 miss=120,799 hit=14.7% (local +08; pair prefix 411,414 B, identical prefixSha256 on both sides).

    Files (machine-readable: hashes / byte counts / ledger rows / parameter diffs only — no message bodies):
    https://gist.github.com/lanxingyun/fefc42c4541e4f632d4546a2480dd157

    • dsh-provider-evidence.json (6.0 KB) — the two flagged pairs plus four more pairs from the same window: reqIso/prevReqIso, gap minutes, message counts, prefixBytes, prefixSha256 (both sides), dumpSha256 + dumpBytes, and the ledger row.
    • dsh-provider-miss-samples.json (234 KB) — all per-request rows from the dumps retained that afternoon: dump file, iso, prev iso, gap, message counts, input/cached/miss/hitPct, model.

    The four dumps are still on disk (dumps/req-*.json), re-hashed just now and matching the gist values:

    • req-1794642362324… 661,083 B 8accd0c1… (sample 1 previous)
    • req-1794642388482… 695,132 B a394d6b2… (sample 1)
    • req-1794644470974… 527,746 B 8ce5f3e6… (sample 2 previous)
    • req-1794644485538… 530,120 B 6659bd88… (sample 2)

    Re #6788 — noted, thanks for turning the dump/retention gap into a tracked issue; the name/format mismatch you describe is exactly what we hit (long-retention dumps, minutes-long raw window).

    Nothing else pending from our side; happy to run an extra pair-diff or ledger check if a specific bucket still looks off.

  10. ranxianglei commented on Nov 15, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Both gist files verified — the loop is closed on all three points.

    1. Artifacts check out. Sizes match what you stated (6,007 B / 234,130 B); all four dumpSha256 + byte counts in dsh-provider-evidence.json are byte-exact against the hashes you posted earlier (8accd0c1…/661,083, a394d6b2…/695,132, 8ce5f3e6…/527,746, 6659bd88…/530,120). All six evidence pairs carry prefixSha256 == prevPrefixSha256 — the byte-identical message-prefix claim is confirmed for every pair, not just the two flagged ones. Both flagged ledger rows are internally consistent (input − cached = miss); I'm treating the gist rows as authoritative over the hand-computed 119,040 from your first post.

    2. The 610-row file makes the classification airtight. Median hit 99.3%, 604/610 rows ≥ 90% — and the only two rows under 50% are exactly your two samples. Both are one-shot dips sandwiched between healthy requests on the same model and same session:

    • sample 1: 98.9% (07:45:24) → 2.2% (07:46:28) → 98.3% (07:47:08);
    • sample 2: 99.1% (08:21:10) → 14.7% (08:21:25) → 96.3% (08:21:36), then 98.9%+ for the next 30+ requests.

    A one-shot dip-and-recover with a byte-identical prefix and unchanged params is the signature of a transient provider/relay-side namespace event (eviction / node or session rotation) — a bili-side seam would persist across turns, and a param change would keep the dip alive. Two details from the join: the aborted request (07:46:26.175Z) and the predecessor dump (07:46:02.324Z) both have no usage row in your all list — a client-side churn burst, with sample 1 settling 14.5 s after the abort, inside the ledger's 30 s abortedNear window. So sample 1 lands exactly where you agreed: provider-side miss + abort correlation, note not action item.

    3. Sample 2 on v0.1.213 — verified by running the tag, not just reading it. I checked out v0.1.213 and ran its effort-dimension suite (tests/cache-ledger-effort-switch.test.ts) against the tag's own kernel: 7/7 green. That suite pins precisely your shape: an output_config.effort flip (default→high) with a byte-identical message prefix is charged to the named effort bucket — not a seam, not the unattributed residual — including for bodies past the seam cap (your 530 KB dump) and with the negative directions (identical effort + genuine mid-history rewrite still reads as a seam; identical effort + pure tail append goes provider-side, never effort). Concretely, your row input=141,535 cached=20,736 miss=120,799 renders as effort switch: 120,799 (1) — host changed reasoning effort / thinking config; provider re-partitions its prompt cache on it (#3803). The attribution flag is persisted on the ledger line itself, so the report shows the bucket even though labels aren't stored — if your own render of that row still says unattributed, that would be a genuine anomaly worth a fresh look; per the code at your exact tag it belongs in the effort bucket.

    Re #6788 — yes, that's the dump/retention gap issue (the GitHub-side number of the one I filed from this thread); the repro, writer-side file map, and fix sketch are in its body.

    Nothing further from our side on this evidence thread. The issue itself stays open for the remaining items: R2 (per-route render control) awaits the owner's decision, and R3 was already classified (host-side resume re-projection + residual model-echoed tag seeds). Thanks for the clean artifacts — this is exactly the kind of byte-level confirmation the ledger buckets were built to consume.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions