Repository navigation
Description
Summary
Upgrading a production proxy from 0.1.200 to 0.1.201 caused every active session to report thousands of "covered id(s) missing" on its first request after the upgrade. Most were re-anchored, but in several sessions thousands of folded messages could not be matched and re-entered the wire unfolded. The worst Anthropic-lane session went from ~126K to ~998K input tokens in one turn (cache hit 2%). Emergency preflight could not recover it: each fold saved ~2.7K tokens (0%), and one preflight ran for 504 s before the client disconnected.
This contradicts the upgrade note in #3658 ("Upgrade cost: one re-key of stored ids for openai/responses/google lanes; #3644's reconcile tiers … absorb it without re-folding", and "the dsh/anthropic lanes take zero re-keying"). The Anthropic /v1/messages lane was affected, and on several sessions the reconcile tiers did not absorb the change.
Rolling back to 0.1.200 caused a second identical drift wave on every session's first request, so the 0.1.201 → 0.1.200 downgrade is not clean either. Some unmatched folds were lost permanently and could only be restored from a backup.
Conversation content was not lost: the host (OpenCode, compaction.auto=false) re-sends the full history every turn, and the unmatched folds reverted to the original messages. What was lost was the fold-to-message mapping, i.e. weeks of compression state per session, with large cost and latency increases.
Environment
- bili 0.1.200 → 0.1.201 (commit 6f2b113) via self-update, then manually rolled back to 0.1.200
- Linux (Debian VM), Node v24.21.0, systemd user service, proxy mode (URL prefix),
autoRestartOnUpdate: false(restarted by an external idle-gated timer) - Storage:
BILI_STORAGE_UNIFIED=legacy(JSON only; the shadow sidecar was disabled on Oct 15 because of the I/O cost reported in [perf] large active sessions (50-60MB payloads, 23k msgs) serialize the main thread: 1276ms avg local processing per turn + full 57MB snapshot rewrite for +270B deltas #3049),compress.ccr.enabled: true - Host: OpenCode 2.0.28 (OpenChamber desktop), native
compaction.auto=false, unchanged throughout. Upstream is a Sub2API relay; lanes are/v1/messages(Claude Opus) and/v1/responses(GPT). - Long-lived sessions: 1K–26K folded messages each, the largest ~15K messages / 36.5 MiB inbound per request
Timeline (UTC, 2026-10-18)
| Time | Event |
|---|---|
| 12:24–16:48 | 0.1.200 running normally. Zero resent history drifted lines in the log. |
| 16:40:53 | [update] installed 0.1.200 → 0.1.201 |
| 16:48:19 | Restarted onto 0.1.201 (idle). No session traffic until 19:02. |
| 19:02:05–19:27 | First request of every session logs [fold-reconcile] resent history drifted (table below) |
| 19:14:29 | Main Anthropic session: [acp-usage] round 1 input=997959 cached=18468 (cache hit 2%) fold=new (previous turn: input 125906) |
| 19:23:58 | preflight compressed 14 range(s), ~70878 tokens saved (1080245 → 960163) in 503739ms → preflight aborted (the client disconnected during preflight compression) |
| 20:56:10 | After autoUpdate was set false on disk, the still-running 0.1.201 process re-installed 0.1.201 over a fresh manual 0.1.200 install (config change not picked up until restart) |
| 21:01–21:06 | Restarted onto 0.1.200. First request of every session drifts again (second table below) |
First request after upgrading to 0.1.201
missing = covered ids missing; unmatched = re-entered the wire unfolded. Session ids truncated.
| Session | Lane | Missing | Re-anchored (how) | Unmatched | "tok unfolded in churn region" |
|---|---|---|---|---|---|
| f20302d59ffe | /v1/messages | 16077 | 10566 (1372 positional, 9194 toolCallId, 10 order breaks) | 5511 | ~1,122,474 |
| f2d47fc7affe | /v1/messages | 10743 | 8254 positional | 2489 | ~92,406 |
| ebae57880ffe | /v1/messages | 5451 | 4948 positional | 503 | ~81,874 |
| ec3389da5ffe | /v1/responses | 24727 | 15454 (12102 positional, 2252 toolCallId, 1100 shape census) | 9273 | ~2,844,782 |
| ee0bcfb25ffe | — | 33742 | 15146 (11719 toolCallId, 3424 shape census, 3 positional) | 18596 (+31 suspected host-rewritten tool-call ids) | — |
| ebf538502ffe | /v1/responses | 19826 | 19290 positional | 536 | ~53,751 |
| eb97f1845ffe | /v1/responses | 16370 | 15834 positional | 536 | ~60,897 |
| ec075c255ffe | /v1/responses | 16628 | 16505 positional | 123 | ~20,528 |
| eb97f1846ffe | /v1/responses | 10474 | 10472 positional | 2 | ~62,238 |
| ec3389d43ffe | /v1/responses | 26027 | 26027 positional | 0 | ~54,188 |
| eb3f5e79dffe, eb3f5e65fffe, eb41a5099ffe, eb3f5db55ffe, eb4303d04ffe, eb4303ceaffe, eb549c932ffe, eb420d17effe, eb4303d23ffe | /v1/responses | 704–5795 | all positional | 0 | 14K–81K |
On the next turns the unmatched remainder was reported again as covered id(s) missing with no anchor match — mutation … or client-side deletion/truncation (main session: 5511, then 3429 on every subsequent turn). The compression-conflict ledger recorded unannounced-rewrite — 3236/11610 incoming message(s) carry pre-turn refs of 14982 known, although nothing outside bili touched the history.
Input-token effect (last request before the upgrade → on 0.1.201): main session 125,906 → 997,959; eb41a5099ffe 176,409 → 350,796; eb420d17effe 98,609 → 157,349; eb4303ceaffe 45,146 → 82,594.
First request after rolling back to 0.1.200
| Session | Missing | Re-anchored | Unmatched |
|---|---|---|---|
| f20302d59ffe (/v1/messages) | 14783 | 10376 (6886 positional, 3448 toolCallId, 42 toolCallId twin) | 4407 |
| ec3389da5ffe | 23884 | 14934 | 8950, then 1775 more on a later turn (~1,008,485 tok unfolded; input 104K → 896K) |
| f2d47fc7affe | 10743 | 8254 | 2489 |
| ebf538502ffe | 20405 | 19872 | 533 |
| ec075c255ffe | 16628 | 16505 | 123 |
| others | 1551–26295 | all | 0–2 |
Expected
An upgrade that changes message identity derivation should either migrate persisted fold state, or re-anchor all of it losslessly before the first forward. #3658 states this for exactly this release. If that is impossible, the release should be marked as needing a migration and held from auto-update (e.g. via the #3619 policy file hold/blocked), and the downgrade path should not re-churn.
Notes / suggestions
- Anthropic lane affected despite "zero re-keying" in [PR] protocol-stable core identity: same logical conversation, same ids in every dialect (#3642) #3658: positional re-anchoring covered only 1372 of 16077 in the main session; toolCallId did most of the work and still left 5511 unmatched.
- Re-anchoring degrades sharply on very long sessions with many unanchored blocks. Short sessions re-anchor 100% positionally; the 15K–34K-message sessions are where unmatched counts explode.
- Downgrade is also lossy: 0.1.201 persisted state under the new identities, so 0.1.200 sees a second churn. A one-time snapshot of fold state before the first request on a new identity epoch would make rollback safe.
- Emergency preflight stalls on drift: after a mass unfold, preflight folds of ~2.7K tokens each (0% shrink, cache ceiling ≥0%) for 504 s, which is the [resilience] Preflight compress/timeout loop on sustained mass fold-coverage loss has no graceful degrade #2438 shape.
- The pending image-identity change in Image message ids re-mint whenever an earlier image leaves the visible tail (membership-based ClusterCounter suffix) — prefix-cache breaks per fold, feeds #3748 storage growth #3751 says "existing image ids re-mint once" at upgrade. Given this incident, please validate that kind of one-time re-key against real long-lived persisted sessions (and the downgrade path) before release.
advisoryCheckcan force-install a target version even withautoUpdate: false. Operators who pin a version also need to know to disable it.
Frozen copies of the affected session state (legacy JSON + content stores) and the full proxy log are retained. I can run diagnostics or a reconcile replay against them if useful.
Activity
ranxianglei commentedon Oct 18, 2026
(ai: vllm-qwen/qwen3.8-27b)
查重: 不重复 —— 按多组关键词检索过 open+closed(identity re-key / mass-unfold / covered id missing / downgrade churn),无既有同类条目。相关但不同:#3658 是引入该 identity 变更的 PR(本 issue 的源头,不是重复报告);#3751 是你 note 5 提到的另一个待上 image identity re-key 提案,后面会一起评估;#2438 仅被你的 note 4 引用为 preflight 症状形状。
标签: bug + opencode + scope/kernel + p1(宿主是 OpenCode 2.0.28;失败域 = 上下文管理的 identity/reconcile 核心;严重度 = cache hit 崩坏导致成本爆炸 + 数周压缩状态丢失,正中 p1 定义)。
版本钉定(version-first):tag 对照确认
kernel/src/wire/identity.ts(#3658/#3642 的规范 identity 模块)在 v0.1.200(kernel 0.0.131)不存在、v0.1.201(kernel 0.0.133)首次出现——id re-key 确实发生在 0.1.200→0.1.201 这一次升级,与你的时间线吻合,排除 #2847 那类版本误判。v0.1.200..v0.1.201 全窗 diff(37 文件,+2217/-394)里 identity 面就是 kernel/src/wire/{identity,openai,responses,google,bili-message}.ts + src/fold-reconcile.ts(+318)。初步层次判断(代码级验证进行中,结论随后):
src/fold-reconcile.ts和 anthropic codec diff 逐条钉死(可用你冻结的会话快照做 replay 验证)。autoUpdate: false下仍强制安装目标版本——pin 语义漏洞)。这两项修法不依赖上面的设计决策,会先修。下一步:① 钉死 5511 unmatched 的构成与机制;② 修复方向我倾向「持久化状态带 identity 纪元标记 + 纪元不一致时加载期做确定性 re-key 迁移(用存储的原始内容对新旧两套 codec 各算一遍,建 old→new 映射重写 fold 元数据)」而不是继续给 reconcile 加档——这涉及 kernel 侧与持久化格式,分析完成后我会给出选项对比请你拍板;③ 不依赖决策的部分(preflight 早停、advisoryCheck 语义)先开修。
另外:你保留的冻结会话快照非常有用,若方便请提供(或告知获取方式),我打算用它做 re-key 迁移方案的回放验证,顺带把 5511 个 unmatched core 的构成一次性钉死。
ranxianglei commentedon Oct 18, 2026
(ai: vllm-qwen/qwen3.8-27b)
分析完成:根因已定位并本地复现,修复方案待拍板
一句话结论
0.1.201 确实重铸了 anthropic 车道所有 tool-call/tool-result core 的 id(这点在 #3658 评审时已被指出「zero re-keying 措辞言过其实」,但发布的 CHANGELOG 行文仍自相矛盾)。但你主会话丢 5511 个 fold 的真正原因不只是 re-key——bili 侧有一个此前无人踩中的容量悬崖:fold 锚点持久化有 16384 的硬上限(
MAX_ANCHORS,#2334 加的性能保护),超出上限的 covered id 只剩一条脆弱的再锚定通道(位置扫描),而位置扫描在累计 8 次失配后死亡。运行数周的长会话必然积累超过 8 次真实漂移(编辑/换模型/删除),于是 re-key 与漂移叠加成 mass-unfold。降级再次 churn 是对称的结构性问题:状态按新 id 落盘,旧版本看到的是又一轮 re-key。证据链(本地复现,非推测)
我把两个版本的 codec + 完整对账引擎(v0.1.200/v0.1.201 源码提取 +
src/fold-reconcile.tsv0.1.201 的planReconciliation全段,可直接执行)搭成了可运行的复现台,生成 OpenCode 形态的 anthropic 历史(3400 turns → 19374 cores,其中 17436 个 folded,越过 16384 上限),模拟 v0.1.200 稳态元数据后喂给 v0.1.201 解析 + 对账:关键事实:
tool_use.id/tool_use_id,新 seed 去掉该槽位并对参数做规范化);text/thinking/image core 字节不变、id 不变。层次判断
MAX_ANCHORS=16384把越限 id 静默降级为单通道恢复,且触发时无任何日志/告警。src/advisory.ts:337advisory 强装路径硬编码autoUpdate: true(搞一个自动强制更新,针对重大bug的版本 #1481 紧急回滚的刻意设计),且配置文件改动无热加载(只有 web UI 的 Apply 路径)——这是 20:56「我设了 autoUpdate:false 它还是重装 0.1.201」的直接原因。与本事件不同失效域,我会单独立项(带来源标记)。修复选项(请拍板)
选项 A(推荐,无决策依赖,我现在就做):轻量锚点越过上限
seed 循环在 16384 之后停止计算昂贵锚点(带 n/m/sh/wh 哈希字段),但继续为越限 id 创建零哈希的轻量锚点
{role, toolCallId}(O(1) 字段读取,内存 ~60B/id)。Pass 1 的 toolCallId 通道恰好只消费这两个字段,其余通道遇缺字段优雅退化。效果:任意长度的会话,tool core 都走协议级唯一的 toolCallId 通道再锚定,误配面为零(匹配纪律不变:每个 (id, role) 要求唯一未认领候选),升级/降级双向生效。残留:越限的非 tool core(image 等)仍靠位置扫描——anthropic 车道占比小,#3641 的 shape census 有部分兜底。选项 B(需拍板):调高或移除昂贵锚点上限
覆盖所有 core 类型的越限恢复(含 image)。成本:每 pass 哈希工作量(#2334 的动机,当时实测 20k id ≈ 3616 次多余哈希,现代硬件可忽略)+ 持久化 JSON 每 id ~300B(17k 时约 5MB)。我的倾向:提到 65536 而非完全移除(保留 #2334 的意图,覆盖绝大多数实际会话)。具体数值你定。
选项 C(需拍板,较大设计):身份纪元标记 + 持久别名表
metadata 增加 free-form 字段
identityEpoch(kernel 身份推导每次变更即递增);纪元切换后的首个 pass 把 old↔new 配对结果固化为显式别名条目并持久(sticky),此后任何方向的纪元切换(升级/降级)都能经别名表映射回去——「降级 re-churn」从此结构性消失,也直接服务 note 5(#3751 的 image re-mint 发布前可用同一机制验证)。实现期发现的约束:完全离线确定性迁移目前做不到——状态里没有成员消息的原始字节(锚点只存指纹),离线 remap 要么引入字节留存(存储成本,与 #3049 交互),要么接受「首个过渡 pass 仍靠(被 A 强化过的)启发式配对,结果固化」。做不做 C、做到哪一档,是存储/产品取舍,你定。发版流程:0.1.201 已出,短会话不受影响(run A 证明干净长会话也全恢复),不建议 hold/block;建议 A(+B)快进 0.1.202。是否用 #3619 策略文件做事前拦截,是独立操作,听你的。
结构化八问
行动清单
ranxianglei commentedon Oct 18, 2026
(ai: vllm-qwen/qwen3.8-27b)
止血 PR 已开:#3753 —— #3753 #3752)。CI 在途,合并需要你。
修复内容(Option A,lite anchors):
MAX_ANCHORS=16384 窗口外的 tool core 现在持久化一个无 hash 的轻量锚点{r, c, t, b}(角色、内容类、协议 toolCallId、文本长度)。Pass 1 的 toolCallId 通道恰好只消费这几个字段——任意长度会话的 tool core 都能再锚定,零新增哈希、零新增误配面(认领纪律不变:每个 (toolCallId, role) 唯一未认领候选)。窗口上限本身不动(#2334 契约原样),非 tool 的超窗 core 维持现状(positional-only)。回归测试 3 条(铸造形状 / 用 reconciler 自己的计数器证明走的是 toolCallId 通道 / 完整事故轨迹收敛回放),不带修复全红、带修复全绿;既有 16 个 fold-reconcile 相关套件 224/224 本地过,全量等 CI。诚实边界(必须说清):re-key 后的第一个 pass 行为不变——死区里的旧 id 在线上没有字节可配对,仍会有一次性 unfold(这正是你看到的 5511 那批);preflight 重折到新 id 后第二个 pass 账本即完整,此后该会话对 tool core 免疫(包括回退路径——只要降回旧版前跑过至少一个 fixed-build pass,旧版的 Pass 1 也能消费这些
{t,r}字段)。要消灭首 pass 损失本身,需要 Option C(epoch 标记 + 粘性别名表),那是设计决策,等你拍板。八问:
n的锚点显式跳过;image 通道本就不收带t的锚点;gap-tier order-bracket 对 lite 锚点新获得资格(同一受认可层级、同证据等级,PR 里已披露);降级兼容有一个已知残留(旧版读新版状态时,lite 锚点会被其 textless 谓词视为空文本族——完全围栏+精确计数的 ≤2 个连续 tool 单元理论上可被 k-th 配对,该层级本就没有角色检查),概率低、有界、仅限降级场景,PR 里逐字记录了,不在本 PR 修(那意味着改已发布代码路径)。其余处置:
autoUpdate:false+ 配置不落盘热载)是独立故障域,按政策另行立项(带来源标记),不塞进本 PR;结论:止血已就绪待合(PR #3753);B/C 两项决策请求维持上一条回复的表述,等你拍板。
ranxianglei commentedon Oct 19, 2026
为什么要有这个上限 怎么来的
ranxianglei commentedon Oct 19, 2026
(ai: vllm-qwen/qwen3.8-27b)
问得好——这个上限的来历正好解释了为什么它会炸。
是什么上限:
MAX_ANCHORS = 16384(src/fold-reconcile.ts:166)——持久化锚点表(foldAnchors)的大小上限。每个 covered id 存一个锚点,里面带约 5 个摘要字段(归一化身份 sha256、去 toolCallId 的 norm、simhash 词草图 ×2、turn key),单条几百字节。怎么来的: 它是个防御性存储护栏,不是设计出来的工作集。引入时的注释就是全部设计文档(
src/fold-reconcile.ts:164-165):即当时判断"16k 远超任何真实会话",上限只为挡病态膨胀。它早于 #2334 就存在(v0.1.187 已可见);#2334/#2338(10-07)没改这个值,只把执行点从「先全量构建再删超限」改成「构建到 16384 即停」——修的是每请求重复计算被丢弃锚点的性能问题(20k-id 会话每请求白付 3616 次 SHA256,函数级 A/B 中位数 171.6ms → 29.6ms)。
为什么现在错了: 前提被证伪了——你的会话 covered id 是 21k–34k,"far beyond any folded session" 不成立。超窗的 id 从此静默失去全部非位置通道(无警告日志),只能靠 Pass 0 位置扫描兜底;而位置扫描在一侧超过 8 个 resync gap(
MAX_RESYNC_GAPS)就死——长会话积累的正常漂移轻松击穿它,于是大面积 unfold。两层参数各自都"合理",组合起来在长会话上是断崖。PR #3753 怎么处理这个上限: 不动 16384 本身(#2334 的性能性质保留——昂贵哈希预算不变),但超窗的 tool core 额外记一个零哈希 lite 锚点(只存 role/contentType/toolCallId/text 长度四个现成字段,O(1)),让 #3641 的 toolCallId 通道在窗口外继续可用。非 tool core 超窗后仍是位置-only,这是诚实残留;如果你想让所有类型都有窗口外通道,那就是待决策项 B(把昂贵锚点上限提到 65536,代价:持久化 ~+6MB/34k id + 一次性 mint 成本)。
另:CI 那边 windows shard 一个测试超时(flake 证据已核,同测试两小时前 master 基线 windows 上通过,本地 9/9 绿),已重跑,结果出来我在本线程报。
ranxianglei commentedon Oct 19, 2026
(ai: vllm-qwen/qwen3.8-27b)
CI 收口了,PR #3753 现在全绿可合:#3753
windows 超时的定性(不是本 PR 的回归):
testTimeoutFailure(恰好 10000ms 超时),不是断言失败;同文件其余测试在同一 shard 上各 ~3s 通过;tests/plugin-terminal-usage.test.ts(responses lane 计费观测)与本 PR diff 零交集(diff 只有src/fold-reconcile.ts+ 新测试 + CHANGELOG,该测试不 import 也不触及 fold-reconcile);过程透明说明:最初 push 触发的 pull_request 矩阵被随后的 label 同步事件(scope=skip)按并发组取消了——这是 ci.yml 注释里记录过的 #3688 label/merge 竞态,不是代码问题;我先 workflow_dispatch 补跑(core 矩阵,遇到上述 flake),flake 排除后给 PR 打了
ci:full标签补上 branch protection 要求的完整 12-leg 矩阵(ubuntu/windows × node22/24),12/12 + 全部静态门禁 success。待你处置:
[no-dispatch],不阻塞本 PR。ranxianglei commentedon Oct 19, 2026
#3759
fix: hash-free lite fold anchors for past-cap tool cores (#3752)
docs: changelog entry for #3752 + correct #3642 upgrade-cost wording
ranxianglei commentedon Oct 19, 2026
(ai: vllm-qwen/qwen3.8-27b)
看了 #3759(已合):这是渲染层的根治——出站前缀改由服务端正史渲染,身份 re-key 从此不再影响线上字节与缓存。它直接覆盖了我此前列的 Option C(epoch 标记+别名表):C 撤回,状态侧不需要再做纪元迁移层,#3759 让线上字节与身份推导解耦了。
但 #3753(lite 锚点止血)与 #3759 是两层,不互相替代,建议两个都合:
FOLD_DRIFT_ESCALATE_PASSES=3(src/fold-reconcile.ts:205)连续多数丢失 pass 后块被reapDestroyedSubstrate删除([Bug][proxy] unframed host-side history deletion leaves zombie fold blocks rendering carriers forever — one carrier flaps in/out every turn (persistent prefix-cache churn) #2695)。没有 [PR] fix: hash-free lite fold anchors for past-cap tool cores (#3752) #3753,re-key 后超窗 tool id 每 pass 都认领不了 → 老块的 z 计数确定性累积 → 3 pass 后块被删 → 块没了,snapshot 渲染器无东西可抑制,内容照样 unfold 回线上。即:[PR] Snapshot rendering: proxy lane outbound prefix rendered from server-side canonical history; existing sessions immune to | 快照渲染:proxy 车道出站前缀从服务端正史渲染,存量会话对内核升级免疫(#3756) #3759 保线上稳定,[PR] fix: hash-free lite fold anchors for past-cap tool cores (#3752) #3753 保基质存活,基质没了渲染器没有可渲之物。{r,c,t,b}字段旧版 Pass 1 原样消费(降回旧版前跑过一个 fixed-build pass 即安全);[PR] Snapshot rendering: proxy lane outbound prefix rendered from server-side canonical history; existing sessions immune to | 快照渲染:proxy 车道出站前缀从服务端正史渲染,存量会话对内核升级免疫(#3756) #3759 在更老版本里根本不存在。八问(决策项:#3753 保留还是按 superseded 关闭)
BILI_ROLLOUT_SNAPSHOT_RENDER/rollout feature(你的 PR,CONFIGURATION.md 已记)。结论:建议两个都合——#3759 管线上,#3753 管基质,合起来才是 id-churn 家族的闭环。若你判断 snapshotRender 会很快 100% 放量、且接受窗口期基质侵蚀风险,说一声我关 #3753;否则等 CI 绿后合并即可。
6 remaining items
feat: wire google lane into snapshot render (#3764)
fix(update): record which channel installed the current version (ranx…
fix(cli): log effective update-lane settings at startup (ranxianglei#…
Merge pull request #3753 from ranxianglei/2026-10-19_lite-fold-anchor…
refactor(#5874 P2): mapEgress direct authority — no per-round diff, e…
refactor(#5874 P2): mapEgress direct authority — no per-round diff, e…
feat: snapshot-table compression foundation — the six-stage pipeline (#…
feat: snapshot-table compression foundation — the six-stage pipeline (#…
feat: snapshot-table compression foundation — the six-stage pipeline (#…
refactor(#5874 P2): mapEgress direct authority — no per-round diff, e…
refactor(#5874 P2): mapEgress direct authority — no per-round diff, e…
feat: snapshot-table compression foundation — the six-stage pipeline (#…
feat: snapshot-table compression foundation — the six-stage pipeline (#…
feat: snapshot-table compression foundation — the six-stage pipeline (#…