Skip to content

[PR] fix(storage): retire superseded same-msg_id generations on non-completeLog saves (#4392 facet C) - #4888

Open
ranxianglei wants to merge 7 commits into
masterfrom
2026-10-26_4392-msgid-content-supersede
Open

ranxianglei wants to merge 7 commits into
masterfrom
2026-10-26_4392-msgid-content-supersede

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What & why

Fixes facet C of #4392 — the [diverged: lastMessages] content divergence reported on v0.1.207 (@BangDoll98): a stable client id whose message content was rewritten (a subagent re-dispatch reusing the parent session id, #4769) left BOTH the old and new byte-rows live in the unified store, so every replay carried the stale generation → permanent DIGEST DRIFT + unbounded row growth.

Root cause

identity is content-derived (identity.ts strips only id/wireIndex/ledgerSeq), so a genuine text edit takes a NEW identity → remapPlanToAbsolute mints a fresh seq for the new generation while the prior same-msg_id row stays resident. The only thing that retires such stale rows is the #3285 sweep, gated on opts.completeLog === true. Envelope renders are completeLog-less pruned views, so the sweep is skipped — correctly, since a global sweep on a partial view would delete live rows merely outside the render window. On the reporter's save class the stale generation therefore never got retired.

Companion facets under the same issue/symptom, tracked separately: facet A (top-level field list) already merged as #4192/#4242 but unreleased; facet B (canonicalize key ordering) is #4685.

Fix

A targeted sibling of that sweep in the full ingest path (src/storage/store.ts, immediately after it): when completeLog !== true and the session is not lane-owned, retire prior same-msg_id generations whose identity is absent from this ingest while their superseding generation is present. Touching only msg_ids actively in the render preserves pruned-render safety (the reason the global sweep stays gated); lane sessions opt out (#3248 positional audit floor). It mirrors the existing sweep's discipline exactly — retire-not-delete (retired_at tombstone), media refs released, text CAS ref kept, reactivate-on-return intact — so full and non-full paths now converge.

Behavior change (disclosed)

  • Old: a content rewrite under a stable client id on a non-completeLog save left both generations LIVE forever (stale row rode every replay).
  • New: the prior same-msg_id generation is RETIRED (tombstoned, excluded from replay; media refs released, bytes kept until archive/reactivation).
  • Scope: non-lane sessions only; only msg_ids present in the current render with a newer-generation sibling. Unchanged identities untouched (no churn); a revert reactivates the tombstone.
  • No config surface, wire, or schema change (reuses the existing retired_at column, schema v8).

Test

tests/issue4392-msgid-supersede.test.ts: after a non-completeLog rewrite of a stable client id, exactly one live row survives per msg_id carrying the latest content; an unchanged re-ingest churns nothing.

Review note

Could not run tests/typecheck locally (no native better-sqlite3 build in this sandbox) — CI is the verifier. This touches p1 load-bearing storage under active development (#4316/#4532/#4540), so it needs careful human review; the change is minimal and mirrors the adjacent sweep's proven discipline. Happy to fold into the maintainers' storage workstream if sequencing prefers that.

…teLog saves (#4392 facet C)

A genuine content rewrite under a stable client id mints a NEW identity
(identity is content-derived; identity.ts strips only id/wireIndex/ledgerSeq),
so remapPlanToAbsolute gave the fresh generation a new seq while the prior
same-msg_id byte-row stayed resident. The #3285 sweep retires such stale rows
only when opts.completeLog === true; envelope renders (completeLog-less) skip
it, so the stale row rode every unified replay forever — permanent DIGEST DRIFT
plus unbounded growth (triggered by subagent session-id sharing, #4769).

Add a targeted sibling of that sweep in the full ingest path: when completeLog
is not true and the session is not lane-owned, retire prior same-msg_id
generations whose identity is absent from this ingest while their superseding
generation is present. Targeting only msg_ids actively in the render keeps
pruned renders safe (the very reason the global sweep stays gated on
completeLog); lane sessions opt out (#3248 positional audit floor). Same
retire-not-delete + media-ref discipline as the existing sweep — text CAS refs
are kept, media refs released, reactivate-on-return still works.

Regression test: tests/issue4392-msgid-supersede.test.ts (one live row per
msg_id after a non-completeLog rewrite carries the latest content; an unchanged
re-ingest churns nothing).

Addresses #4392 (facet C — the reported [diverged: lastMessages] content
divergence). Companion facets: #4685 (facet B, canonicalize key ordering);
#4192/#4242 (facet A, top-level field list, already merged — unreleased).
Trigger: #4769. Related storage workstream: #4316/#4532/#4540.
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-26_4392-msgid-content-supersede (c7d40e6)

Latest build — install this exact version

npm install -g billion-context@0.0.7335-pr.4888.2000ff4

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
c7d40e6 billion-context@0.0.7335-pr.4888.2000ff4 build
bcc90a8 billion-context@0.0.7108-pr.4888.1f0a47f build
390ee67 billion-context@0.0.7103-pr.4888.bd11d1c build
470bb97 billion-context@0.0.6845-pr.4888.f764a4b build
97aed08 billion-context@0.0.6345-pr.4888.d701a54 build
740c471 billion-context@0.0.6341-pr.4888.27d08c7 build
718bd96 billion-context@0.0.6333-pr.4888.ffbee2b build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4888.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(storage): retire superseded same-msg_id generations on non-completeLog saves (#4392 facet C) [PR] fix(storage): retire superseded same-msg_id generations on non-completeLog saves (#4392 facet C) Oct 26, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复 —— 本 PR 是 #4392 下的 facet C(稳定 client id 内容被改写后旧代字节行永久驻留),与 facet A(#4192/#4242,已合未发)和 facet B(#4685)机制不同;未发现同机制的并行 PR。相关项:#4769(subagent 复用父会话 id,触发场景)、#3285/#4390/#4793(相邻 sweep 纪律)。

标签:bug / p1 / scope/storage / claude-code / codex(存储行无界增长属缓存泄露类,按 owner 2026-10-19 修订定 p1);顺手清掉了 #4392 上重复的 p2(严重度应恰好一个)。

本 PR 是我此前开的,按 [PR] 协议复核而非重做:拉了分支 2026-10-26_4392-msgid-content-supersede @ 718bd964c 逐行读 diff、对照 master 现行存储机制逐项验证。CI 全绿(ubuntu/windows × node22/24 × 5 个 test shard + 全部 guard job,含对真 MySQL 8.0 的 mysql-contract;门控 e2e 车道按惯例 skip)。本地沙箱无法跑测试(依赖装不了),CI 即验证者。

结构化八问:

  1. 是什么问题:稳定 client id 的消息内容被原地改写(典型触发:[Bug][claude] Claude Code VS Code subagents still ride the parent session id on 0.1.208/master — each dispatch reaps the parent's folds (#4352 was closed by #4337, which does not fix it) #4769 subagent 重派发复用父会话 id)时,非 completeLog 保存(envelope 渲染这类)下旧、新两代字节行永远都活着,每次回放都带着旧代 → 永久 DIGEST DRIFT + 行数无界增长。
  2. 什么原因导致的:identity 是内容派生的(identityOfStoredContent 只剥 id/wireIndex/ledgerSeq),内容改写 = 新 identity = remapPlanToAbsolute 给新代铸新 seq,旧的同 msg_id 行留在原地;唯一能退役这种陈旧行的机制是 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 sweep,而它被 completeLog === true 门控——envelope 渲染正是没有 completeLog 的裁剪视图,在那里跳过全局 sweep 是对的(部分视图上全局 sweep 会误删窗口外的活行),于是这条保存路径上旧代永远不被退役。
  3. 影响范围与严重程度:p1。报告者在 full mode(其 /__bili/status:requested=effective=full)→ 数据完整性问题:回放永远携带旧代、drift 永不收敛、行只增不减;shadow mode 用户则是假阳性 drift 报告 + watermark 抖动。触发条件:任何稳定 client id 下的原地内容改写(最常见真实场景即 subagent 复用父会话 id)。
  4. 怎么修复的:在 full ingest 路径紧跟 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 sweep 之后加一个定向的兄弟 sweep:当 completeLog !== true 且会话非 lane-owned 时,退役「出现在本次 render 里、但 identity 不在本次 ingest 里」的 msg_id 的先前代(retired_at 墓碑、释放 media ref、保留 text CAS 字节)。只碰本次 render 出现的 msg_id,保住裁剪渲染的安全性(这正是全局 sweep 必须保持门控的原因);lane 会话排除(Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248 positional audit floor)。纪律与现有 sweep 完全一致,full/non-full 两条路径从此收敛到同一套 retire-not-delete。
  5. 是否有回归问题:逐项审过既有行为——completeLog=true 路径不变(新块以 !==true 门控);lane 排除不变;delta 路径不变(严格追加,改写走 full-path fallback,store.ts:696);revert/reactivation 完好(remap 的 seqByIdentity 含墓碑行 store.ts:1217,Four storage-layer defects: identity hash contaminated with positional metadata (truncation triggers identity swap) / sw | 存储层四缺陷:身份哈希掺位置元数据(截断即换血)/清扫删除回退历史/渲染接缝图片只比长度/fork 谱系死列 #4390 复活循环 store.ts:1301-1307 在每次 full 保存——含 non-completeLog——都会跑,内容回退时世代正确翻回);未改动 identity 零扰动(测试 2 用 deepEqual 钉死无 churn);块覆盖解析本来就用 LIVE 行([PR] chore: empty PR (no changes) #4801),与退役语义一致;lastLiveSeq/fork-cut 取最大活行 seq,退役更低的旧代行不影响;导出/归档([PR] JSONL export includes retired (rewound) history rows, marked by retiredAt #4670/[PR] fix(#4665): carry identity + retired_at across archive/restore #4672)带 retired_at 往返无损。无 wire/schema/config 面变更(复用既有 retired_at 列,schema v8)。一个有意行为变更已在 PR 正文披露:旧代从「永远 LIVE」变为「RETIRED」(退出回放,字节保留至归档/复活)。
  6. 是否破坏缓存:改善而非破坏——移除历史中段重写源(旧代残留),存储收敛后前缀缓存反而稳定;退役行退出回放。
  7. 是否改了设计方向:否。补的是既有 sweep 机制的覆盖缺口,不转向。
  8. 是否引入新配置:无。

一处偏差需要你知晓:你在 #4392 批准的 Option B 原话是「删掉旧行」,实现是 retire-not-delete(墓碑)——与 #4390 既定的 historian 纪律一致:回放效果相同(每个 msg_id 一行活行,getMessages 过滤 retired),但字节保留到归档/复活、media ref 释放。比批准措辞更保守更安全,建议保持;若你坚持物理删除我再改。

review 中发现的一个同族残留(未擅自并入本 PR):MySQL 后端有自己的 ingestSession 拷贝(src/storage/mysql.ts:394-428),同样只有 completeLog 门控的 sweep、缺这个兄弟块——同一机制家族在 opt-in MySQL 后端上的同款缺陷。mysql.ts 当前有并行在途工作(#4678 v8 dedup),且本 PR CI 已全绿,我没有单方面扩大 diff;已作为子缺陷记入 #4392 清单。要并入本 PR 说一声即可(机械镜像 + contract-suite 场景,很小)。

判定:可合入,待人工 review —— p1 承重存储面,方向与你批准的 Option B 一致,机制镜像相邻已证明的 sweep 纪律,回归逐项审过无意向漂移,CI 全绿。合并动作仍留给人工。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant