Skip to content

fix: 重启交接后 unified 存储跨进程污染固化(DIGEST DRIFT 永不愈合)— #3285 - #3288

Merged
ranxianglei merged 4 commits into
masterfrom
2026-10-15_fix-3285-unified-dual-writer
Oct 16, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-10-15_fix-3285-unified-dual-writer

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

What & why

[storage] unified shadow DIGEST DRIFT ... migration parity broken fired on boot and never healed across 8h of steady traffic after a host restart (#3285, v0.1.197 field case). Root cause: the unified store's per-session ingest watermark (sessions.last_ingested_seq) carries no writer identity, and the full-ingest path only ever INSERTs (INSERT OR IGNORE) — so under a restart-handoff overlap (old instance still draining while the new one boots; both share the machine-global state dir per the #2671 contract) the new process inherits a foreign watermark's trust AND foreign message rows that no later save removes. Every sampled replay digest keeps flagging drift forever; the #3166 heal (watermark-clear on drift) is insufficient because the subsequent full re-ingest still keeps the orphans.

Two changes, both in the unified store:

  1. Watermark trust does not cross process handoffs. resetAllIngestWatermarks() runs once at writer init (UnifiedShadow.maybeOpen). Each session's first save after boot takes the exact full path, re-establishing trust from this process's own view. Pure readers (export) never call it.
  2. Full path sweeps stale message rows when the plan IS the whole log. Same authoritative-set discipline blocks/refs/ccr already follow, gated on completeLog (plan carried a raw-tail log): rows whose identity (msgId + content hash) is absent from the plan are released (CAS refcount) and deleted. Envelope-only plans are pruned renders (Identity pipeline: key sessions by (agent, conversation id) in the unified store — decouple fold ledgers from protocol/host origin (#2938 / #2847 / #2671 defect-1) #2960⑤ folded view) — sweeping on those would delete raw rows interned beyond the render window, so they stay insertion-only.
  3. Lane exclusion (CI regression fix, commit 2). The incremental lane (Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248) consumes the messages table positionally — its baseSeq epoch is an absolute row count recorded at resync and rejected-branch rows must stay as an audit floor the served history slices past. Sweeping them invalidates the epoch (cold rebuild serves an empty canonical → 409 loop; caught by tests/incremental-lane.test.ts "divergent resync survives a cold restart"). Lane-owned sessions (any inc:* session-ext key) therefore opt out of the sweep.

Behavior changes (disclosed)

  • New: every boot clears all unified watermarks → one extra full ingest per saved session per boot (bounded cost, O(session log) once).
  • New: contaminated sessions' persisted messages row set changes once (foreign orphans removed) on the first complete-log full ingest — this is the repair; clean sessions see no difference (their plan covers every stored identity).
  • Residuals (documented in code): (a) lane-owned sessions keep fix: per-session ingest watermark — storage saves cost O(new messages), not O(full log) (#3133) #3166-class orphans (they opt out of the sweep; watermark reset still protects their delta trust); (b) folded-resident saves carry no raw tail → not sweepable; (c) in unified FULL mode, reads between boot and the session's first save may still see pre-handoff bytes until that save converges the table.

In-pass fix (unrelated, clearly labeled, commit 3)

tests/issue3148-upstream-circuit.test.ts failed on every shard of master itself (verified against the base commit's own CI): the #3148 "exact clamp event" expectations (cap 1360) predate the #3071 4096 capability floor, which is intentional fail-closed behavior (clampOutputBudget, bounded by window - input) with its own pinned test suite. Updated the two stale constants + the explaining comment to 4096. No source change; the recorded event and the wire decision remain exactly equal (the test's actual invariant). Dup-screened: no open issue/PR addresses this red.

Tests

  • New regression test (fails without the fix, verified by stash): restart handoff — previous writer's rows + watermark do not survive; first save = full path, orphans swept, digest converges (verifyDrift: 0), steady-state delta resumes.
  • Targeted local: incremental-lane 29/29, storage battery 90/90 (1 skip), typecheck clean. Full-suite verdict = CI on this head.

Related: #3285 (de-anchor facet there is host-side — DSH Desktop rebuilds/resends reconstructed history across its update+restart; the DIGEST DRIFT facet is what this PR fixes). Not closing #3285.

@github-actions

github-actions Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-15_fix-3285-unified-dual-writer (2170bbd)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3943-pr.3288.3abfa33

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3288.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

独立核验结论:PR 原版(67c3ef2)自身引入一处回归,已在本分支修复并验证;现分支头 d62bf55 本地定向全绿,等 CI。

先说查重与过程:接单后先做本地 A/B——tests/incremental-lane.test.ts 在 master 16/16 通过、在 PR 原头 15/16(第 4 例 "divergent resync survives a cold restart without a 409 loop" 得 409≠200),确认 CI 红不是环境问题而是本 PR 引入。工作期间分支上出现了两个新提交(同一回归的修复 + #3148 期望修正),我对其方案做了独立审核(见下),判定成立,遂放弃自己并行实现的替代方案(retain-floor 跨文件机制,留在本地 reflog 未推),只补了一个存储层单测。

结构化八问

  1. 是什么问题:本 PR 的 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285「full ingest 清扫 plan 外残留行」把另一功能正在使用的行也删了——增量 lane(Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248,rollout-gated 默认关)对分叉 resync 记录一个「行数纪元」并按绝对行位消费 unified messages 表;被它有意留存的 rejected-branch 行被清扫掉后,冷重建 slice 出空 canonical → 409 → resync → 409 死循环。
  2. 什么原因导致的:清扫设计针对「重启交接」(新进程视图即真相、plan 外行皆残留),但没有区分「残留行」与「其他消费者有意保留的行」。lane 的被拒分支行正是后者:不在任何 plan 里,但其绝对行数是冷重建依赖的指针,删掉即指针悬空。(更深一层:我首版用「保留地板」修时,新单测还暴露了 seq 是 0-based、含式边界会永久保护 seq 0 的边界 bug——已按独占边界修正后整体让位于已合入方案。)
  3. 影响范围与严重程度:仅 unified 存储 + 启用 incremental turns(默认 0% rollout)的安装;触发条件 = 分叉 resync 后、下一次 save 落库前发生冷重启。严重度:该会话卡死在 409/resync 循环(需手动 reseed 恢复);其余会话与 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 原目标(DIGEST DRIFT 愈合)均不受此回归影响。
  4. 怎么修复的:排除判定收在存储层——携带任一 inc: 前缀 ext key 的会话(lane 只写 inc:system / inc:cache-marks / inc:base-seq 三个 key,已全仓审计无其他写入方)整体跳过清扫,其余会话保留 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 清扫。时序上 inc:base-seq 在 forward 之前落盘,无竞态窗口;legacy-only 实例不经过该路径。另补存储层单测钉住机制本身(lane 会话行留存 + 无标记对照组照常清扫)。分支上的 [bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148 期望修正(1360→4096)我也独立核过:4096 是 [bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 有意的 capability floor,与实际行为一致,standalone 9/9 绿。
  5. 是否有回归问题:逐项审过既有行为——① [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 核心目标(非 lane 会话清残留)保持(handoff 测试 + 新增对照组均过);② lane 会话回到 master 行为(行永不删),严格不差于 master、严格优于本 PR 首版;③ digest replay 只读 tailSeqs 指针、不依赖全行集,留存行不产生新 drift;④ importRecord/export 不走 completeLog,不受影响;⑤ [bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148 是纯测试期望修正,wire 行为未动。行为变更披露:lane-owned 会话(带任一 inc: key)从此不参与 [bug] Fold anchor detaches after host restart (parent-gone): hydrated and unified-shadow DIGEST DRIFT within the same second, gap solidifies without self-healing (0.1.197, complementary to #3013 restart path) | [bug] 宿主重启(parent-gone)后 fold 锚点脱锚:hydrated 与 unified-shadow DIGEST DRIFT 同秒,缺口固化不自愈(0.1.197,与 #3013 重启路径互补) #3285 残留清扫,其外来行可能长期留存(与 master 一致;boot 水位重置仍会重建 delta 信任)。
  6. 是否破坏缓存:不涉及——水位语义不变(仍每 boot 清空)、delta 锚点逻辑不变、无缓存/重试/去重键改动。
  7. 是否改了设计方向:不涉及——清扫策略仍在存储层,lane 仍按位置消费,无新抽象、无配置面。
  8. 是否引入了新的配置:不涉及——inc:% 只是内部 session_ext key 前缀探测,非配置项。

验证证据(.tmp 双 worktree 定向跑,非全量):incremental-lane 16/16(原失败例转绿)、storage-ingest-watermark 14/14、storage-unified 14p/1s、storage-ext 5/5、issue3148-upstream-circuit 9/9、typecheck clean。master 侧对照:incremental-lane 16/16、issue3148 8/9(红,证明该红与本 PR 无关,已由分支内 0f11198 修正)。

状态:PR 头 d62bf55 的 CI check runs 刚触发(回复时尚未开始出结果),本地定向证据如上。CI 绿即可由人工合并:#3288 。我不会合并——系统规则禁止 agent 合 PR。

awork added 3 commits October 16, 2026 00:02
… full path sweeps foreign message rows (#3285)

Restart handoff with two processes sharing the state dir let a previous writer's index.db rows and last_ingested_seq watermark survive into the new process: the new writer inherited foreign delta trust (empty unverified deltas) and its full re-ingest kept foreign orphan rows via INSERT OR IGNORE + identity remap, so every sampled digest check flagged DIGEST DRIFT forever. The b9e82f6 heal (clear watermark on drift) was insufficient because the subsequent full re-ingest still kept orphans.

Fix: resetAllIngestWatermarks() at writer init (once per boot; readers/export untouched), and the ingestSession full path sweeps message rows absent from the plan when the plan is a complete raw-tail log (completeLog gate — envelope-only plans are pruned folded renders and must never trigger sweeps). Regression test: tests/storage-ingest-watermark.test.ts restart-handoff case (fails without the fix).
… baseSeq epoch consumes the messages table positionally (#3248, CI regression of the #3285 sweep)
…3285)

The e2e incremental-lane case covers the user-visible 409 loop; this unit
case pins the mechanism itself — a session carrying any "inc:" ext key
survives a completeLog ingest untouched, while a marker-less control
session still gets its foreign rows swept.
@ranxianglei
ranxianglei force-pushed the 2026-10-15_fix-3285-unified-dual-writer branch from d62bf55 to d6e703a Compare October 15, 2026 16:06
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(bot) 🏷 (ai: vllm-qwen/qwen3.8-27b)

冲突已解决并 force-push:新头 d6e703a,mergeable=true(现 blocked 仅等 CI)。

冲突面只有一处,且是「双方各修了一遍同一处」的良性重复:

结构化八问

  1. 是什么问题:系统转发的上游 PR 冲突——PR 分支落后 master 后 GitHub 报 dirty;实际内容冲突仅上述一处重复测试修正。
  2. 什么原因导致的:master 与本 PR 在重叠窗口内各自独立修了同一个 stale 期望([bug] 上游在会话中途不可用时,bili 把一次故障放大成四重代价:固定节奏 20 连击全部冷发、输出预算被静默钳制 83 次(最狠 65536→1360)、紧急收缩与 nudge 停摆均不可见 #3148 vs [bug] 0.1.195 输出预算三路饿死仍复现:无 output 上限模型 restore 无下限(1→252、fork 原样转发 1)、#2490 planning floor 钳到 1479–2623 → 3 会话 9 次 finish=length 空轮不重试(#546/#1665/#1840/#2904/#2756 残留) #3071 floor),文本级撞车;非语义分歧。
  3. 影响范围与严重程度:无行为影响——两侧功能改动逐字节等价(断言值相同),只是注释二选一。
  4. 怎么修复的:标准 rebase 到当前 master,冲突处保留 master 版本(union-of-intent 下信息量更大者),空提交按 rebase 语义丢弃,force-with-lease 推回 PR 头。
  5. 是否有回归问题:无新增——相对原头 d62bf55,代码零变化(仅少了一个与 master 重复的测试 commit);floor 5 已披露的行为变更清单(boot 水位重置、completeLog 清扫、lane-owned 会话豁免)原样成立。既有行为逐项结论沿用 floor 5 第 5 问,未复述。
  6. 是否破坏缓存:不涉及——水位/delta/去重逻辑一行未动。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

本地定向验证(§7.2,非全量):typecheck clean;tests/storage-ingest-watermark.test.ts + tests/issue3148-upstream-circuit.test.ts + tests/incremental-lane.test.ts 合计 39/39 pass(incremental-lane 即当初首版回归的守卫套件,在新 master 基上全绿)。全量以远程 CI 为准,CI 绿后可人工合并:#3288

@ranxianglei
ranxianglei merged commit d0ebe24 into master Oct 16, 2026
36 of 38 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.484-master.d0ebe24 (rolling master tag).

npm install -g billion-context@0.0.484-master.d0ebe24  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant