Skip to content

[PR] fix: full mode — label adoption converges the unified index row instead of minting a duplicate (#3831) - #3875

Closed
ranxianglei wants to merge 1 commit into
masterfrom
2026-10-19_label-adoption-dup-row
Closed

ranxianglei wants to merge 1 commit into
masterfrom
2026-10-19_label-adoption-dup-row

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

What

Sub-defect b of #3831 (the issue was rescoped to b after sub-defect a landed with #3819):

In unified FULL mode, a session persisted under a derived id with a native label is adopted under that label on first resume: loadEnvelope remaps the payload id to the label and importRecord runs on the remapped record — minting a second index row while the pre-adoption id-keyed row lingers forever (viewFor hits on the old id block any re-import, so nothing ever removes it). One conversation then carries two rows: double byte accounting in the bounded store, double web/CLI listing, split-brain eviction, and — once the historical file stops aliasing or a post-adoption save lands — a stale boot entry served alongside the fresh one.

How (host-only, no kernel change)

Convergence at every adoption seam in src/persist.ts:

  1. loadSync miss path — after an organic import succeeds, drop the aliased pre-adoption row (convergeAdoptedRow). New row committed first; a crash between the two re-creates the duplicate, which (2) heals.
  2. guardedBuild save path — same convergence after shadow.record on both fresh and stale-disk branches, covering post-adoption saves that would otherwise re-mint the twin row.
  3. boot scan — a twin file whose label is already owned by another (canonical) session skips the warm window, so a pre-adoption file can no longer be loaded under its dead id at boot.
  4. boot heal — pre-existing duplicates minted by pre-fix code are deleted when the label-owned row exists and is not older than the stale row (newer-twin race → warn-log, re-evaluated next boot).

The historical file itself is untouched — tests/bounded-storage.test.ts pins its byte-stability through adoption+save (#2803) and stays green.

Behavior change (old → new)

  • After label adoption: two permanent index rows → exactly one (the label-keyed row).
  • Boot: a pre-adoption twin file could be warmed under its dead id → now always served via the label only.
  • Duplicates created by pre-fix versions: permanent → healed automatically at the next boot.
  • UNCHANGED: shadow/legacy modes, envelope bytes, file layout, writerOff behavior, single-writer contract.
  • Known residual (documented, self-healing): an explicit runtime loadSync(oldId) against the still-present historical file legitimately re-imports its row (file-backed sessions stay addressable by file name); the boot heal removes it again at the next restart. Not addressed here — same failure domain but a different trigger surface than the adoption event this PR fixes.

Config surface

None. No new env var, config field, or CLI flag.

Out of scope

Verification

Part of #3831 (sub-defect b).

CI status note (2026-10-20)

Two fast gates fail on this head but are PRE-EXISTING master-state failures, not caused by this diff (this PR touches only src/persist.ts + one new test file):

Both are tracked in #3870 and fixed by PR #3871 (fully green: docs-sync, unused-exports, all test shards). While #3871 is unmerged these gates sit in the needs chain, so this PR's test/static-gates lanes were SKIPPED, not failed. Once #3871 merges, the two checks will be re-requested here so the full test lanes run against the updated merge ref.

…ad of minting a duplicate (#3831)

Sub-defect b of #3831: when a session persisted under a derived id is
adopted by its label, importRecord ran on the label-remapped payload and
minted a SECOND row while the pre-adoption id-keyed row lingered forever
(viewFor hits on the old id block re-import, so nothing ever removed it).
One conversation carrying two rows = double byte accounting, double web/CLI
listing, split-brain bounded eviction, and a stale boot entry once the
historical file stops aliasing.

Convergence at every adoption seam (host-only, no kernel change):
- loadSync miss path: after an organic import succeeds, drop the aliased
  pre-adoption row (new row committed first; crash between = healed next boot).
- guardedBuild save path: same convergence after shadow.record, covering
  post-adoption saves that would otherwise re-mint the twin row.
- boot scan: twin files of a canonical session skip the warm window, so a
  pre-adoption file can no longer be loaded under its dead id at boot.
- boot heal: pre-existing duplicates (minted by pre-fix code) are deleted
  when the label-owned row is present and not older than the stale row.

The historical file itself is untouched (#2803 pins byte-stability); only
index rows converge.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_label-adoption-dup-row (145682d)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4585-pr.3875.9e84e2a

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3875.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: full mode — label adoption converges the unified index row instead of minting a duplicate (#3831) [PR] fix: full mode — label adoption converges the unified index row instead of minting a duplicate (#3831) Oct 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 本 PR 是 #3831 清单里的子缺陷 b(采纳事件把统一索引行收敛到 label 而不是铸第二行);同家族的 a 已随 #3819(storage full flip)合入、c(createReplica/restoreReplica 丢孤儿原件)另行跟踪,#3841 是 kernel 侧 blockContents carrier——症状同家族但机制/边界不同,各自独立。本 PR 与任何在途条目无重复。

标签: bug、scope/storage、p2 —— 缺陷类;故障域在持久化/索引层(src/persist.ts 统一存储行管理);p2 判定:full mode + label 采纳场景下会话带两行永久索引(双倍字节记账/双列表/分裂驱逐/潜在陈旧 boot 条目),但 resume 本身可用、无数据丢失、增长有界(每次采纳事件多一行,非无限膨胀)——「能用但有明显问题」档。若你认为陈旧 boot 条目风险应升 p1,我改标。

开始审核:diff 已核对(仅 src/persist.ts +83 与新测试 +173,单 commit,parent 为 #3737 合入后的 master),四个收敛缝(loadSync miss / guardedBuild 双分支 / boot scan 跳过 / boot heal)的上下文正在逐段读;接下来本地跑定向测试 + typecheck,再看远端 CI。结论按八问体例跟进。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Oct 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审查结论:拦下,不建议合并(BLOCKED)——新回归测试 5 例中 2 例在本 head 上确定性失败;且修复漏了一条会让重复行复活的隐式 seam。方向本身是对的,细节如下。

① 方向审核(原 issue #3831-b)

方向正确:label adoption 应当收敛行而不是复制行。四个 seam(loadSync miss 收敛 / guardedBuild 保存后收敛 / boot scan 跳过 twin 预热 / boot heal 清存量重复)都打在了真实路径上,守卫齐全(alias 必须存在且异名、stats.disabled 逃生、sessionUpdatedAt(id)===null 防孤儿化唯一行、never throws、newer-twin 留给下次 boot)。历史文件字节稳定性(#2803 钉死的断言)保持通过。shadow/legacy 路径零触碰(全部 gated 在 readsUnified)。

② 本地验证 —— 关键证据

head 145682d93 上 npm install + npm run build 后跑 OP 点名的 6 个定向套件:bounded-storage-scale / bounded-storage / storage-mode-label / storage-unified / storage-json-writer-off 全绿(37 用例 34 过 / 1 skip 缺 better-sqlite3),但新文件 tests/issue3831-label-adoption-dup-row.test.ts 的 5 例中 2 例稳定失败(批量 + 单文件 ×2 + 插桩运行,共 ≥3 次,确定性复现):

  • case 1(line 95):期望 ["ses_native_id"],实际 ["old-derived-id", "ses_native_id"] —— 重复行回来了。
  • case 4(line 152):期望 ["pending-id"],实际 ["ses_pending"]。

即 OP 声称的 “Targeted suites green locally on the rebased head” 不可复现。CI 侧同样没有绿信号:本 head 所有含 test lane 的 suite 均 skipped/cancelled,mergeable_state=blocked(详见 ⑤)。

根因 1(case 1):#3050 R3 懒内容再物化 seam 未被覆盖

boot 本身 adopt+converge 是成功的(日志可见 label adoption converged index row old-derived-id -> ses_native_id,此刻索引只剩 label 行)。把重复行弄回来的是测试自己的 line 94:右值 session.blockContents.get("b1") 触碰了原始 session 对象——它的 blockContents 在 writeNow 后已被 dropResidentBlockContents(persist.ts:2280-2294,#3050 R3 内存优化)折成绑定 "old-derived-id" 的 LazyBlockContents。ensure() 经 loadSync("old-derived-id") 再物化 → full-mode miss(X 行已被收敛掉)→ 对仍然存在的 X 文件做有机导入 → X 行被重新铸造;而 convergeAdoptedRow(X) 是 no-op,因为 alias map 以 label 为键、X 名下没有条目。插桩栈确认:LazyBlockContents.load (src/persist.ts:2289) ← ensure (src/session.ts:67) ← get (src/session.ts:78) ← test line 94 col 82。同类路径(推断,未实测):任何对仍驻留的旧 id session 对象的 eviction flush / save 走 guardedBuild → shadow.record(record.id=X) 同样会重新铸行。

结论:PR 自己声明的不变量「adoption 后恰好一行」被这条 seam 打破。PR 文档化的 residual(“显式 runtime loadSync(oldId) 合法地重新导入其行”)与实现一致,但PR 自己的回归测试断言了超出该 residual 的范围——自相矛盾:要么修 seam,要么改测试。

根因 2(case 4):前提与已钉死行为冲突

boot warm loop 按设计会 adopt 任何带 label 的非 canonical 文件(alias 把 item 映射到 label → loadSync(label) → import+converge)——该行为已被 bounded-storage.test.ts case 12(#2803 “legacy adoption keeps the historical file; canonical copy written beside it”)钉住。因此一个孤立的带 label 文件不可能在未 adopt 的状态下活过 boot(除非 sidecar 降级 / 非 full / 文件落在 warm window 外 / 文件名不匹配,本例均不成立)。case 4 在任何环境都不通过,需重写前提,例如:boot 用 maxBytes:1 让文件落出 warm window → boot 不 adopt → 行存活 → 显式 loadSync(label) adopt+converge——保留原 negative-twin 意图(heal 不得触碰 label 无行的唯一行)。

③ 回归审计(diff 触碰路径的既有行为逐项)

  • shadow/legacy 模式:heal/converge 全 gated readsUnified(full-only),legacy 分支未动 ✅
  • 信封字节 / 文件布局:未变;X 文件字节稳定(case 1 的 Bound startup session loading and preserve historical recovery #2803 断言通过)✅
  • writerOff / 单写者契约:未动 ✅
  • boot 流程顺序:heal 先于 seedIndexedAliases 与文件扫描;twin-skip 在 windowBytes 记账之前(twin 文件不再占 warm window 预算——有意为之);后台 remainder scan 只做 alias 记账、不 import、不能铸行 ✅
  • 行为变更均为 OP 已披露项(两行→一行;boot 不再以死 id 预热 twin;存量重复下次 boot 自愈),无未披露漂移 ✅
  • 唯一缺口即根因 1 的隐式 seam——属修复不完整,不是无意回归。

④ 测试有效性

#3617 正负成对的门纪律形式正确;但 5 例中 2 例在 head 上不通过,「回归测试钉住 bug」的承诺当前不成立;case 4 属假测试(断言了一个系统里不会出现的状态)。

⑤ 顺带发现(与本 PR 无关,另行处理)

master 自身两条 required lane 是红的,且在 base commit 上同样失败(非本 PR 引入):docs-sync(CONFIGURATION.md/zh-CN 相对 config-reference yaml 过期:残留旧的 storage.unified … unset (shadow) 行与缺 #3788 冷却描述的旧 storage.sqliteAutoDownload 行各一份)与 unused-exports(src/log-mask.ts: maskUrlForPublish 导出未用且不在豁免表)。它们阻塞一切 PR 的合并。我将按流程单独立项 + 提小修 PR,不在本线程展开。

结构化八问

  1. 是什么问题:unified FULL 模式下,会话以派生 id 落盘、带原生 label,首次 resume 按 label adopt 时索引铸了第二行,预 adopt 的旧 id 行永远没人删 → 双份字节记账、双份 web/CLI 列表、分裂脑驱逐。
  2. 什么原因导致的:loadEnvelope 把 payload id 重映射为 label 后 importRecord 提交 label 新行,旧 id 行没有任何删除机制(viewFor 命中旧 id 又挡住重新导入)。本 PR 覆盖了四个 adoption seam,但漏了 Memory plane (R3): stop keeping compressed originals, CCR and snapshots fully in RAM now that the unified store has them — measure first, then lazy-load #3050 R3 懒内容再物化这条隐式触发路径(以及同类的 eviction flush/save 路径)。
  3. 影响范围与严重程度:影响所有走「派生 id + 原生 label」adopt 的 full-mode 会话;用户面表现为列表/记账翻倍、boot 陈旧条目、字节压力下驱逐决策失真。数据可从仍在写的 JSON 信封恢复,不构成丢失 → p2(能用但有明显问题;若后续证实驱逐失真会造成实际数据不可达,应升 p1)。
  4. 怎么修复的:在每个 adoption seam 上新行先行提交、随后删除被取代的旧行 + alias 条目;boot 扫描拒绝 twin 预热;boot heal 清理存量重复(新者胜则 warn 留待下轮)。缺的是隐式 seam 的收口(见下方选项)。
  5. 是否有回归问题:审过 shadow/legacy、信封字节、writerOff、boot 顺序、warm window 记账——无无意回归;行为变更全部已披露。修复自身的不变量未达成(根因 1),属不完整而非回归。
  6. 是否破坏缓存:不涉及(前缀缓存/重试策略未触碰);bounded store 驱逐决策的修正正是本 PR 目标。
  7. 是否改了设计方向:不涉及——host-only、无 kernel 变更、单写者契约保留。
  8. 是否引入新配置:无(diff 核实无 env/config/flag 新增,与 OP 声明一致)。

需要你拍板的选项

  • A(推荐):收口隐式 seam。full-mode loadSync miss 路径在有机导入前,检测请求 id 是否为某个存活 label 行的被取代 pre-adoption twin(可用信号:索引行的 label 列、文件内 meta.label、sessionSummaries),命中则改道到 label 行或抑制导入,不再以死 id 铸行。效果:「一会话一行」硬成立,case 1 原样通过。行为变更披露:显式 loadSync(oldId) 从「会重新导入其行、下次 boot 自愈」变为「永不以死 id 铸行,一律经 label 服务」(old→new)。代价:miss 路径多一次检测。
  • B:保留文档化 residual。case 1 改为不触碰旧对象的懒内容(与字面值比较),范围最小;懒 seam 的 churn 留在已知 residual(每次 boot 自愈)。
  • 无论 A/B:case 4 前提都要按 ②根因 2 重写。

拍板后我立即在本分支实施(含 case 4 重写)并重跑定向套件回报。master 侧两条红 lane 的独立小修并行推进,不影响本 PR 决策。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

补充(CI 红灯跟踪线,与上方本 PR 的审核结论相互独立):

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 红灯跟踪线进展:修复 PR 已开出 → #3883(Fixes #3881,分支 2026-10-20_master-ci-gates,从 master tip 39fb841d3 切出,两个小修各一个 commit)。远端分支 tip 与本地验证过的树 SHA 一致。#3883 合入后 master 的 docs-sync / unused-exports 转绿,本 PR 即可解除 blocked——届时按上方选项 A/B 的决策更新分支后,CI 全绿即具备合并条件。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

孤儿票清理:标题指向的 issue 已关闭,修复已由其他路径落地。关单;误判请 reopen。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant