Skip to content

[PR] fix: restore archived originals dropped by full-mode unified reads (#3831) - #3835

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-10-19_full-orphan-blockcontents-guard
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-10-19_full-orphan-blockcontents-guard

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Host-side stopgap (option B, per the recommendation in #3831) for the full-mode reload loss: a session persisted with state.blocks empty but blockContents non-empty (zombie-reaped folds #2695, compaction-archive prunes) served an EMPTY blockContents from the unified view after a clean boot — archived originals were gone from the served session, so decompress could no longer restore them. Pre-existing mechanism, exposed by the default flip in #3819: the unified schema has no standalone blockContents carrier, and ingest rebuilds the view's map only from block rows (src/storage/ingest.ts loadLegacyView), so entries without a blocks slot have nowhere to ride.

How

Every full-mode serving path (loadSync hit + miss, buildForLoadAll boot warm-up) now compares the parsed record's blockContents keys against the view's before serving the view; when the envelope still holds keys the view dropped, it serves the envelope through the new serveFromEnvelope helper — the single home for the original single-buildSession(payload) semantics (plus the #408 clamp rewrite and cold-load log), previously duplicated across two branches. A warn log marks each escape-hatch firing on the cold path.

Behavior change (old → new)

  • Full mode (BILI_STORAGE_UNIFIED=full or config storage.unified: full; becomes the default once [PR] storage: default unified mode flips to full + six #3135 cleanups (#3818) #3819 lands): reloading a session whose record carries orphaned blockContents entries used to serve the lossy unified view (originals silently dropped); it now serves the legacy envelope bytes, so decompress can restore them again.
  • Sessions WITHOUT orphaned entries are unaffected — still served from the index, including index-only reads where the envelope file is absent (pinned by the negative test).
  • Shadow/legacy default modes: no change (the guard lives only inside the readsUnified branches).
  • writerOff users (BILI_STORAGE_JSON_WRITER=off) have no envelope file: unchanged, best effort — the known coverage limit of option B stated in the issue.

Config surface

None — no new env vars, config fields, or defaults.

Out of scope (tracked in #3831)

  • Sub-defect b: lazy label adoption mints a duplicate index row under the label key — separate mechanism facet, stays on the issue checklist.
  • Option A (root fix): a standalone blockContents carrier in the unified schema — kernel change, human-gated, needs owner decision.

Verification

Part of #3831 (sub-defect a).

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-19_full-orphan-blockcontents-guard (c7737b4)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4544-pr.3835.cec6b2e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3835.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: restore archived originals dropped by full-mode unified reads (#3831) [PR] fix: restore archived originals dropped by full-mode unified reads (#3831) Oct 19, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #3831 子缺陷 a 的止血实现(选项 B),与 #3819(默认翻转,只暴露不修复)机制不同;out-of-scope 项(子缺陷 b、选项 A)均留在 #3831 checklist,无并列重复 PR。

标签: bug、scope/storage、p2(full 模式重载丢归档原文;#3819 翻默认尚未合入故当前潜伏,落地后 decompress 恢复不了归档原文但会话仍可用 → p2)。

开始独立审核:拉取 PR 分支核对代码、逐项过回归面、跑定向测试套件、交叉验证 #3819 树上的两个红测试。结论稍后按八问体例给出。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

独立审核完成:rebase 到当前 master(85c27f0eb)验证——PR 落后 8 个提交,无一触碰 src/persist.ts / src/storage/,rebase 零冲突。定向套件 + typecheck 全绿,RED-on-unpatched 已本地复验。判定:可合并(等新 head c7737b4 的 CI test shards 转绿)。审核中发现两处小问题已直接修掉,push 到 PR 分支(两个独立标注 commit)。

1. 是什么问题:full 模式下,持久化记录里 state.blocks 空/缺 slot、但 blockContents 仍持有归档原文的会话(zombie-reaped folds #2695、compaction-archive prunes),重载后从 unified view 拿到空 blockContents,decompress 再也恢复不了原文。

2. 什么原因导致的:unified schema 没有 blockContents 独立载体,ingest.ts:465-470 只从 block 行重建 view map——没有 blocks slot 的记录键无处安放。既有机制,被 #3819 默认翻转暴露。方向正确:本 PR 就是 #3831 推荐的 option B(host 侧止血),不碰 kernel。

3. 影响范围与严重程度:仅 full 模式且含孤儿条目的会话在重载时受影响(#3819 合入后 = 全体默认用户)。p2:会话可用,但归档原文不可再恢复;非数据损坏(envelope 字节完好,改的是读取来源)。

4. 怎么修复的:三条 serving 路径(loadSync hit/miss、buildForLoadAll boot/export warm-up)在 serve view 前比较 record 与 view 的 blockContents 键集;envelope 仍持有 view 丢掉的键 → 回退 serve legacy envelope(收敛进新 serveFromEnvelope,保留原单 buildSession 语义 + #408 clamp + cold-load log)。守卫单向(record 有而 view 无才触发):sparse block 方向由 completeBlockContents 合成补齐、view 只会多不会少,不会误触发。

5. 是否有回归问题(触碰路径既有行为逐项清点):

  • 非孤儿 full hit:行为不变(afterLoad(unified));代价是每次调用多一次 loadEnvelope 读+parse——loadSync 只在冷启动/LRU 驱逐/idle-drop 触发,触发时会话必已空闲且上一轮已落盘,不会 serve 比 view 更新的陈旧字节;止血可接受(根治是 option A)。
  • 非孤儿 miss/import 失败:照旧回退 envelope,不变。
  • index-only 行(envelope 文件缺失):loadEnvelope null → 从 index serve,负向测试钉住。
  • shadow/legacy 模式:守卫只在 readsUnified 分支内,零变化。
  • plain legacy 分支重构进 serveFromEnvelope:字节语义不变(golden-storage-render goldens 原样通过)。
  • writerOff 用户:无 envelope 文件,不变(option B 已知覆盖上限,issue 已声明)。
  • 未发现无意回归;高危模式逐项过(fail-fast 未变吞错、阈值/时序/输出格式不变)。

6. 是否破坏缓存:不涉及——纯持久化读路径,prefix cache/重试/去重游标均不受影响。

7. 是否改了设计方向:否。止活在既有 dual-write 架构内;根治(option A,kernel schema 独立载体)保持 human-gated 待 owner 决策。

8. 是否引入了新的配置:无。

测试有效性(#3617 双向):正向 RED-on-unpatched 本地复验成立(master 上 fail、修复树 pass);负向按设计双世界皆绿。另发现并修复两处(已 push):

  1. test:(3cc9df17e)——原测试注释 "Boot warm path (buildForLoadAll)" 与事实不符:传 limits 的 boot 走 bootBounded→loadSync,buildForLoadAll 这条腿(无限 boot + export 共用的 loadAll)实际没被钉住。probe 实测该腿在 unpatched master 上确实丢原文(RED),补第三个用例钉死并修正注释。生产 boot 恒传 limits(session.ts:957),此腿的真实暴露面是 session export 路径。
  2. fix:(c7737b45e)——PR 描述称 "warn log marks each escape-hatch firing",实际只有 loadSync-hit 路径 warn。抽出 logOrphanFallback(id) 接上全部三条路径。(行为变更披露:miss/boot 路径孤儿回退现在也打 warn 行——纯观测增强,无语义变化。)

对 #3819 交叉验证(重要发现):#3819 自己的 review 轮已落地更深的修复(#3818)——shadow.ts 的 orphanContents() 谓词 + record()/importRecord() 跳过孤儿记录入索引(orphanSkips 计数)+ miss 路径直接 serve 新解析视图。合并世界里本 PR 的三个守卫不是冗余:它们保护存量 index 行(#3818 落地前已写入的行无法追溯修复,record() 的 skip-mirror 会让它们永久 stale)。合并树(#3819 head + 本 PR 三 commit,1 hunk 冲突按 union-of-intent 解决)实测:typecheck 干净;此前红的 bounded-storage-scale "320 MB archive boots under a 128 MB heap…" 与 bounded-storage "first migration adopts legacy labels lazily…" 双双转绿;#3819 自己的 storage-full-flip 套件全绿。合入顺序:谁先合另一个 rebase(与作者描述一致,即我实测的解法)。

CI:新 head c7737b4 静态门禁全绿(unused-exports / config-surface-guard / kernel-guard / docs-sync / version-guard / plan / lock-sync / gate),test shards(ubuntu/windows × node22/24)运行中。

结论:可合并——等 c7737b4 的 test shards 转绿即可合。子缺陷 b 与新发现的 replica 面(已补记 #3831 清单为子缺陷 c)留在 issue 跟踪。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 更新:新 head c7737b4 全部检查转绿——test shards 6/6(ubuntu/windows × node22/24)success,静态门禁全绿,无 failure。审核结论维持:可合并,请合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Superseded by #3819 (merged): the option-B stopgap landed there as an ingest-side orphan-contents guard (record/importRecord skip + envelope-served loads, orphanSkips observable, owner-pinned in 21b70f4 as "never gutted into the index") plus the writerOff safety valve (6d36b45). The remaining representation gap — kernel-side blockContents carrier — is tracked in #3841 options a/b (with the stale-row resurrection bug #3841 reports).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant