Repository navigation
[PR] fix: restore archived originals dropped by full-mode unified reads (#3831) - #3835
ranxianglei wants to merge 3 commits into
Conversation
…s archived originals (#3831)
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4544-pr.3835.cec6b2eThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3835.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复 —— 本 PR 是 #3831 子缺陷 a 的止血实现(选项 B),与 #3819(默认翻转,只暴露不修复)机制不同;out-of-scope 项(子缺陷 b、选项 A)均留在 #3831 checklist,无并列重复 PR。 标签: bug、scope/storage、p2(full 模式重载丢归档原文;#3819 翻默认尚未合入故当前潜伏,落地后 decompress 恢复不了归档原文但会话仍可用 → p2)。 开始独立审核:拉取 PR 分支核对代码、逐项过回归面、跑定向测试套件、交叉验证 #3819 树上的两个红测试。结论稍后按八问体例给出。 |
(ai: vllm-qwen/qwen3.8-27b) 独立审核完成:rebase 到当前 master(85c27f0eb)验证——PR 落后 8 个提交,无一触碰 src/persist.ts / src/storage/,rebase 零冲突。定向套件 + typecheck 全绿,RED-on-unpatched 已本地复验。判定:可合并(等新 head c7737b4 的 CI test shards 转绿)。审核中发现两处小问题已直接修掉,push 到 PR 分支(两个独立标注 commit)。 1. 是什么问题:full 模式下,持久化记录里 2. 什么原因导致的:unified schema 没有 blockContents 独立载体,ingest.ts:465-470 只从 block 行重建 view map——没有 blocks slot 的记录键无处安放。既有机制,被 #3819 默认翻转暴露。方向正确:本 PR 就是 #3831 推荐的 option B(host 侧止血),不碰 kernel。 3. 影响范围与严重程度:仅 full 模式且含孤儿条目的会话在重载时受影响(#3819 合入后 = 全体默认用户)。p2:会话可用,但归档原文不可再恢复;非数据损坏(envelope 字节完好,改的是读取来源)。 4. 怎么修复的:三条 serving 路径(loadSync hit/miss、buildForLoadAll boot/export warm-up)在 serve view 前比较 record 与 view 的 blockContents 键集;envelope 仍持有 view 丢掉的键 → 回退 serve legacy envelope(收敛进新 serveFromEnvelope,保留原单 buildSession 语义 + #408 clamp + cold-load log)。守卫单向(record 有而 view 无才触发):sparse block 方向由 completeBlockContents 合成补齐、view 只会多不会少,不会误触发。 5. 是否有回归问题(触碰路径既有行为逐项清点):
6. 是否破坏缓存:不涉及——纯持久化读路径,prefix cache/重试/去重游标均不受影响。 7. 是否改了设计方向:否。止活在既有 dual-write 架构内;根治(option A,kernel schema 独立载体)保持 human-gated 待 owner 决策。 8. 是否引入了新的配置:无。 测试有效性(#3617 双向):正向 RED-on-unpatched 本地复验成立(master 上 fail、修复树 pass);负向按设计双世界皆绿。另发现并修复两处(已 push):
对 #3819 交叉验证(重要发现):#3819 自己的 review 轮已落地更深的修复(#3818)——shadow.ts 的 orphanContents() 谓词 + record()/importRecord() 跳过孤儿记录入索引(orphanSkips 计数)+ miss 路径直接 serve 新解析视图。合并世界里本 PR 的三个守卫不是冗余:它们保护存量 index 行(#3818 落地前已写入的行无法追溯修复,record() 的 skip-mirror 会让它们永久 stale)。合并树(#3819 head + 本 PR 三 commit,1 hunk 冲突按 union-of-intent 解决)实测:typecheck 干净;此前红的 bounded-storage-scale "320 MB archive boots under a 128 MB heap…" 与 bounded-storage "first migration adopts legacy labels lazily…" 双双转绿;#3819 自己的 storage-full-flip 套件全绿。合入顺序:谁先合另一个 rebase(与作者描述一致,即我实测的解法)。 CI:新 head c7737b4 静态门禁全绿(unused-exports / config-surface-guard / kernel-guard / docs-sync / version-guard / plan / lock-sync / gate),test shards(ubuntu/windows × node22/24)运行中。 结论:可合并——等 c7737b4 的 test shards 转绿即可合。子缺陷 b 与新发现的 replica 面(已补记 #3831 清单为子缺陷 c)留在 issue 跟踪。 |
|
Superseded by #3819 (merged): the option-B stopgap landed there as an ingest-side orphan-contents guard (record/importRecord skip + envelope-served loads, orphanSkips observable, owner-pinned in 21b70f4 as "never gutted into the index") plus the writerOff safety valve (6d36b45). The remaining representation gap — kernel-side blockContents carrier — is tracked in #3841 options a/b (with the stale-row resurrection bug #3841 reports). |
What
Host-side stopgap (option B, per the recommendation in #3831) for the full-mode reload loss: a session persisted with
state.blocksempty butblockContentsnon-empty (zombie-reaped folds #2695, compaction-archive prunes) served an EMPTYblockContentsfrom the unified view after a clean boot — archived originals were gone from the served session, so decompress could no longer restore them. Pre-existing mechanism, exposed by the default flip in #3819: the unified schema has no standalone blockContents carrier, and ingest rebuilds the view's map only from block rows (src/storage/ingest.tsloadLegacyView), so entries without a blocks slot have nowhere to ride.How
Every full-mode serving path (
loadSynchit + miss,buildForLoadAllboot warm-up) now compares the parsed record'sblockContentskeys against the view's before serving the view; when the envelope still holds keys the view dropped, it serves the envelope through the newserveFromEnvelopehelper — the single home for the original single-buildSession(payload)semantics (plus the #408 clamp rewrite and cold-load log), previously duplicated across two branches. Awarnlog marks each escape-hatch firing on the cold path.Behavior change (old → new)
BILI_STORAGE_UNIFIED=fullor configstorage.unified: full; becomes the default once [PR] storage: default unified mode flips to full + six #3135 cleanups (#3818) #3819 lands): reloading a session whose record carries orphanedblockContentsentries used to serve the lossy unified view (originals silently dropped); it now serves the legacy envelope bytes, so decompress can restore them again.readsUnifiedbranches).BILI_STORAGE_JSON_WRITER=off) have no envelope file: unchanged, best effort — the known coverage limit of option B stated in the issue.Config surface
None — no new env vars, config fields, or defaults.
Out of scope (tracked in #3831)
Verification
tests/issue3831-orphan-blockcontents.test.tspins both gate directions (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617): positive (orphaned contents restored on boot-warm AND cold load under explicit full; RED on the unpatched tree) and negative (a non-orphan session keeps being served from the index even with its envelope file deleted — passes with and without the fix by design).Part of #3831 (sub-defect a).