Skip to content

[PR] fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) - #7336

Open
ranxianglei wants to merge 5 commits into
masterfrom
2026-11-20_storage-poison-row-isolation
Open

ranxianglei wants to merge 5 commits into
masterfrom
2026-11-20_storage-poison-row-isolation

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

(ai: sglang-gpu0/qwen3.8-27b)

Fixes #7335
Fixes #7337

改了什么

一条坏数据行(CAS 哈希与字节不匹配 / JSON 解析不了)过去可以炸掉整个会话:冷加载抛异常 → legacy 回退失败 → 状态清零 → 裸上下文从零重压(实例:26 行毒数据 → 32 轮串行 preflight,用户侧卡死 12+ 分钟)。本 PR 关两个结构性缺口:

  1. 写入端拒写毒数据(src/storage/store.ts + src/storage/mysql.ts 的 putContent):调用方提供的摘要现在一律与 sha256(bytes) 复核,不一致即抛错、事务回滚。代价 = 每次写入多哈希一次实际写入的字节(增量稳态亚毫秒;全量重导入罕见且也只是几毫秒)。
  2. 读取端坏行隔离(src/storage/ingest.ts LegacyViewBuilder + src/storage/shadow.ts):解析失败的行跳过并点名记录,加载路径打响亮 warn(unified view degraded … N unparseable row(s) skipped);尾行计入既有 tailGaps。会话带伤加载,块/ref/压缩状态全保住。sidecar 校验器(verify-sample)同样隔离:坏行标 drift,不再让每次保存永久失败重试。

与 #7173(已合并)的关系:互补,不重合

#7173 修的是文件级损坏(双写者炸 WAL、.chk 完整性判词、webSummaryRows 的 json_valid 防护、启动期首写损坏分类);本 PR 修的是行级逻辑毒(CAS 哈希≠字节、单行 JSON 解析不了)——同一事故的两个不同层,文件级防护挡不住行级毒,反之亦然。已 rebase 到含 #7173 的最新 master,无冲突。

证据

  • 新回归测试 tests/storage-poison-row-isolation.test.ts 3/3 绿(毒哈希拒写且零残留;毒消息行 → 视图带 tailGaps 加载、块全在;毒块行 → 视图加载、消息全在)。
  • 定向回归(rebase 后):tests/storage-*.test.ts 全套 324 pass / 0 fail。
  • npm run typecheck 干净。全量套以远端 CI 为准。

评审提示

  • 触达存储写入 CAS 入口 + 冷加载/视图重建 + sidecar 校验,sqlite 与 mysql 两后端一致。
  • 不改 wire / 配置面 / 持久化格式版本;库中既有坏行不受影响。
  • 按 §7.4 属 fallback 行为变更档 → 请人工评审。

Review follow-up(2026-11-20,commit 34da63d)

评审发现原修复漏掉的两种行形状(都能继续炸掉整个会话加载 / 让每次保存失败——即事故的失败模式),本 commit 补上:

  1. 块内容 CAS 行隔离(src/storage/ingest.ts appendBlockContents):每块的 CAS 解析 summary/one 原本无隔离——坏块内容行会让 viewFor 与 resident-fields 读取抛异常。现按行隔离(与 addMessages/addBlocks 同一 skip-and-record sink)。自愈:blocks 写入路径是权威 upsert,下一次全量 ingest 会把该行重指向合成内容(completeBlockContents fallback),drift 收敛。
  2. verify-sample 块行隔离(src/storage/shadow.ts BatchedViewVerification.addBlocks):payload_json 与 summary/one CAS 字符串原本裸解析——坏块行会在 verify-sample 内抛异常、此后每次保存都失败(事故症状 feat: persistent + namespaced sessions, multi-tenant session isolation, stream robustness #5)。现改为标 drift(保持位置对齐)。

新增测试:2 个回归 case(坏块 summary CAS 行 → 视图加载、块文本缺失、fallback 生效;坏尾消息 CAS 行 + live record → verify-sample 标 drift 而非抛异常)。两者在修复前 head 上均失败。

行为变更披露(旧 → 新):

  • 归档导入(putContent(body, hash)):dump 内摘要与字节不符时,旧=照写(毒数据落库);新=抛错、导入失败(loud-fail,与本 PR 写入端哲学一致;罕见路径)。
  • 带坏块内容行的会话冷加载:旧=整体抛异常 → legacy 回退 → 状态清零(事故路径);新=视图带伤加载(坏块文本缺失,payload_json 内联 summary 兜底),下一次全量 ingest 收敛。
  • verify-sample 遇坏块行:旧=抛异常 → 每次保存失败;新=标 drift 走 heal 路径(watermark 清除 / drift latch / 升级)。

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-20_storage-poison-row-isolation (04399e5)

Latest build — install this exact version

npm install -g billion-context@0.0.11045-pr.7336.3b3c789

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
04399e5 billion-context@0.0.11045-pr.7336.3b3c789 build
0dc18f1 billion-context@0.0.11040-pr.7336.4c65e09 build
9d3d890 billion-context@0.0.11037-pr.7336.b302aa0 build
34da63d billion-context@0.0.11034-pr.7336.3d8c875 build
2e395a5 billion-context@0.0.11030-pr.7336.63a65df build
09d8e2d billion-context@0.0.11029-pr.7336.e7744cc build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr7336.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) [PR] fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) Nov 20, 2026
… in unified storage (#7335)

Field incident: 26 content rows of one pi session were poisoned by an
older dev build (CAS hash != stored bytes, JSON unparseable). The next
cold load threw out of LegacyViewBuilder, the legacy fallback found no
file, the state was discarded ('starting fresh') and a ~600K-token
context re-folded from scratch in 32 serial preflight summary rounds
(67-84s each) — a 12+ minute user-visible freeze.

Two structural gaps closed:
- writer (store.ts + mysql.ts putContent): caller-supplied digests are
  now verified against sha256(bytes) and a mismatch fails the save
  loudly instead of persisting a hash/bytes pair readers cannot use.
- reader (ingest.ts LegacyViewBuilder, shadow.ts): unparseable rows are
  skipped and recorded (skippedCorruptRows; tail rows counted into the
  existing tailGaps) and the load path logs which rows were dropped —
  the session loads degraded with all blocks/refs intact. The
  verify-sample verifier gets the same isolation (marks drift instead
  of failing every subsequent save).

Regression tests: tests/storage-poison-row-isolation.test.ts (poisoned
digest refused with full rollback; poisoned message row -> view loads
with tailGaps>=1 and all blocks; poisoned block payload -> view loads
with all messages).

Fixes #7335
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。检索 open+closed 的 issue/PR(poison row / content digest / CAS / 坏行隔离 等关键词)无其他条目触及同一机制;open [PR] 行中只有本 PR (#7336) 指向本线程,无并行 PR。

标签: 已按 triage 打好 bug + pi + scope/storage + p1(会话被清零属数据丢失类,按 2026-10-19 裁定一律 p1;事故客户端为 pi 会话)。

状态: 评审进行中。PR 分支(2026-11-20_storage-poison-row-isolation)已检出并通读 diff,CI 在 merge ref 上运行中(test 车道 in progress)。初步判断:方向正确(写入端摘要复核 + 读取端坏行隔离,正对 #7335 的两个结构性缺口),但审出同机制的两处缺口——block 的 summary/one CAS 内容与校验器(verify-sample)的 block 行仍无隔离解析,事故失败模式对这两种行形状仍然存活。我会在本分支补上(带回归测试),然后给出完整八问评审结论。

@ranxianglei ranxianglei added bug Something isn't working p1 pi scope/storage labels Nov 20, 2026
… verify-sample (#7335)

Review follow-up on the poison-row isolation PR: two row shapes the
original fix left unisolated, both still able to abort a whole session
load or fail every save (the incident's failure modes):

- appendBlockContents (ingest.ts) parsed each block's CAS-resolved
  summary/one without isolation; a corrupt block content row aborted
  the view load (viewFor) and the resident-fields read. Now parsed per
  row with the same skip-and-record sink as addMessages/addBlocks.
  Self-heal: the blocks write path is an authoritative upsert, so the
  next full ingest re-points the row at the synthesized content
  (completeBlockContents fallback) and the drift clears.

- BatchedViewVerification.addBlocks (shadow.ts) parsed payload_json and
  the summary/one CAS strings bare; a corrupt block row threw inside
  the verify-sample and failed every subsequent save (incident symptom
  #5). Now: payload parse failure marks state drift (positional
  alignment kept), summary/one parse failure marks blockContents
  drift. Message-row isolation already landed in the original commit.

Tests: two new regression cases in storage-poison-row-isolation
(corrupt block summary CAS row -> view loads, block text missing,
fallback applies; corrupt tail message CAS row + live record ->
verify-sample marks drift instead of throwing). Both fail on the
pre-fix head. storage-* family: 326 pass / 0 fail.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

评审结论

已完整评审本 PR,并在 PR 分支上补了原修复漏掉的两个缺口(commit 34da63d)。方向正确、机制成立;CI 绿 + 人工评审后可合。

验证过什么

  • 写入端(putContent 摘要复核):逐一核过所有调用方——合法调用方都对实际存储的字节做哈希(splitMediaContent / finishPlan 的 summary+one / ccr / media),无误伤路径。
  • 读取端(消息行 / 块 payload 行隔离):事故 26 行毒数据正是 message content 行,已覆盖;tailGaps 计数正确(resolveSeqs 按缺失 seq 逐条计 gap)。
  • 自愈机制:blocks 写入是权威 upsert,坏块行下次全量 ingest 重指向合成内容(收敛,无永久 drift);坏消息行 INSERT OR IGNORE 不重写,走 drift-latch 路径(已在 PR 正文披露)。

评审发现并已补的两个缺口(commit 34da63d,均已推上 PR 分支)

  1. 块内容 CAS 行(summary/one)无隔离:appendBlockContents 裸解析——坏块内容行仍会让 viewFor 抛异常,事故失败模式对该行形状仍在。现按行隔离(与 addMessages/addBlocks 同一 sink)。
  2. verify-sample 块行无隔离:BatchedViewVerification.addBlocks 裸解析 payload_json 与 summary/one——坏块行会在 verify-sample 内抛异常 → 此后每次保存都失败(事故症状 feat: persistent + namespaced sessions, multi-tenant session isolation, stream robustness #5)。现改为标 drift(保持位置对齐)。
  • 新增 2 个回归测试(在修复前 head 上均失败);storage-* 家族 326 pass / 0 fail;typecheck 干净。

残留(未修,已登记 #7335 作清单项,不另开 issue——同症状同机制)
session 行自身的 JSON(kstateJson/tailJson/metaJson 等)仍未隔离:坏 kstate = 压缩记账清零,属 fallback 语义产品决策(§7.4 人工档),建议与消息/块行相同的 degrade+loud-warn 模式,待你拍板。详见 #7335 评论。

行为变更(旧 → 新,已写入 PR 正文)

  • 归档导入:dump 内摘要与字节不符,旧=照写(毒数据落库);新=抛错、导入失败(罕见路径,与写入端哲学一致)。
  • 带坏块内容行的会话冷加载:旧=整体抛异常 → 状态清零(事故路径);新=视图带伤加载,下次全量 ingest 收敛。
  • verify-sample 遇坏块行:旧=抛异常、每次保存失败;新=标 drift 走 heal 路径。

CI:新 commit(34da63d)的 CI 在跑(旧 run 被本次 push 取消);本地 typecheck + storage 定向回归已绿。


八问

  1. 是什么问题:统一存储里一条坏数据行(CAS 哈希与字节不匹配 / JSON 解析不了)能炸掉整个会话——冷加载抛异常 → 状态清零 → 600K token 从零重压(用户侧卡死 12+ 分钟),且 sidecar verify-sample 此后每次保存都失败。
  2. 什么原因导致的:两个结构性缺口——写入端信任调用方提供的摘要、不与字节复核(旧 dev build 写入了 26 行毒数据);读取端逐行裸解析、无行级隔离(一条坏行 = 整个加载失败)。
  3. 影响范围与严重程度:只影响库中已有坏行的会话(写入端现已封死,不再产生新坏行);触发即 p1——会话状态清零重压 + 每次保存失败。
  4. 怎么修复的:写入端一律 sha256(bytes) 复核、不符即抛错回滚(拒写毒数据);读取端逐行解析、失败跳过并点名记录 + 响亮 warn,会话带伤加载(块/ref/压缩状态全保住);verify-sample 同样隔离并标 drift,走既有 heal 路径(watermark 清除 / drift latch / 升级)。
  5. 是否有回归问题:逐一审过既有行为——合法 putContent 调用方无误伤;tailGaps 计数不变;块 upsert 自愈不变;唯一行为变化是上面披露的 3 项(均有意,均为「丢不了会话」方向的 loud-fail/降级)。
  6. 是否破坏缓存:不涉及 prompt/前缀缓存;存储侧 ingest watermark 语义未变(verify-drift 路径原有的 watermark 清除语义未动)。
  7. 是否改了设计方向:未改——是既有「统一存储 + sidecar 校验」架构的补全,自愈依赖既有权威 upsert + drift-latch 机制,无转向。
  8. 是否引入了新的配置:无新 env/config/默认值。

判定:修复完整(含评审补的两个缺口)、测试钉住行为、行为变更已披露——CI 绿后建议人工评审合入(§7.4 fallback 行为变更档,合并按流程由你执行)。

ci-check added 3 commits November 20, 2026 12:49
…ft in the poison-row fixture (#7335)

Review follow-up on the two poison-row commits:

- The seed fixture stamped its messages with `ts`, which never rides the
  stored content bytes (#4313: ts is a dedicated column). Every
  verify-sample therefore digest-drifted on `fields: ts` regardless of
  any corruption — the message-row verify test passed for the wrong
  reason: its drift came from the fixture, not from the poisoned row it
  claims to pin. Drop the stamp so the drift assertions are anchored to
  the corruption.

- The block-row isolation added to BatchedViewVerification.addBlocks
  (payload/summary/one parse failures mark drift instead of throwing)
  had no regression test — the new test 4 pins the view-load side
  (appendBlockContents), the message-row test pins the addMessages side
  (already landed in the first commit). Add the block-row side: a
  poisoned block summary CAS body must surface as digest drift on the
  sampled read-back, not as a verify-sample throw that fails every save.

Verified: tests 4 and 6 fail on the pre-fix head (2e395a5), all six
pass on the fixed head.
…7335)

CI (windows-latest, node 22, shard 4): all five storage-poison-row tests
failed with failureType=hookFailed — EBUSY unlinking the fixture's
index.db in t.after. The bare rmSync retry budget (5x100ms) lost the
Windows AV/indexer race (#2799/#2908 family); the assertions themselves
passed. Switch the teardown to the repo-standard rmrf (tests/tmp-rm.ts)
exponential ladder, same as every other suite that leaves a fresh db
file behind.
…rder, not AV (#7335)

node:test runs t.after hooks in REGISTRATION order (FIFO), so the fixture's
hook (store.close; db.close; rmrf) ran BEFORE the test's shadow.close() in
every shadow test: rmrf deleted index.db while the shadow's sqlite
connection was still open — a deterministic EBUSY on Windows (unlink of an
open file) that the 9.6s rmrf ladder cannot out-wait, and invisible on
Linux where unlink of an open file succeeds. Test 1 (no shadow) passed;
tests 2-6 failed in both CI runs and both shards, always at the full-ladder
duration (~9.9s).

Split the teardown: the fixture hook now closes only the fixture's own
connections; each test's later-registered hook closes the shadow (if any)
and runs rmrf. FIFO order guarantees every handle is released before the
delete; rmrf keeps the exponential ladder for the genuine external
AV/indexer race (#2799/#2908 family).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment