Repository navigation
[PR] fix: refuse unverifiable content digests and isolate unparseable rows in unified storage (#7335) - #7336
ranxianglei wants to merge 5 commits into
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.11045-pr.7336.3b3c789This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr7336.tgz
npm install -g packageInstall instructions are refreshed on each push. |
… in unified storage (#7335) Field incident: 26 content rows of one pi session were poisoned by an older dev build (CAS hash != stored bytes, JSON unparseable). The next cold load threw out of LegacyViewBuilder, the legacy fallback found no file, the state was discarded ('starting fresh') and a ~600K-token context re-folded from scratch in 32 serial preflight summary rounds (67-84s each) — a 12+ minute user-visible freeze. Two structural gaps closed: - writer (store.ts + mysql.ts putContent): caller-supplied digests are now verified against sha256(bytes) and a mismatch fails the save loudly instead of persisting a hash/bytes pair readers cannot use. - reader (ingest.ts LegacyViewBuilder, shadow.ts): unparseable rows are skipped and recorded (skippedCorruptRows; tail rows counted into the existing tailGaps) and the load path logs which rows were dropped — the session loads degraded with all blocks/refs intact. The verify-sample verifier gets the same isolation (marks drift instead of failing every subsequent save). Regression tests: tests/storage-poison-row-isolation.test.ts (poisoned digest refused with full rollback; poisoned message row -> view loads with tailGaps>=1 and all blocks; poisoned block payload -> view loads with all messages). Fixes #7335
09d8e2d to
2e395a5
Compare
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。检索 open+closed 的 issue/PR(poison row / content digest / CAS / 坏行隔离 等关键词)无其他条目触及同一机制;open [PR] 行中只有本 PR (#7336) 指向本线程,无并行 PR。 标签: 已按 triage 打好 bug + pi + scope/storage + p1(会话被清零属数据丢失类,按 2026-10-19 裁定一律 p1;事故客户端为 pi 会话)。 状态: 评审进行中。PR 分支(2026-11-20_storage-poison-row-isolation)已检出并通读 diff,CI 在 merge ref 上运行中(test 车道 in progress)。初步判断:方向正确(写入端摘要复核 + 读取端坏行隔离,正对 #7335 的两个结构性缺口),但审出同机制的两处缺口——block 的 summary/one CAS 内容与校验器(verify-sample)的 block 行仍无隔离解析,事故失败模式对这两种行形状仍然存活。我会在本分支补上(带回归测试),然后给出完整八问评审结论。 |
… verify-sample (#7335) Review follow-up on the poison-row isolation PR: two row shapes the original fix left unisolated, both still able to abort a whole session load or fail every save (the incident's failure modes): - appendBlockContents (ingest.ts) parsed each block's CAS-resolved summary/one without isolation; a corrupt block content row aborted the view load (viewFor) and the resident-fields read. Now parsed per row with the same skip-and-record sink as addMessages/addBlocks. Self-heal: the blocks write path is an authoritative upsert, so the next full ingest re-points the row at the synthesized content (completeBlockContents fallback) and the drift clears. - BatchedViewVerification.addBlocks (shadow.ts) parsed payload_json and the summary/one CAS strings bare; a corrupt block row threw inside the verify-sample and failed every subsequent save (incident symptom #5). Now: payload parse failure marks state drift (positional alignment kept), summary/one parse failure marks blockContents drift. Message-row isolation already landed in the original commit. Tests: two new regression cases in storage-poison-row-isolation (corrupt block summary CAS row -> view loads, block text missing, fallback applies; corrupt tail message CAS row + live record -> verify-sample marks drift instead of throwing). Both fail on the pre-fix head. storage-* family: 326 pass / 0 fail.
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 评审结论已完整评审本 PR,并在 PR 分支上补了原修复漏掉的两个缺口(commit 34da63d)。方向正确、机制成立;CI 绿 + 人工评审后可合。 验证过什么
评审发现并已补的两个缺口(commit 34da63d,均已推上 PR 分支)
残留(未修,已登记 #7335 作清单项,不另开 issue——同症状同机制) 行为变更(旧 → 新,已写入 PR 正文)
CI:新 commit(34da63d)的 CI 在跑(旧 run 被本次 push 取消);本地 typecheck + storage 定向回归已绿。 八问
判定:修复完整(含评审补的两个缺口)、测试钉住行为、行为变更已披露——CI 绿后建议人工评审合入(§7.4 fallback 行为变更档,合并按流程由你执行)。 |
…ft in the poison-row fixture (#7335) Review follow-up on the two poison-row commits: - The seed fixture stamped its messages with `ts`, which never rides the stored content bytes (#4313: ts is a dedicated column). Every verify-sample therefore digest-drifted on `fields: ts` regardless of any corruption — the message-row verify test passed for the wrong reason: its drift came from the fixture, not from the poisoned row it claims to pin. Drop the stamp so the drift assertions are anchored to the corruption. - The block-row isolation added to BatchedViewVerification.addBlocks (payload/summary/one parse failures mark drift instead of throwing) had no regression test — the new test 4 pins the view-load side (appendBlockContents), the message-row test pins the addMessages side (already landed in the first commit). Add the block-row side: a poisoned block summary CAS body must surface as digest drift on the sampled read-back, not as a verify-sample throw that fails every save. Verified: tests 4 and 6 fail on the pre-fix head (2e395a5), all six pass on the fixed head.
…7335) CI (windows-latest, node 22, shard 4): all five storage-poison-row tests failed with failureType=hookFailed — EBUSY unlinking the fixture's index.db in t.after. The bare rmSync retry budget (5x100ms) lost the Windows AV/indexer race (#2799/#2908 family); the assertions themselves passed. Switch the teardown to the repo-standard rmrf (tests/tmp-rm.ts) exponential ladder, same as every other suite that leaves a fresh db file behind.
…rder, not AV (#7335) node:test runs t.after hooks in REGISTRATION order (FIFO), so the fixture's hook (store.close; db.close; rmrf) ran BEFORE the test's shadow.close() in every shadow test: rmrf deleted index.db while the shadow's sqlite connection was still open — a deterministic EBUSY on Windows (unlink of an open file) that the 9.6s rmrf ladder cannot out-wait, and invisible on Linux where unlink of an open file succeeds. Test 1 (no shadow) passed; tests 2-6 failed in both CI runs and both shards, always at the full-ladder duration (~9.9s). Split the teardown: the fixture hook now closes only the fixture's own connections; each test's later-registered hook closes the shadow (if any) and runs rmrf. FIFO order guarantees every handle is released before the delete; rmrf keeps the exponential ladder for the genuine external AV/indexer race (#2799/#2908 family).
(ai: sglang-gpu0/qwen3.8-27b)
Fixes #7335
Fixes #7337
改了什么
一条坏数据行(CAS 哈希与字节不匹配 / JSON 解析不了)过去可以炸掉整个会话:冷加载抛异常 → legacy 回退失败 → 状态清零 → 裸上下文从零重压(实例:26 行毒数据 → 32 轮串行 preflight,用户侧卡死 12+ 分钟)。本 PR 关两个结构性缺口:
src/storage/store.ts+src/storage/mysql.ts的putContent):调用方提供的摘要现在一律与sha256(bytes)复核,不一致即抛错、事务回滚。代价 = 每次写入多哈希一次实际写入的字节(增量稳态亚毫秒;全量重导入罕见且也只是几毫秒)。src/storage/ingest.tsLegacyViewBuilder+src/storage/shadow.ts):解析失败的行跳过并点名记录,加载路径打响亮 warn(unified view degraded … N unparseable row(s) skipped);尾行计入既有tailGaps。会话带伤加载,块/ref/压缩状态全保住。sidecar 校验器(verify-sample)同样隔离:坏行标 drift,不再让每次保存永久失败重试。与 #7173(已合并)的关系:互补,不重合
#7173 修的是文件级损坏(双写者炸 WAL、
.chk完整性判词、webSummaryRows的 json_valid 防护、启动期首写损坏分类);本 PR 修的是行级逻辑毒(CAS 哈希≠字节、单行 JSON 解析不了)——同一事故的两个不同层,文件级防护挡不住行级毒,反之亦然。已 rebase 到含 #7173 的最新 master,无冲突。证据
tests/storage-poison-row-isolation.test.ts3/3 绿(毒哈希拒写且零残留;毒消息行 → 视图带 tailGaps 加载、块全在;毒块行 → 视图加载、消息全在)。tests/storage-*.test.ts全套 324 pass / 0 fail。npm run typecheck干净。全量套以远端 CI 为准。评审提示
Review follow-up(2026-11-20,commit 34da63d)
评审发现原修复漏掉的两种行形状(都能继续炸掉整个会话加载 / 让每次保存失败——即事故的失败模式),本 commit 补上:
src/storage/ingest.tsappendBlockContents):每块的 CAS 解析 summary/one 原本无隔离——坏块内容行会让viewFor与 resident-fields 读取抛异常。现按行隔离(与 addMessages/addBlocks 同一 skip-and-record sink)。自愈:blocks 写入路径是权威 upsert,下一次全量 ingest 会把该行重指向合成内容(completeBlockContents fallback),drift 收敛。src/storage/shadow.tsBatchedViewVerification.addBlocks):payload_json 与 summary/one CAS 字符串原本裸解析——坏块行会在 verify-sample 内抛异常、此后每次保存都失败(事故症状 feat: persistent + namespaced sessions, multi-tenant session isolation, stream robustness #5)。现改为标 drift(保持位置对齐)。新增测试:2 个回归 case(坏块 summary CAS 行 → 视图加载、块文本缺失、fallback 生效;坏尾消息 CAS 行 + live record → verify-sample 标 drift 而非抛异常)。两者在修复前 head 上均失败。
行为变更披露(旧 → 新):
putContent(body, hash)):dump 内摘要与字节不符时,旧=照写(毒数据落库);新=抛错、导入失败(loud-fail,与本 PR 写入端哲学一致;罕见路径)。