Repository navigation
[PR] fix: isolate block summary/one CAS content parse in the unified storage reader (#7337) - #7338
ranxianglei wants to merge 2 commits into
Conversation
… in unified storage (#7335) Field incident: 26 content rows of one pi session were poisoned by an older dev build (CAS hash != stored bytes, JSON unparseable). The next cold load threw out of LegacyViewBuilder, the legacy fallback found no file, the state was discarded ('starting fresh') and a ~600K-token context re-folded from scratch in 32 serial preflight summary rounds (67-84s each) — a 12+ minute user-visible freeze. Two structural gaps closed: - writer (store.ts + mysql.ts putContent): caller-supplied digests are now verified against sha256(bytes) and a mismatch fails the save loudly instead of persisting a hash/bytes pair readers cannot use. - reader (ingest.ts LegacyViewBuilder, shadow.ts): unparseable rows are skipped and recorded (skippedCorruptRows; tail rows counted into the existing tailGaps) and the load path logs which rows were dropped — the session loads degraded with all blocks/refs intact. The verify-sample verifier gets the same isolation (marks drift instead of failing every subsequent save). Regression tests: tests/storage-poison-row-isolation.test.ts (poisoned digest refused with full rollback; poisoned message row -> view loads with tailGaps>=1 and all blocks; poisoned block payload -> view loads with all messages). Fixes #7335
… aborts the session load #7335 isolated the message-content and block-payload parses in the unified storage reader, but the block summary/one CAS content (appendBlockContents) was left with a bare JSON.parse. A corrupt summary/one row — bad CAS bytes, or a missing CAS row leaving summary="" — threw out of the shared helper and aborted the whole cold load on BOTH the full-view and resident-field paths (the session fell back to the legacy file, or was discarded when none exists). appendBlockContents now skips an unparseable row and returns it; the full-view path folds it into skippedCorruptRows (logged via warnCorruptSkips) and the resident-field path logs it. The rest of the session loads degraded, with all blocks/refs/messages intact — consistent with #7335's isolation. Regression tests: tests/storage-poison-row-isolation.test.ts (unit: corrupt summary/one rows skipped+returned; full-view: one corrupt summary CAS row degrades the view; resident-field: same). Verified the new tests fail on the pre-fix code. Fixes #7337
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.11036-pr.7338.23240f8This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr7338.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 本线程是 PR #7338 的镜像(带 This is a What I verified
CI status (live): in progress — 11 checks pending (test shards ubuntu/windows × node 22/24, static-gates, mysql-contract), 0 failed so far. The full-suite verdict is CI's; my local run was targeted only (per local-testing discipline). I'll follow up when CI completes. Dependency note: this branch is based on #7336 (still open), so its diff vs master currently includes #7336's changes. Per the PR description the author will rebase onto master once #7336 merges, shrinking the diff to just this fix. #7336 must merge first. Labels: already applied (bug, p1, scope/storage) — correct for this diff. 结构化八问
合并判定:代码干净、修复完整且最小、回归测试真实(pre-fix 失败 / fixed 通过)、typecheck 绿。待 CI 全量绿后可合(当前 CI 进行中、0 失败);前置条件为 #7336 先合、本 PR 随后 rebase 到 master 收敛 diff。 |
|
(ai: vllm-qwen/qwen3.8-27b) Superseded by PR #7336 — labelled, not closing (close is owner-only). While this PR was in flight, PR #7336 landed a review follow-up (commit 34da63d) that independently found and fixed the same gap this PR targets — So #7336's fix is a strict superset of this one (same Recommendation: close this as superseded and let #7336 carry the fix. Requesting owner action. |
(ai: vllm-qwen/qwen3.8-27b) Independently verified — the supersession is correct. I did not take the recommendation at face value; I examined #7336's actual code and confirmed the conflict. What I checked
One nuance (test coverage only, not the fix): #7338 carries two tests #7336's branch does not — a direct unit test of I can't close #7338 (close is owner-only). The verification supports your recommendation: close this as superseded and let #7336 carry the fix. CI (live): both still in progress, 0 failed so far — #7336: 12 pending; #7338: 11 pending. #7336's own commit reports 结构化八问
判定:supersede 成立,可关 #7338(关闭动作归 owner)。唯一可选增值:把 #7338 的两个测试补进 #7336 分支(需你点头)。 |
(ai: vllm-qwen/qwen3.8-27b) Status (re-verified live): nothing has changed since my last comment — the thread is waiting on your decision.
Where this stands (from my previous comment, unchanged):
Action needed from you: close #7338 as superseded (your call, owner-only). Optionally approve the 2-test port to #7336. |
What & why
#7335 (PR #7336) isolated the message-content and block-payload parses in the unified storage reader, but the block summary/one CAS content (
appendBlockContents,src/storage/ingest.ts) was left with a bareJSON.parse. A corrupt summary/one row — bad CAS bytes, or a missing CAS row leavingsummary=""— threw out of the shared helper and aborted the whole cold load on BOTH the full-view and resident-field paths (the session fell back to the legacy file, or was discarded when none exists).This closes the last gap in the poison-row isolation #7335 started.
How
appendBlockContentsnow skips an unparseable row and returns it (instead of throwing).LegacyViewBuilder.addBlocks) folds the skipped rows intoskippedCorruptRows(logged viawarnCorruptSkips).readResidentFieldsFor) logs the skipped rows.Behavior change (flagged)
Dependency
This PR is based on and depends on PR #7336 (the #7335 fix that introduces
skippedCorruptRows). It should merge after #7336. The diff against master currently includes #7336's changes because this branch is based on it; once #7336 merges I will rebase this branch on master so the diff shrinks to just this fix.Tests
tests/storage-poison-row-isolation.test.ts— 3 new tests:readResidentFieldsFor.Verified the new tests FAIL on the pre-fix code (real regression tests). Targeted storage/persist suites + typecheck + build green locally; full suite on CI.
<-- ework-agent-pr -->
Fixes #7337