Repository navigation
fix: shadow-storage incremental seq-ledger ingest — steady-state saves cost O(new messages) - #3151
ranxianglei wants to merge 2 commits into
Conversation
…ssages) (#3133) Completes the #3133 default-path perf cliff per the owner design input in the issue thread (per-session high-water mark + identity probing + full fallback on mismatch). The dropped object-identity memo was defeated by per-request wire re-parse; claims now key off durable identity instead: - store.ts: per-session SeqState ledger lazily seeded from the existing (session_id, seq, msg_id, content_hash) rows (zero schema change); commitDelta/dropSeqState manage it around ingest transactions; delta ingests INSERT strictly so a stolen seq throws instead of aliasing. - ingest.ts: planIngest gains an optional ledger — ref-token claims (token unique in the save union + single identity + matching msgId) reuse seqs with no serialization at all; unknown identities serialize once and allocate past the high-water mark. Full mode is byte-identical to before. - shadow.ts: record() drives the ledger with a constraint-conflict heal (reseed + full replay in the same save) and drops it on sampled digest drift; dual-engine constraint classification (better-sqlite3 string codes vs node:sqlite, whose generic ERR_SQLITE_ERROR code carries the real extended result code in errcode). Safety: any doubt falls back to a full ingest (over-write, never under-write); residual same-id-different-content exposure rides the existing sampled verifyDrift net. Tests cover churn steady-state counts, full-vs-delta pipeline byte equivalence, PK-conflict heal on both engines, drift-driven forced-full recovery, and restart re-seeding. No config/schema/kernel/wire change.
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.3593-pr.3151.ff38f1cThis exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3151.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 审核了 PR #3151(分支 1. 是什么问题:默认开启的 SQLite 影子存储,每次保存都把整个会话历史重新 2. 什么原因导致的:根因在 ingest 的建模—— 3. 影响范围与严重程度:所有 0.1.197+ 走默认 shadow 路径的长会话;严重程度是性能悬崖(秒级同步 CPU),不是正确性问题(JSON 仍是事实源,digest 会校验)。 4. 怎么修复的:每会话一张内存 seq 台账( 5. 是否有回归问题:审过以下既有行为——
6. 是否破坏缓存:不涉及。前缀/prompt cache 无关;只动影子存储落盘路径,不碰发给上游的请求体。增量台账是纯内存派生态,不落盘、不进 schema。 7. 是否改了设计方向:没有转向。是把 #3133 里 owner 给的「水印/台账」设计从单边界探测细化成按消息身份认领落地。数据流不变(JSON 信封仍是事实源,影子库是镜像),只是镜像写入成本从 O(全量) 降到 O(新增)。 8. 是否引入新配置:无。没有新 env/config/CLI; 测试有效性(质量关):6 个用例都有牙—— CI:head 判定:可合——rebase 到当前 master 即可(CI 已全绿)。第 5 问的 UNIQUE-sub-8 引擎不对称建议作为一行级 follow-up 补掉(或现在授权我顺手推上),不阻塞本 PR。 |
(ai: vllm-qwen/qwen3.8-27b)
Closes #3133
Post-merge supplement to #3140 (merged as cc58345 while this work was in flight): #3140 shipped the two side fixes (sampled read-back verify, writerOff coalescing) and explicitly deferred the core complaint of #3133 — per-save O(full log) stringify+sha256 on the default shadow path — back to the issue, where the owner posted the watermark design input. This PR implements that design, refined from single-boundary probing to per-message identity claims.
What
Per-session seq ledger (in-memory) + durable-identity claims, replacing full re-intern on every save:
SeqStatelazily seeded ONCE per session from the existing(session_id, seq, msg_id, content_hash)rows — zero schema change;commitDelta/dropSeqStatemanage it around ingest transactions; delta ingests INSERT strictly so a stolen seq throws instead of aliasing.planIngest(legacy, ledger?)— a ref-token claim (token unique in the save union + single identity under it + matching msgId) reuses the stored seq with NO serialization at all; unknown identities serialize once and allocate past the high-water mark; full mode stays byte-identical to pre-change.record()drives the ledger; a constraint conflict heals in the same save (reseed + full replay); sampled digest drift drops the ledger and forces one full replay; dual-engine constraint classification (better-sqlite3 string codes vs node:sqlite, whose genericERR_SQLITE_ERRORcode carries the real extended result code inerrcode).Cost model (per the issue design input): steady-state O(new messages) instead of O(full log); misjudgment costs over-write, never under-write — every doubt path falls back to a full ingest. Residual same-id-different-content exposure rides the existing ≤20-sample verifyDrift net.
Behavior changes (disclosed)
Evidence
tests/storage-incremental-ingest.test.ts(6 tests): churn steady-state exact allocation counts + sampled verify green through n=21; full-vs-delta pipeline byte equivalence on the corpus case; PK-conflict heal on node:sqlite (+ better-sqlite3 variant, runs on CI hosts carrying the native dep); CAS-tamper drift → forced-full recovery → delta resumes; restart re-seed from disk.