Skip to content

fix: shadow-storage incremental seq-ledger ingest — steady-state saves cost O(new messages) - #3151

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-14_shadow-incremental-seq-ledger
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-14_shadow-incremental-seq-ledger

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: vllm-qwen/qwen3.8-27b)

Closes #3133

Post-merge supplement to #3140 (merged as cc58345 while this work was in flight): #3140 shipped the two side fixes (sampled read-back verify, writerOff coalescing) and explicitly deferred the core complaint of #3133 — per-save O(full log) stringify+sha256 on the default shadow path — back to the issue, where the owner posted the watermark design input. This PR implements that design, refined from single-boundary probing to per-message identity claims.

What

Per-session seq ledger (in-memory) + durable-identity claims, replacing full re-intern on every save:

  • store.ts: SeqState lazily seeded ONCE per session from the existing (session_id, seq, msg_id, content_hash) rows — zero schema change; commitDelta/dropSeqState manage it around ingest transactions; delta ingests INSERT strictly so a stolen seq throws instead of aliasing.
  • ingest.ts: planIngest(legacy, ledger?) — a ref-token claim (token unique in the save union + single identity under it + matching msgId) reuses the stored seq with NO serialization at all; unknown identities serialize once and allocate past the high-water mark; full mode stays byte-identical to pre-change.
  • shadow.ts: record() drives the ledger; a constraint conflict heals in the same save (reseed + full replay); sampled digest drift drops the ledger and forces one full replay; dual-engine constraint classification (better-sqlite3 string codes vs node:sqlite, whose generic ERR_SQLITE_ERROR code carries the real extended result code in errcode).

Cost model (per the issue design input): steady-state O(new messages) instead of O(full log); misjudgment costs over-write, never under-write — every doubt path falls back to a full ingest. Residual same-id-different-content exposure rides the existing ≤20-sample verifyDrift net.

Behavior changes (disclosed)

  • Per-save ingest cost on the default shadow path: O(all messages) → O(new messages). Output bytes unchanged (T2 proves full-vs-delta pipeline byte equivalence across the reader surface).
  • No new config/env/CLI surface; no schema change (stays v4); no kernel/; no wire shape.

Evidence

  • New tests/storage-incremental-ingest.test.ts (6 tests): churn steady-state exact allocation counts + sampled verify green through n=21; full-vs-delta pipeline byte equivalence on the corpus case; PK-conflict heal on node:sqlite (+ better-sqlite3 variant, runs on CI hosts carrying the native dep); CAS-tamper drift → forced-full recovery → delta resumes; restart re-seed from disk.
  • Local targeted runs: storage-incremental-ingest 5 pass / 1 skip (better-sqlite3 absent locally); storage-save-path + storage-unified + storage-json-writer-off 19 pass / 1 skip; typecheck clean. Full-suite verdict = CI test lanes on this head.
  • Probe numbers (3000-message session, production-style ref churn per save): before — 3001/3001 serialized per save (~cold cost, ~113 ms); after — 0 serializations for churn-only saves, exactly k for k appended turns.

…ssages) (#3133)

Completes the #3133 default-path perf cliff per the owner design input in
the issue thread (per-session high-water mark + identity probing + full
fallback on mismatch). The dropped object-identity memo was defeated by
per-request wire re-parse; claims now key off durable identity instead:

- store.ts: per-session SeqState ledger lazily seeded from the existing
  (session_id, seq, msg_id, content_hash) rows (zero schema change);
  commitDelta/dropSeqState manage it around ingest transactions; delta
  ingests INSERT strictly so a stolen seq throws instead of aliasing.
- ingest.ts: planIngest gains an optional ledger — ref-token claims (token
  unique in the save union + single identity + matching msgId) reuse seqs
  with no serialization at all; unknown identities serialize once and
  allocate past the high-water mark. Full mode is byte-identical to before.
- shadow.ts: record() drives the ledger with a constraint-conflict heal
  (reseed + full replay in the same save) and drops it on sampled digest
  drift; dual-engine constraint classification (better-sqlite3 string codes
  vs node:sqlite, whose generic ERR_SQLITE_ERROR code carries the real
  extended result code in errcode).

Safety: any doubt falls back to a full ingest (over-write, never
under-write); residual same-id-different-content exposure rides the existing
sampled verifyDrift net. Tests cover churn steady-state counts, full-vs-delta
pipeline byte equivalence, PK-conflict heal on both engines, drift-driven
forced-full recovery, and restart re-seeding. No config/schema/kernel/wire
change.
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-14_shadow-incremental-seq-ledger (8030f2b)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.3593-pr.3151.ff38f1c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3151.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核了 PR #3151(分支 2026-10-14_shadow-incremental-seq-ledger @ 8030f2b0,2 commit,+741/−59,4 文件:store.ts / ingest.ts / shadow.ts + 新测试)。这是 #3133(默认影子存储每次落盘全量序列化+哈希)的核心修复——#3140 交付了两个旁路修复后把主诉留给本 PR。我把三个源文件 diff 逐段读了,并用本机 node:sqlite 实测了约束码分类。结论:方向对、实现稳、测试有牙,可合——CI 已全绿,只差 rebase 到当前 master。逐问如下:

1. 是什么问题:默认开启的 SQLite 影子存储,每次保存都把整个会话历史重新 JSON.stringify+sha256,再整读回算两遍摘要——几千条消息的长会话每次保存都是秒级同步 CPU(占事件循环)。大会话越用越卡,而且这是默认路径。

2. 什么原因导致的:根因在 ingest 的建模——planIngest 把「本次写哪些行」和「每条消息的绝对 seq」耦合在一起,每次都得重新序列化全部消息才能确定身份与序号;且没有跨保存的身份记忆(线上每个请求都重新解析出全新对象图,对象引用活不过两次保存)。#3130/#3140 只加了采样校验和 writerOff 合并,没动这个 O(全量) 序列化本身,所以主诉留给了本 PR。

3. 影响范围与严重程度:所有 0.1.197+ 走默认 shadow 路径的长会话;严重程度是性能悬崖(秒级同步 CPU),不是正确性问题(JSON 仍是事实源,digest 会校验)。

4. 怎么修复的:每会话一张内存 seq 台账(SeqState,LRU 上限 128,首次从 messages 表懒种子),配合三级认领阶梯——① ref-token 认领(token 在本次 union 里唯一 + 台账单身份 + msgId 匹配)→ 零序列化直接复用旧 seq;② 已知按字节(序列化+哈希但命中台账)→ 复用 seq 不发行;③ 新身份 → 分配高水位之后的新 seq。delta 行用严格 INSERT(无 OR IGNORE),seq 被外部写入者偷走就抛错回滚、重种子重放,绝不静默改号。误判代价只会多写不会少写,任何存疑路径都退回全量。

5. 是否有回归问题:审过以下既有行为——

  • 全量模式(无台账)ingest:字节级不变。 我核对了 planIngest:!delta 时三级阶梯的 if (delta && st) 分支全部短路,intern 行为与改前一致,e.seq==log.length==map 索引,tailJson 仍走「相对 seq + remap 覆盖」。T2 测试用真实 corpus(folded)对 full vs delta 两条腿断言 messages/blocks/refs/deadRefs/sessionRow(含 tailJson)deepEqual,钉死了这一点。
  • deleteSession / remove() 各加一行 drop 台账:纯新增,不改既有调用方行为。
  • 无线格式变化、无 schema 变化(保持 v4)、无新增配置/env/CLI(config-surface-guard 绿)、不碰 kernel/。
  • 高危模式逐项:fail-fast 未变(delta 冲突仍是 throw→回滚);阈值/时序/输出格式无漂移;缓存策略不受影响。
  • 一处需留意(非阻塞):isConstraintError 对 node:sqlite 只认 sub 5/sub 6。我用本机 node:sqlite 实测:复合 PK 冲突 errcode=1555=sub 6 ✓ 命中自愈合;但 messages 表还有 UNIQUE(session_id,msg_id,content_hash),跨进程竞争下它报 errcode=2067=sub 8,未被识别 → 走连续失败计数而非自愈合。而 better-sqlite3 侧字符串码已含 SQLITE_CONSTRAINT_UNIQUE,所以两引擎不对称。后果有界:严格 INSERT 仍防静默别名,最坏是影子侧车禁用回退 legacy(即 pre-shadow 行为),不损坏数据。修法是一行(node:sqlite 分支补 || e.errcode === (19 | (8<<8)))对齐 better-sqlite3。按 [PR] 处理纪律我不擅自改,留作建议;要不要顺手补由你定。

6. 是否破坏缓存:不涉及。前缀/prompt cache 无关;只动影子存储落盘路径,不碰发给上游的请求体。增量台账是纯内存派生态,不落盘、不进 schema。

7. 是否改了设计方向:没有转向。是把 #3133 里 owner 给的「水印/台账」设计从单边界探测细化成按消息身份认领落地。数据流不变(JSON 信封仍是事实源,影子库是镜像),只是镜像写入成本从 O(全量) 降到 O(新增)。

8. 是否引入新配置:无。没有新 env/config/CLI;SeqState LRU 上限 128 是内部常量(SEQ_STATE_CAP),不是配置面。stats 多了一个 deltaSaves 观测计数(语义同 verifySamples,计运行次数),供面板看稳态成本,不算配置。

测试有效性(质量关):6 个用例都有牙——allocCursor 直接测「分配游标」这个真实成本单位(churn 后持平、追加 k 条恰 +k);full-vs-delta 用真实 corpus 断言全 reader 面字节相等;PK 冲突自愈往库里真插一条外部行再触发;drift 恢复真篡改 CAS 字节;重启重种子用全新 store。缺口:共享 token / 同 id 不同内容的对抗场景没单独测(它们都会软失败退回序列化,风险低)。作者已明确披露残余风险「同 id 不同内容靠 ≤20 次采样 verifyDrift 兜底」(VERIFY_EVERY_INGESTS=20)——有界且自愈,接受。

CI:head 8030f2b0 上 12/12 test 分片全绿,static-gates ×4、unused-exports、kernel-guard、config-surface-guard、version/lane/pi-guard、lock-sync、docs-sync/coverage、hermetic registry e2e、real dsh/pi/codex 三条 lane 全部通过。第二个 commit(un-export RefClaim)就是修 unused-exports 门禁的,合法且已标注。分支落后 master 2 个提交(release-v0.1.197 + ci-shard-test-suite + docs,均不碰 src/storage/)→ rebase 干净。

判定:可合——rebase 到当前 master 即可(CI 已全绿)。第 5 问的 UNIQUE-sub-8 引擎不对称建议作为一行级 follow-up 补掉(或现在授权我顺手推上),不阻塞本 PR。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

按 superseded 关闭:#3133 的修复改由 PR #3166(水位线方案)承载——owner 已在其上直接迭代,且该机制的增量保证不依赖 byRaw 覆盖、带真机 bench。本分支(seq-ledger 方案)保留在 2026-10-14_shadow-incremental-seq-ledger @ 8030f2b,CI 曾全绿,如需对比可 reopen。审核中发现的 drift-不愈合问题已在 #3166 分支以 6eb8fe5ac 补上。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了

1 participant