Skip to content

[PR] storage: position-independent row identity + historian tombstones (#4390) - #4395

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-23_row-identity-historian
Oct 24, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-23_row-identity-historian

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #4390

这是什么

五个维度存储审查(报告:https://github.com/ranxianglei/billion-context/issues/4390#issuecomment-…)发现的四个 p2 + 三个顺路 p3,一个 PR 修完。核心一句话:让统一存储从「客户端当前视图的镜子」变成「耐久史官」,并把行身份从会被客户端改写污染的原料上解耦。

四个 p2

D4-1 行身份去污染(最重,seq 账记的地基)

  • 病:跨保存 JOIN 键 = ${m.id}\u0000${sha256(整条 CoreMessage)}——序列化掺了 wireIndex(wire 位置)和 m.id 本身。客户端头部截断/中段删一条 → 后续全部消息 wireIndex 移位 → 同一条消息换 hash → 整段行换血(旧行被清扫、seq 重铸、[PR] #4316 Layer 3 (number directly mapped to canonical seq) PR 1/3: Fold-overwrite into real seq range | #4316 层3(编号直连正史 seq)PR 1/3:折叠覆盖写入真实 seq 区间 #4335 的 start_seq/end_seq 区间账记随之抖动)。
  • 修:新增 identity 列 = 剥离 id/wireIndex 后的存储字节哈希(src/storage/identity.ts)。content_hash 保持字节忠实 CAS 键不变。JOIN 三处全迁:intern/remap/delta 严格追加门。
  • 存量:v8 迁移(SchemaVersion 7→8)从 CAS 字节回填 identity,并合并 wireIndex 分裂世代(同 (msg_id, identity) 保最低 seq,重复行删 message_media + 退 CAS refcount)。MySQL 同款(information_schema 探测 + 幂等 ALTER + init 回填——MySQL 没有 ADD COLUMN IF NOT EXISTS)。

D4-2 权威集清扫改墓碑

  • 病:completeLog 保存把「上轮在、本轮缺席」的行物理 DELETE——rewind/undo 的历史被销毁,正史是当前视图的镜像非史官([PR] fold anchoring doctrine: order over hash — shape churn, image offload, host truncation stop reading as destruction (#3641) #3644 epoch 回放、rewind-redo 都需要历史行还在)。
  • 修:缺席 → retired_at = now 墓碑;CAS 引用不释放(行还拥有字节);读面(getMessages / 统一重放 / 视图)默认过滤 retired(默认 exclude,export/diagnostics 可 includeRetired);同一身份回来 → 原地复活保原 seq(sweep 路径 + remap 路径双保险,completeLog-less 保存也覆盖);delta 严格追加门把 retired 算已存在(rewind-redo 回退全量路径)。物理删除仍归 deleteSession/archive/GC。

D4-3 渲染接缝图片真指纹

  • 病:identitySig 图片段 = img:${base64.length}——同尺寸截图等长 → MIN_SEAM_RUN=2 连击 → 错接缝 → 静默截断(车道内错接,降级保护盖不住)。
  • 修:img:${sha256(base64).slice(0,24)}(走 walk 只碰尾部,代价可承受)。

D3-2 fork 谱系入正史

  • 病:parent_session_id/fork_point_seq 恒写 null,谱系只活在信封 metadata.derivedFromSessionId;信封丢失 = 统一存储重建后 fork 家族失忆。
  • 修:ingest 落库 parentSessionId(非空串窄化)。

顺路 p3

  • D5-1:gc full vacuum 改走 this.vacuum('full')(带 TRUNCATE checkpoint,文件真正缩小)。
  • D4-5:loadLegacyView 尾指针缺行从静默过滤改为计数(tailGaps),三个加载路径(replay 校验/统一加载/异步加载)统一 warn。

打桩先行

tests/storage-review-p2s.test.ts 7 桩,实现前全 RED:

  • S1+S2 头部截断后 identity/seq 稳定(旧行为=换血重铸);清扫墓碑不删行;rewind-redo 复活同 seq
  • S3 同长异图 seam 不匹配 + 同图匹配
  • S4 谱系落库
  • S5 v8 迁移回填 + 分裂世代去重(3→2 行,保最低 seq)
  • S6 gc 走 vacuum()(wal_checkpoint TRUNCATE 生效)
  • S7 尾指针缺行计数(不再静默)

验证

  • 本 PR 家族:storage-review-p2s 7/7
  • 回归:storage-unified 14/14(v7→v8 断言更新)、archive-gc 5、full-flip 10、state-fields 2、orphan-representable 1、reap-journal 2、durability 5、maintenance-cli 7、golden-storage-render 5、wire-contract-golden 5、engine-bootstrap 44、fold-reconcile 68、acp-status 7、incremental-lane 24、incremental-ack-pull 3、inc-canonical-pointers 3、snapshot-render 12、snapshot-shadow 5、web-sessions 26
  • npm run typecheck 通过;unused-export gate clean(6 waived)
  • 无 kernel 改动、无新配置/env/CLI、无协议变化

kernel-change: none

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-23_row-identity-historian (5ac5428)

Latest build — install this exact version

npm install -g billion-context@0.0.5398-pr.4395.ca5a06d

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
5ac5428 billion-context@0.0.5398-pr.4395.ca5a06d build
7071b02 billion-context@0.0.5395-pr.4395.a861d69 build
f0b3e8a billion-context@0.0.5343-pr.4395.dcb47f9 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4395.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本条即 #4390 的交付 PR,近 7 天 open/closed issues+PRs 中无同机制在途条目。

标签: bug / scope/storage / p1(统一存储主链路引入两个数据完整性回归,其一为信封冻结→数据丢失向量)。

状态:CI 三个红 shard 已在本地复现、根因钉死、修复完成且本地全绿;即将推送更新分支。

CI 失败(ubuntu22 shard 1/3、windows22 shard 1)对应本地 12 个测试失败,归并后是三个根因:

  1. p1 回归 A——全量 ingest 对「identity 相同、字节不同」的行永远无法自愈(最重)。D4-1 把跨保存 JOIN 键换成 identity(剥离 id/wireIndex 的哈希)后,全量路径 remap 按 identity 复用 seq,INSERT OR IGNORE 又静默保留旧世代字节。两个实证后果(instrumentation 复现):
  2. p2 回归 B——tailGaps 诊断键泄漏进 parity 摘要面。loadLegacyView 新增返回 tailGaps(D4-5),而 expect 侧(live session)结构上没有该字段 → canonicalize 保键 → 摘要必不等 → 每次采样校验都误报漂移(ccr/save-path/watermark 一族测试红的直接原因)。修法:tailGaps 从摘要面剔除(与既有 storageMode 先例同类处理,Storage layer: four concrete defects (filename coupling, multi-writer heuristic, non-atomic content-store write, immortal compressed sessions) #2671);真实缺行仍会让消息数组本身分歧,不掩盖任何东西。
  3. 两个陈旧 v7 断言:schema bump 到 v8 后只更新了 storage-unified.test.ts 一处,storage-ext.test.ts 与 storage-ingest-watermark.test.ts 各剩一个 v7 pin。已修。

验证:typecheck 干净;原先红的 5 个套件 + review-p2s + unified 共 49 例全绿;storage 域 57 文件全量 472 例 0 失败(修前 12 失败);所有 DIGEST DRIFT 日志现在都是真篡改检测、带槽位指纹。

接下来:提交 → rebase 到最新 master → 定向重跑 → push 更新分支 → 看 CI,随后按八问给最终结论。

ranxianglei and others added 4 commits October 24, 2026 02:11
)

Four p2 fixes from the five-dimension storage review, one PR:

- identity column: cross-save JOIN key strips id/wireIndex (D4-1);
  content_hash stays the byte-faithful CAS key. v8 migration backfills
  and heals wireIndex-split generations (keep lowest seq, refund CAS
  refs). MySQL twin via information_schema probe + idempotent ALTER.
- authoritative sweep retires absent rows (retired_at tombstone)
  instead of DELETE; rewind-redo reactivates in place at the original
  seq (remap covers completeLog-less saves too); readers filter
  retired rows (D4-2).
- snapshot-render seam image signature hashes the bytes (24-hex
  sha256), not the length — same-size screenshots can no longer chain
  a false seam into silent truncation (D4-3).
- fork lineage: parent_session_id from metadata.derivedFromSessionId
  (D3-2).
- gc full vacuum routes through vacuum() (TRUNCATE checkpoint included,
  files actually shrink, D5-1); loadLegacyView counts tail pointer gaps
  and all three load paths warn instead of silently shrinking (D4-5).

Stakes first: tests/storage-review-p2s.test.ts (7), storage family +
golden/snapshot/fold-reconcile/incremental/web regressions green.
…ide-p3)

b468656 covered the four p2s and two of the three side-p3s named in the
issue body; the remaining one — fold-reconcile Pass 0 silently disabling
itself when the stored foldPositions copy desyncs from the order backbone
— left no trace at all.

- planReconciliation sets an additive plan.pass0Desync flag (pure core,
  behavior unchanged)
- reconcileFoldCoverage logs a warn naming session id + copy/backbone sizes
- both directions pinned in fold-reconcile-positional.test.ts (desync ->
  flagged; aligned copy -> not flagged); CHANGELOG line extended
Review fix for CI reds on this branch, part 1 of 2.

- loadLegacyView's new tailGaps counter (D4-5) leaked into the shadow-parity digest surface: the expect leg (live session) structurally lacks the key while the got leg (loadLegacyView output) always carries it, so canonicalize()'s key-preserving normalization made every sampled verify report a fake drift. Drop the key from all digests — same class as the existing storageMode precedent (#2671): presence-not-value, and genuine missing rows still diverge the message arrays themselves, so nothing is masked. describeViewDivergence skips it too, keeping drift diagnostics clean (issue3919 forensics pins keys deepEqual).

- Two stale user_version=7 pins the v8 bump missed: tests/storage-ext.test.ts (v3→v7 title/asserts) and tests/storage-ingest-watermark.test.ts (v4→v7 title/assert) now expect v8, matching the updated pin in tests/storage-unified.test.ts.
…s but bytes diverge

Review fix for CI reds on this branch, part 2 of 2.

The D4-1 JOIN migration made remapPlanToAbsolute reuse a seq whenever (msg_id, identity) matched, and INSERT OR IGNORE then silently kept the first generation's bytes for that seq. Any byte divergence under a matching identity became permanently unhealable by full ingest: (a) head truncation — the PR's own flagship scenario — leaves stored rows carrying the old wireIndex generation, so the parity digest never matches the live view again: every 20th save reports drift, the watermark is cleared, the next full save re-runs the same no-op remap, never converges, and after DRIFT_ESCALATE_STREAK consecutive samples #4323 freezes this process's envelope writes (legacy file goes stale = data-loss vector); (b) an externally corrupted CAS row (watermark boundary-tamper test) was healed pre-migration by the byte-faithful join minting fresh rows, and is now kept corrupted forever.

Fix: when the full path reuses an existing seq via the identity join and the stored content_hash differs from the plan row's hash, UPDATE the row in place to the fresher authoritative bytes (CAS ref juggling put-new -> update-ref -> release-old, FK/refcount safe; media mappings untouched because identity includes the {$media:hash} markers). Seq stability is unchanged; the delta path stays append-only. SQLite + MySQL twins. identity.ts doc updated to match; CHANGELOG notes the behavior change.
@ranxianglei ranxianglei added bug Something isn't working p1 scope/storage labels Oct 23, 2026
@ranxianglei
ranxianglei force-pushed the 2026-10-23_row-identity-historian branch from 93acad7 to f0b3e8a Compare October 23, 2026 18:14
…_FILES (#4449 follow-up)

#4449 landed two process-spawning suites without registering them, so
every test shard now aborts at startup ('process-spawning test files
missing from LIVE_FILES') on master and all open PRs. Same disease as
#4333/#4334 — register them.
@ranxianglei
ranxianglei merged commit 03b5734 into master Oct 24, 2026
46 of 50 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.792-master.03b5734 (rolling master tag).

npm install -g billion-context@0.0.792-master.03b5734  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

This was referenced Oct 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p1 scope/storage

Projects

None yet

1 participant