Repository navigation
[PR] storage: position-independent row identity + historian tombstones (#4390) - #4395
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.5398-pr.4395.ca5a06dThis exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4395.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复——本条即 #4390 的交付 PR,近 7 天 open/closed issues+PRs 中无同机制在途条目。 标签: bug / scope/storage / p1(统一存储主链路引入两个数据完整性回归,其一为信封冻结→数据丢失向量)。 状态:CI 三个红 shard 已在本地复现、根因钉死、修复完成且本地全绿;即将推送更新分支。 CI 失败(ubuntu22 shard 1/3、windows22 shard 1)对应本地 12 个测试失败,归并后是三个根因:
验证:typecheck 干净;原先红的 5 个套件 + review-p2s + unified 共 49 例全绿;storage 域 57 文件全量 472 例 0 失败(修前 12 失败);所有 DIGEST DRIFT 日志现在都是真篡改检测、带槽位指纹。 接下来:提交 → rebase 到最新 master → 定向重跑 → push 更新分支 → 看 CI,随后按八问给最终结论。 |
) Four p2 fixes from the five-dimension storage review, one PR: - identity column: cross-save JOIN key strips id/wireIndex (D4-1); content_hash stays the byte-faithful CAS key. v8 migration backfills and heals wireIndex-split generations (keep lowest seq, refund CAS refs). MySQL twin via information_schema probe + idempotent ALTER. - authoritative sweep retires absent rows (retired_at tombstone) instead of DELETE; rewind-redo reactivates in place at the original seq (remap covers completeLog-less saves too); readers filter retired rows (D4-2). - snapshot-render seam image signature hashes the bytes (24-hex sha256), not the length — same-size screenshots can no longer chain a false seam into silent truncation (D4-3). - fork lineage: parent_session_id from metadata.derivedFromSessionId (D3-2). - gc full vacuum routes through vacuum() (TRUNCATE checkpoint included, files actually shrink, D5-1); loadLegacyView counts tail pointer gaps and all three load paths warn instead of silently shrinking (D4-5). Stakes first: tests/storage-review-p2s.test.ts (7), storage family + golden/snapshot/fold-reconcile/incremental/web regressions green.
…ide-p3) b468656 covered the four p2s and two of the three side-p3s named in the issue body; the remaining one — fold-reconcile Pass 0 silently disabling itself when the stored foldPositions copy desyncs from the order backbone — left no trace at all. - planReconciliation sets an additive plan.pass0Desync flag (pure core, behavior unchanged) - reconcileFoldCoverage logs a warn naming session id + copy/backbone sizes - both directions pinned in fold-reconcile-positional.test.ts (desync -> flagged; aligned copy -> not flagged); CHANGELOG line extended
Review fix for CI reds on this branch, part 1 of 2. - loadLegacyView's new tailGaps counter (D4-5) leaked into the shadow-parity digest surface: the expect leg (live session) structurally lacks the key while the got leg (loadLegacyView output) always carries it, so canonicalize()'s key-preserving normalization made every sampled verify report a fake drift. Drop the key from all digests — same class as the existing storageMode precedent (#2671): presence-not-value, and genuine missing rows still diverge the message arrays themselves, so nothing is masked. describeViewDivergence skips it too, keeping drift diagnostics clean (issue3919 forensics pins keys deepEqual). - Two stale user_version=7 pins the v8 bump missed: tests/storage-ext.test.ts (v3→v7 title/asserts) and tests/storage-ingest-watermark.test.ts (v4→v7 title/assert) now expect v8, matching the updated pin in tests/storage-unified.test.ts.
…s but bytes diverge Review fix for CI reds on this branch, part 2 of 2. The D4-1 JOIN migration made remapPlanToAbsolute reuse a seq whenever (msg_id, identity) matched, and INSERT OR IGNORE then silently kept the first generation's bytes for that seq. Any byte divergence under a matching identity became permanently unhealable by full ingest: (a) head truncation — the PR's own flagship scenario — leaves stored rows carrying the old wireIndex generation, so the parity digest never matches the live view again: every 20th save reports drift, the watermark is cleared, the next full save re-runs the same no-op remap, never converges, and after DRIFT_ESCALATE_STREAK consecutive samples #4323 freezes this process's envelope writes (legacy file goes stale = data-loss vector); (b) an externally corrupted CAS row (watermark boundary-tamper test) was healed pre-migration by the byte-faithful join minting fresh rows, and is now kept corrupted forever. Fix: when the full path reuses an existing seq via the identity join and the stored content_hash differs from the plan row's hash, UPDATE the row in place to the fresher authoritative bytes (CAS ref juggling put-new -> update-ref -> release-old, FK/refcount safe; media mappings untouched because identity includes the {$media:hash} markers). Seq stability is unchanged; the delta path stays append-only. SQLite + MySQL twins. identity.ts doc updated to match; CHANGELOG notes the behavior change.
93acad7 to
f0b3e8a
Compare
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.792-master.03b5734 # this exact build
npm install -g billion-context@master # always the newest merged build
|
Fixes #4390
这是什么
五个维度存储审查(报告:https://github.com/ranxianglei/billion-context/issues/4390#issuecomment-…)发现的四个 p2 + 三个顺路 p3,一个 PR 修完。核心一句话:让统一存储从「客户端当前视图的镜子」变成「耐久史官」,并把行身份从会被客户端改写污染的原料上解耦。
四个 p2
D4-1 行身份去污染(最重,seq 账记的地基)
${m.id}\u0000${sha256(整条 CoreMessage)}——序列化掺了wireIndex(wire 位置)和m.id本身。客户端头部截断/中段删一条 → 后续全部消息 wireIndex 移位 → 同一条消息换 hash → 整段行换血(旧行被清扫、seq 重铸、[PR] #4316 Layer 3 (number directly mapped to canonical seq) PR 1/3: Fold-overwrite into real seq range | #4316 层3(编号直连正史 seq)PR 1/3:折叠覆盖写入真实 seq 区间 #4335 的 start_seq/end_seq 区间账记随之抖动)。identity列 = 剥离id/wireIndex后的存储字节哈希(src/storage/identity.ts)。content_hash保持字节忠实 CAS 键不变。JOIN 三处全迁:intern/remap/delta 严格追加门。D4-2 权威集清扫改墓碑
retired_at = now墓碑;CAS 引用不释放(行还拥有字节);读面(getMessages / 统一重放 / 视图)默认过滤 retired(默认 exclude,export/diagnostics 可includeRetired);同一身份回来 → 原地复活保原 seq(sweep 路径 + remap 路径双保险,completeLog-less 保存也覆盖);delta 严格追加门把 retired 算已存在(rewind-redo 回退全量路径)。物理删除仍归 deleteSession/archive/GC。D4-3 渲染接缝图片真指纹
img:${base64.length}——同尺寸截图等长 → MIN_SEAM_RUN=2 连击 → 错接缝 → 静默截断(车道内错接,降级保护盖不住)。img:${sha256(base64).slice(0,24)}(走 walk 只碰尾部,代价可承受)。D3-2 fork 谱系入正史
parent_session_id/fork_point_seq恒写 null,谱系只活在信封metadata.derivedFromSessionId;信封丢失 = 统一存储重建后 fork 家族失忆。顺路 p3
this.vacuum('full')(带 TRUNCATE checkpoint,文件真正缩小)。打桩先行
tests/storage-review-p2s.test.ts7 桩,实现前全 RED:验证
npm run typecheck通过;unused-export gate clean(6 waived)kernel-change: none