Repository navigation
[PR] fix(storage): skip already-retired rows in completeLog historian sweep (#5756) - #5772
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.8298-pr.5772.e1daa10This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5772.tgz
npm install -g packageInstall instructions are refreshed on each push. |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复——本 PR 是 #5756 的唯一在途修复(镜像 [PR] 清单核对:无并行 PR 指向同一线程);#5744/#5755/#3133 为背景/机制来源,非重复项。 标签: bug/scope/storage/p2(存储层稳态保存性能缺陷的修复;无客户端相关标签——纯存储层、宿主无关;可用但有明确性能损失,p2)。 独立审核结论(未复用 issue 侧表述,以下逐条为本轮重做): 先澄清一点:#5756 首楼说「同 PR 顺手修掉 #4793 remap 修复路径把 live 兄弟行退役时不释放其 media mappings」。我独立核对了当前 master:该缺陷不存在——两条车道的 leftovers SELECT( 八问
测试有效性:真实 UnifiedStore + planIngest(非 mock);语句计数器包 验证证据:本地定向套件 23/23 绿(issue5756-lazy-sweep、media-cas、unified-remap-collision、storage-ingest-watermark)+ typecheck 干净;远程 CI ff27ca0 全绿(18 success / 12 skipped 为路径门控 / 0 fail,含覆盖 MySQL 孪生的 mysql-contract)。 diff 整洁度:3 文件 +175/-10,每行都与本修复相关,无跑题改动。 判定:可合入。 |
ranxianglei
left a comment
There was a problem hiding this comment.
(ai: vllm-qwen/qwen3.8-27b)
判定:APPROVE / 可合(head ff27ca005,base = master tip bfabcd1ca 零漂移;3 文件 +175/−10)。作者账号无法自批本 PR(GitHub 规则),按惯例以 COMMENT 事件承载评审。
结构化八问
- 是什么问题:两类永远够不到 #3133 watermark delta 的视图形状(滑动窗口 resident tail、folded-resident envelope)上,每回合的 save 都对全会话历史跑一遍 historian sweep——对窗口外每一行执行 tombstone UPDATE + media SELECT + DELETE(+release 循环),哪怕该行早已退役、media 映射早已清空。稳定 3k 消息 / 160 行窗口会话 ≈ 每 turn 3k×3 条纯记账语句(issue 实测 ≈8900 / 9653 stmts/save)。
- 什么原因导致的:sweep 设计为无条件执行——假设窗口外每行每次都需要(重新)盖墓碑;而 UPDATE 本身一直带
AND retired_at IS NULL守卫,重复执行从来都是 no-op 写,语句成本却每次都付。#5744/#5755 拆掉了更大的 O(H) 件(refs ledger full-replace),本 PR 拆掉剩余的 O(H) churn。 - 影响范围与严重程度:纯性能面——bounded-view 长会话的稳态 save 延迟/语句数;不动数据、wire、配置、schema。严重度:性能劣化(长会话主线程卡顿贡献项),非可用性缺陷。
- 怎么修复的:sweep 跳过已退役行——SQLite
src/storage/store.ts:968(if (row.retired_at !== null) continue;)与 MySQL 孪生src/storage/mysql.ts:699-710(同判据包住三件套)。reader 只查 NULL/non-NULL,已退役行无事可做;首次退役仍付全量工作;tombstone 值、media 映射、refcount 全部不变。 - 是否有回归问题:逐项审过既有行为——(a) 首次退役语义不变(tombstone + media cleanup + ref release 照旧);(b) rewind-redo 再激活路径(#4390/#4506)未触碰,且在 skip 分支之前执行、只认 presented set 内的 identity,此类行永远不会被 skip 误伤;(c) 持久化值字节级不变(旧 UPDATE 本就 IS NULL 守卫不改值;唯一值读取点
src/export.ts:277读的是存储值,从未变化);(d) lane 排除(hasLaneExt)与 delta watermark 路径未触碰;(e) 无 fail-fast→吞错、无阈值/时序/输出格式漂移。结论:无回归。 - 是否破坏缓存:不涉及(不碰 cache/retry/cursor/dedup key;WAL checkpoint cadence 明确 out of scope,留在 issue 线程)。
- 是否改了设计方向:否——同一存储架构,只是不再执行死语句;方向 2/3 按要求 park 在 issue 线程待决策。
- 是否引入新配置:无(PR 声明 Config surface: none,核实属实)。
验证证据(本地,rebased head ff27ca005)
- 新回归
tests/issue5756-lazy-sweep.test.ts通过;负控制实测:把 store.ts 还原为修复前版本,计数断言如预期失败(save2=925 vs save3=935——稳态 save 重付了首退成本),恢复后树干净;该 pin 有牙。 - 相邻套件 media-cas / unified-remap-collision / storage-ingest-watermark 22/22 绿;
tsc --noEmitclean。 - CI(镜像缓存,head
ff27ca005):30 checks = 18 success / 12 gated-skip / 0 failed / 0 pending。 - 诚实边界:新 pin 在 SQLite 侧;MySQL 孪生的 skip 分支骑既有
storage-mysql-seam/storage-backend-mysql-e2elane(seam stub 不产生 retired 行,e2e 在 DSN 在场时覆盖多 save 全路径但不断言语句数)——与 PR 前 #4390 retire 三件套同等覆盖水平,可接受。
§7.4 自动合并画像:单模块(storage)、纯 fix、回归测试带负控制、diff 干净、无 config/wire/kernel/persistence-format 触碰、Closes #5756 —— 满足自动合并条件;合并动作仍归人工。
无顺手修项(未发现小问题)。等 owner 合并。
[bot] 🏷 |
ranxianglei
left a comment
There was a problem hiding this comment.
Review(#5772,#5756 sweep lazy 化)— 本地验证完成,建议可合并(合并权在人)
改动
单 commit ff27ca005,核心一行:completeLog sweep 中已退役行 if (row.retired_at !== null) continue;(首次退役仍做完整 tombstone+媒体释放),MySQL 孪生同形包裹。lane B/C(hasLaneExt)排除逻辑未动。
语义审查(为什么行为完全不变、只省语句)
- 旧 UPDATE 本就带
AND retired_at IS NULL→ 对已退役行匹配 0 行,重打戳从来是空操作;跳过只是省掉这次空执行。 - 被跳过的 tombstone 下不会长出新媒体:planner 不输出
retiredAt(ingest.ts 零匹配),incoming 缺席的行在本次 save 的消息 upsert 里根本不被写,新的 message_media 无从落下;seq 复用的换代先走 remap 的 delMedia+delRow+释放路径。 - rewind/undo 回来先于 skip:identity 重现走 :935-937 复活分支(
SET retired_at = NULL)再继续,新测试专门钉了这条(rewind-redo 后 retiredCount 归 0、图片 refcount 回到 1)。 - reader 审计:全部消费者只判 NULL/非 NULL(store.ts 十处);export/import(:1899/:2007)与会话复制 INSERT..SELECT(:2902-2903)原值搬运永不重生成——sentinel 时间戳原样幸存,测试用 2001 年 sentinel 值直接证明未被触碰。
本地验证(node v25.9.0,分支 pr-5772-check @ ff27ca0)
- 新测试 1/1 pass;相关批(fold-reconcile×3 + watermark-fastpath + bounded-storage)110/110 pass
- 负向对照:去掉那行
continue→ 失败于特征断言steady-state save re-paid the first-time retirement work (save2=925 stmts, save3=935 stmts)(旧码每轮真把 304−W 行的退役重付一遍);恢复即绿 npm run typecheckexit 0- 测试质量:statement 计数器包住 prepare(直接测 issue 指标,非代理量);预算留 2× 余量不误伤;refcount 诚实性跨重复 sweep 钉住
备注(不阻塞)
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.1071-master.5af5f9f # this exact build
npm install -g billion-context@master # always the newest merged build
|
…ed-state shape (second staleness round) The previous alignment targeted df0a402 (#5860/kernel-0.0.184 era). Master has since merged #5772 (lazy-historian-sweep), which rewrote src/storage/{store,mysql}.ts and again changed what the golden drive persists per session. Clean master's own scheduled full matrix (run 42358603178, sha 5af5f9f, tracked as issue #5870) reds on exactly the two suites reading these snapshots: - tests/golden-storage-render.test.ts (4 wires, x3 shards) - tests/storage-engine-proofs.test.ts:191 full-mode restart vs committed goldens, assert.equal(record.stateDigestAligned, golden.stateDigestAligned) (4 wires, x3 shards) Re-generated via scripts/update-storage-render-goldens.ts against clean master@5af5f9fe. Safety check BEFORE trusting the regen: git diff -U0 over the four snapshots shows ZERO turns[].sha lines touched (wire bodies byte-stable), foldAt/editedAt unchanged, only stateDigest/stateDigestAligned/sectionDigests .metadata values move — i.e. persistence-shape drift, not rendering behavior. Local proof on /tmp/mfix (origin/master 5af5f9f, fresh npm ci + full build, CI-parity pins BILI_MESSAGE_TIMESTAMPS/SNAPSHOT_SHADOW/IMAGE_RESCALE=0): tests/golden-storage-render.test.ts + tests/storage-engine-proofs.test.ts = 17 pass / 0 fail after this commit (before it: the CI failure set reproduces). This unblocks both master's :03/:33 scheduled matrix and every in-flight PR whose merge-ref runs against live master.
…4241 teeth at the byte-promised tail CI red on this PR had three causes: 1. storage-render + wire-contract goldens were regenerated against a kernel WITHOUT this PR's config (the committed metadata digest matches what master produces, not this branch): effectiveConfig in the persisted metadata now carries nudge.requireUsageSampleForPressure + nudge. maxEstimateHoldRounds, so every metadata digest moved again. Regenerated both golden sets with the local 0.0.185 kernel. 2. #4241 teeth: the fault dropped outbound[length-2] on the assumption the engine appends a trailing nudge as the final wire message. Since #5772 the snapshot render covers the previous turn's tail too (tailStart advanced), so length-2 lands INSIDE the exempt rendered prefix and the teeth never fire. The fault now drops the client's newest tail row (located by content) — the part the render lane actually owns byte-intact. Verified the same fix also passes on current master, where this test is broken since #5772. Note: storage-render/#4241/wire-contract/storage-engine-proofs also fail on master HEAD (run 42358603178) — #5772/#5616-era fallout, not this PR; the nudge e2e subset that master broke (#728A/C, #1119, #1137, zero-baseline) is fixed by this PR's backstop.
Closes #5756
What & why
#5744 made every save run one big synchronous SQLite transaction whose store leg scales with total session history; #5755 removed the dominant piece (refs ledger full-replace). This removes the remaining O(history) statement churn on the two view shapes that never qualify for the #3133 watermark delta (sliding-window resident tail, folded-resident envelopes): the completeLog historian sweep re-ran the full statement triple — tombstone UPDATE + message_media SELECT + DELETE (+release loop) — for every row outside the presented window on every save, even when the row was already retired and its media mappings were already empty. On a stable 3k-message session with a 160-row window that is ~3k×3 statements/turn of pure bookkeeping (issue measurement: ≈8900 of lane B's 9653 stmts/save).
Fix
ingestSession's completeLog sweep (SQLitesrc/storage/store.ts, MySQL twinsrc/storage/mysql.ts) now skips rows that are already retired: readers check only NULL/non-NULL, so there is nothing to do for them. First retirement still runs the full work (tombstone stamp + media mapping cleanup + ref release); the reactivation path (#4390/#4506 rewind-redo) is untouched — it only fires for rows whose identity IS in the presented set.Both lanes keep the single comparison read (
SELECT seq, msg_id, identity, retired_atover the session) that direction 1 of the issue asks for.Behavior-change disclosure
None observable. The tombstone UPDATE was always guarded by
AND retired_at IS NULL, so repeated sweeps never changed stored values — they only executed statements. After this change those statements stop executing for already-retired rows; every persisted value (retired_at timestamps, media mappings, refcounts) is byte-identical before/after. Net effect is purely CPU/statement-count reduction per steady-state save.Not in scope (parked as a decision request in the issue thread)
WAL_CHECKPOINT_EVERY_INGESTS=20) — product decision.Tests
New
tests/issue5756-lazy-sweep.test.ts: sliding-window steady-state corpus (W=120, K=4/turn) with a statement counter around each save — save 2 pays 3×N first-time-retirement statements; save 3 must not pay them again (assertion requires the gap to exceed half the N-triple cost, leaving a 2× cushion above adjacent-save drift). Sentinel tombstone stamps pin that pre-existing retires are left byte-identical; first-retirement media release, repeat-sweep stability, zero-drift saves, and rewind-redo reactivation (tombstones + image media refs restored) are all asserted.Negative control verified: with the store.ts fix stashed, the counter assertion fails (save2=1041 vs save3=1051 stmts — the steady-state save re-paid the retirement work); with the fix, the gap ≈ 3×N.
Adjacent targeted suites green locally on the rebased head (bfabcd1): tests/media-cas.test.ts, tests/unified-remap-collision.test.ts, tests/storage-ingest-watermark.test.ts (23/23), typecheck clean. The MySQL twin mirrors the same guard and rides the existing
storage-mysql-seam/storage-backend-mysql-e2eCI lanes. Full-suite verdict: remote CI.Config surface: none.