Skip to content

[PR] fix(storage): skip already-retired rows in completeLog historian sweep (#5756) - #5772

Merged
ranxianglei merged 1 commit into
masterfrom
2026-11-05_lazy-historian-sweep
Nov 7, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-11-05_lazy-historian-sweep

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #5756

What & why

#5744 made every save run one big synchronous SQLite transaction whose store leg scales with total session history; #5755 removed the dominant piece (refs ledger full-replace). This removes the remaining O(history) statement churn on the two view shapes that never qualify for the #3133 watermark delta (sliding-window resident tail, folded-resident envelopes): the completeLog historian sweep re-ran the full statement triple — tombstone UPDATE + message_media SELECT + DELETE (+release loop) — for every row outside the presented window on every save, even when the row was already retired and its media mappings were already empty. On a stable 3k-message session with a 160-row window that is ~3k×3 statements/turn of pure bookkeeping (issue measurement: ≈8900 of lane B's 9653 stmts/save).

Fix

ingestSession's completeLog sweep (SQLite src/storage/store.ts, MySQL twin src/storage/mysql.ts) now skips rows that are already retired: readers check only NULL/non-NULL, so there is nothing to do for them. First retirement still runs the full work (tombstone stamp + media mapping cleanup + ref release); the reactivation path (#4390/#4506 rewind-redo) is untouched — it only fires for rows whose identity IS in the presented set.

Both lanes keep the single comparison read (SELECT seq, msg_id, identity, retired_at over the session) that direction 1 of the issue asks for.

Behavior-change disclosure

None observable. The tombstone UPDATE was always guarded by AND retired_at IS NULL, so repeated sweeps never changed stored values — they only executed statements. After this change those statements stop executing for already-retired rows; every persisted value (retired_at timestamps, media mappings, refcounts) is byte-identical before/after. Net effect is purely CPU/statement-count reduction per steady-state save.

Not in scope (parked as a decision request in the issue thread)

Tests

New tests/issue5756-lazy-sweep.test.ts: sliding-window steady-state corpus (W=120, K=4/turn) with a statement counter around each save — save 2 pays 3×N first-time-retirement statements; save 3 must not pay them again (assertion requires the gap to exceed half the N-triple cost, leaving a 2× cushion above adjacent-save drift). Sentinel tombstone stamps pin that pre-existing retires are left byte-identical; first-retirement media release, repeat-sweep stability, zero-drift saves, and rewind-redo reactivation (tombstones + image media refs restored) are all asserted.

Negative control verified: with the store.ts fix stashed, the counter assertion fails (save2=1041 vs save3=1051 stmts — the steady-state save re-paid the retirement work); with the fix, the gap ≈ 3×N.

Adjacent targeted suites green locally on the rebased head (bfabcd1): tests/media-cas.test.ts, tests/unified-remap-collision.test.ts, tests/storage-ingest-watermark.test.ts (23/23), typecheck clean. The MySQL twin mirrors the same guard and rides the existing storage-mysql-seam / storage-backend-mysql-e2e CI lanes. Full-suite verdict: remote CI.

Config surface: none.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-05_lazy-historian-sweep (ff27ca0)

Latest build — install this exact version

npm install -g billion-context@0.0.8298-pr.5772.e1daa10

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
ff27ca0 billion-context@0.0.8298-pr.5772.e1daa10 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5772.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title fix(storage): skip already-retired rows in completeLog historian sweep (#5756) [PR] fix(storage): skip already-retired rows in completeLog historian sweep (#5756) Nov 5, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 是 #5756 的唯一在途修复(镜像 [PR] 清单核对:无并行 PR 指向同一线程);#5744/#5755/#3133 为背景/机制来源,非重复项。

标签: bug/scope/storage/p2(存储层稳态保存性能缺陷的修复;无客户端相关标签——纯存储层、宿主无关;可用但有明确性能损失,p2)。

独立审核结论(未复用 issue 侧表述,以下逐条为本轮重做):

先澄清一点:#5756 首楼说「同 PR 顺手修掉 #4793 remap 修复路径把 live 兄弟行退役时不释放其 media mappings」。我独立核对了当前 master:该缺陷不存在——两条车道的 leftovers SELECT(retired_at IS NOT NULL)都排在 retire UPDATE 之后执行,恰好把刚退役的行一并捞起,统一删行+删 media 映射+释放引用计数(src/storage/store.ts:1419-1446、src/storage/mysql.ts:1011-1032),既有回归钉「#4793: RETIRED duplicate holds the target bytes → dropped outright」通过。所以本 PR diff 里没有那个 commit 是对的,不缺东西:全部退役写点(sweep、remap-repair;fork-copy/import 两侧一致恢复)已满足「retire ⇒ 释放 media」,lazy 化所需前提结构性成立。

八问

  1. 是什么问题:视图形状永远进不了水位 delta 的长会话(滑动窗口驻留尾巴/folded-resident 信封),每轮全量保存仍对窗口外每一行跑完整三件套(tombstone UPDATE + media SELECT+DELETE+释放)——纯记账的 O(历史总量)语句 churn([perf] delta-fallback saves still sweep O(full history) per turn (sliding-window / folded-resident views) #5756 lane B 实测 ≈8900/9653 stmts/save)。
  2. 什么原因导致的:0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744 把每次保存收拢成单事务后,store 腿随历史总量 H 缩放;[PR] fix(storage): refs ledger diffs instead of full-replace per save (#5744) #5755 摘掉了 refs ledger 大头,但 historian sweep 没有「已退役即跳过」短路——tombstone UPDATE 本就带 AND retired_at IS NULL,复扫从不落库,只是白跑语句。
  3. 影响范围与严重程度:仅长会话稳态保存的 CPU/语句数;无数据与正确性影响。程度=性能劣化(p2),无可用性影响。
  4. 怎么修复的:sweep 循环对 retired_at 非空行加一行 skip(SQLite/MySQL 双车道对称各一处)——首次退役仍做全量工作(打戳+media 映射清理+引用释放);rewind-redo 复活分支在守卫之前、不受影响;两车道保留单一比较读。
  5. 是否有回归问题:逐项审过既有行为,全部保持——首次退役工作(测试钉住);复活路径(测试钉住:tombstone 复原+图片 media 引用复原为 1);tombstone 值因 IS NULL 守卫向来自稳定,改前后存储字节相同(旧代码的复扫本就是零匹配 no-op,PR 的「byte-identical」声明成立且比表面更强);media 计数等价(复扫命中空集、本来就没释放任何东西);lane 排除(Delta lane: native-mode clients send only deltas, server maintains full state + one-time fold, upstream receives snapsho | 增量车道:native 模式客户端只发增量,服务端全量维护+一次折叠,上游只收快照(#2926 的泛化) #3248)、delta 快路径(影子存储(默认开启)每次落盘把整个会话重新序列化+哈希,大会话明显变卡;jsonWriter=off 时连防抖合并都没了 #3133)、事务边界(0.1.212: per-turn ingest is still O(history) — 96.5% of proxy CPU in StatementSync.run, event loop starves with 8 concurrent sessions (#4528 / #5469 follow-up) #5744)、WAL cadence 均未触碰;无语义漂移/吞错/阈值漂移等高危模式。两车道守卫写法风格微差(!== null continue vs === null || === undefined 包裹)各自符合本文件局部惯例(mysql 文件他处同为双判),功能等价,不另开修。
  6. 是否破坏缓存:不涉及模型前缀缓存;存储水位/增量游标逻辑未动。
  7. 是否改了设计方向:无——既有 sweep 机制内的纯实现优化。方向 2/3 与 WAL 默认值仍按 [perf] delta-fallback saves still sweep O(full history) per turn (sliding-window / folded-resident views) #5756 线程的 🔶 决策请求挂起,不阻塞本 PR。
  8. 是否引入新配置:无。

测试有效性:真实 UnifiedStore + planIngest(非 mock);语句计数器包 db.prepare(sweep 三件套是逐行内联 prepare,prepare 计数≈执行计数,delta 度量成立);哨兵用 2001 年时间戳钉「既有 tombstone 字节级不被碰」;media 引用双向断言(首退释放→复扫稳定→复活还原)。负控我亲自复跑:注释掉 SQLite 守卫后测试如期失败(save2=925 vs save3=935,断言信息与作者报告一致),随后恢复原树——测试有牙,能抓住本 bug 回归。

验证证据:本地定向套件 23/23 绿(issue5756-lazy-sweep、media-cas、unified-remap-collision、storage-ingest-watermark)+ typecheck 干净;远程 CI ff27ca0 全绿(18 success / 12 skipped 为路径门控 / 0 fail,含覆盖 MySQL 孪生的 mysql-contract)。

diff 整洁度:3 文件 +175/-10,每行都与本修复相关,无跑题改动。

判定:可合入。

@ranxianglei ranxianglei added bug Something isn't working p2 scope/storage labels Nov 5, 2026

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

判定:APPROVE / 可合(head ff27ca005,base = master tip bfabcd1ca 零漂移;3 文件 +175/−10)。作者账号无法自批本 PR(GitHub 规则),按惯例以 COMMENT 事件承载评审。

结构化八问

  1. 是什么问题:两类永远够不到 #3133 watermark delta 的视图形状(滑动窗口 resident tail、folded-resident envelope)上,每回合的 save 都对全会话历史跑一遍 historian sweep——对窗口外每一行执行 tombstone UPDATE + media SELECT + DELETE(+release 循环),哪怕该行早已退役、media 映射早已清空。稳定 3k 消息 / 160 行窗口会话 ≈ 每 turn 3k×3 条纯记账语句(issue 实测 ≈8900 / 9653 stmts/save)。
  2. 什么原因导致的:sweep 设计为无条件执行——假设窗口外每行每次都需要(重新)盖墓碑;而 UPDATE 本身一直带 AND retired_at IS NULL 守卫,重复执行从来都是 no-op 写,语句成本却每次都付。#5744/#5755 拆掉了更大的 O(H) 件(refs ledger full-replace),本 PR 拆掉剩余的 O(H) churn。
  3. 影响范围与严重程度:纯性能面——bounded-view 长会话的稳态 save 延迟/语句数;不动数据、wire、配置、schema。严重度:性能劣化(长会话主线程卡顿贡献项),非可用性缺陷。
  4. 怎么修复的:sweep 跳过已退役行——SQLite src/storage/store.ts:968(if (row.retired_at !== null) continue;)与 MySQL 孪生 src/storage/mysql.ts:699-710(同判据包住三件套)。reader 只查 NULL/non-NULL,已退役行无事可做;首次退役仍付全量工作;tombstone 值、media 映射、refcount 全部不变。
  5. 是否有回归问题:逐项审过既有行为——(a) 首次退役语义不变(tombstone + media cleanup + ref release 照旧);(b) rewind-redo 再激活路径(#4390/#4506)未触碰,且在 skip 分支之前执行、只认 presented set 内的 identity,此类行永远不会被 skip 误伤;(c) 持久化值字节级不变(旧 UPDATE 本就 IS NULL 守卫不改值;唯一值读取点 src/export.ts:277 读的是存储值,从未变化);(d) lane 排除(hasLaneExt)与 delta watermark 路径未触碰;(e) 无 fail-fast→吞错、无阈值/时序/输出格式漂移。结论:无回归。
  6. 是否破坏缓存:不涉及(不碰 cache/retry/cursor/dedup key;WAL checkpoint cadence 明确 out of scope,留在 issue 线程)。
  7. 是否改了设计方向:否——同一存储架构,只是不再执行死语句;方向 2/3 按要求 park 在 issue 线程待决策。
  8. 是否引入新配置:无(PR 声明 Config surface: none,核实属实)。

验证证据(本地,rebased head ff27ca005)

  • 新回归 tests/issue5756-lazy-sweep.test.ts 通过;负控制实测:把 store.ts 还原为修复前版本,计数断言如预期失败(save2=925 vs save3=935——稳态 save 重付了首退成本),恢复后树干净;该 pin 有牙。
  • 相邻套件 media-cas / unified-remap-collision / storage-ingest-watermark 22/22 绿;tsc --noEmit clean。
  • CI(镜像缓存,head ff27ca005):30 checks = 18 success / 12 gated-skip / 0 failed / 0 pending。
  • 诚实边界:新 pin 在 SQLite 侧;MySQL 孪生的 skip 分支骑既有 storage-mysql-seam / storage-backend-mysql-e2e lane(seam stub 不产生 retired 行,e2e 在 DSN 在场时覆盖多 save 全路径但不断言语句数)——与 PR 前 #4390 retire 三件套同等覆盖水平,可接受。

§7.4 自动合并画像:单模块(storage)、纯 fix、回归测试带负控制、diff 干净、无 config/wire/kernel/persistence-format 触碰、Closes #5756 —— 满足自动合并条件;合并动作仍归人工。

无顺手修项(未发现小问题)。等 owner 合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷
(ai: vllm-qwen/qwen3.8-27b) Review lane 收工:APPROVE / 可合(head ff27ca0)。全 diff 通读(3 文件 +175/−10,store.ts:968 + mysql.ts:699-710 skip 已退役行)+ 本地定向验证:新回归通过、负控制实测(修复前代码在计数 pin 上如预期失败 save2=925 vs save3=935)、相邻三套件 22/22、tsc clean;CI 镜像缓存 30 checks = 18 success / 12 gated-skip / 0 failed。无回归(首次退役/rewind-redo 再激活/持久化值字节级不变逐项审过),无顺手修项,§7.4 自动合并画像满足,等 owner 人工合并。完整八问与证据见 PR 评审评论(review id 5791485201);协调行已报 #4599。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review(#5772,#5756 sweep lazy 化)— 本地验证完成,建议可合并(合并权在人)

改动

单 commit ff27ca005,核心一行:completeLog sweep 中已退役行 if (row.retired_at !== null) continue;(首次退役仍做完整 tombstone+媒体释放),MySQL 孪生同形包裹。lane B/C(hasLaneExt)排除逻辑未动。

语义审查(为什么行为完全不变、只省语句)

  1. 旧 UPDATE 本就带 AND retired_at IS NULL → 对已退役行匹配 0 行,重打戳从来是空操作;跳过只是省掉这次空执行。
  2. 被跳过的 tombstone 下不会长出新媒体:planner 不输出 retiredAt(ingest.ts 零匹配),incoming 缺席的行在本次 save 的消息 upsert 里根本不被写,新的 message_media 无从落下;seq 复用的换代先走 remap 的 delMedia+delRow+释放路径。
  3. rewind/undo 回来先于 skip:identity 重现走 :935-937 复活分支(SET retired_at = NULL)再继续,新测试专门钉了这条(rewind-redo 后 retiredCount 归 0、图片 refcount 回到 1)。
  4. reader 审计:全部消费者只判 NULL/非 NULL(store.ts 十处);export/import(:1899/:2007)与会话复制 INSERT..SELECT(:2902-2903)原值搬运永不重生成——sentinel 时间戳原样幸存,测试用 2001 年 sentinel 值直接证明未被触碰。

本地验证(node v25.9.0,分支 pr-5772-check @ ff27ca0)

  • 新测试 1/1 pass;相关批(fold-reconcile×3 + watermark-fastpath + bounded-storage)110/110 pass
  • 负向对照:去掉那行 continue → 失败于特征断言 steady-state save re-paid the first-time retirement work (save2=925 stmts, save3=935 stmts)(旧码每轮真把 304−W 行的退役重付一遍);恢复即绿
  • npm run typecheck exit 0
  • 测试质量:statement 计数器包住 prepare(直接测 issue 指标,非代理量);预算留 2× 余量不误伤;refcount 诚实性跨重复 sweep 钉住

备注(不阻塞)

  • #5744/#5756 家族的最后一块拼图到位后:#5755 的增量 ledger 写入 + 本 PR 的 sweep 跳过,steady-state 每轮同步 SQLite 工作量都应降到 O(diff)+O(新退役窗口)
  • 会话行 JSON blob 整写与 folded-resident 强制 full path 仍是独立后续(见各自 issue)

@ranxianglei
ranxianglei merged commit 5af5f9f into master Nov 7, 2026
82 of 117 checks passed
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.1071-master.5af5f9f (rolling master tag).

npm install -g billion-context@0.0.1071-master.5af5f9f  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei added a commit that referenced this pull request Nov 7, 2026
…ed-state shape (second staleness round)

The previous alignment targeted df0a402 (#5860/kernel-0.0.184 era). Master has
since merged #5772 (lazy-historian-sweep), which rewrote src/storage/{store,mysql}.ts
and again changed what the golden drive persists per session. Clean master's own
scheduled full matrix (run 42358603178, sha 5af5f9f, tracked as issue #5870) reds
on exactly the two suites reading these snapshots:

  - tests/golden-storage-render.test.ts          (4 wires, x3 shards)
  - tests/storage-engine-proofs.test.ts:191      full-mode restart vs committed
    goldens, assert.equal(record.stateDigestAligned, golden.stateDigestAligned)
    (4 wires, x3 shards)

Re-generated via scripts/update-storage-render-goldens.ts against clean
master@5af5f9fe. Safety check BEFORE trusting the regen: git diff -U0 over the
four snapshots shows ZERO turns[].sha lines touched (wire bodies byte-stable),
foldAt/editedAt unchanged, only stateDigest/stateDigestAligned/sectionDigests
.metadata values move — i.e. persistence-shape drift, not rendering behavior.

Local proof on /tmp/mfix (origin/master 5af5f9f, fresh npm ci + full build,
CI-parity pins BILI_MESSAGE_TIMESTAMPS/SNAPSHOT_SHADOW/IMAGE_RESCALE=0):
tests/golden-storage-render.test.ts + tests/storage-engine-proofs.test.ts =
17 pass / 0 fail after this commit (before it: the CI failure set reproduces).

This unblocks both master's :03/:33 scheduled matrix and every in-flight PR
whose merge-ref runs against live master.
ranxianglei added a commit that referenced this pull request Nov 7, 2026
…4241 teeth at the byte-promised tail

CI red on this PR had three causes:

1. storage-render + wire-contract goldens were regenerated against a kernel
   WITHOUT this PR's config (the committed metadata digest matches what
   master produces, not this branch): effectiveConfig in the persisted
   metadata now carries nudge.requireUsageSampleForPressure + nudge.
   maxEstimateHoldRounds, so every metadata digest moved again. Regenerated
   both golden sets with the local 0.0.185 kernel.

2. #4241 teeth: the fault dropped outbound[length-2] on the assumption the
   engine appends a trailing nudge as the final wire message. Since #5772
   the snapshot render covers the previous turn's tail too (tailStart
   advanced), so length-2 lands INSIDE the exempt rendered prefix and the
   teeth never fire. The fault now drops the client's newest tail row
   (located by content) — the part the render lane actually owns
   byte-intact. Verified the same fix also passes on current master, where
   this test is broken since #5772.

Note: storage-render/#4241/wire-contract/storage-engine-proofs also fail on
master HEAD (run 42358603178) — #5772/#5616-era fallout, not this PR; the
nudge e2e subset that master broke (#728A/C, #1119, #1137, zero-baseline)
is fixed by this PR's backstop.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[perf] delta-fallback saves still sweep O(full history) per turn (sliding-window / folded-resident views)

1 participant